Generated by Sphinx

Python Documentation
********************

1 What's New in Python
**********************

The "What's New in Python" series of essays takes tours through the most
important changes between major Python versions.  They are a "must
read" for anyone wishing to stay up-to-date after a new release.

1.1 What's New in Python 2.7
============================

     Author: A.M. Kuchling (amk at amk.ca)

     Release: 2.7.1

     Date: February 08, 2011

  This article explains the new features in Python 2.7.  Python 2.7 was
released on July 3, 2010.

  Numeric handling has been improved in many ways, for both
floating-point numbers and for the Decimal class.  There
are some useful additions to the standard library, such as a greatly
enhanced unittest module, the argparse module for
parsing command-line options, convenient OrderedDict.  and
Counter classes in the collections module, and
many other improvements.

  Python 2.7 is planned to be the last of the 2.x releases, so we worked
on making it a good release for the long term.  To help with porting to
Python 3, several new features from the Python 3.x series have been
included in 2.7.

  This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For full
details, you should refer to the documentation for Python 2.7 at
<http://docs.python.org>. If you want to understand the rationale for
the design and implementation, refer to the PEP for a particular new
feature or the issue on <http://bugs.python.org> in which a change was
discussed.  Whenever possible, "What's New in Python" links to the
bug/patch item for each change.

1.1.1 The Future for Python 2.x
-------------------------------

Python 2.7 is intended to be the last major release in the 2.x series.
The Python maintainers are planning to focus their future efforts on
the Python 3.x series.

  This means that 2.7 will remain in place for a long time, running
production systems that have not been ported to Python 3.x.  Two
consequences of the long-term significance of 2.7 are:

   * It's very likely the 2.7 release will have a longer period of
     maintenance compared to earlier 2.x versions.  Python 2.7 will
     continue to be maintained while the transition to 3.x continues,
     and the developers are planning to support Python 2.7 with bug-fix
     releases beyond the typical two years.

   * A policy decision was made to silence warnings only of interest to
     developers.  DeprecationWarning and its descendants
     are now ignored unless otherwise requested, preventing users from
     seeing warnings triggered by an application.  This change was also
     made in the branch that will become Python 3.2. (Discussed on
     stdlib-sig and carried out in issue 7319
     (http://bugs.python.org/issue7319).)

     In previous releases, see DeprecationWarning messages were
     enabled by default, providing Python developers with a clear
     indication of where their code may break in a future major version
     of Python.

     However, there are increasingly many users of Python-based
     applications who are not directly involved in the development of
     those applications.  DeprecationWarning messages are
     irrelevant to such users, making them worry about an application
     that's actually working correctly and burdening application
     developers with responding to these concerns.

     You can re-enable display of DeprecationWarning.
     messages by running Python with the -Wdefault. (short
     form: -Wd) switch, or by setting the
     PYTHONWARNINGS.  environment variable to `"default"' (or
     `"d"') before running Python.  Python code can also re-enable them
     by calling `warnings.simplefilter('default')'.

1.1.2 Python 3.1 Features
-------------------------

Much as Python 2.6 incorporated features from Python 3.0, version 2.7
incorporates some of the new features in Python 3.1.  The 2.x series
continues to provide tools for migrating to the 3.x series.

  A partial list of 3.1 features that were backported to 2.7:

   * The syntax for set literals (`{1,2,3}' is a mutable set).

   * Dictionary and set comprehensions (`{i: i*2 for i in range(3)}').

   * Multiple context managers in a single with statement.

   * A new version of the io library, rewritten in C for
     performance.

   * The ordered-dictionary type described in PEP 372; Adding an Ordered Dictionary to collections.

   * The new `","' format specifier described in PEP 378; Format Specifier for Thousands Separator.

   * The memoryview object.

   * A small subset of the importlib module, see described below.

   * The repr() of a float `x' is shorter in many cases:
     it's now based on the shortest decimal string that's guaranteed to
     round back to `x'.  As in previous versions of Python, it's
     guaranteed that `float(repr(x))' recovers `x'.

   * Float-to-string and string-to-float conversions are correctly
     rounded.  The round() function is also now correctly
     rounded.

   * The PyCapsule type, used to provide a C API for
     extension modules.

   * The PyLong_AsLongAndOverflow(). C API function.

  Other new Python3-mode warnings include:

   * operator.isCallable() and
     operator.sequenceIncludes(), which are not supported in 3.x,
     now trigger warnings.

   * The -3 switch now automatically enables the
     -Qwarn switch that causes warnings about using classic
     division with integers and long integers.

1.1.3 PEP 372: Adding an Ordered Dictionary to collections
----------------------------------------------------------

Regular Python dictionaries iterate over key/value pairs in arbitrary
order.  Over the years, a number of authors have written alternative
implementations that remember the order that the keys were originally
inserted.  Based on the experiences from those implementations, 2.7
introduces a new OrderedDict class in the
collections module.

  The OrderedDict. API provides the same interface as regular
dictionaries but iterates over keys and values in a guaranteed order
depending on when a key was first inserted:

    >>> from collections import OrderedDict
    >>> d = OrderedDict([('first', 1),
    ...                  ('second', 2),
    ...                  ('third', 3)])
    >>> d.items()
    [('first', 1), ('second', 2), ('third', 3)]

If a new entry overwrites an existing entry, the original insertion
position is left unchanged:

    >>> d['second'] = 4
    >>> d.items()
    [('first', 1), ('second', 4), ('third', 3)]

Deleting an entry and reinserting it will move it to the end:

    >>> del d['second']
    >>> d['second'] = 5
    >>> d.items()
    [('first', 1), ('third', 3), ('second', 5)]

The popitem() method has an optional _last_ argument that
defaults to True.  If _last_ is True, the most recently added key is
returned and removed; if it's False, the oldest key is selected:

    >>> od = OrderedDict([(x,0) for x in range(20)])
    >>> od.popitem()
    (19, 0)
    >>> od.popitem()
    (18, 0)
    >>> od.popitem(last=False)
    (0, 0)
    >>> od.popitem(last=False)
    (1, 0)

Comparing two ordered dictionaries checks both the keys and values, and
requires that the insertion order was the same:

    >>> od1 = OrderedDict([('first', 1),
    ...                    ('second', 2),
    ...                    ('third', 3)])
    >>> od2 = OrderedDict([('third', 3),
    ...                    ('first', 1),
    ...                    ('second', 2)])
    >>> od1 == od2
    False
    >>> # Move 'third' key to the end
    >>> del od2['third']; od2['third'] = 3
    >>> od1 == od2
    True

Comparing an OrderedDict with a regular dictionary ignores
the insertion order and just compares the keys and values.

  How does the OrderedDict work?  It maintains a
doubly-linked list of keys, appending new keys to the list as they're
inserted.  A secondary dictionary maps keys to their corresponding list
node, so deletion doesn't have to traverse the entire linked list and
therefore remains O(1).

  The standard library now supports use of ordered dictionaries in
several modules.

   * The ConfigParser module uses them by default, meaning
     that configuration files can now be read, modified, and then
     written back in their original order.

   * The _asdict() method for
     collections.namedtuple() now returns an ordered dictionary
     with the values appearing in the same order as the underlying
     tuple indices.

   * The json module's JSONDecoder class
     constructor was extended with an _object_pairs_hook_ parameter to
     allow `OrderedDict' instances to be built by the decoder.  Support
     was also added for third-party tools like PyYAML
     (http://pyyaml.org/).

PEP 372 (http://www.python.org/dev/peps/pep-0372) - Adding an ordered dictionary to collections
     PEP written by Armin Ronacher and Raymond Hettinger; implemented
     by Raymond Hettinger.

1.1.4 PEP 378: Format Specifier for Thousands Separator
-------------------------------------------------------

To make program output more readable, it can be useful to add
separators to large numbers, rendering them as
18,446,744,073,709,551,616 instead of 18446744073709551616.

  The fully general solution for doing this is the locale.
module, which can use different separators ("," in North America, "." in
Europe) and different grouping sizes, but locale is
complicated to use and unsuitable for multi-threaded applications where
different threads are producing output for different locales.

  Therefore, a simple comma-grouping mechanism has been added to the
mini-language used by the str.format() method.  When
formatting a floating-point number, simply include a comma between the
width and the precision:

    >>> '{:20,.2f}'.format(18446744073709551616.0)
    '18,446,744,073,709,551,616.00'

When formatting an integer, include the comma after the width:

    >>> '{:20,d}'.format(18446744073709551616)
    '18,446,744,073,709,551,616'

This mechanism is not adaptable at all; commas are always used as the
separator and the grouping is always into three-digit groups.  The
comma-formatting mechanism isn't as general as the locale.
module, but it's easier to use.

PEP 378 (http://www.python.org/dev/peps/pep-0378) - Format Specifier for Thousands Separator
     PEP written by Raymond Hettinger; implemented by Eric Smith.

1.1.5 PEP 389: The argparse Module for Parsing Command Lines
------------------------------------------------------------

The argparse module for parsing command-line arguments was
added as a more powerful replacement for the optparse.
module.

  This means Python now supports three different modules for parsing
command-line arguments: getopt, optparse, and
argparse.  The getopt module closely resembles the C
library's `getopt()' function, so it remains useful if you're writing a
Python prototype that will eventually be rewritten in C.
optparse becomes redundant, but there are no plans to remove it
because there are many scripts still using it, and there's no automated
way to update these scripts.  (Making the argparse.  API
consistent with optparse.'s interface was discussed but
rejected as too messy and difficult.)

  In short, if you're writing a new script and don't need to worry
about compatibility with earlier versions of Python, use
argparse instead of optparse.

  Here's an example:

    import argparse

    parser = argparse.ArgumentParser(description='Command-line example.')

    # Add optional switches
    parser.add_argument('-v', action='store_true', dest='is_verbose',
                        help='produce verbose output')
    parser.add_argument('-o', action='store', dest='output',
                        metavar='FILE',
                        help='direct output to FILE instead of stdout')
    parser.add_argument('-C', action='store', type=int, dest='context',
                        metavar='NUM', default=0,
                        help='display NUM lines of added context')

    # Allow any number of additional arguments.
    parser.add_argument(nargs='*', action='store', dest='inputs',
                        help='input filenames (default is stdin)')

    args = parser.parse_args()
    print args.__dict__

Unless you override it, see -h and -help switches
are automatically added, and produce neatly formatted output:

    -> ./python.exe argparse-example.py --help
    usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]]

    Command-line example.

    positional arguments:
      inputs      input filenames (default is stdin)

    optional arguments:
      -h, --help  show this help message and exit
      -v          produce verbose output
      -o FILE     direct output to FILE instead of stdout
      -C NUM      display NUM lines of added context

As with optparse, the command-line switches and arguments
are returned as an object with attributes named by the _dest_
parameters:

    -> ./python.exe argparse-example.py -v
    {'output': None,
     'is_verbose': True,
     'context': 0,
     'inputs': []}

    -> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2
    {'output': '/tmp/output',
     'is_verbose': True,
     'context': 4,
     'inputs': ['file1', 'file2']}

argparse has much fancier validation than optparse.;
you can specify an exact number of arguments as an integer, 0 or
more arguments by passing `'*'', 1 or more by passing `'+'', or an
optional argument with `'?''.  A top-level parser can contain
sub-parsers to define subcommands that have different sets of switches,
as in `svn commit', `svn checkout', etc.  You can specify an argument's
type as FileType, which will automatically open files for
you and understands that `'-'' means standard input or output.

argparse documentation
     The documentation page of the argparse module.

See Upgrading optparse code.
     Part of the Python documentation, describing how to convert code
     that uses optparse.

PEP 389 (http://www.python.org/dev/peps/pep-0389) - argparse - New Command Line Parsing Module
     PEP written and implemented by Steven Bethard.

1.1.6 PEP 391: Dictionary-Based Configuration For Logging
---------------------------------------------------------

The logging module is very flexible; applications can define
a tree of logging subsystems, and each logger in this tree can filter
out certain messages, format them differently, and direct messages to a
varying number of handlers.

  All this flexibility can require a lot of configuration.  You can
write Python statements to create objects and set their properties, but
a complex set-up requires verbose but boring code.  See logging.
also supports a fileConfig().  function that parses a file,
but the file format doesn't support configuring filters, and it's
messier to generate programmatically.

  Python 2.7 adds a dictConfig() function that uses a
dictionary to configure logging.  There are many ways to produce a
dictionary from different sources: construct one with code; parse a
file containing JSON; or use a YAML parsing library if one is
installed.  For more information see Configuration functions.

  The following example configures two loggers, the root logger and a
logger named "network".  Messages sent to the root logger will be sent
to the system log using the syslog protocol, and messages to the
"network" logger will be written to a `network.log' file that will be
rotated once the log reaches 1MB.

    import logging
    import logging.config

    configdict = {
     'version': 1,    # Configuration schema in use; must be 1 for now
     'formatters': {
         'standard': {
             'format': ('%(asctime)s %(name)-15s '
                        '%(levelname)-8s %(message)s')}},

     'handlers': {'netlog': {'backupCount': 10,
                         'class': 'logging.handlers.RotatingFileHandler',
                         'filename': '/logs/network.log',
                         'formatter': 'standard',
                         'level': 'INFO',
                         'maxBytes': 1000000},
                  'syslog': {'class': 'logging.handlers.SysLogHandler',
                             'formatter': 'standard',
                             'level': 'ERROR'}},

     # Specify all the subordinate loggers
     'loggers': {
                 'network': {
                             'handlers': ['netlog']
                 }
     },
     # Specify properties of the root logger
     'root': {
              'handlers': ['syslog']
     },
    }

    # Set up configuration
    logging.config.dictConfig(configdict)

    # As an example, log two error messages
    logger = logging.getLogger('/')
    logger.error('Database not found')

    netlogger = logging.getLogger('network')
    netlogger.error('Connection failed')

Three smaller enhancements to the logging module, all
implemented by Vinay Sajip, are:

   * The SysLogHandler class now supports syslogging over
     TCP.  The constructor has a _socktype_ parameter giving the type
     of socket to use, either socket.SOCK_DGRAM.  for UDP or
     socket.SOCK_STREAM for TCP.  The default protocol
     remains UDP.

   * `Logger' instances gained a getChild().  method that
     retrieves a descendant logger using a relative path.  For example,
     once you retrieve a logger by doing `log = getLogger('app')',
     calling `log.getChild('network.listen')' is equivalent to
     `getLogger('app.network.listen')'.

   * The LoggerAdapter class gained a `isEnabledFor()'
     method that takes a _level_ and returns whether the underlying
     logger would process a message of that level of importance.

PEP 391 (http://www.python.org/dev/peps/pep-0391) - Dictionary-Based Configuration For Logging
     PEP written and implemented by Vinay Sajip.

1.1.7 PEP 3106: Dictionary Views
--------------------------------

The dictionary methods keys(), values(), and
items() are different in Python 3.x.  They return an object
called a _view_ instead of a fully materialized list.

  It's not possible to change the return values of keys(),
values(), and items() in Python 2.7 because too
much code would break.  Instead the 3.x versions were added under the
new names viewkeys(), viewvalues(), and
viewitems().

    >>> d = dict((i*10, chr(65+i)) for i in range(26))
    >>> d
    {0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'}
    >>> d.viewkeys()
    dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250])

Views can be iterated over, but the key and item views also behave like
sets.  The `&' operator performs intersection, and `|' performs a union:

    >>> d1 = dict((i*10, chr(65+i)) for i in range(26))
    >>> d2 = dict((i**.5, i) for i in range(1000))
    >>> d1.viewkeys() & d2.viewkeys()
    set([0.0, 10.0, 20.0, 30.0])
    >>> d1.viewkeys() | range(0, 30)
    set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250])

The view keeps track of the dictionary and its contents change as the
dictionary is modified:

    >>> vk = d.viewkeys()
    >>> vk
    dict_keys([0, 130, 10, ..., 250])
    >>> d[260] = '&'
    >>> vk
    dict_keys([0, 130, 260, 10, ..., 250])

However, note that you can't add or remove keys while you're iterating
over the view:

    >>> for k in vk:
    ...     d[k*2] = k
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration

You can use the view methods in Python 2.x code, and the 2to3 converter
will change them to the standard keys(), values(),
and items() methods.

PEP 3106 (http://www.python.org/dev/peps/pep-3106) - Revamping dict.keys(), .values() and .items()
     PEP written by Guido van Rossum.  Backported to 2.7 by Alexandre
     Vassalotti; issue 1967 (http://bugs.python.org/issue1967).

1.1.8 PEP 3137: The memoryview Object
-------------------------------------

The memoryview object provides a view of another object's
memory content that matches the `bytes' type's interface.

    >>> import string
    >>> m = memoryview(string.letters)
    >>> m
    <memory at 0x37f850>
    >>> len(m)           # Returns length of underlying object
    52
    >>> m[0], m[25], m[26]   # Indexing returns one byte
    ('a', 'z', 'A')
    >>> m2 = m[0:26]         # Slicing returns another memoryview
    >>> m2
    <memory at 0x37f080>

The content of the view can be converted to a string of bytes or a list
of integers:

    >>> m2.tobytes()
    'abcdefghijklmnopqrstuvwxyz'
    >>> m2.tolist()
    [97, 98, 99, 100, 101, 102, 103, ... 121, 122]
    >>>

memoryview objects allow modifying the underlying object if
it's a mutable object.

    >>> m2[0] = 75
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: cannot modify read-only memory
    >>> b = bytearray(string.letters)  # Creating a mutable object
    >>> b
    bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')
    >>> mb = memoryview(b)
    >>> mb[0] = '*'         # Assign to view, changing the bytearray.
    >>> b[0:5]              # The bytearray has been changed.
    bytearray(b'*bcde')
    >>>

PEP 3137 (http://www.python.org/dev/peps/pep-3137) - Immutable Bytes and Mutable Buffer
     PEP written by Guido van Rossum.  Implemented by Travis Oliphant,
     Antoine Pitrou and others.  Backported to 2.7 by Antoine Pitrou;
     issue 2396 (http://bugs.python.org/issue2396).

1.1.9 Other Language Changes
----------------------------

Some smaller changes made to the core Python language are:

   * The syntax for set literals has been backported from Python 3.x.
     Curly brackets are used to surround the contents of the resulting
     mutable set; set literals are distinguished from dictionaries by
     not containing colons and values.  `{}' continues to represent an
     empty dictionary; use `set()' for an empty set.

         >>> {1, 2, 3, 4, 5}
         set([1, 2, 3, 4, 5])
         >>> set() # empty set
         set([])
         >>> {}    # empty dict
         {}

     Backported by Alexandre Vassalotti; issue 2335
     (http://bugs.python.org/issue2335).

   * Dictionary and set comprehensions are another feature backported
     from 3.x, generalizing list/generator comprehensions to use the
     literal syntax for sets and dictionaries.

         >>> {x: x*x for x in range(6)}
         {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}
         >>> {('a'*x) for x in range(6)}
         set(['', 'a', 'aa', 'aaa', 'aaaa', 'aaaaa'])

     Backported by Alexandre Vassalotti; issue 2333
     (http://bugs.python.org/issue2333).

   * The with statement can now use multiple context
     managers in one statement.  Context managers are processed from
     left to right and each one is treated as beginning a new
     with statement.  This means that:

         with A() as a, B() as b:
             ... suite of statements ...

     is equivalent to:

         with A() as a:
             with B() as b:
                 ... suite of statements ...

     The contextlib.nested() function provides a very
     similar function, so it's no longer necessary and has been
     deprecated.

     (Proposed in <http://codereview.appspot.com/53094>; implemented by
     Georg Brandl.)

   * Conversions between floating-point numbers and strings are now
     correctly rounded on most platforms.  These conversions occur in
     many different places: str() on floats and complex
     numbers; the float and complex.
     constructors; numeric formatting; serializing and deserializing
     floats and complex numbers using the marshal,
     pickle.  and json modules; parsing of float and
     imaginary literals in Python code; and Decimal.
     -to-float conversion.

     Related to this, the repr() of a floating-point number
     _x_ now returns a result based on the shortest decimal string
     that's guaranteed to round back to _x_ under correct rounding (with
     round-half-to-even rounding mode).  Previously it gave a string
     based on rounding x to 17 decimal digits.

     The rounding library responsible for this improvement works on
     Windows and on Unix platforms using the gcc, icc, or suncc
     compilers.  There may be a small number of platforms where correct
     operation of this code cannot be guaranteed, so the code is not
     used on such systems.  You can find out which code is being used
     by checking sys.float_repr_style,  which will be `short'
     if the new code is in use and `legacy' if it isn't.

     Implemented by Eric Smith and Mark Dickinson, using David Gay's
     `dtoa.c' library; issue 7117 (http://bugs.python.org/issue7117).

   * Conversions from long integers and regular integers to floating
     point now round differently, returning the floating-point number
     closest to the number.  This doesn't matter for small integers that
     can be converted exactly, but for large numbers that will
     unavoidably lose precision, Python 2.7 now approximates more
     closely.  For example, Python 2.6 computed the following:

         >>> n = 295147905179352891391
         >>> float(n)
         2.9514790517935283e+20
         >>> n - long(float(n))
         65535L

     Python 2.7's floating-point result is larger, but much closer to
     the true value:

         >>> n = 295147905179352891391
         >>> float(n)
         2.9514790517935289e+20
         >>> n - long(float(n))
         -1L

     (Implemented by Mark Dickinson; issue 3166
     (http://bugs.python.org/issue3166).)

     Integer division is also more accurate in its rounding behaviours.
     (Also implemented by Mark Dickinson; issue 1811
     (http://bugs.python.org/issue1811).)

   * Implicit coercion for complex numbers has been removed; the
     interpreter will no longer ever attempt to call a
     __coerce__() method on complex objects.  (Removed by Meador
     Inge and Mark Dickinson; issue 5211
     (http://bugs.python.org/issue5211).)

   * The str.format() method now supports automatic
     numbering of the replacement fields.  This makes using
     str.format() more closely resemble using `%s' formatting:

         >>> '{}:{}:{}'.format(2009, 04, 'Sunday')
         '2009:4:Sunday'
         >>> '{}:{}:{day}'.format(2009, 4, day='Sunday')
         '2009:4:Sunday'

     The auto-numbering takes the fields from left to right, so the
     first `{...}' specifier will use the first argument to
     str.format(), the next specifier will use the next argument,
     and so on.  You can't mix auto-numbering and explicit numbering -
     either number all of your specifier fields or none of them - but
     you can mix auto-numbering and named fields, as in the second
     example above.  (Contributed by Eric Smith; issue 5237
     (http://bugs.python.org/issue5237).)

     Complex numbers now correctly support usage with format(),
     and default to being right-aligned.  Specifying a precision
     or comma-separation applies to both the real and imaginary parts
     of the number, but a specified field width and alignment is
     applied to the whole of the resulting `1.5+3j' output.
     (Contributed by Eric Smith; issue 1588
     (http://bugs.python.org/issue1588) and issue 7988
     (http://bugs.python.org/issue7988).)

     The 'F' format code now always formats its output using uppercase
     characters, so it will now produce 'INF' and 'NAN'.  (Contributed
     by Eric Smith; issue 3382 (http://bugs.python.org/issue3382).)

     A low-level change: the `object.__format__()' method now triggers
     a PendingDeprecationWarning if it's passed a format
     string, because the `__format__()' method for object.
     converts the object to a string representation and formats that.
     Previously the method silently applied the format string to the
     string representation, but that could hide mistakes in Python
     code.  If you're supplying formatting information such as an
     alignment or precision, presumably you're expecting the formatting
     to be applied in some object-specific way.  (Fixed by Eric Smith;
     issue 7994 (http://bugs.python.org/issue7994).)

   * The int() and long() types gained a
     `bit_length' method that returns the number of bits necessary to
     represent its argument in binary:

         >>> n = 37
         >>> bin(n)
         '0b100101'
         >>> n.bit_length()
         6
         >>> n = 2**123-1
         >>> n.bit_length()
         123
         >>> (n+1).bit_length()
         124

     (Contributed by Fredrik Johansson and Victor Stinner; issue 3439
     (http://bugs.python.org/issue3439).)

   * The import statement will no longer try a relative
     import if an absolute import (e.g. `from .os import sep') fails.
     This fixes a bug, but could possibly break certain import.
     statements that were only working by accident.  (Fixed by
     Meador Inge; issue 7902 (http://bugs.python.org/issue7902).)

   * It's now possible for a subclass of the built-in unicode
     type to override the __unicode__() method.
     (Implemented by Victor Stinner; issue 1583863
     (http://bugs.python.org/issue1583863).)

   * The bytearray type's `translate()' method now accepts
     `None' as its first argument.  (Fixed by Georg Brandl; issue 4759
     (http://bugs.python.org/issue4759).)

   * When using `@classmethod' and `@staticmethod' to wrap methods as
     class or static methods, the wrapper object now exposes the
     wrapped function as their `__func__' attribute.  (Contributed by
     Amaury Forgeot d'Arc, after a suggestion by George Sakkis; issue
     5982 (http://bugs.python.org/issue5982).)

   * When a restricted set of attributes were set using `__slots__',
     deleting an unset attribute would not raise AttributeError.
     as you would expect.  Fixed by Benjamin Peterson; issue 7604
     (http://bugs.python.org/issue7604).)

   * Two new encodings are now supported: "cp720", used primarily for
     Arabic text; and "cp858", a variant of CP 850 that adds the euro
     symbol.  (CP720 contributed by Alexander Belchenko and Amaury
     Forgeot d'Arc in issue 1616979
     (http://bugs.python.org/issue1616979); CP858 contributed by Tim
     Hatch in issue 8016 (http://bugs.python.org/issue8016).)

   * The file object will now set the `filename' attribute
     on the IOError exception when trying to open a
     directory on POSIX platforms (noted by Jan Kaliszewski; issue 4764
     (http://bugs.python.org/issue4764)), and now explicitly checks for
     and forbids writing to read-only file objects instead of trusting
     the C library to catch and report the error (fixed by Stefan Krah;
     issue 5677 (http://bugs.python.org/issue5677)).

   * The Python tokenizer now translates line endings itself, so the
     compile() built-in function now accepts code using any
     line-ending convention.  Additionally, it no longer requires that
     the code end in a newline.

   * Extra parentheses in function definitions are illegal in Python
     3.x, meaning that you get a syntax error from `def f((x)): pass'.
     In Python3-warning mode, Python 2.7 will now warn about this odd
     usage.  (Noted by James Lingard; issue 7362
     (http://bugs.python.org/issue7362).)

   * It's now possible to create weak references to old-style class
     objects.  New-style classes were always weak-referenceable.  (Fixed
     by Antoine Pitrou; issue 8268 (http://bugs.python.org/issue8268).)

   * When a module object is garbage-collected, the module's dictionary
     is now only cleared if no one else is holding a reference to the
     dictionary (issue 7140 (http://bugs.python.org/issue7140)).

1.1.9.1 Interpreter Changes
...........................

A new environment variable, see PYTHONWARNINGS, allows
controlling warnings.  It should be set to a string containing warning
settings, equivalent to those used with the -W switch,
separated by commas.  (Contributed by Brian Curtin; issue 7301
(http://bugs.python.org/issue7301).)

  For example, the following setting will print warnings every time
they occur, but turn warnings from the Cookie module into an
error.  (The exact syntax for setting an environment variable varies
across operating systems and shells.)

    export PYTHONWARNINGS=all,error:::Cookie:0

1.1.9.2 Optimizations
.....................

Several performance enhancements have been added:

   * A new opcode was added to perform the initial setup for
     with statements, looking up the __enter__() and
     __exit__() methods.  (Contributed by Benjamin
     Peterson.)

   * The garbage collector now performs better for one common usage
     pattern: when many objects are being allocated without deallocating
     any of them.  This would previously take quadratic time for
     garbage collection, but now the number of full garbage collections
     is reduced as the number of objects on the heap grows.  The new
     logic only performs a full garbage collection pass when the middle
     generation has been collected 10 times and when the number of
     survivor objects from the middle generation exceeds 10% of the
     number of objects in the oldest generation.  (Suggested by Martin
     von Löwis and implemented by Antoine Pitrou; issue 4074
     (http://bugs.python.org/issue4074).)

   * The garbage collector tries to avoid tracking simple containers
     which can't be part of a cycle. In Python 2.7, this is now true for
     tuples and dicts containing atomic types (such as ints, strings,
     etc.). Transitively, a dict containing tuples of atomic types won't
     be tracked either. This helps reduce the cost of each garbage
     collection by decreasing the number of objects to be considered
     and traversed by the collector.  (Contributed by Antoine Pitrou;
     issue 4688 (http://bugs.python.org/issue4688).)

   * Long integers are now stored internally either in base 2**15 or in
     base 2**30, the base being determined at build time.  Previously,
     they were always stored in base 2**15.  Using base 2**30 gives
     significant performance improvements on 64-bit machines, but
     benchmark results on 32-bit machines have been mixed.  Therefore,
     the default is to use base 2**30 on 64-bit machines and base 2**15
     on 32-bit machines; on Unix, there's a new configure option
     `--enable-big-digits' that can be used to override this default.

     Apart from the performance improvements this change should be
     invisible to end users, with one exception: for testing and
     debugging purposes there's a new structseq sys.long_info
     that provides information about the internal format, giving
     the number of bits per digit and the size in bytes of the C type
     used to store each digit:

         >>> import sys
         >>> sys.long_info
         sys.long_info(bits_per_digit=30, sizeof_digit=4)

     (Contributed by Mark Dickinson; issue 4258
     (http://bugs.python.org/issue4258).)

     Another set of changes made long objects a few bytes smaller: 2
     bytes smaller on 32-bit systems and 6 bytes on 64-bit.
     (Contributed by Mark Dickinson; issue 5260
     (http://bugs.python.org/issue5260).)

   * The division algorithm for long integers has been made faster by
     tightening the inner loop, doing shifts instead of multiplications,
     and fixing an unnecessary extra iteration.  Various benchmarks
     show speedups of between 50% and 150% for long integer divisions
     and modulo operations.  (Contributed by Mark Dickinson; issue 5512
     (http://bugs.python.org/issue5512).)  Bitwise operations are also
     significantly faster (initial patch by Gregory Smith; issue
     1087418 (http://bugs.python.org/issue1087418)).

   * The implementation of `%' checks for the left-side operand being a
     Python string and special-cases it; this results in a 1-3%
     performance increase for applications that frequently use `%' with
     strings, such as templating libraries.  (Implemented by Collin
     Winter; issue 5176 (http://bugs.python.org/issue5176).)

   * List comprehensions with an `if' condition are compiled into
     faster bytecode.  (Patch by Antoine Pitrou, back-ported to 2.7 by
     Jeffrey Yasskin; issue 4715 (http://bugs.python.org/issue4715).)

   * Converting an integer or long integer to a decimal string was made
     faster by special-casing base 10 instead of using a generalized
     conversion function that supports arbitrary bases.  (Patch by
     Gawain Bolton; issue 6713 (http://bugs.python.org/issue6713).)

   * The `split()', `replace()', `rindex()', `rpartition()', and
     `rsplit()' methods of string-like types (strings, Unicode strings,
     and bytearray objects) now use a fast reverse-search
     algorithm instead of a character-by-character scan.  This is
     sometimes faster by a factor of 10.  (Added by Florent Xicluna;
     issue 7462 (http://bugs.python.org/issue7462) and issue 7622
     (http://bugs.python.org/issue7622).)

   * The pickle and cPickle modules now
     automatically intern the strings used for attribute names,
     reducing memory usage of the objects resulting from unpickling.
     (Contributed by Jake McGuire; issue 5084
     (http://bugs.python.org/issue5084).)

   * The cPickle module now special-cases dictionaries,
     nearly halving the time required to pickle them.  (Contributed by
     Collin Winter; issue 5670 (http://bugs.python.org/issue5670).)

1.1.10 New and Improved Modules
-------------------------------

As in every release, Python's standard library received a number of
enhancements and bug fixes.  Here's a partial list of the most notable
changes, sorted alphabetically by module name. Consult the `Misc/NEWS'
file in the source tree for a more complete list of changes, or look
through the Subversion logs for all the details.

   * The bdb module's base debugging class Bdb.
     gained a feature for skipping modules.  The constructor now takes
     an iterable containing glob-style patterns such as `django.*'; the
     debugger will not step into stack frames from a module that
     matches one of these patterns.  (Contributed by Maru Newby after a
     suggestion by Senthil Kumaran; issue 5142
     (http://bugs.python.org/issue5142).)

   * The binascii module now supports the buffer API, so it
     can be used with memoryview instances and other
     similar buffer objects.  (Backported from 3.x by Florent Xicluna;
     issue 7703 (http://bugs.python.org/issue7703).)

   * Updated module: the bsddb module has been updated from
     4.7.2devel9 to version 4.8.4 of the pybsddb package
     (http://www.jcea.es/programacion/pybsddb.htm).  The new version
     features better Python 3.x compatibility, various bug fixes, and
     adds several new BerkeleyDB flags and methods.  (Updated by Jesús
     Cea Avión; issue 8156 (http://bugs.python.org/issue8156).  The
     pybsddb changelog can be read at
     <http://hg.jcea.es/pybsddb/file/tip/ChangeLog>.)

   * The bz2 module's BZ2File now supports the
     context management protocol, so you can write `with
     bz2.BZ2File(...) as f:'.  (Contributed by Hagen Fürstenau; issue
     3860 (http://bugs.python.org/issue3860).)

   * New class: the Counter class in the collections.
     module is useful for tallying data.  See Counter.
     instances behave mostly like dictionaries but return zero for
     missing keys instead of raising a KeyError.:

         >>> from collections import Counter
         >>> c = Counter()
         >>> for letter in 'here is a sample of english text':
         ...   c[letter] += 1
         ...
         >>> c
         Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2,
         'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1,
         'p': 1, 'r': 1, 'x': 1})
         >>> c['e']
         5
         >>> c['z']
         0

     There are three additional Counter methods.
     most_common() returns the N most common elements and their
     counts.  See elements().  returns an iterator over the
     contained elements, repeating each element as many times as its
     count.  subtract() takes an iterable and subtracts one
     for each element instead of adding; if the argument is a
     dictionary or another `Counter', the counts are subtracted.

         >>> c.most_common(5)
         [(' ', 6), ('e', 5), ('s', 3), ('a', 2), ('i', 2)]
         >>> c.elements() ->
            'a', 'a', ' ', ' ', ' ', ' ', ' ', ' ',
            'e', 'e', 'e', 'e', 'e', 'g', 'f', 'i', 'i',
            'h', 'h', 'm', 'l', 'l', 'o', 'n', 'p', 's',
            's', 's', 'r', 't', 't', 'x'
         >>> c['e']
         5
         >>> c.subtract('very heavy on the letter e')
         >>> c['e']    # Count is now lower
         -1

     Contributed by Raymond Hettinger; issue 1696199
     (http://bugs.python.org/issue1696199).

     New class: OrderedDict is described in the earlier
     section PEP 372; Adding an Ordered Dictionary to collections.

     New method: The deque data type now has a
     count() method that returns the number of contained elements
     equal to the supplied argument _x_, and a reverse().
     method that reverses the elements of the deque in-place.
     deque also exposes its maximum length as the read-only
     maxlen attribute.  (Both features added by Raymond
     Hettinger.)

     The namedtuple class now has an optional _rename_
     parameter.  If _rename_ is true, field names that are invalid
     because they've been repeated or aren't legal Python identifiers
     will be renamed to legal names that are derived from the field's
     position within the list of fields:

         >>> from collections import namedtuple
         >>> T = namedtuple('T', ['field1', '$illegal', 'for', 'field2'], rename=True)
         >>> T._fields
         ('field1', '_1', '_2', 'field2')

     (Added by Raymond Hettinger; issue 1818
     (http://bugs.python.org/issue1818).)

     Finally, the `Mapping' abstract base class now returns
     NotImplemented if a mapping is compared to another type that
     isn't a `Mapping'.  (Fixed by Daniel Stutzbach; issue 8729
     (http://bugs.python.org/issue8729).)

   * Constructors for the parsing classes in the ConfigParser
     module now take a _allow_no_value_ parameter, defaulting to
     false; if true, options without values will be allowed.  For
     example:

         >>> import ConfigParser, StringIO
         >>> sample_config = """
         ... [mysqld]
         ... user = mysql
         ... pid-file = /var/run/mysqld/mysqld.pid
         ... skip-bdb
         ... """
         >>> config = ConfigParser.RawConfigParser(allow_no_value=True)
         >>> config.readfp(StringIO.StringIO(sample_config))
         >>> config.get('mysqld', 'user')
         'mysql'
         >>> print config.get('mysqld', 'skip-bdb')
         None
         >>> print config.get('mysqld', 'unknown')
         Traceback (most recent call last):
           ...
         NoOptionError: No option 'unknown' in section: 'mysqld'

     (Contributed by Mats Kindahl; issue 7005
     (http://bugs.python.org/issue7005).)

   * Deprecated function: contextlib.nested(), which allows
     handling more than one context manager with a single with.
     statement, has been deprecated, because the with.
     statement now supports multiple context managers.

   * The cookielib module now ignores cookies that have an
     invalid version field, one that doesn't contain an integer value.
     (Fixed by John J. Lee; issue 3924
     (http://bugs.python.org/issue3924).)

   * The copy module's deepcopy() function will
     now correctly copy bound instance methods.  (Implemented by Robert
     Collins; issue 1515 (http://bugs.python.org/issue1515).)

   * The ctypes module now always converts `None' to a C NULL
     pointer for arguments declared as pointers.  (Changed by Thomas
     Heller; issue 4606 (http://bugs.python.org/issue4606).)  The
     underlying libffi library (http://sourceware.org/libffi/) has been
     updated to version 3.0.9, containing various fixes for different
     platforms.  (Updated by Matthias Klose; issue 8142
     (http://bugs.python.org/issue8142).)

   * New method: the datetime module's timedelta.
     class gained a total_seconds() method that returns the
     number of seconds in the duration.  (Contributed by Brian Quinlan;
     issue 5788 (http://bugs.python.org/issue5788).)

   * New method: the Decimal class gained a
     from_float() class method that performs an exact conversion
     of a floating-point number to a Decimal.  This exact
     conversion strives for the closest decimal approximation to the
     floating-point representation's value; the resulting decimal value
     will therefore still include the inaccuracy, if any.  For example,
     `Decimal.from_float(0.1)' returns
     `Decimal('0.1000000000000000055511151231257827021181583404541015625')'.
     (Implemented by Raymond Hettinger; issue 4796
     (http://bugs.python.org/issue4796).)

     Comparing instances of Decimal with floating-point
     numbers now produces sensible results based on the numeric values
     of the operands.  Previously such comparisons would fall back to
     Python's default rules for comparing objects, which produced
     arbitrary results based on their type.  Note that you still cannot
     combine `Decimal' and floating-point in other operations such as
     addition, since you should be explicitly choosing how to convert
     between float and Decimal.  (Fixed by Mark Dickinson;
     issue 2531 (http://bugs.python.org/issue2531).)

     The constructor for Decimal now accepts floating-point
     numbers (added by Raymond Hettinger; issue 8257
     (http://bugs.python.org/issue8257)) and non-European Unicode
     characters such as Arabic-Indic digits (contributed by Mark
     Dickinson; issue 6595 (http://bugs.python.org/issue6595)).

     Most of the methods of the Context class now accept
     integers as well as Decimal instances; the only
     exceptions are the canonical() and
     is_canonical().  methods.  (Patch by Juan José Conti; issue
     7633 (http://bugs.python.org/issue7633).)

     When using Decimal instances with a string's
     format() method, the default alignment was previously
     left-alignment.  This has been changed to right-alignment, which is
     more sensible for numeric types.  (Changed by Mark Dickinson;
     issue 6857 (http://bugs.python.org/issue6857).)

     Comparisons involving a signaling NaN value (or `sNAN') now signal
     `InvalidOperation' instead of silently returning a true or false
     value depending on the comparison operator.  Quiet NaN values (or
     `NaN') are now hashable.  (Fixed by Mark Dickinson; issue 7279
     (http://bugs.python.org/issue7279).)

   * The difflib module now produces output that is more
     compatible with modern *diff*/*patch* tools through one small
     change, using a tab character instead of spaces as a separator in
     the header giving the filename.  (Fixed by Anatoly Techtonik;
     issue 7585 (http://bugs.python.org/issue7585).)

   * The Distutils `sdist' command now always regenerates the
     `MANIFEST' file, since even if the `MANIFEST.in' or `setup.py'
     files haven't been modified, the user might have created some new
     files that should be included.  (Fixed by Tarek Ziadé; issue 8688
     (http://bugs.python.org/issue8688).)

   * The doctest module's `IGNORE_EXCEPTION_DETAIL' flag
     will now ignore the name of the module containing the exception
     being tested.  (Patch by Lennart Regebro; issue 7490
     (http://bugs.python.org/issue7490).)

   * The email module's Message class will now
     accept a Unicode-valued payload, automatically converting the
     payload to the encoding specified by `output_charset'.  (Added by
     R. David Murray; issue 1368247
     (http://bugs.python.org/issue1368247).)

   * The Fraction class now accepts a single float or
     Decimal instance, or two rational numbers, as arguments to
     its constructor.  (Implemented by Mark Dickinson; rationals added
     in issue 5812 (http://bugs.python.org/issue5812), and
     float/decimal in issue 8294 (http://bugs.python.org/issue8294).)

     Ordering comparisons (`<', `<=', `>', `>=') between fractions and
     complex numbers now raise a TypeError.  This fixes an
     oversight, making the Fraction.  match the other
     numeric types.

   * New class: FTP_TLS in the ftplib module
     provides secure FTP connections using TLS encapsulation of
     authentication as well as subsequent control and data transfers.
     (Contributed by Giampaolo Rodola; issue 2054
     (http://bugs.python.org/issue2054).)

     The storbinary() method for binary uploads can now
     restart uploads thanks to an added _rest_ parameter (patch by
     Pablo Mouzo; issue 6845 (http://bugs.python.org/issue6845).)

   * New class decorator: total_ordering() in the
     functools.  module takes a class that defines an
     __eq__() method and one of __lt__(),
     __le__(), __gt__(), or __ge__(), and
     generates the missing comparison methods.  Since the
     __cmp__() method is being deprecated in Python 3.x, this
     decorator makes it easier to define ordered classes.  (Added by
     Raymond Hettinger; issue 5479 (http://bugs.python.org/issue5479).)

     New function: cmp_to_key() will take an old-style
     comparison function that expects two arguments and return a new
     callable that can be used as the _key_ parameter to functions such
     as sorted(), min() and max(),
     etc.  The primary intended use is to help with making code
     compatible with Python 3.x.  (Added by Raymond Hettinger.)

   * New function: the gc module's is_tracked().
     returns true if a given instance is tracked by the garbage
     collector, false otherwise. (Contributed by Antoine Pitrou; issue
     4688 (http://bugs.python.org/issue4688).)

   * The gzip module's GzipFile now supports the
     context management protocol, so you can write `with
     gzip.GzipFile(...) as f:' (contributed by Hagen Fürstenau; issue
     3860 (http://bugs.python.org/issue3860)), and it now implements
     the io.BufferedIOBase. ABC, so you can wrap it with
     io.BufferedReader for faster processing (contributed
     by Nir Aides; issue 7471 (http://bugs.python.org/issue7471)).
     It's also now possible to override the modification time recorded
     in a gzipped file by providing an optional timestamp to the
     constructor.  (Contributed by Jacques Frechet; issue 4272
     (http://bugs.python.org/issue4272).)

     Files in gzip format can be padded with trailing zero bytes; the
     gzip module will now consume these trailing bytes.
     (Fixed by Tadek Pietraszek and Brian Curtin; issue 2846
     (http://bugs.python.org/issue2846).)

   * New attribute: the hashlib module now has an
     algorithms.  attribute containing a tuple naming the
     supported algorithms.  In Python 2.7, `hashlib.algorithms' contains
     `('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')'.
     (Contributed by Carl Chenet; issue 7418
     (http://bugs.python.org/issue7418).)

   * The default HTTPResponse class used by the
     httplib module now supports buffering, resulting in much
     faster reading of HTTP responses.  (Contributed by Kristján Valur
     Jónsson; issue 4879 (http://bugs.python.org/issue4879).)

     The HTTPConnection and HTTPSConnection.
     classes now support a _source_address_ parameter, a `(host, port)'
     2-tuple giving the source address that will be used for the
     connection.  (Contributed by Eldon Ziegler; issue 3972
     (http://bugs.python.org/issue3972).)

   * The `ihooks' module now supports relative imports.  Note that
     `ihooks' is an older module for customizing imports, superseded by
     the imputil module added in Python 2.0.  (Relative
     import support added by Neil Schemenauer.)

   * The imaplib module now supports IPv6 addresses.
     (Contributed by Derek Morr; issue 1655
     (http://bugs.python.org/issue1655).)

   * New function: the inspect module's getcallargs().
     takes a callable and its positional and keyword arguments,
     and figures out which of the callable's parameters will receive
     each argument, returning a dictionary mapping argument names to
     their values.  For example:

         >>> from inspect import getcallargs
         >>> def f(a, b=1, *pos, **named):
         ...     pass
         >>> getcallargs(f, 1, 2, 3)
         {'a': 1, 'b': 2, 'pos': (3,), 'named': {}}
         >>> getcallargs(f, a=2, x=4)
         {'a': 2, 'b': 1, 'pos': (), 'named': {'x': 4}}
         >>> getcallargs(f)
         Traceback (most recent call last):
         ...
         TypeError: f() takes at least 1 argument (0 given)

     Contributed by George Sakkis; issue 3135
     (http://bugs.python.org/issue3135).

   * Updated module: The io library has been upgraded to the
     version shipped with Python 3.1.  For 3.1, the I/O library was
     entirely rewritten in C and is 2 to 20 times faster depending on
     the task being performed.  The original Python version was renamed
     to the `_pyio' module.

     One minor resulting change: the io.TextIOBase class now
     has an `errors' attribute giving the error setting used for
     encoding and decoding errors (one of `'strict'', `'replace'',
     `'ignore'').

     The io.FileIO class now raises an OSError.
     when passed an invalid file descriptor.  (Implemented by Benjamin
     Peterson; issue 4991 (http://bugs.python.org/issue4991).)  The
     truncate() method now preserves the file position;
     previously it would change the file position to the end of the new
     file.  (Fixed by Pascal Chambon; issue 6939
     (http://bugs.python.org/issue6939).)

   * New function: `itertools.compress(data, selectors)' takes two
     iterators.  Elements of _data_ are returned if the corresponding
     value in _selectors_ is true:

         itertools.compress('ABCDEF', [1,0,1,0,1,1]) =>
           A, C, E, F

     New function: `itertools.combinations_with_replacement(iter, r)'
     returns all the possible _r_-length combinations of elements from
     the iterable _iter_.  Unlike combinations(), individual
     elements can be repeated in the generated combinations:

         itertools.combinations_with_replacement('abc', 2) =>
           ('a', 'a'), ('a', 'b'), ('a', 'c'),
           ('b', 'b'), ('b', 'c'), ('c', 'c')

     Note that elements are treated as unique depending on their
     position in the input, not their actual values.

     The itertools.count() function now has a _step_
     argument that allows incrementing by values other than 1.
     count() also now allows keyword arguments, and using
     non-integer values such as floats or Decimal.
     instances.  (Implemented by Raymond Hettinger; issue 5032
     (http://bugs.python.org/issue5032).)

     itertools.combinations() and
     itertools.product().  previously raised ValueError
     for values of _r_ larger than the input iterable.  This was
     deemed a specification error, so they now return an empty
     iterator.  (Fixed by Raymond Hettinger; issue 4816
     (http://bugs.python.org/issue4816).)

   * Updated module: The json module was upgraded to version
     2.0.9 of the simplejson package, which includes a C extension that
     makes encoding and decoding faster.  (Contributed by Bob Ippolito;
     issue 4136 (http://bugs.python.org/issue4136).)

     To support the new collections.OrderedDict type, see
     json.load().  now has an optional _object_pairs_hook_
     parameter that will be called with any object literal that decodes
     to a list of pairs.  (Contributed by Raymond Hettinger; issue 5381
     (http://bugs.python.org/issue5381).)

   * The mailbox module's Maildir class now
     records the timestamp on the directories it reads, and only
     re-reads them if the modification time has subsequently changed.
     This improves performance by avoiding unneeded directory scans.
     (Fixed by A.M. Kuchling and Antoine Pitrou; issue 1607951
     (http://bugs.python.org/issue1607951), issue 6896
     (http://bugs.python.org/issue6896).)

   * New functions: the math module gained erf()
     and erfc() for the error function and the
     complementary error function, see expm1() which computes
     `e**x - 1' with more precision than using exp() and
     subtracting 1, see gamma() for the Gamma function, and
     lgamma() for the natural log of the Gamma function.
     (Contributed by Mark Dickinson and nirinA raseliarison; issue 3366
     (http://bugs.python.org/issue3366).)

   * The multiprocessing module's `Manager*' classes can
     now be passed a callable that will be called whenever a subprocess
     is started, along with a set of arguments that will be passed to
     the callable.  (Contributed by lekma; issue 5585
     (http://bugs.python.org/issue5585).)

     The `Pool' class, which controls a pool of worker processes, now
     has an optional _maxtasksperchild_ parameter.  Worker processes
     will perform the specified number of tasks and then exit, causing
     the `Pool' to start a new worker.  This is useful if tasks may leak
     memory or other resources, or if some tasks will cause the worker
     to become very large.  (Contributed by Charles Cazabon; issue 6963
     (http://bugs.python.org/issue6963).)

   * The nntplib module now supports IPv6 addresses.
     (Contributed by Derek Morr; issue 1664
     (http://bugs.python.org/issue1664).)

   * New functions: the os module wraps the following POSIX
     system calls: getresgid() and getresuid(),
     which return the real, effective, and saved GIDs and UIDs; see
     setresgid() and setresuid(), which set real,
     effective, and saved GIDs and UIDs to new values; see
     initgroups(), which initialize the group access list for the
     current process.  (GID/UID functions contributed by Travis H.;
     issue 6508 (http://bugs.python.org/issue6508).  Support for
     initgroups added by Jean-Paul Calderone; issue 7333
     (http://bugs.python.org/issue7333).)

     The os.fork() function now re-initializes the import
     lock in the child process; this fixes problems on Solaris when
     fork().  is called from a thread.  (Fixed by Zsolt
     Cserna; issue 7242 (http://bugs.python.org/issue7242).)

   * In the os.path module, the normpath() and
     abspath() functions now preserve Unicode; if their
     input path is a Unicode string, the return value is also a Unicode
     string.  (see normpath() fixed by Matt Giuca in issue 5827
     (http://bugs.python.org/issue5827); see abspath() fixed by
     Ezio Melotti in issue 3426 (http://bugs.python.org/issue3426).)

   * The pydoc module now has help for the various symbols
     that Python uses.  You can now do `help('<<')' or `help('@')', for
     example.  (Contributed by David Laban; issue 4739
     (http://bugs.python.org/issue4739).)

   * The re module's split(), sub(),
     and subn().  now accept an optional _flags_ argument,
     for consistency with the other functions in the module.  (Added by
     Gregory P. Smith.)

   * New function: run_path() in the runpy.
     module will execute the code at a provided _path_ argument.
     _path_ can be the path of a Python source file (`example.py'), a
     compiled bytecode file (`example.pyc'), a directory
     (`./package/'), or a zip archive (`example.zip').  If a directory
     or zip path is provided, it will be added to the front of
     `sys.path' and the module __main__ will be imported.
     It's expected that the directory or zip contains a `__main__.py';
     if it doesn't, some other `__main__.py' might be imported from a
     location later in `sys.path'.  This makes more of the machinery of
     runpy available to scripts that want to mimic the way
     Python's command line processes an explicit path name.  (Added by
     Nick Coghlan; issue 6816 (http://bugs.python.org/issue6816).)

   * New function: in the shutil module, see
     make_archive().  takes a filename, archive type (zip or
     tar-format), and a directory path, and creates an archive
     containing the directory's contents.  (Added by Tarek Ziadé.)

     See shutil.'s copyfile() and copytree().
     functions now raise a `SpecialFileError' exception when
     asked to copy a named pipe.  Previously the code would treat named
     pipes like a regular file by opening them for reading, and this
     would block indefinitely.  (Fixed by Antoine Pitrou; issue 3002
     (http://bugs.python.org/issue3002).)

   * The signal module no longer re-installs the signal
     handler unless this is truly necessary, which fixes a bug that
     could make it impossible to catch the EINTR signal robustly.
     (Fixed by Charles-Francois Natali; issue 8354
     (http://bugs.python.org/issue8354).)

   * New functions: in the site module, three new functions
     return various site- and user-specific paths.
     getsitepackages() returns a list containing all global
     site-packages directories, see getusersitepackages().
     returns the path of the user's site-packages directory, and
     getuserbase() returns the value of the `USER_BASE'
     environment variable, giving the path to a directory that can be
     used to store data.  (Contributed by Tarek Ziadé; issue 6693
     (http://bugs.python.org/issue6693).)

     The site module now reports exceptions occurring when
     the `sitecustomize' module is imported, and will no longer catch
     and swallow the KeyboardInterrupt exception.  (Fixed by
     Victor Stinner; issue 3137 (http://bugs.python.org/issue3137).)

   * The create_connection() function gained a
     _source_address_ parameter, a `(host, port)' 2-tuple giving the
     source address that will be used for the connection.  (Contributed
     by Eldon Ziegler; issue 3972 (http://bugs.python.org/issue3972).)

     The recv_into() and recvfrom_into().
     methods will now write into objects that support the buffer API,
     most usefully the bytearray and memoryview.
     objects.  (Implemented by Antoine Pitrou; issue 8104
     (http://bugs.python.org/issue8104).)

   * The SocketServer module's `TCPServer' class now
     supports socket timeouts and disabling the Nagle algorithm.  The
     `disable_nagle_algorithm' class attribute defaults to False; if
     overridden to be True, new request connections will have the
     TCP_NODELAY option set to prevent buffering many small sends into
     a single TCP packet.  The timeout class attribute can
     hold a timeout in seconds that will be applied to the request
     socket; if no request is received within that time, see
     handle_timeout().  will be called and handle_request()
     will return.  (Contributed by Kristján Valur Jónsson; issue
     6192 (http://bugs.python.org/issue6192) and issue 6267
     (http://bugs.python.org/issue6267).)

   * Updated module: the sqlite3 module has been updated to
     version 2.6.0 of the pysqlite package
     (http://code.google.com/p/pysqlite/). Version 2.6.0 includes a
     number of bugfixes, and adds the ability to load SQLite extensions
     from shared libraries.  Call the `enable_load_extension(True)'
     method to enable extensions, and then call load_extension()
     to load a particular shared library.  (Updated by Gerhard
     Häring.)

   * The ssl module's `SSLSocket' objects now support the
     buffer API, which fixed a test suite failure (fix by Antoine
     Pitrou; issue 7133 (http://bugs.python.org/issue7133)) and
     automatically set OpenSSL's `SSL_MODE_AUTO_RETRY', which will
     prevent an error code being returned from `recv()' operations that
     trigger an SSL renegotiation (fix by Antoine Pitrou; issue 8222
     (http://bugs.python.org/issue8222)).

     The ssl.wrap_socket() constructor function now takes a
     _ciphers_ argument that's a string listing the encryption
     algorithms to be allowed; the format of the string is described in
     the OpenSSL documentation
     (http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT).
     (Added by Antoine Pitrou; issue 8322
     (http://bugs.python.org/issue8322).)

     Another change makes the extension load all of OpenSSL's ciphers
     and digest algorithms so that they're all available.  Some SSL
     certificates couldn't be verified, reporting an "unknown algorithm"
     error.  (Reported by Beda Kosata, and fixed by Antoine Pitrou;
     issue 8484 (http://bugs.python.org/issue8484).)

     The version of OpenSSL being used is now available as the module
     attributes ssl.OPENSSL_VERSION. (a string), see
     ssl.OPENSSL_VERSION_INFO. (a 5-tuple), and
     ssl.OPENSSL_VERSION_NUMBER. (an integer).  (Added by Antoine
     Pitrou; issue 8321 (http://bugs.python.org/issue8321).)

   * The struct module will no longer silently ignore
     overflow errors when a value is too large for a particular integer
     format code (one of `bBhHiIlLqQ'); it now always raises a
     struct.error exception.  (Changed by Mark Dickinson; issue
     1523 (http://bugs.python.org/issue1523).)  The pack().
     function will also attempt to use __index__() to
     convert and pack non-integers before trying the __int__()
     method or reporting an error.  (Changed by Mark Dickinson;
     issue 8300 (http://bugs.python.org/issue8300).)

   * New function: the subprocess module's
     check_output() runs a command with a specified set of
     arguments and returns the command's output as a string when the
     command runs without error, or raises a `CalledProcessError'
     exception otherwise.

         >>> subprocess.check_output(['df', '-h', '.'])
         'Filesystem     Size   Used  Avail Capacity  Mounted on\n
         /dev/disk0s2    52G    49G   3.0G    94%    /\n'

         >>> subprocess.check_output(['df', '-h', '/bogus'])
           ...
         subprocess.CalledProcessError: Command '['df', '-h', '/bogus']' returned non-zero exit status 1

     (Contributed by Gregory P. Smith.)

     The subprocess module will now retry its internal
     system calls on receiving an `EINTR' signal.  (Reported by several
     people; final patch by Gregory P. Smith in issue 1068268
     (http://bugs.python.org/issue1068268).)

   * New function: is_declared_global() in the
     symtable module returns true for variables that are
     explicitly declared to be global, false for ones that are
     implicitly global.  (Contributed by Jeremy Hylton.)

   * The syslog module will now use the value of
     `sys.argv[0]' as the identifier instead of the previous default
     value of `'python''.  (Changed by Sean Reifschneider; issue 8451
     (http://bugs.python.org/issue8451).)

   * The `sys.version_info' value is now a named tuple, with attributes
     named `major', `minor', `micro', `releaselevel', and `serial'.
     (Contributed by Ross Light; issue 4285
     (http://bugs.python.org/issue4285).)

     sys.getwindowsversion() also returns a named tuple,
     with attributes named `major', `minor', `build', see platform,
     `service_pack', `service_pack_major', `service_pack_minor',
     `suite_mask', and `product_type'.  (Contributed by Brian Curtin;
     issue 7766 (http://bugs.python.org/issue7766).)

   * The tarfile module's default error handling has
     changed, to no longer suppress fatal errors.  The default error
     level was previously 0, which meant that errors would only result
     in a message being written to the debug log, but because the debug
     log is not activated by default, these errors go unnoticed.  The
     default error level is now 1, which raises an exception if there's
     an error.  (Changed by Lars Gustäbel; issue 7357
     (http://bugs.python.org/issue7357).)

     tarfile now supports filtering the TarInfo.
     objects being added to a tar file.  When you call add(),
     you may supply an optional _filter_ argument that's a callable.
     The _filter_ callable will be passed the TarInfo for
     every file being added, and can modify and return it.  If the
     callable returns `None', the file will be excluded from the
     resulting archive.  This is more powerful than the existing
     _exclude_ argument, which has therefore been deprecated.  (Added
     by Lars Gustäbel; issue 6856 (http://bugs.python.org/issue6856).)
     The TarFile class also now supports the context
     manager protocol.  (Added by Lars Gustäbel; issue 7232
     (http://bugs.python.org/issue7232).)

   * The wait() method of the threading.Event.
     class now returns the internal flag on exit.  This means the
     method will usually return true because wait() is
     supposed to block until the internal flag becomes true.  The
     return value will only be false if a timeout was provided and the
     operation timed out.  (Contributed by Tim Lesher; issue 1674032
     (http://bugs.python.org/issue1674032).)

   * The Unicode database provided by the unicodedata.
     module is now used internally to determine which characters are
     numeric, whitespace, or represent line breaks.  The database also
     includes information from the `Unihan.txt' data file (patch by
     Anders Chrigström and Amaury Forgeot d'Arc; issue 1571184
     (http://bugs.python.org/issue1571184)) and has been updated to
     version 5.2.0 (updated by Florent Xicluna; issue 8024
     (http://bugs.python.org/issue8024)).

   * The urlparse module's urlsplit() now
     handles unknown URL schemes in a fashion compliant with RFC 3986
     (http://tools.ietf.org/html/rfc3986.html): if the URL is of the
     form `"<something>://..."', the text before the `://' is treated
     as the scheme, even if it's a made-up scheme that the module
     doesn't know about.  This change may break code that worked around
     the old behaviour.  For example, Python 2.6.4 or 2.5 will return
     the following:

         >>> import urlparse
         >>> urlparse.urlsplit('invented://host/filename?query')
         ('invented', '', '//host/filename?query', '', '')

     Python 2.7 (and Python 2.6.5) will return:

         >>> import urlparse
         >>> urlparse.urlsplit('invented://host/filename?query')
         ('invented', 'host', '/filename?query', '', '')

     (Python 2.7 actually produces slightly different output, since it
     returns a named tuple instead of a standard tuple.)

     The urlparse module also supports IPv6 literal
     addresses as defined by RFC 2732
     (http://tools.ietf.org/html/rfc2732.html) (contributed by Senthil
     Kumaran; issue 2987 (http://bugs.python.org/issue2987)).

         >>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo')
         ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]',
                     path='/foo', params='', query='', fragment='')

   * New class: the WeakSet class in the weakref.
     module is a set that only holds weak references to its elements;
     elements will be removed once there are no references pointing to
     them.  (Originally implemented in Python 3.x by Raymond Hettinger,
     and backported to 2.7 by Michael Foord.)

   * The ElementTree library, `xml.etree', no longer escapes ampersands
     and angle brackets when outputting an XML processing instruction
     (which looks like `<?xml-stylesheet href="#style1"?>') or comment
     (which looks like `<!-- comment -->').  (Patch by Neil Muller;
     issue 2746 (http://bugs.python.org/issue2746).)

   * The XML-RPC client and server, provided by the xmlrpclib
     and SimpleXMLRPCServer modules, have improved
     performance by supporting HTTP/1.1 keep-alive and by optionally
     using gzip encoding to compress the XML being exchanged.  The gzip
     compression is controlled by the `encode_threshold' attribute of
     `SimpleXMLRPCRequestHandler', which contains a size in bytes;
     responses larger than this will be compressed.  (Contributed by
     Kristján Valur Jónsson; issue 6267
     (http://bugs.python.org/issue6267).)

   * The zipfile module's ZipFile now supports
     the context management protocol, so you can write `with
     zipfile.ZipFile(...) as f:'.  (Contributed by Brian Curtin; issue
     5511 (http://bugs.python.org/issue5511).)

     zipfile now also supports archiving empty directories
     and extracts them correctly.  (Fixed by Kuba Wieczorek; issue 4710
     (http://bugs.python.org/issue4710).)  Reading files out of an
     archive is faster, and interleaving read() and
     `readline()' now works correctly.  (Contributed by Nir Aides;
     issue 7610 (http://bugs.python.org/issue7610).)

     The is_zipfile() function now accepts a file object,
     in addition to the path names accepted in earlier versions.
     (Contributed by Gabriel Genellina; issue 4756
     (http://bugs.python.org/issue4756).)

     The writestr() method now has an optional
     _compress_type_ parameter that lets you override the default
     compression method specified in the ZipFile.
     constructor.  (Contributed by Ronald Oussoren; issue 6003
     (http://bugs.python.org/issue6003).)

1.1.10.1 New module: importlib
..............................

Python 3.1 includes the importlib package, a
re-implementation of the logic underlying Python's import.
statement.  importlib is useful for implementors of Python
interpreters and to users who wish to write new importers that can
participate in the import process.  Python 2.7 doesn't contain the
complete importlib package, but instead has a tiny subset
that contains a single function, see import_module().

  `import_module(name, package=None)' imports a module.  _name_ is a
string containing the module or package's name.  It's possible to do
relative imports by providing a string that begins with a `.'
character, such as `..utils.errors'.  For relative imports, the
_package_ argument must be provided and is the name of the package that
will be used as the anchor for the relative import.
import_module() both inserts the imported module into
`sys.modules' and returns the module object.

  Here are some examples:

    >>> from importlib import import_module
    >>> anydbm = import_module('anydbm')  # Standard absolute import
    >>> anydbm
    <module 'anydbm' from '/p/python/Lib/anydbm.py'>
    >>> # Relative import
    >>> file_util = import_module('..file_util', 'distutils.command')
    >>> file_util
    <module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'>

importlib was implemented by Brett Cannon and introduced in
Python 3.1.

1.1.10.2 New module: sysconfig
..............................

The sysconfig module has been pulled out of the Distutils
package, becoming a new top-level module in its own right.
sysconfig provides functions for getting information about
Python's build process: compiler switches, installation paths, the
platform name, and whether Python is running from its source directory.

  Some of the functions in the module are:

   * get_config_var() returns variables from Python's
     Makefile and the `pyconfig.h' file.

   * get_config_vars() returns a dictionary containing all
     of the configuration variables.

   * get_path() returns the configured path for a
     particular type of module: the standard library, site-specific
     modules, platform-specific modules, etc.

   * is_python_build() returns true if you're running a
     binary from a Python source tree, and false otherwise.

  Consult the sysconfig documentation for more details and
for a complete list of functions.

  The Distutils package and sysconfig are now maintained by
Tarek Ziadé, who has also started a Distutils2 package (source
repository at <http://hg.python.org/distutils2/>) for developing a
next-generation version of Distutils.

1.1.10.3 ttk: Themed Widgets for Tk
...................................

Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk
widgets but have a more customizable appearance and can therefore more
closely resemble the native platform's widgets.  This widget set was
originally called Tile, but was renamed to Ttk (for "themed Tk") on
being added to Tcl/Tck release 8.5.

  To learn more, read the ttk module documentation.  You
may also wish to read the Tcl/Tk manual page describing the Ttk theme
engine, available at
<http://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm>. Some screenshots of
the Python/Ttk code in use are at
<http://code.google.com/p/python-ttk/wiki/Screenshots>.

  The ttk module was written by Guilherme Polo and added in
issue 2983 (http://bugs.python.org/issue2983).  An alternate version
called `Tile.py', written by Martin Franklin and maintained by Kevin
Walzer, was proposed for inclusion in issue 2618
(http://bugs.python.org/issue2618), but the authors argued that
Guilherme Polo's work was more comprehensive.

1.1.10.4 Updated module: unittest
.................................

The unittest module was greatly enhanced; many new features
were added.  Most of these features were implemented by Michael Foord,
unless otherwise noted.  The enhanced version of the module is
downloadable separately for use with Python versions 2.4 to 2.6,
packaged as the `unittest2' package, from
<http://pypi.python.org/pypi/unittest2>.

  When used from the command line, the module can automatically discover
tests.  It's not as fancy as py.test (http://pytest.org) or nose
(http://code.google.com/p/python-nose/), but provides a simple way to
run tests kept within a set of package directories.  For example, the
following command will search the `test/' subdirectory for any
importable test files named `test*.py':

    python -m unittest discover -s test

Consult the unittest module documentation for more details.
(Developed in issue 6001 (http://bugs.python.org/issue6001).)

  The main() function supports some other new options:

   * `-b' or `--buffer' will buffer the standard output and standard
     error streams during each test.  If the test passes, any resulting
     output will be discarded; on failure, the buffered output will be
     displayed.

   * -c or `--catch' will cause the control-C interrupt to
     be handled more gracefully.  Instead of interrupting the test
     process immediately, the currently running test will be completed
     and then the partial results up to the interruption will be
     reported.  If you're impatient, a second press of control-C will
     cause an immediate interruption.

     This control-C handler tries to avoid causing problems when the
     code being tested or the tests being run have defined a signal
     handler of their own, by noticing that a signal handler was
     already set and calling it.  If this doesn't work for you, there's
     a removeHandler() decorator that can be used to mark
     tests that should have the control-C handling disabled.

   * `-f' or `--failfast' makes test execution stop immediately when a
     test fails instead of continuing to execute further tests.
     (Suggested by Cliff Dyer and implemented by Michael Foord; issue
     8074 (http://bugs.python.org/issue8074).)

  The progress messages now show 'x' for expected failures and 'u' for
unexpected successes when run in verbose mode.  (Contributed by
Benjamin Peterson.)

  Test cases can raise the `SkipTest' exception to skip a test (issue
1034053 (http://bugs.python.org/issue1034053)).

  The error messages for assertEqual(), assertTrue(),
and assertFalse().  failures now provide more
information.  If you set the longMessage attribute of your
TestCase classes to True, both the standard error message
and any additional message you provide will be printed for failures.
(Added by Michael Foord; issue 5663 (http://bugs.python.org/issue5663).)

  The assertRaises() method now returns a context handler
when called without providing a callable object to run.  For example,
you can write this:

    with self.assertRaises(KeyError):
        {}['foo']

(Implemented by Antoine Pitrou; issue 4444
(http://bugs.python.org/issue4444).)

  Module- and class-level setup and teardown fixtures are now supported.
Modules can contain `setUpModule()' and `tearDownModule()' functions.
Classes can have setUpClass() and tearDownClass()
methods that must be defined as class methods (using
`@classmethod' or equivalent).  These functions and methods are invoked
when the test runner switches to a test case in a different module or
class.

  The methods addCleanup() and doCleanups().
were added.  addCleanup() lets you add cleanup functions
that will be called unconditionally (after setUp() if
setUp() fails, otherwise after tearDown()). This
allows for much simpler resource allocation and deallocation during
tests (issue 5679 (http://bugs.python.org/issue5679)).

  A number of new methods were added that provide more specialized
tests.  Many of these methods were written by Google engineers for use
in their test suites; Gregory P. Smith, Michael Foord, and GvR worked
on merging them into Python's version of unittest.

   * assertIsNone() and assertIsNotNone() take
     one expression and verify that the result is or is not `None'.

   * assertIs() and assertIsNot().  take two
     values and check whether the two values evaluate to the same
     object or not.  (Added by Michael Foord; issue 2578
     (http://bugs.python.org/issue2578).)

   * assertIsInstance() and assertNotIsInstance()
     check whether the resulting object is an instance of a
     particular class, or of one of a tuple of classes.  (Added by
     Georg Brandl; issue 7031 (http://bugs.python.org/issue7031).)

   * assertGreater(), assertGreaterEqual(),
     assertLess(), and assertLessEqual() compare two
     quantities.

   * assertMultiLineEqual() compares two strings, and if
     they're not equal, displays a helpful comparison that highlights
     the differences in the two strings.  This comparison is now used by
     default when Unicode strings are compared with
     assertEqual().

   * assertRegexpMatches() and
     assertNotRegexpMatches() checks whether the first argument
     is a string matching or not matching the regular expression
     provided as the second argument (issue 8038
     (http://bugs.python.org/issue8038)).

   * assertRaisesRegexp() checks whether a particular
     exception is raised, and then also checks that the string
     representation of the exception matches the provided regular
     expression.

   * assertIn() and assertNotIn().  tests
     whether _first_ is or is not in  _second_.

   * assertItemsEqual() tests whether two provided sequences
     contain the same elements.

   * assertSetEqual() compares whether two sets are equal,
     and only reports the differences between the sets in case of error.

   * Similarly, see assertListEqual() and
     assertTupleEqual().  compare the specified types and explain
     any differences without necessarily printing their full values;
     these methods are now used by default when comparing lists and
     tuples using assertEqual().  More generally, see
     assertSequenceEqual() compares two sequences and can
     optionally check whether both sequences are of a particular type.

   * assertDictEqual() compares two dictionaries and
     reports the differences; it's now used by default when you compare
     two dictionaries using assertEqual().
     assertDictContainsSubset() checks whether all of the
     key/value pairs in _first_ are found in _second_.

   * assertAlmostEqual() and assertNotAlmostEqual()
     test whether _first_ and _second_ are approximately equal.
     This method can either round their difference to an
     optionally-specified number of _places_ (the default is 7) and
     compare it to zero, or require the difference to be smaller than a
     supplied _delta_ value.

   * loadTestsFromName() properly honors the
     suiteClass attribute of the TestLoader. (Fixed by
     Mark Roddy; issue 6866 (http://bugs.python.org/issue6866).)

   * A new hook lets you extend the assertEqual() method to
     handle new data types.  The addTypeEqualityFunc().
     method takes a type object and a function. The function will be
     used when both of the objects being compared are of the specified
     type.  This function should compare the two objects and raise an
     exception if they don't match; it's a good idea for the function
     to provide additional information about why the two objects aren't
     matching, much as the new sequence comparison methods do.

  unittest.main() now takes an optional `exit' argument.  If
False, see main() doesn't call sys.exit(), allowing
main() to be used from the interactive interpreter.
(Contributed by J. Pablo Fernández; issue 3379
(http://bugs.python.org/issue3379).)

  TestResult has new startTestRun() and
stopTestRun() methods that are called immediately before and
after a test run.  (Contributed by Robert Collins; issue 5728
(http://bugs.python.org/issue5728).)

  With all these changes, the `unittest.py' was becoming awkwardly
large, so the module was turned into a package and the code split into
several files (by Benjamin Peterson).  This doesn't affect how the
module is imported or used.

<http://www.voidspace.org.uk/python/articles/unittest2.shtml>
     Describes the new features, how to use them, and the rationale for
     various design decisions.  (By Michael Foord.)

1.1.10.5 Updated module: ElementTree 1.3
........................................

The version of the ElementTree library included with Python was updated
to version 1.3.  Some of the new features are:

   * The various parsing functions now take a _parser_ keyword argument
     giving an `XMLParser' instance that will be used.  This makes it
     possible to override the file's internal encoding:

         p = ET.XMLParser(encoding='utf-8')
         t = ET.XML("""<root/>""", parser=p)

     Errors in parsing XML now raise a `ParseError' exception, whose
     instances have a `position' attribute containing a (_line_,
     _column_) tuple giving the location of the problem.

   * ElementTree's code for converting trees to a string has been
     significantly reworked, making it roughly twice as fast in many
     cases.  The `ElementTree' `write()' and `Element' `write()'
     methods now have a _method_ parameter that can be "xml" (the
     default), "html", or "text".  HTML mode will output empty elements
     as `<empty></empty>' instead of `<empty/>', and text mode will
     skip over elements and only output the text chunks.  If you set
     the `tag' attribute of an element to `None' but leave its children
     in place, the element will be omitted when the tree is written
     out, so you don't need to do more extensive rearrangement to
     remove a single element.

     Namespace handling has also been improved.  All `xmlns:<whatever>'
     declarations are now output on the root element, not scattered
     throughout the resulting XML.  You can set the default namespace
     for a tree by setting the `default_namespace' attribute and can
     register new prefixes with `register_namespace()'.  In XML mode,
     you can use the true/false _xml_declaration_ parameter to suppress
     the XML declaration.

   * New `Element' method: `extend()' appends the items from a sequence
     to the element's children.  Elements themselves behave like
     sequences, so it's easy to move children from one element to
     another:

         from xml.etree import ElementTree as ET

         t = ET.XML("""<list>
           <item>1</item> <item>2</item>  <item>3</item>
         </list>""")
         new = ET.XML('<root/>')
         new.extend(t)

         # Outputs <root><item>1</item>...</root>
         print ET.tostring(new)

   * New `Element' method: iter() yields the children of the
     element as a generator.  It's also possible to write `for child in
     elem:' to loop over an element's children.  The existing method
     `getiterator()' is now deprecated, as is `getchildren()' which
     constructs and returns a list of children.

   * New `Element' method: `itertext()' yields all chunks of text that
     are descendants of the element.  For example:

         t = ET.XML("""<list>
           <item>1</item> <item>2</item>  <item>3</item>
         </list>""")

         # Outputs ['\n  ', '1', ' ', '2', '  ', '3', '\n']
         print list(t.itertext())

   * Deprecated: using an element as a Boolean (i.e., `if elem:') would
     return true if the element had any children, or false if there were
     no children.  This behaviour is confusing - `None' is false, but
     so is a childless element? - so it will now trigger a
     FutureWarning.  In your code, you should be explicit: write
     `len(elem) != 0' if you're interested in the number of children,
     or `elem is not None'.

  Fredrik Lundh develops ElementTree and produced the 1.3 version; you
can read his article describing 1.3 at
<http://effbot.org/zone/elementtree-13-intro.htm>.  Florent Xicluna
updated the version included with Python, after discussions on
python-dev and in issue 6472 (http://bugs.python.org/issue6472).)

1.1.11 Build and C API Changes
------------------------------

Changes to Python's build process and to the C API include:

   * The latest release of the GNU Debugger, GDB 7, can be scripted
     using Python
     (http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html).
     When you begin debugging an executable program P, GDB will look for
     a file named `P-gdb.py' and automatically read it.  Dave Malcolm
     contributed a `python-gdb.py' that adds a number of commands
     useful when debugging Python itself.  For example, `py-up' and
     `py-down' go up or down one Python stack frame, which usually
     corresponds to several C stack frames.  `py-print' prints the
     value of a Python variable, and `py-bt' prints the Python stack
     trace.  (Added as a result of issue 8032
     (http://bugs.python.org/issue8032).)

   * If you use the `.gdbinit' file provided with Python, the "pyo"
     macro in the 2.7 version now works correctly when the thread being
     debugged doesn't hold the GIL; the macro now acquires it before
     printing.  (Contributed by Victor Stinner; issue 3632
     (http://bugs.python.org/issue3632).)

   * Py_AddPendingCall() is now thread-safe, letting any
     worker thread submit notifications to the main Python thread.  This
     is particularly useful for asynchronous IO operations.
     (Contributed by Kristján Valur Jónsson; issue 4293
     (http://bugs.python.org/issue4293).)

   * New function: PyCode_NewEmpty() creates an empty code
     object; only the filename, function name, and first line number
     are required.  This is useful for extension modules that are
     attempting to construct a more useful traceback stack.  Previously
     such extensions needed to call PyCode_New(), which had
     many more arguments.  (Added by Jeffrey Yasskin.)

   * New function: PyErr_NewExceptionWithDoc() creates a new
     exception class, just as the existing PyErr_NewException()
     does, but takes an extra `char *' argument containing the
     docstring for the new exception class.  (Added by 'lekma' on the
     Python bug tracker; issue 7033 (http://bugs.python.org/issue7033).)

   * New function: PyFrame_GetLineNumber() takes a frame
     object and returns the line number that the frame is currently
     executing.  Previously code would need to get the index of the
     bytecode instruction currently executing, and then look up the
     line number corresponding to that address.  (Added by Jeffrey
     Yasskin.)

   * New functions: PyLong_AsLongAndOverflow() and
     PyLong_AsLongLongAndOverflow().  approximates a Python long
     integer as a C `long' or `long long'.  If the number is too large
     to fit into the output type, an _overflow_ flag is set and
     returned to the caller.  (Contributed by Case Van Horsen; issue
     7528 (http://bugs.python.org/issue7528) and issue 7767
     (http://bugs.python.org/issue7767).)

   * New function: stemming from the rewrite of string-to-float
     conversion, a new PyOS_string_to_double() function was
     added.  The old PyOS_ascii_strtod() and
     PyOS_ascii_atof() functions are now deprecated.

   * New function: PySys_SetArgvEx() sets the value of
     `sys.argv' and can optionally update `sys.path' to include the
     directory containing the script named by `sys.argv[0]' depending
     on the value of an _updatepath_ parameter.

     This function was added to close a security hole for applications
     that embed Python.  The old function, see PySys_SetArgv(),
     would always update `sys.path', and sometimes it would add the
     current directory.  This meant that, if you ran an application
     embedding Python in a directory controlled by someone else,
     attackers could put a Trojan-horse module in the directory (say, a
     file named `os.py') that your application would then import and
     run.

     If you maintain a C/C++ application that embeds Python, check
     whether you're calling PySys_SetArgv() and carefully
     consider whether the application should be using
     PySys_SetArgvEx().  with _updatepath_ set to false.

     Security issue reported as CVE-2008-5983
     (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983);
     discussed in issue 5753 (http://bugs.python.org/issue5753), and
     fixed by Antoine Pitrou.

   * New macros: the Python header files now define the following
     macros: `Py_ISALNUM', `Py_ISALPHA', `Py_ISDIGIT', `Py_ISLOWER',
     `Py_ISSPACE', `Py_ISUPPER', `Py_ISXDIGIT', `Py_TOLOWER', and
     `Py_TOUPPER'.  All of these functions are analogous to the C
     standard macros for classifying characters, but ignore the current
     locale setting, because in several places Python needs to analyze
     characters in a locale-independent way.  (Added by Eric Smith;
     issue 5793 (http://bugs.python.org/issue5793).)

   * Removed function: `PyEval_CallObject' is now only available as a
     macro.  A function version was being kept around to preserve ABI
     linking compatibility, but that was in 1997; it can certainly be
     deleted by now.  (Removed by Antoine Pitrou; issue 8276
     (http://bugs.python.org/issue8276).)

   * New format codes: the `PyFormat_FromString()',
     `PyFormat_FromStringV()', and PyErr_Format() functions
     now accept `%lld' and `%llu' format codes for displaying C's `long
     long' types.  (Contributed by Mark Dickinson; issue 7228
     (http://bugs.python.org/issue7228).)

   * The complicated interaction between threads and process forking has
     been changed.  Previously, the child process created by
     os.fork() might fail because the child is created with only a
     single thread running, the thread performing the os.fork().
     If other threads were holding a lock, such as Python's
     import lock, when the fork was performed, the lock would still be
     marked as "held" in the new process.  But in the child process
     nothing would ever release the lock, since the other threads
     weren't replicated, and the child process would no longer be able
     to perform imports.

     Python 2.7 acquires the import lock before performing an
     os.fork(), and will also clean up any locks created using the
     threading module.  C extension modules that have
     internal locks, or that call `fork()' themselves, will not benefit
     from this clean-up.

     (Fixed by Thomas Wouters; issue 1590864
     (http://bugs.python.org/issue1590864).)

   * The Py_Finalize() function now calls the internal
     `threading._shutdown()' function; this prevents some exceptions
     from being raised when an interpreter shuts down.  (Patch by Adam
     Olsen; issue 1722344 (http://bugs.python.org/issue1722344).)

   * When using the PyMemberDef structure to define
     attributes of a type, Python will no longer let you try to delete
     or set a `T_STRING_INPLACE' attribute.

   * Global symbols defined by the ctypes module are now
     prefixed with `Py', or with `_ctypes'.  (Implemented by Thomas
     Heller; issue 3102 (http://bugs.python.org/issue3102).)

   * New configure option: the `--with-system-expat' switch allows
     building the `pyexpat' module to use the system Expat library.
     (Contributed by Arfrever Frehtes Taifersar Arahesis; issue 7609
     (http://bugs.python.org/issue7609).)

   * New configure option: the `--with-valgrind' option will now
     disable the pymalloc allocator, which is difficult for the
     Valgrind memory-error detector to analyze correctly.  Valgrind
     will therefore be better at detecting memory leaks and overruns.
     (Contributed by James Henstridge; issue 2422
     (http://bugs.python.org/issue2422).)

   * New configure option: you can now supply an empty string to
     `--with-dbmliborder=' in order to disable all of the various DBM
     modules.  (Added by Arfrever Frehtes Taifersar Arahesis; issue
     6491 (http://bugs.python.org/issue6491).)

   * The *configure* script now checks for floating-point rounding bugs
     on certain 32-bit Intel chips and defines a `X87_DOUBLE_ROUNDING'
     preprocessor definition.  No code currently uses this definition,
     but it's available if anyone wishes to use it.  (Added by Mark
     Dickinson; issue 2937 (http://bugs.python.org/issue2937).)

     *configure* also now sets a `LDCXXSHARED' Makefile variable for
     supporting C++ linking.  (Contributed by Arfrever Frehtes
     Taifersar Arahesis; issue 1222585
     (http://bugs.python.org/issue1222585).)

   * The build process now creates the necessary files for pkg-config
     support.  (Contributed by Clinton Roy; issue 3585
     (http://bugs.python.org/issue3585).)

   * The build process now supports Subversion 1.7.  (Contributed by
     Arfrever Frehtes Taifersar Arahesis; issue 6094
     (http://bugs.python.org/issue6094).)

1.1.11.1 Capsules
.................

Python 3.1 adds a new C datatype, see PyCapsule, for providing a
C API to an extension module.  A capsule is essentially the holder of a
C `void *' pointer, and is made available as a module attribute; for
example, the socket module's API is exposed as
`socket.CAPI', and unicodedata exposes `ucnhash_CAPI'.
Other extensions can import the module, access its dictionary to get
the capsule object, and then get the `void *' pointer, which will
usually point to an array of pointers to the module's various API
functions.

  There is an existing data type already used for this, see
PyCObject, but it doesn't provide type safety.  Evil code written
in pure Python could cause a segmentation fault by taking a
PyCObject from module A and somehow substituting it for the
PyCObject in module B.   Capsules know their own name, and
getting the pointer requires providing the name:

    void *vtable;

    if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") {
            PyErr_SetString(PyExc_ValueError, "argument type invalid");
            return NULL;
    }

    vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI");

You are assured that `vtable' points to whatever you're expecting.  If
a different capsule was passed in, see PyCapsule_IsValid() would
detect the mismatched name and return false.  Refer to Providing a C API for an Extension Module
for more information on using
these objects.

  Python 2.7 now uses capsules internally to provide various
extension-module APIs, but the PyCObject_AsVoidPtr() was
modified to handle capsules, preserving compile-time compatibility with
the `CObject' interface.  Use of PyCObject_AsVoidPtr() will
signal a PendingDeprecationWarning, which is silent by
default.

  Implemented in Python 3.1 and backported to 2.7 by Larry Hastings;
discussed in issue 5630 (http://bugs.python.org/issue5630).

1.1.11.2 Port-Specific Changes: Windows
.......................................

   * The msvcrt module now contains some constants from the
     `crtassem.h' header file: `CRT_ASSEMBLY_VERSION',
     `VC_ASSEMBLY_PUBLICKEYTOKEN', and `LIBRARIES_ASSEMBLY_NAME_PREFIX'.
     (Contributed by David Cournapeau; issue 4365
     (http://bugs.python.org/issue4365).)

   * The _winreg module for accessing the registry now
     implements the CreateKeyEx() and DeleteKeyEx().
     functions, extended versions of previously-supported
     functions that take several extra arguments.  The
     DisableReflectionKey(), EnableReflectionKey(), and
     QueryReflectionKey().  were also tested and documented.
     (Implemented by Brian Curtin: issue 7347
     (http://bugs.python.org/issue7347).)

   * The new `_beginthreadex()' API is used to start threads, and the
     native thread-local storage functions are now used.  (Contributed
     by Kristján Valur Jónsson; issue 3582
     (http://bugs.python.org/issue3582).)

   * The os.kill() function now works on Windows.  The
     signal value can be the constants `CTRL_C_EVENT',
     `CTRL_BREAK_EVENT', or any integer.  The first two constants will
     send Control-C and Control-Break keystroke events to subprocesses;
     any other value will use the `TerminateProcess()' API.
     (Contributed by Miki Tebeka; issue 1220212
     (http://bugs.python.org/issue1220212).)

   * The os.listdir() function now correctly fails for an
     empty path.  (Fixed by Hirokazu Yamamoto; issue 5913
     (http://bugs.python.org/issue5913).)

   * The `mimelib' module will now read the MIME database from the
     Windows registry when initializing.  (Patch by Gabriel Genellina;
     issue 4969 (http://bugs.python.org/issue4969).)

1.1.11.3 Port-Specific Changes: Mac OS X
........................................

   * The path `/Library/Python/2.7/site-packages' is now appended to
     `sys.path', in order to share added packages between the system
     installation and a user-installed copy of the same version.
     (Changed by Ronald Oussoren; issue 4865
     (http://bugs.python.org/issue4865).)

1.1.11.4 Port-Specific Changes: FreeBSD
.......................................

   * FreeBSD 7.1's `SO_SETFIB' constant, used with
     `getsockopt()'/`setsockopt()' to select an alternate routing
     table, is now available in the socket.  module.  (Added
     by Kyle VanderBeek; issue 8235 (http://bugs.python.org/issue8235).)

1.1.12 Other Changes and Fixes
------------------------------

   * Two benchmark scripts, `iobench' and `ccbench', were added to the
     `Tools' directory.  `iobench' measures the speed of the built-in
     file I/O objects returned by open().  while performing
     various operations, and `ccbench' is a concurrency benchmark that
     tries to measure computing throughput, thread switching latency,
     and IO processing bandwidth when performing several tasks using a
     varying number of threads.

   * The `Tools/i18n/msgfmt.py' script now understands plural forms in
     `.po' files.  (Fixed by Martin von Löwis; issue 5464
     (http://bugs.python.org/issue5464).)

   * When importing a module from a `.pyc' or `.pyo' file with an
     existing `.py' counterpart, the `co_filename' attributes of the
     resulting code objects are overwritten when the original filename
     is obsolete.  This can happen if the file has been renamed, moved,
     or is accessed through different paths.  (Patch by Ziga Seilnacht
     and Jean-Paul Calderone; issue 1180193
     (http://bugs.python.org/issue1180193).)

   * The `regrtest.py' script now takes a `--randseed=' switch that
     takes an integer that will be used as the random seed for the `-r'
     option that executes tests in random order.  The `-r' option also
     reports the seed that was used (Added by Collin Winter.)

   * Another `regrtest.py' switch is `-j', which takes an integer
     specifying how many tests run in parallel. This allows reducing
     the total runtime on multi-core machines.  This option is
     compatible with several other options, including the `-R' switch
     which is known to produce long runtimes.  (Added by Antoine
     Pitrou, issue 6152 (http://bugs.python.org/issue6152).)  This can
     also be used with a new `-F' switch that runs selected tests in a
     loop until they fail.  (Added by Antoine Pitrou; issue 7312
     (http://bugs.python.org/issue7312).)

   * When executed as a script, the `py_compile.py' module now accepts
     `'-'' as an argument, which will read standard input for the list
     of filenames to be compiled.  (Contributed by Piotr Ożarowski;
     issue 8233 (http://bugs.python.org/issue8233).)

1.1.13 Porting to Python 2.7
----------------------------

This section lists previously described changes and other bugfixes that
may require changes to your code:

   * The range() function processes its arguments more
     consistently; it will now call __int__() on non-float,
     non-integer arguments that are supplied to it.  (Fixed by Alexander
     Belopolsky; issue 1533 (http://bugs.python.org/issue1533).)

   * The string format() method changed the default
     precision used for floating-point and complex numbers from 6
     decimal places to 12, which matches the precision used by
     str().  (Changed by Eric Smith; issue 5920
     (http://bugs.python.org/issue5920).)

   * Because of an optimization for the with statement, the
     special methods __enter__() and __exit__().
     must belong to the object's type, and cannot be directly attached
     to the object's instance.  This affects new-style classes (derived
     from object) and C extension types.  (issue 6101
     (http://bugs.python.org/issue6101).)

   * Due to a bug in Python 2.6, the _exc_value_ parameter to
     __exit__() methods was often the string representation of the
     exception, not an instance.  This was fixed in 2.7, so _exc_value_
     will be an instance as expected.  (Fixed by Florent Xicluna; issue
     7853 (http://bugs.python.org/issue7853).)

   * When a restricted set of attributes were set using `__slots__',
     deleting an unset attribute would not raise AttributeError.
     as you would expect.  Fixed by Benjamin Peterson; issue 7604
     (http://bugs.python.org/issue7604).)

  In the standard library:

   * Operations with datetime instances that resulted in a
     year falling outside the supported range didn't always raise
     OverflowError.  Such errors are now checked more carefully
     and will now raise the exception. (Reported by Mark Leander, patch
     by Anand B. Pillai and Alexander Belopolsky; issue 7150
     (http://bugs.python.org/issue7150).)

   * When using Decimal instances with a string's
     format() method, the default alignment was previously
     left-alignment.  This has been changed to right-alignment, which
     might change the output of your programs.  (Changed by Mark
     Dickinson; issue 6857 (http://bugs.python.org/issue6857).)

     Comparisons involving a signaling NaN value (or `sNAN') now signal
     InvalidOperation instead of silently returning a true
     or false value depending on the comparison operator.  Quiet NaN
     values (or `NaN') are now hashable.  (Fixed by Mark Dickinson;
     issue 7279 (http://bugs.python.org/issue7279).)

   * The ElementTree library, `xml.etree', no longer escapes ampersands
     and angle brackets when outputting an XML processing instruction
     (which looks like `<?xml-stylesheet href="#style1"?>') or comment
     (which looks like `<!- comment ->').  (Patch by Neil Muller; issue
     2746 (http://bugs.python.org/issue2746).)

   * The `readline()' method of StringIO objects now does
     nothing when a negative length is requested, as other file-like
     objects do.  (issue 7348 (http://bugs.python.org/issue7348)).

   * The syslog module will now use the value of
     `sys.argv[0]' as the identifier instead of the previous default
     value of `'python''.  (Changed by Sean Reifschneider; issue 8451
     (http://bugs.python.org/issue8451).)

   * The tarfile module's default error handling has
     changed, to no longer suppress fatal errors.  The default error
     level was previously 0, which meant that errors would only result
     in a message being written to the debug log, but because the debug
     log is not activated by default, these errors go unnoticed.  The
     default error level is now 1, which raises an exception if there's
     an error.  (Changed by Lars Gustäbel; issue 7357
     (http://bugs.python.org/issue7357).)

   * The urlparse module's urlsplit() now
     handles unknown URL schemes in a fashion compliant with RFC 3986
     (http://tools.ietf.org/html/rfc3986.html): if the URL is of the
     form `"<something>://..."', the text before the `://' is treated
     as the scheme, even if it's a made-up scheme that the module
     doesn't know about.  This change may break code that worked around
     the old behaviour.  For example, Python 2.6.4 or 2.5 will return
     the following:

         >>> import urlparse
         >>> urlparse.urlsplit('invented://host/filename?query')
         ('invented', '', '//host/filename?query', '', '')

     Python 2.7 (and Python 2.6.5) will return:

         >>> import urlparse
         >>> urlparse.urlsplit('invented://host/filename?query')
         ('invented', 'host', '/filename?query', '', '')

     (Python 2.7 actually produces slightly different output, since it
     returns a named tuple instead of a standard tuple.)

  For C extensions:

   * C extensions that use integer format codes with the `PyArg_Parse*'
     family of functions will now raise a TypeError.
     exception instead of triggering a DeprecationWarning.
     (issue 5080 (http://bugs.python.org/issue5080)).

   * Use the new PyOS_string_to_double() function instead
     of the old PyOS_ascii_strtod() and
     PyOS_ascii_atof() functions, which are now deprecated.

  For applications that embed Python:

   * The PySys_SetArgvEx() function was added, letting
     applications close a security hole when the existing
     PySys_SetArgv() function was used.  Check whether you're
     calling PySys_SetArgv() and carefully consider whether
     the application should be using PySys_SetArgvEx() with
     _updatepath_ set to false.

1.1.14 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray,
Hugh Secker-Walker.

1.2 What's New in Python 2.6
============================

     Author: A.M. Kuchling (amk at amk.ca)

     Release: 2.7.1

     Date: February 08, 2011

  This article explains the new features in Python 2.6, released on
October 1 2008.  The release schedule is described in PEP 361
(http://www.python.org/dev/peps/pep-0361).

  The major theme of Python 2.6 is preparing the migration path to
Python 3.0, a major redesign of the language.  Whenever possible,
Python 2.6 incorporates new features and syntax from 3.0 while
remaining compatible with existing code by not removing older features
or syntax.  When it's not possible to do that, Python 2.6 tries to do
what it can, adding compatibility functions in a future_builtins
module and a -3 switch to warn about usages that will
become unsupported in 3.0.

  Some significant new packages have been added to the standard library,
such as the multiprocessing and json modules, but
there aren't many new features that aren't related to Python 3.0 in
some way.

  Python 2.6 also sees a number of improvements and bugfixes throughout
the source.  A search through the change logs finds there were 259
patches applied and 612 bugs fixed between Python 2.5 and 2.6.  Both
figures are likely to be underestimates.

  This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For full
details, you should refer to the documentation for Python 2.6. If you
want to understand the rationale for the design and implementation,
refer to the PEP for a particular new feature.  Whenever possible,
"What's New in Python" links to the bug/patch item for each change.

1.2.1 Python 3.0
----------------

The development cycle for Python versions 2.6 and 3.0 was synchronized,
with the alpha and beta releases for both versions being made on the
same days.  The development of 3.0 has influenced many features in 2.6.

  Python 3.0 is a far-ranging redesign of Python that breaks
compatibility with the 2.x series.  This means that existing Python
code will need some conversion in order to run on Python 3.0.  However,
not all the changes in 3.0 necessarily break compatibility.  In cases
where new features won't cause existing code to break, they've been
backported to 2.6 and are described in this document in the appropriate
place.  Some of the 3.0-derived features are:

   * A __complex__() method for converting objects to a
     complex number.

   * Alternate syntax for catching exceptions: `except TypeError as
     exc'.

   * The addition of functools.reduce() as a synonym for
     the built-in reduce() function.

  Python 3.0 adds several new built-in functions and changes the
semantics of some existing builtins.  Functions that are new in 3.0
such as bin() have simply been added to Python 2.6, but
existing builtins haven't been changed; instead, the
future_builtins.  module has versions with the new 3.0 semantics.
Code written to be compatible with 3.0 can do `from future_builtins
import hex, map' as necessary.

  A new command-line switch, see -3, enables warnings about
features that will be removed in Python 3.0.  You can run code with
this switch to see how much work will be necessary to port code to 3.0.
The value of this switch is available to Python code as the boolean
variable sys.py3kwarning, and to C extension code as
`Py_Py3kWarningFlag'.

  The 3xxx series of PEPs, which contains proposals for Python 3.0.  PEP
3000 (http://www.python.org/dev/peps/pep-3000) describes the
development process for Python 3.0.  Start with PEP 3100
(http://www.python.org/dev/peps/pep-3100) that describes the general
goals for Python 3.0, and then explore the higher-numbered PEPS that
propose specific features.

1.2.2 Changes to the Development Process
----------------------------------------

While 2.6 was being developed, the Python development process underwent
two significant changes: we switched from SourceForge's issue tracker
to a customized Roundup installation, and the documentation was
converted from LaTeX to reStructuredText.

1.2.2.1 New Issue Tracker: Roundup
..................................

For a long time, the Python developers had been growing increasingly
annoyed by SourceForge's bug tracker.  SourceForge's hosted solution
doesn't permit much customization; for example, it wasn't possible to
customize the life cycle of issues.

  The infrastructure committee of the Python Software Foundation
therefore posted a call for issue trackers, asking volunteers to set up
different products and import some of the bugs and patches from
SourceForge.  Four different trackers were examined: Jira
(http://www.atlassian.com/software/jira/), Launchpad
(http://www.launchpad.net), Roundup (http://roundup.sourceforge.net/),
and Trac (http://trac.edgewall.org/).  The committee eventually settled
on Jira and Roundup as the two candidates.  Jira is a commercial
product that offers no-cost hosted instances to free-software projects;
Roundup is an open-source project that requires volunteers to
administer it and a server to host it.

  After posting a call for volunteers, a new Roundup installation was
set up at <http://bugs.python.org>.  One installation of Roundup can
host multiple trackers, and this server now also hosts issue trackers
for Jython and for the Python web site.  It will surely find other uses
in the future.  Where possible, this edition of "What's New in Python"
links to the bug/patch item for each change.

  Hosting of the Python bug tracker is kindly provided by Upfront
Systems (http://www.upfrontsystems.co.za/) of Stellenbosch, South
Africa.  Martin von Loewis put a lot of effort into importing existing
bugs and patches from SourceForge; his scripts for this import
operation are at <http://svn.python.org/view/tracker/importer/> and may
be useful to other projects wishing to move from SourceForge to Roundup.

<http://bugs.python.org>
     The Python bug tracker.

<http://bugs.jython.org>:
     The Jython bug tracker.

<http://roundup.sourceforge.net/>
     Roundup downloads and documentation.

<http://svn.python.org/view/tracker/importer/>
     Martin von Loewis's conversion scripts.

1.2.2.2 New Documentation Format: reStructuredText Using Sphinx
...............................................................

The Python documentation was written using LaTeX since the project
started around 1989.  In the 1980s and early 1990s, most documentation
was printed out for later study, not viewed online. LaTeX was widely
used because it provided attractive printed output while remaining
straightforward to write once the basic rules of the markup were
learned.

  Today LaTeX is still used for writing publications destined for
printing, but the landscape for programming tools has shifted.  We no
longer print out reams of documentation; instead, we browse through it
online and HTML has become the most important format to support.
Unfortunately, converting LaTeX to HTML is fairly complicated and Fred
L. Drake Jr., the long-time Python documentation editor, spent a lot of
time maintaining the conversion process.  Occasionally people would
suggest converting the documentation into SGML and later XML, but
performing a good conversion is a major task and no one ever committed
the time required to finish the job.

  During the 2.6 development cycle, Georg Brandl put a lot of effort
into building a new toolchain for processing the documentation.  The
resulting package is called Sphinx, and is available from
<http://sphinx.pocoo.org/>.

  Sphinx concentrates on HTML output, producing attractively styled and
modern HTML; printed output is still supported through conversion to
LaTeX.  The input format is reStructuredText, a markup syntax
supporting custom extensions and directives that is commonly used in
the Python community.

  Sphinx is a standalone package that can be used for writing, and
almost two dozen other projects (listed on the Sphinx web site
(http://sphinx.pocoo.org/examples.html)) have adopted Sphinx as their
documentation tool.

See Documenting Python.
     Describes how to write for Python's documentation.

Sphinx (http://sphinx.pocoo.org/)
     Documentation and code for the Sphinx toolchain.

Docutils (http://docutils.sf.net)
     The underlying reStructuredText parser and toolset.

1.2.3 PEP 343: The 'with' statement
-----------------------------------

The previous version, Python 2.5, added the 'with.'
statement as an optional feature, to be enabled by a `from __future__
import with_statement' directive.  In 2.6 the statement no longer needs
to be specially enabled; this means that with is now always
a keyword.  The rest of this section is a copy of the corresponding
section from the "What's New in Python 2.5" document; if you're
familiar with the 'with.' statement from Python 2.5, you can
skip this section.

  The 'with.' statement clarifies code that previously would
use `try...finally' blocks to ensure that clean-up code is executed.
In this section, I'll discuss the statement as it will commonly be
used.  In the next section, I'll examine the implementation details and
show how to write objects for use with this statement.

  The 'with.' statement is a control-flow structure whose
basic structure is:

    with expression [as variable]:
        with-block

The expression is evaluated, and it should result in an object that
supports the context management protocol (that is, has
__enter__() and __exit__().  methods).

  The object's __enter__() is called before _with-block_ is
executed and therefore can run set-up code. It also may return a value
that is bound to the name _variable_, if given.  (Note carefully that
_variable_ is _not_ assigned the result of _expression_.)

  After execution of the _with-block_ is finished, the object's
__exit__().  method is called, even if the block raised an
exception, and can therefore run clean-up code.

  Some standard Python objects now support the context management
protocol and can be used with the 'with.' statement. File
objects are one example:

    with open('/etc/passwd', 'r') as f:
        for line in f:
            print line
            ... more processing code ...

After this statement has executed, the file object in _f_ will have been
automatically closed, even if the for loop raised an
exception part- way through the block.

     Note: In this case, _f_ is the same object created by
     open(), because `file.__enter__()' returns _self_.

  The threading module's locks and condition variables
also support the 'with.' statement:

    lock = threading.Lock()
    with lock:
        # Critical section of code
        ...

The lock is acquired before the block is executed and always released
once  the block is complete.

  The `localcontext()' function in the decimal module makes
it easy to save and restore the current decimal context, which
encapsulates the desired precision and rounding characteristics for
computations:

    from decimal import Decimal, Context, localcontext

    # Displays with default precision of 28 digits
    v = Decimal('578')
    print v.sqrt()

    with localcontext(Context(prec=16)):
        # All code in this block uses a precision of 16 digits.
        # The original context is restored on exiting the block.
        print v.sqrt()

1.2.3.1 Writing Context Managers
................................

Under the hood, the 'with.' statement is fairly complicated.
Most people will only use 'with.' in company with existing
objects and don't need to know these details, so you can skip the rest
of this section if you like.  Authors of new objects will need to
understand the details of the underlying implementation and should keep
reading.

  A high-level explanation of the context management protocol is:

   * The expression is evaluated and should result in an object called
     a "context manager".  The context manager must have
     __enter__() and __exit__().  methods.

   * The context manager's __enter__() method is called.
     The value returned is assigned to _VAR_.  If no `as VAR' clause is
     present, the value is simply discarded.

   * The code in _BLOCK_ is executed.

   * If _BLOCK_ raises an exception, the context manager's
     __exit__() method is called with three arguments, the
     exception details (`type, value, traceback', the same values
     returned by sys.exc_info(), which can also be `None' if
     no exception occurred).  The method's return value controls
     whether an exception is re-raised: any false value re-raises the
     exception, and `True' will result in suppressing it.  You'll only
     rarely want to suppress the exception, because if you do the
     author of the code containing the 'with.' statement will
     never realize anything went wrong.

   * If _BLOCK_ didn't raise an exception,  the __exit__().
     method is still called, but _type_, _value_, and _traceback_ are
     all `None'.

  Let's think through an example.  I won't present detailed code but
will only sketch the methods necessary for a database that supports
transactions.

  (For people unfamiliar with database terminology: a set of changes to
the database are grouped into a transaction.  Transactions can be
either committed, meaning that all the changes are written into the
database, or rolled back, meaning that the changes are all discarded
and the database is unchanged.  See any database textbook for more
information.)

  Let's assume there's an object representing a database connection.
Our goal will be to let the user write code like this:

    db_connection = DatabaseConnection()
    with db_connection as cursor:
        cursor.execute('insert into ...')
        cursor.execute('delete from ...')
        # ... more operations ...

The transaction should be committed if the code in the block runs
flawlessly or rolled back if there's an exception. Here's the basic
interface for `DatabaseConnection' that I'll assume:

    class DatabaseConnection:
        # Database interface
        def cursor(self):
            "Returns a cursor object and starts a new transaction"
        def commit(self):
            "Commits current transaction"
        def rollback(self):
            "Rolls back current transaction"

The __enter__() method is pretty easy, having only to start
a new transaction.  For this application the resulting cursor object
would be a useful result, so the method will return it.  The user can
then add `as cursor' to their 'with.' statement to bind the
cursor to a variable name.

    class DatabaseConnection:
        ...
        def __enter__(self):
            # Code to start a new transaction
            cursor = self.cursor()
            return cursor

The __exit__() method is the most complicated because it's
where most of the work has to be done.  The method has to check if an
exception occurred.  If there was no exception, the transaction is
committed.  The transaction is rolled back if there was an exception.

  In the code below, execution will just fall off the end of the
function, returning the default value of `None'.  `None' is false, so
the exception will be re-raised automatically.  If you wished, you
could be more explicit and add a return statement at the
marked location.

    class DatabaseConnection:
        ...
        def __exit__(self, type, value, tb):
            if tb is None:
                # No exception, so commit
                self.commit()
            else:
                # Exception occurred, so rollback.
                self.rollback()
                # return False

1.2.3.2 The contextlib module
.............................

The contextlib module provides some functions and a
decorator that are useful when writing objects for use with the '
with.' statement.

  The decorator is called `contextmanager()', and lets you write a
single generator function instead of defining a new class.  The
generator should yield exactly one value.  The code up to the
yield will be executed as the __enter__() method, and
the value yielded will be the method's return value that will get bound
to the variable in the 'with.' statement's as.
clause, if any.  The code after the yield will be executed
in the __exit__() method.  Any exception raised in the
block will be raised by the yield statement.

  Using this decorator, our database example from the previous section
could be written as:

    from contextlib import contextmanager

    @contextmanager
    def db_transaction(connection):
        cursor = connection.cursor()
        try:
            yield cursor
        except:
            connection.rollback()
            raise
        else:
            connection.commit()

    db = DatabaseConnection()
    with db_transaction(db) as cursor:
        ...

The contextlib module also has a `nested(mgr1, mgr2, ...)'
function that combines a number of context managers so you don't need
to write nested 'with.' statements.  In this example, the
single 'with.'  statement both starts a database transaction
and acquires a thread lock:

    lock = threading.Lock()
    with nested (db_transaction(db), lock) as (cursor, locked):
        ...

Finally, the `closing()' function returns its argument so that it can be
bound to a variable, and calls the argument's `.close()' method at the
end of the block.

    import urllib, sys
    from contextlib import closing

    with closing(urllib.urlopen('http://www.yahoo.com')) as f:
        for line in f:
            sys.stdout.write(line)

PEP 343 (http://www.python.org/dev/peps/pep-0343) - The "with" statement
     PEP written by Guido van Rossum and Nick Coghlan; implemented by
     Mike Bland, Guido van Rossum, and Neal Norwitz.  The PEP shows the
     code generated for a 'with.' statement, which can be
     helpful in learning how the statement works.

  The documentation  for the contextlib module.

1.2.4 PEP 366: Explicit Relative Imports From a Main Module
-----------------------------------------------------------

Python's -m switch allows running a module as a script.
When you ran a module that was located inside a package, relative
imports didn't work correctly.

  The fix for Python 2.6 adds a `__package__' attribute to modules.
When this attribute is present, relative imports will be relative to
the value of this attribute instead of the `__name__' attribute.

  PEP 302-style importers can then set `__package__' as necessary.  The
runpy module that implements the -m switch now
does this, so relative imports will now work correctly in scripts
running from inside a package.

1.2.5 PEP 370: Per-user `site-packages' Directory
-------------------------------------------------

When you run Python, the module search path `sys.path' usually includes
a directory whose path ends in `"site-packages"'.  This directory is
intended to hold locally-installed packages available to all users
using a machine or a particular site installation.

  Python 2.6 introduces a convention for user-specific site directories.
The directory varies depending on the platform:

   * Unix and Mac OS X: `~/.local/'

   * Windows: `%APPDATA%/Python'

  Within this directory, there will be version-specific subdirectories,
such as `lib/python2.6/site-packages' on Unix/Mac OS and
`Python26/site-packages' on Windows.

  If you don't like the default directory, it can be overridden by an
environment variable.  PYTHONUSERBASE sets the root
directory used for all Python versions supporting this feature.  On
Windows, the directory for application-specific data can be changed by
setting the `APPDATA' environment variable.  You can also modify the
`site.py' file for your Python installation.

  The feature can be disabled entirely by running Python with the
-s option or setting the PYTHONNOUSERSITE.
environment variable.

PEP 370 (http://www.python.org/dev/peps/pep-0370) - Per-user `site-packages' Directory
     PEP written and implemented by Christian Heimes.

1.2.6 PEP 371: The `multiprocessing' Package
--------------------------------------------

The new multiprocessing package lets Python programs create
new processes that will perform a computation and return a result to the
parent.  The parent and child processes can communicate using queues
and pipes, synchronize their operations using locks and semaphores, and
can share simple arrays of data.

  The multiprocessing module started out as an exact
emulation of the threading module using processes instead
of threads.  That goal was discarded along the path to Python 2.6, but
the general approach of the module is still similar.  The fundamental
class is the `Process', which is passed a callable object and a
collection of arguments.  The `start()' method sets the callable
running in a subprocess, after which you can call the `is_alive()'
method to check whether the subprocess is still running and the
`join()' method to wait for the process to exit.

  Here's a simple example where the subprocess will calculate a
factorial.  The function doing the calculation is written strangely so
that it takes significantly longer when the input argument is a
multiple of 4.

    import time
    from multiprocessing import Process, Queue

    def factorial(queue, N):
        "Compute a factorial."
        # If N is a multiple of 4, this function will take much longer.
        if (N % 4) == 0:
            time.sleep(.05 * N/4)

        # Calculate the result
        fact = 1L
        for i in range(1, N+1):
            fact = fact * i

        # Put the result on the queue
        queue.put(fact)

    if __name__ == '__main__':
        queue = Queue()

        N = 5

        p = Process(target=factorial, args=(queue, N))
        p.start()
        p.join()

        result = queue.get()
        print 'Factorial', N, '=', result

A Queue is used to communicate the input parameter _N_ and
the result.  The Queue object is stored in a global
variable.  The child process will use the value of the variable when
the child was created; because it's a Queue, parent and
child can use the object to communicate.  (If the parent were to change
the value of the global variable, the child's value would be
unaffected, and vice versa.)

  Two other classes, `Pool' and `Manager', provide higher-level
interfaces.  `Pool' will create a fixed number of worker processes, and
requests can then be distributed to the workers by calling
apply() or `apply_async()' to add a single request, and
map() or `map_async()' to add a number of requests.  The
following code uses a `Pool' to spread requests across 5 worker
processes and retrieve a list of results:

    from multiprocessing import Pool

    def factorial(N, dictionary):
        "Compute a factorial."
        ...
    p = Pool(5)
    result = p.map(factorial, range(1, 1000, 10))
    for v in result:
        print v

This produces the following output:

    1
    39916800
    51090942171709440000
    8222838654177922817725562880000000
    33452526613163807108170062053440751665152000000000
    ...

The other high-level interface, the `Manager' class, creates a separate
server process that can hold master copies of Python data structures.
Other processes can then access and modify these data structures using
proxy objects.  The following example creates a shared dictionary by
calling the dict() method; the worker processes then insert
values into the dictionary.  (Locking is not done for you
automatically, which doesn't matter in this example.  `Manager''s
methods also include `Lock()', `RLock()', and `Semaphore()' to create
shared locks.)

    import time
    from multiprocessing import Pool, Manager

    def factorial(N, dictionary):
        "Compute a factorial."
        # Calculate the result
        fact = 1L
        for i in range(1, N+1):
            fact = fact * i

        # Store result in dictionary
        dictionary[N] = fact

    if __name__ == '__main__':
        p = Pool(5)
        mgr = Manager()
        d = mgr.dict()         # Create shared dictionary

        # Run tasks using the pool
        for N in range(1, 1000, 10):
            p.apply_async(factorial, (N, d))

        # Mark pool as closed -- no more tasks can be added.
        p.close()

        # Wait for tasks to exit
        p.join()

        # Output results
        for k, v in sorted(d.items()):
            print k, v

This will produce the output:

    1 1
    11 39916800
    21 51090942171709440000
    31 8222838654177922817725562880000000
    41 33452526613163807108170062053440751665152000000000
    51 15511187532873822802242430164693032110632597200169861120000...

The documentation for the multiprocessing module.

PEP 371 (http://www.python.org/dev/peps/pep-0371) - Addition of the multiprocessing package
     PEP written by Jesse Noller and Richard Oudkerk; implemented by
     Richard Oudkerk and Jesse Noller.

1.2.7 PEP 3101: Advanced String Formatting
------------------------------------------

In Python 3.0, the `%' operator is supplemented by a more powerful
string formatting method, see format().  Support for the
str.format() method has been backported to Python 2.6.

  In 2.6, both 8-bit and Unicode strings have a `.format()' method that
treats the string as a template and takes the arguments to be formatted.
The formatting template uses curly brackets (`{', `}') as special
characters:

    >>> # Substitute positional argument 0 into the string.
    >>> "User ID: {0}".format("root")
    'User ID: root'
    >>> # Use the named keyword arguments
    >>> "User ID: {uid}   Last seen: {last_login}".format(
    ...    uid="root",
    ...    last_login = "5 Mar 2008 07:20")
    'User ID: root   Last seen: 5 Mar 2008 07:20'

Curly brackets can be escaped by doubling them:

    >>> "Empty dict: {{}}".format()
    "Empty dict: {}"

Field names can be integers indicating positional arguments, such as
`{0}', `{1}', etc. or names of keyword arguments.  You can also supply
compound field names that read attributes or access dictionary keys:

    >>> import sys
    >>> print 'Platform: {0.platform}\nPython version: {0.version}'.format(sys)
    Platform: darwin
    Python version: 2.6a1+ (trunk:61261M, Mar  5 2008, 20:29:41)
    [GCC 4.0.1 (Apple Computer, Inc. build 5367)]'

    >>> import mimetypes
    >>> 'Content-type: {0[.mp4]}'.format(mimetypes.types_map)
    'Content-type: video/mp4'

Note that when using dictionary-style notation such as `[.mp4]', you
don't need to put any quotation marks around the string; it will look
up the value using `.mp4' as the key.  Strings beginning with a number
will be converted to an integer.  You can't write more complicated
expressions inside a format string.

  So far we've shown how to specify which field to substitute into the
resulting string.  The precise formatting used is also controllable by
adding a colon followed by a format specifier.  For example:

    >>> # Field 0: left justify, pad to 15 characters
    >>> # Field 1: right justify, pad to 6 characters
    >>> fmt = '{0:15} ${1:>6}'
    >>> fmt.format('Registration', 35)
    'Registration    $    35'
    >>> fmt.format('Tutorial', 50)
    'Tutorial        $    50'
    >>> fmt.format('Banquet', 125)
    'Banquet         $   125'

Format specifiers can reference other fields through nesting:

    >>> fmt = '{0:{1}}'
    >>> width = 15
    >>> fmt.format('Invoice #1234', width)
    'Invoice #1234  '
    >>> width = 35
    >>> fmt.format('Invoice #1234', width)
    'Invoice #1234                      '

The alignment of a field within the desired width can be specified:

Character            Effect
----------------------------------------------------------------------
< (default)          Left-align
>                    Right-align
^                    Center
=                    (For numeric types only) Pad after the sign.

  Format specifiers can also include a presentation type, which
controls how the value is formatted.  For example, floating-point
numbers can be formatted as a general number or in exponential notation:

    >>> '{0:g}'.format(3.75)
    '3.75'
    >>> '{0:e}'.format(3.75)
    '3.750000e+00'

A variety of presentation types are available.  Consult the 2.6
documentation for a complete list.; here's a sample:

`b'       Binary. Outputs the number in base 2.
`c'       Character. Converts the integer to the corresponding Unicode character
          before printing.
`d'       Decimal Integer. Outputs the number in base 10.
`o'       Octal format. Outputs the number in base 8.
`x'       Hex format. Outputs the number in base 16, using lower-case letters for the
          digits above 9.
`e'       Exponent notation. Prints the number in scientific notation using the
          letter 'e' to indicate the exponent.
`g'       General format. This prints the number as a fixed-point number, unless the
          number is too large, in which case it switches to 'e' exponent notation.
`n'       Number. This is the same as 'g' (for floats) or 'd' (for integers), except
          that it uses the current locale setting to insert the appropriate number
          separator characters.
`%'       Percentage. Multiplies the number by 100 and displays in fixed ('f')
          format, followed by a percent sign.

  Classes and types can define a `__format__()' method to control how
they're formatted.  It receives a single argument, the format specifier:

    def __format__(self, format_spec):
        if isinstance(format_spec, unicode):
            return unicode(str(self))
        else:
            return str(self)

There's also a format() builtin that will format a single
value.  It calls the type's `__format__()' method with the provided
specifier:

    >>> format(75.6564, '.2f')
    '75.66'

See Format String Syntax.
     The reference documentation for format fields.

PEP 3101 (http://www.python.org/dev/peps/pep-3101) - Advanced String Formatting
     PEP written by Talin. Implemented by Eric Smith.

1.2.8 PEP 3105: `print' As a Function
-------------------------------------

The `print' statement becomes the print() function in
Python 3.0.  Making print() a function makes it possible to
replace the function by doing `def print(...)' or importing a new
function from somewhere else.

  Python 2.6 has a `__future__' import that removes `print' as language
syntax, letting you use the functional form instead.  For example:

    >>> from __future__ import print_function
    >>> print('# of entries', len(dictionary), file=sys.stderr)

The signature of the new function is:

    def print(*args, sep=' ', end='\n', file=None)

The parameters are:

        * _args_: positional arguments whose values will be printed out.

        * _sep_: the separator, which will be printed between arguments.

        * _end_: the ending text, which will be printed after all of the
          arguments have been output.

        * _file_: the file object to which the output will be sent.

PEP 3105 (http://www.python.org/dev/peps/pep-3105) - Make print a function
     PEP written by Georg Brandl.

1.2.9 PEP 3110: Exception-Handling Changes
------------------------------------------

One error that Python programmers occasionally make is writing the
following code:

    try:
        ...
    except TypeError, ValueError:  # Wrong!
        ...

The author is probably trying to catch both TypeError and
ValueError exceptions, but this code actually does something
different: it will catch TypeError and bind the resulting
exception object to the local name `"ValueError"'.  The
ValueError exception will not be caught at all.  The correct code
specifies a tuple of exceptions:

    try:
        ...
    except (TypeError, ValueError):
        ...

This error happens because the use of the comma here is ambiguous: does
it indicate two different nodes in the parse tree, or a single node
that's a tuple?

  Python 3.0 makes this unambiguous by replacing the comma with the word
"as".  To catch an exception and store the exception object in the
variable `exc', you must write:

    try:
        ...
    except TypeError as exc:
        ...

Python 3.0 will only support the use of "as", and therefore interprets
the first example as catching two different exceptions.  Python 2.6
supports both the comma and "as", so existing code will continue to
work.  We therefore suggest using "as" when writing new Python code
that will only be executed with 2.6.

PEP 3110 (http://www.python.org/dev/peps/pep-3110) - Catching Exceptions in Python 3000
     PEP written and implemented by Collin Winter.

1.2.10 PEP 3112: Byte Literals
------------------------------

Python 3.0 adopts Unicode as the language's fundamental string type and
denotes 8-bit literals differently, either as `b'string'' or using a
`bytes' constructor.  For future compatibility, Python 2.6 adds `bytes'
as a synonym for the str type, and it also supports the
`b''' notation.

  The 2.6 str differs from 3.0's `bytes' type in various
ways; most notably, the constructor is completely different.  In 3.0,
`bytes([65, 66, 67])' is 3 elements long, containing the bytes
representing `ABC'; in 2.6, `bytes([65, 66, 67])' returns the 12-byte
string representing the str() of the list.

  The primary use of `bytes' in 2.6 will be to write tests of object
type such as `isinstance(x, bytes)'.  This will help the 2to3
converter, which can't tell whether 2.x code intends strings to contain
either characters or 8-bit bytes; you can now use either `bytes' or
str to represent your intention exactly, and the resulting
code will also be correct in Python 3.0.

  There's also a `__future__' import that causes all string literals to
become Unicode strings.  This means that `\u' escape sequences can be
used to include Unicode characters:

    from __future__ import unicode_literals

    s = ('\u751f\u3080\u304e\u3000\u751f\u3054'
         '\u3081\u3000\u751f\u305f\u307e\u3054')

    print len(s)               # 12 Unicode characters

At the C level, Python 3.0 will rename the existing 8-bit string type,
called PyStringObject in Python 2.x, to `PyBytesObject'.
Python 2.6 uses `#define' to support using the names `PyBytesObject()',
`PyBytes_Check()', `PyBytes_FromStringAndSize()', and all the other
functions and macros used with strings.

  Instances of the `bytes' type are immutable just as strings are.  A
new bytearray type stores a mutable sequence of bytes:

    >>> bytearray([65, 66, 67])
    bytearray(b'ABC')
    >>> b = bytearray(u'\u21ef\u3244', 'utf-8')
    >>> b
    bytearray(b'\xe2\x87\xaf\xe3\x89\x84')
    >>> b[0] = '\xe3'
    >>> b
    bytearray(b'\xe3\x87\xaf\xe3\x89\x84')
    >>> unicode(str(b), 'utf-8')
    u'\u31ef \u3244'

Byte arrays support most of the methods of string types, such as
`startswith()'/`endswith()', `find()'/`rfind()', and some of the
methods of lists, such as `append()', `pop()',  and `reverse()'.

    >>> b = bytearray('ABC')
    >>> b.append('d')
    >>> b.append(ord('e'))
    >>> b
    bytearray(b'ABCde')

There's also a corresponding C API, with
PyByteArray_FromObject(), PyByteArray_FromStringAndSize(),
and various other functions.

PEP 3112 (http://www.python.org/dev/peps/pep-3112) - Bytes literals in Python 3000
     PEP written by Jason Orendorff; backported to 2.6 by Christian
     Heimes.

1.2.11 PEP 3116: New I/O Library
--------------------------------

Python's built-in file objects support a number of methods, but
file-like objects don't necessarily support all of them.  Objects that
imitate files usually support `read()' and `write()', but they may not
support readline(), for example.  Python 3.0 introduces a
layered I/O library in the io module that separates buffering
and text-handling features from the fundamental read and write
operations.

  There are three levels of abstract base classes provided by the
io module:

   * `RawIOBase' defines raw I/O operations: `read()', `readinto()',
     `write()', `seek()', `tell()', `truncate()', and `close()'.  Most
     of the methods of this class will often map to a single system
     call.  There are also `readable()', `writable()', and `seekable()'
     methods for determining what operations a given object will allow.

     Python 3.0 has concrete implementations of this class for files and
     sockets, but Python 2.6 hasn't restructured its file and socket
     objects in this way.

   * `BufferedIOBase' is an abstract base class that buffers data in
     memory to reduce the number of system calls used, making I/O
     processing more efficient.  It supports all of the methods of
     `RawIOBase', and adds a `raw' attribute holding the underlying raw
     object.

     There are five concrete classes implementing this ABC.
     `BufferedWriter' and `BufferedReader' are for objects that support
     write-only or read-only usage that have a `seek()' method for
     random access.  `BufferedRandom' objects support read and write
     access upon the same underlying stream, and `BufferedRWPair' is
     for objects such as TTYs that have both read and write operations
     acting upon unconnected streams of data.  The `BytesIO' class
     supports reading, writing, and seeking over an in-memory buffer.

   * `TextIOBase': Provides functions for reading and writing strings
     (remember, strings will be Unicode in Python 3.0), and supporting
     universal newlines.  `TextIOBase' defines the readline()
     method and supports iteration upon objects.

     There are two concrete implementations.  `TextIOWrapper' wraps a
     buffered I/O object, supporting all of the methods for text I/O
     and adding a buffer attribute for access to the
     underlying object.  StringIO simply buffers everything
     in memory without ever writing anything to disk.

     (In Python 2.6, see io.StringIO is implemented in pure
     Python, so it's pretty slow.   You should therefore stick with the
     existing StringIO module or cStringIO for
     now.  At some point Python 3.0's io module will be
     rewritten into C for speed, and perhaps the C implementation will
     be  backported to the 2.x releases.)

  In Python 2.6, the underlying implementations haven't been
restructured to build on top of the io module's classes.  The
module is being provided to make it easier to write code that's
forward-compatible with 3.0, and to save developers the effort of
writing their own implementations of buffering and text I/O.

PEP 3116 (http://www.python.org/dev/peps/pep-3116) - New I/O
     PEP written by Daniel Stutzbach, Mike Verdone, and Guido van
     Rossum.  Code by Guido van Rossum, Georg Brandl, Walter Doerwald,
     Jeremy Hylton, Martin von Loewis, Tony Lownds, and others.

1.2.12 PEP 3118: Revised Buffer Protocol
----------------------------------------

The buffer protocol is a C-level API that lets Python types exchange
pointers into their internal representations.  A memory-mapped file can
be viewed as a buffer of characters, for example, and this lets another
module such as re.  treat memory-mapped files as a string of
characters to be searched.

  The primary users of the buffer protocol are numeric-processing
packages such as NumPy, which expose the internal representation of
arrays so that callers can write data directly into an array instead of
going through a slower API.  This PEP updates the buffer protocol in
light of experience from NumPy development, adding a number of new
features such as indicating the shape of an array or locking a memory
region.

  The most important new C API function is `PyObject_GetBuffer(PyObject
*obj, Py_buffer *view, int flags)', which takes an object and a set of
flags, and fills in the `Py_buffer' structure with information about
the object's memory representation.  Objects can use this operation to
lock memory in place while an external caller could be modifying the
contents, so there's a corresponding `PyBuffer_Release(Py_buffer
*view)' to indicate that the external caller is done.

  The _flags_ argument to PyObject_GetBuffer() specifies
constraints upon the memory returned.  Some examples are:

        * `PyBUF_WRITABLE' indicates that the memory must be writable.

        * `PyBUF_LOCK' requests a read-only or exclusive lock on the
          memory.

        * `PyBUF_C_CONTIGUOUS' and `PyBUF_F_CONTIGUOUS' requests a
          C-contiguous (last dimension varies the fastest) or
          Fortran-contiguous (first dimension varies the fastest) array
          layout.

  Two new argument codes for PyArg_ParseTuple(), `s*' and
`z*', return locked buffer objects for a parameter.

PEP 3118 (http://www.python.org/dev/peps/pep-3118) - Revising the buffer protocol
     PEP written by Travis Oliphant and Carl Banks; implemented by
     Travis Oliphant.

1.2.13 PEP 3119: Abstract Base Classes
--------------------------------------

Some object-oriented languages such as Java support interfaces,
declaring that a class has a given set of methods or supports a given
access protocol.  Abstract Base Classes (or ABCs) are an equivalent
feature for Python. The ABC support consists of an abc module
containing a metaclass called `ABCMeta', special handling of this
metaclass by the isinstance() and issubclass().
builtins, and a collection of basic ABCs that the Python developers
think will be widely useful.  Future versions of Python will probably
add more ABCs.

  Let's say you have a particular class and wish to know whether it
supports dictionary-style access.  The phrase "dictionary-style" is
vague, however.  It probably means that accessing items with `obj[1]'
works.  Does it imply that setting items with `obj[2] = value' works?
Or that the object will have `keys()', `values()', and `items()'
methods?  What about the iterative variants  such as `iterkeys()'?
See copy().  and `update()'?  Iterating over the object with
iter().?

  The Python 2.6 collections module includes a number of
different ABCs that represent these distinctions.  `Iterable' indicates
that a class defines __iter__(), and `Container' means the
class defines a __contains__().  method and therefore
supports `x in y' expressions.  The basic dictionary interface of
getting items, setting items, and `keys()', `values()', and `items()',
is defined by the `MutableMapping' ABC.

  You can derive your own classes from a particular ABC to indicate
they support that ABC's interface:

    import collections

    class Storage(collections.MutableMapping):
        ...

Alternatively, you could write the class without deriving from the
desired ABC and instead register the class by calling the ABC's
`register()' method:

    import collections

    class Storage:
        ...

    collections.MutableMapping.register(Storage)

For classes that you write, deriving from the ABC is probably clearer.
The `register()'  method is useful when you've written a new ABC that
can describe an existing type or class, or if you want to declare that
some third-party class implements an ABC.  For example, if you defined
a `PrintableType' ABC, it's legal to do:

    # Register Python's types
    PrintableType.register(int)
    PrintableType.register(float)
    PrintableType.register(str)

Classes should obey the semantics specified by an ABC, but Python can't
check this; it's up to the class author to understand the ABC's
requirements and to implement the code accordingly.

  To check whether an object supports a particular interface, you can
now write:

    def func(d):
        if not isinstance(d, collections.MutableMapping):
            raise ValueError("Mapping object expected, not %r" % d)

Don't feel that you must now begin writing lots of checks as in the
above example.  Python has a strong tradition of duck-typing, where
explicit type-checking is never done and code simply calls methods on
an object, trusting that those methods will be there and raising an
exception if they aren't.  Be judicious in checking for ABCs and only
do it where it's absolutely necessary.

  You can write your own ABCs by using `abc.ABCMeta' as the metaclass
in a class definition:

    from abc import ABCMeta, abstractmethod

    class Drawable():
        __metaclass__ = ABCMeta

        @abstractmethod
        def draw(self, x, y, scale=1.0):
            pass

        def draw_doubled(self, x, y):
            self.draw(x, y, scale=2.0)

    class Square(Drawable):
        def draw(self, x, y, scale):
            ...

In the `Drawable' ABC above, the `draw_doubled()' method renders the
object at twice its size and can be implemented in terms of other
methods described in `Drawable'.  Classes implementing this ABC
therefore don't need to provide their own implementation of
`draw_doubled()', though they can do so.  An implementation of `draw()'
is necessary, though; the ABC can't provide a useful generic
implementation.

  You can apply the `@abstractmethod' decorator to methods such as
`draw()' that must be implemented; Python will then raise an exception
for classes that don't define the method.  Note that the exception is
only raised when you actually try to create an instance of a subclass
lacking the method:

    >>> class Circle(Drawable):
    ...     pass
    ...
    >>> c = Circle()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: Can't instantiate abstract class Circle with abstract methods draw
    >>>

Abstract data attributes can be declared using the `@abstractproperty'
decorator:

    from abc import abstractproperty
    ...

    @abstractproperty
    def readonly(self):
       return self._x

Subclasses must then define a `readonly()' property.

PEP 3119 (http://www.python.org/dev/peps/pep-3119) - Introducing Abstract Base Classes
     PEP written by Guido van Rossum and Talin.  Implemented by Guido
     van Rossum.  Backported to 2.6 by Benjamin Aranguren, with Alex
     Martelli.

1.2.14 PEP 3127: Integer Literal Support and Syntax
---------------------------------------------------

Python 3.0 changes the syntax for octal (base-8) integer literals,
prefixing them with "0o" or "0O" instead of a leading zero, and adds
support for binary (base-2) integer literals, signalled by a "0b" or
"0B" prefix.

  Python 2.6 doesn't drop support for a leading 0 signalling an octal
number, but it does add support for "0o" and "0b":

    >>> 0o21, 2*8 + 1
    (17, 17)
    >>> 0b101111
    47

The oct() builtin still returns numbers prefixed with a
leading zero, and a new bin().  builtin returns the binary
representation for a number:

    >>> oct(42)
    '052'
    >>> future_builtins.oct(42)
    '0o52'
    >>> bin(173)
    '0b10101101'

The int() and long() builtins will now accept
the "0o" and "0b" prefixes when base-8 or base-2 are requested, or when
the _base_ argument is zero (signalling that the base used should be
determined from the string):

    >>> int ('0o52', 0)
    42
    >>> int('1101', 2)
    13
    >>> int('0b1101', 2)
    13
    >>> int('0b1101', 0)
    13

PEP 3127 (http://www.python.org/dev/peps/pep-3127) - Integer Literal Support and Syntax
     PEP written by Patrick Maupin; backported to 2.6 by Eric Smith.

1.2.15 PEP 3129: Class Decorators
---------------------------------

Decorators have been extended from functions to classes.  It's now
legal to write:

    @foo
    @bar
    class A:
      pass

This is equivalent to:

    class A:
      pass

    A = foo(bar(A))

PEP 3129 (http://www.python.org/dev/peps/pep-3129) - Class Decorators
     PEP written by Collin Winter.

1.2.16 PEP 3141: A Type Hierarchy for Numbers
---------------------------------------------

Python 3.0 adds several abstract base classes for numeric types
inspired by Scheme's numeric tower.  These classes were backported to
2.6 as the numbers module.

  The most general ABC is `Number'.  It defines no operations at all,
and only exists to allow checking if an object is a number by doing
`isinstance(obj, Number)'.

  `Complex' is a subclass of `Number'.  Complex numbers can undergo the
basic operations of addition, subtraction, multiplication, division,
and exponentiation, and you can retrieve the real and imaginary parts
and obtain a number's conjugate.  Python's built-in complex type is an
implementation of `Complex'.

  `Real' further derives from `Complex', and adds operations that only
work on real numbers: `floor()', `trunc()', rounding, taking the
remainder mod N, floor division, and comparisons.

  `Rational' numbers derive from `Real', have `numerator' and
`denominator' properties, and can be converted to floats.  Python 2.6
adds a simple rational-number class, `Fraction', in the
fractions module.  (It's called `Fraction' instead of `Rational'
to avoid a name clash with numbers.Rational)

  `Integral' numbers derive from `Rational', and can be shifted left
and right with `<<' and `>>', combined using bitwise operations such as
`&' and `|', and can be used as array indexes and slice boundaries.

  In Python 3.0, the PEP slightly redefines the existing builtins
round(), math.floor(), math.ceil(), and adds
a new one, see math.trunc(), that's been backported to Python
2.6.  math.trunc() rounds toward zero, returning the closest
`Integral' that's between the function's argument and zero.

PEP 3141 (http://www.python.org/dev/peps/pep-3141) - A Type Hierarchy for Numbers
     PEP written by Jeffrey Yasskin.

  Scheme's numerical tower
(http://www.gnu.org/software/guile/manual/html_node/Numerical-Tower.html#Numerical-Tower),
from the Guile manual.

  Scheme's number datatypes
(http://schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_sec_6.2)
from the R5RS Scheme specification.

1.2.16.1 The `fractions' Module
...............................

To fill out the hierarchy of numeric types, the fractions.
module provides a rational-number class.  Rational numbers store their
values as a numerator and denominator forming a fraction, and can
exactly represent numbers such as `2/3' that floating-point numbers can
only approximate.

  The `Fraction' constructor takes two `Integral' values that will be
the numerator and denominator of the resulting fraction.

    >>> from fractions import Fraction
    >>> a = Fraction(2, 3)
    >>> b = Fraction(2, 5)
    >>> float(a), float(b)
    (0.66666666666666663, 0.40000000000000002)
    >>> a+b
    Fraction(16, 15)
    >>> a/b
    Fraction(5, 3)

For converting floating-point numbers to rationals, the float type now
has an `as_integer_ratio()' method that returns the numerator and
denominator for a fraction that evaluates to the same floating-point
value:

    >>> (2.5) .as_integer_ratio()
    (5, 2)
    >>> (3.1415) .as_integer_ratio()
    (7074029114692207L, 2251799813685248L)
    >>> (1./3) .as_integer_ratio()
    (6004799503160661L, 18014398509481984L)

Note that values that can only be approximated by floating-point
numbers, such as 1./3, are not simplified to the number being
approximated; the fraction attempts to match the floating-point value
*exactly*.

  The fractions module is based upon an implementation by
Sjoerd Mullender that was in Python's `Demo/classes/' directory for a
long time.  This implementation was significantly updated by Jeffrey
Yasskin.

1.2.17 Other Language Changes
-----------------------------

Some smaller changes made to the core Python language are:

   * Directories and zip archives containing a `__main__.py' file can
     now be executed directly by passing their name to the interpreter.
     The directory or zip archive is automatically inserted as the
     first entry in sys.path.  (Suggestion and initial patch by Andy
     Chu, subsequently revised by Phillip J. Eby and Nick Coghlan;
     issue 1739468 (http://bugs.python.org/issue1739468).)

   * The hasattr() function was catching and ignoring all
     errors, under the assumption that they meant a
     __getattr__() method was failing somehow and the return
     value of hasattr() would therefore be `False'.  This
     logic shouldn't be applied to KeyboardInterrupt and
     SystemExit, however; Python 2.6 will no longer discard
     such exceptions when hasattr().  encounters them.
     (Fixed by Benjamin Peterson; issue 2196
     (http://bugs.python.org/issue2196).)

   * When calling a function using the `**' syntax to provide keyword
     arguments, you are no longer required to use a Python dictionary;
     any mapping will now work:

         >>> def f(**kw):
         ...    print sorted(kw)
         ...
         >>> ud=UserDict.UserDict()
         >>> ud['a'] = 1
         >>> ud['b'] = 'string'
         >>> f(**ud)
         ['a', 'b']

     (Contributed by Alexander Belopolsky; issue 1686487
     (http://bugs.python.org/issue1686487).)

     It's also become legal to provide keyword arguments after a
     `*args' argument to a function call.

         >>> def f(*args, **kw):
         ...     print args, kw
         ...
         >>> f(1,2,3, *(4,5,6), keyword=13)
         (1, 2, 3, 4, 5, 6) {'keyword': 13}

     Previously this would have been a syntax error.  (Contributed by
     Amaury Forgeot d'Arc; issue 3473
     (http://bugs.python.org/issue3473).)

   * A new builtin, `next(iterator, [default])' returns the next item
     from the specified iterator.  If the _default_ argument is
     supplied, it will be returned if _iterator_ has been exhausted;
     otherwise, the StopIteration exception will be raised.
     (Backported in issue 2719 (http://bugs.python.org/issue2719).)

   * Tuples now have `index()' and `count()' methods matching the list
     type's `index()' and `count()' methods:

         >>> t = (0,1,2,3,4,0,1,2)
         >>> t.index(3)
         3
         >>> t.count(0)
         2

     (Contributed by Raymond Hettinger)

   * The built-in types now have improved support for extended slicing
     syntax, accepting various combinations of `(start, stop, step)'.
     Previously, the support was partial and certain corner cases
     wouldn't work.  (Implemented by Thomas Wouters.)

   * Properties now have three attributes, `getter', `setter' and
     `deleter', that are decorators providing useful shortcuts for
     adding a getter, setter or deleter function to an existing
     property. You would use them like this:

         class C(object):
             @property
             def x(self):
                 return self._x

             @x.setter
             def x(self, value):
                 self._x = value

             @x.deleter
             def x(self):
                 del self._x

         class D(C):
             @C.x.getter
             def x(self):
                 return self._x * 2

             @x.setter
             def x(self, value):
                 self._x = value / 2

   * Several methods of the built-in set types now accept multiple
     iterables: `intersection()', `intersection_update()', `union()',
     `update()', `difference()' and `difference_update()'.

         >>> s=set('1234567890')
         >>> s.intersection('abc123', 'cdf246')  # Intersection between all inputs
         set(['2'])
         >>> s.difference('246', '789')
         set(['1', '0', '3', '5'])

     (Contributed by Raymond Hettinger.)

   * Many floating-point features were added.  The float().
     function will now turn the string `nan' into an IEEE 754 Not A
     Number value, and `+inf' and `-inf' into positive or negative
     infinity.  This works on any platform with IEEE 754 semantics.
     (Contributed by Christian Heimes; issue 1635
     (http://bugs.python.org/issue1635).)

     Other functions in the math module, `isinf()' and
     `isnan()', return true if their floating-point argument is
     infinite or Not A Number.  (issue 1640
     (http://bugs.python.org/issue1640))

     Conversion functions were added to convert floating-point numbers
     into hexadecimal strings (issue 3008
     (http://bugs.python.org/issue3008)).  These functions convert
     floats to and from a string representation without introducing
     rounding errors from the conversion between decimal and binary.
     Floats have a hex() method that returns a string
     representation, and the `float.fromhex()' method converts a string
     back into a number:

         >>> a = 3.75
         >>> a.hex()
         '0x1.e000000000000p+1'
         >>> float.fromhex('0x1.e000000000000p+1')
         3.75
         >>> b=1./3
         >>> b.hex()
         '0x1.5555555555555p-2'

   * A numerical nicety: when creating a complex number from two floats
     on systems that support signed zeros (-0 and +0), the
     complex() constructor will now preserve the sign of the
     zero.  (Fixed by Mark T. Dickinson; issue 1507
     (http://bugs.python.org/issue1507).)

   * Classes that inherit a __hash__() method from a parent
     class can set `__hash__ = None' to indicate that the class isn't
     hashable.  This will make `hash(obj)' raise a TypeError.
     and the class will not be indicated as implementing the `Hashable'
     ABC.

     You should do this when you've defined a __cmp__() or
     __eq__() method that compares objects by their value
     rather than by identity.  All objects have a default hash method
     that uses `id(obj)' as the hash value.  There's no tidy way to
     remove the __hash__() method inherited from a parent
     class, so assigning `None' was implemented as an override.  At the
     C level, extensions can set `tp_hash' to
     PyObject_HashNotImplemented().  (Fixed by Nick Coghlan and
     Amaury Forgeot d'Arc; issue 2235
     (http://bugs.python.org/issue2235).)

   * The GeneratorExit exception now subclasses
     BaseException instead of Exception.  This means
     that an exception handler that does `except Exception:' will not
     inadvertently catch GeneratorExit.  (Contributed by
     Chad Austin; issue 1537 (http://bugs.python.org/issue1537).)

   * Generator objects now have a `gi_code' attribute that refers to
     the original code object backing the generator.  (Contributed by
     Collin Winter; issue 1473257
     (http://bugs.python.org/issue1473257).)

   * The compile() built-in function now accepts keyword
     arguments as well as positional parameters.  (Contributed by
     Thomas Wouters; issue 1444529
     (http://bugs.python.org/issue1444529).)

   * The complex() constructor now accepts strings
     containing parenthesized complex numbers, meaning that
     `complex(repr(cplx))' will now round-trip values.  For example,
     `complex('(3+4j)')' now returns the value (3+4j).  (issue 1491866
     (http://bugs.python.org/issue1491866))

   * The string `translate()' method now accepts `None' as the
     translation table parameter, which is treated as the identity
     transformation.   This makes it easier to carry out operations
     that only delete characters.  (Contributed by Bengt Richter and
     implemented by Raymond Hettinger; issue 1193128
     (http://bugs.python.org/issue1193128).)

   * The built-in dir() function now checks for a
     `__dir__()' method on the objects it receives.  This method must
     return a list of strings containing the names of valid attributes
     for the object, and lets the object control the value that
     dir() produces.  Objects that have __getattr__().
     or __getattribute__().  methods can use this to
     advertise pseudo-attributes they will honor.  (issue 1591665
     (http://bugs.python.org/issue1591665))

   * Instance method objects have new attributes for the object and
     function comprising the method; the new synonym for `im_self' is
     `__self__', and `im_func' is also available as `__func__'.  The
     old names are still supported in Python 2.6, but are gone in 3.0.

   * An obscure change: when you use the locals() function
     inside a class statement, the resulting dictionary no
     longer returns free variables.  (Free variables, in this case, are
     variables referenced in the class statement  that
     aren't attributes of the class.)

1.2.17.1 Optimizations
......................

   * The warnings module has been rewritten in C.  This
     makes it possible to invoke warnings from the parser, and may also
     make the interpreter's startup faster.  (Contributed by Neal
     Norwitz and Brett Cannon; issue 1631171
     (http://bugs.python.org/issue1631171).)

   * Type objects now have a cache of methods that can reduce the work
     required to find the correct method implementation for a
     particular class; once cached, the interpreter doesn't need to
     traverse base classes to figure out the right method to call.  The
     cache is cleared if a base class or the class itself is modified,
     so the cache should remain correct even in the face of Python's
     dynamic nature.  (Original optimization implemented by Armin Rigo,
     updated for Python 2.6 by Kevin Jacobs; issue 1700288
     (http://bugs.python.org/issue1700288).)

     By default, this change is only applied to types that are included
     with the Python core.  Extension modules may not necessarily be
     compatible with this cache, so they must explicitly add
     `Py_TPFLAGS_HAVE_VERSION_TAG' to the module's `tp_flags' field to
     enable the method cache.  (To be compatible with the method cache,
     the extension module's code must not directly access and modify
     the `tp_dict' member of any of the types it implements.  Most
     modules don't do this, but it's impossible for the Python
     interpreter to determine that.  See issue 1878
     (http://bugs.python.org/issue1878) for some discussion.)

   * Function calls that use keyword arguments are significantly faster
     by doing a quick pointer comparison, usually saving the time of a
     full string comparison.  (Contributed by Raymond Hettinger, after
     an initial implementation by Antoine Pitrou; issue 1819
     (http://bugs.python.org/issue1819).)

   * All of the functions in the struct module have been
     rewritten in C, thanks to work at the Need For Speed sprint.
     (Contributed by Raymond Hettinger.)

   * Some of the standard built-in types now set a bit in their type
     objects.  This speeds up checking whether an object is a subclass
     of one of these types.  (Contributed by Neal Norwitz.)

   * Unicode strings now use faster code for detecting whitespace and
     line breaks; this speeds up the `split()' method by about 25% and
     `splitlines()' by 35%.  (Contributed by Antoine Pitrou.)  Memory
     usage is reduced by using pymalloc for the Unicode string's data.

   * The `with' statement now stores the __exit__() method
     on the stack, producing a small speedup.  (Implemented by Jeffrey
     Yasskin.)

   * To reduce memory usage, the garbage collector will now clear
     internal free lists when garbage-collecting the highest generation
     of objects.  This may return memory to the operating system sooner.

1.2.17.2 Interpreter Changes
............................

Two command-line options have been reserved for use by other Python
implementations.  The -J switch has been reserved for use by
Jython for Jython-specific options, such as switches that are passed to
the underlying JVM.  -X has been reserved for options
specific to a particular implementation of Python such as CPython,
Jython, or IronPython.  If either option is used with Python 2.6, the
interpreter will report that the option isn't currently used.

  Python can now be prevented from writing `.pyc' or `.pyo' files by
supplying the -B switch to the Python interpreter, or by
setting the PYTHONDONTWRITEBYTECODE environment variable
before running the interpreter.  This setting is available to Python
programs as the `sys.dont_write_bytecode' variable, and Python code can
change the value to modify the interpreter's behaviour.  (Contributed
by Neal Norwitz and Georg Brandl.)

  The encoding used for standard input, output, and standard error can
be specified by setting the PYTHONIOENCODING environment
variable before running the interpreter.  The value should be a string
in the form `<encoding>' or `<encoding>:<errorhandler>'.  The
_encoding_ part specifies the encoding's name, e.g. `utf-8' or
`latin-1'; the optional _errorhandler_ part specifies what to do with
characters that can't be handled by the encoding, and  should be one of
"error", "ignore", or "replace".   (Contributed by Martin von Loewis.)

1.2.18 New and Improved Modules
-------------------------------

As in every release, Python's standard library received a number of
enhancements and bug fixes.  Here's a partial list of the most notable
changes, sorted alphabetically by module name. Consult the `Misc/NEWS'
file in the source tree for a more complete list of changes, or look
through the Subversion logs for all the details.

   * The asyncore and asynchat modules are being
     actively maintained again, and a number of patches and bugfixes
     were applied.  (Maintained by Josiah Carlson; see issue 1736190
     (http://bugs.python.org/issue1736190) for one patch.)

   * The bsddb module also has a new maintainer, Jesús Cea
     Avion, and the package is now available as a standalone package.
     The web page for the package is
     www.jcea.es/programacion/pybsddb.htm
     (http://www.jcea.es/programacion/pybsddb.htm).  The plan is to
     remove the package from the standard library in Python 3.0,
     because its pace of releases is much more frequent than Python's.

     The `bsddb.dbshelve' module now uses the highest pickling protocol
     available, instead of restricting itself to protocol 1.
     (Contributed by W. Barnes.)

   * The cgi module will now read variables from the query
     string of an HTTP POST request.  This makes it possible to use
     form actions with URLs that include query strings such as
     "/cgi-bin/add.py?category=1".  (Contributed by Alexandre Fiori and
     Nubis; issue 1817 (http://bugs.python.org/issue1817).)

     The `parse_qs()' and `parse_qsl()' functions have been relocated
     from the cgi module to the urlparse module.
     The versions still available in the cgi module will
     trigger PendingDeprecationWarning messages in 2.6
     (issue 600362 (http://bugs.python.org/issue600362)).

   * The cmath module underwent extensive revision,
     contributed by Mark Dickinson and Christian Heimes.  Five new
     functions were added:

        * `polar()' converts a complex number to polar form, returning
          the modulus and argument of the complex number.

        * `rect()' does the opposite, turning a modulus, argument pair
          back into the corresponding complex number.

        * `phase()' returns the argument (also called the angle) of a
          complex number.

        * `isnan()' returns True if either the real or imaginary part
          of its argument is a NaN.

        * `isinf()' returns True if either the real or imaginary part of
          its argument is infinite.

     The revisions also improved the numerical soundness of the
     cmath module.  For all functions, the real and imaginary
     parts of the results are accurate to within a few units of least
     precision (ulps) whenever possible.  See issue 1381
     (http://bugs.python.org/issue1381) for the details.  The branch
     cuts for `asinh()', `atanh()': and `atan()' have also been
     corrected.

     The tests for the module have been greatly expanded; nearly 2000
     new test cases exercise the algebraic functions.

     On IEEE 754 platforms, the cmath module now handles
     IEEE 754 special values and floating-point exceptions in a manner
     consistent with Annex 'G' of the C99 standard.

   * A new data type in the collections module:
     `namedtuple(typename, fieldnames)' is a factory function that
     creates subclasses of the standard tuple whose fields are
     accessible by name as well as index.  For example:

         >>> var_type = collections.namedtuple('variable',
         ...             'id name type size')
         >>> # Names are separated by spaces or commas.
         >>> # 'id, name, type, size' would also work.
         >>> var_type._fields
         ('id', 'name', 'type', 'size')

         >>> var = var_type(1, 'frequency', 'int', 4)
         >>> print var[0], var.id    # Equivalent
         1 1
         >>> print var[2], var.type  # Equivalent
         int int
         >>> var._asdict()
         {'size': 4, 'type': 'int', 'id': 1, 'name': 'frequency'}
         >>> v2 = var._replace(name='amplitude')
         >>> v2
         variable(id=1, name='amplitude', type='int', size=4)

     Several places in the standard library that returned tuples have
     been modified to return `namedtuple' instances.  For example, the
     `Decimal.as_tuple()' method now returns a named tuple with `sign',
     `digits', and `exponent' fields.

     (Contributed by Raymond Hettinger.)

   * Another change to the collections module is that the
     `deque' type now supports an optional _maxlen_ parameter; if
     supplied, the deque's size will be restricted to no more than
     _maxlen_ items.  Adding more items to a full deque causes old
     items to be discarded.

         >>> from collections import deque
         >>> dq=deque(maxlen=3)
         >>> dq
         deque([], maxlen=3)
         >>> dq.append(1) ; dq.append(2) ; dq.append(3)
         >>> dq
         deque([1, 2, 3], maxlen=3)
         >>> dq.append(4)
         >>> dq
         deque([2, 3, 4], maxlen=3)

     (Contributed by Raymond Hettinger.)

   * The Cookie module's `Morsel' objects now support an
     `httponly' attribute.  In some browsers. cookies with this
     attribute set cannot be accessed or manipulated by JavaScript code.
     (Contributed by Arvin Schnell; issue 1638033
     (http://bugs.python.org/issue1638033).)

   * A new window method in the curses module, `chgat()',
     changes the display attributes for a certain number of characters
     on a single line.  (Contributed by Fabian Kreutz.)

         # Boldface text starting at y=0,x=21
         # and affecting the rest of the line.
         stdscr.chgat(0, 21, curses.A_BOLD)

     The `Textbox' class in the curses.textpad module now
     supports editing in insert mode as well as overwrite mode.  Insert
     mode is enabled by supplying a true value for the _insert_mode_
     parameter when creating the `Textbox' instance.

   * The datetime module's `strftime()' methods now support a
     `%f' format code that expands to the number of microseconds in the
     object, zero-padded on the left to six places.  (Contributed by
     Skip Montanaro; issue 1158 (http://bugs.python.org/issue1158).)

   * The decimal module was updated to version 1.66 of the
     General Decimal Specification
     (http://www2.hursley.ibm.com/decimal/decarith.html).  New features
     include some methods for some basic mathematical functions such as
     `exp()' and `log10()':

         >>> Decimal(1).exp()
         Decimal("2.718281828459045235360287471")
         >>> Decimal("2.7182818").ln()
         Decimal("0.9999999895305022877376682436")
         >>> Decimal(1000).log10()
         Decimal("3")

     The `as_tuple()' method of `Decimal' objects now returns a named
     tuple with `sign', `digits', and `exponent' fields.

     (Implemented by Facundo Batista and Mark Dickinson.  Named tuple
     support added by Raymond Hettinger.)

   * The difflib module's `SequenceMatcher' class now
     returns named tuples representing matches, with `a', `b', and
     `size' attributes.  (Contributed by Raymond Hettinger.)

   * An optional `timeout' parameter, specifying a timeout measured in
     seconds, was added to the ftplib.FTP class constructor
     as well as the `connect()' method.  (Added by Facundo Batista.)
     Also, the `FTP' class's `storbinary()' and `storlines()' now take
     an optional _callback_ parameter that will be called with each
     block of data after the data has been sent.  (Contributed by Phil
     Schwartz; issue 1221598 (http://bugs.python.org/issue1221598).)

   * The reduce() built-in function is also available in the
     functools module.  In Python 3.0, the builtin has been
     dropped and reduce() is only available from
     functools.; currently there are no plans to drop the builtin
     in the 2.x series.  (Patched by Christian Heimes; issue 1739906
     (http://bugs.python.org/issue1739906).)

   * When possible, the getpass module will now use
     `/dev/tty' to print a prompt message and read the password,
     falling back to standard error and standard input.  If the
     password may be echoed to the terminal, a warning is printed before
     the prompt is displayed.  (Contributed by Gregory P. Smith.)

   * The glob.glob() function can now return Unicode
     filenames if a Unicode path was used and Unicode filenames are
     matched within the directory.  (issue 1001604
     (http://bugs.python.org/issue1001604))

   * A new function in the heapq module, `merge(iter1,
     iter2, ...)', takes any number of iterables returning data in
     sorted order, and returns a new generator that returns the
     contents of all the iterators, also in sorted order.  For example:

         >>> list(heapq.merge([1, 3, 5, 9], [2, 8, 16]))
         [1, 2, 3, 5, 8, 9, 16]

     Another new function, `heappushpop(heap, item)', pushes _item_
     onto _heap_, then pops off and returns the smallest item.  This is
     more efficient than making a call to `heappush()' and then
     `heappop()'.

     heapq is now implemented to only use less-than
     comparison, instead of the less-than-or-equal comparison it
     previously used.  This makes heapq.'s usage of a type
     match the `list.sort()' method.  (Contributed by Raymond
     Hettinger.)

   * An optional `timeout' parameter, specifying a timeout measured in
     seconds, was added to the httplib.HTTPConnection and
     `HTTPSConnection' class constructors.  (Added by Facundo Batista.)

   * Most of the inspect module's functions, such as
     `getmoduleinfo()' and `getargs()', now return named tuples.  In
     addition to behaving like tuples, the elements of the  return value
     can also be accessed as attributes.  (Contributed by Raymond
     Hettinger.)

     Some new functions in the module include `isgenerator()',
     `isgeneratorfunction()', and `isabstract()'.

   * The itertools module gained several new functions.

     `izip_longest(iter1, iter2, ...[, fillvalue])' makes tuples from
     each of the elements; if some of the iterables are shorter than
     others, the missing values are set to _fillvalue_.  For example:

         >>> tuple(itertools.izip_longest([1,2,3], [1,2,3,4,5]))
         ((1, 1), (2, 2), (3, 3), (None, 4), (None, 5))

     `product(iter1, iter2, ..., [repeat=N])' returns the Cartesian
     product of the supplied iterables, a set of tuples containing
     every possible combination of the elements returned from each
     iterable.

         >>> list(itertools.product([1,2,3], [4,5,6]))
         [(1, 4), (1, 5), (1, 6),
          (2, 4), (2, 5), (2, 6),
          (3, 4), (3, 5), (3, 6)]

     The optional _repeat_ keyword argument is used for taking the
     product of an iterable or a set of iterables with themselves,
     repeated _N_ times.  With a single iterable argument, _N_-tuples
     are returned:

         >>> list(itertools.product([1,2], repeat=3))
         [(1, 1, 1), (1, 1, 2), (1, 2, 1), (1, 2, 2),
          (2, 1, 1), (2, 1, 2), (2, 2, 1), (2, 2, 2)]

     With two iterables, _2N_-tuples are returned.

         >>> list(itertools.product([1,2], [3,4], repeat=2))
         [(1, 3, 1, 3), (1, 3, 1, 4), (1, 3, 2, 3), (1, 3, 2, 4),
          (1, 4, 1, 3), (1, 4, 1, 4), (1, 4, 2, 3), (1, 4, 2, 4),
          (2, 3, 1, 3), (2, 3, 1, 4), (2, 3, 2, 3), (2, 3, 2, 4),
          (2, 4, 1, 3), (2, 4, 1, 4), (2, 4, 2, 3), (2, 4, 2, 4)]

     `combinations(iterable, r)' returns sub-sequences of length _r_
     from the elements of _iterable_.

         >>> list(itertools.combinations('123', 2))
         [('1', '2'), ('1', '3'), ('2', '3')]
         >>> list(itertools.combinations('123', 3))
         [('1', '2', '3')]
         >>> list(itertools.combinations('1234', 3))
         [('1', '2', '3'), ('1', '2', '4'),
          ('1', '3', '4'), ('2', '3', '4')]

     `permutations(iter[, r])' returns all the permutations of length
     _r_ of the iterable's elements.  If _r_ is not specified, it will
     default to the number of elements produced by the iterable.

         >>> list(itertools.permutations([1,2,3,4], 2))
         [(1, 2), (1, 3), (1, 4),
          (2, 1), (2, 3), (2, 4),
          (3, 1), (3, 2), (3, 4),
          (4, 1), (4, 2), (4, 3)]

     `itertools.chain(*iterables)' is an existing function in
     itertools that gained a new constructor in Python 2.6.
     `itertools.chain.from_iterable(iterable)' takes a single iterable
     that should return other iterables.  `chain()' will then return
     all the elements of the first iterable, then all the elements of
     the second, and so on.

         >>> list(itertools.chain.from_iterable([[1,2,3], [4,5,6]]))
         [1, 2, 3, 4, 5, 6]

     (All contributed by Raymond Hettinger.)

   * The logging module's `FileHandler' class and its
     subclasses `WatchedFileHandler', `RotatingFileHandler', and
     `TimedRotatingFileHandler' now have an optional _delay_ parameter
     to their constructors.  If _delay_ is true, opening of the log
     file is deferred until the first `emit()' call is made.
     (Contributed by Vinay Sajip.)

     `TimedRotatingFileHandler' also has a _utc_ constructor parameter.
     If the argument is true, UTC time will be used in determining when
     midnight occurs and in generating filenames; otherwise local time
     will be used.

   * Several new functions were added to the math module:

        * isinf() and isnan() determine whether
          a given float is a (positive or negative) infinity or a NaN
          (Not a Number), respectively.

        * copysign() copies the sign bit of an IEEE 754
          number, returning the absolute value of _x_ combined with the
          sign bit of _y_.  For example, `math.copysign(1, -0.0)'
          returns -1.0.  (Contributed by Christian Heimes.)

        * factorial() computes the factorial of a number.
          (Contributed by Raymond Hettinger; issue 2138
          (http://bugs.python.org/issue2138).)

        * fsum() adds up the stream of numbers from an
          iterable, and is careful to avoid loss of precision through
          using partial sums.  (Contributed by Jean Brouwers, Raymond
          Hettinger, and Mark Dickinson; issue 2819
          (http://bugs.python.org/issue2819).)

        * acosh(), asinh().  and atanh()
          compute the inverse hyperbolic functions.

        * log1p() returns the natural logarithm of _1+x_
          (base _e_).

        * `trunc()' rounds a number toward zero, returning the closest
          `Integral' that's between the function's argument and zero.
          Added as part of the backport of PEP 3141's type hierarchy for numbers.

   * The math module has been improved to give more
     consistent behaviour across platforms, especially with respect to
     handling of floating-point exceptions and IEEE 754 special values.

     Whenever possible, the module follows the recommendations of the
     C99 standard about 754's special values.  For example, `sqrt(-1.)'
     should now give a ValueError across almost all
     platforms, while `sqrt(float('NaN'))' should return a NaN on all
     IEEE 754 platforms.  Where Annex 'F' of the C99 standard
     recommends signaling 'divide-by-zero' or 'invalid', Python will
     raise ValueError.  Where Annex 'F' of the C99 standard
     recommends signaling 'overflow', Python will raise
     OverflowError.  (See issue 711019
     (http://bugs.python.org/issue711019) and issue 1640
     (http://bugs.python.org/issue1640).)

     (Contributed by Christian Heimes and Mark Dickinson.)

   * mmap objects now have a `rfind()' method that searches
     for a substring beginning at the end of the string and searching
     backwards.  The `find()' method also gained an _end_ parameter
     giving an index at which to stop searching.  (Contributed by John
     Lenton.)

   * The operator module gained a `methodcaller()' function
     that takes a name and an optional set of arguments, returning a
     callable that will call the named function on any arguments passed
     to it.  For example:

         >>> # Equivalent to lambda s: s.replace('old', 'new')
         >>> replacer = operator.methodcaller('replace', 'old', 'new')
         >>> replacer('old wine in old bottles')
         'new wine in new bottles'

     (Contributed by Georg Brandl, after a suggestion by Gregory
     Petrosyan.)

     The `attrgetter()' function now accepts dotted names and performs
     the corresponding attribute lookups:

         >>> inst_name = operator.attrgetter(
         ...        '__class__.__name__')
         >>> inst_name('')
         'str'
         >>> inst_name(help)
         '_Helper'

     (Contributed by Georg Brandl, after a suggestion by Barry Warsaw.)

   * The os module now wraps several new system calls.
     `fchmod(fd, mode)' and `fchown(fd, uid, gid)' change the mode and
     ownership of an opened file, and `lchmod(path, mode)' changes the
     mode of a symlink.  (Contributed by Georg Brandl and Christian
     Heimes.)

     `chflags()' and `lchflags()' are wrappers for the corresponding
     system calls (where they're available), changing the flags set on
     a file.  Constants for the flag values are defined in the
     stat module; some possible values include `UF_IMMUTABLE' to
     signal the file may not be changed and `UF_APPEND' to indicate
     that data can only be appended to the file.  (Contributed by M.
     Levinson.)

     `os.closerange(low, high)' efficiently closes all file descriptors
     from _low_ to _high_, ignoring any errors and not including _high_
     itself.  This function is now used by the subprocess.
     module to make starting processes faster.  (Contributed by Georg
     Brandl; issue 1663329 (http://bugs.python.org/issue1663329).)

   * The `os.environ' object's `clear()' method will now unset the
     environment variables using os.unsetenv() in addition
     to clearing the object's keys.  (Contributed by Martin Horcicka;
     issue 1181 (http://bugs.python.org/issue1181).)

   * The os.walk() function now has a `followlinks'
     parameter. If set to True, it will follow symlinks pointing to
     directories and visit the directory's contents.  For backward
     compatibility, the parameter's default value is false.  Note that
     the function can fall into an infinite recursion if there's a
     symlink that points to a parent directory.  (issue 1273829
     (http://bugs.python.org/issue1273829))

   * In the os.path module, the `splitext()' function has
     been changed to not split on leading period characters.  This
     produces better results when operating on Unix's dot-files.  For
     example, `os.path.splitext('.ipython')' now returns `('.ipython',
     '')' instead of `('', '.ipython')'.  (issue 1115886
     (http://bugs.python.org/issue1115886))

     A new function, `os.path.relpath(path, start='.')', returns a
     relative path from the `start' path, if it's supplied, or from the
     current working directory to the destination `path'.  (Contributed
     by Richard Barran; issue 1339796
     (http://bugs.python.org/issue1339796).)

     On Windows, see os.path.expandvars() will now expand
     environment variables given in the form "%var%", and "~user" will
     be expanded into the user's home directory path.  (Contributed by
     Josiah Carlson; issue 957650 (http://bugs.python.org/issue957650).)

   * The Python debugger provided by the pdb module gained
     a new command: "run" restarts the Python program being debugged
     and can optionally take new command-line arguments for the program.
     (Contributed by Rocky Bernstein; issue 1393667
     (http://bugs.python.org/issue1393667).)

   * The pdb.post_mortem() function, used to begin
     debugging a traceback, will now use the traceback returned by
     sys.exc_info().  if no traceback is supplied.
     (Contributed by Facundo Batista; issue 1106316
     (http://bugs.python.org/issue1106316).)

   * The pickletools module now has an `optimize()' function
     that takes a string containing a pickle and removes some unused
     opcodes, returning a shorter pickle that contains the same data
     structure.  (Contributed by Raymond Hettinger.)

   * A `get_data()' function was added to the pkgutil.
     module that returns the contents of resource files included with
     an installed Python package.  For example:

         >>> import pkgutil
         >>> print pkgutil.get_data('test', 'exception_hierarchy.txt')
         BaseException
          +-- SystemExit
          +-- KeyboardInterrupt
          +-- GeneratorExit
          +-- Exception
               +-- StopIteration
               +-- StandardError
          ...

     (Contributed by Paul Moore; issue 2439
     (http://bugs.python.org/issue2439).)

   * The `pyexpat' module's `Parser' objects now allow setting their
     `buffer_size' attribute to change the size of the buffer used to
     hold character data.  (Contributed by Achim Gaedke; issue 1137
     (http://bugs.python.org/issue1137).)

   * The Queue module now provides queue variants that
     retrieve entries in different orders.  The `PriorityQueue' class
     stores queued items in a heap and retrieves them in priority order,
     and `LifoQueue' retrieves the most recently added entries first,
     meaning that it behaves like a stack.  (Contributed by Raymond
     Hettinger.)

   * The random module's `Random' objects can now be
     pickled on a 32-bit system and unpickled on a 64-bit system, and
     vice versa.  Unfortunately, this change also means that Python
     2.6's `Random' objects can't be unpickled correctly on earlier
     versions of Python.  (Contributed by Shawn Ligocki; issue 1727780
     (http://bugs.python.org/issue1727780).)

     The new `triangular(low, high, mode)' function returns random
     numbers following a triangular distribution.   The returned values
     are between _low_ and _high_, not including _high_ itself, and
     with _mode_ as the most frequently occurring value in the
     distribution.  (Contributed by Wladmir van der Laan and Raymond
     Hettinger; issue 1681432 (http://bugs.python.org/issue1681432).)

   * Long regular expression searches carried out by the  re.
     module will check for signals being delivered, so time-consuming
     searches can now be interrupted.  (Contributed by Josh Hoyt and
     Ralf Schmitt; issue 846388 (http://bugs.python.org/issue846388).)

     The regular expression module is implemented by compiling bytecodes
     for a tiny regex-specific virtual machine.  Untrusted code could
     create malicious strings of bytecode directly and cause crashes,
     so Python 2.6 includes a verifier for the regex bytecode.
     (Contributed by Guido van Rossum from work for Google App Engine;
     issue 3487 (http://bugs.python.org/issue3487).)

   * The rlcompleter module's `Completer.complete()' method
     will now ignore exceptions triggered while evaluating a name.
     (Fixed by Lorenz Quack; issue 2250
     (http://bugs.python.org/issue2250).)

   * The sched module's `scheduler' instances now have a
     read-only `queue' attribute that returns the contents of the
     scheduler's queue, represented as a list of named tuples with the
     fields `(time, priority, action, argument)'.  (Contributed by
     Raymond Hettinger; issue 1861 (http://bugs.python.org/issue1861).)

   * The select module now has wrapper functions for the
     Linux `epoll()' and BSD `kqueue()' system calls.  `modify()'
     method was added to the existing `poll' objects;
     `pollobj.modify(fd, eventmask)' takes a file descriptor or file
     object and an event mask, modifying the recorded event mask for
     that file.  (Contributed by Christian Heimes; issue 1657
     (http://bugs.python.org/issue1657).)

   * The shutil.copytree() function now has an optional
     _ignore_ argument that takes a callable object.  This callable
     will receive each directory path and a list of the directory's
     contents, and returns a list of names that will be ignored, not
     copied.

     The shutil module also provides an `ignore_patterns()'
     function for use with this new parameter.  `ignore_patterns()'
     takes an arbitrary number of glob-style patterns and returns a
     callable that will ignore any files and directories that match any
     of these patterns.  The following example copies a directory tree,
     but skips both `.svn' directories and Emacs backup files, which
     have names ending with '~':

         shutil.copytree('Doc/library', '/tmp/library',
                         ignore=shutil.ignore_patterns('*~', '.svn'))

     (Contributed by Tarek Ziadé; issue 2663
     (http://bugs.python.org/issue2663).)

   * Integrating signal handling with GUI handling event loops like
     those used by Tkinter or GTk+ has long been a problem; most
     software ends up polling, waking up every fraction of a second to
     check if any GUI events have occurred.  The signal.
     module can now make this more efficient.  Calling
     `signal.set_wakeup_fd(fd)' sets a file descriptor to be used; when
     a signal is received, a byte is written to that file descriptor.
     There's also a C-level function, see PySignal_SetWakeupFd(),
     for setting the descriptor.

     Event loops will use this by opening a pipe to create two
     descriptors, one for reading and one for writing.  The writable
     descriptor will be passed to `set_wakeup_fd()', and the readable
     descriptor will be added to the list of descriptors monitored by
     the event loop via `select()' or `poll()'.  On receiving a signal,
     a byte will be written and the main event loop will be woken up,
     avoiding the need to poll.

     (Contributed by Adam Olsen; issue 1583
     (http://bugs.python.org/issue1583).)

     The `siginterrupt()' function is now available from Python code,
     and allows changing whether signals can interrupt system calls or
     not.  (Contributed by Ralf Schmitt.)

     The `setitimer()' and `getitimer()' functions have also been added
     (where they're available).  `setitimer()' allows setting interval
     timers that will cause a signal to be delivered to the process
     after a specified time, measured in wall-clock time, consumed
     process time, or combined process+system time.  (Contributed by
     Guilherme Polo; issue 2240 (http://bugs.python.org/issue2240).)

   * The smtplib module now supports SMTP over SSL thanks
     to the addition of the `SMTP_SSL' class. This class supports an
     interface identical to the existing `SMTP' class.  (Contributed by
     Monty Taylor.)  Both class constructors also have an optional
     `timeout' parameter that specifies a timeout for the initial
     connection attempt, measured in seconds.  (Contributed by Facundo
     Batista.)

     An implementation of the LMTP protocol ( RFC 2033
     (http://tools.ietf.org/html/rfc2033.html)) was also added to the
     module.  LMTP is used in place of SMTP when transferring e-mail
     between agents that don't manage a mail queue.  (LMTP implemented
     by Leif Hedstrom; issue 957003
     (http://bugs.python.org/issue957003).)

     `SMTP.starttls()' now complies with RFC 3207
     (http://tools.ietf.org/html/rfc3207.html) and forgets any
     knowledge obtained from the server not obtained from the TLS
     negotiation itself.  (Patch contributed by Bill Fenner; issue
     829951 (http://bugs.python.org/issue829951).)

   * The socket module now supports TIPC
     (<http://tipc.sf.net>), a high-performance non-IP-based protocol
     designed for use in clustered environments.  TIPC addresses are 4-
     or 5-tuples.  (Contributed by Alberto Bertogli; issue 1646
     (http://bugs.python.org/issue1646).)

     A new function, `create_connection()', takes an address and
     connects to it using an optional timeout value, returning the
     connected socket object.  This function also looks up the address's
     type and connects to it using IPv4 or IPv6 as appropriate.
     Changing your code to use `create_connection()' instead of
     `socket(socket.AF_INET, ...)' may be all that's required to make
     your code work with IPv6.

   * The base classes in the SocketServer module now support
     calling a `handle_timeout()' method after a span of inactivity
     specified by the server's `timeout' attribute.  (Contributed by
     Michael Pomraning.)  The `serve_forever()' method now takes an
     optional poll interval measured in seconds, controlling how often
     the server will check for a shutdown request.  (Contributed by
     Pedro Werneck and Jeffrey Yasskin; issue 742598
     (http://bugs.python.org/issue742598), issue 1193577
     (http://bugs.python.org/issue1193577).)

   * The sqlite3 module, maintained by Gerhard Haering, has
     been updated from version 2.3.2 in Python 2.5 to version 2.4.1.

   * The struct module now supports the C99 `_Bool' type,
     using the format character `'?''.  (Contributed by David Remahl.)

   * The `Popen' objects provided by the subprocess module
     now have `terminate()', `kill()', and `send_signal()' methods.  On
     Windows, `send_signal()' only supports the `SIGTERM' signal, and
     all these methods are aliases for the Win32 API function
     `TerminateProcess()'.  (Contributed by Christian Heimes.)

   * A new variable in the sys module, `float_info', is an
     object containing information derived from the `float.h' file
     about the platform's floating-point support.  Attributes of this
     object include `mant_dig' (number of digits in the mantissa),
     `epsilon' (smallest difference between 1.0 and the next largest
     value representable), and several others.  (Contributed by
     Christian Heimes; issue 1534 (http://bugs.python.org/issue1534).)

     Another new variable, `dont_write_bytecode', controls whether
     Python writes any `.pyc' or `.pyo' files on importing a module.
     If this variable is true, the compiled files are not written.  The
     variable is initially set on start-up by supplying the -B.
     switch to the Python interpreter, or by setting the
     PYTHONDONTWRITEBYTECODE environment variable before running
     the interpreter.  Python code can subsequently change the value of
     this variable to control whether bytecode files are written or not.
     (Contributed by Neal Norwitz and Georg Brandl.)

     Information about the command-line arguments supplied to the Python
     interpreter is available by reading attributes of a named tuple
     available as `sys.flags'.  For example, the `verbose' attribute is
     true if Python was executed in verbose mode, `debug' is true in
     debugging mode, etc.  These attributes are all read-only.
     (Contributed by Christian Heimes.)

     A new function, `getsizeof()', takes a Python object and returns
     the amount of memory used by the object, measured in bytes.
     Built-in objects return correct results; third-party extensions
     may not, but can define a `__sizeof__()' method to return the
     object's size.  (Contributed by Robert Schuppenies; issue 2898
     (http://bugs.python.org/issue2898).)

     It's now possible to determine the current profiler and tracer
     functions by calling sys.getprofile() and
     sys.gettrace().  (Contributed by Georg Brandl; issue 1648
     (http://bugs.python.org/issue1648).)

   * The tarfile module now supports POSIX.1-2001 (pax)
     tarfiles in addition to the POSIX.1-1988 (ustar) and GNU tar
     formats that were already supported.  The default format is GNU
     tar; specify the `format' parameter to open a file using a
     different format:

         tar = tarfile.open("output.tar", "w",
                            format=tarfile.PAX_FORMAT)

     The new `encoding' and `errors' parameters specify an encoding and
     an error handling scheme for character conversions.  `'strict'',
     `'ignore'', and `'replace'' are the three standard ways Python can
     handle errors,; `'utf-8'' is a special value that replaces bad
     characters with their UTF-8 representation.  (Character
     conversions occur because the PAX format supports Unicode
     filenames, defaulting to UTF-8 encoding.)

     The `TarFile.add()' method now accepts an `exclude' argument that's
     a function that can be used to exclude certain filenames from an
     archive.  The function must take a filename and return true if the
     file should be excluded or false if it should be archived.  The
     function is applied to both the name initially passed to `add()'
     and to the names of files in recursively-added directories.

     (All changes contributed by Lars Gustäbel).

   * An optional `timeout' parameter was added to the
     telnetlib.Telnet class constructor, specifying a timeout
     measured in seconds.  (Added by Facundo Batista.)

   * The tempfile.NamedTemporaryFile class usually deletes
     the temporary file it created when the file is closed.  This
     behaviour can now be changed by passing `delete=False' to the
     constructor.  (Contributed by Damien Miller; issue 1537850
     (http://bugs.python.org/issue1537850).)

     A new class, `SpooledTemporaryFile', behaves like a temporary file
     but stores its data in memory until a maximum size is exceeded.
     On reaching that limit, the contents will be written to an on-disk
     temporary file.  (Contributed by Dustin J. Mitchell.)

     The `NamedTemporaryFile' and `SpooledTemporaryFile' classes both
     work as context managers, so you can write `with
     tempfile.NamedTemporaryFile() as tmp: ...'.  (Contributed by
     Alexander Belopolsky; issue 2021
     (http://bugs.python.org/issue2021).)

   * The test.test_support module gained a number of
     context managers useful for writing tests.
     `EnvironmentVarGuard()' is a context manager that temporarily
     changes environment variables and automatically restores them to
     their old values.

     Another context manager, `TransientResource', can surround calls
     to resources that may or may not be available; it will catch and
     ignore a specified list of exceptions.  For example, a network
     test may ignore certain failures when connecting to an external
     web site:

         with test_support.TransientResource(IOError,
                                         errno=errno.ETIMEDOUT):
             f = urllib.urlopen('https://sf.net')
             ...

     Finally, `check_warnings()' resets the `warning' module's warning
     filters and returns an object that will record all warning
     messages triggered (issue 3781 (http://bugs.python.org/issue3781)):

         with test_support.check_warnings() as wrec:
             warnings.simplefilter("always")
             # ... code that triggers a warning ...
             assert str(wrec.message) == "function is outdated"
             assert len(wrec.warnings) == 1, "Multiple warnings raised"

     (Contributed by Brett Cannon.)

   * The textwrap module can now preserve existing
     whitespace at the beginnings and ends of the newly-created lines
     by specifying `drop_whitespace=False' as an argument:

         >>> S = """This  sentence  has a bunch   of
         ...   extra   whitespace."""
         >>> print textwrap.fill(S, width=15)
         This  sentence
         has a bunch
         of    extra
         whitespace.
         >>> print textwrap.fill(S, drop_whitespace=False, width=15)
         This  sentence
           has a bunch
            of    extra
            whitespace.
         >>>

     (Contributed by Dwayne Bailey; issue 1581073
     (http://bugs.python.org/issue1581073).)

   * The threading module API is being changed to use
     properties such as `daemon' instead of `setDaemon()' and
     `isDaemon()' methods, and some methods have been renamed to use
     underscores instead of camel-case; for example, the
     `activeCount()' method is renamed to `active_count()'.  Both the
     2.6 and 3.0 versions of the module support the same properties and
     renamed methods, but don't remove the old methods.  No date has
     been set for the deprecation of the old APIs in Python 3.x; the
     old APIs won't be removed in any 2.x version.  (Carried out by
     several people, most notably Benjamin Peterson.)

     The threading module's `Thread' objects gained an
     `ident' property that returns the thread's identifier, a nonzero
     integer.  (Contributed by Gregory P. Smith; issue 2871
     (http://bugs.python.org/issue2871).)

   * The timeit module now accepts callables as well as
     strings for the statement being timed and for the setup code.  Two
     convenience functions were added for creating `Timer' instances:
     `repeat(stmt, setup, time, repeat, number)' and `timeit(stmt,
     setup, time, number)' create an instance and call the
     corresponding method. (Contributed by Erik Demaine; issue 1533909
     (http://bugs.python.org/issue1533909).)

   * The Tkinter module now accepts lists and tuples for
     options, separating the elements by spaces before passing the
     resulting value to Tcl/Tk.  (Contributed by Guilherme Polo; issue
     2906 (http://bugs.python.org/issue2906).)

   * The turtle module for turtle graphics was greatly
     enhanced by Gregor Lingl.  New features in the module include:

        * Better animation of turtle movement and rotation.

        * Control over turtle movement using the new `delay()',
          `tracer()', and `speed()' methods.

        * The ability to set new shapes for the turtle, and to define a
          new coordinate system.

        * Turtles now have an `undo()' method that can roll back
          actions.

        * Simple support for reacting to input events such as mouse and
          keyboard activity, making it possible to write simple games.

        * A `turtle.cfg' file can be used to customize the starting
          appearance of the turtle's screen.

        * The module's docstrings can be replaced by new docstrings
          that have been translated into another language.

     (issue 1513695 (http://bugs.python.org/issue1513695))

   * An optional `timeout' parameter was added to the
     urllib.urlopen() function and the `urllib.ftpwrapper' class
     constructor, as well as the urllib2.urlopen().
     function.  The parameter specifies a timeout measured in seconds.
     For example:

         >>> u = urllib2.urlopen("http://slow.example.com",
                                 timeout=3)
         Traceback (most recent call last):
           ...
         urllib2.URLError: <urlopen error timed out>
         >>>

     (Added by Facundo Batista.)

   * The Unicode database provided by the unicodedata module
     has been updated to version 5.1.0.  (Updated by Martin von Loewis;
     issue 3811 (http://bugs.python.org/issue3811).)

   * The warnings module's `formatwarning()' and
     `showwarning()' gained an optional _line_ argument that can be
     used to supply the line of source code.  (Added as part of issue
     1631171 (http://bugs.python.org/issue1631171), which re-implemented
     part of the warnings module in C code.)

     A new function, `catch_warnings()', is a context manager intended
     for testing purposes that lets you temporarily modify the warning
     filters and then restore their original values (issue 3781
     (http://bugs.python.org/issue3781)).

   * The XML-RPC SimpleXMLRPCServer and
     DocXMLRPCServer.  classes can now be prevented from
     immediately opening and binding to their socket by passing True as
     the `bind_and_activate' constructor parameter.  This can be used
     to modify the instance's `allow_reuse_address' attribute before
     calling the `server_bind()' and `server_activate()' methods to
     open the socket and begin listening for connections.  (Contributed
     by Peter Parente; issue 1599845
     (http://bugs.python.org/issue1599845).)

     SimpleXMLRPCServer also has a `_send_traceback_header'
     attribute; if true, the exception and formatted traceback are
     returned as HTTP headers "X-Exception" and "X-Traceback".  This
     feature is for debugging purposes only and should not be used on
     production servers because the tracebacks might reveal passwords
     or other sensitive information.  (Contributed by Alan McIntyre as
     part of his project for Google's Summer of Code 2007.)

   * The xmlrpclib module no longer automatically converts
     datetime.date and datetime.time to the
     `xmlrpclib.DateTime' type; the conversion semantics were not
     necessarily correct for all applications.  Code using
     xmlrpclib should convert `date' and time.
     instances. (issue 1330538 (http://bugs.python.org/issue1330538))
     The code can also handle dates before 1900 (contributed by Ralf
     Schmitt; issue 2014 (http://bugs.python.org/issue2014)) and 64-bit
     integers represented by using `<i8>' in XML-RPC responses
     (contributed by Riku Lindblad; issue 2985
     (http://bugs.python.org/issue2985)).

   * The zipfile module's `ZipFile' class now has
     `extract()' and `extractall()' methods that will unpack a single
     file or all the files in the archive to the current directory, or
     to a specified directory:

         z = zipfile.ZipFile('python-251.zip')

         # Unpack a single file, writing it relative
         # to the /tmp directory.
         z.extract('Python/sysmodule.c', '/tmp')

         # Unpack all the files in the archive.
         z.extractall()

     (Contributed by Alan McIntyre; issue 467924
     (http://bugs.python.org/issue467924).)

     The open(), `read()' and `extract()' methods can now
     take either a filename or a `ZipInfo' object.  This is useful when
     an archive accidentally contains a duplicated filename.
     (Contributed by Graham Horler; issue 1775025
     (http://bugs.python.org/issue1775025).)

     Finally, see zipfile now supports using Unicode filenames
     for archived files.  (Contributed by Alexey Borzenkov; issue
     1734346 (http://bugs.python.org/issue1734346).)

1.2.18.1 The `ast' module
.........................

The ast module provides an Abstract Syntax Tree
representation of Python code, and Armin Ronacher contributed a set of
helper functions that perform a variety of common tasks.  These will be
useful for HTML templating packages, code analyzers, and similar tools
that process Python code.

  The `parse()' function takes an expression and returns an AST.  The
`dump()' function outputs a representation of a tree, suitable for
debugging:

    import ast

    t = ast.parse("""
    d = {}
    for i in 'abcdefghijklm':
        d[i + i] = ord(i) - ord('a') + 1
    print d
    """)
    print ast.dump(t)

This outputs a deeply nested tree:

    Module(body=[
      Assign(targets=[
        Name(id='d', ctx=Store())
       ], value=Dict(keys=[], values=[]))
      For(target=Name(id='i', ctx=Store()),
          iter=Str(s='abcdefghijklm'), body=[
        Assign(targets=[
          Subscript(value=
            Name(id='d', ctx=Load()),
              slice=
              Index(value=
                BinOp(left=Name(id='i', ctx=Load()), op=Add(),
                 right=Name(id='i', ctx=Load()))), ctx=Store())
         ], value=
         BinOp(left=
          BinOp(left=
           Call(func=
            Name(id='ord', ctx=Load()), args=[
              Name(id='i', ctx=Load())
             ], keywords=[], starargs=None, kwargs=None),
           op=Sub(), right=Call(func=
            Name(id='ord', ctx=Load()), args=[
              Str(s='a')
             ], keywords=[], starargs=None, kwargs=None)),
           op=Add(), right=Num(n=1)))
        ], orelse=[])
       Print(dest=None, values=[
         Name(id='d', ctx=Load())
       ], nl=True)
     ])

The `literal_eval()' method takes a string or an AST representing a
literal expression, parses and evaluates it, and returns the resulting
value.  A literal expression is a Python expression containing only
strings, numbers, dictionaries, etc. but no statements or function
calls.  If you need to evaluate an expression but cannot accept the
security risk of using an eval() call, `literal_eval()'
will handle it safely:

    >>> literal = '("a", "b", {2:4, 3:8, 1:2})'
    >>> print ast.literal_eval(literal)
    ('a', 'b', {1: 2, 2: 4, 3: 8})
    >>> print ast.literal_eval('"a" + "b"')
    Traceback (most recent call last):
      ...
    ValueError: malformed string

The module also includes `NodeVisitor' and `NodeTransformer' classes
for traversing and modifying an AST, and functions for common
transformations such as changing line numbers.

1.2.18.2 The `future_builtins' module
.....................................

Python 3.0 makes many changes to the repertoire of built-in functions,
and most of the changes can't be introduced in the Python 2.x series
because they would break compatibility.  The future_builtins.
module provides versions of these built-in functions that can be
imported when writing 3.0-compatible code.

  The functions in this module currently include:

   * `ascii(obj)': equivalent to repr().  In Python 3.0,
     see repr() will return a Unicode string, while `ascii()'
     will return a pure ASCII bytestring.

   * `filter(predicate, iterable)', `map(func, iterable1, ...)': the
     3.0 versions return iterators, unlike the 2.x builtins which
     return lists.

   * `hex(value)', `oct(value)': instead of calling the
     __hex__() or __oct__() methods, these versions
     will call the __index__() method and convert the
     result to hexadecimal or octal.  oct() will use the
     new `0o' notation for its result.

1.2.18.3 The `json' module: JavaScript Object Notation
......................................................

The new json module supports the encoding and decoding of
Python types in JSON (Javascript Object Notation). JSON is a
lightweight interchange format often used in web applications. For more
information about JSON, see <http://www.json.org>.

  json comes with support for decoding and encoding most
built-in Python types. The following example encodes and decodes a
dictionary:

    >>> import json
    >>> data = {"spam" : "foo", "parrot" : 42}
    >>> in_json = json.dumps(data) # Encode the data
    >>> in_json
    '{"parrot": 42, "spam": "foo"}'
    >>> json.loads(in_json) # Decode into a Python object
    {"spam" : "foo", "parrot" : 42}

It's also possible to write your own decoders and encoders to support
more types. Pretty-printing of the JSON strings is also supported.

  See json. (originally called simplejson) was written by Bob
Ippolito.

1.2.18.4 The `plistlib' module: A Property-List Parser
......................................................

The `.plist' format is commonly used on Mac OS X to store basic data
types (numbers, strings, lists, and dictionaries) by serializing them
into an XML-based format.  It resembles the XML-RPC serialization of
data types.

  Despite being primarily used on Mac OS X, the format has nothing
Mac-specific about it and the Python implementation works on any
platform that Python supports, so the plistlib module has
been promoted to the standard library.

  Using the module is simple:

    import sys
    import plistlib
    import datetime

    # Create data structure
    data_struct = dict(lastAccessed=datetime.datetime.now(),
                       version=1,
                       categories=('Personal','Shared','Private'))

    # Create string containing XML.
    plist_str = plistlib.writePlistToString(data_struct)
    new_struct = plistlib.readPlistFromString(plist_str)
    print data_struct
    print new_struct

    # Write data structure to a file and read it back.
    plistlib.writePlist(data_struct, '/tmp/customizations.plist')
    new_struct = plistlib.readPlist('/tmp/customizations.plist')

    # read/writePlist accepts file-like objects as well as paths.
    plistlib.writePlist(data_struct, sys.stdout)

1.2.18.5 ctypes Enhancements
............................

Thomas Heller continued to maintain and enhance the ctypes.
module.

  ctypes now supports a `c_bool' datatype that represents
the C99 `bool' type.  (Contributed by David Remahl; issue 1649190
(http://bugs.python.org/issue1649190).)

  The ctypes string, buffer and array types have improved
support for extended slicing syntax, where various combinations of
`(start, stop, step)' are supplied.  (Implemented by Thomas Wouters.)

  All ctypes data types now support `from_buffer()' and
`from_buffer_copy()' methods that create a ctypes instance based on a
provided buffer object.  `from_buffer_copy()' copies the contents of
the object, while `from_buffer()' will share the same memory area.

  A new calling convention tells ctypes to clear the `errno'
or Win32 LastError variables at the outset of each wrapped call.
(Implemented by Thomas Heller; issue 1798
(http://bugs.python.org/issue1798).)

  You can now retrieve the Unix `errno' variable after a function call.
When creating a wrapped function, you can supply `use_errno=True' as a
keyword parameter to the `DLL()' function and then call the
module-level methods `set_errno()' and `get_errno()' to set and
retrieve the error value.

  The Win32 LastError variable is similarly supported by the `DLL()',
`OleDLL()', and `WinDLL()' functions.  You supply `use_last_error=True'
as a keyword parameter and then call the module-level methods
`set_last_error()' and `get_last_error()'.

  The `byref()' function, used to retrieve a pointer to a ctypes
instance, now has an optional _offset_ parameter that is a byte count
that will be added to the returned pointer.

1.2.18.6 Improved SSL Support
.............................

Bill Janssen made extensive improvements to Python 2.6's support for
the Secure Sockets Layer by adding a new module, see ssl, that's
built atop the OpenSSL (http://www.openssl.org/) library.  This new
module provides more control over the protocol negotiated, the X.509
certificates used, and has better support for writing SSL servers (as
opposed to clients) in Python.  The existing SSL support in the
socket module hasn't been removed and continues to work, though
it will be removed in Python 3.0.

  To use the new module, you must first create a TCP connection in the
usual way and then pass it to the ssl.wrap_socket().
function.  It's possible to specify whether a certificate is required,
and to obtain certificate info by calling the `getpeercert()' method.

  The documentation for the ssl module.

1.2.19 Deprecations and Removals
--------------------------------

   * String exceptions have been removed.  Attempting to use them
     raises a TypeError.

   * Changes to the `Exception' interface as dictated by PEP 352
     (http://www.python.org/dev/peps/pep-0352) continue to be made.
     For 2.6, the `message' attribute is being deprecated in favor of
     the `args' attribute.

   * (3.0-warning mode) Python 3.0 will feature a reorganized standard
     library that will drop many outdated modules and rename others.
     Python 2.6 running in 3.0-warning mode will warn about these
     modules when they are imported.

     The list of deprecated modules is: `audiodev', `bgenlocations',
     see buildtools, `bundlebuilder', `Canvas', see compiler,
     see dircache, dl, fpformat,
     gensuitemodule, `ihooks', see imageop, imgfile,
     `linuxaudiodev', see mhlib, mimetools,
     multifile, new, `pure', see statvfs,
     sunaudiodev, `test.testall', and `toaiff'.

   * The `gopherlib' module has been removed.

   * The MimeWriter module and mimify module
     have been deprecated; use the email.  package instead.

   * The md5 module has been deprecated; use the
     hashlib module instead.

   * The posixfile module has been deprecated; see
     fcntl.lockf().  provides better locking.

   * The popen2 module has been deprecated; use the
     subprocess.  module.

   * The `rgbimg' module has been removed.

   * The sets module has been deprecated; it's better to
     use the built-in set and frozenset types.

   * The sha module has been deprecated; use the
     hashlib module instead.

1.2.20 Build and C API Changes
------------------------------

Changes to Python's build process and to the C API include:

   * Python now must be compiled with C89 compilers (after 19 years!).
     This means that the Python source tree has dropped its own
     implementations of `memmove()' and `strerror()', which are in the
     C89 standard library.

   * Python 2.6 can be built with Microsoft Visual Studio 2008 (version
     9.0), and this is the new default compiler.  See the `PCbuild'
     directory for the build files.  (Implemented by Christian Heimes.)

   * On Mac OS X, Python 2.6 can be compiled as a 4-way universal build.
     The *configure* script can take a
     `--with-universal-archs=[32-bit|64-bit|all]' switch, controlling
     whether the binaries are built for 32-bit architectures (x86,
     PowerPC), 64-bit (x86-64 and PPC-64), or both.  (Contributed by
     Ronald Oussoren.)

   * The BerkeleyDB module now has a C API object, available as
     `bsddb.db.api'.   This object can be used by other C extensions
     that wish to use the bsddb module for their own
     purposes.  (Contributed by Duncan Grisby.)

   * The new buffer interface, previously described in the PEP 3118 section,
     adds PyObject_GetBuffer() and
     PyBuffer_Release(), as well as a few other functions.

   * Python's use of the C stdio library is now thread-safe, or at least
     as thread-safe as the underlying library is.  A long-standing
     potential bug occurred if one thread closed a file object while
     another thread was reading from or writing to the object.  In 2.6
     file objects have a reference count, manipulated by the
     PyFile_IncUseCount() and PyFile_DecUseCount().
     functions.  File objects can't be closed unless the reference count
     is zero.  PyFile_IncUseCount() should be called while
     the GIL is still held, before carrying out an I/O operation using
     the `FILE *' pointer, and PyFile_DecUseCount() should
     be called immediately after the GIL is re-acquired.  (Contributed
     by Antoine Pitrou and Gregory P. Smith.)

   * Importing modules simultaneously in two different threads no longer
     deadlocks; it will now raise an ImportError.  A new API
     function, see PyImport_ImportModuleNoBlock(), will look for
     a module in `sys.modules' first, then try to import it after
     acquiring an import lock.  If the import lock is held by another
     thread, an ImportError is raised.  (Contributed by
     Christian Heimes.)

   * Several functions return information about the platform's
     floating-point support.  PyFloat_GetMax() returns the
     maximum representable floating point value, and
     PyFloat_GetMin() returns the minimum positive value.
     PyFloat_GetInfo() returns an object containing more
     information from the `float.h' file, such as `"mant_dig"' (number
     of digits in the mantissa), `"epsilon"' (smallest difference
     between 1.0 and the next largest value representable), and several
     others.  (Contributed by Christian Heimes; issue 1534
     (http://bugs.python.org/issue1534).)

   * C functions and methods that use PyComplex_AsCComplex()
     will now accept arguments that have a __complex__()
     method.  In particular, the functions in the cmath.
     module will now accept objects with this method.  This is a
     backport of a Python 3.0 change.  (Contributed by Mark Dickinson;
     issue 1675423 (http://bugs.python.org/issue1675423).)

   * Python's C API now includes two functions for case-insensitive
     string comparisons, `PyOS_stricmp(char*, char*)' and
     `PyOS_strnicmp(char*, char*, Py_ssize_t)'.  (Contributed by
     Christian Heimes; issue 1635 (http://bugs.python.org/issue1635).)

   * Many C extensions define their own little macro for adding
     integers and strings to the module's dictionary in the `init*'
     function.  Python 2.6 finally defines standard macros for adding
     values to a module, see PyModule_AddStringMacro.  and
     `PyModule_AddIntMacro()'.  (Contributed by Christian Heimes.)

   * Some macros were renamed in both 3.0 and 2.6 to make it clearer
     that they are macros, not functions.  `Py_Size()' became
     `Py_SIZE()', `Py_Type()' became `Py_TYPE()', and `Py_Refcnt()'
     became `Py_REFCNT()'.  The mixed-case macros are still available
     in Python 2.6 for backward compatibility.  (issue 1629
     (http://bugs.python.org/issue1629))

   * Distutils now places C extensions it builds in a different
     directory when running on a debug version of Python.  (Contributed
     by Collin Winter; issue 1530959
     (http://bugs.python.org/issue1530959).)

   * Several basic data types, such as integers and strings, maintain
     internal free lists of objects that can be re-used.  The data
     structures for these free lists now follow a naming convention: the
     variable is always named `free_list', the counter is always named
     `numfree', and a macro `Py<typename>_MAXFREELIST' is always
     defined.

   * A new Makefile target, "make patchcheck", prepares the Python
     source tree for making a patch: it fixes trailing whitespace in
     all modified `.py' files, checks whether the documentation has
     been changed, and reports whether the `Misc/ACKS' and `Misc/NEWS'
     files have been updated.  (Contributed by Brett Cannon.)

     Another new target, "make profile-opt", compiles a Python binary
     using GCC's profile-guided optimization.  It compiles Python with
     profiling enabled, runs the test suite to obtain a set of profiling
     results, and then compiles using these results for optimization.
     (Contributed by Gregory P. Smith.)

1.2.20.1 Port-Specific Changes: Windows
.......................................

   * The support for Windows 95, 98, ME and NT4 has been dropped.
     Python 2.6 requires at least Windows 2000 SP4.

   * The new default compiler on Windows is Visual Studio 2008 (version
     9.0). The build directories for Visual Studio 2003 (version 7.1)
     and 2005 (version 8.0) were moved into the PC/ directory. The new
     `PCbuild' directory supports cross compilation for X64, debug
     builds and Profile Guided Optimization (PGO). PGO builds are
     roughly 10% faster than normal builds.  (Contributed by Christian
     Heimes with help from Amaury Forgeot d'Arc and Martin von Loewis.)

   * The msvcrt module now supports both the normal and
     wide char variants of the console I/O API.  The `getwch()'
     function reads a keypress and returns a Unicode value, as does the
     `getwche()' function.  The `putwch()' function takes a Unicode
     character and writes it to the console.  (Contributed by Christian
     Heimes.)

   * os.path.expandvars() will now expand environment
     variables in the form "%var%", and "~user" will be expanded into
     the user's home directory path.  (Contributed by Josiah Carlson;
     issue 957650 (http://bugs.python.org/issue957650).)

   * The socket module's socket objects now have an
     `ioctl()' method that provides a limited interface to the
     `WSAIoctl()' system interface.

   * The _winreg module now has a function,
     `ExpandEnvironmentStrings()', that expands environment variable
     references such as `%NAME%' in an input string.  The handle
     objects provided by this module now support the context protocol,
     so they can be used in with statements. (Contributed
     by Christian Heimes.)

     _winreg also has better support for x64 systems,
     exposing the `DisableReflectionKey()', `EnableReflectionKey()',
     and `QueryReflectionKey()' functions, which enable and disable
     registry reflection for 32-bit processes running on 64-bit systems.
     (issue 1753245 (http://bugs.python.org/issue1753245))

   * The msilib module's `Record' object gained
     `GetInteger()' and `GetString()' methods that return field values
     as an integer or a string.  (Contributed by Floris Bruynooghe;
     issue 2125 (http://bugs.python.org/issue2125).)

1.2.20.2 Port-Specific Changes: Mac OS X
........................................

   * When compiling a framework build of Python, you can now specify the
     framework name to be used by providing the
     `--with-framework-name=' option to the *configure* script.

   * The `macfs' module has been removed.  This in turn required the
     macostools.touched() function to be removed because it
     depended on the `macfs' module.  (issue 1490190
     (http://bugs.python.org/issue1490190))

   * Many other Mac OS modules have been deprecated and will removed in
     Python 3.0: `_builtinSuites', see aepack, aetools,
     aetypes, applesingle, `appletrawmain',
     `appletrunner', `argvemulator', `Audio_mac', see autoGIL,
     `Carbon', see cfmfile, `CodeWarrior', see ColorPicker,
     EasyDialogs, `Explorer', `Finder', see FrameWork,
     findertools, ic, `icglue', see icopen,
     macerrors, MacOS, `macfs', see macostools,
     see macresource, MiniAEFrame, Nav,
     `Netscape', `OSATerminology', `pimp', see PixMapWrapper,
     `StdSuites', `SystemEvents', `Terminal', and
     `terminalcommand'.

1.2.20.3 Port-Specific Changes: IRIX
....................................

A number of old IRIX-specific modules were deprecated and will be
removed in Python 3.0: al and AL, cd,
`cddb', `cdplayer', `CL' and `cl', see DEVICE, `ERRNO', `FILE',
see FL and fl, flp, fm, `GET',
`GLWS', see GL and gl, `IN', `IOCTL', see jpeg,
`panelparser', `readcd', `SV' and `sv', `torgb', see videoreader,
and `WAIT'.

1.2.21 Porting to Python 2.6
----------------------------

This section lists previously described changes and other bugfixes that
may require changes to your code:

   * Classes that aren't supposed to be hashable should set `__hash__ =
     None' in their definitions to indicate the fact.

   * String exceptions have been removed.  Attempting to use them
     raises a TypeError.

   * The __init__() method of collections.deque.
     now clears any existing contents of the deque before adding
     elements from the iterable.  This change makes the behavior match
     `list.__init__()'.

   * object.__init__() previously accepted arbitrary
     arguments and keyword arguments, ignoring them.  In Python 2.6,
     this is no longer allowed and will result in a TypeError.
     This will affect __init__() methods that end up
     calling the corresponding method on object. (perhaps
     through using super()).  See issue 1683368
     (http://bugs.python.org/issue1683368) for discussion.

   * The `Decimal' constructor now accepts leading and trailing
     whitespace when passed a string.  Previously it would raise an
     `InvalidOperation' exception.  On the other hand, the
     `create_decimal()' method of `Context' objects now explicitly
     disallows extra whitespace, raising a `ConversionSyntax' exception.

   * Due to an implementation accident, if you passed a file path to
     the built-in  __import__() function, it would actually
     import the specified file.  This was never intended to work,
     however, and the implementation now explicitly checks for this
     case and raises an ImportError.

   * C API: the PyImport_Import() and
     PyImport_ImportModule().  functions now default to absolute
     imports, not relative imports.  This will affect C extensions that
     import other modules.

   * C API: extension data types that shouldn't be hashable should
     define their `tp_hash' slot to
     PyObject_HashNotImplemented().

   * The socket module exception socket.error.
     now inherits from IOError.  Previously it wasn't a
     subclass of StandardError but now it is, through
     IOError.  (Implemented by Gregory P. Smith; issue 1706815
     (http://bugs.python.org/issue1706815).)

   * The xmlrpclib module no longer automatically converts
     datetime.date and datetime.time to the
     `xmlrpclib.DateTime' type; the conversion semantics were not
     necessarily correct for all applications.  Code using
     xmlrpclib should convert `date' and time.
     instances. (issue 1330538 (http://bugs.python.org/issue1330538))

   * (3.0-warning mode) The `Exception' class now warns when accessed
     using slicing or index access; having `Exception' behave like a
     tuple is being phased out.

   * (3.0-warning mode) inequality comparisons between two dictionaries
     or two objects that don't implement comparison methods are reported
     as warnings.  `dict1 == dict2' still works, but `dict1 < dict2' is
     being phased out.

     Comparisons between cells, which are an implementation detail of
     Python's scoping rules, also cause warnings because such
     comparisons are forbidden entirely in 3.0.

1.2.22 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Georg Brandl, Steve Brown, Nick Coghlan, Ralph Corderoy, Jim
Jewett, Kent Johnson, Chris Lambacher,  Martin Michlmayr, Antoine
Pitrou, Brian Warner.

1.3 What's New in Python 2.5
============================

     Author: A.M. Kuchling

  This article explains the new features in Python 2.5.  The final
release of Python 2.5 is scheduled for August 2006; PEP 356
(http://www.python.org/dev/peps/pep-0356) describes the planned release
schedule.

  The changes in Python 2.5 are an interesting mix of language and
library improvements. The library enhancements will be more important
to Python's user community, I think, because several widely-useful
packages were added.  New modules include ElementTree for XML
processing (`xml.etree'), the SQLite database module (`sqlite'), and
the ctypes.  module for calling C functions.

  The language changes are of middling significance.  Some pleasant new
features were added, but most of them aren't features that you'll use
every day.  Conditional expressions were finally added to the language
using a novel syntax; see section PEP 308; Conditional Expressions.
The new 'with.' statement will make
writing cleanup code easier (section PEP 343; The 'with' statement).
Values can now be passed into generators (section
PEP 342; New Generator Features).  Imports are now visible
as either absolute or relative (section PEP 328; Absolute and Relative Imports).
Some corner cases of exception handling are
handled better (section PEP 341; Unified try/except/finally).
All these improvements are worthwhile, but they're improvements
to one specific language feature or another; none of them are broad
modifications to Python's semantics.

  As well as the language and library additions, other improvements and
bugfixes were made throughout the source tree.  A search through the
SVN change logs finds there were 353 patches applied and 458 bugs fixed
between Python 2.4 and 2.5.  (Both figures are likely to be
underestimates.)

  This article doesn't try to be a complete specification of the new
features; instead changes are briefly introduced using helpful
examples.  For full details, you should always refer to the
documentation for Python 2.5 at <http://docs.python.org>. If you want
to understand the complete implementation and design rationale, refer
to the PEP for a particular new feature.

  Comments, suggestions, and error reports for this document are
welcome; please e-mail them to the author or open a bug in the Python
bug tracker.

1.3.1 PEP 308: Conditional Expressions
--------------------------------------

For a long time, people have been requesting a way to write conditional
expressions, which are expressions that return value A or value B
depending on whether a Boolean value is true or false.  A conditional
expression lets you write a single assignment statement that has the
same effect as the following:

    if condition:
        x = true_value
    else:
        x = false_value

There have been endless tedious discussions of syntax on both
python-dev and comp.lang.python.  A vote was even held that found the
majority of voters wanted conditional expressions in some form, but
there was no syntax that was preferred by a clear majority. Candidates
included C's `cond ? true_v : false_v', `if cond then true_v else
false_v', and 16 other variations.

  Guido van Rossum eventually chose a surprising syntax:

    x = true_value if condition else false_value

Evaluation is still lazy as in existing Boolean expressions, so the
order of evaluation jumps around a bit.  The _condition_ expression in
the middle is evaluated first, and the _true_value_ expression is
evaluated only if the condition was true.  Similarly, the _false_value_
expression is only evaluated when the condition is false.

  This syntax may seem strange and backwards; why does the condition go
in the _middle_ of the expression, and not in the front as in C's `c ?
x : y'?  The decision was checked by applying the new syntax to the
modules in the standard library and seeing how the resulting code read.
In many cases where a conditional expression is used, one value seems
to be the 'common case' and one value is an 'exceptional case', used
only on rarer occasions when the condition isn't met.  The conditional
syntax makes this pattern a bit more obvious:

    contents = ((doc + '\n') if doc else '')

I read the above statement as meaning "here _contents_ is  usually
assigned a value of `doc+'\n''; sometimes  _doc_ is empty, in which
special case an empty string is returned."   I doubt I will use
conditional expressions very often where there  isn't a clear common
and uncommon case.

  There was some discussion of whether the language should require
surrounding conditional expressions with parentheses.  The decision was
made to _not_ require parentheses in the Python language's grammar, but
as a matter of style I think you should always use them. Consider these
two statements:

    # First version -- no parens
    level = 1 if logging else 0

    # Second version -- with parens
    level = (1 if logging else 0)

In the first version, I think a reader's eye might group the statement
into 'level = 1', 'if logging', 'else 0', and think that the condition
decides whether the assignment to _level_ is performed.  The second
version reads better, in my opinion, because it makes it clear that the
assignment is always performed and the choice is being made between two
values.

  Another reason for including the brackets: a few odd combinations of
list comprehensions and lambdas could look like incorrect conditional
expressions.  See PEP 308 (http://www.python.org/dev/peps/pep-0308) for
some examples.  If you put parentheses around your conditional
expressions, you won't run into this case.

PEP 308 (http://www.python.org/dev/peps/pep-0308) - Conditional Expressions
     PEP written by Guido van Rossum and Raymond D. Hettinger;
     implemented by Thomas Wouters.

1.3.2 PEP 309: Partial Function Application
-------------------------------------------

The functools module is intended to contain tools for
functional-style programming.

  One useful tool in this module is the `partial()' function. For
programs written in a functional style, you'll sometimes want to
construct variants of existing functions that have some of the
parameters filled in.  Consider a Python function `f(a, b, c)'; you
could create a new function `g(b, c)' that was equivalent to `f(1, b,
c)'.  This is called "partial function application".

  `partial()' takes the arguments `(function, arg1, arg2, ...
kwarg1=value1, kwarg2=value2)'.  The resulting object is callable, so
you can just call it to invoke _function_ with the filled-in arguments.

  Here's a small but realistic example:

    import functools

    def log (message, subsystem):
        "Write the contents of 'message' to the specified subsystem."
        print '%s: %s' % (subsystem, message)
        ...

    server_log = functools.partial(log, subsystem='server')
    server_log('Unable to open socket')

Here's another example, from a program that uses PyGTK.  Here a context-
sensitive pop-up menu is being constructed dynamically.  The callback
provided for the menu option is a partially applied version of the
`open_item()' method, where the first argument has been provided.

    ...
    class Application:
        def open_item(self, path):
           ...
        def init (self):
            open_func = functools.partial(self.open_item, item_path)
            popup_menu.append( ("Open", open_func, 1) )

Another function in the functools module is the
`update_wrapper(wrapper, wrapped)()' function that helps you write well-
behaved decorators.  `update_wrapper()' copies the name, module, and
docstring attribute to a wrapper function so that tracebacks inside the
wrapped function are easier to understand.  For example, you might
write:

    def my_decorator(f):
        def wrapper(*args, **kwds):
            print 'Calling decorated function'
            return f(*args, **kwds)
        functools.update_wrapper(wrapper, f)
        return wrapper

`wraps()' is a decorator that can be used inside your own decorators to
copy the wrapped function's information.  An alternate  version of the
previous example would be:

    def my_decorator(f):
        @functools.wraps(f)
        def wrapper(*args, **kwds):
            print 'Calling decorated function'
            return f(*args, **kwds)
        return wrapper

PEP 309 (http://www.python.org/dev/peps/pep-0309) - Partial Function Application
     PEP proposed and written by Peter Harris; implemented by Hye-Shik
     Chang and Nick Coghlan, with adaptations by Raymond Hettinger.

1.3.3 PEP 314: Metadata for Python Software Packages v1.1
---------------------------------------------------------

Some simple dependency support was added to Distutils.  The `setup()'
function now has `requires', `provides', and `obsoletes' keyword
parameters.  When you build a source distribution using the `sdist'
command, the dependency information will be recorded in the `PKG-INFO'
file.

  Another new keyword parameter is `download_url', which should be set
to a URL for the package's source code.  This means it's now possible
to look up an entry in the package index, determine the dependencies
for a package, and download the required packages.

    VERSION = '1.0'
    setup(name='PyPackage',
          version=VERSION,
          requires=['numarray', 'zlib (>=1.1.4)'],
          obsoletes=['OldPackage']
          download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'
                        % VERSION),
         )

Another new enhancement to the Python package index at
<http://cheeseshop.python.org> is storing source and binary archives
for a package.  The new *upload* Distutils command will upload a
package to the repository.

  Before a package can be uploaded, you must be able to build a
distribution using the *sdist* Distutils command.  Once that works, you
can run `python setup.py upload' to add your package to the PyPI
archive.  Optionally you can GPG-sign the package by supplying the
`--sign' and `--identity' options.

  Package uploading was implemented by Martin von Löwis and Richard
Jones.

PEP 314 (http://www.python.org/dev/peps/pep-0314) - Metadata for Python Software Packages v1.1
     PEP proposed and written by A.M. Kuchling, Richard Jones, and Fred
     Drake; implemented by Richard Jones and Fred Drake.

1.3.4 PEP 328: Absolute and Relative Imports
--------------------------------------------

The simpler part of PEP 328 was implemented in Python 2.4: parentheses
could now be used to enclose the names imported from a module using the
`from ... import ...' statement, making it easier to import many
different names.

  The more complicated part has been implemented in Python 2.5:
importing a module can be specified to use absolute or package-relative
imports.  The plan is to move toward making absolute imports the
default in future versions of Python.

  Let's say you have a package directory like this:

    pkg/
    pkg/__init__.py
    pkg/main.py
    pkg/string.py

This defines a package named `pkg' containing the `pkg.main' and
`pkg.string' submodules.

  Consider the code in the `main.py' module.  What happens if it
executes the statement `import string'?  In Python 2.4 and earlier, it
will first look in the package's directory to perform a relative
import, finds `pkg/string.py', imports the contents of that file as the
`pkg.string' module, and that module is bound to the name `string' in
the `pkg.main' module's namespace.

  That's fine if `pkg.string' was what you wanted.  But what if you
wanted Python's standard string module?  There's no clean
way to ignore `pkg.string' and look for the standard module; generally
you had to look at the contents of `sys.modules', which is slightly
unclean.    Holger Krekel's `py.std' package provides a tidier way to
perform imports from the standard library, `import py ;
py.std.string.join()', but that package isn't available on all Python
installations.

  Reading code which relies on relative imports is also less clear,
because a reader may be confused about which module, see string.
or `pkg.string', is intended to be used.  Python users soon learned not
to duplicate the names of standard library modules in the names of
their packages' submodules, but you can't protect against having your
submodule's name being used for a new module added in a future version
of Python.

  In Python 2.5, you can switch import.'s behaviour to
absolute imports using a `from __future__ import absolute_import'
directive.  This absolute- import behaviour will become the default in
a future version (probably Python 2.7).  Once absolute imports  are the
default, `import string' will always find the standard library's
version. It's suggested that users should begin using absolute imports
as much as possible, so it's preferable to begin writing `from pkg
import string' in your code.

  Relative imports are still possible by adding a leading period  to
the module name when using the `from ... import' form:

    # Import names from pkg.string
    from .string import name1, name2
    # Import pkg.string
    from . import string

This imports the string module relative to the current
package, so in `pkg.main' this will import _name1_ and _name2_ from
`pkg.string'.  Additional leading periods perform the relative import
starting from the parent of the current package.  For example, code in
the `A.B.C' module can do:

    from . import D                 # Imports A.B.D
    from .. import E                # Imports A.E
    from ..F import G               # Imports A.F.G

Leading periods cannot be used with the `import modname'  form of the
import statement, only the `from ... import' form.

PEP 328 (http://www.python.org/dev/peps/pep-0328) - Imports: Multi-Line and Absolute/Relative
     PEP written by Aahz; implemented by Thomas Wouters.

<http://codespeak.net/py/current/doc/index.html>
     The py library by Holger Krekel, which contains the `py.std'
     package.

1.3.5 PEP 338: Executing Modules as Scripts
-------------------------------------------

The -m switch added in Python 2.4 to execute a module as a
script gained a few more abilities.  Instead of being implemented in C
code inside the Python interpreter, the switch now uses an
implementation in a new module, see runpy.

  The runpy module implements a more sophisticated import
mechanism so that it's now possible to run modules in a package such as
`pychecker.checker'.  The module also supports alternative import
mechanisms such as the zipimport module.  This means you
can add a .zip archive's path to `sys.path' and then use the -m
switch to execute code from the archive.

PEP 338 (http://www.python.org/dev/peps/pep-0338) - Executing modules as scripts
     PEP written and  implemented by Nick Coghlan.

1.3.6 PEP 341: Unified try/except/finally
-----------------------------------------

Until Python 2.5, the try statement came in two flavours.
You could use a finally block to ensure that code is always
executed, or one or more except blocks to catch  specific
exceptions.  You couldn't combine both except blocks and a
finally block, because generating the right bytecode for
the combined version was complicated and it wasn't clear what the
semantics of the combined statement should be.

  Guido van Rossum spent some time working with Java, which does
support the equivalent of combining except blocks and a
finally block, and this clarified what the statement should
mean.  In Python 2.5, you can now write:

    try:
        block-1 ...
    except Exception1:
        handler-1 ...
    except Exception2:
        handler-2 ...
    else:
        else-block
    finally:
        final-block

The code in _block-1_ is executed.  If the code raises an exception,
the various except blocks are tested: if the exception is
of class `Exception1', _handler-1_ is executed; otherwise if it's of
class `Exception2', _handler-2_ is executed, and so forth.  If no
exception is raised, the _else-block_ is executed.

  No matter what happened previously, the _final-block_ is executed
once the code block is complete and any raised exceptions handled. Even
if there's an error in an exception handler or the _else-block_ and a
new exception is raised, the code in the _final-block_ is still run.

PEP 341 (http://www.python.org/dev/peps/pep-0341) - Unifying try-except and try-finally
     PEP written by Georg Brandl;  implementation by Thomas Lee.

1.3.7 PEP 342: New Generator Features
-------------------------------------

Python 2.5 adds a simple way to pass values _into_ a generator. As
introduced in Python 2.3, generators only produce output; once a
generator's code was invoked to create an iterator, there was no way to
pass any new information into the function when its execution is
resumed.  Sometimes the ability to pass in some information would be
useful.  Hackish solutions to this include making the generator's code
look at a global variable and then changing the global variable's
value, or passing in some mutable object that callers then modify.

  To refresh your memory of basic generators, here's a simple example:

    def counter (maximum):
        i = 0
        while i < maximum:
            yield i
            i += 1

When you call `counter(10)', the result is an iterator that returns the
values from 0 up to 9.  On encountering the yield.
statement, the iterator returns the provided value and suspends the
function's execution, preserving the local variables. Execution resumes
on the following call to the iterator's next() method,
picking up after the yield statement.

  In Python 2.3, see yield was a statement; it didn't return
any value.  In 2.5, see yield is now an expression, returning a
value that can be assigned to a variable or otherwise operated on:

    val = (yield i)

I recommend that you always put parentheses around a yield.
expression when you're doing something with the returned value, as in
the above example.  The parentheses aren't always necessary, but it's
easier to always add them instead of having to remember when they're
needed.

  ( PEP 342 (http://www.python.org/dev/peps/pep-0342) explains the
exact rules, which are that a yield.-expression must always
be parenthesized except when it occurs at the top-level expression on
the right-hand side of an assignment.  This means you can write `val =
yield i' but have to use parentheses when there's an operation, as in
`val = (yield i) + 12'.)

  Values are sent into a generator by calling its `send(value)()'
method.  The generator's code is then resumed and the yield.
expression returns the specified _value_.  If the regular next()
method is called, the yield returns None.

  Here's the previous example, modified to allow changing the value of
the internal counter.

    def counter (maximum):
        i = 0
        while i < maximum:
            val = (yield i)
            # If value provided, change counter
            if val is not None:
                i = val
            else:
                i += 1

And here's an example of changing the counter:

    >>> it = counter(10)
    >>> print it.next()
    0
    >>> print it.next()
    1
    >>> print it.send(8)
    8
    >>> print it.next()
    9
    >>> print it.next()
    Traceback (most recent call last):
      File "t.py", line 15, in ?
        print it.next()
    StopIteration

yield will usually return None, so you should
always check for this case.  Don't just use its value in expressions
unless you're sure that the `send()' method will be the only method
used to resume your generator function.

  In addition to `send()', there are two other new methods on
generators:

   * `throw(type, value=None, traceback=None)()' is used to raise an
     exception inside the generator; the exception is raised by the
     yield expression where the generator's execution is
     paused.

   * `close()' raises a new GeneratorExit exception inside
     the generator to terminate the iteration.  On receiving this
     exception, the generator's code must either raise
     GeneratorExit or StopIteration.  Catching the
     GeneratorExit exception and returning a value is
     illegal and will trigger a RuntimeError.; if the
     function raises some other exception, that exception is propagated
     to the caller.  `close()' will also be called by Python's garbage
     collector when the generator is garbage-collected.

     If you need to run cleanup code when a GeneratorExit.
     occurs, I suggest using a `try: ... finally:' suite instead of
     catching GeneratorExit.

  The cumulative effect of these changes is to turn generators from
one-way producers of information into both producers and consumers.

  Generators also become _coroutines_, a more generalized form of
subroutines.  Subroutines are entered at one point and exited at
another point (the top of the function, and a return.
statement), but coroutines can be entered, exited, and resumed at many
different points (the yield statements).  We'll have to
figure out patterns for using coroutines effectively in Python.

  The addition of the `close()' method has one side effect that isn't
obvious.  `close()' is called when a generator is garbage-collected, so
this means the generator's code gets one last chance to run before the
generator is destroyed.  This last chance means that `try...finally'
statements in generators can now be guaranteed to work; the
finally clause will now always get a chance to run.  The
syntactic restriction that you couldn't mix yield.
statements with a `try...finally' suite has therefore been removed.
This seems like a minor bit of language trivia, but using generators and
`try...finally' is actually necessary in order to implement the
with statement described by PEP 343.  I'll look at this new
statement in the following  section.

  Another even more esoteric effect of this change: previously, the
`gi_frame' attribute of a generator was always a frame object. It's now
possible for `gi_frame' to be `None' once the generator has been
exhausted.

PEP 342 (http://www.python.org/dev/peps/pep-0342) - Coroutines via Enhanced Generators
     PEP written by  Guido van Rossum and Phillip J. Eby; implemented
     by Phillip J.  Eby.  Includes examples of  some fancier uses of
     generators as coroutines.

     Earlier versions of these features were proposed in PEP 288
     (http://www.python.org/dev/peps/pep-0288) by Raymond Hettinger and PEP
     325 (http://www.python.org/dev/peps/pep-0325) by Samuele Pedroni.

<http://en.wikipedia.org/wiki/Coroutine>
     The Wikipedia entry for  coroutines.

<http://www.sidhe.org/~dan/blog/archives/000178.html>
     An explanation of coroutines from a Perl point of view, written by
     Dan Sugalski.

1.3.8 PEP 343: The 'with' statement
-----------------------------------

The 'with.' statement clarifies code that previously would
use `try...finally' blocks to ensure that clean-up code is executed.
In this section, I'll discuss the statement as it will commonly be
used.  In the next section, I'll examine the implementation details and
show how to write objects for use with this statement.

  The 'with.' statement is a new control-flow structure
whose basic structure is:

    with expression [as variable]:
        with-block

The expression is evaluated, and it should result in an object that
supports the context management protocol (that is, has
__enter__() and __exit__().  methods.

  The object's __enter__() is called before _with-block_ is
executed and therefore can run set-up code. It also may return a value
that is bound to the name _variable_, if given.  (Note carefully that
_variable_ is _not_ assigned the result of _expression_.)

  After execution of the _with-block_ is finished, the object's
__exit__().  method is called, even if the block raised an
exception, and can therefore run clean-up code.

  To enable the statement in Python 2.5, you need to add the following
directive to your module:

    from __future__ import with_statement

The statement will always be enabled in Python 2.6.

  Some standard Python objects now support the context management
protocol and can be used with the 'with.' statement. File
objects are one example:

    with open('/etc/passwd', 'r') as f:
        for line in f:
            print line
            ... more processing code ...

After this statement has executed, the file object in _f_ will have been
automatically closed, even if the for loop raised an
exception part- way through the block.

     Note: In this case, _f_ is the same object created by
     open(), because `file.__enter__()' returns _self_.

  The threading module's locks and condition variables
also support the 'with.' statement:

    lock = threading.Lock()
    with lock:
        # Critical section of code
        ...

The lock is acquired before the block is executed and always released
once  the block is complete.

  The new `localcontext()' function in the decimal module
makes it easy to save and restore the current decimal context, which
encapsulates the desired precision and rounding characteristics for
computations:

    from decimal import Decimal, Context, localcontext

    # Displays with default precision of 28 digits
    v = Decimal('578')
    print v.sqrt()

    with localcontext(Context(prec=16)):
        # All code in this block uses a precision of 16 digits.
        # The original context is restored on exiting the block.
        print v.sqrt()

1.3.8.1 Writing Context Managers
................................

Under the hood, the 'with.' statement is fairly complicated.
Most people will only use 'with.' in company with existing
objects and don't need to know these details, so you can skip the rest
of this section if you like.  Authors of new objects will need to
understand the details of the underlying implementation and should keep
reading.

  A high-level explanation of the context management protocol is:

   * The expression is evaluated and should result in an object called
     a "context manager".  The context manager must have
     __enter__() and __exit__().  methods.

   * The context manager's __enter__() method is called.
     The value returned is assigned to _VAR_.  If no `'as VAR'' clause
     is present, the value is simply discarded.

   * The code in _BLOCK_ is executed.

   * If _BLOCK_ raises an exception, the `__exit__(type, value,
     traceback)()' is called with the exception details, the same
     values returned by sys.exc_info().  The method's return
     value controls whether the exception is re-raised: any false value
     re-raises the exception, and `True' will result in suppressing it.
     You'll only rarely want to suppress the exception, because if you
     do the author of the code containing the 'with.'
     statement will never realize anything went wrong.

   * If _BLOCK_ didn't raise an exception,  the __exit__().
     method is still called, but _type_, _value_, and _traceback_ are
     all `None'.

  Let's think through an example.  I won't present detailed code but
will only sketch the methods necessary for a database that supports
transactions.

  (For people unfamiliar with database terminology: a set of changes to
the database are grouped into a transaction.  Transactions can be
either committed, meaning that all the changes are written into the
database, or rolled back, meaning that the changes are all discarded
and the database is unchanged.  See any database textbook for more
information.)

  Let's assume there's an object representing a database connection.
Our goal will be to let the user write code like this:

    db_connection = DatabaseConnection()
    with db_connection as cursor:
        cursor.execute('insert into ...')
        cursor.execute('delete from ...')
        # ... more operations ...

The transaction should be committed if the code in the block runs
flawlessly or rolled back if there's an exception. Here's the basic
interface for `DatabaseConnection' that I'll assume:

    class DatabaseConnection:
        # Database interface
        def cursor (self):
            "Returns a cursor object and starts a new transaction"
        def commit (self):
            "Commits current transaction"
        def rollback (self):
            "Rolls back current transaction"

The __enter__() method is pretty easy, having only to start
a new transaction.  For this application the resulting cursor object
would be a useful result, so the method will return it.  The user can
then add `as cursor' to their 'with.' statement to bind the
cursor to a variable name.

    class DatabaseConnection:
        ...
        def __enter__ (self):
            # Code to start a new transaction
            cursor = self.cursor()
            return cursor

The __exit__() method is the most complicated because it's
where most of the work has to be done.  The method has to check if an
exception occurred.  If there was no exception, the transaction is
committed.  The transaction is rolled back if there was an exception.

  In the code below, execution will just fall off the end of the
function, returning the default value of `None'.  `None' is false, so
the exception will be re-raised automatically.  If you wished, you
could be more explicit and add a return statement at the
marked location.

    class DatabaseConnection:
        ...
        def __exit__ (self, type, value, tb):
            if tb is None:
                # No exception, so commit
                self.commit()
            else:
                # Exception occurred, so rollback.
                self.rollback()
                # return False

1.3.8.2 The contextlib module
.............................

The new contextlib module provides some functions and a
decorator that are useful for writing objects for use with the '
with.' statement.

  The decorator is called `contextmanager()', and lets you write a
single generator function instead of defining a new class.  The
generator should yield exactly one value.  The code up to the
yield will be executed as the __enter__() method, and
the value yielded will be the method's return value that will get bound
to the variable in the 'with.' statement's as.
clause, if any.  The code after the yield will be executed
in the __exit__() method.  Any exception raised in the
block will be raised by the yield statement.

  Our database example from the previous section could be written
using this decorator as:

    from contextlib import contextmanager

    @contextmanager
    def db_transaction (connection):
        cursor = connection.cursor()
        try:
            yield cursor
        except:
            connection.rollback()
            raise
        else:
            connection.commit()

    db = DatabaseConnection()
    with db_transaction(db) as cursor:
        ...

The contextlib module also has a `nested(mgr1, mgr2, ...)()'
function that combines a number of context managers so you don't need
to write nested 'with.' statements.  In this example, the
single 'with.'  statement both starts a database transaction
and acquires a thread lock:

    lock = threading.Lock()
    with nested (db_transaction(db), lock) as (cursor, locked):
        ...

Finally, the `closing(object)()' function returns _object_ so that it
can be bound to a variable, and calls `object.close' at the end of the
block.

    import urllib, sys
    from contextlib import closing

    with closing(urllib.urlopen('http://www.yahoo.com')) as f:
        for line in f:
            sys.stdout.write(line)

PEP 343 (http://www.python.org/dev/peps/pep-0343) - The "with" statement
     PEP written by Guido van Rossum and Nick Coghlan; implemented by
     Mike Bland, Guido van Rossum, and Neal Norwitz.  The PEP shows the
     code generated for a 'with.' statement, which can be
     helpful in learning how the statement works.

  The documentation  for the contextlib module.

1.3.9 PEP 352: Exceptions as New-Style Classes
----------------------------------------------

Exception classes can now be new-style classes, not just classic
classes, and the built-in Exception class and all the
standard built-in exceptions (see NameError, ValueError,
etc.) are now new-style classes.

  The inheritance hierarchy for exceptions has been rearranged a bit.
In 2.5, the inheritance relationships are:

    BaseException       # New in Python 2.5
    |- KeyboardInterrupt
    |- SystemExit
    |- Exception
       |- (all other current built-in exceptions)

This rearrangement was done because people often want to catch all
exceptions that indicate program errors.  See KeyboardInterrupt.
and SystemExit.  aren't errors, though, and usually
represent an explicit action such as the user hitting Control-C or code
calling sys.exit().  A bare `except:' will catch all
exceptions, so you commonly need to list KeyboardInterrupt.
and SystemExit in order to re-raise them.  The usual
pattern is:

    try:
        ...
    except (KeyboardInterrupt, SystemExit):
        raise
    except:
        # Log error...
        # Continue running program...

In Python 2.5, you can now write `except Exception' to achieve the same
result, catching all the exceptions that usually indicate errors  but
leaving KeyboardInterrupt and SystemExit alone.
As in previous versions, a bare `except:' still catches all exceptions.

  The goal for Python 3.0 is to require any class raised as an
exception to derive from BaseException or some descendant
of BaseException, and future releases in the Python 2.x
series may begin to enforce this constraint.  Therefore, I suggest you
begin making all your exception classes derive from Exception
now.  It's been suggested that the bare `except:' form should be
removed in Python 3.0, but Guido van Rossum hasn't decided whether to
do this or not.

  Raising of strings as exceptions, as in the statement `raise "Error
occurred"', is deprecated in Python 2.5 and will trigger a warning.
The aim is to be able to remove the string-exception feature in a few
releases.

PEP 352 (http://www.python.org/dev/peps/pep-0352) - Required Superclass for Exceptions
     PEP written by  Brett Cannon and Guido van Rossum; implemented by
     Brett Cannon.

1.3.10 PEP 353: Using ssize_t as the index type
-----------------------------------------------

A wide-ranging change to Python's C API, using a new  `Py_ssize_t' type
definition instead of `int',  will permit the interpreter to handle more
data on 64-bit platforms. This change doesn't affect Python's capacity
on 32-bit platforms.

  Various pieces of the Python interpreter used C's `int' type to store
sizes or counts; for example, the number of items in a list or tuple
were stored in an `int'.  The C compilers for most 64-bit platforms
still define `int' as a 32-bit type, so that meant that lists could
only hold up to `2**31 - 1' = 2147483647 items. (There are actually a
few different programming models that 64-bit C compilers can use - see
<http://www.unix.org/version2/whatsnew/lp64_wp.html> for a discussion -
but the most commonly available model leaves `int' as 32 bits.)

  A limit of 2147483647 items doesn't really matter on a 32-bit
platform because you'll run out of memory before hitting the length
limit. Each list item requires space for a pointer, which is 4 bytes,
plus space for a PyObject representing the item.
2147483647*4 is already more bytes than a 32-bit address space can
contain.

  It's possible to address that much memory on a 64-bit platform,
however.  The pointers for a list that size would only require 16 GiB
of space, so it's not unreasonable that Python programmers might
construct lists that large.  Therefore, the Python interpreter had to
be changed to use some type other than `int', and this will be a 64-bit
type on 64-bit platforms.  The change will cause incompatibilities on
64-bit machines, so it was deemed worth making the transition now,
while the number of 64-bit users is still relatively small.  (In 5 or
10 years, we may _all_ be on 64-bit machines, and the transition would
be more painful then.)

  This change most strongly affects authors of C extension modules.
Python strings and container types such as lists and tuples  now use
`Py_ssize_t' to store their size.   Functions such as
PyList_Size().  now return `Py_ssize_t'.  Code in extension modules
may therefore need to have some variables changed to `Py_ssize_t'.

  The PyArg_ParseTuple() and Py_BuildValue().
functions have a new conversion code, `n', for `Py_ssize_t'.   See
PyArg_ParseTuple().'s `s#' and `t#' still output `int' by default,
but you can define the macro  `PY_SSIZE_T_CLEAN' before including
`Python.h'  to make them return `Py_ssize_t'.

  PEP 353 (http://www.python.org/dev/peps/pep-0353) has a section on
conversion guidelines that  extension authors should read to learn
about supporting 64-bit platforms.

PEP 353 (http://www.python.org/dev/peps/pep-0353) - Using ssize_t as the index type
     PEP written and implemented by Martin von Löwis.

1.3.11 PEP 357: The '__index__' method
--------------------------------------

The NumPy developers had a problem that could only be solved by adding
a new special method, see __index__().  When using slice
notation, as in `[start:stop:step]', the values of the _start_, _stop_,
and _step_ indexes must all be either integers or long integers.  NumPy
defines a variety of specialized integer types corresponding to
unsigned and signed integers of 8, 16, 32, and 64 bits, but there was
no way to signal that these types could be used as slice indexes.

  Slicing can't just use the existing __int__() method
because that method is also used to implement coercion to integers.  If
slicing used __int__(), floating-point numbers would also
become legal slice indexes and that's clearly an undesirable behaviour.

  Instead, a new special method called __index__() was
added.  It takes no arguments and returns an integer giving the slice
index to use.  For example:

    class C:
        def __index__ (self):
            return self.value

The return value must be either a Python integer or long integer. The
interpreter will check that the type returned is correct, and raises a
TypeError if this requirement isn't met.

  A corresponding `nb_index' slot was added to the C-level
PyNumberMethods structure to let C extensions implement this
protocol.  `PyNumber_Index(obj)()' can be used in extension code to
call the __index__() function and retrieve its result.

PEP 357 (http://www.python.org/dev/peps/pep-0357) - Allowing Any Object to be Used for Slicing
     PEP written  and implemented by Travis Oliphant.

1.3.12 Other Language Changes
-----------------------------

Here are all of the changes that Python 2.5 makes to the core Python
language.

   * The dict type has a new hook for letting subclasses
     provide a default value when a key isn't contained in the
     dictionary. When a key isn't found, the dictionary's
     `__missing__(key)()' method will be called.  This hook is used to
     implement the new `defaultdict' class in the collections.
     module.  The following example defines a dictionary  that returns
     zero for any missing key:

         class zerodict (dict):
             def __missing__ (self, key):
                 return 0

         d = zerodict({1:1, 2:2})
         print d[1], d[2]   # Prints 1, 2
         print d[3], d[4]   # Prints 0, 0

   * Both 8-bit and Unicode strings have new `partition(sep)()'  and
     `rpartition(sep)()' methods that simplify a common use case.

     The `find(S)()' method is often used to get an index which is then
     used to slice the string and obtain the pieces that are before and
     after the separator.  `partition(sep)()' condenses this pattern
     into a single method call that returns a 3-tuple containing the
     substring before the separator, the separator itself, and the
     substring after the separator.  If the separator isn't found, the
     first element of the tuple is the entire string and the other two
     elements are empty.  `rpartition(sep)()' also returns a 3-tuple
     but starts searching from the end of the string; the `r' stands
     for 'reverse'.

     Some examples:

         >>> ('http://www.python.org').partition('://')
         ('http', '://', 'www.python.org')
         >>> ('file:/usr/share/doc/index.html').partition('://')
         ('file:/usr/share/doc/index.html', '', '')
         >>> (u'Subject: a quick question').partition(':')
         (u'Subject', u':', u' a quick question')
         >>> 'www.python.org'.rpartition('.')
         ('www.python', '.', 'org')
         >>> 'www.python.org'.rpartition(':')
         ('', '', 'www.python.org')

     (Implemented by Fredrik Lundh following a suggestion by Raymond
     Hettinger.)

   * The `startswith()' and `endswith()' methods of string types now
     accept tuples of strings to check for.

         def is_image_file (filename):
             return filename.endswith(('.gif', '.jpg', '.tiff'))

     (Implemented by Georg Brandl following a suggestion by Tom Lynn.)

   * The min() and max() built-in functions
     gained a `key' keyword parameter analogous to the `key' argument
     for `sort()'.  This parameter supplies a function that takes a
     single argument and is called for every value in the list; see
     min()./see max() will return the element with the
     smallest/largest return value from this function. For example, to
     find the longest string in a list, you can do:

         L = ['medium', 'longest', 'short']
         # Prints 'longest'
         print max(L, key=len)
         # Prints 'short', because lexicographically 'short' has the largest value
         print max(L)

     (Contributed by Steven Bethard and Raymond Hettinger.)

   * Two new built-in functions, see any() and all(),
     evaluate whether an iterator contains any true or false
     values.  any() returns True.  if any value
     returned by the iterator is true; otherwise it will return
     False.  all() returns True only if
     all of the values returned by the iterator evaluate as true.
     (Suggested by Guido van Rossum, and implemented by Raymond
     Hettinger.)

   * The result of a class's __hash__() method can now be
     either a long integer or a regular integer.  If a long integer is
     returned, the hash of that value is taken.  In earlier versions
     the hash value was required to be a regular integer, but in 2.5
     the id() built-in was changed to always return
     non-negative numbers, and users often seem to use `id(self)' in
     __hash__() methods (though this is discouraged).

   * ASCII is now the default encoding for modules.  It's now  a syntax
     error if a module contains string literals with 8-bit characters
     but doesn't have an encoding declaration.  In Python 2.4 this
     triggered a warning, not a syntax error.  See PEP 263
     (http://www.python.org/dev/peps/pep-0263)  for how to declare a
     module's encoding; for example, you might add  a line like this
     near the top of the source file:

         # -*- coding: latin1 -*-

   * A new warning, `UnicodeWarning', is triggered when  you attempt to
     compare a Unicode string and an 8-bit string  that can't be
     converted to Unicode using the default ASCII encoding.   The
     result of the comparison is false:

         >>> chr(128) == unichr(128)   # Can't convert chr(128) to Unicode
         __main__:1: UnicodeWarning: Unicode equal comparison failed
           to convert both arguments to Unicode - interpreting them
           as being unequal
         False
         >>> chr(127) == unichr(127)   # chr(127) can be converted
         True

     Previously this would raise a `UnicodeDecodeError' exception, but
     in 2.5 this could result in puzzling problems when accessing a
     dictionary.  If you looked up `unichr(128)' and `chr(128)' was
     being used as a key, you'd get a `UnicodeDecodeError' exception.
     Other changes in 2.5 resulted in this exception being raised
     instead of suppressed by the code in `dictobject.c' that
     implements dictionaries.

     Raising an exception for such a comparison is strictly correct,
     but the change might have broken code, so instead
     `UnicodeWarning' was introduced.

     (Implemented by Marc-André Lemburg.)

   * One error that Python programmers sometimes make is forgetting to
     include an `__init__.py' module in a package directory. Debugging
     this mistake can be confusing, and usually requires running Python
     with the -v switch to log all the paths searched. In
     Python 2.5, a new ImportWarning warning is triggered
     when an import would have picked up a directory as a package but no
     `__init__.py' was found.  This warning is silently ignored by
     default; provide the `-Wd' option when running the Python
     executable to display the warning message. (Implemented by Thomas
     Wouters.)

   * The list of base classes in a class definition can now be empty.
     As an example, this is now legal:

         class C():
             pass

     (Implemented by Brett Cannon.)

1.3.12.1 Interactive Interpreter Changes
........................................

In the interactive interpreter, `quit' and `exit'  have long been
strings so that new users get a somewhat helpful message when they try
to quit:

    >>> quit
    'Use Ctrl-D (i.e. EOF) to exit.'

In Python 2.5, `quit' and `exit' are now objects that still produce
string representations of themselves, but are also callable. Newbies
who try `quit()' or `exit()' will now exit the interpreter as they
expect.  (Implemented by Georg Brandl.)

  The Python executable now accepts the standard long options
-help.  and -version.; on Windows,  it also accepts the
`/?' option for displaying a help message. (Implemented by Georg
Brandl.)

1.3.12.2 Optimizations
......................

Several of the optimizations were developed at the NeedForSpeed sprint,
an event held in Reykjavik, Iceland, from May 21-28 2006. The sprint
focused on speed enhancements to the CPython implementation and was
funded by EWT LLC with local support from CCP Games.  Those
optimizations added at this sprint are specially marked in the
following list.

   * When they were introduced  in Python 2.4, the built-in set
     and frozenset types were built on top of Python's
     dictionary type.   In 2.5 the internal data structure has been
     customized for implementing sets, and as a result sets will use a
     third less memory and are somewhat faster. (Implemented by Raymond
     Hettinger.)

   * The speed of some Unicode operations, such as finding substrings,
     string splitting, and character map encoding and decoding, has
     been improved.  (Substring search and splitting improvements were
     added by Fredrik Lundh and Andrew Dalke at the NeedForSpeed
     sprint. Character maps were improved by Walter Dörwald and Martin
     von Löwis.)

   * The `long(str, base)()' function is now faster on long digit
     strings because fewer intermediate results are calculated.  The
     peak is for strings of around 800-1000 digits where  the function
     is 6 times faster. (Contributed by Alan McIntyre and committed at
     the NeedForSpeed sprint.)

   * It's now illegal to mix iterating over a file  with `for line in
     file' and calling  the file object's `read()'/see readline().
     /`readlines()' methods.  Iteration uses an internal buffer and
     the  `read*()' methods don't use that buffer.   Instead they would
     return the data following the buffer, causing the data to appear
     out of order.  Mixing iteration and these methods will now trigger
     a ValueError from the `read*()' method.  (Implemented
     by Thomas Wouters.)

   * The struct module now compiles structure format
     strings into an internal representation and caches this
     representation, yielding a 20% speedup.  (Contributed by Bob
     Ippolito at the NeedForSpeed sprint.)

   * The re module got a 1 or 2% speedup by switching to
     Python's allocator functions instead of the system's  `malloc()'
     and `free()'.  (Contributed by Jack Diederich at the NeedForSpeed
     sprint.)

   * The code generator's peephole optimizer now performs simple
     constant folding in expressions.  If you write something like `a =
     2+3', the code generator will do the arithmetic and produce code
     corresponding to `a = 5'.  (Proposed and implemented  by Raymond
     Hettinger.)

   * Function calls are now faster because code objects now keep  the
     most recently finished frame (a "zombie frame") in an internal
     field of the code object, reusing it the next time the code object
     is invoked.  (Original patch by Michael Hudson, modified by Armin
     Rigo and Richard Jones; committed at the NeedForSpeed sprint.)
     Frame objects are also slightly smaller, which may improve cache
     locality and reduce memory usage a bit.  (Contributed by Neal
     Norwitz.)

   * Python's built-in exceptions are now new-style classes, a change
     that speeds up instantiation considerably.  Exception handling in
     Python 2.5 is therefore about 30% faster than in 2.4. (Contributed
     by Richard Jones, Georg Brandl and Sean Reifschneider at the
     NeedForSpeed sprint.)

   * Importing now caches the paths tried, recording whether  they
     exist or not so that the interpreter makes fewer  `open()' and
     `stat()' calls on startup. (Contributed by Martin von Löwis and
     Georg Brandl.)

1.3.13 New, Improved, and Removed Modules
-----------------------------------------

The standard library received many enhancements and bug fixes in Python
2.5.  Here's a partial list of the most notable changes, sorted
alphabetically by module name. Consult the `Misc/NEWS' file in the
source tree for a more complete list of changes, or look through the
SVN logs for all the details.

   * The audioop module now supports the a-LAW encoding, and
     the code for u-LAW encoding has been improved.  (Contributed by
     Lars Immisch.)

   * The codecs module gained support for incremental
     codecs.  The `codec.lookup()' function now returns a `CodecInfo'
     instance instead of a tuple. `CodecInfo' instances behave like a
     4-tuple to preserve backward compatibility but also have the
     attributes `encode', `decode', `incrementalencoder',
     `incrementaldecoder', `streamwriter', and `streamreader'.
     Incremental codecs  can receive input and produce output in
     multiple chunks; the output is the same as if the entire input was
     fed to the non-incremental codec. See the codecs module
     documentation for details. (Designed and implemented by Walter
     Dörwald.)

   * The collections module gained a new type,
     `defaultdict', that subclasses the standard dict type.
     The new type mostly behaves like a dictionary but constructs a
     default value when a key isn't present, automatically adding it to
     the dictionary for the requested key value.

     The first argument to `defaultdict''s constructor is a factory
     function that gets called whenever a key is requested but not
     found. This factory function receives no arguments, so you can use
     built-in type constructors such as list() or
     int().  For example,  you can make an index of words based on
     their initial letter like this:

         words = """Nel mezzo del cammin di nostra vita
         mi ritrovai per una selva oscura
         che la diritta via era smarrita""".lower().split()

         index = defaultdict(list)

         for w in words:
             init_letter = w[0]
             index[init_letter].append(w)

     Printing `index' results in the following output:

         defaultdict(<type 'list'>, {'c': ['cammin', 'che'], 'e': ['era'],
                 'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'],
                 'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'],
                 'p': ['per'], 's': ['selva', 'smarrita'],
                 'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']}

     (Contributed by Guido van Rossum.)

   * The `deque' double-ended queue type supplied by the
     collections.  module now has a `remove(value)()' method that
     removes the first occurrence of _value_ in the queue, raising
     ValueError if the value isn't found.  (Contributed by
     Raymond Hettinger.)

   * New module: The contextlib module contains helper
     functions for use with the new 'with.' statement.  See
     section The contextlib module.  for more about this
     module.

   * New module: The cProfile module is a C implementation
     of  the existing profile module that has much lower
     overhead. The module's interface is the same as profile.:
     you run `cProfile.run('main()')' to profile a function, can
     save profile data to a file, etc.  It's not yet known if the
     Hotshot profiler, which is also written in C but doesn't match the
     profile module's interface, will continue to be
     maintained in future versions of Python.  (Contributed by Armin
     Rigo.)

     Also, the pstats module for analyzing the data
     measured by the profiler now supports directing the output to any
     file object by supplying a _stream_ argument to the `Stats'
     constructor. (Contributed by Skip Montanaro.)

   * The csv module, which parses files in comma-separated
     value format, received several enhancements and a number of
     bugfixes.  You can now set the maximum size in bytes of a field by
     calling the `csv.field_size_limit(new_limit)()' function; omitting
     the _new_limit_ argument will return the currently-set limit.  The
     `reader' class now has a `line_num' attribute that counts the
     number of physical lines read from the source; records can span
     multiple physical lines, so `line_num' is not the same as the
     number of records read.

     The CSV parser is now stricter about multi-line quoted fields.
     Previously, if a line ended within a quoted field without a
     terminating newline character, a newline would be inserted into
     the returned field. This behavior caused problems when reading
     files that contained carriage return characters within fields, so
     the code was changed to return the field without inserting
     newlines. As a consequence, if newlines embedded within fields are
     important, the input should be split into lines in a manner that
     preserves the newline characters.

     (Contributed by Skip Montanaro and Andrew McNamara.)

   * The datetime class in the datetime.  module
     now has a `strptime(string, format)()'  method for parsing date
     strings, contributed by Josh Spoerri. It uses the same format
     characters as time.strptime() and
     time.strftime().:

         from datetime import datetime

         ts = datetime.strptime('10:13:15 2006-03-07',
                                '%H:%M:%S %Y-%m-%d')

   * The `SequenceMatcher.get_matching_blocks()' method in the
     difflib.  module now guarantees to return a minimal list of
     blocks describing matching subsequences.  Previously, the
     algorithm would occasionally break a block of matching elements
     into two list entries. (Enhancement by Tim Peters.)

   * The doctest module gained a `SKIP' option that keeps an
     example from being executed at all.  This is intended for code
     snippets that are usage examples intended for the reader and
     aren't actually test cases.

     An _encoding_ parameter was added to the `testfile()' function and
     the `DocFileSuite' class to specify the file's encoding.  This
     makes it easier to use non-ASCII characters in  tests contained
     within a docstring.  (Contributed by Bjorn Tillenius.)

   * The email package has been updated to version 4.0.
     (Contributed by Barry Warsaw.)

   * The fileinput module was made more flexible. Unicode
     filenames are now supported, and a _mode_ parameter that defaults
     to `"r"' was added to the input() function to allow
     opening files in binary or universal-newline mode.  Another new
     parameter, _openhook_, lets you use a function other than
     open().  to open the input files.  Once you're iterating over
     the set of files, the `FileInput' object's new `fileno()' returns
     the file descriptor for the currently opened file. (Contributed by
     Georg Brandl.)

   * In the gc module, the new `get_count()' function
     returns a 3-tuple containing the current collection counts for the
     three GC generations.  This is accounting information for the
     garbage collector; when these counts reach a specified threshold,
     a garbage collection sweep will be made.  The existing
     gc.collect() function now takes an optional _generation_
     argument of 0, 1, or 2 to specify which generation to collect.
     (Contributed by Barry Warsaw.)

   * The `nsmallest()' and  `nlargest()' functions in the heapq.
     module  now support a `key' keyword parameter similar to the
     one provided by the min()./see max() functions
     and the `sort()' methods.  For example:

         >>> import heapq
         >>> L = ["short", 'medium', 'longest', 'longer still']
         >>> heapq.nsmallest(2, L)  # Return two lowest elements, lexicographically
         ['longer still', 'longest']
         >>> heapq.nsmallest(2, L, key=len)   # Return two shortest elements
         ['short', 'medium']

     (Contributed by Raymond Hettinger.)

   * The itertools.islice() function now accepts `None' for
     the start and step arguments.  This makes it more compatible with
     the attributes of slice objects, so that you can now write the
     following:

         s = slice(5)     # Create slice object
         itertools.islice(iterable, s.start, s.stop, s.step)

     (Contributed by Raymond Hettinger.)

   * The format() function in the locale module
     has been modified and two new functions were added,
     `format_string()' and `currency()'.

     The format() function's _val_ parameter could
     previously be a string as long as no more than one %char specifier
     appeared; now the parameter must be exactly one %char specifier
     with no surrounding text.  An optional _monetary_ parameter was
     also added which, if `True', will use the locale's rules for
     formatting currency in placing a separator between groups of three
     digits.

     To format strings with multiple %char specifiers, use the new
     `format_string()' function that works like format().
     but also supports mixing %char specifiers with arbitrary text.

     A new `currency()' function was also added that formats a number
     according to the current locale's settings.

     (Contributed by Georg Brandl.)

   * The mailbox module underwent a massive rewrite to add
     the capability to modify mailboxes in addition to reading them.  A
     new set of classes that include `mbox', `MH', and `Maildir' are
     used to read mailboxes, and have an `add(message)()' method to add
     messages, `remove(key)()' to remove messages, and
     `lock()'/`unlock()' to lock/unlock the mailbox.  The following
     example converts a maildir-format mailbox into an mbox-format one:

         import mailbox

         # 'factory=None' uses email.Message.Message as the class representing
         # individual messages.
         src = mailbox.Maildir('maildir', factory=None)
         dest = mailbox.mbox('/tmp/mbox')

         for msg in src:
             dest.add(msg)

     (Contributed by Gregory K. Johnson.  Funding was provided by
     Google's 2005 Summer of Code.)

   * New module: the msilib module allows creating
     Microsoft Installer `.msi' files and CAB files.  Some support for
     reading the `.msi' database is also included. (Contributed by
     Martin von Löwis.)

   * The nis module now supports accessing domains other
     than the system default domain by supplying a _domain_ argument to
     the nis.match() and nis.maps() functions.
     (Contributed by Ben Bell.)

   * The operator module's `itemgetter()'  and
     `attrgetter()' functions now support multiple fields.   A call
     such as `operator.attrgetter('a', 'b')' will return a function
     that retrieves the `a' and `b' attributes.  Combining  this new
     feature with the `sort()' method's `key' parameter  lets you
     easily sort lists using multiple fields. (Contributed by Raymond
     Hettinger.)

   * The optparse module was updated to version 1.5.1 of
     the Optik library.  The `OptionParser' class gained an `epilog'
     attribute, a string that will be printed after the help message,
     and a `destroy()' method to break reference cycles created by the
     object. (Contributed by Greg Ward.)

   * The os module underwent several changes.  The
     `stat_float_times' variable now defaults to true, meaning that
     os.stat() will now return time values as floats.
     (This doesn't necessarily mean that os.stat() will
     return times that are precise to fractions of a second; not all
     systems support such precision.)

     Constants named os.SEEK_SET, os.SEEK_CUR, and
     os.SEEK_END have been added; these are the parameters
     to the os.lseek() function.  Two new constants for
     locking are os.O_SHLOCK and os.O_EXLOCK.

     Two new functions, `wait3()' and `wait4()', were added.  They're
     similar the `waitpid()' function which waits for a child process
     to exit and returns a tuple of the process ID and its exit status,
     but `wait3()' and `wait4()' return additional information.
     `wait3()' doesn't take a process ID as input, so it waits for any
     child process to exit and returns a 3-tuple of _process-id_,
     _exit-status_, _resource-usage_ as returned from the
     resource.getrusage() function. `wait4(pid)()' does take a
     process ID.  (Contributed by Chad J. Schroeder.)

     On FreeBSD, the os.stat() function now returns  times
     with nanosecond resolution, and the returned object now has
     `st_gen' and `st_birthtime'. The `st_flags' member is also
     available, if the platform supports it. (Contributed by Antti
     Louko and  Diego Pettenò.)

   * The Python debugger provided by the pdb module can now
     store lists of commands to execute when a breakpoint is reached
     and execution stops.  Once breakpoint #1 has been created, enter
     `commands 1' and enter a series of commands to be executed,
     finishing the list with `end'.  The command list can include
     commands that resume execution, such as `continue' or `next'.
     (Contributed by Grégoire Dooms.)

   * The pickle and cPickle modules no longer
     accept a return value of `None' from the __reduce__().
     method; the method must return a tuple of arguments instead.  The
     ability to return `None' was deprecated in Python 2.4, so this
     completes the removal of the feature.

   * The pkgutil module, containing various utility
     functions for finding packages, was enhanced to support PEP 302's
     import hooks and now also works for packages stored in ZIP-format
     archives. (Contributed by Phillip J. Eby.)

   * The pybench benchmark suite by Marc-André Lemburg is now included
     in the `Tools/pybench' directory.  The pybench suite is an
     improvement on the commonly used `pystone.py' program because
     pybench provides a more detailed measurement of the interpreter's
     speed.  It times particular operations such as function calls,
     tuple slicing, method lookups, and numeric operations, instead of
     performing many different operations and reducing the result to a
     single number as `pystone.py' does.

   * The `pyexpat' module now uses version 2.0 of the Expat parser.
     (Contributed by Trent Mick.)

   * The Queue class provided by the Queue.
     module gained two new methods.  `join()' blocks until all items in
     the queue have been retrieved and all processing work on the items
     have been completed.  Worker threads call the other new method,
     `task_done()', to signal that processing for an item has been
     completed.  (Contributed by Raymond Hettinger.)

   * The old `regex' and `regsub' modules, which have been  deprecated
     ever since Python 2.0, have finally been deleted.   Other deleted
     modules: `statcache', `tzparse', `whrandom'.

   * Also deleted: the `lib-old' directory, which includes ancient
     modules such as `dircmp' and `ni', was removed.  `lib-old' wasn't
     on the default `sys.path', so unless your programs explicitly
     added the directory to `sys.path', this removal shouldn't affect
     your code.

   * The rlcompleter module is no longer  dependent on
     importing the readline module and therefore now works
     on non-Unix platforms. (Patch from Robert Kiendl.)

   * The SimpleXMLRPCServer and DocXMLRPCServer.
     classes now have a `rpc_paths' attribute that constrains XML-RPC
     operations to a limited set of URL paths; the default is to allow
     only `'/'' and `'/RPC2''.  Setting `rpc_paths' to `None' or an
     empty tuple disables  this path checking.

   * The socket module now supports `AF_NETLINK' sockets on
     Linux, thanks to a patch from Philippe Biondi.   Netlink sockets
     are a Linux-specific mechanism for communications between a
     user-space process and kernel code; an introductory  article about
     them is at <http://www.linuxjournal.com/article/7356>.  In Python
     code, netlink addresses are represented as a tuple of 2 integers,
     `(pid, group_mask)'.

     Two new methods on socket objects, `recv_into(buffer)()' and
     `recvfrom_into(buffer)()', store the received data in an object
     that supports the buffer protocol instead of returning the data as
     a string.  This means you can put the data directly into an array
     or a memory-mapped file.

     Socket objects also gained `getfamily()', `gettype()', and
     `getproto()' accessor methods to retrieve the family, type, and
     protocol values for the socket.

   * New module: the spwd module provides functions for
     accessing the shadow password database on systems that support
     shadow passwords.

   * The struct is now faster because it  compiles format
     strings into `Struct' objects with `pack()' and `unpack()'
     methods.  This is similar to how the re module lets
     you create compiled regular expression objects.  You can still use
     the module-level  `pack()' and `unpack()' functions; they'll
     create  `Struct' objects and cache them.  Or you can use  `Struct'
     instances directly:

         s = struct.Struct('ih3s')

         data = s.pack(1972, 187, 'abc')
         year, number, name = s.unpack(data)

     You can also pack and unpack data to and from buffer objects
     directly using the `pack_into(buffer, offset, v1, v2, ...)()' and
     `unpack_from(buffer, offset)()' methods.  This lets you store data
     directly into an array or a memory- mapped file.

     (`Struct' objects were implemented by Bob Ippolito at the
     NeedForSpeed sprint.  Support for buffer objects was added by
     Martin Blais, also at the NeedForSpeed sprint.)

   * The Python developers switched from CVS to Subversion during the
     2.5 development process.  Information about the exact build
     version is available as the `sys.subversion' variable, a 3-tuple
     of `(interpreter-name, branch-name, revision-range)'.  For
     example, at the time of writing my copy of 2.5 was reporting
     `('CPython', 'trunk', '45313:45315')'.

     This information is also available to C extensions via the
     Py_GetBuildInfo() function that returns a  string of build
     information like this: `"trunk:45355:45356M, Apr 13 2006,
     07:42:19"'.   (Contributed by Barry Warsaw.)

   * Another new function, see sys._current_frames(), returns
     the current stack frames for all running threads as a dictionary
     mapping thread identifiers to the topmost stack frame currently
     active in that thread at the time the function is called.
     (Contributed by Tim Peters.)

   * The `TarFile' class in the tarfile module now has an
     `extractall()' method that extracts all members from the archive
     into the current working directory.  It's also possible to set a
     different directory as the extraction target, and to unpack only a
     subset of the archive's members.

     The compression used for a tarfile opened in stream mode can now
     be autodetected using the mode `'r|*''. (Contributed by Lars
     Gustäbel.)

   * The threading module now lets you set the stack size
     used when new threads are created. The `stack_size([*size*])()'
     function returns the currently configured stack size, and
     supplying the optional _size_ parameter sets a new value.  Not all
     platforms support changing the stack size, but Windows, POSIX
     threading, and OS/2 all do. (Contributed by Andrew MacIntyre.)

   * The unicodedata module has been updated to use version
     4.1.0 of the Unicode character database.  Version 3.2.0 is
     required  by some specifications, so it's still available as
     unicodedata.ucd_3_2_0.

   * New module: the  uuid module generates  universally
     unique identifiers (UUIDs) according to RFC 4122
     (http://tools.ietf.org/html/rfc4122.html).  The RFC defines
     several different UUID versions that are generated from a starting
     string, from system properties, or purely randomly.  This module
     contains a `UUID' class and  functions named `uuid1()', `uuid3()',
     `uuid4()',  and  `uuid5()' to generate different versions of UUID.
     (Version 2 UUIDs  are not specified in RFC 4122
     (http://tools.ietf.org/html/rfc4122.html) and are not supported by
     this module.)

         >>> import uuid
         >>> # make a UUID based on the host ID and current time
         >>> uuid.uuid1()
         UUID('a8098c1a-f86e-11da-bd1a-00112444be1e')

         >>> # make a UUID using an MD5 hash of a namespace UUID and a name
         >>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org')
         UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e')

         >>> # make a random UUID
         >>> uuid.uuid4()
         UUID('16fd2706-8baf-433b-82eb-8c7fada847da')

         >>> # make a UUID using a SHA-1 hash of a namespace UUID and a name
         >>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org')
         UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d')

     (Contributed by Ka-Ping Yee.)

   * The weakref module's `WeakKeyDictionary' and
     `WeakValueDictionary' types gained new methods for iterating over
     the weak references contained in the dictionary.  `iterkeyrefs()'
     and `keyrefs()' methods were added to `WeakKeyDictionary', and
     `itervaluerefs()' and `valuerefs()' were added to
     `WeakValueDictionary'.  (Contributed by Fred L. Drake, Jr.)

   * The webbrowser module received a number of
     enhancements. It's now usable as a script with `python -m
     webbrowser', taking a URL as the argument; there are a number of
     switches  to control the behaviour (`-n' for a new browser window,
     see -t for a new tab).  New module-level functions,
     `open_new()' and `open_new_tab()', were added  to support this.
     The module's open() function supports an additional
     feature, an _autoraise_ parameter that signals whether to raise
     the open window when possible. A number of additional browsers
     were added to the supported list such as Firefox, Opera,
     Konqueror, and elinks.  (Contributed by Oleg Broytmann and Georg
     Brandl.)

   * The xmlrpclib module now supports returning
     datetime objects for the XML-RPC date type.  Supply
     `use_datetime=True' to the `loads()' function or the
     `Unmarshaller' class to enable this feature. (Contributed by Skip
     Montanaro.)

   * The zipfile module now supports the ZIP64 version of
     the  format, meaning that a .zip archive can now be larger than 4
     GiB and can contain individual files larger than 4 GiB.
     (Contributed by Ronald Oussoren.)

   * The zlib module's `Compress' and `Decompress' objects
     now support a copy() method that makes a copy of the
     object's internal state and returns a new  `Compress' or
     `Decompress' object.  (Contributed by Chris AtLee.)

1.3.13.1 The ctypes package
...........................

The ctypes package, written by Thomas Heller, has been added
to the standard library.  ctypes lets you call arbitrary
functions  in shared libraries or DLLs.  Long-time users may remember
the dl module, which provides functions for loading shared
libraries and calling functions in them.  The ctypes package
is much fancier.

  To load a shared library or DLL, you must create an instance of the
`CDLL' class and provide the name or path of the shared library or DLL.
Once that's done, you can call arbitrary functions by accessing them as
attributes of the `CDLL' object.

    import ctypes

    libc = ctypes.CDLL('libc.so.6')
    result = libc.printf("Line of output\n")

Type constructors for the various C types are provided: `c_int()',
`c_float()', `c_double()', `c_char_p()' (equivalent to `char *'), and
so forth.  Unlike Python's types, the C versions are all mutable; you
can assign to their `value' attribute to change the wrapped value.
Python integers and strings will be automatically converted to the
corresponding C types, but for other types you  must call the correct
type constructor.  (And I mean _must_;  getting it wrong will often
result in the interpreter crashing with a segmentation fault.)

  You shouldn't use `c_char_p()' with a Python string when the C
function will be modifying the memory area, because Python strings are
supposed to be immutable; breaking this rule will cause puzzling bugs.
When you need a modifiable memory area, use `create_string_buffer()':

    s = "this is a string"
    buf = ctypes.create_string_buffer(s)
    libc.strfry(buf)

C functions are assumed to return integers, but you can set the
`restype' attribute of the function object to  change this:

    >>> libc.atof('2.71828')
    -1783957616
    >>> libc.atof.restype = ctypes.c_double
    >>> libc.atof('2.71828')
    2.71828

ctypes also provides a wrapper for Python's C API  as the
`ctypes.pythonapi' object.  This object does _not_  release the global
interpreter lock before calling a function, because the lock must be
held when calling into the interpreter's code.   There's a
`py_object()' type constructor that will create a  PyObject *
pointer.  A simple usage:

    import ctypes

    d = {}
    ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d),
              ctypes.py_object("abc"),  ctypes.py_object(1))
    # d is now {'abc', 1}.

Don't forget to use `py_object()'; if it's omitted you end  up with a
segmentation fault.

  ctypes has been around for a while, but people still write
and distribution hand-coded extension modules because you can't rely on
ctypes being present. Perhaps developers will begin to write
Python wrappers atop a library accessed through ctypes.
instead of extension modules, now that ctypes is included
with core Python.

<http://starship.python.net/crew/theller/ctypes/>
     The ctypes web page, with a tutorial, reference, and FAQ.

  The documentation  for the ctypes module.

1.3.13.2 The ElementTree package
................................

A subset of Fredrik Lundh's ElementTree library for processing XML has
been added to the standard library as `xml.etree'.  The available
modules are `ElementTree', `ElementPath', and `ElementInclude' from
ElementTree 1.2.6.    The `cElementTree' accelerator module is also
included.

  The rest of this section will provide a brief overview of using
ElementTree.  Full documentation for ElementTree is available at
<http://effbot.org/zone/element-index.htm>.

  ElementTree represents an XML document as a tree of element nodes.
The text content of the document is stored as the `text' and `tail'
attributes of  (This is one of the major differences between
ElementTree and the Document Object Model; in the DOM there are many
different types of node, including `TextNode'.)

  The most commonly used parsing function is `parse()', that takes
either a string (assumed to contain a filename) or a file-like object
and returns an `ElementTree' instance:

    from xml.etree import ElementTree as ET

    tree = ET.parse('ex-1.xml')

    feed = urllib.urlopen(
              'http://planet.python.org/rss10.xml')
    tree = ET.parse(feed)

Once you have an `ElementTree' instance, you can call its `getroot()'
method to get the root `Element' node.

  There's also an `XML()' function that takes a string literal and
returns an `Element' node (not an `ElementTree').   This function
provides a tidy way to incorporate XML fragments, approaching the
convenience of an XML literal:

    svg = ET.XML("""<svg width="10px" version="1.0">
                 </svg>""")
    svg.set('height', '320px')
    svg.append(elem1)

Each XML element supports some dictionary-like and some list-like access
methods.  Dictionary-like operations are used to access attribute
values, and list-like operations are used to access child nodes.

Operation                           Result
-------------------------------------------------------------------------------------
`elem[n]'                           Returns n'th child element.
`elem[m:n]'                         Returns list of m'th through n'th child
                                    elements.
`len(elem)'                         Returns number of child elements.
`list(elem)'                        Returns list of child elements.
`elem.append(elem2)'                Adds _elem2_ as a child.
`elem.insert(index, elem2)'         Inserts _elem2_ at the specified location.
`del elem[n]'                       Deletes n'th child element.
`elem.keys()'                       Returns list of attribute names.
`elem.get(name)'                    Returns value of attribute _name_.
`elem.set(name, value)'             Sets new value for attribute _name_.
`elem.attrib'                       Retrieves the dictionary containing attributes.
`del elem.attrib[name]'             Deletes attribute _name_.

  Comments and processing instructions are also represented as `Element'
nodes.  To check if a node is a comment or processing instructions:

    if elem.tag is ET.Comment:
        ...
    elif elem.tag is ET.ProcessingInstruction:
        ...

To generate XML output, you should call the `ElementTree.write()'
method.  Like `parse()', it can take either a string or a file-like
object:

    # Encoding is US-ASCII
    tree.write('output.xml')

    # Encoding is UTF-8
    f = open('output.xml', 'w')
    tree.write(f, encoding='utf-8')

(Caution: the default encoding used for output is ASCII.  For general
XML work, where an element's name may contain arbitrary Unicode
characters, ASCII isn't a very useful encoding because it will raise an
exception if an element's name contains any characters with values
greater than 127.  Therefore, it's best to specify a different encoding
such as UTF-8 that can handle any Unicode character.)

  This section is only a partial description of the ElementTree
interfaces. Please read the package's official documentation for more
details.

<http://effbot.org/zone/element-index.htm>
     Official documentation for ElementTree.

1.3.13.3 The hashlib package
............................

A new hashlib module, written by Gregory P. Smith,  has been
added to replace the md5 and sha modules.
hashlib adds support for additional secure hashes (SHA-224,
SHA-256, SHA-384, and SHA-512). When available, the module uses OpenSSL
for fast platform optimized implementations of algorithms.

  The old md5 and sha modules still exist as
wrappers around hashlib to preserve backwards compatibility.  The new
module's interface is very close to that of the old modules, but not
identical. The most significant difference is that the constructor
functions for creating new hashing objects are named differently.

    # Old versions
    h = md5.md5()
    h = md5.new()

    # New version
    h = hashlib.md5()

    # Old versions
    h = sha.sha()
    h = sha.new()

    # New version
    h = hashlib.sha1()

    # Hash that weren't previously available
    h = hashlib.sha224()
    h = hashlib.sha256()
    h = hashlib.sha384()
    h = hashlib.sha512()

    # Alternative form
    h = hashlib.new('md5')          # Provide algorithm as a string

Once a hash object has been created, its methods are the same as before:
`update(string)()' hashes the specified string into the  current digest
state, `digest()' and `hexdigest()' return the digest value as a binary
string or a string of hex digits, and copy() returns a new
hashing object with the same digest state.

  The documentation  for the hashlib module.

1.3.13.4 The sqlite3 package
............................

The pysqlite module (<http://www.pysqlite.org>), a wrapper for the
SQLite embedded database, has been added to the standard library under
the package name sqlite3.

  SQLite is a C library that provides a lightweight disk-based database
that doesn't require a separate server process and allows accessing the
database using a nonstandard variant of the SQL query language. Some
applications can use SQLite for internal data storage.  It's also
possible to prototype an application using SQLite and then port the
code to a larger database such as PostgreSQL or Oracle.

  pysqlite was written by Gerhard Häring and provides a SQL interface
compliant with the DB-API 2.0 specification described by PEP 249
(http://www.python.org/dev/peps/pep-0249).

  If you're compiling the Python source yourself, note that the source
tree doesn't include the SQLite code, only the wrapper module. You'll
need to have the SQLite libraries and headers installed before
compiling Python, and the build process will compile the module when
the necessary headers are available.

  To use the module, you must first create a `Connection' object that
represents the database.  Here the data will be stored in the
`/tmp/example' file:

    conn = sqlite3.connect('/tmp/example')

You can also supply the special name `:memory:' to create a database in
RAM.

  Once you have a `Connection', you can create a `Cursor'  object and
call its `execute()' method to perform SQL commands:

    c = conn.cursor()

    # Create table
    c.execute('''create table stocks
    (date text, trans text, symbol text,
     qty real, price real)''')

    # Insert a row of data
    c.execute("""insert into stocks
              values ('2006-01-05','BUY','RHAT',100,35.14)""")

Usually your SQL operations will need to use values from Python
variables.  You shouldn't assemble your query using Python's string
operations because doing so is insecure; it makes your program
vulnerable to an SQL injection attack.

  Instead, use the DB-API's parameter substitution.  Put `?' as a
placeholder wherever you want to use a value, and then provide a tuple
of values as the second argument to the cursor's `execute()' method.
(Other database modules may use a different placeholder, such as `%s'
or `:1'.) For example:

    # Never do this -- insecure!
    symbol = 'IBM'
    c.execute("... where symbol = '%s'" % symbol)

    # Do this instead
    t = (symbol,)
    c.execute('select * from stocks where symbol=?', t)

    # Larger example
    for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
              ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
              ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
             ):
        c.execute('insert into stocks values (?,?,?,?,?)', t)

To retrieve data after executing a SELECT statement, you can either
treat the cursor as an iterator, call the cursor's `fetchone()' method
to retrieve a single matching row,  or call `fetchall()' to get a list
of the matching rows.

  This example uses the iterator form:

    >>> c = conn.cursor()
    >>> c.execute('select * from stocks order by price')
    >>> for row in c:
    ...    print row
    ...
    (u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001)
    (u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
    (u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
    (u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)
    >>>

For more information about the SQL dialect supported by SQLite, see
<http://www.sqlite.org>.

<http://www.pysqlite.org>
     The pysqlite web page.

<http://www.sqlite.org>
     The SQLite web page; the documentation describes the syntax and
     the available data types for the supported SQL dialect.

  The documentation  for the sqlite3 module.

PEP 249 (http://www.python.org/dev/peps/pep-0249) - Database API Specification 2.0
     PEP written by Marc-André Lemburg.

1.3.13.5 The wsgiref package
............................

The Web Server Gateway Interface (WSGI) v1.0 defines a standard
interface between web servers and Python web applications and is
described in PEP 333 (http://www.python.org/dev/peps/pep-0333).  The
wsgiref package is a reference implementation of the WSGI
specification.

  The package includes a basic HTTP server that will run a WSGI
application; this server is useful for debugging but isn't intended for
production use.  Setting up a server takes only a few lines of code:

    from wsgiref import simple_server

    wsgi_app = ...

    host = ''
    port = 8000
    httpd = simple_server.make_server(host, port, wsgi_app)
    httpd.serve_forever()

<http://www.wsgi.org>
     A central web site for WSGI-related resources.

PEP 333 (http://www.python.org/dev/peps/pep-0333) - Python Web Server Gateway Interface v1.0
     PEP written by Phillip J. Eby.

1.3.14 Build and C API Changes
------------------------------

Changes to Python's build process and to the C API include:

   * The Python source tree was converted from CVS to Subversion,  in a
     complex migration procedure that was supervised and flawlessly
     carried out by Martin von Löwis.  The procedure was developed as PEP
     347 (http://www.python.org/dev/peps/pep-0347).

   * Coverity, a company that markets a source code analysis tool
     called Prevent, provided the results of their examination of the
     Python source code.  The analysis found about 60 bugs that  were
     quickly fixed.  Many of the bugs were refcounting problems, often
     occurring in error-handling code.  See <http://scan.coverity.com>
     for the statistics.

   * The largest change to the C API came from PEP 353
     (http://www.python.org/dev/peps/pep-0353), which modifies the
     interpreter to use a `Py_ssize_t' type definition instead of
     `int'.  See the earlier section PEP 353; Using ssize_t as the index type
     for a discussion of this change.

   * The design of the bytecode compiler has changed a great deal,  no
     longer generating bytecode by traversing the parse tree.  Instead
     the parse tree is converted to an abstract syntax tree (or AST),
     and it is  the abstract syntax tree that's traversed to produce
     the bytecode.

     It's possible for Python code to obtain AST objects by using the
     compile() built-in and specifying `_ast.PyCF_ONLY_AST'
     as the value of the  _flags_ parameter:

         from _ast import PyCF_ONLY_AST
         ast = compile("""a=0
         for i in range(10):
             a += i
         """, "<string>", 'exec', PyCF_ONLY_AST)

         assignment = ast.body[0]
         for_loop = ast.body[1]

     No official documentation has been written for the AST code yet,
     but PEP 339 (http://www.python.org/dev/peps/pep-0339) discusses
     the design.  To start learning about the code, read the definition
     of the various AST nodes in `Parser/Python.asdl'.  A Python script
     reads this file and generates a set of C structure definitions in
     `Include/Python-ast.h'.  The `PyParser_ASTFromString()' and
     `PyParser_ASTFromFile()', defined in `Include/pythonrun.h', take
     Python source as input and return the root of an AST representing
     the contents.  This AST can then be turned into a code object by
     `PyAST_Compile()'.  For more information, read the source code,
     and then ask questions on python-dev.

     The AST code was developed under Jeremy Hylton's management, and
     implemented by (in alphabetical order) Brett Cannon, Nick Coghlan,
     Grant Edwards, John Ehresman, Kurt Kaiser, Neal Norwitz, Tim
     Peters, Armin Rigo, and Neil Schemenauer, plus the participants in
     a number of AST sprints at conferences such as PyCon.

   * Evan Jones's patch to obmalloc, first described in a talk at PyCon
     DC 2005, was applied.  Python 2.4 allocated small objects in
     256K-sized arenas, but never freed arenas.  With this patch,
     Python will free arenas when they're empty.  The net effect is
     that on some platforms, when you allocate many objects, Python's
     memory usage may actually drop when you delete them and the memory
     may be returned to the operating system.  (Implemented by Evan
     Jones, and reworked by Tim Peters.)

     Note that this change means extension modules must be more careful
     when allocating memory.  Python's API has many different functions
     for allocating memory that are grouped into families.  For
     example, see PyMem_Malloc(), PyMem_Realloc(),
     and PyMem_Free() are one family that allocates raw
     memory, while `PyObject_Malloc()', `PyObject_Realloc()', and
     `PyObject_Free()' are another family that's supposed to be used for
     creating Python objects.

     Previously these different families all reduced to the platform's
     `malloc()' and `free()' functions.  This meant  it didn't matter if
     you got things wrong and allocated memory with the `PyMem()'
     function but freed it with the PyObject() function.
     With 2.5's changes to obmalloc, these families now do different
     things and mismatches will probably result in a segfault.  You
     should carefully test your C extension modules with Python 2.5.

   * The built-in set types now have an official C API.  Call
     PySet_New().  and PyFrozenSet_New() to create a
     new set, see PySet_Add() and PySet_Discard().
     to add and remove elements, and PySet_Contains().  and
     PySet_Size() to examine the set's state. (Contributed
     by Raymond Hettinger.)

   * C code can now obtain information about the exact revision of the
     Python interpreter by calling the  Py_GetBuildInfo().
     function that returns a string of build information like this:
     `"trunk:45355:45356M, Apr 13 2006, 07:42:19"'.   (Contributed by
     Barry Warsaw.)

   * Two new macros can be used to indicate C functions that are local
     to the current file so that a faster calling convention can be
     used.  `Py_LOCAL(type)()' declares the function as returning a
     value of the specified _type_ and uses a fast-calling qualifier.
     `Py_LOCAL_INLINE(type)()' does the same thing and also requests the
     function be inlined.  If `PY_LOCAL_AGGRESSIVE()' is defined before
     `python.h' is included, a set of more aggressive optimizations are
     enabled for the module; you should benchmark the results to find
     out if these optimizations actually make the code faster.
     (Contributed by Fredrik Lundh at the NeedForSpeed sprint.)

   * `PyErr_NewException(name, base, dict)()' can now accept a tuple of
     base classes as its _base_ argument.  (Contributed by Georg
     Brandl.)

   * The PyErr_Warn() function for issuing warnings is now
     deprecated in favour of `PyErr_WarnEx(category, message,
     stacklevel)()' which lets you specify the number of stack frames
     separating this function and the caller.  A _stacklevel_ of 1 is
     the function calling PyErr_WarnEx(), 2 is the function
     above that, and so forth.  (Added by Neal Norwitz.)

   * The CPython interpreter is still written in C, but  the code can
     now be compiled with a C++ compiler without errors.   (Implemented
     by Anthony Baxter, Martin von Löwis, Skip Montanaro.)

   * The `PyRange_New()' function was removed.  It was never
     documented, never used in the core code, and had dangerously lax
     error checking.  In the unlikely case that your extensions were
     using it, you can replace it by something like the following:

         range = PyObject_CallFunction((PyObject*) &PyRange_Type, "lll",
                                       start, stop, step);

1.3.14.1 Port-Specific Changes
..............................

   * MacOS X (10.3 and higher): dynamic loading of modules now uses the
     `dlopen()' function instead of MacOS-specific functions.

   * MacOS X: an `--enable-universalsdk' switch was added to the
     *configure* script that compiles the interpreter as a universal
     binary able to run on both PowerPC and Intel processors.
     (Contributed by Ronald Oussoren; issue 2573
     (http://bugs.python.org/issue2573).)

   * Windows: `.dll' is no longer supported as a filename extension for
     extension modules.  `.pyd' is now the only filename extension that
     will be searched for.

1.3.15 Porting to Python 2.5
----------------------------

This section lists previously described changes that may require
changes to your code:

   * ASCII is now the default encoding for modules.  It's now  a syntax
     error if a module contains string literals with 8-bit characters
     but doesn't have an encoding declaration.  In Python 2.4 this
     triggered a warning, not a syntax error.

   * Previously, the `gi_frame' attribute of a generator was always a
     frame object.  Because of the PEP 342
     (http://www.python.org/dev/peps/pep-0342) changes described in
     section PEP 342; New Generator Features, it's now
     possible for `gi_frame' to be `None'.

   * A new warning, `UnicodeWarning', is triggered when  you attempt to
     compare a Unicode string and an 8-bit string that can't be
     converted to Unicode using the default ASCII encoding.  Previously
     such comparisons would raise a `UnicodeDecodeError' exception.

   * Library: the csv module is now stricter about
     multi-line quoted fields.  If your files contain newlines embedded
     within fields, the input should be split into lines in a manner
     which preserves the newline characters.

   * Library: the locale module's  format().
     function's would previously  accept any string as long as no more
     than one %char specifier appeared.  In Python 2.5, the argument
     must be exactly one %char specifier with no surrounding text.

   * Library: The pickle and cPickle modules no
     longer accept a return value of `None' from the
     __reduce__() method; the method must return a tuple of
     arguments instead.  The modules also no longer accept the
     deprecated _bin_ keyword parameter.

   * Library: The SimpleXMLRPCServer and
     DocXMLRPCServer.  classes now have a `rpc_paths' attribute
     that constrains XML-RPC operations to a limited set of URL paths;
     the default is to allow only `'/'' and `'/RPC2''.  Setting
     `rpc_paths' to `None' or an empty tuple disables  this path
     checking.

   * C API: Many functions now use `Py_ssize_t'  instead of `int' to
     allow processing more data on 64-bit machines.  Extension code may
     need to make the same change to avoid warnings and to support
     64-bit machines.  See the earlier section PEP 353; Using ssize_t as the index type
     for a discussion of this change.

   * C API:  The obmalloc changes mean that  you must be careful to not
     mix usage of the `PyMem_*()' and `PyObject_*()' families of
     functions. Memory allocated with  one family's `*_Malloc()' must
     be  freed with the corresponding family's `*_Free()' function.

1.3.16 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Georg Brandl, Nick Coghlan, Phillip J. Eby, Lars Gustäbel,
Raymond Hettinger, Ralf W. Grosse- Kunstleve, Kent Johnson, Iain Lowe,
Martin von Löwis, Fredrik Lundh, Andrew McNamara, Skip Montanaro,
Gustavo Niemeyer, Paul Prescod, James Pryor, Mike Rovner, Scott
Weikart, Barry Warsaw, Thomas Wouters.

1.4 What's New in Python 2.4
============================

     Author: A.M. Kuchling

  This article explains the new features in Python 2.4.1, released on
March 30, 2005.

  Python 2.4 is a medium-sized release.  It doesn't introduce as many
changes as the radical Python 2.2, but introduces more features than
the conservative 2.3 release.  The most significant new language
features are function decorators and generator expressions; most other
changes are to the standard library.

  According to the CVS change logs, there were 481 patches applied and
502 bugs fixed between Python 2.3 and 2.4.  Both figures are likely to
be underestimates.

  This article doesn't attempt to provide a complete specification of
every single new feature, but instead provides a brief introduction to
each feature.  For full details, you should refer to the documentation
for Python 2.4, such as the Python Library Reference and the Python
Reference Manual.  Often you will be referred to the PEP for a
particular new feature for explanations of the implementation and
design rationale.

1.4.1 PEP 218: Built-In Set Objects
-----------------------------------

Python 2.3 introduced the sets module.  C implementations
of set data types have now been added to the Python core as two new
built-in types, `set(iterable)()' and `frozenset(iterable)()'.  They
provide high speed operations for membership testing, for eliminating
duplicates from sequences, and for mathematical operations like unions,
intersections, differences, and symmetric differences.

    >>> a = set('abracadabra')              # form a set from a string
    >>> 'z' in a                            # fast membership testing
    False
    >>> a                                   # unique letters in a
    set(['a', 'r', 'b', 'c', 'd'])
    >>> ''.join(a)                          # convert back into a string
    'arbcd'

    >>> b = set('alacazam')                 # form a second set
    >>> a - b                               # letters in a but not in b
    set(['r', 'd', 'b'])
    >>> a | b                               # letters in either a or b
    set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
    >>> a & b                               # letters in both a and b
    set(['a', 'c'])
    >>> a ^ b                               # letters in a or b but not both
    set(['r', 'd', 'b', 'm', 'z', 'l'])

    >>> a.add('z')                          # add a new element
    >>> a.update('wxy')                     # add multiple new elements
    >>> a
    set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
    >>> a.remove('x')                       # take one element out
    >>> a
    set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])

The frozenset() type is an immutable version of
set(). Since it is immutable and hashable, it may be used as a
dictionary key or as a member of another set.

  The sets module remains in the standard library, and may
be useful if you wish to subclass the `Set' or `ImmutableSet' classes.
There are currently no plans to deprecate the module.

PEP 218 (http://www.python.org/dev/peps/pep-0218) - Adding a Built-In Set Object Type
     Originally proposed by Greg Wilson and ultimately implemented by
     Raymond Hettinger.

1.4.2 PEP 237: Unifying Long Integers and Integers
--------------------------------------------------

The lengthy transition process for this PEP, begun in Python 2.2, takes
another step forward in Python 2.4.  In 2.3, certain integer operations
that would behave differently after int/long unification triggered
FutureWarning.  warnings and returned values limited to 32
or 64 bits (depending on your platform).  In 2.4, these expressions no
longer produce a warning and instead produce a different result that's
usually a long integer.

  The problematic expressions are primarily left shifts and lengthy
hexadecimal and octal constants.  For example, `2 << 32' results in a
warning in 2.3, evaluating to 0 on 32-bit platforms.  In Python 2.4,
this expression now returns the correct answer, 8589934592.

PEP 237 (http://www.python.org/dev/peps/pep-0237) - Unifying Long Integers and Integers
     Original PEP written by Moshe Zadka and GvR.  The changes for 2.4
     were implemented by  Kalle Svensson.

1.4.3 PEP 289: Generator Expressions
------------------------------------

The iterator feature introduced in Python 2.2 and the itertools
module make it easier to write programs that loop through large
data sets without having the entire data set in memory at one time.
List comprehensions don't fit into this picture very well because they
produce a Python list object containing all of the items.  This
unavoidably pulls all of the objects into memory, which can be a
problem if your data set is very large.  When trying to write a
functionally-styled program, it would be natural to write something
like:

    links = [link for link in get_all_links() if not link.followed]
    for link in links:
        ...

instead of

    for link in get_all_links():
        if link.followed:
            continue
        ...

The first form is more concise and perhaps more readable, but if you're
dealing with a large number of link objects you'd have to write the
second form to avoid having all link objects in memory at the same time.

  Generator expressions work similarly to list comprehensions but don't
materialize the entire list; instead they create a generator that will
return elements one by one.  The above example could be written as:

    links = (link for link in get_all_links() if not link.followed)
    for link in links:
        ...

Generator expressions always have to be written inside parentheses, as
in the above example.  The parentheses signalling a function call also
count, so if you want to create an iterator that will be immediately
passed to a function you could write:

    print sum(obj.count for obj in list_all_objects())

Generator expressions differ from list comprehensions in various small
ways.  Most notably, the loop variable (_obj_ in the above example) is
not accessible outside of the generator expression.  List
comprehensions leave the variable assigned to its last value; future
versions of Python will change this, making list comprehensions match
generator expressions in this respect.

PEP 289 (http://www.python.org/dev/peps/pep-0289) - Generator Expressions
     Proposed by Raymond Hettinger and implemented by Jiwon Seo with
     early efforts steered by Hye-Shik Chang.

1.4.4 PEP 292: Simpler String Substitutions
-------------------------------------------

Some new classes in the standard library provide an alternative
mechanism for substituting variables into strings; this style of
substitution may be better for applications where untrained users need
to edit templates.

  The usual way of substituting variables by name is the `%' operator:

    >>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}
    '2: The Best of Times'

When writing the template string, it can be easy to forget the `i' or
`s' after the closing parenthesis.  This isn't a big problem if the
template is in a Python module, because you run the code, get an
"Unsupported format character" ValueError, and fix the
problem.  However, consider an application such as Mailman where
template strings or translations are being edited by users who aren't
aware of the Python language.  The format string's syntax is complicated
to explain to such users, and if they make a mistake, it's difficult to
provide helpful feedback to them.

  PEP 292 adds a `Template' class to the string module that
uses `$' to indicate a substitution:

    >>> import string
    >>> t = string.Template('$page: $title')
    >>> t.substitute({'page':2, 'title': 'The Best of Times'})
    '2: The Best of Times'

If a key is missing from the dictionary, the `substitute()' method will
raise a KeyError.  There's also a `safe_substitute()' method
that ignores missing keys:

    >>> t = string.Template('$page: $title')
    >>> t.safe_substitute({'page':3})
    '3: $title'

PEP 292 (http://www.python.org/dev/peps/pep-0292) - Simpler String Substitutions
     Written and implemented  by Barry Warsaw.

1.4.5 PEP 318: Decorators for Functions and Methods
---------------------------------------------------

Python 2.2 extended Python's object model by adding static methods and
class methods, but it didn't extend Python's syntax to provide any new
way of defining static or class methods.  Instead, you had to write a
def statement in the usual way, and pass the resulting
method to a staticmethod() or classmethod().
function that would wrap up the function as a method of the new type.
Your code would look like this:

    class C:
       def meth (cls):
           ...

       meth = classmethod(meth)   # Rebind name to wrapped-up class method

If the method was very long, it would be easy to miss or forget the
classmethod() invocation after the function body.

  The intention was always to add some syntax to make such definitions
more readable, but at the time of 2.2's release a good syntax was not
obvious.  Today a good syntax _still_ isn't obvious but users are
asking for easier access to the feature; a new syntactic feature has
been added to meet this need.

  The new feature is called "function decorators".  The name comes from
the idea that classmethod(), staticmethod(), and
friends are storing additional information on a function object;
they're _decorating_ functions with more details.

  The notation borrows from Java and uses the `'@'' character as an
indicator.  Using the new syntax, the example above would be written:

    class C:

       @classmethod
       def meth (cls):
           ...

The `@classmethod' is shorthand for the `meth=classmethod(meth)'
assignment.  More generally, if you have the following:

    @A
    @B
    @C
    def f ():
        ...

It's equivalent to the following pre-decorator code:

    def f(): ...
    f = A(B(C(f)))

Decorators must come on the line before a function definition, one
decorator per line, and can't be on the same line as the def statement,
meaning that `@A def f(): ...' is illegal.  You can only decorate
function definitions, either at the module level or inside a class; you
can't decorate class definitions.

  A decorator is just a function that takes the function to be
decorated as an argument and returns either the same function or some
new object.  The return value of the decorator need not be callable
(though it typically is), unless further decorators will be applied to
the result.  It's easy to write your own decorators.  The following
simple example just sets an attribute on the function object:

    >>> def deco(func):
    ...    func.attr = 'decorated'
    ...    return func
    ...
    >>> @deco
    ... def f(): pass
    ...
    >>> f
    <function f at 0x402ef0d4>
    >>> f.attr
    'decorated'
    >>>

As a slightly more realistic example, the following decorator checks
that the supplied argument is an integer:

    def require_int (func):
        def wrapper (arg):
            assert isinstance(arg, int)
            return func(arg)

        return wrapper

    @require_int
    def p1 (arg):
        print arg

    @require_int
    def p2(arg):
        print arg*2

An example in PEP 318 (http://www.python.org/dev/peps/pep-0318)
contains a fancier version of this idea that lets you both specify the
required type and check the returned type.

  Decorator functions can take arguments.  If arguments are supplied,
your decorator function is called with only those arguments and must
return a new decorator function; this function must take a single
function and return a function, as previously described.  In other
words, `@A @B @C(args)' becomes:

    def f(): ...
    _deco = C(args)
    f = A(B(_deco(f)))

Getting this right can be slightly brain-bending, but it's not too
difficult.

  A small related change makes the `func_name' attribute of functions
writable.  This attribute is used to display function names in
tracebacks, so decorators should change the name of any new function
that's constructed and returned.

PEP 318 (http://www.python.org/dev/peps/pep-0318) - Decorators for Functions, Methods and Classes
     Written  by Kevin D. Smith, Jim Jewett, and Skip Montanaro.
     Several people wrote patches implementing function decorators, but
     the one that was actually checked in was patch #979728, written by
     Mark Russell.

<http://www.python.org/moin/PythonDecoratorLibrary>
     This Wiki page contains several examples of decorators.

1.4.6 PEP 322: Reverse Iteration
--------------------------------

A new built-in function, `reversed(seq)()', takes a sequence and
returns an iterator that loops over the elements of the sequence  in
reverse order.

    >>> for i in reversed(xrange(1,4)):
    ...    print i
    ...
    3
    2
    1

Compared to extended slicing, such as `range(1,4)[::-1]', see
reversed() is easier to read, runs faster, and uses substantially
less memory.

  Note that reversed() only accepts sequences, not
arbitrary iterators.  If you want to reverse an iterator, first convert
it to  a list with list().

    >>> input = open('/etc/passwd', 'r')
    >>> for line in reversed(list(input)):
    ...   print line
    ...
    root:*:0:0:System Administrator:/var/root:/bin/tcsh
      ...

PEP 322 (http://www.python.org/dev/peps/pep-0322) - Reverse Iteration
     Written and implemented by Raymond Hettinger.

1.4.7 PEP 324: New subprocess Module
------------------------------------

The standard library provides a number of ways to execute a subprocess,
offering different features and different levels of complexity.
`os.system(command)()' is easy to use, but slow (it runs a shell process
which executes the command) and dangerous (you have to be careful about
escaping the shell's metacharacters).  The popen2 module
offers classes that can capture standard output and standard error from
the subprocess, but the naming is confusing.  The subprocess
module cleans  this up, providing a unified interface that offers
all the features you might need.

  Instead of popen2.'s collection of classes, see
subprocess contains a single class called `Popen'  whose
constructor supports a number of different keyword arguments.

    class Popen(args, bufsize=0, executable=None,
                stdin=None, stdout=None, stderr=None,
                preexec_fn=None, close_fds=False, shell=False,
                cwd=None, env=None, universal_newlines=False,
                startupinfo=None, creationflags=0):

_args_ is commonly a sequence of strings that will be the arguments to
the program executed as the subprocess.  (If the _shell_ argument is
true, _args_ can be a string which will then be passed on to the shell
for interpretation, just as os.system() does.)

  _stdin_, _stdout_, and _stderr_ specify what the subprocess's input,
output, and error streams will be.  You can provide a file object or a
file descriptor, or you can use the constant `subprocess.PIPE' to
create a pipe between the subprocess and the parent.

  The constructor has a number of handy options:

   * _close_fds_ requests that all file descriptors be closed before
     running the subprocess.

   * _cwd_ specifies the working directory in which the subprocess will
     be executed (defaulting to whatever the parent's working directory
     is).

   * _env_ is a dictionary specifying environment variables.

   * _preexec_fn_ is a function that gets called before the child is
     started.

   * _universal_newlines_ opens the child's input and output using
     Python's universal newline feature.

  Once you've created the `Popen' instance,  you can call its `wait()'
method to pause until the subprocess has exited, `poll()' to check if
it's exited without pausing,  or `communicate(data)()' to send the
string _data_ to the subprocess's standard input.
`communicate(data)()'  then reads any data that the subprocess has sent
to its standard output  or standard error, returning a tuple
`(stdout_data, stderr_data)'.

  `call()' is a shortcut that passes its arguments along to the `Popen'
constructor, waits for the command to complete, and returns the status
code of the subprocess.  It can serve as a safer analog to
os.system().:

    sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])
    if sts == 0:
        # Success
        ...
    else:
        # dpkg returned an error
        ...

The command is invoked without use of the shell.  If you really do want
to  use the shell, you can add `shell=True' as a keyword argument and
provide a string instead of a sequence:

    sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)

The PEP takes various examples of shell and Python code and shows how
they'd be translated into Python code that uses subprocess.
Reading this section of the PEP is highly recommended.

PEP 324 (http://www.python.org/dev/peps/pep-0324) - subprocess - New process module
     Written and implemented by Peter Åstrand, with assistance from
     Fredrik Lundh and others.

1.4.8 PEP 327: Decimal Data Type
--------------------------------

Python has always supported floating-point (FP) numbers, based on the
underlying C `double' type, as a data type.  However, while most
programming languages provide a floating-point type, many people (even
programmers) are unaware that floating-point numbers don't represent
certain decimal fractions accurately.  The new `Decimal' type can
represent these fractions accurately, up to a user-specified precision
limit.

1.4.8.1 Why is Decimal needed?
..............................

The limitations arise from the representation used for floating-point
numbers.  FP numbers are made up of three components:

   * The sign, which is positive or negative.

   * The mantissa, which is a single-digit binary number   followed by
     a fractional part.  For example, `1.01' in base-2 notation is `1 +
     0/2 + 1/4', or 1.25 in decimal notation.

   * The exponent, which tells where the decimal point is located in
     the number represented.

  For example, the number 1.25 has positive sign, a mantissa value of
1.01 (in binary), and an exponent of 0 (the decimal point doesn't need
to be shifted).  The number 5 has the same sign and mantissa, but the
exponent is 2 because the mantissa is multiplied by 4 (2 to the power
of the exponent 2); 1.25 * 4 equals 5.

  Modern systems usually provide floating-point support that conforms
to a standard called IEEE 754.  C's `double' type is usually
implemented as a 64-bit IEEE 754 number, which uses 52 bits of space
for the mantissa.  This means that numbers can only be specified to 52
bits of precision.  If you're trying to represent numbers whose
expansion repeats endlessly, the expansion is cut off after 52 bits.
Unfortunately, most software needs to produce output in base 10, and
common fractions in base 10 are often repeating decimals in binary.
For example, 1.1 decimal is binary `1.0001100110011 ...'; .1 = 1/16 +
1/32 + 1/256 plus an infinite number of additional terms.  IEEE 754 has
to chop off that infinitely repeated decimal after 52 digits, so the
representation is slightly inaccurate.

  Sometimes you can see this inaccuracy when the number is printed:

    >>> 1.1
    1.1000000000000001

The inaccuracy isn't always visible when you print the number because
the FP-to- decimal-string conversion is provided by the C library, and
most C libraries try to produce sensible output.  Even if it's not
displayed, however, the inaccuracy is still there and subsequent
operations can magnify the error.

  For many applications this doesn't matter.  If I'm plotting points and
displaying them on my monitor, the difference between 1.1 and
1.1000000000000001 is too small to be visible.  Reports often limit
output to a certain number of decimal places, and if you round the
number to two or three or even eight decimal places, the error is never
apparent.  However, for applications where it does matter,  it's a lot
of work to implement your own custom arithmetic routines.

  Hence, the `Decimal' type was created.

1.4.8.2 The `Decimal' type
..........................

A new module, see decimal, was added to Python's standard
library.  It contains two classes, `Decimal' and `Context'.  `Decimal'
instances represent numbers, and `Context' instances are used to wrap up
various settings such as the precision and default rounding mode.

  `Decimal' instances are immutable, like regular Python integers and FP
numbers; once it's been created, you can't change the value an instance
represents.  `Decimal' instances can be created from integers or
strings:

    >>> import decimal
    >>> decimal.Decimal(1972)
    Decimal("1972")
    >>> decimal.Decimal("1.1")
    Decimal("1.1")

You can also provide tuples containing the sign, the mantissa
represented  as a tuple of decimal digits, and the exponent:

    >>> decimal.Decimal((1, (1, 4, 7, 5), -2))
    Decimal("-14.75")

Cautionary note: the sign bit is a Boolean value, so 0 is positive and
1 is negative.

  Converting from floating-point numbers poses a bit of a problem:
should the FP number representing 1.1 turn into the decimal number for
exactly 1.1, or for 1.1 plus whatever inaccuracies are introduced? The
decision was to dodge the issue and leave such a conversion out of the
API.  Instead, you should convert the floating-point number into a
string using the desired precision and pass the string to the `Decimal'
constructor:

    >>> f = 1.1
    >>> decimal.Decimal(str(f))
    Decimal("1.1")
    >>> decimal.Decimal('%.12f' % f)
    Decimal("1.100000000000")

Once you have `Decimal' instances, you can perform the usual
mathematical operations on them.  One limitation: exponentiation
requires an integer exponent:

    >>> a = decimal.Decimal('35.72')
    >>> b = decimal.Decimal('1.73')
    >>> a+b
    Decimal("37.45")
    >>> a-b
    Decimal("33.99")
    >>> a*b
    Decimal("61.7956")
    >>> a/b
    Decimal("20.64739884393063583815028902")
    >>> a ** 2
    Decimal("1275.9184")
    >>> a**b
    Traceback (most recent call last):
      ...
    decimal.InvalidOperation: x ** (non-integer)

You can combine `Decimal' instances with integers, but not with
floating- point numbers:

    >>> a + 4
    Decimal("39.72")
    >>> a + 4.5
    Traceback (most recent call last):
      ...
    TypeError: You can interact Decimal only with int, long or Decimal data types.
    >>>

`Decimal' numbers can be used with the math and
cmath.  modules, but note that they'll be immediately converted to
floating-point numbers before the operation is performed, resulting in
a possible loss of precision and accuracy.  You'll also get back a
regular floating-point number and not a `Decimal'.

    >>> import math, cmath
    >>> d = decimal.Decimal('123456789012.345')
    >>> math.sqrt(d)
    351364.18288201344
    >>> cmath.sqrt(-d)
    351364.18288201344j

`Decimal' instances have a `sqrt()' method that returns a `Decimal',
but if you need other things such as trigonometric functions you'll
have to implement them.

    >>> d.sqrt()
    Decimal("351364.1828820134592177245001")

1.4.8.3 The `Context' type
..........................

Instances of the `Context' class encapsulate several settings for
decimal operations:

   * `prec' is the precision, the number of decimal places.

   * `rounding' specifies the rounding mode.  The decimal.
     module has constants for the various possibilities: `ROUND_DOWN',
     `ROUND_CEILING',  `ROUND_HALF_EVEN', and various others.

   * `traps' is a dictionary specifying what happens on encountering
     certain error conditions: either  an exception is raised or  a
     value is returned.  Some examples of error conditions are division
     by zero, loss of precision, and overflow.

  There's a thread-local default context available by calling
`getcontext()'; you can change the properties of this context to alter
the default precision, rounding, or trap handling.  The following
example shows the effect of changing the precision of the default
context:

    >>> decimal.getcontext().prec
    28
    >>> decimal.Decimal(1) / decimal.Decimal(7)
    Decimal("0.1428571428571428571428571429")
    >>> decimal.getcontext().prec = 9
    >>> decimal.Decimal(1) / decimal.Decimal(7)
    Decimal("0.142857143")

The default action for error conditions is selectable; the module can
either return a special value such as infinity or not-a-number, or
exceptions can be raised:

    >>> decimal.Decimal(1) / decimal.Decimal(0)
    Traceback (most recent call last):
      ...
    decimal.DivisionByZero: x / 0
    >>> decimal.getcontext().traps[decimal.DivisionByZero] = False
    >>> decimal.Decimal(1) / decimal.Decimal(0)
    Decimal("Infinity")
    >>>

The `Context' instance also has various methods for formatting  numbers
such as `to_eng_string()' and `to_sci_string()'.

  For more information, see the documentation for the decimal
module, which includes a quick-start tutorial and a reference.

PEP 327 (http://www.python.org/dev/peps/pep-0327) - Decimal Data Type
     Written by Facundo Batista and implemented by Facundo Batista,
     Eric Price, Raymond Hettinger, Aahz, and Tim Peters.

<http://www.lahey.com/float.htm>
     The article uses Fortran code to illustrate many of the problems
     that floating- point inaccuracy can cause.

<http://www2.hursley.ibm.com/decimal/>
     A description of a decimal-based representation.  This
     representation is being proposed as a standard, and underlies the
     new Python decimal type.  Much of this material was written by
     Mike Cowlishaw, designer of the Rexx language.

1.4.9 PEP 328: Multi-line Imports
---------------------------------

One language change is a small syntactic tweak aimed at making it
easier to import many names from a module.  In a `from module import
names' statement, _names_ is a sequence of names separated by commas.
If the sequence is  very long, you can either write multiple imports
from the same module, or you can use backslashes to escape the line
endings like this:

    from SimpleXMLRPCServer import SimpleXMLRPCServer,\
                SimpleXMLRPCRequestHandler,\
                CGIXMLRPCRequestHandler,\
                resolve_dotted_attribute

The syntactic change in Python 2.4 simply allows putting the names
within parentheses.  Python ignores newlines within a parenthesized
expression, so the backslashes are no longer needed:

    from SimpleXMLRPCServer import (SimpleXMLRPCServer,
                                    SimpleXMLRPCRequestHandler,
                                    CGIXMLRPCRequestHandler,
                                    resolve_dotted_attribute)

The PEP also proposes that all import statements be
absolute imports, with a leading `.' character to indicate a relative
import.  This part of the PEP was not implemented for Python 2.4, but
was completed for Python 2.5.

PEP 328 (http://www.python.org/dev/peps/pep-0328) - Imports: Multi-Line and Absolute/Relative
     Written by Aahz.  Multi-line imports were implemented by Dima
     Dorfman.

1.4.10 PEP 331: Locale-Independent Float/String Conversions
-----------------------------------------------------------

The locale modules lets Python software select various
conversions and display conventions that are localized to a particular
country or language.  However, the module was careful to not change the
numeric locale because various functions in Python's implementation
required that the numeric locale remain set to the `'C'' locale.  Often
this was because the code was using the C library's `atof()' function.

  Not setting the numeric locale caused trouble for extensions that
used third- party C libraries, however, because they wouldn't have the
correct locale set.  The motivating example was GTK+, whose user
interface widgets weren't displaying numbers in the current locale.

  The solution described in the PEP is to add three new functions to
the Python API that perform ASCII-only conversions, ignoring the locale
setting:

   * `PyOS_ascii_strtod(str, ptr)()'  and `PyOS_ascii_atof(str, ptr)()'
     both convert a string to a C `double'.

   * `PyOS_ascii_formatd(buffer, buf_len, format, d)()' converts a
     `double' to an ASCII string.

  The code for these functions came from the GLib library
(<http://library.gnome.org/devel/glib/stable/>), whose developers kindly
relicensed the relevant functions and donated them to the Python
Software Foundation.  The locale module  can now change the
numeric locale, letting extensions such as GTK+  produce the correct
results.

PEP 331 (http://www.python.org/dev/peps/pep-0331) - Locale-Independent Float/String Conversions
     Written by Christian R. Reis, and implemented by Gustavo Carneiro.

1.4.11 Other Language Changes
-----------------------------

Here are all of the changes that Python 2.4 makes to the core Python
language.

   * Decorators for functions and methods were added ( PEP 318
     (http://www.python.org/dev/peps/pep-0318)).

   * Built-in set() and frozenset() types were
     added ( PEP 218 (http://www.python.org/dev/peps/pep-0218)).  Other
     new built-ins include the `reversed(seq)()' function ( PEP 322
     (http://www.python.org/dev/peps/pep-0322)).

   * Generator expressions were added ( PEP 289
     (http://www.python.org/dev/peps/pep-0289)).

   * Certain numeric expressions no longer return values restricted to
     32 or 64 bits ( PEP 237 (http://www.python.org/dev/peps/pep-0237)).

   * You can now put parentheses around the list of names in a `from
     module import names' statement ( PEP 328
     (http://www.python.org/dev/peps/pep-0328)).

   * The dict.update() method now accepts the same argument
     forms as the dict constructor.  This includes any
     mapping, any iterable of key/value pairs, and keyword arguments.
     (Contributed by Raymond Hettinger.)

   * The string methods `ljust()', `rjust()', and `center()' now take
     an optional argument for specifying a fill character other than a
     space.  (Contributed by Raymond Hettinger.)

   * Strings also gained an `rsplit()' method that works like the
     `split()' method but splits from the end of the string.
     (Contributed by Sean Reifschneider.)

         >>> 'www.python.org'.split('.', 1)
         ['www', 'python.org']
         'www.python.org'.rsplit('.', 1)
         ['www.python', 'org']

   * Three keyword parameters, _cmp_, _key_, and _reverse_, were added
     to the `sort()' method of lists. These parameters make some common
     usages of `sort()' simpler. All of these parameters are optional.

     For the _cmp_ parameter, the value should be a comparison function
     that takes two parameters and returns -1, 0, or +1 depending on
     how the parameters compare.  This function will then be used to
     sort the list.  Previously this was the only parameter that could
     be provided to `sort()'.

     _key_ should be a single-parameter function that takes a list
     element and returns a comparison key for the element.  The list is
     then sorted using the comparison keys.  The following example
     sorts a list case-insensitively:

         >>> L = ['A', 'b', 'c', 'D']
         >>> L.sort()                 # Case-sensitive sort
         >>> L
         ['A', 'D', 'b', 'c']
         >>> # Using 'key' parameter to sort list
         >>> L.sort(key=lambda x: x.lower())
         >>> L
         ['A', 'b', 'c', 'D']
         >>> # Old-fashioned way
         >>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
         >>> L
         ['A', 'b', 'c', 'D']

     The last example, which uses the _cmp_ parameter, is the old way
     to perform a case-insensitive sort.  It works but is slower than
     using a _key_ parameter.  Using _key_ calls `lower()' method once
     for each element in the list while using _cmp_ will call it twice
     for each comparison, so using _key_ saves on invocations of the
     `lower()' method.

     For simple key functions and comparison functions, it is often
     possible to avoid a lambda expression by using an
     unbound method instead.  For example, the above case-insensitive
     sort is best written as:

         >>> L.sort(key=str.lower)
         >>> L
         ['A', 'b', 'c', 'D']

     Finally, the _reverse_ parameter takes a Boolean value.  If the
     value is true, the list will be sorted into reverse order. Instead
     of `L.sort() ; L.reverse()', you can now write
     `L.sort(reverse=True)'.

     The results of sorting are now guaranteed to be stable.  This
     means that two entries with equal keys will be returned in the
     same order as they were input.  For example, you can sort a list
     of people by name, and then sort the list by age, resulting in a
     list sorted by age where people with the same age are in
     name-sorted order.

     (All changes to `sort()' contributed by Raymond Hettinger.)

   * There is a new built-in function `sorted(iterable)()' that works
     like the in-place `list.sort()' method but can be used in
     expressions.  The differences are:

   * the input may be any iterable;

   * a newly formed copy is sorted, leaving the original intact; and

   * the expression returns the new sorted copy

         >>> L = [9,7,8,3,2,4,1,6,5]
         >>> [10+i for i in sorted(L)]       # usable in a list comprehension
         [11, 12, 13, 14, 15, 16, 17, 18, 19]
         >>> L                               # original is left unchanged
         [9,7,8,3,2,4,1,6,5]
         >>> sorted('Monty Python')          # any iterable may be an input
         [' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']

         >>> # List the contents of a dict sorted by key values
         >>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
         >>> for k, v in sorted(colormap.iteritems()):
         ...     print k, v
         ...
         black 4
         blue 2
         green 3
         red 1
         yellow 5

     (Contributed by Raymond Hettinger.)

   * Integer operations will no longer trigger an `OverflowWarning'. The
     `OverflowWarning' warning will disappear in Python 2.5.

   * The interpreter gained a new switch, see -m, that takes a
     name, searches for the corresponding  module on `sys.path', and
     runs the module as a script.  For example,  you can now run the
     Python profiler with `python -m profile'.  (Contributed by Nick
     Coghlan.)

   * The `eval(expr, globals, locals)()' and `execfile(filename,
     globals, locals)()' functions and the exec statement
     now accept any mapping type for the _locals_ parameter.
     Previously this had to be a regular Python dictionary.
     (Contributed by Raymond Hettinger.)

   * The zip() built-in function and
     itertools.izip() now return an empty list if called with no
     arguments. Previously they raised a TypeError.
     exception.  This makes them more suitable for use with variable
     length argument lists:

         >>> def transpose(array):
         ...    return zip(*array)
         ...
         >>> transpose([(1,2,3), (4,5,6)])
         [(1, 4), (2, 5), (3, 6)]
         >>> transpose([])
         []

     (Contributed by Raymond Hettinger.)

   * Encountering a failure while importing a module no longer leaves a
     partially- initialized module object in `sys.modules'.  The
     incomplete module object left behind would fool further imports of
     the same module into succeeding, leading to confusing errors.
     (Fixed by Tim Peters.)

   * None is now a constant; code that binds a new value to
     the name `None' is now a syntax error. (Contributed by Raymond
     Hettinger.)

1.4.11.1 Optimizations
......................

   * The inner loops for list and tuple slicing were optimized and now
     run about one-third faster.  The inner loops for dictionaries were
     also optimized, resulting in performance boosts for `keys()',
     `values()', `items()', `iterkeys()', `itervalues()', and
     `iteritems()'. (Contributed by Raymond Hettinger.)

   * The machinery for growing and shrinking lists was optimized for
     speed and for space efficiency.  Appending and popping from lists
     now runs faster due to more efficient code paths and less frequent
     use of the underlying system `realloc()'.  List comprehensions
     also benefit.   `list.extend()' was also optimized and no longer
     converts its argument into a temporary list before extending the
     base list.  (Contributed by Raymond Hettinger.)

   * list(), tuple(), map(),
     filter(), and zip() now run several times faster
     with non-sequence arguments that supply a __len__().
     method.  (Contributed by Raymond Hettinger.)

   * The methods `list.__getitem__()', `dict.__getitem__()', and
     `dict.__contains__()' are are now implemented as
     `method_descriptor' objects rather than `wrapper_descriptor'
     objects.  This form of  access doubles their performance and makes
     them more suitable for use as arguments to functionals:
     `map(mydict.__getitem__, keylist)'. (Contributed by Raymond
     Hettinger.)

   * Added a new opcode, `LIST_APPEND', that simplifies the generated
     bytecode for list comprehensions and speeds them up by about a
     third.  (Contributed by Raymond Hettinger.)

   * The peephole bytecode optimizer has been improved to  produce
     shorter, faster bytecode; remarkably, the resulting bytecode is
     more readable.  (Enhanced by Raymond Hettinger.)

   * String concatenations in statements of the form `s = s + "abc"'
     and `s += "abc"' are now performed more efficiently in certain
     circumstances.  This optimization won't be present in other Python
     implementations such as Jython, so you shouldn't rely on it; using
     the `join()' method of strings is still recommended when you want
     to efficiently glue a large number of strings together.
     (Contributed by Armin Rigo.)

  The net result of the 2.4 optimizations is that Python 2.4 runs the
pystone benchmark around 5% faster than Python 2.3 and 35% faster than
Python 2.2.  (pystone is not a particularly good benchmark, but it's
the most commonly used measurement of Python's performance.  Your own
applications may show greater or smaller benefits from Python 2.4.)

1.4.12 New, Improved, and Deprecated Modules
--------------------------------------------

As usual, Python's standard library received a number of enhancements
and bug fixes.  Here's a partial list of the most notable changes,
sorted alphabetically by module name. Consult the `Misc/NEWS' file in
the source tree for a more complete list of changes, or look through
the CVS logs for all the details.

   * The asyncore module's `loop()' function now has a
     _count_ parameter that lets you perform a limited number of passes
     through the polling loop.  The default is still to loop forever.

   * The base64 module now has more complete RFC 3548
     support for Base64, Base32, and Base16 encoding and decoding,
     including optional case folding and optional alternative
     alphabets. (Contributed by Barry Warsaw.)

   * The bisect module now has an underlying C
     implementation for improved performance. (Contributed by Dmitry
     Vasiliev.)

   * The CJKCodecs collections of East Asian codecs, maintained by
     Hye-Shik Chang, was integrated into 2.4.   The new encodings are:

   * Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz

   * Chinese (ROC): big5, cp950

   *
    Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp,
          iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext,
          iso-2022-jp-2004, shift-jis, shift-jisx0213, shift-jis-2004

   * Korean: cp949, euc-kr, johab, iso-2022-kr

   * Some other new encodings were added: HP Roman8,  ISO_8859-11,
     ISO_8859-16, PCTP-154, and TIS-620.

   * The UTF-8 and UTF-16 codecs now cope better with receiving partial
     input.  Previously the `StreamReader' class would try to read more
     data, making it impossible to resume decoding from the stream.
     The `read()' method will now return as much data as it can and
     future calls will resume decoding where previous ones left off.
     (Implemented by Walter Dörwald.)

   * There is a new collections module for  various
     specialized collection datatypes.   Currently it contains just one
     type, `deque',  a double- ended queue that supports efficiently
     adding and removing elements from either end:

         >>> from collections import deque
         >>> d = deque('ghi')        # make a new deque with three items
         >>> d.append('j')           # add a new entry to the right side
         >>> d.appendleft('f')       # add a new entry to the left side
         >>> d                       # show the representation of the deque
         deque(['f', 'g', 'h', 'i', 'j'])
         >>> d.pop()                 # return and remove the rightmost item
         'j'
         >>> d.popleft()             # return and remove the leftmost item
         'f'
         >>> list(d)                 # list the contents of the deque
         ['g', 'h', 'i']
         >>> 'h' in d                # search the deque
         True

     Several modules, such as the Queue and
     threading modules, now take advantage of
     collections.deque for improved performance.  (Contributed by
     Raymond Hettinger.)

   * The ConfigParser classes have been enhanced slightly.
     The `read()' method now returns a list of the files that were
     successfully parsed, and the set() method raises
     TypeError if passed a _value_ argument that isn't a string.
     (Contributed by John Belmonte and David Goodger.)

   * The curses module now supports the ncurses extension
     `use_default_colors()'.  On platforms where the terminal supports
     transparency, this makes it possible to use a transparent
     background.  (Contributed by Jörg Lehmann.)

   * The difflib module now includes an `HtmlDiff' class
     that creates an HTML table showing a side by side comparison of
     two versions of a text.  (Contributed by Dan Gass.)

   * The email package was updated to version 3.0,  which
     dropped various deprecated APIs and removes support for Python
     versions earlier than 2.3.  The 3.0 version of the package uses a
     new incremental parser for MIME messages, available in the
     `email.FeedParser' module.  The new parser doesn't require reading
     the entire message into memory, and doesn't raise exceptions if a
     message is malformed; instead it records any problems in the
     `defect' attribute of the message.  (Developed by Anthony Baxter,
     Barry Warsaw, Thomas Wouters, and others.)

   * The heapq module has been converted to C.  The
     resulting tenfold improvement in speed makes the module suitable
     for handling high volumes of data.  In addition, the module has
     two new functions `nlargest()' and `nsmallest()' that use heaps to
     find the N largest or smallest values in a dataset without the
     expense of a full sort.  (Contributed by Raymond Hettinger.)

   * The httplib module now contains constants for HTTP
     status codes defined in various HTTP-related RFC documents.
     Constants have names such as `OK', `CREATED', `CONTINUE', and
     `MOVED_PERMANENTLY'; use pydoc to get a full list.  (Contributed by
     Andrew Eland.)

   * The imaplib module now supports IMAP's THREAD command
     (contributed by Yves Dionne) and new `deleteacl()' and
     `myrights()' methods (contributed by Arnaud Mazin).

   * The itertools module gained a `groupby(iterable[,
     *func*])()' function. _iterable_ is something that can be iterated
     over to return a stream of elements, and the optional _func_
     parameter is a function that takes an element and returns a key
     value; if omitted, the key is simply the element itself.
     `groupby()' then groups the elements into subsequences which have
     matching values of the key, and returns a series of 2-tuples
     containing the key value and an iterator over the subsequence.

     Here's an example to make this clearer.  The _key_ function simply
     returns whether a number is even or odd, so the result of
     `groupby()' is to return consecutive runs of odd or even numbers.

         >>> import itertools
         >>> L = [2, 4, 6, 7, 8, 9, 11, 12, 14]
         >>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
         ...    print key_val, list(it)
         ...
         0 [2, 4, 6]
         1 [7]
         0 [8]
         1 [9, 11]
         0 [12, 14]
         >>>

     `groupby()' is typically used with sorted input.  The logic for
     `groupby()' is similar to the Unix `uniq' filter which makes it
     handy for eliminating, counting, or identifying duplicate elements:

         >>> word = 'abracadabra'
         >>> letters = sorted(word)   # Turn string into a sorted list of letters
         >>> letters
         ['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
         >>> for k, g in itertools.groupby(letters):
         ...    print k, list(g)
         ...
         a ['a', 'a', 'a', 'a', 'a']
         b ['b', 'b']
         c ['c']
         d ['d']
         r ['r', 'r']
         >>> # List unique letters
         >>> [k for k, g in groupby(letters)]
         ['a', 'b', 'c', 'd', 'r']
         >>> # Count letter occurrences
         >>> [(k, len(list(g))) for k, g in groupby(letters)]
         [('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]

     (Contributed by Hye-Shik Chang.)

   * itertools also gained a function named `tee(iterator,
     N)()' that returns _N_ independent iterators that replicate
     _iterator_.  If _N_ is omitted, the default is 2.

         >>> L = [1,2,3]
         >>> i1, i2 = itertools.tee(L)
         >>> i1,i2
         (<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
         >>> list(i1)               # Run the first iterator to exhaustion
         [1, 2, 3]
         >>> list(i2)               # Run the second iterator to exhaustion
         [1, 2, 3]

     Note that `tee()' has to keep copies of the values returned  by the
     iterator; in the worst case, it may need to keep all of them.
     This should therefore be used carefully if the leading iterator
     can run far ahead of the trailing iterator in a long stream of
     inputs. If the separation is large, then you might as well use
     list() instead.  When the iterators track closely with
     one another, `tee()' is ideal.  Possible applications include
     bookmarking, windowing, or lookahead iterators. (Contributed by
     Raymond Hettinger.)

   * A number of functions were added to the locale.
     module, such as `bind_textdomain_codeset()' to specify a
     particular encoding and a family of `l*gettext()' functions that
     return messages in the chosen encoding.  (Contributed by Gustavo
     Niemeyer.)

   * Some keyword arguments were added to the logging.
     package's `basicConfig()' function to simplify log configuration.
     The default behavior is to log messages to standard error, but
     various keyword arguments can be specified to log to a particular
     file, change the logging format, or set the logging level. For
     example:

         import logging
         logging.basicConfig(filename='/var/log/application.log',
             level=0,  # Log all messages
             format='%(levelname):%(process):%(thread):%(message)')

     Other additions to the logging package include a
     `log(level, msg)()' convenience method, as well as a
     `TimedRotatingFileHandler' class that rotates its log files at a
     timed interval.  The module already had `RotatingFileHandler',
     which rotated logs once the file exceeded a certain size.  Both
     classes derive from a new `BaseRotatingHandler' class that can be
     used to implement other rotating handlers.

     (Changes implemented by Vinay Sajip.)

   * The marshal module now shares interned strings on
     unpacking a  data structure.  This may shrink the size of certain
     pickle strings, but the primary effect is to make `.pyc' files
     significantly smaller. (Contributed by Martin von Löwis.)

   * The nntplib module's `NNTP' class gained
     `description()' and `descriptions()' methods to retrieve
     newsgroup descriptions for a single group or for a range of
     groups. (Contributed by Jürgen A. Erhard.)

   * Two new functions were added to the operator module,
     `attrgetter(attr)()' and `itemgetter(index)()'. Both functions
     return callables that take a single argument and return the
     corresponding attribute or item; these callables make excellent
     data extractors when used with map().  or
     sorted().  For example:

         >>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
         >>> map(operator.itemgetter(0), L)
         ['c', 'd', 'a', 'b']
         >>> map(operator.itemgetter(1), L)
         [2, 1, 4, 3]
         >>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
         [('d', 1), ('c', 2), ('b', 3), ('a', 4)]

     (Contributed by Raymond Hettinger.)

   * The optparse module was updated in various ways.  The
     module now passes its messages through gettext.gettext(),
     making it possible to internationalize Optik's help and error
     messages.  Help messages for options can now include the string
     `'%default'', which will be replaced by the option's default
     value.  (Contributed by Greg Ward.)

   * The long-term plan is to deprecate the rfc822 module
     in some future Python release in favor of the email.
     package. To this end, the `email.Utils.formatdate()' function has
     been changed to make it usable as a replacement for
     `rfc822.formatdate()'.  You may want to write new e-mail
     processing code with this in mind.  (Change implemented by Anthony
     Baxter.)

   * A new `urandom(n)()' function was added to the os.
     module, returning a string containing _n_ bytes of random data.
     This function provides access to platform-specific sources of
     randomness such as `/dev/urandom' on Linux or the Windows
     CryptoAPI.  (Contributed by Trevor Perrin.)

   * Another new function: `os.path.lexists(path)()'  returns true if
     the file specified by _path_ exists, whether or not it's a
     symbolic link.  This differs from the existing
     `os.path.exists(path)()' function, which returns false if _path_
     is a symlink that points to a destination that doesn't exist.
     (Contributed by Beni Cherniavsky.)

   * A new `getsid()' function was added to the posix.
     module that underlies the os module. (Contributed by
     J. Raynor.)

   * The poplib module now supports POP over SSL.
     (Contributed by Hector Urtubia.)

   * The profile module can now profile C extension
     functions. (Contributed by Nick Bastin.)

   * The random module has a new method called
     `getrandbits(N)()' that returns a long integer _N_ bits in length.
     The existing `randrange()' method now uses `getrandbits()' where
     appropriate, making generation of arbitrarily large random numbers
     more efficient.  (Contributed by Raymond Hettinger.)

   * The regular expression language accepted by the re.
     module was extended with simple conditional expressions, written
     as `(?(group)A|B)'.  _group_ is either a numeric group ID or a
     group name defined with `(?P<group>...)' earlier in the
     expression.  If the specified group matched, the regular
     expression pattern _A_ will be tested against the string; if the
     group didn't match, the pattern _B_ will be used instead.
     (Contributed by Gustavo Niemeyer.)

   * The re module is also no longer recursive, thanks to a
     massive amount of work by Gustavo Niemeyer.  In a recursive
     regular expression engine, certain patterns result in a large
     amount of C stack space being consumed, and it was possible to
     overflow the stack. For example, if you matched a 30000-byte string
     of `a' characters against the expression `(a|b)+', one stack frame
     was consumed per character.  Python 2.3 tried to check for stack
     overflow and raise a RuntimeError exception, but
     certain patterns could sidestep the checking and if you were
     unlucky Python could segfault. Python 2.4's regular expression
     engine can match this pattern without problems.

   * The signal module now performs tighter error-checking
     on the parameters to the signal.signal() function.
     For example, you can't set a handler on the `SIGKILL' signal;
     previous versions of Python would quietly accept this, but 2.4
     will raise a RuntimeError exception.

   * Two new functions were added to the socket module.
     `socketpair()' returns a pair of connected sockets and
     `getservbyport(port)()' looks up the service name for a given port
     number. (Contributed by Dave Cole and Barry Warsaw.)

   * The sys.exitfunc() function has been deprecated.  Code
     should be using the existing atexit module, which
     correctly handles calling multiple exit functions.  Eventually
     sys.exitfunc() will become a purely internal
     interface, accessed only by atexit.

   * The tarfile module now generates GNU-format tar files
     by default.  (Contributed by Lars Gustaebel.)

   * The threading module now has an elegantly simple way
     to support thread-local data.  The module contains a `local' class
     whose attribute values are local to different threads.

         import threading

         data = threading.local()
         data.number = 42
         data.url = ('www.python.org', 80)

     Other threads can assign and retrieve their own values for the
     `number' and `url' attributes.  You can subclass `local' to
     initialize attributes or to add methods. (Contributed by Jim
     Fulton.)

   * The timeit module now automatically disables periodic
     garbage collection during the timing loop.  This change makes
     consecutive timings more comparable.  (Contributed by Raymond
     Hettinger.)

   * The weakref module now supports a wider variety of
     objects including Python functions, class instances, sets,
     frozensets, deques, arrays, files, sockets, and regular expression
     pattern objects. (Contributed by Raymond Hettinger.)

   * The xmlrpclib module now supports a multi-call
     extension for transmitting multiple XML-RPC calls in a single HTTP
     operation. (Contributed by Brian Quinlan.)

   * The `mpz', `rotor', and `xreadlines' modules have  been removed.

1.4.12.1 cookielib
..................

The cookielib library supports client-side handling for HTTP
cookies, mirroring the Cookie module's server-side cookie
support. Cookies are stored in cookie jars; the library transparently
stores cookies offered by the web server in the cookie jar, and fetches
the cookie from the jar when connecting to the server. As in web
browsers, policy objects control whether cookies are accepted or not.

  In order to store cookies across sessions, two implementations of
cookie jars are provided: one that stores cookies in the Netscape
format so applications can use the Mozilla or Lynx cookie files, and
one that stores cookies in the same format as the Perl libwww library.

  urllib2 has been changed to interact with
cookielib.: `HTTPCookieProcessor' manages a cookie jar that is used
when accessing URLs.

  This module was contributed by John J. Lee.

1.4.12.2 doctest
................

The doctest module underwent considerable refactoring thanks
to Edward Loper and Tim Peters.  Testing can still be as simple as
running doctest.testmod(), but the refactorings allow
customizing the module's operation in various ways

  The new `DocTestFinder' class extracts the tests from a given
object's docstrings:

    def f (x, y):
        """>>> f(2,2)
    4
    >>> f(3,2)
    6
        """
        return x*y

    finder = doctest.DocTestFinder()

    # Get list of DocTest instances
    tests = finder.find(f)

The new `DocTestRunner' class then runs individual tests and can produce
a summary of the results:

    runner = doctest.DocTestRunner()
    for t in tests:
        tried, failed = runner.run(t)

    runner.summarize(verbose=1)

The above example produces the following output:

    1 items passed all tests:
       2 tests in f
    2 tests in 1 items.
    2 passed and 0 failed.
    Test passed.

`DocTestRunner' uses an instance of the `OutputChecker' class to
compare the expected output with the actual output.  This class takes a
number of different flags that customize its behaviour; ambitious users
can also write a completely new subclass of `OutputChecker'.

  The default output checker provides a number of handy features. For
example, with the doctest.ELLIPSIS option flag, an ellipsis
(`...') in the expected output matches any substring,  making it easier
to accommodate outputs that vary in minor ways:

    def o (n):
        """>>> o(1)
    <__main__.C instance at 0x...>
    >>>
    """

Another special string, `<BLANKLINE>', matches a blank line:

    def p (n):
        """>>> p(1)
    <BLANKLINE>
    >>>
    """

Another new capability is producing a diff-style display of the output
by specifying the doctest.REPORT_UDIFF. (unified diffs),
see doctest.REPORT_CDIFF. (context diffs), or
doctest.REPORT_NDIFF.  (delta-style) option flags.  For example:

    def g (n):
        """>>> g(4)
    here
    is
    a
    lengthy
    >>>"""
        L = 'here is a rather lengthy list of words'.split()
        for word in L[:n]:
            print word

Running the above function's tests with doctest.REPORT_UDIFF
specified, you get the following output:

    **********************************************************************
    File "t.py", line 15, in g
    Failed example:
        g(4)
    Differences (unified diff with -expected +actual):
        @@ -2,3 +2,3 @@
         is
         a
        -lengthy
        +rather
    **********************************************************************

1.4.13 Build and C API Changes
------------------------------

Some of the changes to Python's build process and to the C API are:

   * Three new convenience macros were added for common return values
     from extension functions: Py_RETURN_NONE,
     Py_RETURN_TRUE, and Py_RETURN_FALSE. (Contributed
     by Brett Cannon.)

   * Another new macro, `Py_CLEAR(obj)',  decreases the reference count
     of _obj_ and sets _obj_ to the null pointer.  (Contributed by Jim
     Fulton.)

   * A new function, `PyTuple_Pack(N, obj1, obj2, ..., objN)()',
     constructs tuples from a variable length argument list of Python
     objects.  (Contributed by Raymond Hettinger.)

   * A new function, `PyDict_Contains(d, k)()', implements fast
     dictionary lookups without masking exceptions raised during the
     look-up process.  (Contributed by Raymond Hettinger.)

   * The `Py_IS_NAN(X)' macro returns 1 if  its float or double argument
     _X_ is a NaN.   (Contributed by Tim Peters.)

   * C code can avoid unnecessary locking by using the new
     PyEval_ThreadsInitialized() function to tell  if any thread
     operations have been performed.  If this function  returns false,
     no lock operations are needed. (Contributed by Nick Coghlan.)

   * A new function, see PyArg_VaParseTupleAndKeywords(), is the
     same as PyArg_ParseTupleAndKeywords() but takes a
     `va_list' instead of a number of arguments. (Contributed by Greg
     Chapman.)

   * A new method flag, `METH_COEXISTS', allows a function defined in
     slots to co-exist with a PyCFunction having the same
     name.  This can halve the access time for a method such as
     `set.__contains__()'.  (Contributed by Raymond Hettinger.)

   * Python can now be built with additional profiling for the
     interpreter itself, intended as an aid to people developing the
     Python core.  Providing `----enable-profiling' to the *configure*
     script will let you profile the interpreter with *gprof*, and
     providing the `----with-tsc' switch enables profiling using the
     Pentium's Time-Stamp- Counter register.  Note that the
     `----with-tsc' switch is slightly misnamed, because the profiling
     feature also works on the PowerPC platform, though that processor
     architecture doesn't call that register "the TSC register".
     (Contributed by Jeremy Hylton.)

   * The `tracebackobject' type has been renamed to `PyTracebackObject'.

1.4.13.1 Port-Specific Changes
..............................

   * The Windows port now builds under MSVC++ 7.1 as well as version 6.
     (Contributed by Martin von Löwis.)

1.4.14 Porting to Python 2.4
----------------------------

This section lists previously described changes that may require
changes to your code:

   * Left shifts and hexadecimal/octal constants that are too  large no
     longer trigger a FutureWarning and return  a value
     limited to 32 or 64 bits; instead they return a long integer.

   * Integer operations will no longer trigger an `OverflowWarning'. The
     `OverflowWarning' warning will disappear in Python 2.5.

   * The zip() built-in function and
     itertools.izip() now return  an empty list instead of
     raising a TypeError exception if called with no
     arguments.

   * You can no longer compare the `date' and datetime.
     instances provided by the datetime module.  Two
     instances of different classes will now always be unequal, and
     relative comparisons (`<', `>') will raise a TypeError.

   * dircache.listdir() now passes exceptions to the caller
     instead of returning empty lists.

   * `LexicalHandler.startDTD()' used to receive the public and system
     IDs in the wrong order.  This has been corrected; applications
     relying on the wrong order need to be fixed.

   * fcntl.ioctl() now warns if the _mutate_  argument is
     omitted and relevant.

   * The tarfile module now generates GNU-format tar files
     by default.

   * Encountering a failure while importing a module no longer leaves a
     partially- initialized module object in `sys.modules'.

   * None is now a constant; code that binds a new value to
     the name `None' is now a syntax error.

   * The `signals.signal()' function now raises a RuntimeError
     exception for certain illegal values; previously these errors
     would pass silently.  For example, you can no longer set a handler
     on the `SIGKILL' signal.

1.4.15 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Koray Can, Hye- Shik Chang, Michael Dyck, Raymond Hettinger,
Brian Hurt, Hamish Lawson, Fredrik Lundh, Sean Reifschneider, Sadruddin
Rejeb.

1.5 What's New in Python 2.3
============================

     Author: A.M. Kuchling

  This article explains the new features in Python 2.3.  Python 2.3 was
released on July 29, 2003.

  The main themes for Python 2.3 are polishing some of the features
added in 2.2, adding various small but useful enhancements to the core
language, and expanding the standard library.  The new object model
introduced in the previous version has benefited from 18 months of
bugfixes and from optimization efforts that have improved the
performance of new-style classes.  A few new built-in functions have
been added such as sum() and enumerate().  The
in.  operator can now be used for substring searches (e.g.
`"ab" in "abc"' returns True).

  Some of the many new library features include Boolean, set, heap, and
date/time data types, the ability to import modules from ZIP-format
archives, metadata support for the long-awaited Python catalog, an
updated version of IDLE, and modules for logging messages, wrapping
text, parsing CSV files, processing command-line options, using
BerkeleyDB databases...  the list of new and enhanced modules is
lengthy.

  This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For full
details, you should refer to the documentation for Python 2.3, such as
the Python Library Reference and the Python Reference Manual.  If you
want to understand the complete implementation and design rationale,
refer to the PEP for a particular new feature.

1.5.1 PEP 218: A Standard Set Datatype
--------------------------------------

The new sets module contains an implementation of a set
datatype.  The `Set' class is for mutable sets, sets that can have
members added and removed.  The `ImmutableSet' class is for sets that
can't be modified, and instances of `ImmutableSet' can therefore be
used as dictionary keys.  Sets are built on top of dictionaries, so the
elements within a set must be hashable.

  Here's a simple example:

    >>> import sets
    >>> S = sets.Set([1,2,3])
    >>> S
    Set([1, 2, 3])
    >>> 1 in S
    True
    >>> 0 in S
    False
    >>> S.add(5)
    >>> S.remove(3)
    >>> S
    Set([1, 2, 5])
    >>>

The union and intersection of sets can be computed with the `union()'
and `intersection()' methods; an alternative notation uses the bitwise
operators `&' and `|'. Mutable sets also have in-place versions of
these methods, `union_update()' and `intersection_update()'.

    >>> S1 = sets.Set([1,2,3])
    >>> S2 = sets.Set([4,5,6])
    >>> S1.union(S2)
    Set([1, 2, 3, 4, 5, 6])
    >>> S1 | S2                  # Alternative notation
    Set([1, 2, 3, 4, 5, 6])
    >>> S1.intersection(S2)
    Set([])
    >>> S1 & S2                  # Alternative notation
    Set([])
    >>> S1.union_update(S2)
    >>> S1
    Set([1, 2, 3, 4, 5, 6])
    >>>

It's also possible to take the symmetric difference of two sets.  This
is the set of all elements in the union that aren't in the
intersection.  Another way of putting it is that the symmetric
difference contains all elements that are in exactly one set.  Again,
there's an alternative notation (`^'), and an in- place version with
the ungainly name `symmetric_difference_update()'.

    >>> S1 = sets.Set([1,2,3,4])
    >>> S2 = sets.Set([3,4,5,6])
    >>> S1.symmetric_difference(S2)
    Set([1, 2, 5, 6])
    >>> S1 ^ S2
    Set([1, 2, 5, 6])
    >>>

There are also `issubset()' and `issuperset()' methods for checking
whether one set is a subset or superset of another:

    >>> S1 = sets.Set([1,2,3])
    >>> S2 = sets.Set([2,3])
    >>> S2.issubset(S1)
    True
    >>> S1.issubset(S2)
    False
    >>> S1.issuperset(S2)
    True
    >>>

PEP 218 (http://www.python.org/dev/peps/pep-0218) - Adding a Built-In Set Object Type
     PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex
     Martelli, and GvR.

1.5.2 PEP 255: Simple Generators
--------------------------------

In Python 2.2, generators were added as an optional feature, to be
enabled by a `from __future__ import generators' directive.  In 2.3
generators no longer need to be specially enabled, and are now always
present; this means that yield is now always a keyword.
The rest of this section is a copy of the description of generators
from the "What's New in Python 2.2" document; if you read it back when
Python 2.2 came out, you can skip the rest of this section.

  You're doubtless familiar with how function calls work in Python or
C. When you call a function, it gets a private namespace where its
local variables are created.  When the function reaches a return
statement, the local variables are destroyed and the resulting
value is returned to the caller.  A later call to the same function
will get a fresh new set of local variables.  But, what if the local
variables weren't thrown away on exiting a function?  What if you could
later resume the function where it left off?  This is what generators
provide; they can be thought of as resumable functions.

  Here's the simplest example of a generator function:

    def generate_ints(N):
        for i in range(N):
            yield i

A new keyword, see yield, was introduced for generators.  Any
function containing a yield statement is a generator
function; this is detected by Python's bytecode compiler which compiles
the function specially as a result.

  When you call a generator function, it doesn't return a single value;
instead it returns a generator object that supports the iterator
protocol.  On executing the yield statement, the generator
outputs the value of `i', similar to a return statement.
The big difference between yield and a return.
statement is that on reaching a yield the generator's state
of execution is suspended and local variables are preserved.  On the
next call to the generator's `.next()' method, the function will resume
executing immediately after the yield statement.  (For
complicated reasons, the yield.  statement isn't allowed
inside the try block of a try...finally
statement; read PEP 255 (http://www.python.org/dev/peps/pep-0255)
for a full explanation of the interaction between yield and
exceptions.)

  Here's a sample usage of the `generate_ints()' generator:

    >>> gen = generate_ints(3)
    >>> gen
    <generator object at 0x8117f90>
    >>> gen.next()
    0
    >>> gen.next()
    1
    >>> gen.next()
    2
    >>> gen.next()
    Traceback (most recent call last):
      File "stdin", line 1, in ?
      File "stdin", line 2, in generate_ints
    StopIteration

You could equally write `for i in generate_ints(5)', or `a,b,c =
generate_ints(3)'.

  Inside a generator function, the return statement can
only be used without a value, and signals the end of the procession of
values; afterwards the generator cannot return any further values.
return with a value, such as `return 5', is a syntax error
inside a generator function.  The end of the generator's results can
also be indicated by raising StopIteration.  manually, or by
just letting the flow of execution fall off the bottom of the function.

  You could achieve the effect of generators manually by writing your
own class and storing all the local variables of the generator as
instance variables.  For example, returning a list of integers could be
done by setting `self.count' to 0, and having the next().
method increment `self.count' and return it.  However, for a moderately
complicated generator, writing a corresponding class would be much
messier. `Lib/test/test_generators.py' contains a number of more
interesting examples.  The simplest one implements an in-order
traversal of a tree using generators recursively.

    # A recursive generator that generates Tree leaves in in-order.
    def inorder(t):
        if t:
            for x in inorder(t.left):
                yield x
            yield t.label
            for x in inorder(t.right):
                yield x

Two other examples in `Lib/test/test_generators.py' produce solutions
for the N-Queens problem (placing $N$ queens on an $NxN$ chess board so
that no queen threatens another) and the Knight's Tour (a route that
takes a knight to every square of an $NxN$ chessboard without visiting
any square twice).

  The idea of generators comes from other programming languages,
especially Icon (<http://www.cs.arizona.edu/icon/>), where the idea of
generators is central.  In Icon, every expression and function call
behaves like a generator.  One example from "An Overview of the Icon
Programming Language" at
<http://www.cs.arizona.edu/icon/docs/ipd266.htm> gives an idea of what
this looks like:

    sentence := "Store it in the neighboring harbor"
    if (i := find("or", sentence)) > 5 then write(i)

In Icon the `find()' function returns the indexes at which the substring
"or" is found: 3, 23, 33.  In the if statement, `i' is first
assigned a value of 3, but 3 is less than 5, so the comparison fails,
and Icon retries it with the second value of 23.  23 is greater than 5,
so the comparison now succeeds, and the code prints the value 23 to the
screen.

  Python doesn't go nearly as far as Icon in adopting generators as a
central concept.  Generators are considered part of the core Python
language, but learning or using them isn't compulsory; if they don't
solve any problems that you have, feel free to ignore them. One novel
feature of Python's interface as compared to Icon's is that a
generator's state is represented as a concrete object (the iterator)
that can be passed around to other functions or stored in a data
structure.

PEP 255 (http://www.python.org/dev/peps/pep-0255) - Simple Generators
     Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.
     Implemented mostly by Neil Schemenauer and Tim Peters, with other
     fixes from the Python Labs crew.

1.5.3 PEP 263: Source Code Encodings
------------------------------------

Python source files can now be declared as being in different character
set encodings.  Encodings are declared by including a specially
formatted comment in the first or second line of the source file.  For
example, a UTF-8 file can be declared with:

    #!/usr/bin/env python
    # -*- coding: UTF-8 -*-

Without such an encoding declaration, the default encoding used is
7-bit ASCII.  Executing or importing modules that contain string
literals with 8-bit characters and have no encoding declaration will
result in a DeprecationWarning being signalled by Python
2.3; in 2.4 this will be a syntax error.

  The encoding declaration only affects Unicode string literals, which
will be converted to Unicode using the specified encoding.  Note that
Python identifiers are still restricted to ASCII characters, so you
can't have variable names that use characters outside of the usual
alphanumerics.

PEP 263 (http://www.python.org/dev/peps/pep-0263) - Defining Python Source Code Encodings
     Written by Marc-André Lemburg and Martin von Löwis; implemented by
     Suzuki Hisao and Martin von Löwis.

1.5.4 PEP 273: Importing Modules from ZIP Archives
--------------------------------------------------

The new zipimport module adds support for importing modules
from a ZIP- format archive.  You don't need to import the module
explicitly; it will be automatically imported if a ZIP archive's
filename is added to `sys.path'.  For example:

    amk@nyman:~/src/python$ unzip -l /tmp/example.zip
    Archive:  /tmp/example.zip
      Length     Date   Time    Name
     --------    ----   ----    ----
         8467  11-26-02 22:30   jwzthreading.py
     --------                   -------
         8467                   1 file
    amk@nyman:~/src/python$ ./python
    Python 2.3 (#1, Aug 1 2003, 19:54:32)
    >>> import sys
    >>> sys.path.insert(0, '/tmp/example.zip')  # Add .zip file to front of path
    >>> import jwzthreading
    >>> jwzthreading.__file__
    '/tmp/example.zip/jwzthreading.py'
    >>>

An entry in `sys.path' can now be the filename of a ZIP archive. The ZIP
archive can contain any kind of files, but only files named `*.py',
`*.pyc', or `*.pyo' can be imported.  If an archive only contains
`*.py' files, Python will not attempt to modify the archive by adding
the corresponding `*.pyc' file, meaning that if a ZIP archive doesn't
contain `*.pyc' files, importing may be rather slow.

  A path within the archive can also be specified to only import from a
subdirectory; for example, the path `/tmp/example.zip/lib/' would only
import from the `lib/' subdirectory within the archive.

PEP 273 (http://www.python.org/dev/peps/pep-0273) - Import Modules from Zip Archives
     Written by James C. Ahlstrom,  who also provided an
     implementation. Python 2.3 follows the specification in PEP 273
     (http://www.python.org/dev/peps/pep-0273),  but uses an
     implementation written by Just van Rossum  that uses the import
     hooks described in PEP 302
     (http://www.python.org/dev/peps/pep-0302). See section PEP 302; New Import Hooks
     for a description of the new import
     hooks.

1.5.5 PEP 277: Unicode file name support for Windows NT
-------------------------------------------------------

On Windows NT, 2000, and XP, the system stores file names as Unicode
strings.  Traditionally, Python has represented file names as byte
strings, which is inadequate because it renders some file names
inaccessible.

  Python now allows using arbitrary Unicode strings (within the
limitations of the file system) for all functions that expect file
names, most notably the open() built-in function. If a
Unicode string is passed to os.listdir(), Python now returns
a list of Unicode strings.  A new function, see os.getcwdu(),
returns the current directory as a Unicode string.

  Byte strings still work as file names, and on Windows Python will
transparently convert them to Unicode using the `mbcs' encoding.

  Other systems also allow Unicode strings as file names but convert
them to byte strings before passing them to the system, which can cause
a UnicodeError.  to be raised. Applications can test whether
arbitrary Unicode strings are supported as file names by checking
os.path.supports_unicode_filenames, a Boolean value.

  Under MacOS, see os.listdir() may now return Unicode
filenames.

PEP 277 (http://www.python.org/dev/peps/pep-0277) - Unicode file name support for Windows NT
     Written by Neil Hodgson; implemented by Neil Hodgson, Martin von
     Löwis, and Mark Hammond.

1.5.6 PEP 278: Universal Newline Support
----------------------------------------

The three major operating systems used today are Microsoft Windows,
Apple's Macintosh OS, and the various Unix derivatives.  A minor
irritation of cross- platform work  is that these three platforms all
use different characters to mark the ends of lines in text files.  Unix
uses the linefeed (ASCII character 10), MacOS uses the carriage return
(ASCII character 13), and Windows uses a two-character sequence of a
carriage return plus a newline.

  Python's file objects can now support end of line conventions other
than the one followed by the platform on which Python is running.
Opening a file with the mode `'U'' or `'rU'' will open a file for
reading in universal newline mode.  All three line ending conventions
will be translated to a `'\n'' in the strings returned by the various
file methods such as `read()' and readline().

  Universal newline support is also used when importing modules and
when executing a file with the execfile() function.  This
means that Python modules can be shared between all three operating
systems without needing to convert the line-endings.

  This feature can be disabled when compiling Python by specifying the
`--without-universal-newlines' switch when running Python's *configure*
script.

PEP 278 (http://www.python.org/dev/peps/pep-0278) - Universal Newline Support
     Written and implemented by Jack Jansen.

1.5.7 PEP 279: enumerate()
--------------------------

A new built-in function, see enumerate(), will make certain
loops a bit clearer.  `enumerate(thing)', where _thing_ is either an
iterator or a sequence, returns a iterator that will return `(0,
thing[0])', `(1, thing[1])', `(2, thing[2])', and so forth.

  A common idiom to change every element of a list looks like this:

    for i in range(len(L)):
        item = L[i]
        # ... compute some result based on item ...
        L[i] = result

This can be rewritten using enumerate() as:

    for i, item in enumerate(L):
        # ... compute some result based on item ...
        L[i] = result

PEP 279 (http://www.python.org/dev/peps/pep-0279) - The enumerate() built-in function
     Written and implemented by Raymond D. Hettinger.

1.5.8 PEP 282: The logging Package
----------------------------------

A standard package for writing logs, see logging, has been added
to Python 2.3.  It provides a powerful and flexible mechanism for
generating logging output which can then be filtered and processed in
various ways.  A configuration file written in a standard format can be
used to control the logging behavior of a program.  Python includes
handlers that will write log records to standard error or to a file or
socket, send them to the system log, or even e-mail them to a
particular address; of course, it's also possible to write your own
handler classes.

  The `Logger' class is the primary class. Most application code will
deal with one or more `Logger' objects, each one used by a particular
subsystem of the application. Each `Logger' is identified by a name, and
names are organized into a hierarchy using `.'  as the component
separator.  For example, you might have `Logger' instances named
`server', `server.auth' and `server.network'.  The latter two instances
are below `server' in the hierarchy.  This means that if you turn up
the verbosity for `server' or direct `server' messages to a different
handler, the changes will also apply to records logged to `server.auth'
and `server.network'.  There's also a root `Logger' that's the parent
of all other loggers.

  For simple uses, the logging package contains some
convenience functions that always use the root log:

    import logging

    logging.debug('Debugging information')
    logging.info('Informational message')
    logging.warning('Warning:config file %s not found', 'server.conf')
    logging.error('Error occurred')
    logging.critical('Critical error -- shutting down')

This produces the following output:

    WARNING:root:Warning:config file server.conf not found
    ERROR:root:Error occurred
    CRITICAL:root:Critical error -- shutting down

In the default configuration, informational and debugging messages are
suppressed and the output is sent to standard error.  You can enable
the display of informational and debugging messages by calling the
`setLevel()' method on the root logger.

  Notice the `warning()' call's use of string formatting operators; all
of the functions for logging messages take the arguments `(msg, arg1,
arg2, ...)' and log the string resulting from `msg % (arg1, arg2, ...)'.

  There's also an `exception()' function that records the most recent
traceback.  Any of the other functions will also record the traceback
if you specify a true value for the keyword argument _exc_info_.

    def f():
        try:    1/0
        except: logging.exception('Problem recorded')

    f()

This produces the following output:

    ERROR:root:Problem recorded
    Traceback (most recent call last):
      File "t.py", line 6, in f
        1/0
    ZeroDivisionError: integer division or modulo by zero

Slightly more advanced programs will use a logger other than the root
logger.  The `getLogger(name)()' function is used to get a particular
log, creating it if it doesn't exist yet. `getLogger(None)()' returns
the root logger.

    log = logging.getLogger('server')
     ...
    log.info('Listening on port %i', port)
     ...
    log.critical('Disk full')
     ...

Log records are usually propagated up the hierarchy, so a message
logged to `server.auth' is also seen by `server' and `root', but a
`Logger' can prevent this by setting its `propagate' attribute to
False.

  There are more classes provided by the logging package
that can be customized.  When a `Logger' instance is told to log a
message, it creates a `LogRecord' instance that is sent to any number
of different `Handler' instances.  Loggers and handlers can also have
an attached list of filters, and each filter can cause the `LogRecord'
to be ignored or can modify the record before passing it along.  When
they're finally output, `LogRecord' instances are converted to text by
a `Formatter' class.  All of these classes can be replaced by your own
specially-written classes.

  With all of these features the logging package should
provide enough flexibility for even the most complicated applications.
This is only an incomplete overview of its features, so please see the
package's reference documentation for all of the details.  Reading PEP
282 (http://www.python.org/dev/peps/pep-0282) will also be helpful.

PEP 282 (http://www.python.org/dev/peps/pep-0282) - A Logging System
     Written by Vinay Sajip and Trent Mick; implemented by Vinay Sajip.

1.5.9 PEP 285: A Boolean Type
-----------------------------

A Boolean type was added to Python 2.3.  Two new constants were added
to the __builtin__ module, see True and False.
(see True and False constants were added to
the built-ins in Python 2.2.1, but the 2.2.1 versions are simply set to
integer values of 1 and 0 and aren't a different type.)

  The type object for this new type is named bool.; the
constructor for it takes any Python value and converts it to
True or False.

    >>> bool(1)
    True
    >>> bool(0)
    False
    >>> bool([])
    False
    >>> bool( (1,) )
    True

Most of the standard library modules and built-in functions have been
changed to return Booleans.

    >>> obj = []
    >>> hasattr(obj, 'append')
    True
    >>> isinstance(obj, list)
    True
    >>> isinstance(obj, tuple)
    False

Python's Booleans were added with the primary goal of making code
clearer.  For example, if you're reading a function and encounter the
statement `return 1', you might wonder whether the `1' represents a
Boolean truth value, an index, or a coefficient that multiplies some
other quantity.  If the statement is `return True', however, the
meaning of the return value is quite clear.

  Python's Booleans were _not_ added for the sake of strict
type-checking.  A very strict language such as Pascal would also
prevent you performing arithmetic with Booleans, and would require that
the expression in an if statement always evaluate to a
Boolean result.  Python is not this strict and never will be, as PEP
285 (http://www.python.org/dev/peps/pep-0285) explicitly says.  This
means you can still use any expression in an if statement,
even ones that evaluate to a list or tuple or some random object.  The
Boolean type is a subclass of the int class so that
arithmetic using a Boolean still works.

    >>> True + 1
    2
    >>> False + 1
    1
    >>> False * 75
    0
    >>> True * 75
    75

To sum up True and False in a sentence: they're
alternative ways to spell the integer values 1 and 0, with the single
difference that str() and repr() return the
strings `'True'' and `'False'' instead of `'1'' and `'0''.

PEP 285 (http://www.python.org/dev/peps/pep-0285) - Adding a bool type
     Written and implemented by GvR.

1.5.10 PEP 293: Codec Error Handling Callbacks
----------------------------------------------

When encoding a Unicode string into a byte string, unencodable
characters may be encountered.  So far, Python has allowed specifying
the error processing as either "strict" (raising UnicodeError),
"ignore" (skipping the character), or "replace" (using a
question mark in the output string), with "strict" being the default
behavior. It may be desirable to specify alternative processing of such
errors, such as inserting an XML character reference or HTML entity
reference into the converted string.

  Python now has a flexible framework to add different processing
strategies.  New error handlers can be added with
codecs.register_error(), and codecs then can access the error
handler with codecs.lookup_error(). An equivalent C API has
been added for codecs written in C. The error handler gets the necessary
state information such as the string being converted, the position in
the string where the error was detected, and the target encoding.  The
handler can then either raise an exception or return a replacement
string.

  Two additional error handlers have been implemented using this
framework: "backslashreplace" uses Python backslash quoting to
represent unencodable characters and "xmlcharrefreplace" emits XML
character references.

PEP 293 (http://www.python.org/dev/peps/pep-0293) - Codec Error Handling Callbacks
     Written and implemented by Walter Dörwald.

1.5.11 PEP 301: Package Index and Metadata for Distutils
--------------------------------------------------------

Support for the long-requested Python catalog makes its first
appearance in 2.3.

  The heart of the catalog is the new Distutils *register* command.
Running `python setup.py register' will collect the metadata describing
a package, such as its name, version, maintainer, description, &c., and
send it to a central catalog server.  The resulting catalog is
available from <http://www.python.org/pypi>.

  To make the catalog a bit more useful, a new optional _classifiers_
keyword argument has been added to the Distutils `setup()' function.  A
list of Trove (http://catb.org/~esr/trove/)-style strings can be
supplied to help classify the software.

  Here's an example `setup.py' with classifiers, written to be
compatible with older versions of the Distutils:

    from distutils import core
    kw = {'name': "Quixote",
          'version': "0.5.1",
          'description': "A highly Pythonic Web application framework",
          # ...
          }

    if (hasattr(core, 'setup_keywords') and
        'classifiers' in core.setup_keywords):
        kw['classifiers'] = \
            ['Topic :: Internet :: WWW/HTTP :: Dynamic Content',
             'Environment :: No Input/Output (Daemon)',
             'Intended Audience :: Developers'],

    core.setup(**kw)

The full list of classifiers can be obtained by running  `python
setup.py register --list-classifiers'.

PEP 301 (http://www.python.org/dev/peps/pep-0301) - Package Index and Metadata for Distutils
     Written and implemented by Richard Jones.

1.5.12 PEP 302: New Import Hooks
--------------------------------

While it's been possible to write custom import hooks ever since the
`ihooks' module was introduced in Python 1.3, no one has ever been
really happy with it because writing new import hooks is difficult and
messy.  There have been various proposed alternatives such as the
imputil and `iu' modules, but none of them has ever gained much
acceptance, and none of them were easily usable from C code.

  PEP 302 (http://www.python.org/dev/peps/pep-0302) borrows ideas from
its predecessors, especially from Gordon McMillan's `iu' module.  Three
new items  are added to the sys.  module:

   * `sys.path_hooks' is a list of callable objects; most  often
     they'll be classes.  Each callable takes a string containing a
     path and either returns an importer object that will handle
     imports from this path or raises an ImportError.
     exception if it can't handle this path.

   * `sys.path_importer_cache' caches importer objects for each path, so
     `sys.path_hooks' will only need to be traversed once for each path.

   * `sys.meta_path' is a list of importer objects that will be
     traversed before `sys.path' is checked.  This list is initially
     empty, but user code can add objects to it.  Additional built-in
     and frozen modules can be imported by an object added to this list.

  Importer objects must have a single method, `find_module(fullname,
path=None)()'.  _fullname_ will be a module or package name, e.g.
`string' or `distutils.core'.  `find_module()' must return a loader
object that has a single method, `load_module(fullname)()', that
creates and returns the corresponding module object.

  Pseudo-code for Python's new import logic, therefore, looks something
like this (simplified a bit; see PEP 302
(http://www.python.org/dev/peps/pep-0302) for the full details):

    for mp in sys.meta_path:
        loader = mp(fullname)
        if loader is not None:
            <module> = loader.load_module(fullname)

    for path in sys.path:
        for hook in sys.path_hooks:
            try:
                importer = hook(path)
            except ImportError:
                # ImportError, so try the other path hooks
                pass
            else:
                loader = importer.find_module(fullname)
                <module> = loader.load_module(fullname)

    # Not found!
    raise ImportError

PEP 302 (http://www.python.org/dev/peps/pep-0302) - New Import Hooks
     Written by Just van Rossum and Paul Moore. Implemented by Just van
     Rossum.

1.5.13 PEP 305: Comma-separated Files
-------------------------------------

Comma-separated files are a format frequently used for exporting data
from databases and spreadsheets.  Python 2.3 adds a parser for
comma-separated files.

  Comma-separated format is deceptively simple at first glance:

    Costs,150,200,3.95

Read a line and call `line.split(',')': what could be simpler? But toss
in string data that can contain commas, and things get more complicated:

    "Costs",150,200,3.95,"Includes taxes, shipping, and sundry items"

A big ugly regular expression can parse this, but using the new
csv.  package is much simpler:

    import csv

    input = open('datafile', 'rb')
    reader = csv.reader(input)
    for line in reader:
        print line

The `reader()' function takes a number of different options. The field
separator isn't limited to the comma and can be changed to any
character, and so can the quoting and line-ending characters.

  Different dialects of comma-separated files can be defined and
registered; currently there are two dialects, both used by Microsoft
Excel. A separate csv.writer class will generate
comma-separated files from a succession of tuples or lists, quoting
strings that contain the delimiter.

PEP 305 (http://www.python.org/dev/peps/pep-0305) - CSV File API
     Written and implemented  by Kevin Altis, Dave Cole, Andrew
     McNamara, Skip Montanaro, Cliff Wells.

1.5.14 PEP 307: Pickle Enhancements
-----------------------------------

The pickle and cPickle modules received some
attention during the 2.3 development cycle.  In 2.2, new-style classes
could be pickled without difficulty, but they weren't pickled very
compactly; PEP 307 (http://www.python.org/dev/peps/pep-0307) quotes a
trivial example where a new-style class results in a pickled string
three times longer than that for a classic class.

  The solution was to invent a new pickle protocol.  The
pickle.dumps().  function has supported a text-or-binary flag  for
a long time.  In 2.3, this flag is redefined from a Boolean to an
integer: 0 is the old text-mode pickle format, 1 is the old binary
format, and now 2 is a new 2.3-specific format.  A new constant, see
pickle.HIGHEST_PROTOCOL, can be used to select the fanciest
protocol available.

  Unpickling is no longer considered a safe operation.  2.2's
pickle.  provided hooks for trying to prevent unsafe classes from
being unpickled (specifically, a `__safe_for_unpickling__' attribute),
but none of this code was ever audited and therefore it's all been
ripped out in 2.3.  You should not unpickle untrusted data in any
version of Python.

  To reduce the pickling overhead for new-style classes, a new
interface for customizing pickling was added using three special
methods: __getstate__(), __setstate__(), and
__getnewargs__().  Consult PEP 307
(http://www.python.org/dev/peps/pep-0307) for the full semantics  of
these methods.

  As a way to compress pickles yet further, it's now possible to use
integer codes instead of long strings to identify pickled classes. The
Python Software Foundation will maintain a list of standardized codes;
there's also a range of codes for private use.  Currently no codes have
been specified.

PEP 307 (http://www.python.org/dev/peps/pep-0307) - Extensions to the pickle protocol
     Written and implemented  by Guido van Rossum and Tim Peters.

1.5.15 Extended Slices
----------------------

Ever since Python 1.4, the slicing syntax has supported an optional
third "step" or "stride" argument.  For example, these are all legal
Python syntax: `L[1:10:2]', `L[:-1:1]', `L[::-1]'.  This was added to
Python at the request of the developers of Numerical Python, which uses
the third argument extensively.  However, Python's built-in list,
tuple, and string sequence types have never supported this feature,
raising a TypeError if you tried it.  Michael Hudson
contributed a patch to fix this shortcoming.

  For example, you can now easily extract the elements of a list that
have even indexes:

    >>> L = range(10)
    >>> L[::2]
    [0, 2, 4, 6, 8]

Negative values also work to make a copy of the same list in reverse
order:

    >>> L[::-1]
    [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

This also works for tuples, arrays, and strings:

    >>> s='abcd'
    >>> s[::2]
    'ac'
    >>> s[::-1]
    'dcba'

If you have a mutable sequence such as a list or an array you can
assign to or delete an extended slice, but there are some differences
between assignment to extended and regular slices.  Assignment to a
regular slice can be used to change the length of the sequence:

    >>> a = range(3)
    >>> a
    [0, 1, 2]
    >>> a[1:3] = [4, 5, 6]
    >>> a
    [0, 4, 5, 6]

Extended slices aren't this flexible.  When assigning to an extended
slice, the list on the right hand side of the statement must contain
the same number of items as the slice it is replacing:

    >>> a = range(4)
    >>> a
    [0, 1, 2, 3]
    >>> a[::2]
    [0, 2]
    >>> a[::2] = [0, -1]
    >>> a
    [0, 1, -1, 3]
    >>> a[::2] = [0,1,2]
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: attempt to assign sequence of size 3 to extended slice of size 2

Deletion is more straightforward:

    >>> a = range(4)
    >>> a
    [0, 1, 2, 3]
    >>> a[::2]
    [0, 2]
    >>> del a[::2]
    >>> a
    [1, 3]

One can also now pass slice objects to the __getitem__().
methods of the built-in sequences:

    >>> range(10).__getitem__(slice(0, 5, 2))
    [0, 2, 4]

Or use slice objects directly in subscripts:

    >>> range(10)[slice(0, 5, 2)]
    [0, 2, 4]

To simplify implementing sequences that support extended slicing, slice
objects now have a method `indices(length)()' which, given the length
of a sequence, returns a `(start, stop, step)' tuple that can be passed
directly to range(). `indices()' handles omitted and
out-of-bounds indices in a manner consistent with regular slices (and
this innocuous phrase hides a welter of confusing details!).  The
method is intended to be used like this:

    class FakeSeq:
        ...
        def calc_item(self, i):
            ...
        def __getitem__(self, item):
            if isinstance(item, slice):
                indices = item.indices(len(self))
                return FakeSeq([self.calc_item(i) for i in range(*indices)])
            else:
                return self.calc_item(i)

From this example you can also see that the built-in slice.
object is now the type object for the slice type, and is no longer a
function.  This is consistent with Python 2.2, where int,
str, etc., underwent the same change.

1.5.16 Other Language Changes
-----------------------------

Here are all of the changes that Python 2.3 makes to the core Python
language.

   * The yield statement is now always a keyword, as
     described in section PEP 255; Simple Generators of
     this document.

   * A new built-in function enumerate() was added, as
     described in section PEP 279; enumerate() of this
     document.

   * Two new constants, see True and False were
     added along with the built-in bool type, as described
     in section PEP 285; A Boolean Type of this document.

   * The int() type constructor will now return a long
     integer instead of raising an OverflowError when a
     string or floating-point number is too large to fit into an
     integer.  This can lead to the paradoxical result that
     `isinstance(int(expression), int)' is false, but that seems
     unlikely to cause problems in practice.

   * Built-in types now support the extended slicing syntax, as
     described in section Extended Slices of this document.

   * A new built-in function, `sum(iterable, start=0)()',  adds up the
     numeric items in the iterable object and returns their sum.
     sum() only accepts numbers, meaning that you can't use it to
     concatenate a bunch of strings.  (Contributed by Alex Martelli.)

   * `list.insert(pos, value)' used to  insert _value_ at the front of
     the list when _pos_ was negative.  The behaviour has now been
     changed to be consistent with slice indexing, so when _pos_ is -1
     the value will be inserted before the last element, and so forth.

   * `list.index(value)', which searches for _value_  within the list
     and returns its index, now takes optional  _start_ and _stop_
     arguments to limit the search to  only part of the list.

   * Dictionaries have a new method, `pop(key[, *default*])()', that
     returns the value corresponding to _key_ and removes that
     key/value pair from the dictionary.  If the requested key isn't
     present in the dictionary, _default_ is returned if it's specified
     and KeyError raised if it isn't.

         >>> d = {1:2}
         >>> d
         {1: 2}
         >>> d.pop(4)
         Traceback (most recent call last):
           File "stdin", line 1, in ?
         KeyError: 4
         >>> d.pop(1)
         2
         >>> d.pop(1)
         Traceback (most recent call last):
           File "stdin", line 1, in ?
         KeyError: 'pop(): dictionary is empty'
         >>> d
         {}
         >>>

     There's also a new class method,  `dict.fromkeys(iterable,
     value)()', that creates a dictionary with keys taken from the
     supplied iterator _iterable_ and all values set to _value_,
     defaulting to `None'.

     (Patches contributed by Raymond Hettinger.)

     Also, the dict() constructor now accepts keyword
     arguments to simplify creating small dictionaries:

         >>> dict(red=1, blue=2, green=3, black=4)
         {'blue': 2, 'black': 4, 'green': 3, 'red': 1}

     (Contributed by Just van Rossum.)

   * The assert statement no longer checks the `__debug__'
     flag, so you can no longer disable assertions by assigning to
     `__debug__'. Running Python with the -O switch will
     still generate code that doesn't execute any assertions.

   * Most type objects are now callable, so you can use them to create
     new objects such as functions, classes, and modules.  (This means
     that the new module can be deprecated in a future
     Python version, because you can now use the type objects available
     in the types module.) For example, you can create a new
     module object with the following code:

         >>> import types
         >>> m = types.ModuleType('abc','docstring')
         >>> m
         <module 'abc' (built-in)>
         >>> m.__doc__
         'docstring'

   * A new warning, see PendingDeprecationWarning was added to
     indicate features which are in the process of being deprecated.
     The warning will _not_ be printed by default.  To check for use of
     features that will be deprecated in the future, supply
     `-Walways::PendingDeprecationWarning::' on the command line or use
     warnings.filterwarnings().

   * The process of deprecating string-based exceptions, as in `raise
     "Error occurred"', has begun.  Raising a string will now trigger
     PendingDeprecationWarning.

   * Using `None' as a variable name will now result in a
     SyntaxWarning.  warning.  In a future version of Python,
     `None' may finally become a keyword.

   * The `xreadlines()' method of file objects, introduced in Python
     2.1, is no longer necessary because files now behave as their own
     iterator.  `xreadlines()' was originally introduced as a faster
     way to loop over all the lines in a file, but now you can simply
     write `for line in file_obj'.  File objects also have a new
     read-only `encoding' attribute that gives the encoding used by the
     file; Unicode strings written to the file will be automatically
     converted to bytes using the given encoding.

   * The method resolution order used by new-style classes has changed,
     though you'll only notice the difference if you have a really
     complicated inheritance hierarchy.  Classic classes are unaffected
     by this change.  Python 2.2 originally used a topological sort of
     a class's ancestors, but 2.3 now uses the C3 algorithm as
     described in the paper "A Monotonic Superclass Linearization for
     Dylan"
     (http://www.webcom.com/haahr/dylan/linearization-oopsla96.html). To
     understand the motivation for this change,  read Michele
     Simionato's article "Python 2.3 Method Resolution Order"
     (http://www.python.org/2.3/mro.html), or read the thread on
     python-dev starting with the message at
     <http://mail.python.org/pipermail/python-dev/2002-October/029035.html>.
     Samuele Pedroni first pointed out the problem and also implemented
     the fix by coding the C3 algorithm.

   * Python runs multithreaded programs by switching between threads
     after executing N bytecodes.  The default value for N has been
     increased from 10 to 100 bytecodes, speeding up single-threaded
     applications by reducing the switching overhead.  Some
     multithreaded applications may suffer slower response time, but
     that's easily fixed by setting the limit back to a lower number
     using `sys.setcheckinterval(N)()'. The limit can be retrieved with
     the new sys.getcheckinterval() function.

   * One minor but far-reaching change is that the names of extension
     types defined by the modules included with Python now contain the
     module and a `'.'' in front of the type name.  For example, in
     Python 2.2, if you created a socket and printed its `__class__',
     you'd get this output:

         >>> s = socket.socket()
         >>> s.__class__
         <type 'socket'>

     In 2.3, you get this:

         >>> s.__class__
         <type '_socket.socket'>

   * One of the noted incompatibilities between old- and new-style
     classes has been removed: you can now assign to the `__name__' and
     `__bases__' attributes of new-style classes.  There are some
     restrictions on what can be assigned to `__bases__' along the
     lines of those relating to assigning to an instance's `__class__'
     attribute.

1.5.16.1 String Changes
.......................

   * The in operator now works differently for strings.
     Previously, when evaluating `X in Y' where _X_ and _Y_ are
     strings, _X_ could only be a single character. That's now changed;
     _X_ can be a string of any length, and `X in Y' will return
     True if _X_ is a substring of _Y_.  If _X_ is the empty
     string, the result is always True.

         >>> 'ab' in 'abcd'
         True
         >>> 'ad' in 'abcd'
         False
         >>> '' in 'abcd'
         True

     Note that this doesn't tell you where the substring starts; if you
     need that information, use the `find()' string method.

   * The `strip()', `lstrip()', and `rstrip()' string methods now have
     an optional argument for specifying the characters to strip.  The
     default is still to remove all whitespace characters:

         >>> '   abc '.strip()
         'abc'
         >>> '><><abc<><><>'.strip('<>')
         'abc'
         >>> '><><abc<><><>\n'.strip('<>')
         'abc<><><>\n'
         >>> u'\u4000\u4001abc\u4000'.strip(u'\u4000')
         u'\u4001abc'
         >>>

     (Suggested by Simon Brunning and implemented by Walter Dörwald.)

   * The `startswith()' and `endswith()' string methods now accept
     negative numbers for the _start_ and _end_ parameters.

   * Another new string method is `zfill()', originally a function in
     the string module.  `zfill()' pads a numeric string
     with zeros on the left until it's the specified width. Note that
     the `%' operator is still more flexible and powerful than
     `zfill()'.

         >>> '45'.zfill(4)
         '0045'
         >>> '12345'.zfill(4)
         '12345'
         >>> 'goofy'.zfill(6)
         '0goofy'

     (Contributed by Walter Dörwald.)

   * A new type object, see basestring, has been added. Both
     8-bit strings and Unicode strings inherit from this type, so
     `isinstance(obj, basestring)' will return True for
     either kind of string.  It's a completely abstract type, so you
     can't create basestring instances.

   * Interned strings are no longer immortal and will now be
     garbage-collected in the usual way when the only reference to them
     is from the internal dictionary of interned strings.  (Implemented
     by Oren Tirosh.)

1.5.16.2 Optimizations
......................

   * The creation of new-style class instances has been made much
     faster; they're now faster than classic classes!

   * The `sort()' method of list objects has been extensively rewritten
     by Tim Peters, and the implementation is significantly faster.

   * Multiplication of large long integers is now much faster thanks to
     an implementation of Karatsuba multiplication, an algorithm that
     scales better than the O(n*n) required for the grade-school
     multiplication algorithm.  (Original patch by Christopher A.
     Craig, and significantly reworked by Tim Peters.)

   * The `SET_LINENO' opcode is now gone.  This may provide a small
     speed increase, depending on your compiler's idiosyncrasies. See
     section Other Changes and Fixes for a longer
     explanation. (Removed by Michael Hudson.)

   * xrange() objects now have their own iterator, making
     `for i in xrange(n)' slightly faster than `for i in range(n)'.
     (Patch by Raymond Hettinger.)

   * A number of small rearrangements have been made in various
     hotspots to improve performance, such as inlining a function or
     removing some code.  (Implemented mostly by GvR, but lots of
     people have contributed single changes.)

  The net result of the 2.3 optimizations is that Python 2.3 runs the
pystone benchmark around 25% faster than Python 2.2.

1.5.17 New, Improved, and Deprecated Modules
--------------------------------------------

As usual, Python's standard library received a number of enhancements
and bug fixes.  Here's a partial list of the most notable changes,
sorted alphabetically by module name. Consult the `Misc/NEWS' file in
the source tree for a more complete list of changes, or look through
the CVS logs for all the details.

   * The array module now supports arrays of Unicode
     characters using the `'u'' format character.  Arrays also now
     support using the `+=' assignment operator to add another array's
     contents, and the `*=' assignment operator to repeat an array.
     (Contributed by Jason Orendorff.)

   * The bsddb module has been replaced by version 4.1.6 of
     the PyBSDDB (http://pybsddb.sourceforge.net)package, providing a
     more complete interface to the transactional features of the
     BerkeleyDB library.

     The old version of the module has been renamed to  `bsddb185' and
     is no longer built automatically; you'll  have to edit
     `Modules/Setup' to enable it.  Note that the new bsddb.
     package is intended to be compatible with the  old module, so be
     sure to file bugs if you discover any incompatibilities.  When
     upgrading to Python 2.3, if the new interpreter is compiled with a
     new version of  the underlying BerkeleyDB library, you will almost
     certainly have to convert your database files to the new version.
     You can do this fairly easily with the new scripts `db2pickle.py'
     and `pickle2db.py' which you will find in the distribution's
     `Tools/scripts' directory.  If you've already been using the
     PyBSDDB package and importing it as `bsddb3', you will have to
     change your `import' statements to import it as bsddb.

   * The new bz2 module is an interface to the bz2 data
     compression library.  bz2-compressed data is usually smaller than
     corresponding zlib.-compressed data. (Contributed by
     Gustavo Niemeyer.)

   * A set of standard date/time types has been added in the new
     datetime.  module.  See the following section for more details.

   * The Distutils `Extension' class now supports an extra constructor
     argument named _depends_ for listing additional source files that
     an extension depends on.  This lets Distutils recompile the module
     if any of the dependency files are modified.  For example, if
     `sampmodule.c' includes the header file `sample.h', you would
     create the `Extension' object like this:

         ext = Extension("samp",
                         sources=["sampmodule.c"],
                         depends=["sample.h"])

     Modifying `sample.h' would then cause the module to be recompiled.
     (Contributed by Jeremy Hylton.)

   * Other minor changes to Distutils: it now checks for the `CC', `CFLAGS', `CPP', `LDFLAGS',
     and `CPPFLAGS' environment variables, using them to override the
     settings in Python's configuration (contributed by Robert Weber).

   * Previously the doctest module would only search the
     docstrings of public methods and functions for test cases, but it
     now also examines private ones as well.  The `DocTestSuite(()'
     function creates a unittest.TestSuite object from a
     set of doctest tests.

   * The new `gc.get_referents(object)()' function returns a list of
     all the objects referenced by _object_.

   * The getopt module gained a new function,
     `gnu_getopt()', that supports the same arguments as the existing
     getopt() function but uses GNU-style scanning mode. The
     existing getopt() stops processing options as soon as a
     non-option argument is encountered, but in GNU-style mode
     processing continues, meaning that options and arguments can be
     mixed.  For example:

         >>> getopt.getopt(['-f', 'filename', 'output', '-v'], 'f:v')
         ([('-f', 'filename')], ['output', '-v'])
         >>> getopt.gnu_getopt(['-f', 'filename', 'output', '-v'], 'f:v')
         ([('-f', 'filename'), ('-v', '')], ['output'])

     (Contributed by Peter Åstrand.)

   * The grp, pwd, and resource.
     modules now return enhanced tuples:

         >>> import grp
         >>> g = grp.getgrnam('amk')
         >>> g.gr_name, g.gr_gid
         ('amk', 500)

   * The gzip module can now handle files exceeding 2 GiB.

   * The new heapq module contains an implementation of a
     heap queue algorithm.  A heap is an array-like data structure that
     keeps items in a partially sorted order such that, for every index
     _k_, `heap[k] <= heap[2*k+1]' and `heap[k] <= heap[2*k+2]'.  This
     makes it quick to remove the smallest item, and inserting a new
     item while maintaining the heap property is O(lg n).  (See
     <http://www.nist.gov/dads/HTML/priorityque.html> for more
     information about the priority queue data structure.)

     The heapq module provides `heappush()' and `heappop()'
     functions for adding and removing items while maintaining the heap
     property on top of some other mutable Python sequence type.
     Here's an example that uses a Python list:

         >>> import heapq
         >>> heap = []
         >>> for item in [3, 7, 5, 11, 1]:
         ...    heapq.heappush(heap, item)
         ...
         >>> heap
         [1, 3, 5, 11, 7]
         >>> heapq.heappop(heap)
         1
         >>> heapq.heappop(heap)
         3
         >>> heap
         [5, 7, 11]

     (Contributed by Kevin O'Connor.)

   * The IDLE integrated development environment has been updated using
     the code from the IDLEfork project (<http://idlefork.sf.net>).
     The most notable feature is that the code being developed is now
     executed in a subprocess, meaning that there's no longer any need
     for manual `reload()' operations. IDLE's core code has been
     incorporated into the standard library as the `idlelib' package.

   * The imaplib module now supports IMAP over SSL.
     (Contributed by Piers Lauder and Tino Lange.)

   * The itertools contains a number of useful functions for
     use with iterators, inspired by various functions provided by the
     ML and Haskell languages.  For example,
     `itertools.ifilter(predicate, iterator)' returns all elements in
     the iterator for which the function `predicate()' returns
     True, and `itertools.repeat(obj, N)' returns `obj' _N_ times.
     There are a number of other functions in the module; see the
     package's reference documentation for details.  (Contributed by
     Raymond Hettinger.)

   * Two new functions in the math module,
     `degrees(rads)()' and `radians(degs)()', convert between radians
     and degrees.  Other functions in the math module such
     as math.sin() and math.cos() have always
     required input values measured in radians.  Also, an optional
     _base_ argument was added to math.log() to make it
     easier to compute logarithms for bases other than `e' and `10'.
     (Contributed by Raymond Hettinger.)

   * Several new POSIX functions (`getpgid()', `killpg()', `lchown()',
     `loadavg()', `major()', `makedev()', `minor()', and `mknod()')
     were added to the posix module that underlies the
     os module. (Contributed by Gustavo Niemeyer, Geert
     Jansen, and Denis S.  Otkidach.)

   * In the os module, the `*stat()' family of functions
     can now report fractions of a second in a timestamp.  Such time
     stamps are represented as floats, similar to the value returned by
     time.time().

     During testing, it was found that some applications will break if
     time stamps are floats.  For compatibility, when using the tuple
     interface of the `stat_result' time stamps will be represented as
     integers. When using named fields (a feature first introduced in
     Python 2.2), time stamps are still represented as integers, unless
     os.stat_float_times() is invoked to enable float
     return values:

         >>> os.stat("/tmp").st_mtime
         1034791200
         >>> os.stat_float_times(True)
         >>> os.stat("/tmp").st_mtime
         1034791200.6335014

     In Python 2.4, the default will change to always returning floats.

     Application developers should enable this feature only if all
     their libraries work properly when confronted with floating point
     time stamps, or if they use the tuple API. If used, the feature
     should be activated on an application level instead of trying to
     enable it on a per-use basis.

   * The optparse module contains a new parser for
     command-line arguments that can convert option values to a
     particular Python type  and will automatically generate a usage
     message.  See the following section for  more details.

   * The old and never-documented `linuxaudiodev' module has been
     deprecated, and a new version named ossaudiodev has
     been added.  The module was renamed because the OSS sound drivers
     can be used on platforms other than Linux, and the interface has
     also been tidied and brought up to date in various ways.
     (Contributed by Greg Ward and Nicholas FitzRoy-Dale.)

   * The new platform module contains a number of functions
     that try to determine various properties of the platform you're
     running on.  There are functions for getting the architecture, CPU
     type, the Windows OS version, and even the Linux distribution
     version. (Contributed by Marc-André Lemburg.)

   * The parser objects provided by the `pyexpat' module can now
     optionally buffer character data, resulting in fewer calls to your
     character data handler and therefore faster performance.  Setting
     the parser object's `buffer_text' attribute to True.
     will enable buffering.

   * The `sample(population, k)()' function was added to the
     random.  module.  _population_ is a sequence or xrange
     object containing the elements of a population, and
     `sample()' chooses _k_ elements from the population without
     replacing chosen elements.  _k_ can be any value up to
     `len(population)'. For example:

         >>> days = ['Mo', 'Tu', 'We', 'Th', 'Fr', 'St', 'Sn']
         >>> random.sample(days, 3)      # Choose 3 elements
         ['St', 'Sn', 'Th']
         >>> random.sample(days, 7)      # Choose 7 elements
         ['Tu', 'Th', 'Mo', 'We', 'St', 'Fr', 'Sn']
         >>> random.sample(days, 7)      # Choose 7 again
         ['We', 'Mo', 'Sn', 'Fr', 'Tu', 'St', 'Th']
         >>> random.sample(days, 8)      # Can't choose eight
         Traceback (most recent call last):
           File "<stdin>", line 1, in ?
           File "random.py", line 414, in sample
               raise ValueError, "sample larger than population"
         ValueError: sample larger than population
         >>> random.sample(xrange(1,10000,2), 10)   # Choose ten odd nos. under 10000
         [3407, 3805, 1505, 7023, 2401, 2267, 9733, 3151, 8083, 9195]

     The random module now uses a new algorithm, the
     Mersenne Twister, implemented in C.  It's faster and more
     extensively studied than the previous algorithm.

     (All changes contributed by Raymond Hettinger.)

   * The readline module also gained a number of new
     functions: `get_history_item()', `get_current_history_length()',
     and `redisplay()'.

   * The rexec and Bastion modules have been
     declared dead, and attempts to import them will fail with a
     RuntimeError.  New-style classes provide new ways to break
     out of the restricted execution environment provided by
     rexec, and no one has interest in fixing them or time to do
     so.  If you have applications using rexec, rewrite them
     to use something else.

     (Sticking with Python 2.2 or 2.1 will not make your applications
     any safer because there are known bugs in the rexec.
     module in those versions.  To repeat: if you're using rexec,
     stop using it immediately.)

   * The `rotor' module has been deprecated because the  algorithm it
     uses for encryption is not believed to be secure.  If you need
     encryption, use one of the several AES Python modules that are
     available separately.

   * The shutil module gained a `move(src, dest)()'
     function that recursively moves a file or directory to a new
     location.

   * Support for more advanced POSIX signal handling was added to the
     signal.  but then removed again as it proved impossible
     to make it work reliably across platforms.

   * The socket module now supports timeouts.  You can call
     the `settimeout(t)()' method on a socket object to set a timeout
     of _t_ seconds.  Subsequent socket operations that take longer
     than _t_ seconds to complete will abort and raise a
     socket.timeout exception.

     The original timeout implementation was by Tim O'Malley.  Michael
     Gilfix integrated it into the Python socket module and
     shepherded it through a lengthy review.  After the code was
     checked in, Guido van Rossum rewrote parts of it.  (This is a good
     example of a collaborative development process in action.)

   * On Windows, the socket module now ships with Secure
     Sockets Layer (SSL) support.

   * The value of the C `PYTHON_API_VERSION' macro is now exposed at the
     Python level as `sys.api_version'.  The current exception can be
     cleared by calling the new sys.exc_clear() function.

   * The new tarfile module  allows reading from and
     writing to *tar*-format archive files. (Contributed by Lars
     Gustäbel.)

   * The new textwrap module contains functions for
     wrapping strings containing paragraphs of text.  The `wrap(text,
     width)()' function takes a string and returns a list containing
     the text split into lines of no more than the chosen width.  The
     `fill(text, width)()' function returns a single string,
     reformatted to fit into lines no longer than the chosen width. (As
     you can guess, `fill()' is built on top of `wrap()'.  For example:

         >>> import textwrap
         >>> paragraph = "Not a whit, we defy augury: ... more text ..."
         >>> textwrap.wrap(paragraph, 60)
         ["Not a whit, we defy augury: there's a special providence in",
          "the fall of a sparrow. If it be now, 'tis not to come; if it",
          ...]
         >>> print textwrap.fill(paragraph, 35)
         Not a whit, we defy augury: there's
         a special providence in the fall of
         a sparrow. If it be now, 'tis not
         to come; if it be not to come, it
         will be now; if it be not now, yet
         it will come: the readiness is all.
         >>>

     The module also contains a `TextWrapper' class that actually
     implements the text wrapping strategy.   Both the `TextWrapper'
     class and the `wrap()' and `fill()' functions support a number of
     additional keyword arguments for fine-tuning the formatting;
     consult the module's documentation for details. (Contributed by
     Greg Ward.)

   * The thread and threading modules now have
     companion modules, see dummy_thread and
     dummy_threading, that provide a do-nothing implementation of
     the thread module's interface for platforms where
     threads are not supported.  The intention is to simplify
     thread-aware modules (ones that _don't_ rely on threads to run) by
     putting the following code at the top:

         try:
             import threading as _threading
         except ImportError:
             import dummy_threading as _threading

     In this example, `_threading' is used as the module name to make
     it clear that the module being used is not necessarily the actual
     threading.  module. Code can call functions and use
     classes in `_threading' whether or not threads are supported,
     avoiding an if statement and making the code slightly
     clearer.  This module will not magically make multithreaded code
     run without threads; code that waits for another thread to return
     or to do something will simply hang forever.

   * The time module's `strptime()' function has long been
     an annoyance because it uses the platform C library's `strptime()'
     implementation, and different platforms sometimes have odd bugs.
     Brett Cannon contributed a portable implementation that's written
     in pure Python and should behave identically on all platforms.

   * The new timeit module helps measure how long snippets
     of Python code take to execute.  The `timeit.py' file can be run
     directly from the command line, or the module's `Timer' class can
     be imported and used directly.  Here's a short example that
     figures out whether it's faster to convert an 8-bit string to
     Unicode by appending an empty Unicode string to it or by using the
     unicode() function:

         import timeit

         timer1 = timeit.Timer('unicode("abc")')
         timer2 = timeit.Timer('"abc" + u""')

         # Run three trials
         print timer1.repeat(repeat=3, number=100000)
         print timer2.repeat(repeat=3, number=100000)

         # On my laptop this outputs:
         # [0.36831796169281006, 0.37441694736480713, 0.35304892063140869]
         # [0.17574405670166016, 0.18193507194519043, 0.17565798759460449]

   * The Tix module has received various bug fixes and
     updates for the current version of the Tix package.

   * The Tkinter module now works with a thread-enabled
     version of Tcl.  Tcl's threading model requires that widgets only
     be accessed from the thread in which they're created; accesses
     from another thread can cause Tcl to panic.  For certain Tcl
     interfaces, see Tkinter will now automatically avoid this
     when a widget is accessed from a different thread by marshalling a
     command, passing it to the correct thread, and waiting for the
     results.  Other interfaces can't be handled automatically but
     Tkinter will now raise an exception on such an access
     so that you can at least find out about the problem.  See
     <http://mail.python.org/pipermail/python-dev/2002-December/031107.html>
     for a more detailed explanation of this change.  (Implemented by
     Martin von Löwis.)

   * Calling Tcl methods through `_tkinter' no longer  returns only
     strings.  Instead, if Tcl returns other objects those objects are
     converted to their Python equivalent, if one exists, or wrapped
     with a `_tkinter.Tcl_Obj' object if no Python equivalent exists.
     This behavior can be controlled through the `wantobjects()' method
     of `tkapp' objects.

     When using `_tkinter' through the Tkinter module (as
     most Tkinter applications will), this feature is always activated.
     It should not cause compatibility problems, since Tkinter would
     always convert string results to Python types where possible.

     If any incompatibilities are found, the old behavior can be
     restored by setting the `wantobjects' variable in the
     Tkinter module to false before creating the first `tkapp'
     object.

         import Tkinter
         Tkinter.wantobjects = 0

     Any breakage caused by this change should be reported as a bug.

   * The UserDict module has a new `DictMixin' class which
     defines all dictionary methods for classes that already have a
     minimum mapping interface.  This greatly simplifies writing
     classes that need to be substitutable for dictionaries, such as
     the classes in  the shelve.  module.

     Adding the mix-in as a superclass provides the full dictionary
     interface whenever the class defines __getitem__(),
     __setitem__(), __delitem__(), and `keys()'.
     For example:

         >>> import UserDict
         >>> class SeqDict(UserDict.DictMixin):
         ...     """Dictionary lookalike implemented with lists."""
         ...     def __init__(self):
         ...         self.keylist = []
         ...         self.valuelist = []
         ...     def __getitem__(self, key):
         ...         try:
         ...             i = self.keylist.index(key)
         ...         except ValueError:
         ...             raise KeyError
         ...         return self.valuelist[i]
         ...     def __setitem__(self, key, value):
         ...         try:
         ...             i = self.keylist.index(key)
         ...             self.valuelist[i] = value
         ...         except ValueError:
         ...             self.keylist.append(key)
         ...             self.valuelist.append(value)
         ...     def __delitem__(self, key):
         ...         try:
         ...             i = self.keylist.index(key)
         ...         except ValueError:
         ...             raise KeyError
         ...         self.keylist.pop(i)
         ...         self.valuelist.pop(i)
         ...     def keys(self):
         ...         return list(self.keylist)
         ...
         >>> s = SeqDict()
         >>> dir(s)      # See that other dictionary methods are implemented
         ['__cmp__', '__contains__', '__delitem__', '__doc__', '__getitem__',
          '__init__', '__iter__', '__len__', '__module__', '__repr__',
          '__setitem__', 'clear', 'get', 'has_key', 'items', 'iteritems',
          'iterkeys', 'itervalues', 'keylist', 'keys', 'pop', 'popitem',
          'setdefault', 'update', 'valuelist', 'values']

     (Contributed by Raymond Hettinger.)

   * The DOM implementation in xml.dom.minidom can now
     generate XML output in a particular encoding by providing an
     optional encoding argument to the `toxml()' and `toprettyxml()'
     methods of DOM nodes.

   * The xmlrpclib module now supports an XML-RPC extension
     for handling nil data values such as Python's `None'.  Nil values
     are always supported on unmarshalling an XML-RPC response.  To
     generate requests containing `None', you must supply a true value
     for the _allow_none_ parameter when creating a `Marshaller'
     instance.

   * The new DocXMLRPCServer module allows writing
     self-documenting XML-RPC servers. Run it in demo mode (as a
     program) to see it in action.   Pointing the Web browser to the
     RPC server produces pydoc-style documentation; pointing xmlrpclib
     to the server allows invoking the actual methods. (Contributed by
     Brian Quinlan.)

   * Support for internationalized domain names (RFCs 3454, 3490, 3491,
     and 3492) has been added. The "idna" encoding can be used to
     convert between a Unicode domain name and the ASCII-compatible
     encoding (ACE) of that name.

         >{}>{}> u"www.Alliancefrançaise.nu".encode("idna")
         'www.xn--alliancefranaise-npb.nu'

     The socket module has also been extended to
     transparently convert Unicode hostnames to the ACE version before
     passing them to the C library.  Modules that deal with hostnames
     such as httplib and ftplib)  also support
     Unicode host names; see httplib also sends HTTP `Host'
     headers using the ACE version of the domain name.  urllib
     supports Unicode URLs with non-ASCII host names as long as
     the `path' part of the URL is ASCII only.

     To implement this change, the stringprep module, the
     `mkstringprep' tool and the `punycode' encoding have been added.

1.5.17.1 Date/Time Type
.......................

Date and time types suitable for expressing timestamps were added as the
datetime module.  The types don't support different
calendars or many fancy features, and just stick to the basics of
representing time.

  The three primary types are: `date', representing a day, month, and
year; see time, consisting of hour, minute, and second; and
datetime, which contains all the attributes of both `date'
and time.  There's also a `timedelta' class representing
differences between two points in time, and time zone logic is
implemented by classes inheriting from the abstract `tzinfo' class.

  You can create instances of `date' and time by either
supplying keyword arguments to the appropriate constructor, e.g.
`datetime.date(year=1972, month=10, day=15)', or by using one of a
number of class methods.  For example, the `date.today()' class method
returns the current local date.

  Once created, instances of the date/time classes are all immutable.
There are a number of methods for producing formatted strings from
objects:

    >>> import datetime
    >>> now = datetime.datetime.now()
    >>> now.isoformat()
    '2002-12-30T21:27:03.994956'
    >>> now.ctime()  # Only available on date, datetime
    'Mon Dec 30 21:27:03 2002'
    >>> now.strftime('%Y %d %b')
    '2002 30 Dec'

The `replace()' method allows modifying one or more fields  of a `date'
or datetime instance, returning a new instance:

    >>> d = datetime.datetime.now()
    >>> d
    datetime.datetime(2002, 12, 30, 22, 15, 38, 827738)
    >>> d.replace(year=2001, hour = 12)
    datetime.datetime(2001, 12, 30, 12, 15, 38, 827738)
    >>>

Instances can be compared, hashed, and converted to strings (the result
is the same as that of `isoformat()').  `date' and datetime.
instances can be subtracted from each other, and added to `timedelta'
instances.  The largest missing feature is that there's no standard
library support for parsing strings and getting back a `date' or
datetime.

  For more information, refer to the module's reference documentation.
(Contributed by Tim Peters.)

1.5.17.2 The optparse Module
............................

The getopt module provides simple parsing of command-line
arguments.  The new optparse module (originally named
Optik) provides more elaborate command-line parsing that follows the
Unix conventions, automatically creates the output for -help,
and can perform different actions for different options.

  You start by creating an instance of `OptionParser' and telling it
what your program's options are.

    import sys
    from optparse import OptionParser

    op = OptionParser()
    op.add_option('-i', '--input',
                  action='store', type='string', dest='input',
                  help='set input filename')
    op.add_option('-l', '--length',
                  action='store', type='int', dest='length',
                  help='set maximum length of output')

Parsing a command line is then done by calling the `parse_args()'
method.

    options, args = op.parse_args(sys.argv[1:])
    print options
    print args

This returns an object containing all of the option values, and a list
of strings containing the remaining arguments.

  Invoking the script with the various arguments now works as you'd
expect it to.  Note that the length argument is automatically converted
to an integer.

    $ ./python opt.py -i data arg1
    <Values at 0x400cad4c: {'input': 'data', 'length': None}>
    ['arg1']
    $ ./python opt.py --input=data --length=4
    <Values at 0x400cad2c: {'input': 'data', 'length': 4}>
    []
    $

The help message is automatically generated for you:

    $ ./python opt.py --help
    usage: opt.py [options]

    options:
      -h, --help            show this help message and exit
      -iINPUT, --input=INPUT
                            set input filename
      -lLENGTH, --length=LENGTH
                            set maximum length of output
    $

See the module's documentation for more details.

  Optik was written by Greg Ward, with suggestions from the readers of
the Getopt SIG.

1.5.18 Pymalloc: A Specialized Object Allocator
-----------------------------------------------

Pymalloc, a specialized object allocator written by Vladimir
Marangozov, was a feature added to Python 2.1.  Pymalloc is intended to
be faster than the system `malloc()' and to have less memory overhead
for allocation patterns typical of Python programs. The allocator uses
C's `malloc()' function to get large pools of memory and then fulfills
smaller memory requests from these pools.

  In 2.1 and 2.2, pymalloc was an experimental feature and wasn't
enabled by default; you had to explicitly enable it when compiling
Python by providing the `--with-pymalloc' option to the *configure*
script.  In 2.3, pymalloc has had further enhancements and is now
enabled by default; you'll have to supply `--without-pymalloc' to
disable it.

  This change is transparent to code written in Python; however,
pymalloc may expose bugs in C extensions.  Authors of C extension
modules should test their code with pymalloc enabled, because some
incorrect code may cause core dumps at runtime.

  There's one particularly common error that causes problems.  There
are a number of memory allocation functions in Python's C API that have
previously just been aliases for the C library's `malloc()' and
`free()', meaning that if you accidentally called mismatched functions
the error wouldn't be noticeable.  When the object allocator is
enabled, these functions aren't aliases of `malloc()' and `free()' any
more, and calling the wrong function to free memory may get you a core
dump.  For example, if memory was allocated using `PyObject_Malloc()',
it has to be freed using `PyObject_Free()', not `free()'.  A few
modules included with Python fell afoul of this and had to be fixed;
doubtless there are more third-party modules that will have the same
problem.

  As part of this change, the confusing multiple interfaces for
allocating memory have been consolidated down into two API families.
Memory allocated with one family must not be manipulated with functions
from the other family.  There is one family for allocating chunks of
memory and another family of functions specifically for allocating
Python objects.

   * To allocate and free an undistinguished chunk of memory use the
     "raw memory" family: PyMem_Malloc(),
     PyMem_Realloc(), and PyMem_Free().

   * The "object memory" family is the interface to the pymalloc
     facility described above and is biased towards a large number of
     "small" allocations: `PyObject_Malloc()', `PyObject_Realloc()',
     and `PyObject_Free()'.

   * To allocate and free Python objects, use the "object" family
     PyObject_New(), PyObject_NewVar(), and
     PyObject_Del().

  Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides
debugging features to catch memory overwrites and doubled frees in both
extension modules and in the interpreter itself.  To enable this
support, compile a debugging version of the Python interpreter by
running *configure* with `--with-pydebug'.

  To aid extension writers, a header file `Misc/pymemcompat.h' is
distributed with the source to Python 2.3 that allows Python extensions
to use the 2.3 interfaces to memory allocation while compiling against
any version of Python since 1.5.2.  You would copy the file from
Python's source distribution and bundle it with the source of your
extension.

<http://svn.python.org/view/python/trunk/Objects/obmalloc.c>
     For the full details of the pymalloc implementation, see the
     comments at the top of the file `Objects/obmalloc.c' in the Python
     source code.  The above link points to the file within the
     python.org SVN browser.

1.5.19 Build and C API Changes
------------------------------

Changes to Python's build process and to the C API include:

   * The cycle detection implementation used by the garbage collection
     has proven to be stable, so it's now been made mandatory.  You can
     no longer compile Python without it, and the `--with-cycle-gc'
     switch to *configure* has been removed.

   * Python can now optionally be built as a shared library
     (`libpython2.3.so') by supplying `--enable-shared' when running
     Python's *configure* script.  (Contributed by Ondrej Palkovsky.)

   * The `DL_EXPORT' and `DL_IMPORT' macros are now deprecated.
     Initialization functions for Python extension modules should now
     be declared using the new macro `PyMODINIT_FUNC', while the Python
     core will generally use the `PyAPI_FUNC' and `PyAPI_DATA' macros.

   * The interpreter can be compiled without any docstrings for the
     built-in functions and modules by supplying
     `--without-doc-strings' to the *configure* script. This makes the
     Python executable about 10% smaller, but will also mean that you
     can't get help for Python's built-ins.  (Contributed by Gustavo
     Niemeyer.)

   * The `PyArg_NoArgs()' macro is now deprecated, and code that uses it
     should be changed.  For Python 2.2 and later, the method
     definition table can specify the METH_NOARGS flag,
     signalling that there are no arguments, and the argument checking
     can then be removed.  If compatibility with pre-2.2 versions of
     Python is important, the code could use `PyArg_ParseTuple(args,
     "")' instead, but this will be slower than using
     METH_NOARGS.

   * PyArg_ParseTuple() accepts new format characters for
     various sizes of unsigned integers: `B' for `unsigned char', `H'
     for `unsigned short int',  `I' for `unsigned int',  and `K' for
     `unsigned long long'.

   * A new function, `PyObject_DelItemString(mapping, char *key)()' was
     added as shorthand for `PyObject_DelItem(mapping,
     PyString_New(key))'.

   * File objects now manage their internal string buffer differently,
     increasing it exponentially when needed.  This results in the
     benchmark tests in `Lib/test/test_bufio.py' speeding up
     considerably (from 57 seconds to 1.7 seconds, according to one
     measurement).

   * It's now possible to define class and static methods for a C
     extension type by setting either the METH_CLASS or
     METH_STATIC flags in a method's PyMethodDef
     structure.

   * Python now includes a copy of the Expat XML parser's source code,
     removing any dependence on a system version or local installation
     of Expat.

   * If you dynamically allocate type objects in your extension, you
     should be aware of a change in the rules relating to the
     `__module__' and `__name__' attributes.  In summary, you will want
     to ensure the type's dictionary contains a `'__module__'' key;
     making the module name the part of the type name leading up to the
     final period will no longer have the desired effect.  For more
     detail, read the API reference documentation or the  source.

1.5.19.1 Port-Specific Changes
..............................

Support for a port to IBM's OS/2 using the EMX runtime environment was
merged into the main Python source tree.  EMX is a POSIX emulation
layer over the OS/2 system APIs.  The Python port for EMX tries to
support all the POSIX-like capability exposed by the EMX runtime, and
mostly succeeds; `fork()' and fcntl() are restricted by the
limitations of the underlying emulation layer.  The standard OS/2 port,
which uses IBM's Visual Age compiler, also gained support for
case-sensitive import semantics as part of the integration of the EMX
port into CVS.  (Contributed by Andrew MacIntyre.)

  On MacOS, most toolbox modules have been weaklinked to improve
backward compatibility.  This means that modules will no longer fail to
load if a single routine is missing on the current OS version. Instead
calling the missing routine will raise an exception. (Contributed by
Jack Jansen.)

  The RPM spec files, found in the `Misc/RPM/' directory in the Python
source distribution, were updated for 2.3.  (Contributed by Sean
Reifschneider.)

  Other new platforms now supported by Python include AtheOS
(<http://www.atheos.cx/>), GNU/Hurd, and OpenVMS.

1.5.20 Other Changes and Fixes
------------------------------

As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree.  A search through the CVS change
logs finds there were 523 patches applied and 514 bugs fixed between
Python 2.2 and 2.3.  Both figures are likely to be underestimates.

  Some of the more notable changes are:

   * If the PYTHONINSPECT environment variable is set, the
     Python interpreter will enter the interactive prompt after running
     a Python program, as if Python had been invoked with the -i
     option. The environment variable can be set before running
     the Python interpreter, or it can be set by the Python program as
     part of its execution.

   * The `regrtest.py' script now provides a way to allow "all resources
     except _foo_."  A resource name passed to the -u.
     option can now be prefixed with a hyphen (`'-'') to mean "remove
     this resource."  For example, the option '`-uall,-bsddb'' could be
     used to enable the use of all resources except `bsddb'.

   * The tools used to build the documentation now work under Cygwin as
     well as Unix.

   * The `SET_LINENO' opcode has been removed.  Back in the mists of
     time, this opcode was needed to produce line numbers in tracebacks
     and support trace functions (for, e.g., see pdb). Since
     Python 1.5, the line numbers in tracebacks have been computed
     using a different mechanism that works with "python -O".  For
     Python 2.3 Michael Hudson implemented a similar scheme to
     determine when to call the trace function, removing the need for
     `SET_LINENO' entirely.

     It would be difficult to detect any resulting difference from
     Python code, apart from a slight speed up when Python is run
     without -O.

     C extensions that access the `f_lineno' field of frame objects
     should instead call `PyCode_Addr2Line(f->f_code, f->f_lasti)'.
     This will have the added effect of making the code work as desired
     under "python -O" in earlier versions of Python.

     A nifty new feature is that trace functions can now assign to the
     `f_lineno' attribute of frame objects, changing the line that will
     be executed next.  A `jump' command has been added to the
     pdb debugger taking advantage of this new feature.
     (Implemented by Richie Hindle.)

1.5.21 Porting to Python 2.3
----------------------------

This section lists previously described changes that may require
changes to your code:

   * yield is now always a keyword; if it's used as a
     variable name in your code, a different name must be chosen.

   * For strings _X_ and _Y_, `X in Y' now works if _X_ is more than one
     character long.

   * The int() type constructor will now return a long
     integer instead of raising an OverflowError when a
     string or floating-point number is too large to fit into an
     integer.

   * If you have Unicode strings that contain 8-bit characters, you
     must declare the file's encoding (UTF-8, Latin-1, or whatever) by
     adding a comment to the top of the file.  See section PEP 263; Source Code Encodings
     for more information.

   * Calling Tcl methods through `_tkinter' no longer  returns only
     strings.  Instead, if Tcl returns other objects those objects are
     converted to their Python equivalent, if one exists, or wrapped
     with a `_tkinter.Tcl_Obj' object if no Python equivalent exists.

   * Large octal and hex literals such as `0xffffffff' now trigger a
     FutureWarning. Currently they're stored as 32-bit
     numbers and result in a negative value, but in Python 2.4 they'll
     become positive long integers.

     There are a few ways to fix this warning.  If you really need a
     positive number, just add an `L' to the end of the literal.  If
     you're trying to get a 32-bit integer with low bits set and have
     previously used an expression such as `~(1 << 31)', it's probably
     clearest to start with all bits set and clear the desired upper
     bits. For example, to clear just the top bit (bit 31), you could
     write `0xffffffffL &~(1L<<31)'.

   * You can no longer disable assertions by assigning to `__debug__'.

   * The Distutils `setup()' function has gained various new keyword
     arguments such as _depends_.  Old versions of the Distutils will
     abort if passed unknown keywords.  A solution is to check for the
     presence of the new `get_distutil_options()' function in your
     `setup.py' and only uses the new keywords with a version of the
     Distutils that supports them:

         from distutils import core

         kw = {'sources': 'foo.c', ...}
         if hasattr(core, 'get_distutil_options'):
             kw['depends'] = ['foo.h']
         ext = Extension(**kw)

   * Using `None' as a variable name will now result in a
     SyntaxWarning.  warning.

   * Names of extension types defined by the modules included with
     Python now contain the module and a `'.'' in front of the type
     name.

1.5.22 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Jeff Bauer, Simon Brunning, Brett Cannon, Michael Chermside,
Andrew Dalke, Scott David Daniels, Fred L. Drake, Jr., David Fraser,
Kelly Gerber, Raymond Hettinger, Michael Hudson, Chris Lambert, Detlef
Lannert, Martin von Löwis, Andrew MacIntyre, Lalo Martins, Chad Netzer,
Gustavo Niemeyer, Neal Norwitz, Hans Nowak, Chris Reedy, Francesco
Ricciardi, Vinay Sajip, Neil Schemenauer, Roman Suzi, Jason Tishler,
Just van Rossum.

1.6 What's New in Python 2.2
============================

     Author: A.M. Kuchling

1.6.1 Introduction
------------------

This article explains the new features in Python 2.2.2, released on
October 14, 2002.  Python 2.2.2 is a bugfix release of Python 2.2,
originally released on December 21, 2001.

  Python 2.2 can be thought of as the "cleanup release".  There are
some features such as generators and iterators that are completely new,
but most of the changes, significant and far-reaching though they may
be, are aimed at cleaning up irregularities and dark corners of the
language design.

  This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For full
details, you should refer to the documentation for Python 2.2, such as
the Python Library Reference
(http://www.python.org/doc/2.2/lib/lib.html)and the Python Reference
Manual (http://www.python.org/doc/2.2/ref/ref.html).  If you want to
understand the complete implementation and design rationale for a
change, refer to the PEP for a particular new feature.

1.6.2 PEPs 252 and 253: Type and Class Changes
----------------------------------------------

The largest and most far-reaching changes in Python 2.2 are to Python's
model of objects and classes.  The changes should be backward
compatible, so it's likely that your code will continue to run
unchanged, but the changes provide some amazing new capabilities.
Before beginning this, the longest and most complicated section of this
article, I'll provide an overview of the changes and offer some
comments.

  A long time ago I wrote a Web page listing flaws in Python's design.
One of the most significant flaws was that it's impossible to subclass
Python types implemented in C.  In particular, it's not possible to
subclass built-in types, so you can't just subclass, say, lists in
order to add a single useful method to them. The UserList.
module provides a class that supports all of the methods of lists and
that can be subclassed further, but there's lots of C code that expects
a regular Python list and won't accept a UserList.  instance.

  Python 2.2 fixes this, and in the process adds some exciting new
capabilities.  A brief summary:

   * You can subclass built-in types such as lists and even integers,
     and your subclasses should work in every place that requires the
     original type.

   * It's now possible to define static and class methods, in addition
     to the instance methods available in previous versions of Python.

   * It's also possible to automatically call methods on accessing or
     setting an instance attribute by using a new mechanism called
     _properties_.  Many uses of __getattr__() can be
     rewritten to use properties instead, making the resulting code
     simpler and faster.  As a small side benefit, attributes can now
     have docstrings, too.

   * The list of legal attributes for an instance can be limited to a
     particular set using _slots_, making it possible to safeguard
     against typos and perhaps make more optimizations possible in
     future versions of Python.

  Some users have voiced concern about all these changes.  Sure, they
say, the new features are neat and lend themselves to all sorts of
tricks that weren't possible in previous versions of Python, but they
also make the language more complicated.  Some people have said that
they've always recommended Python for its simplicity, and feel that its
simplicity is being lost.

  Personally, I think there's no need to worry.  Many of the new
features are quite esoteric, and you can write a lot of Python code
without ever needed to be aware of them.  Writing a simple class is no
more difficult than it ever was, so you don't need to bother learning
or teaching them unless they're actually needed.  Some very complicated
tasks that were previously only possible from C will now be possible in
pure Python, and to my mind that's all for the better.

  I'm not going to attempt to cover every single corner case and small
change that were required to make the new features work.  Instead this
section will paint only the broad strokes.  See section Related Links,
"Related Links", for further sources of information about
Python 2.2's new object model.

1.6.2.1 Old and New Classes
...........................

First, you should know that Python 2.2 really has two kinds of classes:
classic or old-style classes, and new-style classes.  The old-style
class model is exactly the same as the class model in earlier versions
of Python.  All the new features described in this section apply only
to new-style classes. This divergence isn't intended to last forever;
eventually old-style classes will be dropped, possibly in Python 3.0.

  So how do you define a new-style class?  You do it by subclassing an
existing new-style class.  Most of Python's built-in types, such as
integers, lists, dictionaries, and even files, are new-style classes
now.  A new-style class named object, the base class for all
built-in types, has also been added so if no built-in type is suitable,
you can just subclass object.:

    class C(object):
        def __init__ (self):
            ...
        ...

This means that class statements that don't have any base
classes are always classic classes in Python 2.2.  (Actually you can
also change this by setting a module-level variable named
__metaclass__. -- see PEP 253
(http://www.python.org/dev/peps/pep-0253) for the details -- but it's
easier to just subclass object)

  The type objects for the built-in types are available as built-ins,
named using a clever trick.  Python has always had built-in functions
named int(), float(), and str().  In
2.2, they aren't functions any more, but type objects that behave as
factories when called.

    >>> int
    <type 'int'>
    >>> int('123')
    123

To make the set of types complete, new type objects such as
dict() and file() have been added.  Here's a more
interesting example, adding a `lock()' method to file objects:

    class LockableFile(file):
        def lock (self, operation, length=0, start=0, whence=0):
            import fcntl
            return fcntl.lockf(self.fileno(), operation,
                               length, start, whence)

The now-obsolete posixfile module contained a class that
emulated all of a file object's methods and also added a `lock()'
method, but this class couldn't be passed to internal functions that
expected a built-in file, something which is possible with our new
`LockableFile'.

1.6.2.2 Descriptors
...................

In previous versions of Python, there was no consistent way to discover
what attributes and methods were supported by an object. There were
some informal conventions, such as defining `__members__' and
`__methods__' attributes that were lists of names, but often the author
of an extension type or a class wouldn't bother to define them.  You
could fall back on inspecting the `__dict__' of an object, but when
class inheritance or an arbitrary __getattr__() hook were
in use this could still be inaccurate.

  The one big idea underlying the new class model is that an API for
describing the attributes of an object using _descriptors_ has been
formalized.  Descriptors specify the value of an attribute, stating
whether it's a method or a field.  With the descriptor API, static
methods and class methods become possible, as well as more exotic
constructs.

  Attribute descriptors are objects that live inside class objects, and
have a few attributes of their own:

   * `__name__' is the attribute's name.

   * `__doc__' is the attribute's docstring.

   * `__get__(object)()' is a method that retrieves the attribute value
     from _object_.

   * `__set__(object, value)()' sets the attribute on _object_ to
     _value_.

   * `__delete__(object, value)()' deletes the _value_  attribute of
     _object_.

  For example, when you write `obj.x', the steps that Python actually
performs are:

    descriptor = obj.__class__.x
    descriptor.__get__(obj)

For methods, `descriptor.__get__()' returns a temporary object that's
callable, and wraps up the instance and the method to be called on it.
This is also why static methods and class methods are now possible;
they have descriptors that wrap up just the method, or the method and
the class.  As a brief explanation of these new kinds of methods,
static methods aren't passed the instance, and therefore resemble
regular functions.  Class methods are passed the class of the object,
but not the object itself.  Static and class methods are defined like
this:

    class C(object):
        def f(arg1, arg2):
            ...
        f = staticmethod(f)

        def g(cls, arg1, arg2):
            ...
        g = classmethod(g)

The staticmethod() function takes the function `f()', and
returns it wrapped up in a descriptor so it can be stored in the class
object.  You might expect there to be special syntax for creating such
methods (`def static f', `defstatic f()', or something like that) but
no such syntax has been defined yet; that's been left for future
versions of Python.

  More new features, such as slots and properties, are also implemented
as new kinds of descriptors, and it's not difficult to write a
descriptor class that does something novel.  For example, it would be
possible to write a descriptor class that made it possible to write
Eiffel-style preconditions and postconditions for a method.  A class
that used this feature might be defined like this:

    from eiffel import eiffelmethod

    class C(object):
        def f(self, arg1, arg2):
            # The actual function
            ...
        def pre_f(self):
            # Check preconditions
            ...
        def post_f(self):
            # Check postconditions
            ...

        f = eiffelmethod(f, pre_f, post_f)

Note that a person using the new `eiffelmethod()' doesn't have to
understand anything about descriptors.  This is why I think the new
features don't increase the basic complexity of the language. There
will be a few wizards who need to know about it in order to write
`eiffelmethod()' or the ZODB or whatever, but most users will just
write code on top of the resulting libraries and ignore the
implementation details.

1.6.2.3 Multiple Inheritance: The Diamond Rule
..............................................

Multiple inheritance has also been made more useful through changing
the rules under which names are resolved.  Consider this set of classes
(diagram taken from PEP 253 (http://www.python.org/dev/peps/pep-0253)
by Guido van Rossum):

          class A:
            ^ ^  def save(self): ...
           /   \
          /     \
         /       \
        /         \
    class B     class C:
        ^         ^  def save(self): ...
         \       /
          \     /
           \   /
            \ /
          class D

The lookup rule for classic classes is simple but not very smart; the
base classes are searched depth-first, going from left to right.  A
reference to `D.save()' will search the classes `D', `B', and then `A',
where `save()' would be found and returned.  `C.save()' would never be
found at all.  This is bad, because if `C''s `save()' method is saving
some internal state specific to `C', not calling it will result in that
state never getting saved.

  New-style classes follow a different algorithm that's a bit more
complicated to explain, but does the right thing in this situation.
(Note that Python 2.3 changes this algorithm to one that produces the
same results in most cases, but produces more useful results for really
complicated inheritance graphs.)

  1. List all the base classes, following the classic lookup rule and
     include a class multiple times if it's visited repeatedly.  In the
     above example, the list of visited classes is [`D', `B', `A', `C',
     `A'].

  2. Scan the list for duplicated classes.  If any are found, remove
     all but one occurrence, leaving the _last_ one in the list.  In
     the above example, the list becomes [`D', `B', `C', `A'] after
     dropping duplicates.

  Following this rule, referring to `D.save()' will return `C.save()',
which is the behaviour we're after.  This lookup rule is the same as
the one followed by Common Lisp.  A new built-in function, see
super(), provides a way to get at a class's superclasses without
having to reimplement Python's algorithm. The most commonly used form
will be  `super(class, obj)()', which returns  a bound superclass
object (not the actual class object).  This form will be used in
methods to call a method in the superclass; for example, `D''s `save()'
method would look like this:

    class D (B,C):
        def save (self):
            # Call superclass .save()
            super(D, self).save()
            # Save D's private information here
            ...

super() can also return unbound superclass objects when
called as `super(class)()' or `super(class1, class2)()', but this
probably won't often be useful.

1.6.2.4 Attribute Access
........................

A fair number of sophisticated Python classes define hooks for
attribute access using __getattr__().; most commonly this is
done for convenience, to make code more readable by automatically
mapping an attribute access such as `obj.parent' into a method call
such as `obj.get_parent'.  Python 2.2 adds some new ways of controlling
attribute access.

  First, `__getattr__(attr_name)()' is still supported by new-style
classes, and nothing about it has changed.  As before, it will be
called when an attempt is made to access `obj.foo' and no attribute
named `foo' is found in the instance's dictionary.

  New-style classes also support a new method,
`__getattribute__(attr_name)()'.  The difference between the two
methods is that __getattribute__() is _always_ called
whenever any attribute is accessed, while the old __getattr__()
is only called if `foo' isn't found in the instance's dictionary.

  However, Python 2.2's support for _properties_ will often be a
simpler way to trap attribute references.  Writing a
__getattr__() method is complicated because to avoid recursion
you can't use regular attribute accesses inside them, and instead have
to mess around with the contents of `__dict__'. __getattr__()
methods also end up being called by Python when it checks for
other methods such as __repr__() or __coerce__(),
and so have to be written with this in mind. Finally, calling a
function on every attribute access results in a sizable performance
loss.

  property is a new built-in type that packages up three
functions that get, set, or delete an attribute, and a docstring.  For
example, if you want to define a `size' attribute that's computed, but
also settable, you could write:

    class C(object):
        def get_size (self):
            result = ... computation ...
            return result
        def set_size (self, size):
            ... compute something based on the size
            and set internal state appropriately ...

        # Define a property.  The 'delete this attribute'
        # method is defined as None, so the attribute
        # can't be deleted.
        size = property(get_size, set_size,
                        None,
                        "Storage size of this instance")

That is certainly clearer and easier to write than a pair of
__getattr__()./see __setattr__() methods that check for
the `size' attribute and handle it specially while retrieving all other
attributes from the instance's `__dict__'.  Accesses to `size' are also
the only ones which have to perform the work of calling a function, so
references to other attributes run at their usual speed.

  Finally, it's possible to constrain the list of attributes that can be
referenced on an object using the new __slots__ class
attribute. Python objects are usually very dynamic; at any time it's
possible to define a new attribute on an instance by just doing
`obj.new_attr=1'.   A new-style class can define a class attribute
named __slots__ to limit the legal attributes  to a
particular set of names.  An example will make this clear:

    >>> class C(object):
    ...     __slots__ = ('template', 'name')
    ...
    >>> obj = C()
    >>> print obj.template
    None
    >>> obj.template = 'Test'
    >>> print obj.template
    Test
    >>> obj.newattr = None
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    AttributeError: 'C' object has no attribute 'newattr'

Note how you get an AttributeError on the attempt to assign
to an attribute not listed in __slots__.

1.6.2.5 Related Links
.....................

This section has just been a quick overview of the new features, giving
enough of an explanation to start you programming, but many details
have been simplified or ignored.  Where should you go to get a more
complete picture?

  <http://www.python.org/2.2/descrintro.html> is a lengthy tutorial
introduction to the descriptor features, written by Guido van Rossum.
If my description has whetted your appetite, go read this tutorial
next, because it goes into much more detail about the new features
while still remaining quite easy to read.

  Next, there are two relevant PEPs, PEP 252
(http://www.python.org/dev/peps/pep-0252) and PEP 253
(http://www.python.org/dev/peps/pep-0253).  PEP 252
(http://www.python.org/dev/peps/pep-0252) is titled "Making Types Look
More Like Classes", and covers the descriptor API.  PEP 253
(http://www.python.org/dev/peps/pep-0253) is titled "Subtyping Built-in
Types", and describes the changes to type objects that make it possible
to subtype built-in objects.  PEP 253
(http://www.python.org/dev/peps/pep-0253) is the more complicated PEP
of the two, and at a few points the necessary explanations of types and
meta-types may cause your head to explode.  Both PEPs were written and
implemented by Guido van Rossum, with substantial assistance from the
rest of the Zope Corp. team.

  Finally, there's the ultimate authority: the source code.  Most of
the machinery for the type handling is in `Objects/typeobject.c', but
you should only resort to it after all other avenues have been
exhausted, including posting a question to python-list or python-dev.

1.6.3 PEP 234: Iterators
------------------------

Another significant addition to 2.2 is an iteration interface at both
the C and Python levels.  Objects can define how they can be looped
over by callers.

  In Python versions up to 2.1, the usual way to make `for item in obj'
work is to define a __getitem__() method that looks
something like this:

    def __getitem__(self, index):
        return <next item>

__getitem__() is more properly used to define an indexing
operation on an object so that you can write `obj[5]' to retrieve the
sixth element.  It's a bit misleading when you're using this only to
support for loops.  Consider some file-like object that
wants to be looped over; the _index_ parameter is essentially
meaningless, as the class probably assumes that a series of
__getitem__() calls will be made with _index_ incrementing by one
each time.  In other words, the presence of the __getitem__()
method doesn't mean that using `file[5]'  to randomly access the
sixth element will work, though it really should.

  In Python 2.2, iteration can be implemented separately, and
__getitem__().  methods can be limited to classes that really do
support random access.  The basic idea of iterators is  simple.  A new
built-in function, `iter(obj)()' or `iter(C, sentinel)', is used to get
an iterator. `iter(obj)()' returns an iterator for the object _obj_,
while `iter(C, sentinel)' returns an iterator that will invoke the
callable object _C_ until it returns _sentinel_ to signal that the
iterator is done.

  Python classes can define an __iter__() method, which
should create and return a new iterator for the object; if the object
is its own iterator, this method can just return `self'.  In
particular, iterators will usually be their own iterators.  Extension
types implemented in C can implement a `tp_iter' function in order to
return an iterator, and extension types that want to behave as
iterators can define a `tp_iternext' function.

  So, after all this, what do iterators actually do?  They have one
required method, see next(), which takes no arguments and
returns the next value.  When there are no more values to be returned,
calling next() should raise the StopIteration.
exception.

    >>> L = [1,2,3]
    >>> i = iter(L)
    >>> print i
    <iterator object at 0x8116870>
    >>> i.next()
    1
    >>> i.next()
    2
    >>> i.next()
    3
    >>> i.next()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    StopIteration
    >>>

In 2.2, Python's for statement no longer expects a
sequence; it expects something for which iter() will return
an iterator. For backward compatibility and convenience, an iterator is
automatically constructed for sequences that don't implement
__iter__() or a `tp_iter' slot, so `for i in [1,2,3]' will still
work.  Wherever the Python interpreter loops over a sequence, it's been
changed to use the iterator protocol.  This means you can do things
like this:

    >>> L = [1,2,3]
    >>> i = iter(L)
    >>> a,b,c = i
    >>> a,b,c
    (1, 2, 3)

Iterator support has been added to some of Python's basic types.
Calling iter() on a dictionary will return an iterator
which loops over its keys:

    >>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
    ...      'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
    >>> for key in m: print key, m[key]
    ...
    Mar 3
    Feb 2
    Aug 8
    Sep 9
    May 5
    Jun 6
    Jul 7
    Jan 1
    Apr 4
    Nov 11
    Dec 12
    Oct 10

That's just the default behaviour.  If you want to iterate over keys,
values, or key/value pairs, you can explicitly call the `iterkeys()',
`itervalues()', or `iteritems()' methods to get an appropriate iterator.
In a minor related change, the in operator now works on
dictionaries, so `key in dict' is now equivalent to `dict.has_key(key)'.

  Files also provide an iterator, which calls the readline()
method until there are no more lines in the file.  This means you
can now read each line of a file using code like this:

    for line in file:
        # do something for each line
        ...

Note that you can only go forward in an iterator; there's no way to get
the previous element, reset the iterator, or make a copy of it. An
iterator object could provide such additional capabilities, but the
iterator protocol only requires a next() method.

PEP 234 (http://www.python.org/dev/peps/pep-0234) - Iterators
     Written by Ka-Ping Yee and GvR; implemented  by the Python Labs
     crew, mostly by GvR and Tim Peters.

1.6.4 PEP 255: Simple Generators
--------------------------------

Generators are another new feature, one that interacts with the
introduction of iterators.

  You're doubtless familiar with how function calls work in Python or
C.  When you call a function, it gets a private namespace where its
local variables are created.  When the function reaches a return
statement, the local variables are destroyed and the resulting
value is returned to the caller.  A later call to the same function
will get a fresh new set of local variables.  But, what if the local
variables weren't thrown away on exiting a function?  What if you could
later resume the function where it left off?  This is what generators
provide; they can be thought of as resumable functions.

  Here's the simplest example of a generator function:

    def generate_ints(N):
        for i in range(N):
            yield i

A new keyword, see yield, was introduced for generators.  Any
function containing a yield statement is a generator
function; this is detected by Python's bytecode compiler which compiles
the function specially as a result.  Because a new keyword was
introduced, generators must be explicitly enabled in a module by
including a `from __future__ import generators' statement near the top
of the module's source code.  In Python 2.3 this statement will become
unnecessary.

  When you call a generator function, it doesn't return a single value;
instead it returns a generator object that supports the iterator
protocol.  On executing the yield statement, the generator
outputs the value of `i', similar to a return statement.
The big difference between yield and a return.
statement is that on reaching a yield the generator's state
of execution is suspended and local variables are preserved.  On the
next call to the generator's `next()' method, the function will resume
executing immediately after the yield.  statement.  (For
complicated reasons, the yield statement isn't allowed
inside the try block of a try...finally
statement; read PEP 255 (http://www.python.org/dev/peps/pep-0255)
for a full explanation of the interaction between yield and
exceptions.)

  Here's a sample usage of the `generate_ints()' generator:

    >>> gen = generate_ints(3)
    >>> gen
    <generator object at 0x8117f90>
    >>> gen.next()
    0
    >>> gen.next()
    1
    >>> gen.next()
    2
    >>> gen.next()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "<stdin>", line 2, in generate_ints
    StopIteration

You could equally write `for i in generate_ints(5)', or `a,b,c =
generate_ints(3)'.

  Inside a generator function, the return statement can
only be used without a value, and signals the end of the procession of
values; afterwards the generator cannot return any further values.
return with a value, such as `return 5', is a syntax error
inside a generator function.  The end of the generator's results can
also be indicated by raising StopIteration.  manually, or by
just letting the flow of execution fall off the bottom of the function.

  You could achieve the effect of generators manually by writing your
own class and storing all the local variables of the generator as
instance variables.  For example, returning a list of integers could be
done by setting `self.count' to 0, and having the next().
method increment `self.count' and return it.  However, for a moderately
complicated generator, writing a corresponding class would be much
messier. `Lib/test/test_generators.py' contains a number of more
interesting examples.  The simplest one implements an in-order
traversal of a tree using generators recursively.

    # A recursive generator that generates Tree leaves in in-order.
    def inorder(t):
        if t:
            for x in inorder(t.left):
                yield x
            yield t.label
            for x in inorder(t.right):
                yield x

Two other examples in `Lib/test/test_generators.py' produce solutions
for the N-Queens problem (placing $N$ queens on an $NxN$ chess board so
that no queen threatens another) and the Knight's Tour (a route that
takes a knight to every square of an $NxN$ chessboard without visiting
any square twice).

  The idea of generators comes from other programming languages,
especially Icon (<http://www.cs.arizona.edu/icon/>), where the idea of
generators is central.  In Icon, every expression and function call
behaves like a generator.  One example from "An Overview of the Icon
Programming Language" at
<http://www.cs.arizona.edu/icon/docs/ipd266.htm> gives an idea of what
this looks like:

    sentence := "Store it in the neighboring harbor"
    if (i := find("or", sentence)) > 5 then write(i)

In Icon the `find()' function returns the indexes at which the substring
"or" is found: 3, 23, 33.  In the if statement, `i' is first
assigned a value of 3, but 3 is less than 5, so the comparison fails,
and Icon retries it with the second value of 23.  23 is greater than 5,
so the comparison now succeeds, and the code prints the value 23 to the
screen.

  Python doesn't go nearly as far as Icon in adopting generators as a
central concept.  Generators are considered a new part of the core
Python language, but learning or using them isn't compulsory; if they
don't solve any problems that you have, feel free to ignore them. One
novel feature of Python's interface as compared to Icon's is that a
generator's state is represented as a concrete object (the iterator)
that can be passed around to other functions or stored in a data
structure.

PEP 255 (http://www.python.org/dev/peps/pep-0255) - Simple Generators
     Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.
     Implemented mostly by Neil Schemenauer and Tim Peters, with other
     fixes from the Python Labs crew.

1.6.5 PEP 237: Unifying Long Integers and Integers
--------------------------------------------------

In recent versions, the distinction between regular integers, which are
32-bit values on most machines, and long integers, which can be of
arbitrary size, was becoming an annoyance.  For example, on platforms
that support files larger than `2**32' bytes, the `tell()' method of
file objects has to return a long integer. However, there were various
bits of Python that expected plain integers and would raise an error if
a long integer was provided instead.  For example, in Python 1.5, only
regular integers could be used as a slice index, and `'abc'[1L:]' would
raise a TypeError exception with the message 'slice index
must be int'.

  Python 2.2 will shift values from short to long integers as required.
The 'L' suffix is no longer needed to indicate a long integer literal,
as now the compiler will choose the appropriate type.  (Using the 'L'
suffix will be discouraged in future 2.x versions of Python, triggering
a warning in Python 2.4, and probably dropped in Python 3.0.)  Many
operations that used to raise an OverflowError will now
return a long integer as their result.  For example:

    >>> 1234567890123
    1234567890123L
    >>> 2 ** 64
    18446744073709551616L

In most cases, integers and long integers will now be treated
identically.  You can still distinguish them with the type()
built-in function, but that's rarely needed.

PEP 237 (http://www.python.org/dev/peps/pep-0237) - Unifying Long Integers and Integers
     Written by Moshe Zadka and Guido van Rossum.  Implemented mostly
     by Guido van Rossum.

1.6.6 PEP 238: Changing the Division Operator
---------------------------------------------

The most controversial change in Python 2.2 heralds the start of an
effort to fix an old design flaw that's been in Python from the
beginning. Currently Python's division operator, `/', behaves like C's
division operator when presented with two integer arguments: it returns
an integer result that's truncated down when there would be a
fractional part.  For example, `3/2' is 1, not 1.5, and `(-1)/2' is -1,
not -0.5.  This means that the results of division can vary
unexpectedly depending on the type of the two operands and because
Python is dynamically typed, it can be difficult to determine the
possible types of the operands.

  (The controversy is over whether this is _really_ a design flaw, and
whether it's worth breaking existing code to fix this.  It's caused
endless discussions on python-dev, and in July 2001 erupted into an
storm of acidly sarcastic postings on `comp.lang.python'. I won't argue
for either side here and will stick to describing what's  implemented
in 2.2.  Read PEP 238 (http://www.python.org/dev/peps/pep-0238) for a
summary of arguments and counter-arguments.)

  Because this change might break code, it's being introduced very
gradually.  Python 2.2 begins the transition, but the switch won't be
complete until Python 3.0.

  First, I'll borrow some terminology from PEP 238
(http://www.python.org/dev/peps/pep-0238).  "True division" is the
division that most non-programmers are familiar with: 3/2 is 1.5, 1/4
is 0.25, and so forth.  "Floor division" is what Python's `/' operator
currently does when given integer operands; the result is the floor of
the value returned by true division.  "Classic division" is the current
mixed behaviour of `/'; it returns the result of floor division when
the operands are integers, and returns the result of true division when
one of the operands is a floating-point number.

  Here are the changes 2.2 introduces:

   * A new operator, `//', is the floor division operator. (Yes, we
     know it looks like C++'s comment symbol.)  `//' _always_ performs
     floor division no matter what the types of its operands are, so `1
     // 2' is 0 and `1.0 // 2.0' is also 0.0.

     `//' is always available in Python 2.2; you don't need to enable
     it using a `__future__' statement.

   * By including a `from __future__ import division' in a module, the
     `/' operator will be changed to return the result of true
     division, so `1/2' is 0.5.  Without the `__future__' statement,
     `/' still means classic division.  The default meaning of `/' will
     not change until Python 3.0.

   * Classes can define methods called __truediv__() and
     __floordiv__().  to overload the two division
     operators.  At the C level, there are also slots in the
     PyNumberMethods structure so extension types can define the
     two operators.

   * Python 2.2 supports some command-line arguments for testing
     whether code will works with the changed division semantics.
     Running python with `-Q warn' will cause a warning to be issued
     whenever division is applied to two integers.  You can use this to
     find code that's affected by the change and fix it.  By default,
     Python 2.2 will simply perform classic division without a warning;
     the warning will be turned on by default in Python 2.3.

PEP 238 (http://www.python.org/dev/peps/pep-0238) - Changing the Division Operator
     Written by Moshe Zadka and  Guido van Rossum.  Implemented by
     Guido van Rossum..

1.6.7 Unicode Changes
---------------------

Python's Unicode support has been enhanced a bit in 2.2.  Unicode
strings are usually stored as UCS-2, as 16-bit unsigned integers.
Python 2.2 can also be compiled to use UCS-4, 32-bit unsigned integers,
as its internal encoding by supplying `--enable-unicode=ucs4' to the
configure script.   (It's also possible to specify `--disable-unicode'
to completely disable Unicode support.)

  When built to use UCS-4 (a "wide Python"), the interpreter can
natively handle Unicode characters from U+000000 to U+110000, so the
range of legal values for the unichr() function is expanded
accordingly.  Using an interpreter compiled to use UCS-2 (a "narrow
Python"), values greater than 65535 will still cause unichr()
to raise a ValueError exception. This is all described
in PEP 261 (http://www.python.org/dev/peps/pep-0261), "Support for
'wide' Unicode characters"; consult it for further details.

  Another change is simpler to explain. Since their introduction,
Unicode strings have supported an `encode()' method to convert the
string to a selected encoding such as UTF-8 or Latin-1.  A symmetric
`decode([*encoding*])()' method has been added to 8-bit strings (though
not to Unicode strings) in 2.2.  `decode()' assumes that the string is
in the specified encoding and decodes it, returning whatever is
returned by the codec.

  Using this new feature, codecs have been added for tasks not directly
related to Unicode.  For example, codecs have been added for
uu-encoding, MIME's base64 encoding, and compression with the
zlib module:

    >>> s = """Here is a lengthy piece of redundant, overly verbose,
    ... and repetitive text.
    ... """
    >>> data = s.encode('zlib')
    >>> data
    'x\x9c\r\xc9\xc1\r\x80 \x10\x04\xc0?Ul...'
    >>> data.decode('zlib')
    'Here is a lengthy piece of redundant, overly verbose,\nand repetitive text.\n'
    >>> print s.encode('uu')
    begin 666 <data>
    M2&5R92!I<R!A(&QE;F=T:'D@<&EE8V4@;V8@<F5D=6YD86YT+"!O=F5R;'D@
    >=F5R8F]S92P*86YD(')E<&5T:71I=F4@=&5X="X*

    end
    >>> "sheesh".encode('rot-13')
    'furrfu'

To convert a class instance to Unicode, a __unicode__().
method can be defined by a class, analogous to __str__().

  `encode()', `decode()', and __unicode__() were
implemented by Marc-André Lemburg.  The changes to support using UCS-4
internally were implemented by Fredrik Lundh and Martin von Löwis.

PEP 261 (http://www.python.org/dev/peps/pep-0261) - Support for 'wide' Unicode characters
     Written by Paul Prescod.

1.6.8 PEP 227: Nested Scopes
----------------------------

In Python 2.1, statically nested scopes were added as an optional
feature, to be enabled by a `from __future__ import nested_scopes'
directive.  In 2.2 nested scopes no longer need to be specially
enabled, and are now always present.  The rest of this section is a
copy of the description of nested scopes from my "What's New in Python
2.1" document; if you read it when 2.1 came out, you can skip the rest
of this section.

  The largest change introduced in Python 2.1, and made complete in
2.2, is to Python's scoping rules.  In Python 2.0, at any given time
there are at most three namespaces used to look up variable names:
local, module-level, and the built-in namespace.  This often surprised
people because it didn't match their intuitive expectations.  For
example, a nested recursive function definition doesn't work:

    def f():
        ...
        def g(value):
            ...
            return g(value-1) + 1
        ...

The function `g()' will always raise a NameError exception,
because the binding of the name `g' isn't in either its local namespace
or in the module-level namespace.  This isn't much of a problem in
practice (how often do you recursively define interior functions like
this?), but this also made using the lambda statement
clumsier, and this was a problem in practice.  In code which uses
lambda you can often find local variables being copied by passing
them as the default values of arguments.

    def find(self, name):
        "Return list of any entries equal to 'name'"
        L = filter(lambda x, name=name: x == name,
                   self.list_attribute)
        return L

The readability of Python code written in a strongly functional style
suffers greatly as a result.

  The most significant change to Python 2.2 is that static scoping has
been added to the language to fix this problem.  As a first effect, the
`name=name' default argument is now unnecessary in the above example.
Put simply, when a given variable name is not assigned a value within a
function (by an assignment, or the def, class, or
import statements), references to the variable will be
looked up in the local namespace of the enclosing scope.  A more
detailed explanation of the rules, and a dissection of the
implementation, can be found in the PEP.

  This change may cause some compatibility problems for code where the
same variable name is used both at the module level and as a local
variable within a function that contains further function definitions.
This seems rather unlikely though, since such code would have been
pretty confusing to read in the first place.

  One side effect of the change is that the `from module import *' and
exec statements have been made illegal inside a function
scope under certain conditions.  The Python reference manual has said
all along that `from module import *' is only legal at the top level of
a module, but the CPython interpreter has never enforced this before.
As part of the implementation of nested scopes, the compiler which
turns Python source into bytecodes has to generate different code to
access variables in a containing scope.  `from module import *' and
exec make it impossible for the compiler to figure this
out, because they add names to the local namespace that are unknowable
at compile time. Therefore, if a function contains function definitions
or lambda expressions with free variables, the compiler
will flag this by raising a SyntaxError exception.

  To make the preceding explanation a bit clearer, here's an example:

    x = 1
    def f():
        # The next line is a syntax error
        exec 'x=2'
        def g():
            return x

Line 4 containing the exec statement is a syntax error,
since exec would define a new local variable named `x'
whose value should be accessed by `g()'.

  This shouldn't be much of a limitation, since exec is
rarely used in most Python code (and when it is used, it's often a sign
of a poor design anyway).

PEP 227 (http://www.python.org/dev/peps/pep-0227) - Statically Nested Scopes
     Written and implemented by Jeremy Hylton.

1.6.9 New and Improved Modules
------------------------------

   * The xmlrpclib module was contributed to the standard
     library by Fredrik Lundh, providing support for writing XML-RPC
     clients.  XML-RPC is a simple remote procedure call protocol built
     on top of HTTP and XML. For example, the following snippet
     retrieves a list of RSS channels from the O'Reilly Network, and
     then  lists the recent headlines for one channel:

         import xmlrpclib
         s = xmlrpclib.Server(
               'http://www.oreillynet.com/meerkat/xml-rpc/server.php')
         channels = s.meerkat.getChannels()
         # channels is a list of dictionaries, like this:
         # [{'id': 4, 'title': 'Freshmeat Daily News'}
         #  {'id': 190, 'title': '32Bits Online'},
         #  {'id': 4549, 'title': '3DGamers'}, ... ]

         # Get the items for one channel
         items = s.meerkat.getItems( {'channel': 4} )

         # 'items' is another list of dictionaries, like this:
         # [{'link': 'http://freshmeat.net/releases/52719/',
         #   'description': 'A utility which converts HTML to XSL FO.',
         #   'title': 'html2fo 0.3 (Default)'}, ... ]

     The SimpleXMLRPCServer module makes it easy to create
     straightforward XML-RPC servers.  See <http://www.xmlrpc.com/> for
     more information about XML-RPC.

   * The new hmac module implements the HMAC algorithm
     described by RFC 2104 (http://tools.ietf.org/html/rfc2104.html).
     (Contributed by Gerhard Häring.)

   * Several functions that originally returned lengthy tuples now
     return pseudo- sequences that still behave like tuples but also
     have mnemonic attributes such as memberst_mtime or `tm_year'. The
     enhanced functions include stat(), `fstat()', see
     statvfs(), and `fstatvfs()' in the os module, and
     `localtime()', `gmtime()', and `strptime()' in the time
     module.

     For example, to obtain a file's size using the old tuples, you'd
     end up writing something like `file_size =
     os.stat(filename)[stat.ST_SIZE]', but now this can be written more
     clearly as `file_size = os.stat(filename).st_size'.

     The original patch for this feature was contributed by Nick
     Mathewson.

   * The Python profiler has been extensively reworked and various
     errors in its output have been corrected.  (Contributed by Fred L.
     Drake, Jr. and Tim Peters.)

   * The socket module can be compiled to support IPv6;
     specify the `--enable-ipv6' option to Python's configure script.
     (Contributed by Jun-ichiro "itojun" Hagino.)

   * Two new format characters were added to the struct.
     module for 64-bit integers on platforms that support the C `long
     long' type.  `q' is for a signed 64-bit integer, and `Q' is for an
     unsigned one.  The value is returned in Python's long integer
     type.  (Contributed by Tim Peters.)

   * In the interpreter's interactive mode, there's a new built-in
     function help() that uses the pydoc module
     introduced in Python 2.1 to provide interactive help.
     `help(object)' displays any available help text about _object_.
     help() with no argument puts you in an online help
     utility, where you can enter the names of functions, classes, or
     modules to read their help text. (Contributed by Guido van Rossum,
     using Ka-Ping Yee's pydoc module.)

   * Various bugfixes and performance improvements have been made to
     the SRE engine underlying the re module.  For example,
     the re.sub() and re.split() functions have
     been rewritten in C.  Another contributed patch speeds up certain
     Unicode character ranges by a factor of two, and a new
     `finditer()'  method that returns an iterator over all the
     non-overlapping matches in  a given string.  (SRE is maintained by
     Fredrik Lundh.  The BIGCHARSET patch was contributed by Martin von
     Löwis.)

   * The smtplib module now supports RFC 2487
     (http://tools.ietf.org/html/rfc2487.html), "Secure SMTP over TLS",
     so it's now possible to encrypt the SMTP traffic between a Python
     program and the mail transport agent being handed a message.
     smtplib also supports SMTP authentication.
     (Contributed by Gerhard Häring.)

   * The imaplib module, maintained by Piers Lauder, has
     support for several new extensions: the NAMESPACE extension
     defined in RFC 2342 (http://tools.ietf.org/html/rfc2342.html),
     SORT, GETACL and SETACL.  (Contributed by Anthony Baxter and
     Michel Pelletier.)

   * The rfc822 module's parsing of email addresses is now
     compliant with RFC 2822 (http://tools.ietf.org/html/rfc2822.html),
     an update to RFC 822 (http://tools.ietf.org/html/rfc822.html).
     (The module's name is _not_ going to be changed to `rfc2822'.)  A
     new package, see email, has also been added for parsing and
     generating e-mail messages.  (Contributed by Barry Warsaw, and
     arising out of his work on Mailman.)

   * The difflib module now contains a new `Differ' class for
     producing human-readable lists of changes (a "delta") between two
     sequences of lines of text.  There are also two generator
     functions, `ndiff()' and `restore()', which respectively return a
     delta from two sequences, or one of the original sequences from a
     delta. (Grunt work contributed by David Goodger, from ndiff.py
     code by Tim Peters who then did the generatorization.)

   * New constants `ascii_letters', `ascii_lowercase', and
     `ascii_uppercase' were added to the string module.
     There were several modules in the standard library that used
     string.letters to mean the ranges A-Za-z, but that
     assumption is incorrect when locales are in use, because
     string.letters varies depending on the set of legal
     characters defined by the current locale.  The buggy modules have
     all been fixed to use `ascii_letters' instead. (Reported by an
     unknown person; fixed by Fred L. Drake, Jr.)

   * The mimetypes module now makes it easier to use
     alternative MIME-type databases by the addition of a `MimeTypes'
     class, which takes a list of filenames to be parsed.  (Contributed
     by Fred L. Drake, Jr.)

   * A `Timer' class was added to the threading module that
     allows scheduling an activity to happen at some future time.
     (Contributed by Itamar Shtull-Trauring.)

1.6.10 Interpreter Changes and Fixes
------------------------------------

Some of the changes only affect people who deal with the Python
interpreter at the C level because they're writing Python extension
modules, embedding the interpreter, or just hacking on the interpreter
itself. If you only write Python code, none of the changes described
here will affect you very much.

   * Profiling and tracing functions can now be implemented in C, which
     can operate at much higher speeds than Python-based functions and
     should reduce the overhead of profiling and tracing.  This  will
     be of interest to authors of development environments for Python.
     Two new C functions were added to Python's API, see
     PyEval_SetProfile() and PyEval_SetTrace(). The
     existing sys.setprofile() and sys.settrace()
     functions still exist, and have simply been changed to use
     the new C-level interface.  (Contributed by Fred L.  Drake, Jr.)

   * Another low-level API, primarily of interest to implementors of
     Python debuggers and development tools, was added.
     PyInterpreterState_Head() and
     PyInterpreterState_Next() let a caller walk through all the
     existing interpreter objects; see
     PyInterpreterState_ThreadHead() and
     PyThreadState_Next() allow looping over all the thread
     states for a given interpreter.  (Contributed by David Beazley.)

   * The C-level interface to the garbage collector has been changed to
     make it easier to write extension types that support garbage
     collection and to debug misuses of the functions. Various
     functions have slightly different semantics, so a bunch of
     functions had to be renamed.  Extensions that use the old API will
     still compile but will _not_ participate in garbage collection, so
     updating them for 2.2 should be considered fairly high priority.

     To upgrade an extension module to the new API, perform the
     following steps:

   * Rename `Py_TPFLAGS_GC()' to `PyTPFLAGS_HAVE_GC()'.

   *
    Use PyObject_GC_New() or PyObject_GC_NewVar() to allocate
          objects, and PyObject_GC_Del() to deallocate them.

   *
    Rename `PyObject_GC_Init()' to PyObject_GC_Track() and
          `PyObject_GC_Fini()' to PyObject_GC_UnTrack().

   * Remove `PyGC_HEAD_SIZE()' from object size calculations.

   * Remove calls to `PyObject_AS_GC()' and `PyObject_FROM_GC()'.

   * A new `et' format sequence was added to PyArg_ParseTuple().;
     `et' takes both a parameter and an encoding name, and
     converts the parameter to the given encoding if the parameter
     turns out to be a Unicode string, or leaves it alone if it's an
     8-bit string, assuming it to already be in the desired encoding.
     This differs from the `es' format character, which assumes that
     8-bit strings are in Python's default ASCII encoding and converts
     them to the specified new encoding. (Contributed by M.-A. Lemburg,
     and used for the MBCS support on Windows described in the
     following section.)

   * A different argument parsing function, see PyArg_UnpackTuple(),
     has been added that's simpler and presumably faster.  Instead
     of specifying a format string, the caller simply gives the minimum
     and maximum number of arguments expected, and a set of pointers to
     PyObject* variables that will be filled in with
     argument values.

   * Two new flags METH_NOARGS and METH_O are
     available in method definition tables to simplify implementation
     of methods with no arguments or a single untyped argument. Calling
     such methods is more efficient than calling a corresponding method
     that uses METH_VARARGS.  Also, the old
     METH_OLDARGS style of writing C methods is  now officially
     deprecated.

   * Two new wrapper functions, see PyOS_snprintf() and
     PyOS_vsnprintf().  were added to provide  cross-platform
     implementations for the relatively new `snprintf()' and
     `vsnprintf()' C lib APIs. In contrast to the standard `sprintf()'
     and `vsprintf()' functions, the Python versions check the bounds
     of the buffer used to protect against buffer overruns.
     (Contributed by M.-A. Lemburg.)

   * The _PyTuple_Resize() function has lost an unused
     parameter, so now it takes 2 parameters instead of 3.  The third
     argument was never used, and can simply be discarded when porting
     code from earlier versions to Python 2.2.

1.6.11 Other Changes and Fixes
------------------------------

As usual there were a bunch of other improvements and bugfixes scattered
throughout the source tree.  A search through the CVS change logs finds
there were 527 patches applied and 683 bugs fixed between Python 2.1
and 2.2; 2.2.1 applied 139 patches and fixed 143 bugs; 2.2.2 applied
106 patches and fixed 82 bugs.  These figures are likely to be
underestimates.

  Some of the more notable changes are:

   * The code for the MacOS port for Python, maintained by Jack Jansen,
     is now kept in the main Python CVS tree, and many changes have
     been made to support MacOS X.

     The most significant change is the ability to build Python as a
     framework, enabled by supplying the `--enable-framework' option to
     the configure script when compiling Python.  According to Jack
     Jansen, "This installs a self- contained Python installation plus
     the OS X framework "glue" into
     `/Library/Frameworks/Python.framework' (or another location of
     choice).  For now there is little immediate added benefit to this
     (actually, there is the disadvantage that you have to change your
     PATH to be able to find Python), but it is the basis for creating
     a full-blown Python application, porting the MacPython IDE,
     possibly using Python as a standard OSA scripting language and
     much more."

     Most of the MacPython toolbox modules, which interface to MacOS
     APIs such as windowing, QuickTime, scripting, etc. have been
     ported to OS X, but they've been left commented out in `setup.py'.
     People who want to experiment with these modules can uncomment
     them manually.

   * Keyword arguments passed to built-in functions that don't take
     them now cause a TypeError exception to be raised,
     with the message "_function_ takes no keyword arguments".

   * Weak references, added in Python 2.1 as an extension module, are
     now part of the core because they're used in the implementation of
     new-style classes.  The ReferenceError exception has
     therefore moved from the weakref.  module to become a
     built-in exception.

   * A new script, `Tools/scripts/cleanfuture.py' by Tim Peters,
     automatically removes obsolete `__future__' statements from Python
     source code.

   * An additional _flags_ argument has been added to the built-in
     function compile(), so the behaviour of `__future__'
     statements can now be correctly observed in simulated shells, such
     as those presented by IDLE and other development environments.
     This is described in PEP 264
     (http://www.python.org/dev/peps/pep-0264). (Contributed by Michael
     Hudson.)

   * The new license introduced with Python 1.6 wasn't GPL-compatible.
     This is fixed by some minor textual changes to the 2.2 license, so
     it's now legal to embed Python inside a GPLed program again.  Note
     that Python itself is not GPLed, but instead is under a license
     that's essentially equivalent to the BSD license, same as it
     always was.  The license changes were also applied to the Python
     2.0.1 and 2.1.1 releases.

   * When presented with a Unicode filename on Windows, Python will now
     convert it to an MBCS encoded string, as used by the Microsoft
     file APIs.  As MBCS is explicitly used by the file APIs, Python's
     choice of ASCII as the default encoding turns out to be an
     annoyance.  On Unix, the locale's character set is used if
     `locale.nl_langinfo(CODESET)()' is available.  (Windows support was
     contributed by Mark Hammond with assistance from Marc-André
     Lemburg. Unix support was added by Martin von Löwis.)

   * Large file support is now enabled on Windows.  (Contributed by Tim
     Peters.)

   * The `Tools/scripts/ftpmirror.py' script now parses a `.netrc'
     file, if you have one. (Contributed by Mike Romberg.)

   * Some features of the object returned by the xrange().
     function are now deprecated, and trigger warnings when they're
     accessed; they'll disappear in Python 2.3. See xrange.
     objects tried to pretend they were full sequence types by
     supporting slicing, sequence multiplication, and the in.
     operator, but these features were rarely used and therefore buggy.
     The `tolist()' method and the `start', `stop', and `step'
     attributes are also being deprecated.  At the C level, the fourth
     argument to the `PyRange_New()' function, `repeat', has also been
     deprecated.

   * There were a bunch of patches to the dictionary implementation,
     mostly to fix potential core dumps if a dictionary contains
     objects that sneakily changed their hash value, or mutated the
     dictionary they were contained in. For a while python-dev fell
     into a gentle rhythm of Michael Hudson finding a case that dumped
     core, Tim Peters fixing the bug, Michael finding another case, and
     round and round it went.

   * On Windows, Python can now be compiled with Borland C thanks to a
     number of patches contributed by Stephen Hansen, though the result
     isn't fully functional yet.  (But this _is_ progress...)

   * Another Windows enhancement: Wise Solutions generously offered
     PythonLabs use of their InstallerMaster 8.1 system.  Earlier
     PythonLabs Windows installers used Wise 5.0a, which was beginning
     to show its age.  (Packaged up by Tim Peters.)

   * Files ending in `.pyw' can now be imported on Windows. `.pyw' is a
     Windows-only thing, used to indicate that a script needs to be run
     using PYTHONW.EXE instead of PYTHON.EXE in order to prevent a DOS
     console from popping up to display the output.  This patch makes
     it possible to import such scripts, in case they're also usable as
     modules.  (Implemented by David Bolen.)

   * On platforms where Python uses the C `dlopen()' function  to load
     extension modules, it's now possible to set the flags used  by
     `dlopen()' using the sys.getdlopenflags() and
     sys.setdlopenflags() functions.  (Contributed by Bram Stolk.)

   * The pow() built-in function no longer supports 3
     arguments when floating-point numbers are supplied. `pow(x, y, z)'
     returns `(x**y) % z', but this is never useful for floating point
     numbers, and the final result varies unpredictably depending on
     the platform.  A call such as `pow(2.0, 8.0, 7.0)' will now raise
     a TypeError exception.

1.6.12 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Fred Bremmer, Keith Briggs, Andrew Dalke, Fred L. Drake, Jr.,
Carel Fellinger, David Goodger, Mark Hammond, Stephen Hansen, Michael
Hudson, Jack Jansen, Marc-André Lemburg, Martin von Löwis, Fredrik
Lundh, Michael McLay, Nick Mathewson, Paul Moore, Gustavo Niemeyer, Don
O'Donnell, Joonas Paalasma, Tim Peters, Jens Quade, Tom Reinhardt, Neil
Schemenauer, Guido van Rossum, Greg Ward, Edward Welbourne.

1.7 What's New in Python 2.1
============================

     Author: A.M. Kuchling

1.7.1 Introduction
------------------

This article explains the new features in Python 2.1.  While there
aren't as many changes in 2.1 as there were in Python 2.0, there are
still some pleasant surprises in store.  2.1 is the first release to be
steered through the use of Python Enhancement Proposals, or PEPs, so
most of the sizable changes have accompanying PEPs that provide more
complete documentation and a design rationale for the change.  This
article doesn't attempt to document the new features completely, but
simply provides an overview of the new features for Python programmers.
Refer to the Python 2.1 documentation, or to the specific PEP, for more
details about any new feature that particularly interests you.

  One recent goal of the Python development team has been to accelerate
the pace of new releases, with a new release coming every 6 to 9
months. 2.1 is the first release to come out at this faster pace, with
the first alpha appearing in January, 3 months after the final version
of 2.0 was released.

  The final release of Python 2.1 was made on April 17, 2001.

1.7.2 PEP 227: Nested Scopes
----------------------------

The largest change in Python 2.1 is to Python's scoping rules.  In
Python 2.0, at any given time there are at most three namespaces used
to look up variable names: local, module-level, and the built-in
namespace.  This often surprised people because it didn't match their
intuitive expectations.  For example, a nested recursive function
definition doesn't work:

    def f():
        ...
        def g(value):
            ...
            return g(value-1) + 1
        ...

The function `g()' will always raise a NameError exception,
because the binding of the name `g' isn't in either its local namespace
or in the module-level namespace.  This isn't much of a problem in
practice (how often do you recursively define interior functions like
this?), but this also made using the lambda statement
clumsier, and this was a problem in practice.  In code which uses
lambda you can often find local variables being copied by passing
them as the default values of arguments.

    def find(self, name):
        "Return list of any entries equal to 'name'"
        L = filter(lambda x, name=name: x == name,
                   self.list_attribute)
        return L

The readability of Python code written in a strongly functional style
suffers greatly as a result.

  The most significant change to Python 2.1 is that static scoping has
been added to the language to fix this problem.  As a first effect, the
`name=name' default argument is now unnecessary in the above example.
Put simply, when a given variable name is not assigned a value within a
function (by an assignment, or the def, class, or
import statements), references to the variable will be
looked up in the local namespace of the enclosing scope.  A more
detailed explanation of the rules, and a dissection of the
implementation, can be found in the PEP.

  This change may cause some compatibility problems for code where the
same variable name is used both at the module level and as a local
variable within a function that contains further function definitions.
This seems rather unlikely though, since such code would have been
pretty confusing to read in the first place.

  One side effect of the change is that the `from module import *' and
exec statements have been made illegal inside a function
scope under certain conditions.  The Python reference manual has said
all along that `from module import *' is only legal at the top level of
a module, but the CPython interpreter has never enforced this before.
As part of the implementation of nested scopes, the compiler which
turns Python source into bytecodes has to generate different code to
access variables in a containing scope.  `from module import *' and
exec make it impossible for the compiler to figure this
out, because they add names to the local namespace that are unknowable
at compile time. Therefore, if a function contains function definitions
or lambda expressions with free variables, the compiler
will flag this by raising a SyntaxError exception.

  To make the preceding explanation a bit clearer, here's an example:

    x = 1
    def f():
        # The next line is a syntax error
        exec 'x=2'
        def g():
            return x

Line 4 containing the exec statement is a syntax error,
since exec would define a new local variable named `x'
whose value should be accessed by `g()'.

  This shouldn't be much of a limitation, since exec is
rarely used in most Python code (and when it is used, it's often a sign
of a poor design anyway).

  Compatibility concerns have led to nested scopes being introduced
gradually; in Python 2.1, they aren't enabled by default, but can be
turned on within a module by using a future statement as described in
PEP 236.  (See the following section for further discussion of PEP
236.)  In Python 2.2, nested scopes will become the default and there
will be no way to turn them off, but users will have had all of 2.1's
lifetime to fix any breakage resulting from their introduction.

PEP 227 (http://www.python.org/dev/peps/pep-0227) - Statically Nested Scopes
     Written and implemented by Jeremy Hylton.

1.7.3 PEP 236: __future__ Directives
------------------------------------

The reaction to nested scopes was widespread concern about the dangers
of breaking code with the 2.1 release, and it was strong enough to make
the Pythoneers take a more conservative approach.  This approach
consists of introducing a convention for enabling optional
functionality in release N that will become compulsory in release N+1.

  The syntax uses a `from...import' statement using the reserved module
name __future__.  Nested scopes can be enabled by the
following statement:

    from __future__ import nested_scopes

While it looks like a normal import statement, it's not;
there are strict rules on where such a future statement can be put.
They can only be at the top of a module, and must precede any Python
code or regular import statements.  This is because such
statements can affect how the Python bytecode compiler parses code and
generates bytecode, so they must precede any statement that will result
in bytecodes being produced.

PEP 236 (http://www.python.org/dev/peps/pep-0236) - Back to the __future__.
     Written by Tim Peters, and primarily implemented by Jeremy Hylton.

1.7.4 PEP 207: Rich Comparisons
-------------------------------

In earlier versions, Python's support for implementing comparisons on
user- defined classes and extension types was quite simple. Classes
could implement a __cmp__() method that was given two
instances of a class, and could only return 0 if they were equal or +1
or -1 if they weren't; the method couldn't raise an exception or return
anything other than a Boolean value.  Users of Numeric Python often
found this model too weak and restrictive, because in the
number-crunching programs that numeric Python is used for, it would be
more useful to be able to perform elementwise comparisons of two
matrices, returning a matrix containing the results of a given
comparison for each element.  If the two matrices are of different
sizes, then the compare has to be able to raise an exception to signal
the error.

  In Python 2.1, rich comparisons were added in order to support this
need.  Python classes can now individually overload each of the `<',
`<=', `>', `>=', `==', and `!=' operations.  The new magic method names
are:

Operation       Method name
-------------------------------------
`<'             __lt__().
`<='            __le__().
`>'             __gt__().
`>='            __ge__().
`=='            __eq__().
`!='            __ne__().

  (The magic methods are named after the corresponding Fortran
operators `.LT.'.  `.LE.', &c.  Numeric programmers are almost
certainly quite familiar with these names and will find them easy to
remember.)

  Each of these magic methods is of the form `method(self, other)',
where `self' will be the object on the left-hand side of the operator,
while `other' will be the object on the right-hand side.  For example,
the expression `A < B' will cause `A.__lt__(B)' to be called.

  Each of these magic methods can return anything at all: a Boolean, a
matrix, a list, or any other Python object.  Alternatively they can
raise an exception if the comparison is impossible, inconsistent, or
otherwise meaningless.

  The built-in `cmp(A,B)()' function can use the rich comparison
machinery, and now accepts an optional argument specifying which
comparison operation to use; this is given as one of the strings `"<"',
`"<="', `">"', `">="', `"=="', or `"!="'.  If called without the
optional third argument, see cmp() will only return -1, 0, or
+1 as in previous versions of Python; otherwise it will call the
appropriate method and can return any Python object.

  There are also corresponding changes of interest to C programmers;
there's a new slot `tp_richcmp' in type objects and an API for
performing a given rich comparison.  I won't cover the C API here, but
will refer you to PEP 207, or to 2.1's C API documentation, for the
full list of related functions.

PEP 207 (http://www.python.org/dev/peps/pep-0207) - Rich Comparisions
     Written by Guido van Rossum, heavily based on earlier work by
     David Ascher, and implemented by Guido van Rossum.

1.7.5 PEP 230: Warning Framework
--------------------------------

Over its 10 years of existence, Python has accumulated a certain number
of obsolete modules and features along the way.  It's difficult to know
when a feature is safe to remove, since there's no way of knowing how
much code uses it -- perhaps no programs depend on the feature, or
perhaps many do.  To enable removing old features in a more structured
way, a warning framework was added.  When the Python developers want to
get rid of a feature, it will first trigger a warning in the next
version of Python.  The following Python version can then drop the
feature, and users will have had a full release cycle to remove uses of
the old feature.

  Python 2.1 adds the warning framework to be used in this scheme.  It
adds a warnings module that provide functions to issue
warnings, and to filter out warnings that you don't want to be
displayed. Third-party modules can also use this framework to deprecate
old features that they no longer wish to support.

  For example, in Python 2.1 the `regex' module is deprecated, so
importing it causes a warning to be printed:

    >>> import regex
    __main__:1: DeprecationWarning: the regex module
             is deprecated; please use the re module
    >>>

Warnings can be issued by calling the warnings.warn().
function:

    warnings.warn("feature X no longer supported")

The first parameter is the warning message; an additional optional
parameters can be used to specify a particular warning category.

  Filters can be added to disable certain warnings; a regular
expression pattern can be applied to the message or to the module name
in order to suppress a warning.  For example, you may have a program
that uses the `regex' module and not want to spare the time to convert
it to use the re module right now.  The warning can be
suppressed by calling

    import warnings
    warnings.filterwarnings(action = 'ignore',
                            message='.*regex module is deprecated',
                            category=DeprecationWarning,
                            module = '__main__')

This adds a filter that will apply only to warnings of the class
`DeprecationWarning' triggered in the __main__ module, and
applies a regular expression to only match the message about the
`regex' module being deprecated, and will cause such warnings to be
ignored.  Warnings can also be printed only once, printed every time
the offending code is executed, or turned into exceptions that will
cause the program to stop (unless the exceptions are caught in the
usual way, of course).

  Functions were also added to Python's C API for issuing warnings;
refer to PEP 230 or to Python's API documentation for the details.

PEP 5 (http://www.python.org/dev/peps/pep-0005) - Guidelines for Language Evolution
     Written by Paul Prescod, to specify procedures to be followed when
     removing old features from Python.  The policy described in this
     PEP hasn't been officially adopted, but the eventual policy
     probably won't be too different from Prescod's proposal.

PEP 230 (http://www.python.org/dev/peps/pep-0230) - Warning Framework
     Written and implemented by Guido van Rossum.

1.7.6 PEP 229: New Build System
-------------------------------

When compiling Python, the user had to go in and edit the
`Modules/Setup' file in order to enable various additional modules; the
default set is relatively small and limited to modules that compile on
most Unix platforms.  This means that on Unix platforms with many more
features, most notably Linux, Python installations often don't contain
all useful modules they could.

  Python 2.0 added the Distutils, a set of modules for distributing and
installing extensions.  In Python 2.1, the Distutils are used to
compile much of the standard library of extension modules,
autodetecting which ones are supported on the current machine.  It's
hoped that this will make Python installations easier and more
featureful.

  Instead of having to edit the `Modules/Setup' file in order to enable
modules, a `setup.py' script in the top directory of the Python source
distribution is run at build time, and attempts to discover which
modules can be enabled by examining the modules and header files on the
system.  If a module is configured in `Modules/Setup', the `setup.py'
script won't attempt to compile that module and will defer to the
`Modules/Setup' file's contents.  This provides a way to specific any
strange command-line flags or libraries that are required for a
specific platform.

  In another far-reaching change to the build mechanism, Neil
Schemenauer restructured things so Python now uses a single makefile
that isn't recursive, instead of makefiles in the top directory and in
each of the `Python/', `Parser/', `Objects/', and `Modules/'
subdirectories.  This makes building Python faster and also makes
hacking the Makefiles clearer and simpler.

PEP 229 (http://www.python.org/dev/peps/pep-0229) - Using Distutils to Build Python
     Written and implemented by A.M. Kuchling.

1.7.7 PEP 205: Weak References
------------------------------

Weak references, available through the weakref module, are
a minor but useful new data type in the Python programmer's toolbox.

  Storing a reference to an object (say, in a dictionary or a list) has
the side effect of keeping that object alive forever.  There are a few
specific cases where this behaviour is undesirable, object caches being
the most common one, and another being circular references in data
structures such as trees.

  For example, consider a memoizing function that caches the results of
another function `f(x)()' by storing the function's argument and its
result in a dictionary:

    _cache = {}
    def memoize(x):
        if _cache.has_key(x):
            return _cache[x]

        retval = f(x)

        # Cache the returned object
        _cache[x] = retval

        return retval

This version works for simple things such as integers, but it has a
side effect; the `_cache' dictionary holds a reference to the return
values, so they'll never be deallocated until the Python process exits
and cleans up This isn't very noticeable for integers, but if `f()'
returns an object, or a data structure that takes up a lot of memory,
this can be a problem.

  Weak references provide a way to implement a cache that won't keep
objects alive beyond their time.  If an object is only accessible
through weak references, the object will be deallocated and the weak
references will now indicate that the object it referred to no longer
exists.  A weak reference to an object _obj_ is created by calling `wr
= weakref.ref(obj)'.  The object being referred to is returned by
calling the weak reference as if it were a function: `wr()'.  It will
return the referenced object, or `None' if the object no longer exists.

  This makes it possible to write a `memoize()' function whose cache
doesn't keep objects alive, by storing weak references in the cache.

    _cache = {}
    def memoize(x):
        if _cache.has_key(x):
            obj = _cache[x]()
            # If weak reference object still exists,
            # return it
            if obj is not None: return obj

        retval = f(x)

        # Cache a weak reference
        _cache[x] = weakref.ref(retval)

        return retval

The weakref module also allows creating proxy objects which
behave like weak references -- an object referenced only by proxy
objects is deallocated - but instead of requiring an explicit call to
retrieve the object, the proxy transparently forwards all operations to
the object as long as the object still exists.  If the object is
deallocated, attempting to use a proxy will cause a
weakref.ReferenceError exception to be raised.

    proxy = weakref.proxy(obj)
    proxy.attr   # Equivalent to obj.attr
    proxy.meth() # Equivalent to obj.meth()
    del obj
    proxy.attr   # raises weakref.ReferenceError

PEP 205 (http://www.python.org/dev/peps/pep-0205) - Weak References
     Written and implemented by Fred L. Drake, Jr.

1.7.8 PEP 232: Function Attributes
----------------------------------

In Python 2.1, functions can now have arbitrary information attached to
them.  People were often using docstrings to hold information about
functions and methods, because the `__doc__' attribute was the only way
of attaching any information to a function.  For example, in the Zope
Web application server, functions are marked as safe for public access
by having a docstring, and in John Aycock's SPARK parsing framework,
docstrings hold parts of the BNF grammar to be parsed.  This
overloading is unfortunate, since docstrings are really intended to
hold a function's documentation; for example, it means you can't
properly document functions intended for private use in Zope.

  Arbitrary attributes can now be set and retrieved on functions using
the regular Python syntax:

    def f(): pass

    f.publish = 1
    f.secure = 1
    f.grammar = "A ::= B (C D)*"

The dictionary containing attributes can be accessed as the function's
`__dict__'. Unlike the `__dict__' attribute of class instances, in
functions you can actually assign a new dictionary to `__dict__', though
the new value is restricted to a regular Python dictionary; you _can't_
be tricky and set it to a UserDict instance, or any other
random object that behaves like a mapping.

PEP 232 (http://www.python.org/dev/peps/pep-0232) - Function Attributes
     Written and implemented by Barry Warsaw.

1.7.9 PEP 235: Importing Modules on Case-Insensitive Platforms
--------------------------------------------------------------

Some operating systems have filesystems that are case-insensitive,
MacOS and Windows being the primary examples; on these systems, it's
impossible to distinguish the filenames `FILE.PY' and `file.py', even
though they do store the file's name  in its original case (they're
case-preserving, too).

  In Python 2.1, the import statement will work to simulate
case- sensitivity on case-insensitive platforms.  Python will now
search for the first case-sensitive match by default, raising an
ImportError if no such file is found, so `import file' will not
import a module named `FILE.PY'.  Case- insensitive matching can be
requested by setting the PYTHONCASEOK.  environment variable
before starting the Python interpreter.

1.7.10 PEP 217: Interactive Display Hook
----------------------------------------

When using the Python interpreter interactively, the output of commands
is displayed using the built-in repr() function. In Python
2.1, the variable sys.displayhook() can be set to a
callable object which will be called instead of repr(). For
example, you can set it to a special pretty- printing function:

    >>> # Create a recursive data structure
    ... L = [1,2,3]
    >>> L.append(L)
    >>> L # Show Python's default output
    [1, 2, 3, [...]]
    >>> # Use pprint.pprint() as the display function
    ... import sys, pprint
    >>> sys.displayhook = pprint.pprint
    >>> L
    [1, 2, 3,  <Recursion on list with id=135143996>]
    >>>

PEP 217 (http://www.python.org/dev/peps/pep-0217) - Display Hook for Interactive Use
     Written and implemented by Moshe Zadka.

1.7.11 PEP 208: New Coercion Model
----------------------------------

How numeric coercion is done at the C level was significantly modified.
This will only affect the authors of C extensions to Python, allowing
them more flexibility in writing extension types that support numeric
operations.

  Extension types can now set the type flag `Py_TPFLAGS_CHECKTYPES' in
their `PyTypeObject' structure to indicate that they support the new
coercion model.  In such extension types, the numeric slot functions
can no longer assume that they'll be passed two arguments of the same
type; instead they may be passed two arguments of differing types, and
can then perform their own internal coercion.  If the slot function is
passed a type it can't handle, it can indicate the failure by returning
a reference to the `Py_NotImplemented' singleton value.  The numeric
functions of the other type will then be tried, and perhaps they can
handle the operation; if the other type also returns
`Py_NotImplemented', then a TypeError will be raised.
Numeric methods written in Python can also return `Py_NotImplemented',
causing the interpreter to act as if the method did not exist (perhaps
raising a TypeError, perhaps trying another object's numeric
methods).

PEP 208 (http://www.python.org/dev/peps/pep-0208) - Reworking the Coercion Model
     Written and implemented by Neil Schemenauer, heavily based upon
     earlier work by Marc-André Lemburg.  Read this to understand the
     fine points of how numeric operations will now be processed at the
     C level.

1.7.12 PEP 241: Metadata in Python Packages
-------------------------------------------

A common complaint from Python users is that there's no single catalog
of all the Python modules in existence.  T. Middleton's Vaults of
Parnassus at <http://www.vex.net/parnassus/> are the largest catalog of
Python modules, but registering software at the Vaults is optional, and
many people don't bother.

  As a first small step toward fixing the problem, Python software
packaged using the Distutils *sdist* command will include a file named
`PKG-INFO' containing information about the package such as its name,
version, and author (metadata, in cataloguing terminology).  PEP 241
contains the full list of fields that can be present in the `PKG-INFO'
file.  As people began to package their software using Python 2.1, more
and more packages will include metadata, making it possible to build
automated cataloguing systems and experiment with them.  With the
result experience, perhaps it'll be possible to design a really good
catalog and then build support for it into Python 2.2.  For example,
the Distutils *sdist* and *bdist_** commands could support a `upload'
option that would automatically upload your package to a catalog server.

  You can start creating packages containing `PKG-INFO' even if you're
not using Python 2.1, since a new release of the Distutils will be made
for users of earlier Python versions.  Version 1.0.2 of the Distutils
includes the changes described in PEP 241, as well as various bugfixes
and enhancements.  It will be available from  the Distutils SIG at
<http://www.python.org/sigs/distutils-sig/>.

PEP 241 (http://www.python.org/dev/peps/pep-0241) - Metadata for Python Software Packages
     Written and implemented by A.M. Kuchling.

PEP 243 (http://www.python.org/dev/peps/pep-0243) - Module Repository Upload Mechanism
     Written by Sean Reifschneider, this draft PEP describes a proposed
     mechanism for uploading  Python packages to a central server.

1.7.13 New and Improved Modules
-------------------------------

   * Ka-Ping Yee contributed two new modules: `inspect.py', a module for
     getting information about live Python code, and `pydoc.py', a
     module for interactively converting docstrings to HTML or text.
     As a bonus, `Tools/scripts/pydoc', which is now automatically
     installed, uses `pydoc.py' to display documentation given a Python
     module, package, or class name.  For example, `pydoc xml.dom'
     displays the following:

         Python Library Documentation: package xml.dom in xml

         NAME
             xml.dom - W3C Document Object Model implementation for Python.

         FILE
             /usr/local/lib/python2.1/xml/dom/__init__.pyc

         DESCRIPTION
             The Python mapping of the Document Object Model is documented in the
             Python Library Reference in the section on the xml.dom package.

             This package contains the following modules:
               ...

     `pydoc' also includes a Tk-based interactive help browser.
     `pydoc' quickly becomes addictive; try it out!

   * Two different modules for unit testing were added to the standard
     library.  The doctest module, contributed by Tim
     Peters, provides a testing framework based on running embedded
     examples in docstrings and comparing the results against the
     expected output.  PyUnit, contributed by Steve Purcell, is a unit
     testing framework inspired by JUnit, which was in turn an
     adaptation of Kent Beck's Smalltalk testing framework.  See
     <http://pyunit.sourceforge.net/> for more information about PyUnit.

   * The difflib module contains a class, `SequenceMatcher',
     which compares two sequences and computes the changes required to
     transform one sequence into the other.  For example, this module
     can be used to write a tool similar to the Unix *diff* program,
     and in fact the sample program `Tools/scripts/ndiff.py'
     demonstrates how to write such a script.

   * curses.panel, a wrapper for the panel library, part of
     ncurses and of SYSV curses, was contributed by Thomas Gellekum.
     The panel library provides windows with the additional feature of
     depth. Windows can be moved higher or lower in the depth ordering,
     and the panel library figures out where panels overlap and which
     sections are visible.

   * The PyXML package has gone through a few releases since Python
     2.0, and Python 2.1 includes an updated version of the `xml'
     package.  Some of the noteworthy changes include support for Expat
     1.2 and later versions, the ability for Expat parsers to handle
     files in any encoding supported by Python, and various bugfixes
     for SAX, DOM, and the `minidom' module.

   * Ping also contributed another hook for handling uncaught
     exceptions.  sys.excepthook() can be set to a callable
     object.  When an exception isn't caught by any try...
     except blocks, the exception will be passed to
     sys.excepthook(), which can then do whatever it likes.
     At the Ninth Python Conference, Ping demonstrated an application
     for this hook: printing an extended traceback that not only lists
     the stack frames, but also lists the function arguments and the
     local variables for each frame.

   * Various functions in the time module, such as
     `asctime()' and `localtime()', require a floating point argument
     containing the time in seconds since the epoch.  The most common
     use of these functions is to work with the current time, so the
     floating point argument has been made optional; when a value isn't
     provided, the current time will be used.  For example, log file
     entries usually need a string containing the current time; in
     Python 2.1, `time.asctime()' can be used, instead of the lengthier
     `time.asctime(time.localtime(time.time()))' that was previously
     required.

     This change was proposed and implemented by Thomas Wouters.

   * The ftplib module now defaults to retrieving files in
     passive mode, because passive mode is more likely to work from
     behind a firewall.  This request came from the Debian bug tracking
     system, since other Debian packages use ftplib to
     retrieve files and then don't work from behind a firewall.  It's
     deemed unlikely that this will cause problems for anyone, because
     Netscape defaults to passive mode and few people complain, but if
     passive mode is unsuitable for your application or network setup,
     call `set_pasv(0)()' on FTP objects to disable passive mode.

   * Support for raw socket access has been added to the socket
     module, contributed by Grant Edwards.

   * The pstats module now contains a simple interactive
     statistics browser for displaying timing profiles for Python
     programs, invoked when the module is run as a script.  Contributed
     by  Eric S. Raymond.

   * A new implementation-dependent function,
     `sys._getframe([depth])()', has been added to return a given frame
     object from the current call stack.  See sys._getframe().
     returns the frame at the top of the call stack;  if the optional
     integer argument _depth_ is supplied, the function returns the
     frame that is _depth_ calls below the top of the stack.  For
     example, `sys._getframe(1)' returns the caller's frame object.

     This function is only present in CPython, not in Jython or the .NET
     implementation.  Use it for debugging, and resist the temptation
     to put it into production code.

1.7.14 Other Changes and Fixes
------------------------------

There were relatively few smaller changes made in Python 2.1 due to the
shorter release cycle.  A search through the CVS change logs turns up
117 patches applied, and 136 bugs fixed; both figures are likely to be
underestimates.  Some of the more notable changes are:

   * A specialized object allocator is now optionally available, that
     should be faster than the system `malloc()' and have less memory
     overhead.  The allocator uses C's `malloc()' function to get large
     pools of memory, and then fulfills smaller memory requests from
     these pools.  It can be enabled by providing the `--with-pymalloc'
     option to the *configure* script; see `Objects/obmalloc.c' for the
     implementation details.

     Authors of C extension modules should test their code with the
     object allocator enabled, because some incorrect code may break,
     causing core dumps at runtime.  There are a bunch of memory
     allocation functions in Python's C API that have previously been
     just aliases for the C library's `malloc()' and `free()', meaning
     that if you accidentally called mismatched functions, the error
     wouldn't be noticeable.  When the object allocator is enabled,
     these functions aren't aliases of `malloc()' and `free()' any
     more, and calling the wrong function to free memory will get you a
     core dump.  For example, if memory was allocated using
     `PyMem_New()', it has to be freed using `PyMem_Del()', not
     `free()'.  A few modules included with Python fell afoul of this
     and had to be fixed; doubtless there are more third-party modules
     that will have the same problem.

     The object allocator was contributed by Vladimir Marangozov.

   * The speed of line-oriented file I/O has been improved because
     people often complain about its lack of speed, and because it's
     often been used as a naïve benchmark.  The readline().
     method of file objects has therefore been rewritten to be much
     faster.  The exact amount of the speedup will vary from platform
     to platform depending on how slow the C library's `getc()' was, but
     is around 66%, and potentially much faster on some particular
     operating systems.  Tim Peters did much of the benchmarking and
     coding for this change, motivated by a discussion in
     comp.lang.python.

     A new module and method for file objects was also added,
     contributed by Jeff Epler. The new method, `xreadlines()', is
     similar to the existing xrange() built-in.
     `xreadlines()' returns an opaque sequence object that only
     supports being iterated over, reading a line on every iteration but
     not reading the entire file into memory as the existing
     `readlines()' method does. You'd use it like this:

         for line in sys.stdin.xreadlines():
             # ... do something for each line ...
             ...

     For a fuller discussion of the line I/O changes, see the
     python-dev summary for January 1-15, 2001 at
     <http://www.python.org/dev/summary/2001-01-1/>.

   * A new method, `popitem()', was added to dictionaries to enable
     destructively iterating through the contents of a dictionary; this
     can be faster for large dictionaries because there's no need to
     construct a list containing all the keys or values. `D.popitem()'
     removes a random `(key, value)' pair from the dictionary `D' and
     returns it as a 2-tuple.  This was implemented mostly by Tim
     Peters and Guido van Rossum, after a suggestion and preliminary
     patch by Moshe Zadka.

   * Modules can now control which names are imported when `from module
     import *' is used, by defining an `__all__' attribute containing a
     list of names that will be imported.  One common complaint is that
     if the module imports other modules such as sys or
     string, `from module import *' will add them to the
     importing module's namespace.  To fix this, simply list the public
     names in `__all__':

         # List public names
         __all__ = ['Database', 'open']

     A stricter version of this patch was first suggested and
     implemented by Ben Wolfson, but after some python-dev discussion,
     a weaker final version was checked in.

   * Applying repr() to strings previously used octal
     escapes for non-printable characters; for example, a newline was
     `'\012''.  This was a vestigial trace of Python's C ancestry, but
     today octal is of very little practical use.  Ka-Ping Yee
     suggested using hex escapes instead of octal ones, and using the
     `\n', `\t', `\r' escapes for the appropriate characters, and
     implemented this new formatting.

   * Syntax errors detected at compile-time can now raise exceptions
     containing the filename and line number of the error, a pleasant
     side effect of the compiler reorganization done by Jeremy Hylton.

   * C extensions which import other modules have been changed to use
     `PyImport_ImportModule()', which means that they will use any
     import hooks that have been installed.  This is also encouraged
     for third-party extensions that need to import some other module
     from C code.

   * The size of the Unicode character database was shrunk by another
     340K thanks to Fredrik Lundh.

   * Some new ports were contributed: MacOS X (by Steven Majewski),
     Cygwin (by Jason Tishler); RISCOS (by Dietmar Schwertberger);
     Unixware 7  (by Billy G.  Allie).

  And there's the usual list of minor bugfixes, minor memory leaks,
docstring edits, and other tweaks, too lengthy to be worth itemizing;
see the CVS logs for the full details if you want them.

1.7.15 Acknowledgements
-----------------------

The author would like to thank the following people for offering
suggestions on various drafts of this article: Graeme Cross, David
Goodger, Jay Graves, Michael Hudson, Marc-André Lemburg, Fredrik Lundh,
Neil Schemenauer, Thomas Wouters.

1.8 What's New in Python 2.0
============================

     Author: A.M. Kuchling and Moshe Zadka

1.8.1 Introduction
------------------

A new release of Python, version 2.0, was released on October 16, 2000.
This article covers the exciting new features in 2.0, highlights some
other useful changes, and points out a few incompatible changes that
may require rewriting code.

  Python's development never completely stops between releases, and a
steady flow of bug fixes and improvements are always being submitted. A
host of minor fixes, a few optimizations, additional docstrings, and
better error messages went into 2.0; to list them all would be
impossible, but they're certainly significant.  Consult the
publicly-available CVS logs if you want to see the full list.  This
progress is due to the five developers working for  PythonLabs are now
getting paid to spend their days fixing bugs, and also due to the
improved communication resulting  from moving to SourceForge.

1.8.2 What About Python 1.6?
----------------------------

Python 1.6 can be thought of as the Contractual Obligations Python
release.  After the core development team left CNRI in May 2000, CNRI
requested that a 1.6 release be created, containing all the work on
Python that had been performed at CNRI.  Python 1.6 therefore
represents the state of the CVS tree as of May 2000, with the most
significant new feature being Unicode support.  Development continued
after May, of course, so the 1.6 tree received a few fixes to ensure
that it's forward-compatible with Python 2.0.  1.6 is therefore part of
Python's evolution, and not a side branch.

  So, should you take much interest in Python 1.6?  Probably not.  The
1.6final and 2.0beta1 releases were made on the same day (September 5,
2000), the plan being to finalize Python 2.0 within a month or so.  If
you have applications to maintain, there seems little point in breaking
things by moving to 1.6, fixing them, and then having another round of
breakage within a month by moving to 2.0; you're better off just going
straight to 2.0.  Most of the really interesting features described in
this document are only in 2.0, because a lot of work was done between
May and September.

1.8.3 New Development Process
-----------------------------

The most important change in Python 2.0 may not be to the code at all,
but to how Python is developed: in May 2000 the Python developers began
using the tools made available by SourceForge for storing  source code,
tracking bug reports, and managing the queue of patch submissions.  To
report bugs or submit patches for Python 2.0, use the bug tracking and
patch manager tools available from Python's project page, located at
<http://sourceforge.net/projects/python/>.

  The most important of the services now hosted at SourceForge is the
Python CVS tree, the version-controlled repository containing the
source code for Python.  Previously, there were roughly 7 or so people
who had write access to the CVS tree, and all patches had to be
inspected and checked in by one of the people on this short list.
Obviously, this wasn't very scalable.  By moving the CVS tree to
SourceForge, it became possible to grant write access to more people;
as of September 2000 there were 27 people able to check in changes, a
fourfold increase.  This makes possible large-scale changes that
wouldn't be attempted if they'd have to be filtered through the small
group of core developers.  For example, one day Peter Schneider-Kamp
took it into his head to drop K&R C compatibility and convert the C
source for Python to ANSI C. After getting approval on the python-dev
mailing list, he launched into a flurry of checkins that lasted about a
week, other developers joined in to help, and the job was done.  If
there were only 5 people with write access, probably that task would
have been viewed as "nice, but not worth the time and effort needed"
and it would never have gotten done.

  The shift to using SourceForge's services has resulted in a
remarkable increase in the speed of development.  Patches now get
submitted, commented on, revised by people other than the original
submitter, and bounced back and forth between people until the patch is
deemed worth checking in.  Bugs are tracked in one central location and
can be assigned to a specific person for fixing, and we can count the
number of open bugs to measure progress.  This didn't come without a
cost: developers now have more e-mail to deal with, more mailing lists
to follow, and special tools had to be written for the new environment.
For example, SourceForge sends default patch and bug notification
e-mail messages that are completely unhelpful, so Ka-Ping Yee wrote an
HTML screen-scraper that sends more useful messages.

  The ease of adding code caused a few initial growing pains, such as
code was checked in before it was ready or without getting clear
agreement from the developer group.  The approval process that has
emerged is somewhat similar to that used by the Apache group.
Developers can vote +1, +0, -0, or -1 on a patch; +1 and -1 denote
acceptance or rejection, while +0 and -0 mean the developer is mostly
indifferent to the change, though with a slight positive or negative
slant.  The most significant change from the Apache model is that the
voting is essentially advisory, letting Guido van Rossum, who has
Benevolent Dictator For Life status, know what the general opinion is.
He can still ignore the result of a vote, and approve or reject a
change even if the community disagrees with him.

  Producing an actual patch is the last step in adding a new feature,
and is usually easy compared to the earlier task of coming up with a
good design.  Discussions of new features can often explode into
lengthy mailing list threads, making the discussion hard to follow, and
no one can read every posting to python-dev.  Therefore, a relatively
formal process has been set up to write Python Enhancement Proposals
(PEPs), modelled on the Internet RFC process.  PEPs are draft documents
that describe a proposed new feature, and are continually revised until
the community reaches a consensus, either accepting or rejecting the
proposal.  Quoting from the introduction to PEP 1, "PEP Purpose and
Guidelines":

     PEP stands for Python Enhancement Proposal.  A PEP is a design
     document providing information to the Python community, or
     describing a new feature for Python.  The PEP should provide a
     concise technical specification of the feature and a rationale for
     the feature.

     We intend PEPs to be the primary mechanisms for proposing new
     features, for collecting community input on an issue, and for
     documenting the design decisions that have gone into Python.  The
     PEP author is responsible for building consensus within the
     community and documenting dissenting opinions.

  Read the rest of PEP 1 for the details of the PEP editorial process,
style, and format.  PEPs are kept in the Python CVS tree on
SourceForge, though they're not part of the Python 2.0 distribution,
and are also available in HTML form from <http://www.python.org/peps/>.
As of September 2000, there are 25 PEPS, ranging from PEP 201,
"Lockstep Iteration", to PEP 225, "Elementwise/Objectwise Operators".

1.8.4 Unicode
-------------

The largest new feature in Python 2.0 is a new fundamental data type:
Unicode strings.  Unicode uses 16-bit numbers to represent characters
instead of the 8-bit number used by ASCII, meaning that 65,536 distinct
characters can be supported.

  The final interface for Unicode support was arrived at through
countless often- stormy discussions on the python-dev mailing list, and
mostly implemented by Marc-André Lemburg, based on a Unicode string
type implementation by Fredrik Lundh.  A detailed explanation of the
interface was written up as PEP 100
(http://www.python.org/dev/peps/pep-0100), "Python Unicode
Integration". This article will simply cover the most significant
points about the Unicode interfaces.

  In Python source code, Unicode strings are written as `u"string"'.
Arbitrary Unicode characters can be written using a new escape
sequence, `\uHHHH', where _HHHH_ is a 4-digit hexadecimal number from
0000 to FFFF.  The existing `\xHHHH' escape sequence can also be used,
and octal escapes can be used for characters up to U+01FF, which is
represented by `\777'.

  Unicode strings, just like regular strings, are an immutable sequence
type.  They can be indexed and sliced, but not modified in place.
Unicode strings have an `encode( [encoding] )' method that returns an
8-bit string in the desired encoding.  Encodings are named by strings,
such as `'ascii'', `'utf-8'', `'iso-8859-1'', or whatever.  A codec API
is defined for implementing and registering new encodings that are then
available throughout a Python program.  If an encoding isn't specified,
the default encoding is usually 7-bit ASCII, though it can be changed
for your Python installation by calling the
`sys.setdefaultencoding(encoding)()' function in a customised version of
`site.py'.

  Combining 8-bit and Unicode strings always coerces to Unicode, using
the default ASCII encoding; the result of `'a' + u'bc'' is `u'abc''.

  New built-in functions have been added, and existing built-ins
modified to support Unicode:

   * `unichr(ch)' returns a Unicode string 1 character long, containing
     the character _ch_.

   * `ord(u)', where _u_ is a 1-character regular or Unicode string,
     returns the number of the character as an integer.

   * `unicode(string [, encoding]  [, errors] )' creates a Unicode
     string from an 8-bit string.  `encoding' is a string naming the
     encoding to use. The `errors' parameter specifies the treatment of
     characters that are invalid for the current encoding; passing
     `'strict'' as the value causes an exception to be raised on any
     encoding error, while `'ignore'' causes errors to be silently
     ignored and `'replace'' uses U+FFFD, the official replacement
     character, in case of any problems.

   * The exec statement, and various built-ins such as
     `eval()', `getattr()', and `setattr()' will also accept Unicode
     strings as well as regular strings.  (It's possible that the
     process of fixing this missed some built-ins; if you find a
     built-in function that accepts strings but doesn't accept Unicode
     strings at all, please report it as a bug.)

  A new module, see unicodedata, provides an interface to
Unicode character properties.  For example,
`unicodedata.category(u'A')' returns the 2-character string 'Lu', the
'L' denoting it's a letter, and 'u' meaning that it's uppercase.
`unicodedata.bidirectional(u'\u0660')' returns 'AN', meaning that
U+0660 is an Arabic number.

  The codecs module contains functions to look up existing
encodings and register new ones.  Unless you want to implement a new
encoding, you'll most often use the `codecs.lookup(encoding)()'
function, which returns a 4-element tuple: `(encode_func, decode_func,
stream_reader, stream_writer)'.

   * _encode_func_ is a function that takes a Unicode string, and
     returns a 2-tuple `(string, length)'.  _string_ is an 8-bit string
     containing a portion (perhaps all) of the Unicode string converted
     into the given encoding, and _length_ tells you how much of the
     Unicode string was converted.

   * _decode_func_ is the opposite of _encode_func_, taking an 8-bit
     string and returning a 2-tuple `(ustring, length)', consisting of
     the resulting Unicode string _ustring_ and the integer _length_
     telling how much of the 8-bit string was consumed.

   * _stream_reader_ is a class that supports decoding input from a
     stream.  _stream_reader(file_obj)_ returns an object that supports
     the `read()', see readline(), and `readlines()' methods.
     These methods will all translate from the given encoding and
     return Unicode strings.

   * _stream_writer_, similarly, is a class that supports encoding
     output to a stream.  _stream_writer(file_obj)_ returns an object
     that supports the `write()' and `writelines()' methods.  These
     methods expect Unicode strings, translating them to the given
     encoding on output.

  For example, the following code writes a Unicode string into a file,
encoding it as UTF-8:

    import codecs

    unistr = u'\u0660\u2000ab ...'

    (UTF8_encode, UTF8_decode,
     UTF8_streamreader, UTF8_streamwriter) = codecs.lookup('UTF-8')

    output = UTF8_streamwriter( open( '/tmp/output', 'wb') )
    output.write( unistr )
    output.close()

The following code would then read UTF-8 input from the file:

    input = UTF8_streamreader( open( '/tmp/output', 'rb') )
    print repr(input.read())
    input.close()

Unicode-aware regular expressions are available through the re
module, which has a new underlying implementation called SRE
written by Fredrik Lundh of Secret Labs AB.

  A `-U' command line option was added which causes the Python compiler
to interpret all string literals as Unicode string literals. This is
intended to be used in testing and future-proofing your Python code,
since some future version of Python may drop support for 8-bit strings
and provide only Unicode strings.

1.8.5 List Comprehensions
-------------------------

Lists are a workhorse data type in Python, and many programs manipulate
a list at some point.  Two common operations on lists are to loop over
them, and either pick out the elements that meet a certain criterion,
or apply some function to each element.  For example, given a list of
strings, you might want to pull out all the strings containing a given
substring, or strip off trailing whitespace from each line.

  The existing map() and filter() functions can
be used for this purpose, but they require a function as one of their
arguments.  This is fine if there's an existing built-in function that
can be passed directly, but if there isn't, you have to create a little
function to do the required work, and Python's scoping rules make the
result ugly if the little function needs additional information.  Take
the first example in the previous paragraph, finding all the strings in
the list containing a given substring.  You could write the following
to do it:

    # Given the list L, make a list of all strings
    # containing the substring S.
    sublist = filter( lambda s, substring=S:
                         string.find(s, substring) != -1,
                      L)

Because of Python's scoping rules, a default argument is used so that
the anonymous function created by the lambda statement
knows what substring is being searched for.  List comprehensions make
this cleaner:

    sublist = [ s for s in L if string.find(s, S) != -1 ]

List comprehensions have the form:

    [ expression for expr in sequence1
                 for expr2 in sequence2 ...
                 for exprN in sequenceN
                 if condition ]

The for...in clauses contain the sequences to be
iterated over.  The sequences do not have to be the same length,
because they are _not_ iterated over in parallel, but from left to
right; this is explained more clearly in the following paragraphs.  The
elements of the generated list will be the successive values of
_expression_.  The final if clause is optional; if present,
_expression_ is only evaluated and added to the result if _condition_
is true.

  To make the semantics very clear, a list comprehension is equivalent
to the following Python code:

    for expr1 in sequence1:
        for expr2 in sequence2:
        ...
            for exprN in sequenceN:
                 if (condition):
                      # Append the value of
                      # the expression to the
                      # resulting list.

This means that when there are multiple for...See in.
clauses, the resulting list will be equal to the product of the lengths
of all the sequences.  If you have two lists of length 3, the output
list is 9 elements long:

    seq1 = 'abc'
    seq2 = (1,2,3)
    >>> [ (x,y) for x in seq1 for y in seq2]
    [('a', 1), ('a', 2), ('a', 3), ('b', 1), ('b', 2), ('b', 3), ('c', 1),
    ('c', 2), ('c', 3)]

To avoid introducing an ambiguity into Python's grammar, if
_expression_ is creating a tuple, it must be surrounded with
parentheses.  The first list comprehension below is a syntax error,
while the second one is correct:

    # Syntax error
    [ x,y for x in seq1 for y in seq2]
    # Correct
    [ (x,y) for x in seq1 for y in seq2]

The idea of list comprehensions originally comes from the functional
programming language Haskell (<http://www.haskell.org>).  Greg Ewing
argued most effectively for adding them to Python and wrote the initial
list comprehension patch, which was then discussed for a seemingly
endless time on the python-dev mailing list and kept up-to-date by Skip
Montanaro.

1.8.6 Augmented Assignment
--------------------------

Augmented assignment operators, another long-requested feature, have
been added to Python 2.0.  Augmented assignment operators include `+=',
`-=', `*=', and so forth.  For example, the statement `a += 2'
increments the value of the variable  `a' by 2, equivalent to the
slightly lengthier `a = a + 2'.

  The full list of supported assignment operators is `+=', `-=', `*=',
`/=', `%=', `**=', `&=', `|=', `^=', `>>=', and `<<='.  Python classes
can override the augmented assignment operators by defining methods
named __iadd__(), __isub__(), etc.  For example,
the following `Number' class stores a number and supports using += to
create a new instance with an incremented value.

    class Number:
        def __init__(self, value):
            self.value = value
        def __iadd__(self, increment):
            return Number( self.value + increment)

    n = Number(5)
    n += 3
    print n.value

The __iadd__() special method is called with the value of
the increment, and should return a new instance with an appropriately
modified value; this return value is bound as the new value of the
variable on the left-hand side.

  Augmented assignment operators were first introduced in the C
programming language, and most C-derived languages, such as *awk*, C++,
Java, Perl, and PHP also support them.  The augmented assignment patch
was implemented by Thomas Wouters.

1.8.7 String Methods
--------------------

Until now string-manipulation functionality was in the string
module, which was usually a front-end for the `strop' module
written in C.  The addition of Unicode posed a difficulty for the
`strop' module, because the functions would all need to be rewritten in
order to accept either 8-bit or Unicode strings.  For functions such as
string.replace(), which takes 3 string arguments, that means
eight possible permutations, and correspondingly complicated code.

  Instead, Python 2.0 pushes the problem onto the string type, making
string manipulation functionality available through methods on both
8-bit strings and Unicode strings.

    >>> 'andrew'.capitalize()
    'Andrew'
    >>> 'hostname'.replace('os', 'linux')
    'hlinuxtname'
    >>> 'moshe'.find('sh')
    2

One thing that hasn't changed, a noteworthy April Fools' joke
notwithstanding, is that Python strings are immutable. Thus, the string
methods return new strings, and do not modify the string on which they
operate.

  The old string module is still around for backwards
compatibility, but it mostly acts as a front-end to the new string
methods.

  Two methods which have no parallel in pre-2.0 versions, although they
did exist in JPython for quite some time, are `startswith()' and
`endswith()'.  `s.startswith(t)' is equivalent to `s[:len(t)] == t',
while `s.endswith(t)' is equivalent to `s[-len(t):] == t'.

  One other method which deserves special mention is `join()'.  The
`join()' method of a string receives one parameter, a sequence of
strings, and is equivalent to the string.join() function
from the old string.  module, with the arguments reversed.
In other words, `s.join(seq)' is equivalent to the old
`string.join(seq, s)'.

1.8.8 Garbage Collection of Cycles
----------------------------------

The C implementation of Python uses reference counting to implement
garbage collection.  Every Python object maintains a count of the
number of references pointing to itself, and adjusts the count as
references are created or destroyed.  Once the reference count reaches
zero, the object is no longer accessible, since you need to have a
reference to an object to access it, and if the count is zero, no
references exist any longer.

  Reference counting has some pleasant properties: it's easy to
understand and implement, and the resulting implementation is portable,
fairly fast, and reacts well with other libraries that implement their
own memory handling schemes.  The major problem with reference counting
is that it sometimes doesn't realise that objects are no longer
accessible, resulting in a memory leak.  This happens when there are
cycles of references.

  Consider the simplest possible cycle,  a class instance which has a
reference to itself:

    instance = SomeClass()
    instance.myself = instance

After the above two lines of code have been executed, the reference
count of `instance' is 2; one reference is from the variable named
`'instance'', and the other is from the `myself' attribute of the
instance.

  If the next line of code is `del instance', what happens?  The
reference count of `instance' is decreased by 1, so it has a reference
count of 1; the reference in the `myself' attribute still exists.  Yet
the instance is no longer accessible through Python code, and it could
be deleted.  Several objects can participate in a cycle if they have
references to each other, causing all of the objects to be leaked.

  Python 2.0 fixes this problem by periodically executing a cycle
detection algorithm which looks for inaccessible cycles and deletes the
objects involved.  A new gc module provides functions to
perform a garbage collection, obtain debugging statistics, and tuning
the collector's parameters.

  Running the cycle detection algorithm takes some time, and therefore
will result in some additional overhead.  It is hoped that after we've
gotten experience with the cycle collection from using 2.0, Python 2.1
will be able to minimize the overhead with careful tuning.  It's not
yet obvious how much performance is lost, because benchmarking this is
tricky and depends crucially on how often the program creates and
destroys objects.  The detection of cycles can be disabled when Python
is compiled, if you can't afford even a tiny speed penalty or suspect
that the cycle collection is buggy, by specifying the
`--without-cycle-gc' switch when running the *configure* script.

  Several people tackled this problem and contributed to a solution.
An early implementation of the cycle detection approach was written by
Toby Kelsey.  The current algorithm was suggested by Eric Tiedemann
during a visit to CNRI, and Guido van Rossum and Neil Schemenauer wrote
two different implementations, which were later integrated by Neil.
Lots of other people offered suggestions along the way; the March 2000
archives of the python-dev mailing list contain most of the relevant
discussion, especially in the threads titled "Reference cycle
collection for Python" and "Finalization again".

1.8.9 Other Core Changes
------------------------

Various minor changes have been made to Python's syntax and built-in
functions.  None of the changes are very far-reaching, but they're
handy conveniences.

1.8.9.1 Minor Language Changes
..............................

A new syntax makes it more convenient to call a given function with a
tuple of arguments and/or a dictionary of keyword arguments. In Python
1.5 and earlier, you'd use the apply() built-in function:
`apply(f, args, kw)' calls the function `f()' with the argument tuple
_args_ and the keyword arguments in the dictionary _kw_.  See
apply().  is the same in 2.0, but thanks to a patch from Greg
Ewing, `f(*args, **kw)' as a shorter and clearer way to achieve the
same effect.  This syntax is symmetrical with the syntax for defining
functions:

    def f(*args, **kw):
        # args is a tuple of positional args,
        # kw is a dictionary of keyword args
        ...

The print statement can now have its output directed to a
file-like object by following the print with  `>> file',
similar to the redirection operator in Unix shells. Previously you'd
either have to use the `write()' method of the file-like object, which
lacks the convenience and simplicity of print, or you could
assign a new value to `sys.stdout' and then restore the old value.  For
sending output to standard error, it's much easier to write this:

    print >> sys.stderr, "Warning: action field not supplied"

Modules can now be renamed on importing them, using the syntax `import
module as name' or `from module import name as othername'.  The patch
was submitted by Thomas Wouters.

  A new format style is available when using the `%' operator; '%r'
will insert the repr() of its argument.  This was also
added from symmetry considerations, this time for symmetry with the
existing '%s' format style, which inserts the str() of its
argument.  For example, `'%r %s' % ('abc', 'abc')' returns a string
containing `'abc' abc'.

  Previously there was no way to implement a class that overrode
Python's built-in in operator and implemented a custom
version.  `obj in seq' returns true if _obj_ is present in the sequence
_seq_; Python computes this by simply trying every index of the
sequence until either _obj_ is found or an IndexError is
encountered.  Moshe Zadka contributed a patch which adds a
__contains__() magic method for providing a custom implementation
for in. Additionally, new built-in objects written in C can
define what in means for them via a new slot in the
sequence protocol.

  Earlier versions of Python used a recursive algorithm for deleting
objects.  Deeply nested data structures could cause the interpreter to
fill up the C stack and crash; Christian Tismer rewrote the deletion
logic to fix this problem.  On a related note, comparing recursive
objects recursed infinitely and crashed; Jeremy Hylton rewrote the code
to no longer crash, producing a useful result instead.  For example,
after this code:

    a = []
    b = []
    a.append(a)
    b.append(b)

The comparison `a==b' returns true, because the two recursive data
structures are isomorphic. See the thread "trashcan and PR#7" in the
April 2000 archives of the python-dev mailing list for the discussion
leading up to this implementation, and some useful relevant links.
Note that comparisons can now also raise exceptions. In earlier
versions of Python, a comparison operation such as `cmp(a,b)' would
always produce an answer, even if a user-defined __cmp__().
method encountered an error, since the resulting exception would simply
be silently swallowed.

  Work has been done on porting Python to 64-bit Windows on the Itanium
processor, mostly by Trent Mick of ActiveState.  (Confusingly,
`sys.platform' is still `'win32'' on Win64 because it seems that for
ease of porting, MS Visual C++ treats code as 32 bit on Itanium.)
PythonWin also supports Windows CE; see the Python CE page at
<http://pythonce.sourceforge.net/> for more information.

  Another new platform is Darwin/MacOS X; initial support for it is in
Python 2.0.  Dynamic loading works, if you specify "configure
-with-dyld -with-suffix=.x".  Consult the README in the Python source
distribution for more instructions.

  An attempt has been made to alleviate one of Python's warts, the
often-confusing NameError exception when code refers to a
local variable before the variable has been assigned a value.  For
example, the following code raises an exception on the print
statement in both 1.5.2 and 2.0; in 1.5.2 a NameError.
exception is raised, while 2.0 raises a new UnboundLocalError
exception. UnboundLocalError is a subclass of
NameError, so any existing code that expects NameError.
to be raised should still work.

    def f():
        print "i=",i
        i = i + 1
    f()

Two new exceptions, `TabError' and `IndentationError', have been
introduced.  They're both subclasses of SyntaxError, and are
raised when Python code is found to be improperly indented.

1.8.9.2 Changes to Built-in Functions
.....................................

A new built-in, `zip(seq1, seq2, ...)()', has been added.  See zip().
returns a list of tuples where each tuple contains the i-th
element from each of the argument sequences.  The difference between
zip() and `map(None, seq1, seq2)' is that map().
pads the sequences with `None' if the sequences aren't all of the same
length, while zip() truncates the returned list to the
length of the shortest argument sequence.

  The int() and long() functions now accept an
optional "base" parameter when the first argument is a string.
`int('123', 10)' returns 123, while `int('123', 16)' returns 291.
`int(123, 16)' raises a TypeError exception with the
message "can't convert non-string with explicit base".

  A new variable holding more detailed version information has been
added to the sys module.  `sys.version_info' is a tuple
`(major, minor, micro, level, serial)' For example, in a hypothetical
2.0.1beta1, `sys.version_info' would be `(2, 0, 1, 'beta', 1)'. _level_
is a string such as `"alpha"', `"beta"', or `"final"' for a final
release.

  Dictionaries have an odd new method, `setdefault(key, default)()',
which behaves similarly to the existing `get()' method.  However, if
the key is missing, `setdefault()' both returns the value of _default_
as `get()' would do, and also inserts it into the dictionary as the
value for _key_.  Thus, the following lines of code:

    if dict.has_key( key ): return dict[key]
    else:
        dict[key] = []
        return dict[key]

can be reduced to a single `return dict.setdefault(key, [])' statement.

  The interpreter sets a maximum recursion depth in order to catch
runaway recursion before filling the C stack and causing a core dump or
GPF..  Previously this limit was fixed when you compiled Python, but in
2.0 the maximum recursion depth can be read and modified using
sys.getrecursionlimit() and sys.setrecursionlimit().
The default value is 1000, and a rough maximum value for a given
platform can be found by running a new script,
`Misc/find_recursionlimit.py'.

1.8.10 Porting to 2.0
---------------------

New Python releases try hard to be compatible with previous releases,
and the record has been pretty good.  However, some changes are
considered useful enough, usually because they fix initial design
decisions that turned out to be actively mistaken, that breaking
backward compatibility can't always be avoided.  This section lists the
changes in Python 2.0 that may cause old Python code to break.

  The change which will probably break the most code is tightening up
the arguments accepted by some methods.  Some methods would take
multiple arguments and treat them as a tuple, particularly various list
methods such as `append()' and `insert()'. In earlier versions of
Python, if `L' is a list, `L.append( 1,2 )' appends the tuple `(1,2)'
to the list.  In Python 2.0 this causes a TypeError.
exception to be raised, with the message: 'append requires exactly 1
argument; 2 given'.  The fix is to simply add an extra set of
parentheses to pass both values as a tuple:  `L.append( (1,2) )'.

  The earlier versions of these methods were more forgiving because
they used an old function in Python's C interface to parse their
arguments; 2.0 modernizes them to use `PyArg_ParseTuple()', the current
argument parsing function, which provides more helpful error messages
and treats multi-argument calls as errors.  If you absolutely must use
2.0 but can't fix your code, you can edit `Objects/listobject.c' and
define the preprocessor symbol `NO_STRICT_LIST_APPEND' to preserve the
old behaviour; this isn't recommended.

  Some of the functions in the socket module are still
forgiving in this way.  For example, `socket.connect( ('hostname', 25)
)()' is the correct form, passing a tuple representing an IP address,
but `socket.connect( 'hostname', 25 )()' also works.
`socket.connect_ex()' and `socket.bind()' are similarly easy-going.
2.0alpha1 tightened these functions up, but because the documentation
actually used the erroneous multiple argument form, many people wrote
code which would break with the stricter checking.  GvR backed out the
changes in the face of public reaction, so for the socket.
module, the documentation was fixed and the multiple argument form is
simply marked as deprecated; it _will_ be tightened up again in a
future Python version.

  The `\x' escape in string literals now takes exactly 2 hex digits.
Previously it would consume all the hex digits following the 'x' and
take the lowest 8 bits of the result, so `\x123456' was equivalent to
`\x56'.

  The AttributeError and NameError exceptions
have a more friendly error message, whose text will be something like
`'Spam' instance has no attribute 'eggs'' or `name 'eggs' is not
defined'.  Previously the error message was just the missing attribute
name `eggs', and code written to take advantage of this fact will break
in 2.0.

  Some work has been done to make integers and long integers a bit more
interchangeable.  In 1.5.2, large-file support was added for Solaris,
to allow reading files larger than 2 GiB; this made the `tell()' method
of file objects return a long integer instead of a regular integer.
Some code would subtract two file offsets and attempt to use the result
to multiply a sequence or slice a string, but this raised a
TypeError.  In 2.0, long integers can be used to multiply or slice
a sequence, and it'll behave as you'd intuitively expect it to; `3L *
'abc'' produces 'abcabcabc', and `(0,1,2,3)[2L:4L]' produces (2,3).
Long integers can also be used in various contexts where previously
only integers were accepted, such as in the `seek()' method of file
objects, and in the formats supported by the `%' operator (`%d', `%i',
`%x', etc.).  For example, `"%d" % 2L**64' will produce the string
`18446744073709551616'.

  The subtlest long integer change of all is that the str().
of a long integer no longer has a trailing 'L' character, though
repr() still includes it.  The 'L' annoyed many people who wanted
to print long integers that looked just like regular integers, since
they had to go out of their way to chop off the character.  This is no
longer a problem in 2.0, but code which does `str(longval)[:-1]' and
assumes the 'L' is there, will now lose the final digit.

  Taking the repr() of a float now uses a different
formatting precision than str().  repr() uses
`%.17g' format string for C's `sprintf()', while str() uses
`%.12g' as before.  The effect is that repr() may
occasionally show more decimal places than  str(), for
certain numbers.  For example, the number 8.1 can't be represented
exactly in binary, so `repr(8.1)' is `'8.0999999999999996'', while
str(8.1) is `'8.1''.

  The `-X' command-line option, which turned all standard exceptions
into strings instead of classes, has been removed; the standard
exceptions will now always be classes.  The exceptions.
module containing the standard exceptions was translated from Python to
a built-in C module, written by Barry Warsaw and Fredrik Lundh.

1.8.11 Extending/Embedding Changes
----------------------------------

Some of the changes are under the covers, and will only be apparent to
people writing C extension modules or embedding a Python interpreter in
a larger application.  If you aren't dealing with Python's C API, you
can safely skip this section.

  The version number of the Python C API was incremented, so C
extensions compiled for 1.5.2 must be recompiled in order to work with
2.0.  On Windows, it's not possible for Python 2.0 to import a third
party extension built for Python 1.5.x due to how Windows DLLs work, so
Python will raise an exception and the import will fail.

  Users of Jim Fulton's ExtensionClass module will be pleased to find
out that hooks have been added so that ExtensionClasses are now
supported by isinstance() and issubclass(). This
means you no longer have to remember to write code such as `if
type(obj) == myExtensionClass', but can use the more natural `if
isinstance(obj, myExtensionClass)'.

  The `Python/importdl.c' file, which was a mass of #ifdefs to support
dynamic loading on many different platforms, was cleaned up and
reorganised by Greg Stein.  `importdl.c' is now quite small, and
platform-specific code has been moved into a bunch of
`Python/dynload_*.c' files.  Another cleanup: there were also a number
of `my*.h' files in the Include/ directory that held various
portability hacks; they've been merged into a single file,
`Include/pyport.h'.

  Vladimir Marangozov's long-awaited malloc restructuring was
completed, to make it easy to have the Python interpreter use a custom
allocator instead of C's standard `malloc()'.  For documentation, read
the comments in `Include/pymem.h' and `Include/objimpl.h'.  For the
lengthy discussions during which the interface was hammered out, see
the Web archives of the 'patches' and 'python-dev' lists at python.org.

  Recent versions of the GUSI development environment for MacOS support
POSIX threads.  Therefore, Python's POSIX threading support now works
on the Macintosh.  Threading support using the user-space GNU `pth'
library was also contributed.

  Threading support on Windows was enhanced, too.  Windows supports
thread locks that use kernel objects only in case of contention; in the
common case when there's no contention, they use simpler functions
which are an order of magnitude faster.  A threaded version of Python
1.5.2 on NT is twice as slow as an unthreaded version; with the 2.0
changes, the difference is only 10%.  These improvements were
contributed by Yakov Markovitch.

  Python 2.0's source now uses only ANSI C prototypes, so compiling
Python now requires an ANSI C compiler, and can no longer be done using
a compiler that only supports K&R C.

  Previously the Python virtual machine used 16-bit numbers in its
bytecode, limiting the size of source files.  In particular, this
affected the maximum size of literal lists and dictionaries in Python
source; occasionally people who are generating Python code would run
into this limit.  A patch by Charles G.  Waldman raises the limit from
`2^16' to `2^{32}'.

  Three new convenience functions intended for adding constants to a
module's dictionary at module initialization time were added:
`PyModule_AddObject()', `PyModule_AddIntConstant()', and
`PyModule_AddStringConstant()'.  Each of these functions takes a module
object, a null-terminated C string containing the name to be added, and
a third argument for the value to be assigned to the name.  This third
argument is, respectively, a Python object, a C long, or a C string.

  A wrapper API was added for Unix-style signal handlers.
`PyOS_getsig()' gets a signal handler and `PyOS_setsig()' will set a
new handler.

1.8.12 Distutils: Making Modules Easy to Install
------------------------------------------------

Before Python 2.0, installing modules was a tedious affair - there was
no way to figure out automatically where Python is installed, or what
compiler options to use for extension modules.  Software authors had to
go through an arduous ritual of editing Makefiles and configuration
files, which only really work on Unix and leave Windows and MacOS
unsupported.  Python users faced wildly differing installation
instructions which varied between different extension packages, which
made administering a Python installation something of  a chore.

  The SIG for distribution utilities, shepherded by Greg Ward, has
created the Distutils, a system to make package installation much
easier.  They form the distutils package, a new part of
Python's standard library. In the best case, installing a Python module
from source will require the same steps: first you simply mean unpack
the tarball or zip archive, and the run "`python setup.py install'".
The platform will be automatically detected, the compiler will be
recognized, C extension modules will be compiled, and the distribution
installed into the proper directory.  Optional command-line arguments
provide more control over the installation process, the distutils
package offers many places to override defaults - separating the build
from the install, building or installing in non-default directories,
and more.

  In order to use the Distutils, you need to write a `setup.py' script.
For the simple case, when the software contains only .py files, a
minimal `setup.py' can be just a few lines long:

    from distutils.core import setup
    setup (name = "foo", version = "1.0",
           py_modules = ["module1", "module2"])

The `setup.py' file isn't much more complicated if the software consists
of a few packages:

    from distutils.core import setup
    setup (name = "foo", version = "1.0",
           packages = ["package", "package.subpackage"])

A C extension can be the most complicated case; here's an example taken
from the PyXML package:

    from distutils.core import setup, Extension

    expat_extension = Extension('xml.parsers.pyexpat',
         define_macros = [('XML_NS', None)],
         include_dirs = [ 'extensions/expat/xmltok',
                          'extensions/expat/xmlparse' ],
         sources = [ 'extensions/pyexpat.c',
                     'extensions/expat/xmltok/xmltok.c',
                     'extensions/expat/xmltok/xmlrole.c', ]
           )
    setup (name = "PyXML", version = "0.5.4",
           ext_modules =[ expat_extension ] )

The Distutils can also take care of creating source and binary
distributions.  The "sdist" command, run by "`python setup.py sdist'',
builds a source distribution such as `foo-1.0.tar.gz'. Adding new
commands isn't difficult, "bdist_rpm" and "bdist_wininst" commands have
already been contributed to create an RPM distribution and a Windows
installer for the software, respectively.  Commands to create other
distribution formats such as Debian packages and Solaris `.pkg' files
are in various stages of development.

  All this is documented in a new manual, _Distributing Python
Modules_, that joins the basic set of Python documentation.

1.8.13 XML Modules
------------------

Python 1.5.2 included a simple XML parser in the form of the `xmllib'
module, contributed by Sjoerd Mullender.  Since 1.5.2's release, two
different interfaces for processing XML have become common: SAX2
(version 2 of the Simple API for XML) provides an event-driven
interface with some similarities to `xmllib', and the DOM (Document
Object Model) provides a tree-based interface, transforming an XML
document into a tree of nodes that can be traversed and modified.
Python 2.0 includes a SAX2 interface and a stripped- down DOM interface
as part of the `xml' package. Here we will give a brief overview of
these new interfaces; consult the Python documentation or the source
code for complete details. The Python XML SIG is also working on
improved documentation.

1.8.13.1 SAX2 Support
.....................

SAX defines an event-driven interface for parsing XML.  To use SAX, you
must write a SAX handler class.  Handler classes inherit from various
classes provided by SAX, and override various methods that will then be
called by the XML parser.  For example, the `startElement()' and
`endElement()' methods are called for every starting and end tag
encountered by the parser, the `characters()' method is called for
every chunk of character data, and so forth.

  The advantage of the event-driven approach is that the whole document
doesn't have to be resident in memory at any one time, which matters if
you are processing really huge documents.  However, writing the SAX
handler class can get very complicated if you're trying to modify the
document structure in some elaborate way.

  For example, this little example program defines a handler that
prints a message for every starting and ending tag, and then parses the
file `hamlet.xml' using it:

    from xml import sax

    class SimpleHandler(sax.ContentHandler):
        def startElement(self, name, attrs):
            print 'Start of element:', name, attrs.keys()

        def endElement(self, name):
            print 'End of element:', name

    # Create a parser object
    parser = sax.make_parser()

    # Tell it what handler to use
    handler = SimpleHandler()
    parser.setContentHandler( handler )

    # Parse a file!
    parser.parse( 'hamlet.xml' )

For more information, consult the Python documentation, or the XML
HOWTO at <http://pyxml.sourceforge.net/topics/howto/xml-howto.html>.

1.8.13.2 DOM Support
....................

The Document Object Model is a tree-based representation for an XML
document.  A top-level `Document' instance is the root of the tree, and
has a single child which is the top-level `Element' instance. This
`Element' has children nodes representing character data and any
sub-elements, which may have further children of their own, and so
forth.  Using the DOM you can traverse the resulting tree any way you
like, access element and attribute values, insert and delete nodes, and
convert the tree back into XML.

  The DOM is useful for modifying XML documents, because you can create
a DOM tree, modify it by adding new nodes or rearranging subtrees, and
then produce a new XML document as output.  You can also construct a
DOM tree manually and convert it to XML, which can be a more flexible
way of producing XML output than simply writing `<tag1>'...`</tag1>' to
a file.

  The DOM implementation included with Python lives in the
xml.dom.minidom.  module.  It's a lightweight implementation of
the Level 1 DOM with support for XML namespaces.  The  `parse()' and
`parseString()' convenience functions are provided for generating a DOM
tree:

    from xml.dom import minidom
    doc = minidom.parse('hamlet.xml')

`doc' is a `Document' instance.  `Document', like all the other DOM
classes such as `Element' and `Text', is a subclass of the `Node' base
class.  All the nodes in a DOM tree therefore support certain common
methods, such as `toxml()' which returns a string containing the XML
representation of the node and its children.  Each class also has
special methods of its own; for example, `Element' and `Document'
instances have a method to find all child elements with a given tag
name.  Continuing from the previous 2-line example:

    perslist = doc.getElementsByTagName( 'PERSONA' )
    print perslist[0].toxml()
    print perslist[1].toxml()

For the _Hamlet_ XML file, the above few lines output:

    <PERSONA>CLAUDIUS, king of Denmark. </PERSONA>
    <PERSONA>HAMLET, son to the late, and nephew to the present king.</PERSONA>

The root element of the document is available as `doc.documentElement',
and its children can be easily modified by deleting, adding, or
removing nodes:

    root = doc.documentElement

    # Remove the first child
    root.removeChild( root.childNodes[0] )

    # Move the new first child to the end
    root.appendChild( root.childNodes[0] )

    # Insert the new first child (originally,
    # the third child) before the 20th child.
    root.insertBefore( root.childNodes[0], root.childNodes[20] )

Again, I will refer you to the Python documentation for a complete
listing of the different `Node' classes and their various methods.

1.8.13.3 Relationship to PyXML
..............................

The XML Special Interest Group has been working on XML-related Python
code for a while.  Its code distribution, called PyXML, is available
from the SIG's Web pages at <http://www.python.org/sigs/xml-sig/>. The
PyXML distribution also used the package name `xml'.  If you've written
programs that used PyXML, you're probably wondering about its
compatibility with the 2.0 `xml' package.

  The answer is that Python 2.0's `xml' package isn't compatible with
PyXML, but can be made compatible by installing a recent version PyXML.
Many applications can get by with the XML support that is included with
Python 2.0, but more complicated applications will require that the
full PyXML package will be installed.  When installed, PyXML versions
0.6.0 or greater will replace the `xml' package shipped with Python,
and will be a strict superset of the standard package, adding a bunch
of additional features.  Some of the additional features in PyXML
include:

   * 4DOM, a full DOM implementation from FourThought, Inc.

   * The xmlproc validating parser, written by Lars Marius Garshol.

   * The `sgmlop' parser accelerator module, written by Fredrik Lundh.

1.8.14 Module changes
---------------------

Lots of improvements and bugfixes were made to Python's extensive
standard library; some of the affected modules include readline,
see ConfigParser, cgi, calendar,
posix, readline, `xmllib', see aifc, `chunk,
wave', see random, shelve, and nntplib.
Consult the CVS logs for the exact patch-by-patch details.

  Brian Gallew contributed OpenSSL support for the socket.
module.  OpenSSL is an implementation of the Secure Socket Layer, which
encrypts the data being sent over a socket.  When compiling Python, you
can edit `Modules/Setup' to include SSL support, which adds an
additional function to the socket.  module:
`socket.ssl(socket, keyfile, certfile)()', which takes a socket object
and returns an SSL socket.  The httplib and urllib
modules were also changed to support `https://' URLs, though no
one has implemented FTP or SMTP over SSL.

  The httplib module has been rewritten by Greg Stein to
support HTTP/1.1.  Backward compatibility with the 1.5 version of
httplib is provided, though using HTTP/1.1 features such as
pipelining will require rewriting code to use a different set of
interfaces.

  The Tkinter module now supports Tcl/Tk version 8.1, 8.2,
or 8.3, and support for the older 7.x versions has been dropped.  The
Tkinter module now supports displaying Unicode strings in Tk widgets.
Also, Fredrik Lundh contributed an optimization which makes operations
like `create_line' and `create_polygon' much faster, especially when
using lots of coordinates.

  The curses module has been greatly extended, starting from
Oliver Andrich's enhanced version, to provide many additional functions
from ncurses and SYSV curses, such as colour, alternative character set
support, pads, and mouse support.  This means the module is no longer
compatible with operating systems that only have BSD curses, but there
don't seem to be any currently maintained OSes that fall into this
category.

  As mentioned in the earlier discussion of 2.0's Unicode support, the
underlying implementation of the regular expressions provided by the
re module has been changed.  SRE, a new regular expression
engine written by Fredrik Lundh and partially funded by Hewlett
Packard, supports matching against both 8-bit strings and Unicode
strings.

1.8.15 New modules
------------------

A number of new modules were added.  We'll simply list them with brief
descriptions; consult the 2.0 documentation for the details of a
particular module.

   * atexit.:  For registering functions to be called before
     the Python interpreter exits. Code that currently sets
     `sys.exitfunc' directly should be changed to  use the
     atexit module instead, importing atexit and
     calling atexit.register() with  the function to be
     called on exit.  (Contributed by Skip Montanaro.)

   * codecs, `encodings', see unicodedata.:  Added as
     part of the new Unicode support.

   * filecmp.: Supersedes the old cmp, `cmpcache'
     and `dircmp' modules, which have now become deprecated.
     (Contributed by Gordon MacMillan and Moshe Zadka.)

   * gettext.: This module provides internationalization
     (I18N) and localization (L10N) support for Python programs by
     providing an interface to the GNU gettext message catalog library.
     (Integrated by Barry Warsaw, from separate contributions by Martin
     von Löwis, Peter Funk, and James Henstridge.)

   * `linuxaudiodev': Support for the `/dev/audio' device on Linux, a
     twin to the existing sunaudiodev module. (Contributed
     by Peter Bosch, with fixes by Jeremy Hylton.)

   * mmap.: An interface to memory-mapped files on both
     Windows and Unix.  A file's contents can be mapped directly into
     memory, at which point it behaves like a mutable string, so its
     contents can be read and modified.  They can even be passed to
     functions that expect ordinary strings, such as the re.
     module. (Contributed by Sam Rushing, with some extensions by A.M.
     Kuchling.)

   * `pyexpat': An interface to the Expat XML parser. (Contributed by
     Paul Prescod.)

   * robotparser.: Parse a `robots.txt' file, which is used
     for writing Web spiders that politely avoid certain areas of a Web
     site.  The parser accepts the contents of a `robots.txt' file,
     builds a set of rules from it, and can then answer questions about
     the fetchability of a given URL.  (Contributed by Skip Montanaro.)

   * tabnanny.: A module/script to  check Python source code
     for ambiguous indentation. (Contributed by Tim Peters.)

   * UserString.: A base class useful for deriving objects
     that behave like strings.

   * webbrowser.: A module that provides a platform
     independent way to launch a web browser on a specific URL. For
     each platform, various browsers are tried in a specific order. The
     user can alter which browser is launched by setting the _BROWSER_
     environment variable.  (Originally inspired by Eric S. Raymond's
     patch to urllib which added similar functionality, but
     the final module comes from code originally  implemented by Fred
     Drake as `Tools/idle/BrowserControl.py', and adapted for the
     standard library by Fred.)

   * _winreg.: An interface to the Windows registry.
     _winreg is an adaptation of functions that have been part of
     PythonWin since 1995, but has now been added to the core
     distribution, and enhanced to support Unicode.  See _winreg.
     was written by Bill Tutt and Mark Hammond.

   * zipfile.: A module for reading and writing ZIP-format
     archives.  These are archives produced by *PKZIP* on DOS/Windows
     or *zip* on Unix, not to be confused with *gzip*-format files
     (which are supported by the gzip module) (Contributed
     by James C. Ahlstrom.)

   * imputil.: A module that provides a simpler way for
     writing customised import hooks, in comparison to the existing
     `ihooks' module.  (Implemented by Greg Stein, with much discussion
     on python-dev along the way.)

1.8.16 IDLE Improvements
------------------------

IDLE is the official Python cross-platform IDE, written using Tkinter.
Python 2.0 includes IDLE 0.6, which adds a number of new features and
improvements.  A partial list:

   * UI improvements and optimizations, especially in the area of syntax
     highlighting and auto-indentation.

   * The class browser now shows more information, such as the top
     level functions in a module.

   * Tab width is now a user settable option. When opening an existing
     Python file, IDLE automatically detects the indentation
     conventions, and adapts.

   * There is now support for calling browsers on various platforms,
     used to open the Python documentation in a browser.

   * IDLE now has a command line, which is largely similar to  the
     vanilla Python interpreter.

   * Call tips were added in many places.

   * IDLE can now be installed as a package.

   * In the editor window, there is now a line/column bar at the bottom.

   * Three new keystroke commands: Check module (Alt-F5), Import module
     (F5) and Run script (Ctrl-F5).

1.8.17 Deleted and Deprecated Modules
-------------------------------------

A few modules have been dropped because they're obsolete, or because
there are now better ways to do the same thing.  The `stdwin' module is
gone; it was for a platform-independent windowing toolkit that's no
longer developed.

  A number of modules have been moved to the `lib-old' subdirectory:
cmp, `cmpcache', `dircmp', `dump',  `find', `grep',
`packmail',  `poly', `util', `whatsound', `zmod'.  If you have code
which relies on a module  that's been moved to `lib-old', you can
simply add that directory to `sys.path'   to get them back, but you're
encouraged to update any code that uses these modules.

1.8.18 Acknowledgements
-----------------------

The authors would like to thank the following people for offering
suggestions on various drafts of this article: David Bolen, Mark
Hammond, Gregg Hauser, Jeremy Hylton, Fredrik Lundh, Detlef Lannert,
Aahz Maruch, Skip Montanaro, Vladimir Marangozov, Tobias Polzin, Guido
van Rossum, Neil Schemenauer, and Russ Schmidt.

2 The Python Tutorial
*********************

     Release: 2.7

     Date: February 08, 2011

  Python is an easy to learn, powerful programming language. It has
efficient high-level data structures and a simple but effective
approach to object-oriented programming. Python's elegant syntax and
dynamic typing, together with its interpreted nature, make it an ideal
language for scripting and rapid application development in many areas
on most platforms.

  The Python interpreter and the extensive standard library are freely
available in source or binary form for all major platforms from the
Python Web site, <http://www.python.org/>, and may be freely
distributed. The same site also contains distributions of and pointers
to many free third party Python modules, programs and tools, and
additional documentation.

  The Python interpreter is easily extended with new functions and data
types implemented in C or C++ (or other languages callable from C).
Python is also suitable as an extension language for customizable
applications.

  This tutorial introduces the reader informally to the basic concepts
and features of the Python language and system. It helps to have a
Python interpreter handy for hands-on experience, but all examples are
self-contained, so the tutorial can be read off-line as well.

  For a description of standard objects and modules, see The Python Standard Library.
The Python Language Reference
gives a more formal definition of the language.  To write
extensions in C or C++, read Extending and Embedding the Python Interpreter
and Python/C API Reference Manual. There
are also several books covering Python in depth.

  This tutorial does not attempt to be comprehensive and cover every
single feature, or even every commonly used feature. Instead, it
introduces many of Python's most noteworthy features, and will give you
a good idea of the language's flavor and style. After reading it, you
will be able to read and write Python modules and programs, and you
will be ready to learn more about the various Python library modules
described in The Python Standard Library.

  The Glossary is also worth going through.

2.1 Whetting Your Appetite
==========================

If you do much work on computers, eventually you find that there's some
task you'd like to automate.  For example, you may wish to perform a
search-and-replace over a large number of text files, or rename and
rearrange a bunch of photo files in a complicated way. Perhaps you'd
like to write a small custom database, or a specialized GUI
application, or a simple game.

  If you're a professional software developer, you may have to work
with several C/C++/Java libraries but find the usual
write/compile/test/re-compile cycle is too slow.  Perhaps you're
writing a test suite for such a library and find writing the testing
code a tedious task.  Or maybe you've written a program that could use
an extension language, and you don't want to design and implement a
whole new language for your application.

  Python is just the language for you.

  You could write a Unix shell script or Windows batch files for some
of these tasks, but shell scripts are best at moving around files and
changing text data, not well-suited for GUI applications or games. You
could write a C/C++/Java program, but it can take a lot of development
time to get even a first-draft program.  Python is simpler to use,
available on Windows, Mac OS X, and Unix operating systems, and will
help you get the job done more quickly.

  Python is simple to use, but it is a real programming language,
offering much more structure and support for large programs than shell
scripts or batch files can offer.  On the other hand, Python also
offers much more error checking than C, and, being a _very-high-level
language_, it has high-level data types built in, such as flexible
arrays and dictionaries.  Because of its more general data types Python
is applicable to a much larger problem domain than Awk or even Perl,
yet many things are at least as easy in Python as in those languages.

  Python allows you to split your program into modules that can be
reused in other Python programs.  It comes with a large collection of
standard modules that you can use as the basis of your programs -- or
as examples to start learning to program in Python.  Some of these
modules provide things like file I/O, system calls, sockets, and even
interfaces to graphical user interface toolkits like Tk.

  Python is an interpreted language, which can save you considerable
time during program development because no compilation and linking is
necessary.  The interpreter can be used interactively, which makes it
easy to experiment with features of the language, to write throw-away
programs, or to test functions during bottom-up program development. It
is also a handy desk calculator.

  Python enables programs to be written compactly and readably.
Programs written in Python are typically much shorter than equivalent
C,  C++, or Java programs, for several reasons:

   * the high-level data types allow you to express complex operations
     in a single statement;

   * statement grouping is done by indentation instead of beginning and
     ending brackets;

   * no variable or argument declarations are necessary.

  Python is _extensible_: if you know how to program in C it is easy to
add a new built-in function or module to the interpreter, either to
perform critical operations at maximum speed, or to link Python
programs to libraries that may only be available in binary form (such
as a vendor-specific graphics library).  Once you are really hooked,
you can link the Python interpreter into an application written in C
and use it as an extension or command language for that application.

  By the way, the language is named after the BBC show "Monty Python's
Flying Circus" and has nothing to do with reptiles.  Making references
to Monty Python skits in documentation is not only allowed, it is
encouraged!

  Now that you are all excited about Python, you'll want to examine it
in some more detail.  Since the best way to learn a language is to use
it, the tutorial invites you to play with the Python interpreter as you
read.

  In the next chapter, the mechanics of using the interpreter are
explained.  This is rather mundane information, but essential for
trying out the examples shown later.

  The rest of the tutorial introduces various features of the Python
language and system through examples, beginning with simple
expressions, statements and data types, through functions and modules,
and finally touching upon advanced concepts like exceptions and
user-defined classes.

2.2 Using the Python Interpreter
================================

2.2.1 Invoking the Interpreter
------------------------------

The Python interpreter is usually installed as `/usr/local/bin/python'
on those machines where it is available; putting `/usr/local/bin' in
your Unix shell's search path makes it possible to start it by typing
the command

    python

to the shell.  Since the choice of the directory where the interpreter
lives is an installation option, other places are possible; check with
your local Python guru or system administrator.  (E.g.,
`/usr/local/python' is a popular alternative location.)

  On Windows machines, the Python installation is usually placed in
`C:\Python27', though you can change this when you're running the
installer.  To add this directory to your path,  you can type the
following command into the command prompt in a DOS box:

    set path=%path%;C:\python27

Typing an end-of-file character (`Control-D' on Unix, `Control-Z' on
Windows) at the primary prompt causes the interpreter to exit with a
zero exit status.  If that doesn't work, you can exit the interpreter
by typing the following command: `quit()'.

  The interpreter's line-editing features usually aren't very
sophisticated.  On Unix, whoever installed the interpreter may have
enabled support for the GNU readline library, which adds more elaborate
interactive editing and history features. Perhaps the quickest check to
see whether command line editing is supported is typing Control-P to
the first Python prompt you get.  If it beeps, you have command line
editing; see Appendix Interactive Input Editing and History Substitution
for an introduction to the keys.  If nothing appears
to happen, or if `^P' is echoed, command line editing isn't available;
you'll only be able to use backspace to remove characters from the
current line.

  The interpreter operates somewhat like the Unix shell: when called
with standard input connected to a tty device, it reads and executes
commands interactively; when called with a file name argument or with a
file as standard input, it reads and executes a _script_ from that file.

  A second way of starting the interpreter is `python -c command [arg]
...', which executes the statement(s) in _command_, analogous to the
shell's -c option.  Since Python statements often contain
spaces or other characters that are special to the shell, it is usually
advised to quote _command_ in its entirety with single quotes.

  Some Python modules are also useful as scripts.  These can be invoked
using `python -m module [arg] ...', which executes the source file for
_module_ as if you had spelled out its full name on the command line.

  Note that there is a difference between `python file' and `python
<file'.  In the latter case, input requests from the program, such as
calls to input() and raw_input(), are satisfied
from _file_.  Since this file has already been read until the end by
the parser before the program starts executing, the program will
encounter end-of-file immediately.  In the former case (which is
usually what you want) they are satisfied from whatever file or device
is connected to standard input of the Python interpreter.

  When a script file is used, it is sometimes useful to be able to run
the script and enter interactive mode afterwards.  This can be done by
passing -i.  before the script.  (This does not work if the
script is read from standard input, for the same reason as explained in
the previous paragraph.)

2.2.1.1 Argument Passing
........................

When known to the interpreter, the script name and additional arguments
thereafter are passed to the script in the variable `sys.argv', which
is a list of strings.  Its length is at least one; when no script and
no arguments are given, `sys.argv[0]' is an empty string.  When the
script name is given as `'-'' (meaning  standard input), `sys.argv[0]'
is set to `'-''.  When -c. _command_ is used, `sys.argv[0]'
is set to `'-c''.  When -m. _module_ is used, `sys.argv[0]'
is set to the full name of the located module.  Options found after
-c. _command_ or -m.  _module_ are not consumed
by the Python interpreter's option processing but left in `sys.argv'
for  the command or module to handle.

2.2.1.2 Interactive Mode
........................

When commands are read from a tty, the interpreter is said to be in
_interactive mode_.  In this mode it prompts for the next command with
the _primary prompt_, usually three greater-than signs (`>>>'); for
continuation lines it prompts with the _secondary prompt_, by default
three dots (`...'). The interpreter prints a welcome message stating
its version number and a copyright notice before printing the first
prompt:

    python
    Python 2.7 (#1, Feb 28 2010, 00:02:06)
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

Continuation lines are needed when entering a multi-line construct. As
an example, take a look at this if statement:

    >>> the_world_is_flat = 1
    >>> if the_world_is_flat:
    ...     print "Be careful not to fall off!"
    ...
    Be careful not to fall off!

2.2.2 The Interpreter and Its Environment
-----------------------------------------

2.2.2.1 Error Handling
......................

When an error occurs, the interpreter prints an error message and a
stack trace.  In interactive mode, it then returns to the primary
prompt; when input came from a file, it exits with a nonzero exit
status after printing the stack trace.  (Exceptions handled by an
except clause in a try statement are not errors in
this context.)  Some errors are unconditionally fatal and cause an exit
with a nonzero exit; this applies to internal inconsistencies and some
cases of running out of memory.  All error messages are written to the
standard error stream; normal output from executed commands is written
to standard output.

  Typing the interrupt character (usually Control-C or DEL) to the
primary or secondary prompt cancels the input and returns to the
primary prompt. (1) Typing an interrupt while a command is executing
raises the KeyboardInterrupt exception, which may be
handled by a try.  statement.

2.2.2.2 Executable Python Scripts
.................................

On BSD'ish Unix systems, Python scripts can be made directly
executable, like shell scripts, by putting the line

    #! /usr/bin/env python

(assuming that the interpreter is on the user's `PATH') at the beginning
of the script and giving the file an executable mode.  The `#!' must be
the first two characters of the file.  On some platforms, this first
line must end with a Unix-style line ending (`'\n''), not a Windows
(`'\r\n'') line ending.  Note that the hash, or pound, character,
`'#'', is used to start a comment in Python.

  The script can be given an executable mode, or permission, using the
*chmod* command:

    $ chmod +x myscript.py

On Windows systems, there is no notion of an "executable mode".  The
Python installer automatically associates `.py' files with `python.exe'
so that a double-click on a Python file will run it as a script.  The
extension can also be `.pyw', in that case, the console window that
normally appears is suppressed.

2.2.2.3 Source Code Encoding
............................

It is possible to use encodings different than ASCII in Python source
files. The best way to do it is to put one more special comment line
right after the `#!' line to define the source file encoding:

    # -*- coding: encoding -*-

With that declaration, all characters in the source file will be
treated as having the encoding _encoding_, and it will be possible to
directly write Unicode string literals in the selected encoding.  The
list of possible encodings can be found in the Python Library
Reference, in the section on codecs.

  For example, to write Unicode literals including the Euro currency
symbol, the ISO-8859-15 encoding can be used, with the Euro symbol
having the ordinal value 164.  This script will print the value 8364
(the Unicode codepoint corresponding to the Euro symbol) and then exit:

    # -*- coding: iso-8859-15 -*-

    currency = u"€"
    print ord(currency)

If your editor supports saving files as `UTF-8' with a UTF-8 _byte
order mark_ (aka BOM), you can use that instead of an encoding
declaration. IDLE supports this capability if `Options/General/Default
Source Encoding/UTF-8' is set.  Notice that this signature is not
understood in older Python releases (2.2 and earlier), and also not
understood by the operating system for script files with `#!' lines
(only used on Unix systems).

  By using UTF-8 (either through the signature or an encoding
declaration), characters of most languages in the world can be used
simultaneously in string literals and comments.  Using non-ASCII
characters in identifiers is not supported. To display all these
characters properly, your editor must recognize that the file is UTF-8,
and it must use a font that supports all the characters in the file.

2.2.2.4 The Interactive Startup File
....................................

When you use Python interactively, it is frequently handy to have some
standard commands executed every time the interpreter is started.  You
can do this by setting an environment variable named
PYTHONSTARTUP to the name of a file containing your start-up
commands.  This is similar to the `.profile' feature of the Unix shells.

  This file is only read in interactive sessions, not when Python reads
commands from a script, and not when `/dev/tty' is given as the
explicit source of commands (which otherwise behaves like an
interactive session).  It is executed in the same namespace where
interactive commands are executed, so that objects that it defines or
imports can be used without qualification in the interactive session.
You can also change the prompts `sys.ps1' and `sys.ps2' in this file.

  If you want to read an additional start-up file from the current
directory, you can program this in the global start-up file using code
like `if os.path.isfile('.pythonrc.py'): execfile('.pythonrc.py')'.  If
you want to use the startup file in a script, you must do this
explicitly in the script:

    import os
    filename = os.environ.get('PYTHONSTARTUP')
    if filename and os.path.isfile(filename):
        execfile(filename)

Footnotes
.........

     (1) A problem with the GNU Readline package may prevent this.

2.3 An Informal Introduction to Python
======================================

In the following examples, input and output are distinguished by the
presence or absence of prompts (`>>>' and `...'): to repeat the
example, you must type everything after the prompt, when the prompt
appears; lines that do not begin with a prompt are output from the
interpreter. Note that a secondary prompt on a line by itself in an
example means you must type a blank line; this is used to end a
multi-line command.

  Many of the examples in this manual, even those entered at the
interactive prompt, include comments.  Comments in Python start with
the hash character, `#', and extend to the end of the physical line.  A
comment may appear at the start of a line or following whitespace or
code, but not within a string literal.  A hash character within a
string literal is just a hash character.  Since comments are to clarify
code and are not interpreted by Python, they may be omitted when typing
in examples.

  Some examples:

    # this is the first comment
    SPAM = 1                 # and this is the second comment
                             # ... and now a third!
    STRING = "# This is not a comment."

2.3.1 Using Python as a Calculator
----------------------------------

Let's try some simple Python commands.  Start the interpreter and wait
for the primary prompt, `>>>'.  (It shouldn't take long.)

2.3.1.1 Numbers
...............

The interpreter acts as a simple calculator: you can type an expression
at it and it will write the value.  Expression syntax is
straightforward: the operators `+', `-', `*' and `/' work just like in
most other languages (for example, Pascal or C); parentheses can be
used for grouping.  For example:

    >>> 2+2
    4
    >>> # This is a comment
    ... 2+2
    4
    >>> 2+2  # and a comment on the same line as code
    4
    >>> (50-5*6)/4
    5
    >>> # Integer division returns the floor:
    ... 7/3
    2
    >>> 7/-3
    -3

The equal sign (`'='') is used to assign a value to a variable.
Afterwards, no result is displayed before the next interactive prompt:

    >>> width = 20
    >>> height = 5*9
    >>> width * height
    900

A value can be assigned to several variables simultaneously:

    >>> x = y = z = 0  # Zero x, y and z
    >>> x
    0
    >>> y
    0
    >>> z
    0

Variables must be "defined" (assigned a value) before they can be used,
or an error will occur:

    >>> # try to access an undefined variable
    ... n
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'n' is not defined

There is full support for floating point; operators with mixed type
operands convert the integer operand to floating point:

    >>> 3 * 3.75 / 1.5
    7.5
    >>> 7.0 / 2
    3.5

Complex numbers are also supported; imaginary numbers are written with
a suffix of `j' or `J'.  Complex numbers with a nonzero real component
are written as `(real+imagj)', or can be created with the
`complex(real, imag)' function.

    >>> 1j * 1J
    (-1+0j)
    >>> 1j * complex(0,1)
    (-1+0j)
    >>> 3+1j*3
    (3+3j)
    >>> (3+1j)*3
    (9+3j)
    >>> (1+2j)/(1+1j)
    (1.5+0.5j)

Complex numbers are always represented as two floating point numbers,
the real and imaginary part.  To extract these parts from a complex
number _z_, use `z.real' and `z.imag'.

    >>> a=1.5+0.5j
    >>> a.real
    1.5
    >>> a.imag
    0.5

The conversion functions to floating point and integer (see float(),
see int() and long()) don't work for complex
numbers -- there is no one correct way to convert a complex number to a
real number.  Use `abs(z)' to get its magnitude (as a float) or
`z.real' to get its real part.

    >>> a=3.0+4.0j
    >>> float(a)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: can't convert complex to float; use abs(z)
    >>> a.real
    3.0
    >>> a.imag
    4.0
    >>> abs(a)  # sqrt(a.real**2 + a.imag**2)
    5.0

In interactive mode, the last printed expression is assigned to the
variable `_'.  This means that when you are using Python as a desk
calculator, it is somewhat easier to continue calculations, for example:

    >>> tax = 12.5 / 100
    >>> price = 100.50
    >>> price * tax
    12.5625
    >>> price + _
    113.0625
    >>> round(_, 2)
    113.06

This variable should be treated as read-only by the user.  Don't
explicitly assign a value to it -- you would create an independent
local variable with the same name masking the built-in variable with
its magic behavior.

2.3.1.2 Strings
...............

Besides numbers, Python can also manipulate strings, which can be
expressed in several ways.  They can be enclosed in single quotes or
double quotes:

    >>> 'spam eggs'
    'spam eggs'
    >>> 'doesn\'t'
    "doesn't"
    >>> "doesn't"
    "doesn't"
    >>> '"Yes," he said.'
    '"Yes," he said.'
    >>> "\"Yes,\" he said."
    '"Yes," he said.'
    >>> '"Isn\'t," she said.'
    '"Isn\'t," she said.'

The interpreter prints the result of string operations in the same way
as they are typed for input: inside quotes, and with quotes and other
funny characters escaped by backslashes, to show the precise value.
The string is enclosed in double quotes if the string contains a single
quote and no double quotes, else it's enclosed in single quotes.  The
print statement produces a more readable output for such
input strings.

  String literals can span multiple lines in several ways.
Continuation lines can be used, with a backslash as the last character
on the line indicating that the next line is a logical continuation of
the line:

    hello = "This is a rather long string containing\n\
    several lines of text just as you would do in C.\n\
        Note that whitespace at the beginning of the line is\
     significant."

    print hello

Note that newlines still need to be embedded in the string using `\n' -
the newline following the trailing backslash is discarded.  This
example would print the following:

    This is a rather long string containing
    several lines of text just as you would do in C.
        Note that whitespace at the beginning of the line is significant.

Or, strings can be surrounded in a pair of matching triple-quotes:
`"""' or `''''.  End of lines do not need to be escaped when using
triple-quotes, but they will be included in the string.

    print """
    Usage: thingy [OPTIONS]
         -h                        Display this usage message
         -H hostname               Hostname to connect to
    """

produces the following output:

    Usage: thingy [OPTIONS]
         -h                        Display this usage message
         -H hostname               Hostname to connect to

If we make the string literal a "raw" string, `\n' sequences are not
converted to newlines, but the backslash at the end of the line, and
the newline character in the source, are both included in the string as
data.  Thus, the example:

    hello = r"This is a rather long string containing\n\
    several lines of text much as you would do in C."

    print hello

would print:

    This is a rather long string containing\n\
    several lines of text much as you would do in C.

The interpreter prints the result of string operations in the same way
as they are typed for input: inside quotes, and with quotes and other
funny characters escaped by backslashes, to show the precise value.
The string is enclosed in double quotes if the string contains a single
quote and no double quotes, else it's enclosed in single quotes.  (The
print statement, described later, can be used to write
strings without quotes or escapes.)

  Strings can be concatenated (glued together) with the `+' operator,
and repeated with `*':

    >>> word = 'Help' + 'A'
    >>> word
    'HelpA'
    >>> '<' + word*5 + '>'
    '<HelpAHelpAHelpAHelpAHelpA>'

Two string literals next to each other are automatically concatenated;
the first line above could also have been written `word = 'Help' 'A'';
this only works with two literals, not with arbitrary string
expressions:

    >>> 'str' 'ing'                   #  <-  This is ok
    'string'
    >>> 'str'.strip() + 'ing'   #  <-  This is ok
    'string'
    >>> 'str'.strip() 'ing'     #  <-  This is invalid
      File "<stdin>", line 1, in ?
        'str'.strip() 'ing'
                          ^
    SyntaxError: invalid syntax

Strings can be subscripted (indexed); like in C, the first character of
a string has subscript (index) 0.  There is no separate character type;
a character is simply a string of size one.  Like in Icon, substrings
can be specified with the _slice notation_: two indices separated by a
colon.

    >>> word[4]
    'A'
    >>> word[0:2]
    'He'
    >>> word[2:4]
    'lp'

Slice indices have useful defaults; an omitted first index defaults to
zero, an omitted second index defaults to the size of the string being
sliced.

    >>> word[:2]    # The first two characters
    'He'
    >>> word[2:]    # Everything except the first two characters
    'lpA'

Unlike a C string, Python strings cannot be changed.  Assigning to an
indexed position in the string results in an error:

    >>> word[0] = 'x'
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: object does not support item assignment
    >>> word[:1] = 'Splat'
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: object does not support slice assignment

However, creating a new string with the combined content is easy and
efficient:

    >>> 'x' + word[1:]
    'xelpA'
    >>> 'Splat' + word[4]
    'SplatA'

Here's a useful invariant of slice operations: `s[:i] + s[i:]' equals
`s'.

    >>> word[:2] + word[2:]
    'HelpA'
    >>> word[:3] + word[3:]
    'HelpA'

Degenerate slice indices are handled gracefully: an index that is too
large is replaced by the string size, an upper bound smaller than the
lower bound returns an empty string.

    >>> word[1:100]
    'elpA'
    >>> word[10:]
    ''
    >>> word[2:1]
    ''

Indices may be negative numbers, to start counting from the right. For
example:

    >>> word[-1]     # The last character
    'A'
    >>> word[-2]     # The last-but-one character
    'p'
    >>> word[-2:]    # The last two characters
    'pA'
    >>> word[:-2]    # Everything except the last two characters
    'Hel'

But note that -0 is really the same as 0, so it does not count from the
right!

    >>> word[-0]     # (since -0 equals 0)
    'H'

Out-of-range negative slice indices are truncated, but don't try this
for single-element (non-slice) indices:

    >>> word[-100:]
    'HelpA'
    >>> word[-10]    # error
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    IndexError: string index out of range

One way to remember how slices work is to think of the indices as
pointing _between_ characters, with the left edge of the first
character numbered 0.  Then the right edge of the last character of a
string of _n_ characters has index _n_, for example:

     +---+---+---+---+---+
     | H | e | l | p | A |
     +---+---+---+---+---+
     0   1   2   3   4   5
    -5  -4  -3  -2  -1

The first row of numbers gives the position of the indices 0...5 in the
string; the second row gives the corresponding negative indices. The
slice from _i_ to _j_ consists of all characters between the edges
labeled _i_ and _j_, respectively.

  For non-negative indices, the length of a slice is the difference of
the indices, if both are within bounds.  For example, the length of
`word[1:3]' is 2.

  The built-in function len() returns the length of a
string:

    >>> s = 'supercalifragilisticexpialidocious'
    >>> len(s)
    34

See Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange.
     Strings, and the Unicode strings described in the next section, are
     examples of _sequence types_, and support the common operations
     supported by such types.

See String Methods.
     Both strings and Unicode strings support a large number of methods
     for basic transformations and searching.

See String Formatting.
     Information about string formatting with str.format().
     is described here.

See String Formatting Operations.
     The old formatting operations invoked when strings and Unicode
     strings are the left operand of the `%' operator are described in
     more detail here.

2.3.1.3 Unicode Strings
.......................

Starting with Python 2.0 a new data type for storing text data is
available to the programmer: the Unicode object. It can be used to
store and manipulate Unicode data (see <http://www.unicode.org/>) and
integrates well with the existing string objects, providing
auto-conversions where necessary.

  Unicode has the advantage of providing one ordinal for every
character in every script used in modern and ancient texts. Previously,
there were only 256 possible ordinals for script characters. Texts were
typically bound to a code page which mapped the ordinals to script
characters. This lead to very much confusion especially with respect to
internationalization (usually written as `i18n' -- `'i'' + 18
characters + `'n'') of software.  Unicode solves these problems by
defining one code page for all scripts.

  Creating Unicode strings in Python is just as simple as creating
normal strings:

    >>> u'Hello World !'
    u'Hello World !'

The small `'u'' in front of the quote indicates that a Unicode string is
supposed to be created. If you want to include special characters in
the string, you can do so by using the Python _Unicode-Escape_
encoding. The following example shows how:

    >>> u'Hello\u0020World !'
    u'Hello World !'

The escape sequence `\u0020' indicates to insert the Unicode character
with the ordinal value 0x0020 (the space character) at the given
position.

  Other characters are interpreted by using their respective ordinal
values directly as Unicode ordinals.  If you have literal strings in
the standard Latin-1 encoding that is used in many Western countries,
you will find it convenient that the lower 256 characters of Unicode
are the same as the 256 characters of Latin-1.

  For experts, there is also a raw mode just like the one for normal
strings. You have to prefix the opening quote with 'ur' to have Python
use the _Raw-Unicode-Escape_ encoding. It will only apply the above
`\uXXXX' conversion if there is an uneven number of backslashes in
front of the small 'u'.

    >>> ur'Hello\u0020World !'
    u'Hello World !'
    >>> ur'Hello\\u0020World !'
    u'Hello\\\\u0020World !'

The raw mode is most useful when you have to enter lots of backslashes,
as can be necessary in regular expressions.

  Apart from these standard encodings, Python provides a whole set of
other ways of creating Unicode strings on the basis of a known encoding.

  The built-in function unicode() provides access to all
registered Unicode codecs (COders and DECoders). Some of the more well
known encodings which these codecs can convert are _Latin-1_, _ASCII_,
_UTF-8_, and _UTF-16_. The latter two are variable-length encodings
that store each Unicode character in one or more bytes. The default
encoding is normally set to ASCII, which passes through characters in
the range 0 to 127 and rejects any other characters with an error.
When a Unicode string is printed, written to a file, or converted with
str(), conversion takes place using this default encoding.

    >>> u"abc"
    u'abc'
    >>> str(u"abc")
    'abc'
    >>> u"äöü"
    u'\xe4\xf6\xfc'
    >>> str(u"äöü")
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

To convert a Unicode string into an 8-bit string using a specific
encoding, Unicode objects provide an `encode()' method that takes one
argument, the name of the encoding.  Lowercase names for encodings are
preferred.

    >>> u"äöü".encode('utf-8')
    '\xc3\xa4\xc3\xb6\xc3\xbc'

If you have data in a specific encoding and want to produce a
corresponding Unicode string from it, you can use the unicode()
function with the encoding name as the second argument.

    >>> unicode('\xc3\xa4\xc3\xb6\xc3\xbc', 'utf-8')
    u'\xe4\xf6\xfc'

2.3.1.4 Lists
.............

Python knows a number of _compound_ data types, used to group together
other values.  The most versatile is the _list_, which can be written
as a list of comma-separated values (items) between square brackets.
List items need not all have the same type.

    >>> a = ['spam', 'eggs', 100, 1234]
    >>> a
    ['spam', 'eggs', 100, 1234]

Like string indices, list indices start at 0, and lists can be sliced,
concatenated and so on:

    >>> a[0]
    'spam'
    >>> a[3]
    1234
    >>> a[-2]
    100
    >>> a[1:-1]
    ['eggs', 100]
    >>> a[:2] + ['bacon', 2*2]
    ['spam', 'eggs', 'bacon', 4]
    >>> 3*a[:3] + ['Boo!']
    ['spam', 'eggs', 100, 'spam', 'eggs', 100, 'spam', 'eggs', 100, 'Boo!']

All slice operations return a new list containing the requested
elements.  This means that the following slice returns a shallow copy
of the list _a_:

    >>> a[:]
    ['spam', 'eggs', 100, 1234]

Unlike strings, which are _immutable_, it is possible to change
individual elements of a list:

    >>> a
    ['spam', 'eggs', 100, 1234]
    >>> a[2] = a[2] + 23
    >>> a
    ['spam', 'eggs', 123, 1234]

Assignment to slices is also possible, and this can even change the
size of the list or clear it entirely:

    >>> # Replace some items:
    ... a[0:2] = [1, 12]
    >>> a
    [1, 12, 123, 1234]
    >>> # Remove some:
    ... a[0:2] = []
    >>> a
    [123, 1234]
    >>> # Insert some:
    ... a[1:1] = ['bletch', 'xyzzy']
    >>> a
    [123, 'bletch', 'xyzzy', 1234]
    >>> # Insert (a copy of) itself at the beginning
    >>> a[:0] = a
    >>> a
    [123, 'bletch', 'xyzzy', 1234, 123, 'bletch', 'xyzzy', 1234]
    >>> # Clear the list: replace all items with an empty list
    >>> a[:] = []
    >>> a
    []

The built-in function len() also applies to lists:

    >>> a = ['a', 'b', 'c', 'd']
    >>> len(a)
    4

It is possible to nest lists (create lists containing other lists), for
example:

    >>> q = [2, 3]
    >>> p = [1, q, 4]
    >>> len(p)
    3
    >>> p[1]
    [2, 3]
    >>> p[1][0]
    2
    >>> p[1].append('xtra')     # See section 5.1
    >>> p
    [1, [2, 3, 'xtra'], 4]
    >>> q
    [2, 3, 'xtra']

Note that in the last example, `p[1]' and `q' really refer to the same
object!  We'll come back to _object semantics_ later.

2.3.2 First Steps Towards Programming
-------------------------------------

Of course, we can use Python for more complicated tasks than adding two
and two together.  For instance, we can write an initial sub-sequence
of the _Fibonacci_ series as follows:

    >>> # Fibonacci series:
    ... # the sum of two elements defines the next
    ... a, b = 0, 1
    >>> while b < 10:
    ...     print b
    ...     a, b = b, a+b
    ...
    1
    1
    2
    3
    5
    8

This example introduces several new features.

   * The first line contains a _multiple assignment_: the variables `a'
     and `b' simultaneously get the new values 0 and 1.  On the last
     line this is used again, demonstrating that the expressions on the
     right-hand side are all evaluated first before any of the
     assignments take place.  The right-hand side expressions are
     evaluated  from the left to the right.

   * The while loop executes as long as the condition
     (here: `b < 10') remains true.  In Python, like in C, any non-zero
     integer value is true; zero is false.  The condition may also be a
     string or list value, in fact any sequence; anything with a
     non-zero length is true, empty sequences are false.  The test used
     in the example is a simple comparison.  The standard comparison
     operators are written the same as in C: `<' (less than), `>'
     (greater than), `==' (equal to), `<=' (less than or equal to),
     `>=' (greater than or equal to) and `!=' (not equal to).

   * The _body_ of the loop is _indented_: indentation is Python's way
     of grouping statements.  Python does not (yet!) provide an
     intelligent input line editing facility, so you have to type a tab
     or space(s) for each indented line.  In practice you will prepare
     more complicated input for Python with a text editor; most text
     editors have an auto-indent facility.  When a compound statement is
     entered interactively, it must be followed by a blank line to
     indicate completion (since the parser cannot guess when you have
     typed the last line).  Note that each line within a basic block
     must be indented by the same amount.

   * The print statement writes the value of the
     expression(s) it is given.  It differs from just writing the
     expression you want to write (as we did earlier in the calculator
     examples) in the way it handles multiple expressions and strings.
     Strings are printed without quotes, and a space is inserted
     between items, so you can format things nicely, like this:

         >>> i = 256*256
         >>> print 'The value of i is', i
         The value of i is 65536

     A trailing comma avoids the newline after the output:

         >>> a, b = 0, 1
         >>> while b < 1000:
         ...     print b,
         ...     a, b = b, a+b
         ...
         1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987

     Note that the interpreter inserts a newline before it prints the
     next prompt if the last line was not completed.

2.4 More Control Flow Tools
===========================

Besides the while statement just introduced, Python knows
the usual control flow statements known from other languages, with some
twists.

2.4.1 `if' Statements
---------------------

Perhaps the most well-known statement type is the if.
statement.  For example:

    >>> x = int(raw_input("Please enter an integer: "))
    Please enter an integer: 42
    >>> if x < 0:
    ...      x = 0
    ...      print 'Negative changed to zero'
    ... elif x == 0:
    ...      print 'Zero'
    ... elif x == 1:
    ...      print 'Single'
    ... else:
    ...      print 'More'
    ...
    More

There can be zero or more elif parts, and the else
part is optional.  The keyword 'elif.' is short for
'else if', and is useful to avoid excessive indentation.  An  if...
See elif...  See elif... sequence is a
substitute for the `switch' or `case' statements found in other
languages.

2.4.2 `for' Statements
----------------------

The for statement in Python differs a bit from what you may
be used to in C or Pascal.  Rather than always iterating over an
arithmetic progression of numbers (like in Pascal), or giving the user
the ability to define both the iteration step and halting condition (as
C), Python's for statement iterates over the items of any
sequence (a list or a string), in the order that they appear in the
sequence.  For example (no pun intended):

    >>> # Measure some strings:
    ... a = ['cat', 'window', 'defenestrate']
    >>> for x in a:
    ...     print x, len(x)
    ...
    cat 3
    window 6
    defenestrate 12

It is not safe to modify the sequence being iterated over in the loop
(this can only happen for mutable sequence types, such as lists).  If
you need to modify the list you are iterating over (for example, to
duplicate selected items) you must iterate over a copy.  The slice
notation makes this particularly convenient:

    >>> for x in a[:]: # make a slice copy of the entire list
    ...    if len(x) > 6: a.insert(0, x)
    ...
    >>> a
    ['defenestrate', 'cat', 'window', 'defenestrate']

2.4.3 The `range()' Function
----------------------------

If you do need to iterate over a sequence of numbers, the built-in
function range() comes in handy.  It generates lists
containing arithmetic progressions:

    >>> range(10)
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The given end point is never part of the generated list; `range(10)'
generates a list of 10 values, the legal indices for items of a
sequence of length 10.  It is possible to let the range start at
another number, or to specify a different increment (even negative;
sometimes this is called the 'step'):

    >>> range(5, 10)
    [5, 6, 7, 8, 9]
    >>> range(0, 10, 3)
    [0, 3, 6, 9]
    >>> range(-10, -100, -30)
    [-10, -40, -70]

To iterate over the indices of a sequence, you can combine
range() and len() as follows:

    >>> a = ['Mary', 'had', 'a', 'little', 'lamb']
    >>> for i in range(len(a)):
    ...     print i, a[i]
    ...
    0 Mary
    1 had
    2 a
    3 little
    4 lamb

In most such cases, however, it is convenient to use the
enumerate().  function, see Looping Techniques.

2.4.4 `break' and `continue' Statements, and `else' Clauses on Loops
--------------------------------------------------------------------

The break statement, like in C, breaks out of the smallest
enclosing for or while loop.

  The continue statement, also borrowed from C, continues
with the next iteration of the loop.

  Loop statements may have an `else' clause; it is executed when the
loop terminates through exhaustion of the list (with for)
or when the condition becomes false (with while), but not
when the loop is terminated by a break statement.  This is
exemplified by the following loop, which searches for prime numbers:

    >>> for n in range(2, 10):
    ...     for x in range(2, n):
    ...         if n % x == 0:
    ...             print n, 'equals', x, '*', n/x
    ...             break
    ...     else:
    ...         # loop fell through without finding a factor
    ...         print n, 'is a prime number'
    ...
    2 is a prime number
    3 is a prime number
    4 equals 2 * 2
    5 is a prime number
    6 equals 2 * 3
    7 is a prime number
    8 equals 2 * 4
    9 equals 3 * 3

2.4.5 `pass' Statements
-----------------------

The pass statement does nothing. It can be used when a
statement is required syntactically but the program requires no action.
For example:

    >>> while True:
    ...     pass  # Busy-wait for keyboard interrupt (Ctrl+C)
    ...

This is commonly used for creating minimal classes:

    >>> class MyEmptyClass:
    ...     pass
    ...

Another place pass can be used is as a place-holder for a
function or conditional body when you are working on new code, allowing
you to keep thinking at a more abstract level.  The pass is
silently ignored:

    >>> def initlog(*args):
    ...     pass   # Remember to implement this!
    ...

2.4.6 Defining Functions
------------------------

We can create a function that writes the Fibonacci series to an
arbitrary boundary:

    >>> def fib(n):    # write Fibonacci series up to n
    ...     """Print a Fibonacci series up to n."""
    ...     a, b = 0, 1
    ...     while a < n:
    ...         print a,
    ...         a, b = b, a+b
    ...
    >>> # Now call the function we just defined:
    ... fib(2000)
    0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597

The keyword def introduces a function _definition_.  It
must be followed by the function name and the parenthesized list of
formal parameters.  The statements that form the body of the function
start at the next line, and must be indented.

  The first statement of the function body can optionally be a string
literal; this string literal is the function's documentation string, or
_docstring_.  (More about docstrings can be found in the section
Documentation Strings)  There are tools which use docstrings to
automatically produce online or printed documentation, or to let the
user interactively browse through code; it's good practice to include
docstrings in code that you write, so make a habit of it.

  The _execution_ of a function introduces a new symbol table used for
the local variables of the function.  More precisely, all variable
assignments in a function store the value in the local symbol table;
whereas variable references first look in the local symbol table, then
in the local symbol tables of enclosing functions, then in the global
symbol table, and finally in the table of built-in names. Thus, global
variables cannot be directly assigned a value within a function (unless
named in a global statement), although they may be
referenced.

  The actual parameters (arguments) to a function call are introduced
in the local symbol table of the called function when it is called;
thus, arguments are passed using _call by value_ (where the _value_ is
always an object _reference_, not the value of the object). (1) When a
function calls another function, a new local symbol table is created
for that call.

  A function definition introduces the function name in the current
symbol table.  The value of the function name has a type that is
recognized by the interpreter as a user-defined function.  This value
can be assigned to another name which can then also be used as a
function.  This serves as a general renaming mechanism:

    >>> fib
    <function fib at 10042ed0>
    >>> f = fib
    >>> f(100)
    0 1 1 2 3 5 8 13 21 34 55 89

Coming from other languages, you might object that `fib' is not a
function but a procedure since it doesn't return a value.  In fact,
even functions without a return statement do return a
value, albeit a rather boring one.  This value is called `None' (it's a
built-in name).  Writing the value `None' is normally suppressed by the
interpreter if it would be the only value written.  You can see it if
you really want to using print.:

    >>> fib(0)
    >>> print fib(0)
    None

It is simple to write a function that returns a list of the numbers of
the Fibonacci series, instead of printing it:

    >>> def fib2(n): # return Fibonacci series up to n
    ...     """Return a list containing the Fibonacci series up to n."""
    ...     result = []
    ...     a, b = 0, 1
    ...     while a < n:
    ...         result.append(a)    # see below
    ...         a, b = b, a+b
    ...     return result
    ...
    >>> f100 = fib2(100)    # call it
    >>> f100                # write the result
    [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]

This example, as usual, demonstrates some new Python features:

   * The return statement returns with a value from a
     function.  return without an expression argument
     returns `None'. Falling off the end of a function also returns
     `None'.

   * The statement `result.append(a)' calls a _method_ of the list
     object `result'.  A method is a function that 'belongs' to an
     object and is named `obj.methodname', where `obj' is some object
     (this may be an expression), and `methodname' is the name of a
     method that is defined by the object's type.  Different types
     define different methods.  Methods of different types may have the
     same name without causing ambiguity.  (It is possible to define
     your own object types and methods, using _classes_, see
     Classes)  The method `append()' shown in the example is
     defined for list objects; it adds a new element at the end of the
     list.  In this example it is equivalent to `result = result +
     [a]', but more efficient.

2.4.7 More on Defining Functions
--------------------------------

It is also possible to define functions with a variable number of
arguments.  There are three forms, which can be combined.

2.4.7.1 Default Argument Values
...............................

The most useful form is to specify a default value for one or more
arguments.  This creates a function that can be called with fewer
arguments than it is defined to allow.  For example:

    def ask_ok(prompt, retries=4, complaint='Yes or no, please!'):
        while True:
            ok = raw_input(prompt)
            if ok in ('y', 'ye', 'yes'):
                return True
            if ok in ('n', 'no', 'nop', 'nope'):
                return False
            retries = retries - 1
            if retries < 0:
                raise IOError('refusenik user')
            print complaint

This function can be called in several ways:

   * giving only the mandatory argument: `ask_ok('Do you really want to
     quit?')'

   * giving one of the optional arguments: `ask_ok('OK to overwrite the
     file?', 2)'

   * or even giving all arguments: `ask_ok('OK to overwrite the file?',
     2, 'Come on, only yes or no!')'

  This example also introduces the in keyword. This tests
whether or not a sequence contains a certain value.

  The default values are evaluated at the point of function definition
in the _defining_ scope, so that

    i = 5

    def f(arg=i):
        print arg

    i = 6
    f()

will print `5'.

  *Important warning:*  The default value is evaluated only once. This
makes a difference when the default is a mutable object such as a list,
dictionary, or instances of most classes.  For example, the following
function accumulates the arguments passed to it on subsequent calls:

    def f(a, L=[]):
        L.append(a)
        return L

    print f(1)
    print f(2)
    print f(3)

This will print

    [1]
    [1, 2]
    [1, 2, 3]

If you don't want the default to be shared between subsequent calls,
you can write the function like this instead:

    def f(a, L=None):
        if L is None:
            L = []
        L.append(a)
        return L

2.4.7.2 Keyword Arguments
.........................

Functions can also be called using keyword arguments of the form
`keyword = value'.  For instance, the following function:

    def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'):
        print "-- This parrot wouldn't", action,
        print "if you put", voltage, "volts through it."
        print "-- Lovely plumage, the", type
        print "-- It's", state, "!"

could be called in any of the following ways:

    parrot(1000)
    parrot(action = 'VOOOOOM', voltage = 1000000)
    parrot('a thousand', state = 'pushing up the daisies')
    parrot('a million', 'bereft of life', 'jump')

but the following calls would all be invalid:

    parrot()                     # required argument missing
    parrot(voltage=5.0, 'dead')  # non-keyword argument following keyword
    parrot(110, voltage=220)     # duplicate value for argument
    parrot(actor='John Cleese')  # unknown keyword

In general, an argument list must have any positional arguments
followed by any keyword arguments, where the keywords must be chosen
from the formal parameter names.  It's not important whether a formal
parameter has a default value or not.  No argument may receive a value
more than once -- formal parameter names corresponding to positional
arguments cannot be used as keywords in the same calls. Here's an
example that fails due to this restriction:

    >>> def function(a):
    ...     pass
    ...
    >>> function(0, a=0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: function() got multiple values for keyword argument 'a'

When a final formal parameter of the form `**name' is present, it
receives a dictionary (see Mapping Types -- dict)
containing all keyword arguments except for those corresponding to a
formal parameter.  This may be combined with a formal parameter of the
form `*name' (described in the next subsection) which receives a tuple
containing the positional arguments beyond the formal parameter list.
(`*name' must occur before `**name'.) For example, if we define a
function like this:

    def cheeseshop(kind, *arguments, **keywords):
        print "-- Do you have any", kind, "?"
        print "-- I'm sorry, we're all out of", kind
        for arg in arguments:
            print arg
        print "-" * 40
        keys = sorted(keywords.keys())
        for kw in keys:
            print kw, ":", keywords[kw]

It could be called like this:

    cheeseshop("Limburger", "It's very runny, sir.",
               "It's really very, VERY runny, sir.",
               shopkeeper='Michael Palin',
               client="John Cleese",
               sketch="Cheese Shop Sketch")

and of course it would print:

    -- Do you have any Limburger ?
    -- I'm sorry, we're all out of Limburger
    It's very runny, sir.
    It's really very, VERY runny, sir.
    ----------------------------------------
    client : John Cleese
    shopkeeper : Michael Palin
    sketch : Cheese Shop Sketch

Note that the `sort()' method of the list of keyword argument names is
called before printing the contents of the `keywords' dictionary; if
this is not done, the order in which the arguments are printed is
undefined.

2.4.7.3 Arbitrary Argument Lists
................................

Finally, the least frequently used option is to specify that a function
can be called with an arbitrary number of arguments.  These arguments
will be wrapped up in a tuple (see Tuples and Sequences).
Before the variable number of arguments, zero or more normal arguments
may occur.

    def write_multiple_items(file, separator, *args):
        file.write(separator.join(args))

2.4.7.4 Unpacking Argument Lists
................................

The reverse situation occurs when the arguments are already in a list
or tuple but need to be unpacked for a function call requiring separate
positional arguments.  For instance, the built-in range().
function expects separate _start_ and _stop_ arguments.  If they are
not available separately, write the function call with the
`*'-operator to unpack the arguments out of a list or tuple:

    >>> range(3, 6)             # normal call with separate arguments
    [3, 4, 5]
    >>> args = [3, 6]
    >>> range(*args)            # call with arguments unpacked from a list
    [3, 4, 5]

In the same fashion, dictionaries can deliver keyword arguments with
the `**'-operator:

    >>> def parrot(voltage, state='a stiff', action='voom'):
    ...     print "-- This parrot wouldn't", action,
    ...     print "if you put", voltage, "volts through it.",
    ...     print "E's", state, "!"
    ...
    >>> d = {"voltage": "four million", "state": "bleedin' demised", "action": "VOOM"}
    >>> parrot(**d)
    -- This parrot wouldn't VOOM if you put four million volts through it. E's bleedin' demised !

2.4.7.5 Lambda Forms
....................

By popular demand, a few features commonly found in functional
programming languages like Lisp have been added to Python.  With the
lambda.  keyword, small anonymous functions can be created.
Here's a function that returns the sum of its two arguments: `lambda a,
b: a+b'.  Lambda forms can be used wherever function objects are
required.  They are syntactically restricted to a single expression.
Semantically, they are just syntactic sugar for a normal function
definition.  Like nested function definitions, lambda forms can
reference variables from the containing scope:

    >>> def make_incrementor(n):
    ...     return lambda x: x + n
    ...
    >>> f = make_incrementor(42)
    >>> f(0)
    42
    >>> f(1)
    43

2.4.7.6 Documentation Strings
.............................

There are emerging conventions about the content and formatting of
documentation strings.

  The first line should always be a short, concise summary of the
object's purpose.  For brevity, it should not explicitly state the
object's name or type, since these are available by other means (except
if the name happens to be a verb describing a function's operation).
This line should begin with a capital letter and end with a period.

  If there are more lines in the documentation string, the second line
should be blank, visually separating the summary from the rest of the
description.  The following lines should be one or more paragraphs
describing the object's calling conventions, its side effects, etc.

  The Python parser does not strip indentation from multi-line string
literals in Python, so tools that process documentation have to strip
indentation if desired.  This is done using the following convention.
The first non-blank line _after_ the first line of the string
determines the amount of indentation for the entire documentation
string.  (We can't use the first line since it is generally adjacent to
the string's opening quotes so its indentation is not apparent in the
string literal.)  Whitespace "equivalent" to this indentation is then
stripped from the start of all lines of the string.  Lines that are
indented less should not occur, but if they occur all their leading
whitespace should be stripped.  Equivalence of whitespace should be
tested after expansion of tabs (to 8 spaces, normally).

  Here is an example of a multi-line docstring:

    >>> def my_function():
    ...     """Do nothing, but document it.
    ...
    ...     No, really, it doesn't do anything.
    ...     """
    ...     pass
    ...
    >>> print my_function.__doc__
    Do nothing, but document it.

        No, really, it doesn't do anything.

2.4.8 Intermezzo: Coding Style
------------------------------

Now that you are about to write longer, more complex pieces of Python,
it is a good time to talk about _coding style_.  Most languages can be
written (or more concise, _formatted_) in different styles; some are
more readable than others.  Making it easy for others to read your code
is always a good idea, and adopting a nice coding style helps
tremendously for that.

  For Python, PEP 8 (http://www.python.org/dev/peps/pep-0008) has
emerged as the style guide that most projects adhere to; it promotes a
very readable and eye-pleasing coding style.  Every Python developer
should read it at some point; here are the most important points
extracted for you:

   * Use 4-space indentation, and no tabs.

     4 spaces are a good compromise between small indentation (allows
     greater nesting depth) and large indentation (easier to read).
     Tabs introduce confusion, and are best left out.

   * Wrap lines so that they don't exceed 79 characters.

     This helps users with small displays and makes it possible to have
     several code files side-by-side on larger displays.

   * Use blank lines to separate functions and classes, and larger
     blocks of code inside functions.

   * When possible, put comments on a line of their own.

   * Use docstrings.

   * Use spaces around operators and after commas, but not directly
     inside bracketing constructs: `a = f(1, 2) + g(3, 4)'.

   * Name your classes and functions consistently; the convention is to
     use `CamelCase' for classes and `lower_case_with_underscores' for
     functions and methods.  Always use `self' as the name for the
     first method argument (see A First Look at Classes for
     more on classes and methods).

   * Don't use fancy encodings if your code is meant to be used in
     international environments.  Plain ASCII works best in any case.

Footnotes
.........

     (1) Actually, _call by object reference_ would be a better
     description, since if a mutable object is passed, the caller will
     see any changes the callee makes to it (items inserted into a
     list).

2.5 Data Structures
===================

This chapter describes some things you've learned about already in more
detail, and adds some new things as well.

2.5.1 More on Lists
-------------------

The list data type has some more methods.  Here are all of the methods
of list objects:

 -- Method: list.append (x)
     Add an item to the end of the list; equivalent to `a[len(a):] =
     [x]'.

 -- Method: list.extend (L)
     Extend the list by appending all the items in the given list;
     equivalent to `a[len(a):] = L'.

 -- Method: list.insert (i, x)
     Insert an item at a given position.  The first argument is the
     index of the element before which to insert, so `a.insert(0, x)'
     inserts at the front of the list, and `a.insert(len(a), x)' is
     equivalent to `a.append(x)'.

 -- Method: list.remove (x)
     Remove the first item from the list whose value is _x_. It is an
     error if there is no such item.

 -- Method: list.pop ([i])
     Remove the item at the given position in the list, and return it.
     If no index is specified, `a.pop()' removes and returns the last
     item in the list.  (The square brackets around the _i_ in the
     method signature denote that the parameter is optional, not that
     you should type square brackets at that position.  You will see
     this notation frequently in the Python Library Reference.)

 -- Method: list.index (x)
     Return the index in the list of the first item whose value is _x_.
     It is an error if there is no such item.

 -- Method: list.count (x)
     Return the number of times _x_ appears in the list.

 -- Method: list.sort ()
     Sort the items of the list, in place.

 -- Method: list.reverse ()
     Reverse the elements of the list, in place.

  An example that uses most of the list methods:

    >>> a = [66.25, 333, 333, 1, 1234.5]
    >>> print a.count(333), a.count(66.25), a.count('x')
    2 1 0
    >>> a.insert(2, -1)
    >>> a.append(333)
    >>> a
    [66.25, 333, -1, 333, 1, 1234.5, 333]
    >>> a.index(333)
    1
    >>> a.remove(333)
    >>> a
    [66.25, -1, 333, 1, 1234.5, 333]
    >>> a.reverse()
    >>> a
    [333, 1234.5, 1, 333, -1, 66.25]
    >>> a.sort()
    >>> a
    [-1, 1, 66.25, 333, 333, 1234.5]

2.5.1.1 Using Lists as Stacks
.............................

The list methods make it very easy to use a list as a stack, where the
last element added is the first element retrieved ("last-in,
first-out").  To add an item to the top of the stack, use `append()'.
To retrieve an item from the top of the stack, use `pop()' without an
explicit index.  For example:

    >>> stack = [3, 4, 5]
    >>> stack.append(6)
    >>> stack.append(7)
    >>> stack
    [3, 4, 5, 6, 7]
    >>> stack.pop()
    7
    >>> stack
    [3, 4, 5, 6]
    >>> stack.pop()
    6
    >>> stack.pop()
    5
    >>> stack
    [3, 4]

2.5.1.2 Using Lists as Queues
.............................

It is also possible to use a list as a queue, where the first element
added is the first element retrieved ("first-in, first-out"); however,
lists are not efficient for this purpose.  While appends and pops from
the end of list are fast, doing inserts or pops from the beginning of a
list is slow (because all of the other elements have to be shifted by
one).

  To implement a queue, use collections.deque which was
designed to have fast appends and pops from both ends.  For example:

    >>> from collections import deque
    >>> queue = deque(["Eric", "John", "Michael"])
    >>> queue.append("Terry")           # Terry arrives
    >>> queue.append("Graham")          # Graham arrives
    >>> queue.popleft()                 # The first to arrive now leaves
    'Eric'
    >>> queue.popleft()                 # The second to arrive now leaves
    'John'
    >>> queue                           # Remaining queue in order of arrival
    deque(['Michael', 'Terry', 'Graham'])

2.5.1.3 Functional Programming Tools
....................................

There are three built-in functions that are very useful when used with
lists: filter(), map(), and reduce().

  `filter(function, sequence)' returns a sequence consisting of those
items from the sequence for which `function(item)' is true. If
_sequence_ is a string or tuple, the result will
be of the same type; otherwise, it is always a list. For
example, to compute primes up to 25:

    >>> def f(x): return x % 2 != 0 and x % 3 != 0
    ...
    >>> filter(f, range(2, 25))
    [5, 7, 11, 13, 17, 19, 23]

`map(function, sequence)' calls `function(item)' for each of the
sequence's items and returns a list of the return values.  For example,
to compute some cubes:

    >>> def cube(x): return x*x*x
    ...
    >>> map(cube, range(1, 11))
    [1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]

More than one sequence may be passed; the function must then have as
many arguments as there are sequences and is called with the
corresponding item from each sequence (or `None' if some sequence is
shorter than another).  For example:

    >>> seq = range(8)
    >>> def add(x, y): return x+y
    ...
    >>> map(add, seq, seq)
    [0, 2, 4, 6, 8, 10, 12, 14]

`reduce(function, sequence)' returns a single value constructed by
calling the binary function _function_ on the first two items of the
sequence, then on the result and the next item, and so on.  For
example, to compute the sum of the numbers 1 through 10:

    >>> def add(x,y): return x+y
    ...
    >>> reduce(add, range(1, 11))
    55

If there's only one item in the sequence, its value is returned; if the
sequence is empty, an exception is raised.

  A third argument can be passed to indicate the starting value.  In
this case the starting value is returned for an empty sequence, and the
function is first applied to the starting value and the first sequence
item, then to the result and the next item, and so on.  For example,

    >>> def sum(seq):
    ...     def add(x,y): return x+y
    ...     return reduce(add, seq, 0)
    ...
    >>> sum(range(1, 11))
    55
    >>> sum([])
    0

Don't use this example's definition of sum().: since summing
numbers is such a common need, a built-in function `sum(sequence)' is
already provided, and works exactly like this.

  New in version 2.3.

2.5.1.4 List Comprehensions
...........................

List comprehensions provide a concise way to create lists without
resorting to use of map(), filter() and/or
lambda. The resulting list definition tends often to be clearer
than lists built using those constructs.  Each list comprehension
consists of an expression followed by a for.  clause, then
zero or more for or if clauses.  The result
will be a list resulting from evaluating the expression in the context
of the for and if clauses which follow it.  If
the expression would evaluate to a tuple, it must be parenthesized.

    >>> freshfruit = ['  banana', '  loganberry ', 'passion fruit  ']
    >>> [weapon.strip() for weapon in freshfruit]
    ['banana', 'loganberry', 'passion fruit']
    >>> vec = [2, 4, 6]
    >>> [3*x for x in vec]
    [6, 12, 18]
    >>> [3*x for x in vec if x > 3]
    [12, 18]
    >>> [3*x for x in vec if x < 2]
    []
    >>> [[x,x**2] for x in vec]
    [[2, 4], [4, 16], [6, 36]]
    >>> [x, x**2 for x in vec]  # error - parens required for tuples
      File "<stdin>", line 1, in ?
        [x, x**2 for x in vec]
                   ^
    SyntaxError: invalid syntax
    >>> [(x, x**2) for x in vec]
    [(2, 4), (4, 16), (6, 36)]
    >>> vec1 = [2, 4, 6]
    >>> vec2 = [4, 3, -9]
    >>> [x*y for x in vec1 for y in vec2]
    [8, 6, -18, 16, 12, -36, 24, 18, -54]
    >>> [x+y for x in vec1 for y in vec2]
    [6, 5, -7, 8, 7, -5, 10, 9, -3]
    >>> [vec1[i]*vec2[i] for i in range(len(vec1))]
    [8, 12, -54]

List comprehensions are much more flexible than map() and
can be applied to complex expressions and nested functions:

    >>> [str(round(355/113.0, i)) for i in range(1,6)]
    ['3.1', '3.14', '3.142', '3.1416', '3.14159']

2.5.1.5 Nested List Comprehensions
..................................

If you've got the stomach for it, list comprehensions can be nested.
They are a powerful tool but - like all powerful tools - they need to
be used carefully, if at all.

  Consider the following example of a 3x3 matrix held as a list
containing three lists, one list per row:

    >>> mat = [
    ...        [1, 2, 3],
    ...        [4, 5, 6],
    ...        [7, 8, 9],
    ...       ]

Now, if you wanted to swap rows and columns, you could use a list
comprehension:

    >>> print [[row[i] for row in mat] for i in [0, 1, 2]]
    [[1, 4, 7], [2, 5, 8], [3, 6, 9]]

Special care has to be taken for the _nested_ list comprehension:

     To avoid apprehension when nesting list comprehensions, read from
     right to left.

  A more verbose version of this snippet shows the flow explicitly:

    for i in [0, 1, 2]:
        for row in mat:
            print row[i],
        print

In real world, you should prefer built-in functions to complex flow
statements.  The zip() function would do a great job for
this use case:

    >>> zip(*mat)
    [(1, 4, 7), (2, 5, 8), (3, 6, 9)]

See Unpacking Argument Lists for details on the asterisk in
this line.

2.5.2 The `del' statement
-------------------------

There is a way to remove an item from a list given its index instead of
its value: the del statement.  This differs from the
`pop()' method which returns a value.  The del statement
can also be used to remove slices from a list or clear the entire list
(which we did earlier by assignment of an empty list to the slice).
For example:

    >>> a = [-1, 1, 66.25, 333, 333, 1234.5]
    >>> del a[0]
    >>> a
    [1, 66.25, 333, 333, 1234.5]
    >>> del a[2:4]
    >>> a
    [1, 66.25, 1234.5]
    >>> del a[:]
    >>> a
    []

del can also be used to delete entire variables:

    >>> del a

Referencing the name `a' hereafter is an error (at least until another
value is assigned to it).  We'll find other uses for del.
later.

2.5.3 Tuples and Sequences
--------------------------

We saw that lists and strings have many common properties, such as
indexing and slicing operations.  They are two examples of _sequence_
data types (see Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange).
Since Python is an evolving
language, other sequence data types may be added.  There is also
another standard sequence data type: the _tuple_.

  A tuple consists of a number of values separated by commas, for
instance:

    >>> t = 12345, 54321, 'hello!'
    >>> t[0]
    12345
    >>> t
    (12345, 54321, 'hello!')
    >>> # Tuples may be nested:
    ... u = t, (1, 2, 3, 4, 5)
    >>> u
    ((12345, 54321, 'hello!'), (1, 2, 3, 4, 5))

As you see, on output tuples are always enclosed in parentheses, so
that nested tuples are interpreted correctly; they may be input with or
without surrounding parentheses, although often parentheses are
necessary anyway (if the tuple is part of a larger expression).

  Tuples have many uses.  For example: (x, y) coordinate pairs,
employee records from a database, etc.  Tuples, like strings, are
immutable: it is not possible to assign to the individual items of a
tuple (you can simulate much of the same effect with slicing and
concatenation, though).  It is also possible to create tuples which
contain mutable objects, such as lists.

  A special problem is the construction of tuples containing 0 or 1
items: the syntax has some extra quirks to accommodate these.  Empty
tuples are constructed by an empty pair of parentheses; a tuple with
one item is constructed by following a value with a comma (it is not
sufficient to enclose a single value in parentheses). Ugly, but
effective.  For example:

    >>> empty = ()
    >>> singleton = 'hello',    # <-- note trailing comma
    >>> len(empty)
    0
    >>> len(singleton)
    1
    >>> singleton
    ('hello',)

The statement `t = 12345, 54321, 'hello!'' is an example of _tuple
packing_: the values `12345', `54321' and `'hello!'' are packed
together in a tuple.  The reverse operation is also possible:

    >>> x, y, z = t

This is called, appropriately enough, _sequence unpacking_ and works
for any sequence on the right-hand side.  Sequence unpacking requires
the list of variables on the left to have the same number of elements
as the length of the sequence.  Note that multiple assignment is really
just a combination of tuple packing and sequence unpacking.

2.5.4 Sets
----------

Python also includes a data type for _sets_.  A set is an unordered
collection with no duplicate elements.  Basic uses include membership
testing and eliminating duplicate entries.  Set objects also support
mathematical operations like union, intersection, difference, and
symmetric difference.

  Here is a brief demonstration:

    >>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
    >>> fruit = set(basket)               # create a set without duplicates
    >>> fruit
    set(['orange', 'pear', 'apple', 'banana'])
    >>> 'orange' in fruit                 # fast membership testing
    True
    >>> 'crabgrass' in fruit
    False

    >>> # Demonstrate set operations on unique letters from two words
    ...
    >>> a = set('abracadabra')
    >>> b = set('alacazam')
    >>> a                                  # unique letters in a
    set(['a', 'r', 'b', 'c', 'd'])
    >>> a - b                              # letters in a but not in b
    set(['r', 'd', 'b'])
    >>> a | b                              # letters in either a or b
    set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
    >>> a & b                              # letters in both a and b
    set(['a', 'c'])
    >>> a ^ b                              # letters in a or b but not both
    set(['r', 'd', 'b', 'm', 'z', 'l'])

2.5.5 Dictionaries
------------------

Another useful data type built into Python is the _dictionary_ (see
Mapping Types -- dict). Dictionaries are sometimes found in
other languages as "associative memories" or "associative arrays".
Unlike sequences, which are indexed by a range of numbers, dictionaries
are indexed by _keys_, which can be any immutable type; strings and
numbers can always be keys.  Tuples can be used as keys if they contain
only strings, numbers, or tuples; if a tuple contains any mutable
object either directly or indirectly, it cannot be used as a key.  You
can't use lists as keys, since lists can be modified in place using
index assignments, slice assignments, or methods like `append()' and
`extend()'.

  It is best to think of a dictionary as an unordered set of _key:
value_ pairs, with the requirement that the keys are unique (within one
dictionary). A pair of braces creates an empty dictionary: `{}'.
Placing a comma-separated list of key:value pairs within the braces
adds initial key:value pairs to the dictionary; this is also the way
dictionaries are written on output.

  The main operations on a dictionary are storing a value with some key
and extracting the value given the key.  It is also possible to delete
a key:value pair with `del'. If you store using a key that is already
in use, the old value associated with that key is forgotten.  It is an
error to extract a value using a non-existent key.

  The `keys()' method of a dictionary object returns a list of all the
keys used in the dictionary, in arbitrary order (if you want it sorted,
just apply the sorted() function to it).  To check whether
a single key is in the dictionary, use the in keyword.

  Here is a small example using a dictionary:

    >>> tel = {'jack': 4098, 'sape': 4139}
    >>> tel['guido'] = 4127
    >>> tel
    {'sape': 4139, 'guido': 4127, 'jack': 4098}
    >>> tel['jack']
    4098
    >>> del tel['sape']
    >>> tel['irv'] = 4127
    >>> tel
    {'guido': 4127, 'irv': 4127, 'jack': 4098}
    >>> tel.keys()
    ['guido', 'irv', 'jack']
    >>> 'guido' in tel
    True

The dict() constructor builds dictionaries directly from
lists of key-value pairs stored as tuples.  When the pairs form a
pattern, list comprehensions can compactly specify the key-value list.

    >>> dict([('sape', 4139), ('guido', 4127), ('jack', 4098)])
    {'sape': 4139, 'jack': 4098, 'guido': 4127}
    >>> dict([(x, x**2) for x in (2, 4, 6)])     # use a list comprehension
    {2: 4, 4: 16, 6: 36}

Later in the tutorial, we will learn about Generator Expressions which
are even better suited for the task of supplying key-values pairs to
the dict().  constructor.

  When the keys are simple strings, it is sometimes easier to specify
pairs using keyword arguments:

    >>> dict(sape=4139, guido=4127, jack=4098)
    {'sape': 4139, 'jack': 4098, 'guido': 4127}

2.5.6 Looping Techniques
------------------------

When looping through dictionaries, the key and corresponding value can
be retrieved at the same time using the `iteritems()' method.

    >>> knights = {'gallahad': 'the pure', 'robin': 'the brave'}
    >>> for k, v in knights.iteritems():
    ...     print k, v
    ...
    gallahad the pure
    robin the brave

When looping through a sequence, the position index and corresponding
value can be retrieved at the same time using the enumerate()
function.

    >>> for i, v in enumerate(['tic', 'tac', 'toe']):
    ...     print i, v
    ...
    0 tic
    1 tac
    2 toe

To loop over two or more sequences at the same time, the entries can be
paired with the zip() function.

    >>> questions = ['name', 'quest', 'favorite color']
    >>> answers = ['lancelot', 'the holy grail', 'blue']
    >>> for q, a in zip(questions, answers):
    ...     print 'What is your {0}?  It is {1}.'.format(q, a)
    ...
    What is your name?  It is lancelot.
    What is your quest?  It is the holy grail.
    What is your favorite color?  It is blue.

To loop over a sequence in reverse, first specify the sequence in a
forward direction and then call the reversed() function.

    >>> for i in reversed(xrange(1,10,2)):
    ...     print i
    ...
    9
    7
    5
    3
    1

To loop over a sequence in sorted order, use the sorted().
function which returns a new sorted list while leaving the source
unaltered.

    >>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
    >>> for f in sorted(set(basket)):
    ...     print f
    ...
    apple
    banana
    orange
    pear

2.5.7 More on Conditions
------------------------

The conditions used in `while' and `if' statements can contain any
operators, not just comparisons.

  The comparison operators `in' and `not in' check whether a value
occurs (does not occur) in a sequence.  The operators `is' and `is not'
compare whether two objects are really the same object; this only
matters for mutable objects like lists.  All comparison operators have
the same priority, which is lower than that of all numerical operators.

  Comparisons can be chained.  For example, `a < b == c' tests whether
`a' is less than `b' and moreover `b' equals `c'.

  Comparisons may be combined using the Boolean operators `and' and
`or', and the outcome of a comparison (or of any other Boolean
expression) may be negated with `not'.  These have lower priorities
than comparison operators; between them, `not' has the highest priority
and `or' the lowest, so that `A and not B or C' is equivalent to `(A
and (not B)) or C'. As always, parentheses can be used to express the
desired composition.

  The Boolean operators `and' and `or' are so-called _short-circuit_
operators: their arguments are evaluated from left to right, and
evaluation stops as soon as the outcome is determined.  For example, if
`A' and `C' are true but `B' is false, `A and B and C' does not
evaluate the expression `C'.  When used as a general value and not as a
Boolean, the return value of a short-circuit operator is the last
evaluated argument.

  It is possible to assign the result of a comparison or other Boolean
expression to a variable.  For example,

    >>> string1, string2, string3 = '', 'Trondheim', 'Hammer Dance'
    >>> non_null = string1 or string2 or string3
    >>> non_null
    'Trondheim'

Note that in Python, unlike C, assignment cannot occur inside
expressions. C programmers may grumble about this, but it avoids a
common class of problems encountered in C programs: typing `=' in an
expression when `==' was intended.

2.5.8 Comparing Sequences and Other Types
-----------------------------------------

Sequence objects may be compared to other objects with the same
sequence type.  The comparison uses _lexicographical_ ordering: first
the first two items are compared, and if they differ this determines
the outcome of the comparison; if they are equal, the next two items
are compared, and so on, until either sequence is exhausted. If two
items to be compared are themselves sequences of the same type, the
lexicographical comparison is carried out recursively.  If all items of
two sequences compare equal, the sequences are considered equal.  If
one sequence is an initial sub-sequence of the other, the shorter
sequence is the smaller (lesser) one.  Lexicographical ordering for
strings uses the ASCII ordering for individual characters.  Some
examples of comparisons between sequences of the same type:

    (1, 2, 3)              < (1, 2, 4)
    [1, 2, 3]              < [1, 2, 4]
    'ABC' < 'C' < 'Pascal' < 'Python'
    (1, 2, 3, 4)           < (1, 2, 4)
    (1, 2)                 < (1, 2, -1)
    (1, 2, 3)             == (1.0, 2.0, 3.0)
    (1, 2, ('aa', 'ab'))   < (1, 2, ('abc', 'a'), 4)

Note that comparing objects of different types is legal.  The outcome is
deterministic but arbitrary: the types are ordered by their name. Thus,
a list is always smaller than a string, a string is always smaller than
a tuple, etc.  (1) Mixed numeric types are compared according to their
numeric value, so 0 equals 0.0, etc.

Footnotes
.........

     (1) The rules for comparing objects of different types should not
     be relied upon; they may change in a future version of the
     language.

2.6 Modules
===========

If you quit from the Python interpreter and enter it again, the
definitions you have made (functions and variables) are lost.
Therefore, if you want to write a somewhat longer program, you are
better off using a text editor to prepare the input for the interpreter
and running it with that file as input instead.  This is known as
creating a _script_.  As your program gets longer, you may want to
split it into several files for easier maintenance.  You may also want
to use a handy function that you've written in several programs without
copying its definition into each program.

  To support this, Python has a way to put definitions in a file and
use them in a script or in an interactive instance of the interpreter.
Such a file is called a _module_; definitions from a module can be
_imported_ into other modules or into the _main_ module (the collection
of variables that you have access to in a script executed at the top
level and in calculator mode).

  A module is a file containing Python definitions and statements.  The
file name is the module name with the suffix `.py' appended.  Within a
module, the module's name (as a string) is available as the value of
the global variable `__name__'.  For instance, use your favorite text
editor to create a file called `fibo.py' in the current directory with
the following contents:

    # Fibonacci numbers module

    def fib(n):    # write Fibonacci series up to n
        a, b = 0, 1
        while b < n:
            print b,
            a, b = b, a+b

    def fib2(n): # return Fibonacci series up to n
        result = []
        a, b = 0, 1
        while b < n:
            result.append(b)
            a, b = b, a+b
        return result

Now enter the Python interpreter and import this module with the
following command:

    >>> import fibo

This does not enter the names of the functions defined in `fibo'
directly in the current symbol table; it only enters the module name
`fibo' there. Using the module name you can access the functions:

    >>> fibo.fib(1000)
    1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
    >>> fibo.fib2(100)
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
    >>> fibo.__name__
    'fibo'

If you intend to use a function often you can assign it to a local name:

    >>> fib = fibo.fib
    >>> fib(500)
    1 1 2 3 5 8 13 21 34 55 89 144 233 377

2.6.1 More on Modules
---------------------

A module can contain executable statements as well as function
definitions.  These statements are intended to initialize the module.
They are executed only the _first_ time the module is imported
somewhere. (1)

  Each module has its own private symbol table, which is used as the
global symbol table by all functions defined in the module. Thus, the
author of a module can use global variables in the module without
worrying about accidental clashes with a user's global variables. On
the other hand, if you know what you are doing you can touch a module's
global variables with the same notation used to refer to its functions,
`modname.itemname'.

  Modules can import other modules.  It is customary but not required
to place all import statements at the beginning of a module
(or script, for that matter).  The imported module names are placed in
the importing module's global symbol table.

  There is a variant of the import statement that imports
names from a module directly into the importing module's symbol table.
For example:

    >>> from fibo import fib, fib2
    >>> fib(500)
    1 1 2 3 5 8 13 21 34 55 89 144 233 377

This does not introduce the module name from which the imports are
taken in the local symbol table (so in the example, `fibo' is not
defined).

  There is even a variant to import all names that a module defines:

    >>> from fibo import *
    >>> fib(500)
    1 1 2 3 5 8 13 21 34 55 89 144 233 377

This imports all names except those beginning with an underscore (`_').

  Note that in general the practice of importing `*' from a module or
package is frowned upon, since it often causes poorly readable code.
However, it is okay to use it to save typing in interactive sessions.

     Note: For efficiency reasons, each module is only imported once
     per interpreter session.  Therefore, if you change your modules,
     you must restart the interpreter - or, if it's just one module you
     want to test interactively, use reload(), e.g.
     `reload(modulename)'.

2.6.1.1 Executing modules as scripts
....................................

When you run a Python module with

    python fibo.py <arguments>

the code in the module will be executed, just as if you imported it,
but with the `__name__' set to `"__main__"'.  That means that by adding
this code at the end of your module:

    if __name__ == "__main__":
        import sys
        fib(int(sys.argv[1]))

you can make the file usable as a script as well as an importable
module, because the code that parses the command line only runs if the
module is executed as the "main" file:

    $ python fibo.py 50
    1 1 2 3 5 8 13 21 34

If the module is imported, the code is not run:

    >>> import fibo
    >>>

This is often used either to provide a convenient user interface to a
module, or for testing purposes (running the module as a script
executes a test suite).

2.6.1.2 The Module Search Path
..............................

When a module named `spam' is imported, the interpreter searches for a
file named `spam.py' in the current directory, and then in the list of
directories specified by the environment variable PYTHONPATH.
This has the same syntax as the shell variable `PATH', that is, a
list of directory names.  When PYTHONPATH is not set, or
when the file is not found there, the search continues in an
installation-dependent default path; on Unix, this is usually
`.:/usr/local/lib/python'.

  Actually, modules are searched in the list of directories given by
the variable `sys.path' which is initialized from the directory
containing the input script (or the current directory), see
PYTHONPATH and the installation- dependent default.  This allows
Python programs that know what they're doing to modify or replace the
module search path.  Note that because the directory containing the
script being run is on the search path, it is important that the script
not have the same name as a standard module, or Python will attempt to
load the script as a module when that module is imported. This will
generally be an error.  See section Standard Modules for
more information.

2.6.1.3 "Compiled" Python files
...............................

As an important speed-up of the start-up time for short programs that
use a lot of standard modules, if a file called `spam.pyc' exists in
the directory where `spam.py' is found, this is assumed to contain an
already-"byte-compiled" version of the module `spam'. The modification
time of the version of `spam.py' used to create `spam.pyc' is recorded
in `spam.pyc', and the `.pyc' file is ignored if these don't match.

  Normally, you don't need to do anything to create the `spam.pyc' file.
Whenever `spam.py' is successfully compiled, an attempt is made to write
the compiled version to `spam.pyc'.  It is not an error if this attempt
fails; if for any reason the file is not written completely, the
resulting `spam.pyc' file will be recognized as invalid and thus
ignored later.  The contents of the `spam.pyc' file are platform
independent, so a Python module directory can be shared by machines of
different architectures.

  Some tips for experts:

   * When the Python interpreter is invoked with the -O.
     flag, optimized code is generated and stored in `.pyo' files.  The
     optimizer currently doesn't help much; it only removes
     assert statements.  When -O is used, _all_
     bytecode is optimized; `.pyc' files are ignored and `.py'
     files are compiled to optimized bytecode.

   * Passing two -O flags to the Python interpreter (see
     -OO) will cause the bytecode compiler to perform
     optimizations that could in some rare cases result in
     malfunctioning programs.  Currently only `__doc__' strings are
     removed from the bytecode, resulting in more compact `.pyo' files.
     Since some programs may rely on having these available, you should
     only use this option if you know what you're doing.

   * A program doesn't run any faster when it is read from a `.pyc' or
     `.pyo' file than when it is read from a `.py' file; the only thing
     that's faster about `.pyc' or `.pyo' files is the speed with which
     they are loaded.

   * When a script is run by giving its name on the command line, the
     bytecode for the script is never written to a `.pyc' or `.pyo'
     file.  Thus, the startup time of a script may be reduced by moving
     most of its code to a module and having a small bootstrap script
     that imports that module.  It is also possible to name a `.pyc' or
     `.pyo' file directly on the command line.

   * It is possible to have a file called `spam.pyc' (or `spam.pyo'
     when -O is used) without a file `spam.py' for the same
     module.  This can be used to distribute a library of Python code
     in a form that is moderately hard to reverse engineer.

   * The module compileall can create `.pyc' files (or `.pyo'
     files when -O is used) for all modules in a directory.

2.6.2 Standard Modules
----------------------

Python comes with a library of standard modules, described in a separate
document, the Python Library Reference ("Library Reference" hereafter).
Some modules are built into the interpreter; these provide access to
operations that are not part of the core of the language but are
nevertheless built in, either for efficiency or to provide access to
operating system primitives such as system calls.  The set of such
modules is a configuration option which also depends on the underlying
platform For example, the `winreg' module is only provided on Windows
systems. One particular module deserves some attention: sys,
which is built into every Python interpreter.  The variables `sys.ps1'
and `sys.ps2' define the strings used as primary and secondary prompts:

    >>> import sys
    >>> sys.ps1
    '>>> '
    >>> sys.ps2
    '... '
    >>> sys.ps1 = 'C> '
    C> print 'Yuck!'
    Yuck!
    C>

These two variables are only defined if the interpreter is in
interactive mode.

  The variable `sys.path' is a list of strings that determines the
interpreter's search path for modules. It is initialized to a default
path taken from the environment variable PYTHONPATH, or from
a built-in default if PYTHONPATH is not set.  You can
modify it using standard list operations:

    >>> import sys
    >>> sys.path.append('/ufs/guido/lib/python')

2.6.3 The `dir()' Function
--------------------------

The built-in function dir() is used to find out which names
a module defines.  It returns a sorted list of strings:

    >>> import fibo, sys
    >>> dir(fibo)
    ['__name__', 'fib', 'fib2']
    >>> dir(sys)
    ['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__',
     '__stdin__', '__stdout__', '_getframe', 'api_version', 'argv',
     'builtin_module_names', 'byteorder', 'callstats', 'copyright',
     'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook',
     'exec_prefix', 'executable', 'exit', 'getdefaultencoding', 'getdlopenflags',
     'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode',
     'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache',
     'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags',
     'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout',
     'version', 'version_info', 'warnoptions']

Without arguments, see dir() lists the names you have defined
currently:

    >>> a = [1, 2, 3, 4, 5]
    >>> import fibo
    >>> fib = fibo.fib
    >>> dir()
    ['__builtins__', '__doc__', '__file__', '__name__', 'a', 'fib', 'fibo', 'sys']

Note that it lists all types of names: variables, modules, functions,
etc.

  dir() does not list the names of built-in functions and
variables.  If you want a list of those, they are defined in the
standard module __builtin__.:

    >>> import __builtin__
    >>> dir(__builtin__)
    ['ArithmeticError', 'AssertionError', 'AttributeError', 'DeprecationWarning',
     'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False',
     'FloatingPointError', 'FutureWarning', 'IOError', 'ImportError',
     'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt',
     'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented',
     'NotImplementedError', 'OSError', 'OverflowError',
     'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError',
     'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError',
     'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True',
     'TypeError', 'UnboundLocalError', 'UnicodeDecodeError',
     'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError',
     'UserWarning', 'ValueError', 'Warning', 'WindowsError',
     'ZeroDivisionError', '_', '__debug__', '__doc__', '__import__',
     '__name__', 'abs', 'apply', 'basestring', 'bool', 'buffer',
     'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile',
     'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod',
     'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float',
     'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex',
     'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter',
     'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'memoryview',
     'min', 'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit', 'range',
     'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set',
     'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super',
     'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']

2.6.4 Packages
--------------

Packages are a way of structuring Python's module namespace by using
"dotted module names".  For example, the module name `A.B' designates a
submodule named `B' in a package named `A'.  Just like the use of
modules saves the authors of different modules from having to worry
about each other's global variable names, the use of dotted module
names saves the authors of multi-module packages like NumPy or the
Python Imaging Library from having to worry about each other's module
names.

  Suppose you want to design a collection of modules (a "package") for
the uniform handling of sound files and sound data.  There are many
different sound file formats (usually recognized by their extension,
for example: `.wav', `.aiff', `.au'), so you may need to create and
maintain a growing collection of modules for the conversion between the
various file formats.  There are also many different operations you
might want to perform on sound data (such as mixing, adding echo,
applying an equalizer function, creating an artificial stereo effect),
so in addition you will be writing a never-ending stream of modules to
perform these operations.  Here's a possible structure for your package
(expressed in terms of a hierarchical filesystem):

    sound/                          Top-level package
          __init__.py               Initialize the sound package
          formats/                  Subpackage for file format conversions
                  __init__.py
                  wavread.py
                  wavwrite.py
                  aiffread.py
                  aiffwrite.py
                  auread.py
                  auwrite.py
                  ...
          effects/                  Subpackage for sound effects
                  __init__.py
                  echo.py
                  surround.py
                  reverse.py
                  ...
          filters/                  Subpackage for filters
                  __init__.py
                  equalizer.py
                  vocoder.py
                  karaoke.py
                  ...

When importing the package, Python searches through the directories on
`sys.path' looking for the package subdirectory.

  The `__init__.py' files are required to make Python treat the
directories as containing packages; this is done to prevent directories
with a common name, such as `string', from unintentionally hiding valid
modules that occur later on the module search path. In the simplest
case, `__init__.py' can just be an empty file, but it can also execute
initialization code for the package or set the `__all__' variable,
described later.

  Users of the package can import individual modules from the package,
for example:

    import sound.effects.echo

This loads the submodule `sound.effects.echo'.  It must be referenced
with its full name.

    sound.effects.echo.echofilter(input, output, delay=0.7, atten=4)

An alternative way of importing the submodule is:

    from sound.effects import echo

This also loads the submodule `echo', and makes it available without its
package prefix, so it can be used as follows:

    echo.echofilter(input, output, delay=0.7, atten=4)

Yet another variation is to import the desired function or variable
directly:

    from sound.effects.echo import echofilter

Again, this loads the submodule `echo', but this makes its function
`echofilter()' directly available:

    echofilter(input, output, delay=0.7, atten=4)

Note that when using `from package import item', the item can be either
a submodule (or subpackage) of the package, or some  other name defined
in the package, like a function, class or variable.  The `import'
statement first tests whether the item is defined in the package; if
not, it assumes it is a module and attempts to load it.  If it fails to
find it, an ImportError.  exception is raised.

  Contrarily, when using syntax like `import item.subitem.subsubitem',
each item except for the last must be a package; the last item can be a
module or a package but can't be a class or function or variable
defined in the previous item.

2.6.4.1 Importing * From a Package
..................................

Now what happens when the user writes `from sound.effects import *'?
Ideally, one would hope that this somehow goes out to the filesystem,
finds which submodules are present in the package, and imports them
all.  This could take a long time and importing sub-modules might have
unwanted side-effects that should only happen when the sub-module is
explicitly imported.

  The only solution is for the package author to provide an explicit
index of the package.  The import statement uses the
following convention: if a package's `__init__.py' code defines a list
named `__all__', it is taken to be the list of module names that should
be imported when `from package import *' is encountered.  It is up to
the package author to keep this list up-to-date when a new version of
the package is released.  Package authors may also decide not to
support it, if they don't see a use for importing * from their package.
For example, the file `sounds/effects/__init__.py' could contain the
following code:

    __all__ = ["echo", "surround", "reverse"]

This would mean that `from sound.effects import *' would import the
three named submodules of the `sound' package.

  If `__all__' is not defined, the statement `from sound.effects import
*' does _not_ import all submodules from the package `sound.effects'
into the current namespace; it only ensures that the package
`sound.effects' has been imported (possibly running any initialization
code in `__init__.py') and then imports whatever names are defined in
the package.  This includes any names defined (and submodules
explicitly loaded) by `__init__.py'.  It also includes any submodules
of the package that were explicitly loaded by previous import
statements.  Consider this code:

    import sound.effects.echo
    import sound.effects.surround
    from sound.effects import *

In this example, the `echo' and `surround' modules are imported in the
current namespace because they are defined in the `sound.effects'
package when the `from...import' statement is executed.  (This also
works when `__all__' is defined.)

  Although certain modules are designed to export only names that
follow certain patterns when you use `import *', it is still considered
bad practise in production code.

  Remember, there is nothing wrong with using `from Package import
specific_submodule'!  In fact, this is the recommended notation unless
the importing module needs to use submodules with the same name from
different packages.

2.6.4.2 Intra-package References
................................

The submodules often need to refer to each other.  For example, the
`surround' module might use the `echo' module.  In fact, such
references are so common that the import statement first
looks in the containing package before looking in the standard module
search path. Thus, the `surround' module can simply use `import echo'
or `from echo import echofilter'.  If the imported module is not found
in the current package (the package of which the current module is a
submodule), the import.  statement looks for a top-level
module with the given name.

  When packages are structured into subpackages (as with the `sound'
package in the example), you can use absolute imports to refer to
submodules of siblings packages.  For example, if the module
`sound.filters.vocoder' needs to use the `echo' module in the
`sound.effects' package, it can use `from sound.effects import echo'.

  Starting with Python 2.5, in addition to the implicit relative
imports described above, you can write explicit relative imports with
the `from module import name' form of import statement. These explicit
relative imports use leading dots to indicate the current and parent
packages involved in the relative import. From the `surround' module
for example, you might use:

    from . import echo
    from .. import formats
    from ..filters import equalizer

Note that both explicit and implicit relative imports are based on the
name of the current module. Since the name of the main module is always
`"__main__"', modules intended for use as the main module of a Python
application should always use absolute imports.

2.6.4.3 Packages in Multiple Directories
........................................

Packages support one more special attribute, `__path__'.  This is
initialized to be a list containing the name of the directory holding
the package's `__init__.py' before the code in that file is executed.
This variable can be modified; doing so affects future searches for
modules and subpackages contained in the package.

  While this feature is not often needed, it can be used to extend the
set of modules found in a package.

Footnotes
.........

     (1) In fact function definitions are also 'statements' that are
     'executed'; the execution of a module-level function enters the
     function name in the module's global symbol table.

2.7 Input and Output
====================

There are several ways to present the output of a program; data can be
printed in a human-readable form, or written to a file for future use.
This chapter will discuss some of the possibilities.

2.7.1 Fancier Output Formatting
-------------------------------

So far we've encountered two ways of writing values: _expression
statements_ and the print statement.  (A third way is using
the `write()' method of file objects; the standard output file can be
referenced as `sys.stdout'.  See the Library Reference for more
information on this.)

  Often you'll want more control over the formatting of your output
than simply printing space-separated values.  There are two ways to
format your output; the first way is to do all the string handling
yourself; using string slicing and concatenation operations you can
create any layout you can imagine.  The standard module string
contains some useful operations for padding strings to a given
column width; these will be discussed shortly.  The second way is to
use the str.format() method.

  One question remains, of course: how do you convert values to
strings? Luckily, Python has ways to convert any value to a string:
pass it to the repr().  or str() functions.

  The str() function is meant to return representations of
values which are fairly human-readable, while repr() is
meant to generate representations which can be read by the interpreter
(or will force a SyntaxError if there is not equivalent
syntax).  For objects which don't have a particular representation for
human consumption, see str() will return the same value as
repr().  Many values, such as numbers or structures like
lists and dictionaries, have the same representation using either
function.  Strings and floating point numbers, in particular, have two
distinct representations.

  Some examples:

    >>> s = 'Hello, world.'
    >>> str(s)
    'Hello, world.'
    >>> repr(s)
    "'Hello, world.'"
    >>> str(1.0/7.0)
    '0.142857142857'
    >>> repr(1.0/7.0)
    '0.14285714285714285'
    >>> x = 10 * 3.25
    >>> y = 200 * 200
    >>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...'
    >>> print s
    The value of x is 32.5, and y is 40000...
    >>> # The repr() of a string adds string quotes and backslashes:
    ... hello = 'hello, world\n'
    >>> hellos = repr(hello)
    >>> print hellos
    'hello, world\n'
    >>> # The argument to repr() may be any Python object:
    ... repr((x, y, ('spam', 'eggs')))
    "(32.5, 40000, ('spam', 'eggs'))"

Here are two ways to write a table of squares and cubes:

    >>> for x in range(1, 11):
    ...     print repr(x).rjust(2), repr(x*x).rjust(3),
    ...     # Note trailing comma on previous line
    ...     print repr(x*x*x).rjust(4)
    ...
     1   1    1
     2   4    8
     3   9   27
     4  16   64
     5  25  125
     6  36  216
     7  49  343
     8  64  512
     9  81  729
    10 100 1000

    >>> for x in range(1,11):
    ...     print '{0:2d} {1:3d} {2:4d}'.format(x, x*x, x*x*x)
    ...
     1   1    1
     2   4    8
     3   9   27
     4  16   64
     5  25  125
     6  36  216
     7  49  343
     8  64  512
     9  81  729
    10 100 1000

(Note that in the first example, one space between each column was
added by the way print works: it always adds spaces between
its arguments.)

  This example demonstrates the `rjust()' method of string objects,
which right-justifies a string in a field of a given width by padding
it with spaces on the left.  There are similar methods `ljust()' and
`center()'.  These methods do not write anything, they just return a
new string.  If the input string is too long, they don't truncate it,
but return it unchanged; this will mess up your column lay-out but
that's usually better than the alternative, which would be lying about
a value.  (If you really want truncation you can always add a slice
operation, as in `x.ljust(n)[:n]'.)

  There is another method, `zfill()', which pads a numeric string on
the left with zeros.  It understands about plus and minus signs:

    >>> '12'.zfill(5)
    '00012'
    >>> '-3.14'.zfill(7)
    '-003.14'
    >>> '3.14159265359'.zfill(5)
    '3.14159265359'

Basic usage of the str.format() method looks like this:

    >>> print 'We are the {} who say "{}!"'.format('knights', 'Ni')
    We are the knights who say "Ni!"

The brackets and characters within them (called format fields) are
replaced with the objects passed into the format() method.
A number in the brackets refers to the position of the object passed
into the format() method.

    >>> print '{0} and {1}'.format('spam', 'eggs')
    spam and eggs
    >>> print '{1} and {0}'.format('spam', 'eggs')
    eggs and spam

If keyword arguments are used in the format() method, their
values are referred to by using the name of the argument.

    >>> print 'This {food} is {adjective}.'.format(
    ...       food='spam', adjective='absolutely horrible')
    This spam is absolutely horrible.

Positional and keyword arguments can be arbitrarily combined:

    >>> print 'The story of {0}, {1}, and {other}.'.format('Bill', 'Manfred',
    ...                                                    other='Georg')
    The story of Bill, Manfred, and Georg.

`'!s'' (apply str()) and `'!r'' (apply repr())
can be used to convert the value before it is formatted.

    >>> import math
    >>> print 'The value of PI is approximately {}.'.format(math.pi)
    The value of PI is approximately 3.14159265359.
    >>> print 'The value of PI is approximately {!r}.'.format(math.pi)
    The value of PI is approximately 3.141592653589793.

An optional `':'' and format specifier can follow the field name. This
allows greater control over how the value is formatted.  The following
example truncates Pi to three places after the decimal.

    >>> import math
    >>> print 'The value of PI is approximately {0:.3f}.'.format(math.pi)
    The value of PI is approximately 3.142.

Passing an integer after the `':'' will cause that field to be a minimum
number of characters wide.  This is useful for making tables pretty.

    >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678}
    >>> for name, phone in table.items():
    ...     print '{0:10} ==> {1:10d}'.format(name, phone)
    ...
    Jack       ==>       4098
    Dcab       ==>       7678
    Sjoerd     ==>       4127

If you have a really long format string that you don't want to split
up, it would be nice if you could reference the variables to be
formatted by name instead of by position.  This can be done by simply
passing the dict and using square brackets `'[]'' to access the keys

    >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
    >>> print ('Jack: {0[Jack]:d}; Sjoerd: {0[Sjoerd]:d}; '
    ...        'Dcab: {0[Dcab]:d}'.format(table))
    Jack: 4098; Sjoerd: 4127; Dcab: 8637678

This could also be done by passing the table as keyword arguments with
the '**' notation.

    >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
    >>> print 'Jack: {Jack:d}; Sjoerd: {Sjoerd:d}; Dcab: {Dcab:d}'.format(**table)
    Jack: 4098; Sjoerd: 4127; Dcab: 8637678

This is particularly useful in combination with the new built-in
vars().  function, which returns a dictionary containing all local
variables.

  For a complete overview of string formatting with str.format(),
see Format String Syntax.

2.7.1.1 Old string formatting
.............................

The `%' operator can also be used for string formatting. It interprets
the left argument much like a `sprintf()'-style format string to be
applied to the right argument, and returns the string resulting from
this formatting operation. For example:

    >>> import math
    >>> print 'The value of PI is approximately %5.3f.' % math.pi
    The value of PI is approximately 3.142.

Since str.format() is quite new, a lot of Python code still
uses the `%' operator. However, because this old style of formatting
will eventually be removed from the language, see str.format().
should generally be used.

  More information can be found in the String Formatting Operations
section.

2.7.2 Reading and Writing Files
-------------------------------

open() returns a file object, and is most commonly used
with two arguments: `open(filename, mode)'.

    >>> f = open('/tmp/workfile', 'w')
    >>> print f
    <open file '/tmp/workfile', mode 'w' at 80a0960>

The first argument is a string containing the filename.  The second
argument is another string containing a few characters describing the
way in which the file will be used.  _mode_ can be `'r'' when the file
will only be read, `'w'' for only writing (an existing file with the
same name will be erased), and `'a'' opens the file for appending; any
data written to the file is automatically added to the end.  `'r+''
opens the file for both reading and writing. The _mode_ argument is
optional; `'r'' will be assumed if it's omitted.

  On Windows, `'b'' appended to the mode opens the file in binary mode,
so there are also modes like `'rb'', `'wb'', and `'r+b''.  Python on
Windows makes a distinction between text and binary files; the
end-of-line characters in text files are automatically altered slightly
when data is read or written.  This behind-the-scenes modification to
file data is fine for ASCII text files, but it'll corrupt binary data
like that in `JPEG' or `EXE' files.  Be very careful to use binary mode
when reading and writing such files.  On Unix, it doesn't hurt to
append a `'b'' to the mode, so you can use it platform-independently
for all binary files.

2.7.2.1 Methods of File Objects
...............................

The rest of the examples in this section will assume that a file object
called `f' has already been created.

  To read a file's contents, call `f.read(size)', which reads some
quantity of data and returns it as a string.  _size_ is an optional
numeric argument.  When _size_ is omitted or negative, the entire
contents of the file will be read and returned; it's your problem if
the file is twice as large as your machine's memory. Otherwise, at most
_size_ bytes are read and returned.  If the end of the file has been
reached, `f.read()' will return an empty string (`""').

    >>> f.read()
    'This is the entire file.\n'
    >>> f.read()
    ''

`f.readline()' reads a single line from the file; a newline character
(`\n') is left at the end of the string, and is only omitted on the
last line of the file if the file doesn't end in a newline.  This makes
the return value unambiguous; if `f.readline()' returns an empty
string, the end of the file has been reached, while a blank line is
represented by `'\n'', a string containing only a single newline.

    >>> f.readline()
    'This is the first line of the file.\n'
    >>> f.readline()
    'Second line of the file\n'
    >>> f.readline()
    ''

`f.readlines()' returns a list containing all the lines of data in the
file.  If given an optional parameter _sizehint_, it reads that many
bytes from the file and enough more to complete a line, and returns the
lines from that.  This is often used to allow efficient reading of a
large file by lines, but without having to load the entire file in
memory.  Only complete lines will be returned.

    >>> f.readlines()
    ['This is the first line of the file.\n', 'Second line of the file\n']

An alternative approach to reading lines is to loop over the file
object. This is memory efficient, fast, and leads to simpler code:

    >>> for line in f:
            print line,

    This is the first line of the file.
    Second line of the file

The alternative approach is simpler but does not provide as fine-grained
control.  Since the two approaches manage line buffering differently,
they should not be mixed.

  `f.write(string)' writes the contents of _string_ to the file,
returning `None'.

    >>> f.write('This is a test\n')

To write something other than a string, it needs to be converted to a
string first:

    >>> value = ('the answer', 42)
    >>> s = str(value)
    >>> f.write(s)

`f.tell()' returns an integer giving the file object's current position
in the file, measured in bytes from the beginning of the file.  To
change the file object's position, use `f.seek(offset, from_what)'.
The position is computed from adding _offset_ to a reference point; the
reference point is selected by the _from_what_ argument.  A _from_what_
value of 0 measures from the beginning of the file, 1 uses the current
file position, and 2 uses the end of the file as the reference point.
_from_what_ can be omitted and defaults to 0, using the beginning of
the file as the reference point.

    >>> f = open('/tmp/workfile', 'r+')
    >>> f.write('0123456789abcdef')
    >>> f.seek(5)     # Go to the 6th byte in the file
    >>> f.read(1)
    '5'
    >>> f.seek(-3, 2) # Go to the 3rd byte before the end
    >>> f.read(1)
    'd'

When you're done with a file, call `f.close()' to close it and free up
any system resources taken up by the open file.  After calling
`f.close()', attempts to use the file object will automatically fail.

    >>> f.close()
    >>> f.read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: I/O operation on closed file

It is good practice to use the with keyword when dealing
with file objects.  This has the advantage that the file is properly
closed after its suite finishes, even if an exception is raised on the
way.  It is also much shorter than writing equivalent try.
-finally blocks:

    >>> with open('/tmp/workfile', 'r') as f:
    ...     read_data = f.read()
    >>> f.closed
    True

File objects have some additional methods, such as isatty().
and truncate() which are less frequently used; consult the
Library Reference for a complete guide to file objects.

2.7.2.2 The `pickle' Module
...........................

Strings can easily be written to and read from a file. Numbers take a
bit more effort, since the `read()' method only returns strings, which
will have to be passed to a function like int(), which takes
a string like `'123'' and returns its numeric value 123.  However, when
you want to save more complex data types like lists, dictionaries, or
class instances, things get a lot more complicated.

  Rather than have users be constantly writing and debugging code to
save complicated data types, Python provides a standard module called
pickle.  This is an amazing module that can take almost any
Python object (even some forms of Python code!), and convert it to a
string representation; this process is called _pickling_.
Reconstructing the object from the string representation is called
_unpickling_.  Between pickling and unpickling, the string representing
the object may have been stored in a file or data, or sent over a
network connection to some distant machine.

  If you have an object `x', and a file object `f' that's been opened
for writing, the simplest way to pickle the object takes only one line
of code:

    pickle.dump(x, f)

To unpickle the object again, if `f' is a file object which has been
opened for reading:

    x = pickle.load(f)

(There are other variants of this, used when pickling many objects or
when you don't want to write the pickled data to a file; consult the
complete documentation for pickle in the Python Library
Reference.)

  pickle is the standard way to make Python objects which
can be stored and reused by other programs or by a future invocation of
the same program; the technical term for this is a _persistent_ object.
Because pickle is so widely used, many authors who write
Python extensions take care to ensure that new data types such as
matrices can be properly pickled and unpickled.

2.8 Errors and Exceptions
=========================

Until now error messages haven't been more than mentioned, but if you
have tried out the examples you have probably seen some.  There are (at
least) two distinguishable kinds of errors: _syntax errors_ and
_exceptions_.

2.8.1 Syntax Errors
-------------------

Syntax errors, also known as parsing errors, are perhaps the most
common kind of complaint you get while you are still learning Python:

    >>> while True print 'Hello world'
      File "<stdin>", line 1, in ?
        while True print 'Hello world'
                       ^
    SyntaxError: invalid syntax

The parser repeats the offending line and displays a little 'arrow'
pointing at the earliest point in the line where the error was
detected.  The error is caused by (or at least detected at) the token
_preceding_ the arrow: in the example, the error is detected at the
keyword print, since a colon (`':'') is missing before it.
File name and line number are printed so you know where to look in case
the input came from a script.

2.8.2 Exceptions
----------------

Even if a statement or expression is syntactically correct, it may
cause an error when an attempt is made to execute it. Errors detected
during execution are called _exceptions_ and are not unconditionally
fatal: you will soon learn how to handle them in Python programs.  Most
exceptions are not handled by programs, however, and result in error
messages as shown here:

    >>> 10 * (1/0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ZeroDivisionError: integer division or modulo by zero
    >>> 4 + spam*3
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    NameError: name 'spam' is not defined
    >>> '2' + 2
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: cannot concatenate 'str' and 'int' objects

The last line of the error message indicates what happened. Exceptions
come in different types, and the type is printed as part of the
message: the types in the example are ZeroDivisionError,
NameError and TypeError.  The string printed as
the exception type is the name of the built-in exception that occurred.
This is true for all built-in exceptions, but need not be true for
user-defined exceptions (although it is a useful convention). Standard
exception names are built-in identifiers (not reserved keywords).

  The rest of the line provides detail based on the type of exception
and what caused it.

  The preceding part of the error message shows the context where the
exception happened, in the form of a stack traceback. In general it
contains a stack traceback listing source lines; however, it will not
display lines read from standard input.

  Built-in Exceptions lists the built-in exceptions and
their meanings.

2.8.3 Handling Exceptions
-------------------------

It is possible to write programs that handle selected exceptions. Look
at the following example, which asks the user for input until a valid
integer has been entered, but allows the user to interrupt the program
(using `Control-C' or whatever the operating system supports); note
that a user-generated interruption is signalled by raising the
KeyboardInterrupt exception.

    >>> while True:
    ...     try:
    ...         x = int(raw_input("Please enter a number: "))
    ...         break
    ...     except ValueError:
    ...         print "Oops!  That was no valid number.  Try again..."
    ...

The try statement works as follows.

   * First, the _try clause_ (the statement(s) between the try
     and except keywords) is executed.

   * If no exception occurs, the _except clause_ is skipped and
     execution of the try statement is finished.

   * If an exception occurs during execution of the try clause, the
     rest of the clause is skipped.  Then if its type matches the
     exception named after the except keyword, the except
     clause is executed, and then execution continues after the
     try statement.

   * If an exception occurs which does not match the exception named in
     the except clause, it is passed on to outer try.
     statements; if no handler is found, it is an _unhandled exception_
     and execution stops with a message as shown above.

  A try statement may have more than one except clause, to
specify handlers for different exceptions.  At most one handler will be
executed.  Handlers only handle exceptions that occur in the
corresponding try clause, not in other handlers of the same try
statement.  An except clause may name multiple exceptions as a
parenthesized tuple, for example:

    ... except (RuntimeError, TypeError, NameError):
    ...     pass

The last except clause may omit the exception name(s), to serve as a
wildcard.  Use this with extreme caution, since it is easy to mask a
real programming error in this way!  It can also be used to print an
error message and then re-raise the exception (allowing a caller to
handle the exception as well):

    import sys

    try:
        f = open('myfile.txt')
        s = f.readline()
        i = int(s.strip())
    except IOError as (errno, strerror):
        print "I/O error({0}): {1}".format(errno, strerror)
    except ValueError:
        print "Could not convert data to an integer."
    except:
        print "Unexpected error:", sys.exc_info()[0]
        raise

The try... except statement has an optional
_else clause_, which, when present, must follow all except clauses.  It
is useful for code that must be executed if the try clause does not
raise an exception.  For example:

    for arg in sys.argv[1:]:
        try:
            f = open(arg, 'r')
        except IOError:
            print 'cannot open', arg
        else:
            print arg, 'has', len(f.readlines()), 'lines'
            f.close()

The use of the else clause is better than adding additional
code to the try clause because it avoids accidentally
catching an exception that wasn't raised by the code being protected by
the try...  except statement.

  When an exception occurs, it may have an associated value, also known
as the exception's _argument_. The presence and type of the argument
depend on the exception type.

  The except clause may specify a variable after the exception name (or
tuple).  The variable is bound to an exception instance with the
arguments stored in `instance.args'.  For convenience, the exception
instance defines __str__() so the arguments can be printed
directly without having to reference `.args'.

  One may also instantiate an exception first before raising it and add
any attributes to it as desired.

    >>> try:
    ...    raise Exception('spam', 'eggs')
    ... except Exception as inst:
    ...    print type(inst)     # the exception instance
    ...    print inst.args      # arguments stored in .args
    ...    print inst           # __str__ allows args to printed directly
    ...    x, y = inst          # __getitem__ allows args to be unpacked directly
    ...    print 'x =', x
    ...    print 'y =', y
    ...
    <type 'exceptions.Exception'>
    ('spam', 'eggs')
    ('spam', 'eggs')
    x = spam
    y = eggs

If an exception has an argument, it is printed as the last part
('detail') of the message for unhandled exceptions.

  Exception handlers don't just handle exceptions if they occur
immediately in the try clause, but also if they occur inside functions
that are called (even indirectly) in the try clause. For example:

    >>> def this_fails():
    ...     x = 1/0
    ...
    >>> try:
    ...     this_fails()
    ... except ZeroDivisionError as detail:
    ...     print 'Handling run-time error:', detail
    ...
    Handling run-time error: integer division or modulo by zero

2.8.4 Raising Exceptions
------------------------

The raise statement allows the programmer to force a
specified exception to occur. For example:

    >>> raise NameError('HiThere')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    NameError: HiThere

The sole argument to raise indicates the exception to be
raised.  This must be either an exception instance or an exception
class (a class that derives from `Exception').

  If you need to determine whether an exception was raised but don't
intend to handle it, a simpler form of the raise statement
allows you to re-raise the exception:

    >>> try:
    ...     raise NameError('HiThere')
    ... except NameError:
    ...     print 'An exception flew by!'
    ...     raise
    ...
    An exception flew by!
    Traceback (most recent call last):
      File "<stdin>", line 2, in ?
    NameError: HiThere

2.8.5 User-defined Exceptions
-----------------------------

Programs may name their own exceptions by creating a new exception
class (see Classes for more about Python classes).
Exceptions should typically be derived from the Exception.
class, either directly or indirectly.  For example:

    >>> class MyError(Exception):
    ...     def __init__(self, value):
    ...         self.value = value
    ...     def __str__(self):
    ...         return repr(self.value)
    ...
    >>> try:
    ...     raise MyError(2*2)
    ... except MyError as e:
    ...     print 'My exception occurred, value:', e.value
    ...
    My exception occurred, value: 4
    >>> raise MyError('oops!')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    __main__.MyError: 'oops!'

In this example, the default __init__() of `Exception' has
been overridden.  The new behavior simply creates the _value_
attribute.  This replaces the default behavior of creating the _args_
attribute.

  Exception classes can be defined which do anything any other class
can do, but are usually kept simple, often only offering a number of
attributes that allow information about the error to be extracted by
handlers for the exception.  When creating a module that can raise
several distinct errors, a common practice is to create a base class
for exceptions defined by that module, and subclass that to create
specific exception classes for different error conditions:

    class Error(Exception):
        """Base class for exceptions in this module."""
        pass

    class InputError(Error):
        """Exception raised for errors in the input.

        Attributes:
            expr -- input expression in which the error occurred
            msg  -- explanation of the error
        """

        def __init__(self, expr, msg):
            self.expr = expr
            self.msg = msg

    class TransitionError(Error):
        """Raised when an operation attempts a state transition that's not
        allowed.

        Attributes:
            prev -- state at beginning of transition
            next -- attempted new state
            msg  -- explanation of why the specific transition is not allowed
        """

        def __init__(self, prev, next, msg):
            self.prev = prev
            self.next = next
            self.msg = msg

Most exceptions are defined with names that end in "Error," similar to
the naming of the standard exceptions.

  Many standard modules define their own exceptions to report errors
that may occur in functions they define.  More information on classes
is presented in chapter Classes.

2.8.6 Defining Clean-up Actions
-------------------------------

The try statement has another optional clause which is
intended to define clean-up actions that must be executed under all
circumstances.  For example:

    >>> try:
    ...     raise KeyboardInterrupt
    ... finally:
    ...     print 'Goodbye, world!'
    ...
    Goodbye, world!
    Traceback (most recent call last):
      File "<stdin>", line 2, in ?
    KeyboardInterrupt

A _finally clause_ is always executed before leaving the try.
statement, whether an exception has occurred or not. When an exception
has occurred in the try clause and has not been handled by
an except clause (or it has occurred in a except
or else clause), it is re-raised after the
finally clause has been executed.  The finally clause
is also executed "on the way out" when any other clause of the
try statement is left via a break, continue
or return statement.  A more complicated example
(having except and finally clauses in the same
try statement works as of Python 2.5):

    >>> def divide(x, y):
    ...     try:
    ...         result = x / y
    ...     except ZeroDivisionError:
    ...         print "division by zero!"
    ...     else:
    ...         print "result is", result
    ...     finally:
    ...         print "executing finally clause"
    ...
    >>> divide(2, 1)
    result is 2
    executing finally clause
    >>> divide(2, 0)
    division by zero!
    executing finally clause
    >>> divide("2", "1")
    executing finally clause
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "<stdin>", line 3, in divide
    TypeError: unsupported operand type(s) for /: 'str' and 'str'

As you can see, the finally clause is executed in any
event.  The TypeError raised by dividing two strings is not
handled by the except clause and therefore re-raised after
the finally.  clause has been executed.

  In real world applications, the finally clause is useful
for releasing external resources (such as files or network
connections), regardless of whether the use of the resource was
successful.

2.8.7 Predefined Clean-up Actions
---------------------------------

Some objects define standard clean-up actions to be undertaken when the
object is no longer needed, regardless of whether or not the operation
using the object succeeded or failed. Look at the following example,
which tries to open a file and print its contents to the screen.

    for line in open("myfile.txt"):
        print line

The problem with this code is that it leaves the file open for an
indeterminate amount of time after the code has finished executing.
This is not an issue in simple scripts, but can be a problem for larger
applications. The with statement allows objects like files
to be used in a way that ensures they are always cleaned up promptly
and correctly.

    with open("myfile.txt") as f:
        for line in f:
            print line

After the statement is executed, the file _f_ is always closed, even if
a problem was encountered while processing the lines. Other objects
which provide predefined clean-up actions will indicate this in their
documentation.

2.9 Classes
===========

Compared with other programming languages, Python's class mechanism
adds classes with a minimum of new syntax and semantics.  It is a
mixture of the class mechanisms found in C++ and Modula-3.  Python
classes provide all the standard features of Object Oriented
Programming: the class inheritance mechanism allows multiple base
classes, a derived class can override any methods of its base class or
classes, and a method can call the method of a base class with the same
name.  Objects can contain arbitrary amounts and kinds of data.  As is
true for modules, classes partake of the dynamic nature of Python: they
are created at runtime, and can be modified further after creation.

  In C++ terminology, normally class members (including the data
members) are _public_ (except see below Private Variables),
and all member functions are _virtual_.  As in Modula-3, there are no
shorthands for referencing the object's members from its methods: the
method function is declared with an explicit first argument
representing the object, which is provided implicitly by the call.  As
in Smalltalk, classes themselves are objects.  This provides semantics
for importing and renaming.  Unlike C++ and Modula-3, built-in types
can be used as base classes for extension by the user.  Also, like in
C++, most built-in operators with special syntax (arithmetic operators,
subscripting etc.) can be redefined for class instances.

  (Lacking universally accepted terminology to talk about classes, I
will make occasional use of Smalltalk and C++ terms.  I would use
Modula-3 terms, since its object-oriented semantics are closer to those
of Python than C++, but I expect that few readers have heard of it.)

2.9.1 A Word About Names and Objects
------------------------------------

Objects have individuality, and multiple names (in multiple scopes) can
be bound to the same object.  This is known as aliasing in other
languages.  This is usually not appreciated on a first glance at
Python, and can be safely ignored when dealing with immutable basic
types (numbers, strings, tuples).  However, aliasing has a possibly
surprising effect on the semantics of Python code involving mutable
objects such as lists, dictionaries, and most other types.  This is
usually used to the benefit of the program, since aliases behave like
pointers in some respects.  For example, passing an object is cheap
since only a pointer is passed by the implementation; and if a function
modifies an object passed as an argument, the caller will see the
change -- this eliminates the need for two different argument passing
mechanisms as in Pascal.

2.9.2 Python Scopes and Namespaces
----------------------------------

Before introducing classes, I first have to tell you something about
Python's scope rules.  Class definitions play some neat tricks with
namespaces, and you need to know how scopes and namespaces work to
fully understand what's going on.  Incidentally, knowledge about this
subject is useful for any advanced Python programmer.

  Let's begin with some definitions.

  A _namespace_ is a mapping from names to objects.  Most namespaces
are currently implemented as Python dictionaries, but that's normally
not noticeable in any way (except for performance), and it may change
in the future.  Examples of namespaces are: the set of built-in names
(containing functions such as abs(), and built-in exception
names); the global names in a module; and the local names in a function
invocation.  In a sense the set of attributes of an object also form a
namespace.  The important thing to know about namespaces is that there
is absolutely no relation between names in different namespaces; for
instance, two different modules may both define a function `maximize'
without confusion -- users of the modules must prefix it with the
module name.

  By the way, I use the word _attribute_ for any name following a dot
-- for example, in the expression `z.real', `real' is an attribute of
the object `z'.  Strictly speaking, references to names in modules are
attribute references: in the expression `modname.funcname', `modname'
is a module object and `funcname' is an attribute of it.  In this case
there happens to be a straightforward mapping between the module's
attributes and the global names defined in the module: they share the
same namespace!  (1)

  Attributes may be read-only or writable.  In the latter case,
assignment to attributes is possible.  Module attributes are writable:
you can write `modname.the_answer = 42'.  Writable attributes may also
be deleted with the del statement.  For example, `del
modname.the_answer' will remove the attribute `the_answer' from the
object named by `modname'.

  Namespaces are created at different moments and have different
lifetimes.  The namespace containing the built-in names is created when
the Python interpreter starts up, and is never deleted.  The global
namespace for a module is created when the module definition is read
in; normally, module namespaces also last until the interpreter quits.
The statements executed by the top-level invocation of the interpreter,
either read from a script file or interactively, are considered part of
a module called __main__, so they have their own global
namespace.  (The built-in names actually also live in a module; this is
called __builtin__)

  The local namespace for a function is created when the function is
called, and deleted when the function returns or raises an exception
that is not handled within the function.  (Actually, forgetting would
be a better way to describe what actually happens.)  Of course,
recursive invocations each have their own local namespace.

  A _scope_ is a textual region of a Python program where a namespace
is directly accessible.  "Directly accessible" here means that an
unqualified reference to a name attempts to find the name in the
namespace.

  Although scopes are determined statically, they are used dynamically.
At any time during execution, there are at least three nested scopes
whose namespaces are directly accessible:

   * the innermost scope, which is searched first, contains the local
     names

   * the scopes of any enclosing functions, which are searched starting
     with the nearest enclosing scope, contains non-local, but also
     non-global names

   * the next-to-last scope contains the current module's global names

   * the outermost scope (searched last) is the namespace containing
     built-in names

  If a name is declared global, then all references and assignments go
directly to the middle scope containing the module's global names.
Otherwise, all variables found outside of the innermost scope are
read-only (an attempt to write to such a variable will simply create a
_new_ local variable in the innermost scope, leaving the identically
named outer variable unchanged).

  Usually, the local scope references the local names of the
(textually) current function.  Outside functions, the local scope
references the same namespace as the global scope: the module's
namespace. Class definitions place yet another namespace in the local
scope.

  It is important to realize that scopes are determined textually: the
global scope of a function defined in a module is that module's
namespace, no matter from where or by what alias the function is
called.  On the other hand, the actual search for names is done
dynamically, at run time -- however, the language definition is
evolving towards static name resolution, at "compile" time, so don't
rely on dynamic name resolution!  (In fact, local variables are already
determined statically.)

  A special quirk of Python is that - if no global.
statement is in effect - assignments to names always go into the
innermost scope.  Assignments do not copy data -- they just bind names
to objects.  The same is true for deletions: the statement `del x'
removes the binding of `x' from the namespace referenced by the local
scope.  In fact, all operations that introduce new names use the local
scope: in particular, see import statements and function
definitions bind the module or function name in the local scope.  (The
global statement can be used to indicate that particular
variables live in the global scope.)

2.9.3 A First Look at Classes
-----------------------------

Classes introduce a little bit of new syntax, three new object types,
and some new semantics.

2.9.3.1 Class Definition Syntax
...............................

The simplest form of class definition looks like this:

    class ClassName:
        <statement-1>
        .
        .
        .
        <statement-N>

Class definitions, like function definitions (see def.
statements) must be executed before they have any effect.  (You could
conceivably place a class definition in a branch of an if.
statement, or inside a function.)

  In practice, the statements inside a class definition will usually be
function definitions, but other statements are allowed, and sometimes
useful -- we'll come back to this later.  The function definitions
inside a class normally have a peculiar form of argument list, dictated
by the calling conventions for methods -- again, this is explained
later.

  When a class definition is entered, a new namespace is created, and
used as the local scope -- thus, all assignments to local variables go
into this new namespace.  In particular, function definitions bind the
name of the new function here.

  When a class definition is left normally (via the end), a _class
object_ is created.  This is basically a wrapper around the contents of
the namespace created by the class definition; we'll learn more about
class objects in the next section.  The original local scope (the one
in effect just before the class definition was entered) is reinstated,
and the class object is bound here to the class name given in the class
definition header (`ClassName' in the example).

2.9.3.2 Class Objects
.....................

Class objects support two kinds of operations: attribute references and
instantiation.

  _Attribute references_ use the standard syntax used for all attribute
references in Python: `obj.name'.  Valid attribute names are all the
names that were in the class's namespace when the class object was
created.  So, if the class definition looked like this:

    class MyClass:
        """A simple example class"""
        i = 12345
        def f(self):
            return 'hello world'

then `MyClass.i' and `MyClass.f' are valid attribute references,
returning an integer and a function object, respectively. Class
attributes can also be assigned to, so you can change the value of
`MyClass.i' by assignment.  `__doc__' is also a valid attribute,
returning the docstring belonging to the class: `"A simple example
class"'.

  Class _instantiation_ uses function notation.  Just pretend that the
class object is a parameterless function that returns a new instance of
the class.  For example (assuming the above class):

    x = MyClass()

creates a new _instance_ of the class and assigns this object to the
local variable `x'.

  The instantiation operation ("calling" a class object) creates an
empty object.  Many classes like to create objects with instances
customized to a specific initial state. Therefore a class may define a
special method named __init__(), like this:

    def __init__(self):
        self.data = []

When a class defines an __init__() method, class
instantiation automatically invokes __init__() for the
newly-created class instance.  So in this example, a new, initialized
instance can be obtained by:

    x = MyClass()

Of course, the __init__() method may have arguments for
greater flexibility.  In that case, arguments given to the class
instantiation operator are passed on to __init__().  For
example,

    >>> class Complex:
    ...     def __init__(self, realpart, imagpart):
    ...         self.r = realpart
    ...         self.i = imagpart
    ...
    >>> x = Complex(3.0, -4.5)
    >>> x.r, x.i
    (3.0, -4.5)

2.9.3.3 Instance Objects
........................

Now what can we do with instance objects?  The only operations
understood by instance objects are attribute references.  There are two
kinds of valid attribute names, data attributes and methods.

  _data attributes_ correspond to "instance variables" in Smalltalk,
and to "data members" in C++.  Data attributes need not be declared;
like local variables, they spring into existence when they are first
assigned to.  For example, if `x' is the instance of `MyClass' created
above, the following piece of code will print the value `16', without
leaving a trace:

    x.counter = 1
    while x.counter < 10:
        x.counter = x.counter * 2
    print x.counter
    del x.counter

The other kind of instance attribute reference is a _method_. A method
is a function that "belongs to" an object.  (In Python, the term method
is not unique to class instances: other object types can have methods
as well.  For example, list objects have methods called append, insert,
remove, sort, and so on.  However, in the following discussion, we'll
use the term method exclusively to mean methods of class instance
objects, unless explicitly stated otherwise.)

  Valid method names of an instance object depend on its class.  By
definition, all attributes of a class that are function  objects define
corresponding methods of its instances.  So in our example, `x.f' is a
valid method reference, since `MyClass.f' is a function, but `x.i' is
not, since `MyClass.i' is not.  But `x.f' is not the same thing as
`MyClass.f' -- it is a _method object_, not a function object.

2.9.3.4 Method Objects
......................

Usually, a method is called right after it is bound:

    x.f()

In the `MyClass' example, this will return the string `'hello world''.
However, it is not necessary to call a method right away: `x.f' is a
method object, and can be stored away and called at a later time.  For
example:

    xf = x.f
    while True:
        print xf()

will continue to print `hello world' until the end of time.

  What exactly happens when a method is called?  You may have noticed
that `x.f()' was called without an argument above, even though the
function definition for `f()' specified an argument.  What happened to
the argument?  Surely Python raises an exception when a function that
requires an argument is called without any -- even if the argument
isn't actually used...

  Actually, you may have guessed the answer: the special thing about
methods is that the object is passed as the first argument of the
function.  In our example, the call `x.f()' is exactly equivalent to
`MyClass.f(x)'.  In general, calling a method with a list of _n_
arguments is equivalent to calling the corresponding function with an
argument list that is created by inserting the method's object before
the first argument.

  If you still don't understand how methods work, a look at the
implementation can perhaps clarify matters.  When an instance attribute
is referenced that isn't a data attribute, its class is searched.  If
the name denotes a valid class attribute that is a function object, a
method object is created by packing (pointers to) the instance object
and the function object just found together in an abstract object: this
is the method object.  When the method object is called with an
argument list, a new argument list is constructed from the instance
object and the argument list, and the function object is called with
this new argument list.

2.9.4 Random Remarks
--------------------

Data attributes override method attributes with the same name; to avoid
accidental name conflicts, which may cause hard-to-find bugs in large
programs, it is wise to use some kind of convention that minimizes the
chance of conflicts.  Possible conventions include capitalizing method
names, prefixing data attribute names with a small unique string
(perhaps just an underscore), or using verbs for methods and nouns for
data attributes.

  Data attributes may be referenced by methods as well as by ordinary
users ("clients") of an object.  In other words, classes are not usable
to implement pure abstract data types.  In fact, nothing in Python
makes it possible to enforce data hiding -- it is all based upon
convention.  (On the other hand, the Python implementation, written in
C, can completely hide implementation details and control access to an
object if necessary; this can be used by extensions to Python written
in C.)

  Clients should use data attributes with care -- clients may mess up
invariants maintained by the methods by stamping on their data
attributes.  Note that clients may add data attributes of their own to
an instance object without affecting the validity of the methods, as
long as name conflicts are avoided -- again, a naming convention can
save a lot of headaches here.

  There is no shorthand for referencing data attributes (or other
methods!) from within methods.  I find that this actually increases the
readability of methods: there is no chance of confusing local variables
and instance variables when glancing through a method.

  Often, the first argument of a method is called `self'.  This is
nothing more than a convention: the name `self' has absolutely no
special meaning to Python.  Note, however, that by not following the
convention your code may be less readable to other Python programmers,
and it is also conceivable that a _class browser_ program might be
written that relies upon such a convention.

  Any function object that is a class attribute defines a method for
instances of that class.  It is not necessary that the function
definition is textually enclosed in the class definition: assigning a
function object to a local variable in the class is also ok.  For
example:

    # Function defined outside the class
    def f1(self, x, y):
        return min(x, x+y)

    class C:
        f = f1
        def g(self):
            return 'hello world'
        h = g

Now `f', `g' and `h' are all attributes of class `C' that refer to
function objects, and consequently they are all methods of instances of
`C' -- `h' being exactly equivalent to `g'.  Note that this practice
usually only serves to confuse the reader of a program.

  Methods may call other methods by using method attributes of the
`self' argument:

    class Bag:
        def __init__(self):
            self.data = []
        def add(self, x):
            self.data.append(x)
        def addtwice(self, x):
            self.add(x)
            self.add(x)

Methods may reference global names in the same way as ordinary
functions.  The global scope associated with a method is the module
containing the class definition.  (The class itself is never used as a
global scope.)  While one rarely encounters a good reason for using
global data in a method, there are many legitimate uses of the global
scope: for one thing, functions and modules imported into the global
scope can be used by methods, as well as functions and classes defined
in it.  Usually, the class containing the method is itself defined in
this global scope, and in the next section we'll find some good reasons
why a method would want to reference its own class.

  Each value is an object, and therefore has a _class_ (also called its
_type_).  It is stored as `object.__class__'.

2.9.5 Inheritance
-----------------

Of course, a language feature would not be worthy of the name "class"
without supporting inheritance.  The syntax for a derived class
definition looks like this:

    class DerivedClassName(BaseClassName):
        <statement-1>
        .
        .
        .
        <statement-N>

The name `BaseClassName' must be defined in a scope containing the
derived class definition.  In place of a base class name, other
arbitrary expressions are also allowed.  This can be useful, for
example, when the base class is defined in another module:

    class DerivedClassName(modname.BaseClassName):

Execution of a derived class definition proceeds the same as for a base
class.  When the class object is constructed, the base class is
remembered.  This is used for resolving attribute references: if a
requested attribute is not found in the class, the search proceeds to
look in the base class.  This rule is applied recursively if the base
class itself is derived from some other class.

  There's nothing special about instantiation of derived classes:
`DerivedClassName()' creates a new instance of the class.  Method
references are resolved as follows: the corresponding class attribute
is searched, descending down the chain of base classes if necessary,
and the method reference is valid if this yields a function object.

  Derived classes may override methods of their base classes.  Because
methods have no special privileges when calling other methods of the
same object, a method of a base class that calls another method defined
in the same base class may end up calling a method of a derived class
that overrides it.  (For C++ programmers: all methods in Python are
effectively `virtual'.)

  An overriding method in a derived class may in fact want to extend
rather than simply replace the base class method of the same name.
There is a simple way to call the base class method directly: just call
`BaseClassName.methodname(self, arguments)'.  This is occasionally
useful to clients as well.  (Note that this only works if the base
class is accessible as `BaseClassName' in the global scope.)

  Python has two built-in functions that work with inheritance:

   * Use isinstance() to check an instance's type:
     `isinstance(obj, int)' will be `True' only if `obj.__class__' is
     int or some class derived from int.

   * Use issubclass() to check class inheritance:
     `issubclass(bool, int)' is `True' since bool is a
     subclass of int.  However, `issubclass(unicode, str)'
     is `False' since unicode is not a subclass of
     str. (they only share a common ancestor, see basestring).

2.9.5.1 Multiple Inheritance
............................

Python supports a limited form of multiple inheritance as well.  A class
definition with multiple base classes looks like this:

    class DerivedClassName(Base1, Base2, Base3):
        <statement-1>
        .
        .
        .
        <statement-N>

For old-style classes, the only rule is depth-first, left-to-right.
Thus, if an attribute is not found in `DerivedClassName', it is
searched in `Base1', then (recursively) in the base classes of `Base1',
and only if it is not found there, it is searched in `Base2', and so on.

  (To some people breadth first -- searching `Base2' and `Base3' before
the base classes of `Base1' -- looks more natural.  However, this would
require you to know whether a particular attribute of `Base1' is
actually defined in `Base1' or in one of its base classes before you can
figure out the consequences of a name conflict with an attribute of
`Base2'.  The depth-first rule makes no differences between direct and
inherited attributes of `Base1'.)

  For new-style class.es, the method resolution order
changes dynamically to support cooperative calls to super().
This approach is known in some other multiple-inheritance languages as
call-next-method and is more powerful than the super call found in
single-inheritance languages.

  With new-style classes, dynamic ordering is necessary because all
cases of multiple inheritance exhibit one or more diamond relationships
(where one at least one of the parent classes can be accessed through
multiple paths from the bottommost class).  For example, all new-style
classes inherit from object, so any case of multiple
inheritance provides more than one path to reach object.  To
keep the base classes from being accessed more than once, the dynamic
algorithm linearizes the search order in a way that preserves the
left-to-right ordering specified in each class, that calls each parent
only once, and that is monotonic (meaning that a class can be subclassed
without affecting the precedence order of its parents).  Taken
together, these properties make it possible to design reliable and
extensible classes with multiple inheritance.  For more detail, see
<http://www.python.org/download/releases/2.3/mro/>.

2.9.6 Private Variables
-----------------------

"Private" instance variables that cannot be accessed except from inside
an object don't exist in Python.  However, there is a convention that
is followed by most Python code: a name prefixed with an underscore
(e.g. `_spam') should be treated as a non-public part of the API
(whether it is a function, a method or a data member).  It should be
considered an implementation detail and subject to change without
notice.

  Since there is a valid use-case for class-private members (namely to
avoid name clashes of names with names defined by subclasses), there is
limited support for such a mechanism, called _name mangling_.  Any
identifier of the form `__spam' (at least two leading underscores, at
most one trailing underscore) is textually replaced with
`_classname__spam', where `classname' is the current class name with
leading underscore(s) stripped.  This mangling is done without regard
to the syntactic position of the identifier, as long as it occurs
within the definition of a class.

  Note that the mangling rules are designed mostly to avoid accidents;
it still is possible to access or modify a variable that is considered
private.  This can even be useful in special circumstances, such as in
the debugger.

  Notice that code passed to `exec', `eval()' or `execfile()' does not
consider the classname of the invoking  class to be the current class;
this is similar to the effect of the  `global' statement, the effect of
which is likewise restricted to  code that is byte-compiled together.
The same restriction applies to `getattr()', `setattr()' and
`delattr()', as well as when referencing `__dict__' directly.

2.9.7 Odds and Ends
-------------------

Sometimes it is useful to have a data type similar to the Pascal
"record" or C "struct", bundling together a few named data items.  An
empty class definition will do nicely:

    class Employee:
        pass

    john = Employee() # Create an empty employee record

    # Fill the fields of the record
    john.name = 'John Doe'
    john.dept = 'computer lab'
    john.salary = 1000

A piece of Python code that expects a particular abstract data type can
often be passed a class that emulates the methods of that data type
instead.  For instance, if you have a function that formats some data
from a file object, you can define a class with methods `read()' and
readline() that get the data from a string buffer instead,
and pass it as an argument.

  Instance method objects have attributes, too: `m.im_self' is the
instance object with the method `m()', and `m.im_func' is the function
object corresponding to the method.

2.9.8 Exceptions Are Classes Too
--------------------------------

User-defined exceptions are identified by classes as well.  Using this
mechanism it is possible to create extensible hierarchies of exceptions.

  There are two new valid (semantic) forms for the raise.
statement:

    raise Class, instance

    raise instance

In the first form, `instance' must be an instance of `Class' or of a
class derived from it.  The second form is a shorthand for:

    raise instance.__class__, instance

A class in an except clause is compatible with an exception
if it is the same class or a base class thereof (but not the other way
around -- an except clause listing a derived class is not compatible
with a base class).  For example, the following code will print B, C, D
in that order:

    class B:
        pass
    class C(B):
        pass
    class D(C):
        pass

    for c in [B, C, D]:
        try:
            raise c()
        except D:
            print "D"
        except C:
            print "C"
        except B:
            print "B"

Note that if the except clauses were reversed (with `except B' first),
it would have printed B, B, B -- the first matching except clause is
triggered.

  When an error message is printed for an unhandled exception, the
exception's class name is printed, then a colon and a space, and
finally the instance converted to a string using the built-in function
str().

2.9.9 Iterators
---------------

By now you have probably noticed that most container objects can be
looped over using a for statement:

    for element in [1, 2, 3]:
        print element
    for element in (1, 2, 3):
        print element
    for key in {'one':1, 'two':2}:
        print key
    for char in "123":
        print char
    for line in open("myfile.txt"):
        print line

This style of access is clear, concise, and convenient.  The use of
iterators pervades and unifies Python.  Behind the scenes, the
for statement calls iter() on the container object.
The function returns an iterator object that defines the method
next() which accesses elements in the container one at a time.
When there are no more elements, see next() raises a
StopIteration exception which tells the for loop to
terminate.  This example shows how it all works:

    >>> s = 'abc'
    >>> it = iter(s)
    >>> it
    <iterator object at 0x00A1DB50>
    >>> it.next()
    'a'
    >>> it.next()
    'b'
    >>> it.next()
    'c'
    >>> it.next()

    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
        it.next()
    StopIteration

Having seen the mechanics behind the iterator protocol, it is easy to
add iterator behavior to your classes.  Define an __iter__()
method which returns an object with a next() method.
If the class defines next(), then __iter__() can
just return `self':

    class Reverse:
        "Iterator for looping over a sequence backwards"
        def __init__(self, data):
            self.data = data
            self.index = len(data)
        def __iter__(self):
            return self
        def next(self):
            if self.index == 0:
                raise StopIteration
            self.index = self.index - 1
            return self.data[self.index]

    >>> rev = Reverse('spam')
    >>> iter(rev)
    <__main__.Reverse object at 0x00A1DB50>
    >>> for char in rev:
    ...     print char
    ...
    m
    a
    p
    s

2.9.10 Generators
-----------------

See Generator.s are a simple and powerful tool for creating
iterators.  They are written like regular functions but use the
yield statement whenever they want to return data.  Each time
next() is called, the generator resumes where it left-off
(it remembers all the data values and which statement was last
executed).  An example shows that generators can be trivially easy to
create:

    def reverse(data):
        for index in range(len(data)-1, -1, -1):
            yield data[index]

    >>> for char in reverse('golf'):
    ...     print char
    ...
    f
    l
    o
    g

Anything that can be done with generators can also be done with class
based iterators as described in the previous section.  What makes
generators so compact is that the __iter__() and
next() methods are created automatically.

  Another key feature is that the local variables and execution state
are automatically saved between calls.  This made the function easier
to write and much more clear than an approach using instance variables
like `self.index' and `self.data'.

  In addition to automatic method creation and saving program state,
when generators terminate, they automatically raise
StopIteration. In combination, these features make it easy to
create iterators with no more effort than writing a regular function.

2.9.11 Generator Expressions
----------------------------

Some simple generators can be coded succinctly as expressions using a
syntax similar to list comprehensions but with parentheses instead of
brackets.  These expressions are designed for situations where the
generator is used right away by an enclosing function.  Generator
expressions are more compact but less versatile than full generator
definitions and tend to be more memory friendly than equivalent list
comprehensions.

  Examples:

    >>> sum(i*i for i in range(10))                 # sum of squares
    285

    >>> xvec = [10, 20, 30]
    >>> yvec = [7, 5, 3]
    >>> sum(x*y for x,y in zip(xvec, yvec))         # dot product
    260

    >>> from math import pi, sin
    >>> sine_table = dict((x, sin(x*pi/180)) for x in range(0, 91))

    >>> unique_words = set(word  for line in page  for word in line.split())

    >>> valedictorian = max((student.gpa, student.name) for student in graduates)

    >>> data = 'golf'
    >>> list(data[i] for i in range(len(data)-1,-1,-1))
    ['f', 'l', 'o', 'g']

Footnotes
.........

     (1) Except for one thing.  Module objects have a secret read-only
     attribute called `__dict__' which returns the dictionary used to
     implement the module's namespace; the name `__dict__' is an
     attribute but not a global name.  Obviously, using this violates
     the abstraction of namespace implementation, and should be
     restricted to things like post-mortem debuggers.

2.10 Brief Tour of the Standard Library
=======================================

2.10.1 Operating System Interface
---------------------------------

The os module provides dozens of functions for interacting
with the operating system:

    >>> import os
    >>> os.getcwd()      # Return the current working directory
    'C:\\Python26'
    >>> os.chdir('/server/accesslogs')   # Change current working directory
    >>> os.system('mkdir today')   # Run the command mkdir in the system shell
    0

Be sure to use the `import os' style instead of `from os import *'.
This will keep os.open() from shadowing the built-in
open() function which operates much differently.

  The built-in dir() and help() functions are
useful as interactive aids for working with large modules like
os.:

    >>> import os
    >>> dir(os)
    <returns a list of all module functions>
    >>> help(os)
    <returns an extensive manual page created from the module's docstrings>

For daily file and directory management tasks, the shutil.
module provides a higher level interface that is easier to use:

    >>> import shutil
    >>> shutil.copyfile('data.db', 'archive.db')
    >>> shutil.move('/build/executables', 'installdir')

2.10.2 File Wildcards
---------------------

The glob module provides a function for making file lists
from directory wildcard searches:

    >>> import glob
    >>> glob.glob('*.py')
    ['primes.py', 'random.py', 'quote.py']

2.10.3 Command Line Arguments
-----------------------------

Common utility scripts often need to process command line arguments.
These arguments are stored in the sys module's _argv_
attribute as a list.  For instance the following output results from
running `python demo.py one two three' at the command line:

    >>> import sys
    >>> print sys.argv
    ['demo.py', 'one', 'two', 'three']

The getopt module processes _sys.argv_ using the conventions
of the Unix getopt() function.  More powerful and flexible
command line processing is provided by the argparse module.

2.10.4 Error Output Redirection and Program Termination
-------------------------------------------------------

The sys module also has attributes for _stdin_, _stdout_,
and _stderr_.  The latter is useful for emitting warnings and error
messages to make them visible even when _stdout_ has been redirected:

    >>> sys.stderr.write('Warning, log file not found starting a new one\n')
    Warning, log file not found starting a new one

The most direct way to terminate a script is to use `sys.exit()'.

2.10.5 String Pattern Matching
------------------------------

The re module provides regular expression tools for
advanced string processing. For complex matching and manipulation,
regular expressions offer succinct, optimized solutions:

    >>> import re
    >>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')
    ['foot', 'fell', 'fastest']
    >>> re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat')
    'cat in the hat'

When only simple capabilities are needed, string methods are preferred
because they are easier to read and debug:

    >>> 'tea for too'.replace('too', 'two')
    'tea for two'

2.10.6 Mathematics
------------------

The math module gives access to the underlying C library
functions for floating point math:

    >>> import math
    >>> math.cos(math.pi / 4.0)
    0.70710678118654757
    >>> math.log(1024, 2)
    10.0

The random module provides tools for making random
selections:

    >>> import random
    >>> random.choice(['apple', 'pear', 'banana'])
    'apple'
    >>> random.sample(xrange(100), 10)   # sampling without replacement
    [30, 83, 16, 4, 8, 81, 41, 50, 18, 33]
    >>> random.random()    # random float
    0.17970987693706186
    >>> random.randrange(6)    # random integer chosen from range(6)
    4

2.10.7 Internet Access
----------------------

There are a number of modules for accessing the internet and processing
internet protocols. Two of the simplest are urllib2 for
retrieving data from urls and smtplib for sending mail:

    >>> import urllib2
    >>> for line in urllib2.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
    ...     if 'EST' in line or 'EDT' in line:  # look for Eastern Time
    ...         print line

    <BR>Nov. 25, 09:43:32 PM EST

    >>> import smtplib
    >>> server = smtplib.SMTP('localhost')
    >>> server.sendmail('soothsayer@example.org', 'jcaesar@example.org',
    ... """To: jcaesar@example.org
    ... From: soothsayer@example.org
    ...
    ... Beware the Ides of March.
    ... """)
    >>> server.quit()

(Note that the second example needs a mailserver running on localhost.)

2.10.8 Dates and Times
----------------------

The datetime module supplies classes for manipulating dates
and times in both simple and complex ways. While date and time
arithmetic is supported, the focus of the implementation is on
efficient member extraction for output formatting and manipulation.
The module also supports objects that are timezone aware.

    >>> # dates are easily constructed and formatted
    >>> from datetime import date
    >>> now = date.today()
    >>> now
    datetime.date(2003, 12, 2)
    >>> now.strftime("%m-%d-%y. %d %b %Y is a %A on the %d day of %B.")
    '12-02-03. 02 Dec 2003 is a Tuesday on the 02 day of December.'

    >>> # dates support calendar arithmetic
    >>> birthday = date(1964, 7, 31)
    >>> age = now - birthday
    >>> age.days
    14368

2.10.9 Data Compression
-----------------------

Common data archiving and compression formats are directly supported by
modules including: zlib, gzip, bz2,
zipfile and tarfile.

    >>> import zlib
    >>> s = 'witch which has which witches wrist watch'
    >>> len(s)
    41
    >>> t = zlib.compress(s)
    >>> len(t)
    37
    >>> zlib.decompress(t)
    'witch which has which witches wrist watch'
    >>> zlib.crc32(s)
    226805979

2.10.10 Performance Measurement
-------------------------------

Some Python users develop a deep interest in knowing the relative
performance of different approaches to the same problem. Python
provides a measurement tool that answers those questions immediately.

  For example, it may be tempting to use the tuple packing and
unpacking feature instead of the traditional approach to swapping
arguments. The timeit.  module quickly demonstrates a modest
performance advantage:

    >>> from timeit import Timer
    >>> Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
    0.57535828626024577
    >>> Timer('a,b = b,a', 'a=1; b=2').timeit()
    0.54962537085770791

In contrast to timeit.'s fine level of granularity, the
profile and pstats modules provide tools for
identifying time critical sections in larger blocks of code.

2.10.11 Quality Control
-----------------------

One approach for developing high quality software is to write tests for
each function as it is developed and to run those tests frequently
during the development process.

  The doctest module provides a tool for scanning a module
and validating tests embedded in a program's docstrings.  Test
construction is as simple as cutting-and-pasting a typical call along
with its results into the docstring.  This improves the documentation
by providing the user with an example and it allows the doctest module
to make sure the code remains true to the documentation:

    def average(values):
        """Computes the arithmetic mean of a list of numbers.

        >>> print average([20, 30, 70])
        40.0
        """
        return sum(values, 0.0) / len(values)

    import doctest
    doctest.testmod()   # automatically validate the embedded tests

The unittest module is not as effortless as the
doctest module, but it allows a more comprehensive set of tests to
be maintained in a separate file:

    import unittest

    class TestStatisticalFunctions(unittest.TestCase):

        def test_average(self):
            self.assertEqual(average([20, 30, 70]), 40.0)
            self.assertEqual(round(average([1, 5, 7]), 1), 4.3)
            self.assertRaises(ZeroDivisionError, average, [])
            self.assertRaises(TypeError, average, 20, 30, 70)

    unittest.main() # Calling from the command line invokes all tests

2.10.12 Batteries Included
--------------------------

Python has a "batteries included" philosophy.  This is best seen
through the sophisticated and robust capabilities of its larger
packages. For example:

   * The xmlrpclib and SimpleXMLRPCServer.
     modules make implementing remote procedure calls into an almost
     trivial task.  Despite the modules names, no direct knowledge or
     handling of XML is needed.

   * The email package is a library for managing email
     messages, including MIME and other RFC 2822-based message
     documents. Unlike smtplib and poplib which
     actually send and receive messages, the email package has a
     complete toolset for building or decoding complex message
     structures (including attachments) and for implementing internet
     encoding and header protocols.

   * The xml.dom and xml.sax packages provide
     robust support for parsing this popular data interchange format.
     Likewise, the csv module supports direct reads and
     writes in a common database format. Together, these modules and
     packages greatly simplify data interchange between Python
     applications and other tools.

   * Internationalization is supported by a number of modules including
     gettext, locale, and the codecs.
     package.

2.11 Brief Tour of the Standard Library - Part II
=================================================

This second tour covers more advanced modules that support professional
programming needs.  These modules rarely occur in small scripts.

2.11.1 Output Formatting
------------------------

The repr module provides a version of repr().
customized for abbreviated displays of large or deeply nested
containers:

    >>> import repr
    >>> repr.repr(set('supercalifragilisticexpialidocious'))
    "set(['a', 'c', 'd', 'e', 'f', 'g', ...])"

The pprint module offers more sophisticated control over
printing both built-in and user defined objects in a way that is
readable by the interpreter.  When the result is longer than one line,
the "pretty printer" adds line breaks and indentation to more clearly
reveal data structure:

    >>> import pprint
    >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta',
    ...     'yellow'], 'blue']]]
    ...
    >>> pprint.pprint(t, width=30)
    [[[['black', 'cyan'],
       'white',
       ['green', 'red']],
      [['magenta', 'yellow'],
       'blue']]]

The textwrap module formats paragraphs of text to fit a
given screen width:

    >>> import textwrap
    >>> doc = """The wrap() method is just like fill() except that it returns
    ... a list of strings instead of one big string with newlines to separate
    ... the wrapped lines."""
    ...
    >>> print textwrap.fill(doc, width=40)
    The wrap() method is just like fill()
    except that it returns a list of strings
    instead of one big string with newlines
    to separate the wrapped lines.

The locale module accesses a database of culture specific
data formats.  The grouping attribute of locale's format function
provides a direct way of formatting numbers with group separators:

    >>> import locale
    >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
    'English_United States.1252'
    >>> conv = locale.localeconv()          # get a mapping of conventions
    >>> x = 1234567.8
    >>> locale.format("%d", x, grouping=True)
    '1,234,567'
    >>> locale.format_string("%s%.*f", (conv['currency_symbol'],
    ...                      conv['frac_digits'], x), grouping=True)
    '$1,234,567.80'

2.11.2 Templating
-----------------

The string module includes a versatile `Template' class
with a simplified syntax suitable for editing by end-users.  This
allows users to customize their applications without having to alter
the application.

  The format uses placeholder names formed by `$' with valid Python
identifiers (alphanumeric characters and underscores).  Surrounding the
placeholder with braces allows it to be followed by more alphanumeric
letters with no intervening spaces.  Writing `$$' creates a single
escaped `$':

    >>> from string import Template
    >>> t = Template('${village}folk send $$10 to $cause.')
    >>> t.substitute(village='Nottingham', cause='the ditch fund')
    'Nottinghamfolk send $10 to the ditch fund.'

The `substitute()' method raises a KeyError when a
placeholder is not supplied in a dictionary or a keyword argument. For
mail-merge style applications, user supplied data may be incomplete and
the `safe_substitute()' method may be more appropriate -- it will leave
placeholders unchanged if data is missing:

    >>> t = Template('Return the $item to $owner.')
    >>> d = dict(item='unladen swallow')
    >>> t.substitute(d)
    Traceback (most recent call last):
      . . .
    KeyError: 'owner'
    >>> t.safe_substitute(d)
    'Return the unladen swallow to $owner.'

Template subclasses can specify a custom delimiter.  For example, a
batch renaming utility for a photo browser may elect to use percent
signs for placeholders such as the current date, image sequence number,
or file format:

    >>> import time, os.path
    >>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg']
    >>> class BatchRename(Template):
    ...     delimiter = '%'
    >>> fmt = raw_input('Enter rename style (%d-date %n-seqnum %f-format):  ')
    Enter rename style (%d-date %n-seqnum %f-format):  Ashley_%n%f

    >>> t = BatchRename(fmt)
    >>> date = time.strftime('%d%b%y')
    >>> for i, filename in enumerate(photofiles):
    ...     base, ext = os.path.splitext(filename)
    ...     newname = t.substitute(d=date, n=i, f=ext)
    ...     print '{0} --> {1}'.format(filename, newname)

    img_1074.jpg --> Ashley_0.jpg
    img_1076.jpg --> Ashley_1.jpg
    img_1077.jpg --> Ashley_2.jpg

Another application for templating is separating program logic from the
details of multiple output formats.  This makes it possible to
substitute custom templates for XML files, plain text reports, and HTML
web reports.

2.11.3 Working with Binary Data Record Layouts
----------------------------------------------

The struct module provides `pack()' and `unpack()'
functions for working with variable length binary record formats.  The
following example shows how to loop through header information in a ZIP
file without using the zipfile module.  Pack codes `"H"'
and `"I"' represent two and four byte unsigned numbers respectively.
The `"<"' indicates that they are standard size and in little-endian
byte order:

    import struct

    data = open('myfile.zip', 'rb').read()
    start = 0
    for i in range(3):                      # show the first 3 file headers
        start += 14
        fields = struct.unpack('<IIIHH', data[start:start+16])
        crc32, comp_size, uncomp_size, filenamesize, extra_size = fields

        start += 16
        filename = data[start:start+filenamesize]
        start += filenamesize
        extra = data[start:start+extra_size]
        print filename, hex(crc32), comp_size, uncomp_size

        start += extra_size + comp_size     # skip to the next header

2.11.4 Multi-threading
----------------------

Threading is a technique for decoupling tasks which are not sequentially
dependent.  Threads can be used to improve the responsiveness of
applications that accept user input while other tasks run in the
background.  A related use case is running I/O in parallel with
computations in another thread.

  The following code shows how the high level threading.
module can run tasks in background while the main program continues to
run:

    import threading, zipfile

    class AsyncZip(threading.Thread):
        def __init__(self, infile, outfile):
            threading.Thread.__init__(self)
            self.infile = infile
            self.outfile = outfile
        def run(self):
            f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
            f.write(self.infile)
            f.close()
            print 'Finished background zip of: ', self.infile

    background = AsyncZip('mydata.txt', 'myarchive.zip')
    background.start()
    print 'The main program continues to run in foreground.'

    background.join()    # Wait for the background task to finish
    print 'Main program waited until background was done.'

The principal challenge of multi-threaded applications is coordinating
threads that share data or other resources.  To that end, the threading
module provides a number of synchronization primitives including locks,
events, condition variables, and semaphores.

  While those tools are powerful, minor design errors can result in
problems that are difficult to reproduce.  So, the preferred approach
to task coordination is to concentrate all access to a resource in a
single thread and then use the Queue module to feed that
thread with requests from other threads.  Applications using
Queue.Queue objects for inter-thread communication and
coordination are easier to design, more readable, and more reliable.

2.11.5 Logging
--------------

The logging module offers a full featured and flexible
logging system.  At its simplest, log messages are sent to a file or to
`sys.stderr':

    import logging
    logging.debug('Debugging information')
    logging.info('Informational message')
    logging.warning('Warning:config file %s not found', 'server.conf')
    logging.error('Error occurred')
    logging.critical('Critical error -- shutting down')

This produces the following output:

    WARNING:root:Warning:config file server.conf not found
    ERROR:root:Error occurred
    CRITICAL:root:Critical error -- shutting down

By default, informational and debugging messages are suppressed and the
output is sent to standard error.  Other output options include routing
messages through email, datagrams, sockets, or to an HTTP Server.  New
filters can select different routing based on message priority:
`DEBUG', `INFO', `WARNING', `ERROR', and `CRITICAL'.

  The logging system can be configured directly from Python or can be
loaded from a user editable configuration file for customized logging
without altering the application.

2.11.6 Weak References
----------------------

Python does automatic memory management (reference counting for most
objects and garbage collection to eliminate cycles).  The
memory is freed shortly after the last reference to it has been
eliminated.

  This approach works fine for most applications but occasionally there
is a need to track objects only as long as they are being used by
something else.  Unfortunately, just tracking them creates a reference
that makes them permanent.  The weakref module provides
tools for tracking objects without creating a reference.  When the
object is no longer needed, it is automatically removed from a weakref
table and a callback is triggered for weakref objects.  Typical
applications include caching objects that are expensive to create:

    >>> import weakref, gc
    >>> class A:
    ...     def __init__(self, value):
    ...             self.value = value
    ...     def __repr__(self):
    ...             return str(self.value)
    ...
    >>> a = A(10)                   # create a reference
    >>> d = weakref.WeakValueDictionary()
    >>> d['primary'] = a            # does not create a reference
    >>> d['primary']                # fetch the object if it is still alive
    10
    >>> del a                       # remove the one reference
    >>> gc.collect()                # run garbage collection right away
    0
    >>> d['primary']                # entry was automatically removed
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
        d['primary']                # entry was automatically removed
      File "C:/python26/lib/weakref.py", line 46, in __getitem__
        o = self.data[key]()
    KeyError: 'primary'

2.11.7 Tools for Working with Lists
-----------------------------------

Many data structure needs can be met with the built-in list type.
However, sometimes there is a need for alternative implementations with
different performance trade-offs.

  The array module provides an array() object that
is like a list that stores only homogeneous data and stores it more
compactly.  The following example shows an array of numbers stored as
two byte unsigned binary numbers (typecode `"H"') rather than the usual
16 bytes per entry for regular lists of Python int objects:

    >>> from array import array
    >>> a = array('H', [4000, 10, 700, 22222])
    >>> sum(a)
    26932
    >>> a[1:3]
    array('H', [10, 700])

The collections module provides a `deque()' object that is
like a list with faster appends and pops from the left side but slower
lookups in the middle. These objects are well suited for implementing
queues and breadth first tree searches:

    >>> from collections import deque
    >>> d = deque(["task1", "task2", "task3"])
    >>> d.append("task4")
    >>> print "Handling", d.popleft()
    Handling task1

    unsearched = deque([starting_node])
    def breadth_first_search(unsearched):
        node = unsearched.popleft()
        for m in gen_moves(node):
            if is_goal(m):
                return m
            unsearched.append(m)

In addition to alternative list implementations, the library also
offers other tools such as the bisect module with functions
for manipulating sorted lists:

    >>> import bisect
    >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')]
    >>> bisect.insort(scores, (300, 'ruby'))
    >>> scores
    [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]

The heapq module provides functions for implementing heaps
based on regular lists.  The lowest valued entry is always kept at
position zero.  This is useful for applications which repeatedly access
the smallest element but do not want to run a full list sort:

    >>> from heapq import heapify, heappop, heappush
    >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
    >>> heapify(data)                      # rearrange the list into heap order
    >>> heappush(data, -5)                 # add a new entry
    >>> [heappop(data) for i in range(3)]  # fetch the three smallest entries
    [-5, 0, 1]

2.11.8 Decimal Floating Point Arithmetic
----------------------------------------

The decimal module offers a `Decimal' datatype for decimal
floating point arithmetic.  Compared to the built-in float.
implementation of binary floating point, the class is especially
helpful for

   * financial applications and other uses which require exact decimal
     representation,

   * control over precision,

   * control over rounding to meet legal or regulatory requirements,

   * tracking of significant decimal places, or

   * applications where the user expects the results to match
     calculations done by hand.

  For example, calculating a 5% tax on a 70 cent phone charge gives
different results in decimal floating point and binary floating point.
The difference becomes significant if the results are rounded to the
nearest cent:

    >>> from decimal import *
    >>> x = Decimal('0.70') * Decimal('1.05')
    >>> x
    Decimal('0.7350')
    >>> x.quantize(Decimal('0.01'))  # round to nearest cent
    Decimal('0.74')
    >>> round(.70 * 1.05, 2)         # same calculation with floats
    0.73

The `Decimal' result keeps a trailing zero, automatically inferring four
place significance from multiplicands with two place significance.
Decimal reproduces mathematics as done by hand and avoids issues that
can arise when binary floating point cannot exactly represent decimal
quantities.

  Exact representation enables the `Decimal' class to perform modulo
calculations and equality tests that are unsuitable for binary floating
point:

    >>> Decimal('1.00') % Decimal('.10')
    Decimal('0.00')
    >>> 1.00 % 0.10
    0.09999999999999995

    >>> sum([Decimal('0.1')]*10) == Decimal('1.0')
    True
    >>> sum([0.1]*10) == 1.0
    False

The decimal module provides arithmetic with as much
precision as needed:

    >>> getcontext().prec = 36
    >>> Decimal(1) / Decimal(7)
    Decimal('0.142857142857142857142857142857142857')

2.12 What Now?
==============

Reading this tutorial has probably reinforced your interest in using
Python -- you should be eager to apply Python to solving your
real-world problems. Where should you go to learn more?

  This tutorial is part of Python's documentation set.   Some other
documents in the set are:

   * The Python Standard Library.:

     You should browse through this manual, which gives complete
     (though terse) reference material about types, functions, and the
     modules in the standard library.  The standard Python distribution
     includes a _lot_ of additional code.  There are modules to read
     Unix mailboxes, retrieve documents via HTTP, generate random
     numbers, parse command-line options, write CGI programs, compress
     data, and many other tasks. Skimming through the Library Reference
     will give you an idea of what's available.

   * Installing Python Modules explains how to install
     external modules written by other Python users.

   * The Python Language Reference.: A detailed explanation
     of Python's syntax and semantics.  It's heavy reading, but is
     useful as a complete guide to the language itself.

  More Python resources:

   * <http://www.python.org>:  The major Python Web site.  It contains
     code, documentation, and pointers to Python-related pages around
     the Web.  This Web site is mirrored in various places around the
     world, such as Europe, Japan, and Australia; a mirror may be
     faster than the main site, depending on your geographical location.

   * <http://docs.python.org>:  Fast access to Python's  documentation.

   * <http://pypi.python.org>: The Python Package Index, previously
     also nicknamed the Cheese Shop, is an index of user-created Python
     modules that are available for download.  Once you begin releasing
     code, you can register it here so that others can find it.

   * <http://aspn.activestate.com/ASPN/Python/Cookbook/>: The Python
     Cookbook is a sizable collection of code examples, larger modules,
     and useful scripts.  Particularly notable contributions are
     collected in a book also titled Python Cookbook (O'Reilly &
     Associates, ISBN 0-596-00797-3.)

  For Python-related questions and problem reports, you can post to the
newsgroup `comp.lang.python', or send them to the mailing list at
<python-list@python.org>.  The newsgroup and mailing list are
gatewayed, so messages posted to one will automatically be forwarded to
the other.  There are around 120 postings a day (with peaks up to
several hundred), asking (and answering) questions, suggesting new
features, and announcing new modules.  Before posting, be sure to check
the list of Frequently Asked Questions
(http://www.python.org/doc/faq/)(also called the FAQ), or look for it
in the `Misc/' directory of the Python source distribution.  Mailing
list archives are available at <http://mail.python.org/pipermail/>. The
FAQ answers many of the questions that come up again and again, and may
already contain the solution for your problem.

2.13 Interactive Input Editing and History Substitution
=======================================================

Some versions of the Python interpreter support editing of the current
input line and history substitution, similar to facilities found in the
Korn shell and the GNU Bash shell.  This is implemented using the GNU
Readline (http://tiswww.case.edu/php/chet/readline/rltop.html) library,
which supports Emacs-style and vi-style editing.  This library has its
own documentation which I won't duplicate here; however, the basics are
easily explained.  The interactive editing and history described here
are optionally available in the Unix and Cygwin versions of the
interpreter.

  This chapter does _not_ document the editing facilities of Mark
Hammond's PythonWin package or the Tk-based environment, IDLE,
distributed with Python.  The command line history recall which
operates within DOS boxes on NT and some other DOS and Windows flavors
is yet another beast.

2.13.1 Line Editing
-------------------

If supported, input line editing is active whenever the interpreter
prints a primary or secondary prompt.  The current line can be edited
using the conventional Emacs control characters.  The most important of
these are: `C-A' (Control-A) moves the cursor to the beginning of the
line, `C-E' to the end, `C-B' moves it one position to the left, `C-F'
to the right.  Backspace erases the character to the left of the
cursor, `C-D' the character to its right. `C-K' kills (erases) the rest
of the line to the right of the cursor, `C-Y' yanks back the last
killed string.  `C-underscore' undoes the last change you made; it can
be repeated for cumulative effect.

2.13.2 History Substitution
---------------------------

History substitution works as follows.  All non-empty input lines
issued are saved in a history buffer, and when a new prompt is given
you are positioned on a new line at the bottom of this buffer. `C-P'
moves one line up (back) in the history buffer, `C-N' moves one down.
Any line in the history buffer can be edited; an asterisk appears in
front of the prompt to mark a line as modified.  Pressing the `Return'
key passes the current line to the interpreter.  `C-R' starts an
incremental reverse search; `C-S' starts a forward search.

2.13.3 Key Bindings
-------------------

The key bindings and some other parameters of the Readline library can
be customized by placing commands in an initialization file called
`~/.inputrc'.  Key bindings have the form

    key-name: function-name

or

    "string": function-name

and options can be set with

    set option-name value

For example:

    # I prefer vi-style editing:
    set editing-mode vi

    # Edit using a single line:
    set horizontal-scroll-mode On

    # Rebind some keys:
    Meta-h: backward-kill-word
    "\C-u": universal-argument
    "\C-x\C-r": re-read-init-file

Note that the default binding for `Tab' in Python is to insert a `Tab'
character instead of Readline's default filename completion function.
If you insist, you can override this by putting

    Tab: complete

in your `~/.inputrc'.  (Of course, this makes it harder to type indented
continuation lines if you're accustomed to using `Tab' for that
purpose.)

  Automatic completion of variable and module names is optionally
available.  To enable it in the interpreter's interactive mode, add the
following to your startup file: (1)

    import rlcompleter, readline
    readline.parse_and_bind('tab: complete')

This binds the `Tab' key to the completion function, so hitting the
`Tab' key twice suggests completions; it looks at Python statement
names, the current local variables, and the available module names.
For dotted expressions such as `string.a', it will evaluate the
expression up to the final `'.'' and then suggest completions from the
attributes of the resulting object.  Note that this may execute
application-defined code if an object with a __getattr__().
method is part of the expression.

  A more capable startup file might look like this example.  Note that
this deletes the names it creates once they are no longer needed; this
is done since the startup file is executed in the same namespace as the
interactive commands, and removing the names avoids creating side
effects in the interactive environment.  You may find it convenient to
keep some of the imported modules, such as os, which turn
out to be needed in most sessions with the interpreter.

    # Add auto-completion and a stored history file of commands to your Python
    # interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
    # bound to the Esc key by default (you can change it - see readline docs).
    #
    # Store the file in ~/.pystartup, and set an environment variable to point
    # to it:  "export PYTHONSTARTUP=/home/user/.pystartup" in bash.
    #
    # Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the
    # full path to your home directory.

    import atexit
    import os
    import readline
    import rlcompleter

    historyPath = os.path.expanduser("~/.pyhistory")

    def save_history(historyPath=historyPath):
        import readline
        readline.write_history_file(historyPath)

    if os.path.exists(historyPath):
        readline.read_history_file(historyPath)

    atexit.register(save_history)
    del os, atexit, readline, rlcompleter, save_history, historyPath

2.13.4 Alternatives to the Interactive Interpreter
--------------------------------------------------

This facility is an enormous step forward compared to earlier versions
of the interpreter; however, some wishes are left: It would be nice if
the proper indentation were suggested on continuation lines (the parser
knows if an indent token is required next).  The completion mechanism
might use the interpreter's symbol table.  A command to check (or even
suggest) matching parentheses, quotes, etc., would also be useful.

  One alternative enhanced interactive interpreter that has been around
for quite some time is IPython (http://ipython.scipy.org/), which
features tab completion, object exploration and advanced history
management.  It can also be thoroughly customized and embedded into
other applications.  Another similar enhanced interactive environment is
bpython (http://www.bpython-interpreter.org/).

Footnotes
.........

     (1) Python will execute the contents of a file identified by the
     PYTHONSTARTUP environment variable when you start an
     interactive interpreter.

2.14 Floating Point Arithmetic:  Issues and Limitations
=======================================================

Floating-point numbers are represented in computer hardware as base 2
(binary) fractions.  For example, the decimal fraction

    0.125

has value 1/10 + 2/100 + 5/1000, and in the same way the binary fraction

    0.001

has value 0/2 + 0/4 + 1/8.  These two fractions have identical values,
the only real difference being that the first is written in base 10
fractional notation, and the second in base 2.

  Unfortunately, most decimal fractions cannot be represented exactly
as binary fractions.  A consequence is that, in general, the decimal
floating-point numbers you enter are only approximated by the binary
floating-point numbers actually stored in the machine.

  The problem is easier to understand at first in base 10.  Consider
the fraction 1/3.  You can approximate that as a base 10 fraction:

    0.3

or, better,

    0.33

or, better,

    0.333

and so on.  No matter how many digits you're willing to write down, the
result will never be exactly 1/3, but will be an increasingly better
approximation of 1/3.

  In the same way, no matter how many base 2 digits you're willing to
use, the decimal value 0.1 cannot be represented exactly as a base 2
fraction.  In base 2, 1/10 is the infinitely repeating fraction

    0.0001100110011001100110011001100110011001100110011...

Stop at any finite number of bits, and you get an approximation.

  On a typical machine running Python, there are 53 bits of precision
available for a Python float, so the value stored internally when you
enter the decimal number `0.1' is the binary fraction

    0.00011001100110011001100110011001100110011001100110011010

which is close to, but not exactly equal to, 1/10.

  It's easy to forget that the stored value is an approximation to the
original decimal fraction, because of the way that floats are displayed
at the interpreter prompt.  Python only prints a decimal approximation
to the true decimal value of the binary approximation stored by the
machine.  If Python were to print the true decimal value of the binary
approximation stored for 0.1, it would have to display

    >>> 0.1
    0.1000000000000000055511151231257827021181583404541015625

That is more digits than most people find useful, so Python keeps the
number of digits manageable by displaying a rounded value instead

    >>> 0.1
    0.1

It's important to realize that this is, in a real sense, an illusion:
the value in the machine is not exactly 1/10, you're simply rounding
the _display_ of the true machine value.  This fact becomes apparent as
soon as you try to do arithmetic with these values

    >>> 0.1 + 0.2
    0.30000000000000004

Note that this is in the very nature of binary floating-point: this is
not a bug in Python, and it is not a bug in your code either.  You'll
see the same kind of thing in all languages that support your
hardware's floating-point arithmetic (although some languages may not
_display_ the difference by default, or in all output modes).

  Other surprises follow from this one.  For example, if you try to
round the value 2.675 to two decimal places, you get this

    >>> round(2.675, 2)
    2.67

The documentation for the built-in round() function says
that it rounds to the nearest value, rounding ties away from zero.
Since the decimal fraction 2.675 is exactly halfway between 2.67 and
2.68, you might expect the result here to be (a binary approximation
to) 2.68.  It's not, because when the decimal string `2.675' is
converted to a binary floating-point number, it's again replaced with a
binary approximation, whose exact value is

    2.67499999999999982236431605997495353221893310546875

Since this approximation is slightly closer to 2.67 than to 2.68, it's
rounded down.

  If you're in a situation where you care which way your decimal
halfway-cases are rounded, you should consider using the decimal
module.  Incidentally, the decimal module also provides
a nice way to "see" the exact value that's stored in any particular
Python float

    >>> from decimal import Decimal
    >>> Decimal(2.675)
    Decimal('2.67499999999999982236431605997495353221893310546875')

Another consequence is that since 0.1 is not exactly 1/10, summing ten
values of 0.1 may not yield exactly 1.0, either:

    >>> sum = 0.0
    >>> for i in range(10):
    ...     sum += 0.1
    ...
    >>> sum
    0.9999999999999999

Binary floating-point arithmetic holds many surprises like this.  The
problem with "0.1" is explained in precise detail below, in the
"Representation Error" section.  See The Perils of Floating Point
(http://www.lahey.com/float.htm)for a more complete account of other
common surprises.

  As that says near the end, "there are no easy answers."  Still, don't
be unduly wary of floating-point!  The errors in Python float
operations are inherited from the floating-point hardware, and on most
machines are on the order of no more than 1 part in 2**53 per
operation.  That's more than adequate for most tasks, but you do need
to keep in mind that it's not decimal arithmetic, and that every float
operation can suffer a new rounding error.

  While pathological cases do exist, for most casual use of
floating-point arithmetic you'll see the result you expect in the end
if you simply round the display of your final results to the number of
decimal digits you expect.  For fine control over how a float is
displayed see the str.format() method's format specifiers
in Format String Syntax.

2.14.1 Representation Error
---------------------------

This section explains the "0.1" example in detail, and shows how you can
perform an exact analysis of cases like this yourself.  Basic
familiarity with binary floating-point representation is assumed.

  _Representation error_ refers to the fact that some (most, actually)
decimal fractions cannot be represented exactly as binary (base 2)
fractions.  This is the chief reason why Python (or Perl, C, C++, Java,
Fortran, and many others) often won't display the exact decimal number
you expect:

    >>> 0.1 + 0.2
    0.30000000000000004

Why is that?  1/10 and 2/10 are not exactly representable as a binary
fraction. Almost all machines today (July 2010) use IEEE-754 floating
point arithmetic, and almost all platforms map Python floats to
IEEE-754 "double precision".  754 doubles contain 53 bits of precision,
so on input the computer strives to convert 0.1 to the closest fraction
it can of the form _J_/2**_N_ where _J_ is an integer containing
exactly 53 bits.  Rewriting

    1 / 10 ~= J / (2**N)

as

    J ~= 2**N / 10

and recalling that _J_ has exactly 53 bits (is `>= 2**52' but `<
2**53'), the best value for _N_ is 56:

    >>> 2**52
    4503599627370496
    >>> 2**53
    9007199254740992
    >>> 2**56/10
    7205759403792793

That is, 56 is the only value for _N_ that leaves _J_ with exactly 53
bits.  The best possible value for _J_ is then that quotient rounded:

    >>> q, r = divmod(2**56, 10)
    >>> r
    6

Since the remainder is more than half of 10, the best approximation is
obtained by rounding up:

    >>> q+1
    7205759403792794

Therefore the best possible approximation to 1/10 in 754 double
precision is that over 2**56, or

    7205759403792794 / 72057594037927936

Note that since we rounded up, this is actually a little bit larger
than 1/10; if we had not rounded up, the quotient would have been a
little bit smaller than 1/10.  But in no case can it be _exactly_ 1/10!

  So the computer never "sees" 1/10:  what it sees is the exact
fraction given above, the best 754 double approximation it can get:

    >>> .1 * 2**56
    7205759403792794.0

If we multiply that fraction by 10**30, we can see the (truncated)
value of its 30 most significant decimal digits:

    >>> 7205759403792794 * 10**30 // 2**56
    100000000000000005551115123125L

meaning that the exact number stored in the computer is approximately
equal to the decimal value 0.100000000000000005551115123125.  In
versions prior to Python 2.7 and Python 3.1, Python rounded this value
to 17 significant digits, giving '0.10000000000000001'.  In current
versions, Python displays a value based on the shortest decimal
fraction that rounds correctly back to the true binary value, resulting
simply in '0.1'.

3 Python Setup and Usage
************************

This part of the documentation is devoted to general information on the
setup of the Python environment on different platform, the invocation
of the interpreter and things that make working with Python easier.

3.1 Command line and environment
================================

The CPython interpreter scans the command line and the environment for
various settings.

*CPython implementation detail:* Other implementations' command line
schemes may differ.  See Alternate Implementations for
further resources.

3.1.1 Command line
------------------

When invoking Python, you may specify any of these options:

    python [-BdEiOQsStuUvVWxX3?] [-c command | -m module-name | script | - ] [args]

The most common use case is, of course, a simple invocation of a script:

    python myscript.py

3.1.1.1 Interface options
.........................

The interpreter interface resembles that of the UNIX shell, but
provides some additional methods of invocation:

   * When called with standard input connected to a tty device, it
     prompts for commands and executes them until an EOF (an
     end-of-file character, you can produce that with _Ctrl-D_ on UNIX
     or _Ctrl-Z, Enter_ on Windows) is read.

   * When called with a file name argument or with a file as standard
     input, it reads and executes a script from that file.

   * When called with a directory name argument, it reads and executes
     an appropriately named script from that directory.

   * When called with `-c command', it executes the Python statement(s)
     given as _command_.  Here _command_ may contain multiple
     statements separated by newlines. Leading whitespace is
     significant in Python statements!

   * When called with `-m module-name', the given module is located on
     the Python module path and executed as a script.

  In non-interactive mode, the entire input is parsed before it is
executed.

  An interface option terminates the list of options consumed by the
interpreter, all consecutive arguments will end up in sys.argv.
- note that the first element, subscript zero (`sys.argv[0]'), is
a string reflecting the program's source.

 -- Option: -c <command>
     Execute the Python code in _command_.  _command_ can be one or more
     statements separated by newlines, with significant leading
     whitespace as in normal module code.

     If this option is given, the first element of sys.argv.
     will be `"-c"' and the current directory will be added to the
     start of sys.path. (allowing modules in that directory
     to be imported as top level modules).

 -- Option: -m <module-name>
     Search sys.path for the named module and execute its
     contents as the __main__ module.

     Since the argument is a _module_ name, you must not give a file
     extension (`.py').  The `module-name' should be a valid Python
     module name, but the implementation may not always enforce this
     (e.g. it may allow you to use a name that includes a hyphen).

     Package names are also permitted. When a package name is supplied
     instead of a normal module, the interpreter will execute
     `<pkg>.__main__' as the main module. This behaviour is
     deliberately similar to the handling of directories and zipfiles
     that are passed to the interpreter as the script argument.

          Note: This option cannot be used with built-in modules and
          extension modules written in C, since they do not have Python
          module files. However, it can still be used for precompiled
          modules, even if the original source file is not available.

     If this option is given, the first element of sys.argv.
     will be the full path to the module file. As with the -c
     option, the current directory will be added to the start of
     sys.path.

     Many standard library modules contain code that is invoked on
     their execution as a script.  An example is the timeit.
     module:

         python -mtimeit -s 'setup here' 'benchmarked code here'
         python -mtimeit -h # for details

     See runpy.run_module().     Equivalent functionality
     directly available to Python code

     PEP 338 (http://www.python.org/dev/peps/pep-0338) - Executing
     modules as scripts

     New in version 2.4.

     Changed in version 2.5: The named module can now be located inside
     a package.

     Changed in version 2.7: Supply the package name to run a
     `__main__' submodule.  sys.argv[0] is now set to `"-m"' while
     searching for the module (it was previously incorrectly set to
     `"-c"')

 -- Description: -
     Read commands from standard input (see sys.stdin).  If
     standard input is a terminal, see -i is implied.

     If this option is given, the first element of sys.argv.
     will be `"-"' and the current directory will be added to the start
     of sys.path.

 -- Description: <script>
     Execute the Python code contained in _script_, which must be a
     filesystem path (absolute or relative) referring to either a
     Python file, a directory containing a `__main__.py' file, or a
     zipfile containing a `__main__.py' file.

     If this option is given, the first element of sys.argv.
     will be the script name as given on the command line.

     If the script name refers directly to a Python file, the directory
     containing that file is added to the start of sys.path,
     and the file is executed as the __main__ module.

     If the script name refers to a directory or zipfile, the script
     name is added to the start of sys.path and the
     `__main__.py' file in that location is executed as the
     __main__ module.

     Changed in version 2.5: Directories and zipfiles containing a
     `__main__.py' file at the top level are now considered valid
     Python scripts.

  If no interface option is given, see -i is implied,
`sys.argv[0]' is an empty string (`""') and the current directory will
be added to the start of sys.path.

  See Invoking the Interpreter.

3.1.1.2 Generic options
.......................

 -- Option: -?
 -- Option: -h
 -- Option: -help
     Print a short description of all command line options.

     Changed in version 2.5: The `--help' variant.

 -- Option: -V
 -- Option: -version
     Print the Python version number and exit.  Example output could be:

         Python 2.5.1

     Changed in version 2.5: The `--version' variant.

3.1.1.3 Miscellaneous options
.............................

 -- Option: -B
     If given, Python won't try to write `.pyc' or `.pyo' files on the
     import of source modules.  See also PYTHONDONTWRITEBYTECODE.

     New in version 2.6.

 -- Option: -d
     Turn on parser debugging output (for wizards only, depending on
     compilation options).  See also PYTHONDEBUG.

 -- Option: -E
     Ignore all `PYTHON*' environment variables, e.g.  see
     PYTHONPATH and PYTHONHOME, that might be set.

     New in version 2.2.

 -- Option: -i
     When a script is passed as first argument or the -c.
     option is used, enter interactive mode after executing the script
     or the command, even when sys.stdin does not appear to
     be a terminal.  The PYTHONSTARTUP file is not read.

     This can be useful to inspect global variables or a stack trace
     when a script raises an exception.  See also PYTHONINSPECT.

 -- Option: -O
     Turn on basic optimizations.  This changes the filename extension
     for compiled (see bytecode) files from `.pyc' to `.pyo'.
     See also PYTHONOPTIMIZE.

 -- Option: -OO
     Discard docstrings in addition to the -O optimizations.

 -- Option: -Q <arg>
     Division control. The argument must be one of the following:

    `old'
          division of int/int and long/long return an int or long
          (_default_)

    `new'
          new division semantics, i.e. division of int/int and
          long/long returns a float

    `warn'
          old division semantics with a warning for int/int and
          long/long

    `warnall'
          old division semantics with a warning for all uses of the
          division operator

     `Tools/scripts/fixdiv.py'    for a use of `warnall'

     PEP 238 (http://www.python.org/dev/peps/pep-0238) - Changing the
     division operator

 -- Option: -s
     Don't add user site directory to sys.path

     New in version 2.6.

     PEP 370 (http://www.python.org/dev/peps/pep-0370) - Per user
     site-packages directory

 -- Option: -S
     Disable the import of the module site and the
     site-dependent manipulations of sys.path that it
     entails.

 -- Option: -t
     Issue a warning when a source file mixes tabs and spaces for
     indentation in a way that makes it depend on the worth of a tab
     expressed in spaces.  Issue an error when the option is given
     twice (`-tt').

 -- Option: -u
     Force stdin, stdout and stderr to be totally unbuffered.  On
     systems where it matters, also put stdin, stdout and stderr in
     binary mode.

     Note that there is internal buffering in file.readlines()
     and File Objects. (`for line in sys.stdin') which
     is not influenced by this option.  To work around this, you will
     want to use file.readline() inside a `while 1:' loop.

     See also PYTHONUNBUFFERED.

 -- Option: -v
     Print a message each time a module is initialized, showing the
     place (filename or built-in module) from which it is loaded.  When
     given twice (`-vv'), print a message for each file that is checked
     for when searching for a module.  Also provides information on
     module cleanup at exit.  See also PYTHONVERBOSE.

 -- Option: -W arg
     Warning control.  Python's warning machinery by default prints
     warning messages to sys.stderr.  A typical warning
     message has the following form:

         file:line: category: message

     By default, each warning is printed once for each source line
     where it occurs.  This option controls how often warnings are
     printed.

     Multiple -W options may be given; when a warning
     matches more than one option, the action for the last matching
     option is performed.  Invalid -W options are ignored
     (though, a warning message is printed about invalid options when
     the first warning is issued).

     Starting from Python 2.7, see DeprecationWarning and its
     descendants are ignored by default.  The `-Wd' option can be used
     to re-enable them.

     Warnings can also be controlled from within a Python program using
     the warnings module.

     The simplest form of argument is one of the following action
     strings (or a unique abbreviation) by themselves:

    `ignore'
          Ignore all warnings.

    `default'
          Explicitly request the default behavior (printing each
          warning once per source line).

    `all'
          Print a warning each time it occurs (this may generate many
          messages if a warning is triggered repeatedly for the same
          source line, such as inside a loop).

    `module'
          Print each warning only the first time it occurs in each
          module.

    `once'
          Print each warning only the first time it occurs in the
          program.

    `error'
          Raise an exception instead of printing a warning message.

     The full form of argument is:

         action:message:category:module:line

     Here, _action_ is as explained above but only applies to messages
     that match the remaining fields.  Empty fields match all values;
     trailing empty fields may be omitted.  The _message_ field matches
     the start of the warning message printed; this match is
     case-insensitive.  The _category_ field matches the warning
     category.  This must be a class name; the match tests whether the
     actual warning category of the message is a subclass of the
     specified warning category.  The full class name must be given.
     The _module_ field matches the (fully-qualified) module name; this
     match is case-sensitive.  The _line_ field matches the line
     number, where zero matches all line numbers and is thus equivalent
     to an omitted line number.

     See warnings. - the warnings module

     PEP 230 (http://www.python.org/dev/peps/pep-0230) - Warning
     framework

     See PYTHONWARNINGS.

 -- Option: -x
     Skip the first line of the source, allowing use of non-Unix forms
     of `#!cmd'.  This is intended for a DOS specific hack only.

          Note: The line numbers in error messages will be off by one.

 -- Option: -3
     Warn about Python 3.x incompatibilities which cannot be fixed
     trivially by 2to3. Among these are:

        * dict.has_key().

        * apply().

        * callable().

        * coerce().

        * execfile().

        * reduce().

        * reload().

     Using these will emit a DeprecationWarning.

     New in version 2.6.

3.1.1.4 Options you shouldn't use
.................................

 -- Option: -J
     Reserved for use by Jython (http://jython.org).

 -- Option: -U
     Turns all string literals into unicodes globally.  Do not be
     tempted to use this option as it will probably break your world.
     It also produces `.pyc' files with a different magic number than
     normal.  Instead, you can enable unicode literals on a per-module
     basis by using:

         from __future__ import unicode_literals

     at the top of the file.  See __future__ for details.

 -- Option: -X
     Reserved for alternative implementations of Python to use for
     their own purposes.

3.1.2 Environment variables
---------------------------

These environment variables influence Python's behavior.

 -- envvar: PYTHONHOME
     Change the location of the standard Python libraries.  By default,
     the libraries are searched in `_prefix_/lib/python_version_' and
     `_exec_prefix_/lib/python_version_', where `_prefix_' and
     `_exec_prefix_' are installation-dependent directories, both
     defaulting to `/usr/local'.

     When PYTHONHOME is set to a single directory, its
     value replaces both `_prefix_' and `_exec_prefix_'.  To specify
     different values for these, set PYTHONHOME to
     `_prefix_:_exec_prefix_'.

 -- envvar: PYTHONPATH
     Augment the default search path for module files.  The format is
     the same as the shell's `PATH': one or more directory pathnames
     separated by os.pathsep. (e.g. colons on Unix or
     semicolons on Windows).  Non-existent directories are silently
     ignored.

     In addition to normal directories, individual PYTHONPATH
     entries may refer to zipfiles containing pure Python modules
     (in either source or compiled form). Extension modules cannot be
     imported from zipfiles.

     The default search path is installation dependent, but generally
     begins with `_prefix_/lib/python_version_' (see PYTHONHOME
     above).  It is _always_ appended to PYTHONPATH.

     An additional directory will be inserted in the search path in
     front of PYTHONPATH as described above under
     Interface options. The search path can be manipulated from
     within a Python program as the variable sys.path.

 -- envvar: PYTHONSTARTUP
     If this is the name of a readable file, the Python commands in
     that file are executed before the first prompt is displayed in
     interactive mode.  The file is executed in the same namespace
     where interactive commands are executed so that objects defined or
     imported in it can be used without qualification in the
     interactive session.  You can also change the prompts
     sys.ps1 and sys.ps2 in this file.

 -- envvar: PYTHONY2K
     Set this to a non-empty string to cause the time.
     module to require dates specified as strings to include 4-digit
     years, otherwise 2-digit years are converted based on rules
     described in the time module documentation.

 -- envvar: PYTHONOPTIMIZE
     If this is set to a non-empty string it is equivalent to
     specifying the -O option.  If set to an integer, it is
     equivalent to specifying -O multiple times.

 -- envvar: PYTHONDEBUG
     If this is set to a non-empty string it is equivalent to
     specifying the -d option.  If set to an integer, it is
     equivalent to specifying -d multiple times.

 -- envvar: PYTHONINSPECT
     If this is set to a non-empty string it is equivalent to
     specifying the -i option.

     This variable can also be modified by Python code using
     os.environ.  to force inspect mode on program termination.

 -- envvar: PYTHONUNBUFFERED
     If this is set to a non-empty string it is equivalent to
     specifying the -u option.

 -- envvar: PYTHONVERBOSE
     If this is set to a non-empty string it is equivalent to
     specifying the -v option.  If set to an integer, it is
     equivalent to specifying -v multiple times.

 -- envvar: PYTHONCASEOK
     If this is set, Python ignores case in import.
     statements.  This only works on Windows.

 -- envvar: PYTHONDONTWRITEBYTECODE
     If this is set, Python won't try to write `.pyc' or `.pyo' files
     on the import of source modules.

     New in version 2.6.

 -- envvar: PYTHONIOENCODING
     Overrides the encoding used for stdin/stdout/stderr, in the syntax
     `encodingname:errorhandler'.  The `:errorhandler' part is optional
     and has the same meaning as in str.encode().

     New in version 2.6.

 -- envvar: PYTHONNOUSERSITE
     If this is set, Python won't add the user site directory to
     sys.path

     New in version 2.6.

     PEP 370 (http://www.python.org/dev/peps/pep-0370) - Per user
     site-packages directory

 -- envvar: PYTHONUSERBASE
     Sets the base directory for the user site directory

     New in version 2.6.

     PEP 370 (http://www.python.org/dev/peps/pep-0370) - Per user
     site-packages directory

 -- envvar: PYTHONEXECUTABLE
     If this environment variable is set, `sys.argv[0]' will be set to
     its value instead of the value got through the C runtime.  Only
     works on Mac OS X.

 -- envvar: PYTHONWARNINGS
     This is equivalent to the -W option. If set to a comma
     separated string, it is equivalent to specifying -W.
     multiple times.

3.1.2.1 Debug-mode variables
............................

Setting these variables only has an effect in a debug build of Python,
that is, if Python was configured with the `--with-pydebug' build
option.

 -- envvar: PYTHONTHREADDEBUG
     If set, Python will print threading debug info.

     Changed in version 2.6: Previously, this variable was called
     `THREADDEBUG'.

 -- envvar: PYTHONDUMPREFS
     If set, Python will dump objects and reference counts still alive
     after shutting down the interpreter.

 -- envvar: PYTHONMALLOCSTATS
     If set, Python will print memory allocation statistics every time
     a new object arena is created, and on shutdown.

3.2 Using Python on Unix platforms
==================================

3.2.1 Getting and installing the latest version of Python
---------------------------------------------------------

3.2.1.1 On Linux
................

Python comes preinstalled on most Linux distributions, and is available
as a package on all others.  However there are certain features you
might want to use that are not available on your distro's package.  You
can easily compile the latest version of Python from source.

  In the event that Python doesn't come preinstalled and isn't in the
repositories as well, you can easily make packages for your own distro.
Have a look at the following links:

<http://www.linux.com/articles/60383>
     for Debian users

<http://linuxmafia.com/pub/linux/suse-linux-internals/chapter35.html>
     for OpenSuse users

<http://docs.fedoraproject.org/drafts/rpm-guide-en/ch-creating-rpms.html>
     for Fedora users

<http://www.slackbook.org/html/package-management-making-packages.html>
     for Slackware users

3.2.1.2 On FreeBSD and OpenBSD
..............................

   * FreeBSD users, to add the package use:

         pkg_add -r python

   * OpenBSD users use:

         pkg_add ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages/<insert your architecture here>/python-<version>.tgz

     For example i386 users get the 2.5.1 version of Python using:

         pkg_add ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages/i386/python-2.5.1p2.tgz

3.2.1.3 On OpenSolaris
......................

To install the newest Python versions on OpenSolaris, install blastwave
(<http://www.blastwave.org/howto.html>) and type "pkg_get -i python" at
the prompt.

3.2.2 Building Python
---------------------

If you want to compile CPython yourself, first thing you should do is
get the source (http://python.org/download/source/). You can download
either the latest release's source or just grab a fresh checkout
(http://www.python.org/dev/faq/#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write).

  The build process consists the usual

    ./configure
    make
    make install

invocations. Configuration options and caveats for specific Unix
platforms are extensively documented in the `README' file in the root
of the Python source tree.

     Warning: `make install' can overwrite or masquerade the `python'
     binary.  `make altinstall' is therefore recommended instead of
     `make install' since it only installs
     `_exec_prefix_/bin/python_version_'.

3.2.3 Python-related paths and files
------------------------------------

These are subject to difference depending on local installation
conventions; `prefix' (`${prefix}') and `exec_prefix' (`${exec_prefix}')
are installation-dependent and should be interpreted as for GNU
software; they may be the same.

  For example, on most Linux systems, the default for both is `/usr'.

File/directory                                      Meaning
---------------------------------------------------------------------------------------------------
`_exec_prefix_/bin/python'                          Recommended location of the interpreter.
`_prefix_/lib/python_version_',                     Recommended locations of the directories
`_exec_prefix_/lib/python_version_'                 containing the standard modules.
`_prefix_/include/python_version_',                 Recommended locations of the directories
`_exec_prefix_/include/python_version_'             containing the include files needed for
                                                    developing Python extensions and embedding
                                                    the interpreter.
`~/.pythonrc.py'                                    User-specific initialization file loaded by
                                                    the user module; not used by default or by
                                                    most applications.

3.2.4 Miscellaneous
-------------------

To easily use Python scripts on Unix, you need to make them executable,
e.g. with

    $ chmod +x script

and put an appropriate Shebang line at the top of the script.  A good
choice is usually

    #!/usr/bin/env python

which searches for the Python interpreter in the whole `PATH'.  However,
some Unices may not have the *env* command, so you may need to hardcode
`/usr/bin/python' as the interpreter path.

  To use shell commands in your Python scripts, look at the
subprocess module.

3.2.5 Editors
-------------

Vim and Emacs are excellent editors which support Python very well.
For more information on how to code in Python in these editors, look at:

   * <http://www.vim.org/scripts/script.php?script_id=790>

   * <http://sourceforge.net/projects/python-mode>

  Geany is an excellent IDE with support for a lot of languages. For
more information, read: <http://geany.uvena.de/>

  Komodo edit is another extremely good IDE.  It also has support for a
lot of languages. For more information, read:
<http://www.activestate.com/store/productdetail.aspx?prdGuid=20f4ed15-6684-4118-a78b-d37ff4058c5f>

3.3 Using Python on Windows
===========================

This document aims to give an overview of Windows-specific behaviour
you should know about when using Python on Microsoft Windows.

3.3.1 Installing Python
-----------------------

Unlike most Unix systems and services, Windows does not require Python
natively and thus does not pre-install a version of Python.  However,
the CPython team has compiled Windows installers (MSI packages) with
every release (http://www.python.org/download/releases/)for many years.

  With ongoing development of Python, some platforms that used to be
supported earlier are no longer supported (due to the lack of users or
developers).  Check PEP 11 (http://www.python.org/dev/peps/pep-0011)
for details on all unsupported platforms.

   * DOS and Windows 3.x are deprecated since Python 2.0 and code
     specific to these systems was removed in Python 2.1.

   * Up to 2.5, Python was still compatible with Windows 95, 98 and ME
     (but already raised a deprecation warning on installation).  For
     Python 2.6 (and all following releases), this support was dropped
     and new releases are just expected to work on the Windows NT
     family.

   * Windows CE (http://pythonce.sourceforge.net/)is still supported.

   * The Cygwin (http://cygwin.com/)installer offers to install the
     Python interpreter (http://cygwin.com/packages/python)as well; it
     is located under "Interpreters." (cf. Cygwin package source
     (ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/mirrors/cygnus/release/python),
     Maintainer releases
     (http://www.tishler.net/jason/software/python/))

  See Python for Windows (and DOS)
(http://www.python.org/download/windows/)for detailed information about
platforms with precompiled installers.

Python on XP (http://www.richarddooling.com/index.php/2006/03/14/python-on-xp-7-minutes-to-hello-world/)
     "7 Minutes to "Hello World!""  by Richard Dooling, 2006

Installing on Windows (http://diveintopython.org/installing_python/windows.html)
     in "Dive into Python: Python from novice to pro
     (http://diveintopython.org/index.html)" by Mark Pilgrim, 2004,
     ISBN 1-59059-356-1

For Windows users (http://swaroopch.com/text/Byte_of_Python:Installing_Python#For_Windows_users)
     in "Installing Python" in "A Byte of Python
     (http://www.byteofpython.info)" by Swaroop C H, 2003

3.3.2 Alternative bundles
-------------------------

Besides the standard CPython distribution, there are modified packages
including additional functionality.  The following is a list of popular
versions and their key features:

ActivePython (http://www.activestate.com/Products/activepython/)
     Installer with multi-platform compatibility, documentation, PyWin32

Enthought Python Distribution (http://www.enthought.com/products/epd.php)
     Popular modules (such as PyWin32) with their respective
     documentation, tool suite for building extensible Python
     applications

  Notice that these packages are likely to install _older_ versions of
Python.

3.3.3 Configuring Python
------------------------

In order to run Python flawlessly, you might have to change certain
environment settings in Windows.

3.3.3.1 Excursus: Setting environment variables
...............................................

Windows has a built-in dialog for changing environment variables
(following guide applies to XP classical view): Right-click the icon
for your machine (usually located on your Desktop and called "My
Computer") and choose _Properties_ there.  Then, open the _Advanced_ tab
and click the _Environment Variables_ button.

  In short, your path is:

     _My Computer ‣ Properties ‣ Advanced ‣ Environment Variables_

  In this dialog, you can add or modify User and System variables. To
change System variables, you need non-restricted access to your machine
(i.e. Administrator rights).

  Another way of adding variables to your environment is using the *set*
command:

    set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib

To make this setting permanent, you could add the corresponding command
line to your `autoexec.bat'. *msconfig* is a graphical interface to this
file.

  Viewing environment variables can also be done more straight-forward:
The command prompt will expand strings wrapped into percent signs
automatically:

    echo %PATH%

Consult *set /?* for details on this behaviour.

<http://support.microsoft.com/kb/100843>
     Environment variables in Windows NT

<http://support.microsoft.com/kb/310519>
     How To Manage Environment Variables in Windows XP

<http://www.chem.gla.ac.uk/~louis/software/faq/q1.html>
     Setting Environment variables, Louis J. Farrugia

3.3.3.2 Finding the Python executable
.....................................

Besides using the automatically created start menu entry for the Python
interpreter, you might want to start Python in the DOS prompt.  To make
this work, you need to set your `%PATH%' environment variable to
include the directory of your Python distribution, delimited by a
semicolon from other entries.  An example variable could look like this
(assuming the first two entries are Windows' default):

    C:\WINDOWS\system32;C:\WINDOWS;C:\Python25

Typing *python* on your command prompt will now fire up the Python
interpreter.  Thus, you can also execute your scripts with command line
options, see Command line documentation.

3.3.3.3 Finding modules
.......................

Python usually stores its library (and thereby your site-packages
folder) in the installation directory.  So, if you had installed Python
to `C:\Python\', the default library would reside in `C:\Python\Lib\'
and third-party modules should be stored in
`C:\Python\Lib\site-packages\'.

  This is how sys.path is populated on Windows:

   * An empty entry is added at the start, which corresponds to the
     current directory.

   * If the environment variable PYTHONPATH exists, as
     described in Environment variables, its entries are
     added next.  Note that on Windows, paths in this variable must be
     separated by semicolons, to distinguish them from the colon used
     in drive identifiers (`C:\' etc.).

   * Additional "application paths" can be added in the registry as
     subkeys of `\SOFTWARE\Python\PythonCore\_version_\PythonPath'
     under both the `HKEY_CURRENT_USER' and `HKEY_LOCAL_MACHINE' hives.
     Subkeys which have semicolon-delimited path strings as their
     default value will cause each path to be added to sys.path.
     (Note that all known installers only use HKLM, so HKCU is
     typically empty.)

   * If the environment variable PYTHONHOME is set, it is
     assumed as "Python Home".  Otherwise, the path of the main Python
     executable is used to locate a "landmark file" (`Lib\os.py') to
     deduce the "Python Home".  If a Python home is found, the relevant
     sub-directories added to sys.path.  (`Lib', `plat-win',
     etc) are based on that folder.  Otherwise, the core Python path is
     constructed from the PythonPath stored in the registry.

   * If the Python Home cannot be located, no PYTHONPATH is
     specified in the environment, and no registry entries can be
     found, a default path with relative entries is used (e.g.
     `.\Lib;.\plat-win', etc).

  The end result of all this is:

   * When running `python.exe', or any other .exe in the main Python
     directory (either an installed version, or directly from the
     PCbuild directory), the core path is deduced, and the core paths
     in the registry are ignored.  Other "application paths" in the
     registry are always read.

   * When Python is hosted in another .exe (different directory,
     embedded via COM, etc), the "Python Home" will not be deduced, so
     the core path from the registry is used.  Other "application
     paths" in the registry are always read.

   * If Python can't find its home and there is no registry (eg, frozen
     .exe, some very strange installation setup) you get a path with
     some default, but relative, paths.

3.3.3.4 Executing scripts
.........................

Python scripts (files with the extension `.py') will be executed by
*python.exe* by default.  This executable opens a terminal, which stays
open even if the program uses a GUI.  If you do not want this to
happen, use the extension `.pyw' which will cause the script to be
executed by *pythonw.exe* by default (both executables are located in
the top-level of your Python installation directory).  This suppresses
the terminal window on startup.

  You can also make all `.py' scripts execute with *pythonw.exe*,
setting this through the usual facilities, for example (might require
administrative rights):

  1. Launch a command prompt.

  2. Associate the correct file group with `.py' scripts:

         assoc .py=Python.File

  3. Redirect all Python files to the new executable:

         ftype Python.File=C:\Path\to\pythonw.exe "%1" %*

3.3.4 Additional modules
------------------------

Even though Python aims to be portable among all platforms, there are
features that are unique to Windows.  A couple of modules, both in the
standard library and external, and snippets exist to use these features.

  The Windows-specific standard modules are documented in MS Windows Specific Services.

3.3.4.1 PyWin32
...............

The PyWin32 (http://python.net/crew/mhammond/win32/)module by Mark
Hammond is a collection of modules for advanced Windows-specific
support.  This includes utilities for:

   * Component Object Model (http://www.microsoft.com/com/)(COM)

   * Win32 API calls

   * Registry

   * Event log

   * Microsoft Foundation Classes
     (http://msdn.microsoft.com/en-us/library/fe1cf721%28VS.80%29.aspx)(MFC)
     user interfaces

  PythonWin
(http://web.archive.org/web/20060524042422/http://www.python.org/windows/pythonwin/)is
a sample MFC application shipped with PyWin32.  It is an embeddable IDE
with a built-in debugger.

Win32 How Do I...? (http://timgolden.me.uk/python/win32_how_do_i.html)
     by Tim Golden

Python and COM (http://www.boddie.org.uk/python/COM.html)
     by David and Paul Boddie

3.3.4.2 Py2exe
..............

Py2exe (http://www.py2exe.org/)is a distutils extension (see
Extending Distutils) which wraps Python scripts into
executable Windows programs (`_*_.exe' files).  When you have done
this, you can distribute your application without requiring your users
to install Python.

3.3.4.3 WConio
..............

Since Python's advanced terminal handling layer, see curses, is
restricted to Unix-like systems, there is a library exclusive to
Windows as well: Windows Console I/O for Python.

  WConio (http://newcenturycomputers.net/projects/wconio.html)is a
wrapper for Turbo-C's `CONIO.H', used to create text user interfaces.

3.3.5 Compiling Python on Windows
---------------------------------

If you want to compile CPython yourself, first thing you should do is
get the source (http://python.org/download/source/). You can download
either the latest release's source or just grab a fresh checkout
(http://www.python.org/dev/faq/#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write).

  For Microsoft Visual C++, which is the compiler with which official
Python releases are built, the source tree contains solutions/project
files.  View the `readme.txt' in their respective directories:

Directory                MSVC version       Visual Studio version
------------------------------------------------------------------------
`PC/VC6/'                6.0                97
`PC/VS7.1/'              7.1                2003
`PC/VS8.0/'              8.0                2005
`PCbuild/'               9.0                2008

  Note that not all of these build directories are fully supported.
Read the release notes to see which compiler version the official
releases for your version are built with.

  Check `PC/readme.txt' for general information on the build process.

  For extension modules, consult Building C and C++ Extensions on Windows.

Python + Windows + distutils + SWIG + gcc MinGW (http://sebsauvage.net/python/mingw.html)
     or "Creating Python extensions in C/C++ with SWIG and compiling
     them with MinGW gcc under Windows" or "Installing Python extension
     with distutils and without Microsoft Visual C++" by Sébastien
     Sauvage, 2003

MingW - Python extensions (http://oldwiki.mingw.org/index.php/Python%20extensions)
     by Trent Apted et al, 2007

3.3.6 Other resources
---------------------

Python Programming On Win32 (http://www.oreilly.com/catalog/pythonwin32/)
     "Help for Windows Programmers" by Mark Hammond and Andy Robinson,
     O'Reilly Media, 2000, ISBN 1-56592-621-8

A Python for Windows Tutorial (http://www.imladris.com/Scripts/PythonForWindows.html)
     by Amanda Birmingham, 2004

3.4 Using Python on a Macintosh
===============================

     Author: Bob Savage <<bobsavage@mac.com>>

  Python on a Macintosh running Mac OS X is in principle very similar
to Python on any other Unix platform, but there are a number of
additional features such as the IDE and the Package Manager that are
worth pointing out.

  The Mac-specific modules are documented in Mac OS X specific services.

  Python on Mac OS 9 or earlier can be quite different from Python on
Unix or Windows, but is beyond the scope of this manual, as that
platform is no longer supported, starting with Python 2.4. See
<http://www.cwi.nl/~jack/macpython> for installers for the latest 2.3
release for Mac OS 9 and related documentation.

3.4.1 Getting and Installing MacPython
--------------------------------------

Mac OS X 10.5 comes with Python 2.5.1 pre-installed by Apple.  If you
wish, you are invited to install the most recent version of Python from
the Python website (<http://www.python.org>).  A current "universal
binary" build of Python, which runs natively on the Mac's new Intel and
legacy PPC CPU's, is available there.

  What you get after installing is a number of things:

   * A `MacPython 2.5' folder in your `Applications' folder. In here
     you find IDLE, the development environment that is a standard part
     of official Python distributions; PythonLauncher, which handles
     double-clicking Python scripts from the Finder; and the "Build
     Applet" tool, which allows you to package Python scripts as
     standalone applications on your system.

   * A framework `/Library/Frameworks/Python.framework', which includes
     the Python executable and libraries. The installer adds this
     location to your shell path. To uninstall MacPython, you can
     simply remove these three things. A symlink to the Python
     executable is placed in /usr/local/bin/.

  The Apple-provided build of Python is installed in
`/System/Library/Frameworks/Python.framework' and `/usr/bin/python',
respectively. You should never modify or delete these, as they are
Apple-controlled and are used by Apple- or third-party software.
Remember that if you choose to install a newer Python version from
python.org, you will have two different but functional Python
installations on your computer, so it will be important that your paths
and usages are consistent with what you want to do.

  IDLE includes a help menu that allows you to access Python
documentation. If you are completely new to Python you should start
reading the tutorial introduction in that document.

  If you are familiar with Python on other Unix platforms you should
read the section on running Python scripts from the Unix shell.

3.4.1.1 How to run a Python script
..................................

Your best way to get started with Python on Mac OS X is through the IDLE
integrated development environment, see section The IDE and
use the Help menu when the IDE is running.

  If you want to run Python scripts from the Terminal window command
line or from the Finder you first need an editor to create your script.
Mac OS X comes with a number of standard Unix command line editors,
*vim* and *emacs* among them. If you want a more Mac-like editor,
*BBEdit* or *TextWrangler* from Bare Bones Software (see
<http://www.barebones.com/products/bbedit/index.shtml>) are good
choices, as is *TextMate* (see <http://macromates.com/>). Other editors
include *Gvim* (<http://macvim.org>) and *Aquamacs*
(<http://aquamacs.org/>).

  To run your script from the Terminal window you must make sure that
`/usr/local/bin' is in your shell search path.

  To run your script from the Finder you have two options:

   * Drag it to *PythonLauncher*

   * Select *PythonLauncher* as the default application to open your
     script (or any .py script) through the finder Info window and
     double-click it.  *PythonLauncher* has various preferences to
     control how your script is launched. Option-dragging allows you to
     change these for one invocation, or use its Preferences menu to
     change things globally.

3.4.1.2 Running scripts with a GUI
..................................

With older versions of Python, there is one Mac OS X quirk that you
need to be aware of: programs that talk to the Aqua window manager (in
other words, anything that has a GUI) need to be run in a special way.
Use *pythonw* instead of *python* to start such scripts.

  With Python 2.5, you can use either *python* or *pythonw*.

3.4.1.3 Configuration
.....................

Python on OS X honors all standard Unix environment variables such as
PYTHONPATH, but setting these variables for programs started from
the Finder is non-standard as the Finder does not read your `.profile'
or `.cshrc' at startup. You need to create a file `~
/.MacOSX/environment.plist'. See Apple's Technical Document QA1067 for
details.

  For more information on installation Python packages in MacPython,
see section Installing Additional Python Packages.

3.4.2 The IDE
-------------

MacPython ships with the standard IDLE development environment. A good
introduction to using IDLE can be found at
<http://hkn.eecs.berkeley.edu/~dyoo/python/idle_intro/index.html>.

3.4.3 Installing Additional Python Packages
-------------------------------------------

There are several methods to install additional Python packages:

   * <http://pythonmac.org/packages/> contains selected compiled
     packages for Python 2.5, 2.4, and 2.3.

   * Packages can be installed via the standard Python distutils mode
     (`python setup.py install').

   * Many packages can also be installed via the *setuptools* extension.

3.4.4 GUI Programming on the Mac
--------------------------------

There are several options for building GUI applications on the Mac with
Python.

  _PyObjC_ is a Python binding to Apple's Objective-C/Cocoa framework,
which is the foundation of most modern Mac development. Information on
PyObjC is available from <http://pyobjc.sourceforge.net>.

  The standard Python GUI toolkit is Tkinter, based on the
cross-platform Tk toolkit (<http://www.tcl.tk>). An Aqua-native version
of Tk is bundled with OS X by Apple, and the latest version can be
downloaded and installed from <http://www.activestate.com>; it can also
be built from source.

  _wxPython_ is another popular cross-platform GUI toolkit that runs
natively on Mac OS X. Packages and documentation are available from
<http://www.wxpython.org>.

  _PyQt_ is another popular cross-platform GUI toolkit that runs
natively on Mac OS X. More information can be found at
<http://www.riverbankcomputing.co.uk/software/pyqt/intro>.

3.4.5 Distributing Python Applications on the Mac
-------------------------------------------------

The "Build Applet" tool that is placed in the MacPython 2.5 folder is
fine for packaging small Python scripts on your own machine to run as a
standard Mac application. This tool, however, is not robust enough to
distribute Python applications to other users.

  The standard tool for deploying standalone Python applications on the
Mac is *py2app*. More information on installing and using py2app can be
found at <http://undefined.org/python/#py2app>.

3.4.6 Application Scripting
---------------------------

Python can also be used to script other Mac applications via Apple's
Open Scripting Architecture (OSA); see
<http://appscript.sourceforge.net>. Appscript is a high-level,
user-friendly Apple event bridge that allows you to control scriptable
Mac OS X applications using ordinary Python scripts. Appscript makes
Python a serious alternative to Apple's own _AppleScript_ language for
automating your Mac. A related package, _PyOSA_, is an OSA language
component for the Python scripting language, allowing Python code to be
executed by any OSA-enabled application (Script Editor, Mail, iTunes,
etc.). PyOSA makes Python a full peer to AppleScript.

3.4.7 Other Resources
---------------------

The MacPython mailing list is an excellent support resource for Python
users and developers on the Mac:

  <http://www.python.org/community/sigs/current/pythonmac-sig/>

  Another useful resource is the MacPython wiki:

  <http://wiki.python.org/moin/MacPython>

4 The Python Language Reference
*******************************

     Release: 2.7

     Date: February 08, 2011

  This reference manual describes the syntax and "core semantics" of the
language. It is terse, but attempts to be exact and complete. The
semantics of non-essential built-in object types and of the built-in
functions and modules are described in The Python Standard Library.
For an informal introduction to the language, see
The Python Tutorial. For C or C++ programmers, two additional
manuals exist: Extending and Embedding the Python Interpreter
describes the high-level picture of how to write a Python
extension module, and the Python/C API Reference Manual.
describes the interfaces available to C/C++ programmers in detail.

4.1 Introduction
================

This reference manual describes the Python programming language. It is
not intended as a tutorial.

  While I am trying to be as precise as possible, I chose to use
English rather than formal specifications for everything except syntax
and lexical analysis.  This should make the document more
understandable to the average reader, but will leave room for
ambiguities. Consequently, if you were coming from Mars and tried to
re-implement Python from this document alone, you might have to guess
things and in fact you would probably end up implementing quite a
different language. On the other hand, if you are using Python and
wonder what the precise rules about a particular area of the language
are, you should definitely be able to find them here. If you would like
to see a more formal definition of the language, maybe you could
volunteer your time -- or invent a cloning machine :-).

  It is dangerous to add too many implementation details to a language
reference document -- the implementation may change, and other
implementations of the same language may work differently.  On the
other hand, there is currently only one Python implementation in
widespread use (although alternate implementations exist), and its
particular quirks are sometimes worth being mentioned, especially where
the implementation imposes additional limitations.  Therefore, you'll
find short "implementation notes" sprinkled throughout the text.

  Every Python implementation comes with a number of built-in and
standard modules.  These are documented in The Python Standard Library.
A few built-in modules are mentioned when they interact
in a significant way with the language definition.

4.1.1 Alternate Implementations
-------------------------------

Though there is one Python implementation which is by far the most
popular, there are some alternate implementations which are of
particular interest to different audiences.

  Known implementations include:

CPython
     This is the original and most-maintained implementation of Python,
     written in C.  New language features generally appear here first.

Jython
     Python implemented in Java.  This implementation can be used as a
     scripting language for Java applications, or can be used to create
     applications using the Java class libraries.  It is also often
     used to create tests for Java libraries.  More information can be
     found at the Jython website (http://www.jython.org/).

Python for .NET
     This implementation actually uses the CPython implementation, but
     is a managed .NET application and makes .NET libraries available.
     It was created by Brian Lloyd.  For more information, see the
     Python for .NET home page (http://pythonnet.sourceforge.net).

IronPython
     An alternate Python for .NET.  Unlike Python.NET, this is a
     complete Python implementation that generates IL, and compiles
     Python code directly to .NET assemblies.  It was created by Jim
     Hugunin, the original creator of Jython.  For more information,
     see the IronPython website (http://www.ironpython.com/).

PyPy
     An implementation of Python written completely in Python. It
     supports several advanced features not found in other
     implementations like stackless support and a Just in Time
     compiler. One of the goals of the project is to encourage
     experimentation with the language itself by making it easier to
     modify the interpreter (since it is written in Python).
     Additional information is available on the PyPy project's home
     page (http://pypy.org/).

  Each of these implementations varies in some way from the language as
documented in this manual, or introduces specific information beyond
what's covered in the standard Python documentation.  Please refer to
the implementation-specific documentation to determine what else you
need to know about the specific implementation you're using.

4.1.2 Notation
--------------

The descriptions of lexical analysis and syntax use a modified BNF
grammar notation.  This uses the following style of definition:

    name      ::= lc_letter (lc_letter | "_")*
    lc_letter ::= "a"..."z"

The first line says that a `name' is an `lc_letter' followed by a
sequence of zero or more `lc_letter's and underscores.  An `lc_letter'
in turn is any of the single characters `'a'' through `'z''.  (This
rule is actually adhered to for the names defined in lexical and
grammar rules in this document.)

  Each rule begins with a name (which is the name defined by the rule)
and `::='.  A vertical bar (`|') is used to separate alternatives; it
is the least binding operator in this notation.  A star (`*') means
zero or more repetitions of the preceding item; likewise, a plus (`+')
means one or more repetitions, and a phrase enclosed in square brackets
(`[ ]') means zero or one occurrences (in other words, the enclosed
phrase is optional).  The `*' and `+' operators bind as tightly as
possible; parentheses are used for grouping.  Literal strings are
enclosed in quotes.  White space is only meaningful to separate tokens.
Rules are normally contained on a single line; rules with many
alternatives may be formatted alternatively with each line after the
first beginning with a vertical bar.

  In lexical definitions (as the example above), two more conventions
are used: Two literal characters separated by three dots mean a choice
of any single character in the given (inclusive) range of ASCII
characters.  A phrase between angular brackets (`<...>') gives an
informal description of the symbol defined; e.g., this could be used to
describe the notion of 'control character' if needed.

  Even though the notation used is almost the same, there is a big
difference between the meaning of lexical and syntactic definitions: a
lexical definition operates on the individual characters of the input
source, while a syntax definition operates on the stream of tokens
generated by the lexical analysis.  All uses of BNF in the next chapter
("Lexical Analysis") are lexical definitions; uses in subsequent
chapters are syntactic definitions.

4.2 Lexical analysis
====================

A Python program is read by a _parser_.  Input to the parser is a
stream of _tokens_, generated by the _lexical analyzer_.  This chapter
describes how the lexical analyzer breaks a file into tokens.

  Python uses the 7-bit ASCII character set for program text.

  New in version 2.3: An encoding declaration can be used to indicate
that  string literals and comments use an encoding different from ASCII.

  For compatibility with older versions, Python only warns if it finds
8-bit characters; those warnings should be corrected by either
declaring an explicit encoding, or using escape sequences if those
bytes are binary data, instead of characters.

  The run-time character set depends on the I/O devices connected to
the program but is generally a superset of ASCII.

  *Future compatibility note:* It may be tempting to assume that the
character set for 8-bit characters is ISO Latin-1 (an ASCII superset
that covers most western languages that use the Latin alphabet), but it
is possible that in the future Unicode text editors will become common.
These generally use the UTF-8 encoding, which is also an ASCII
superset, but with very different use for the characters with ordinals
128-255.  While there is no consensus on this subject yet, it is unwise
to assume either Latin-1 or UTF-8, even though the current
implementation appears to favor Latin-1.  This applies both to the
source character set and the run-time character set.

4.2.1 Line structure
--------------------

A Python program is divided into a number of _logical lines_.

4.2.1.1 Logical lines
.....................

The end of a logical line is represented by the token NEWLINE.
Statements cannot cross logical line boundaries except where NEWLINE is
allowed by the syntax (e.g., between statements in compound
statements). A logical line is constructed from one or more _physical
lines_ by following the explicit or implicit _line joining_ rules.

4.2.1.2 Physical lines
......................

A physical line is a sequence of characters terminated by an end-of-line
sequence.  In source files, any of the standard platform line
termination sequences can be used - the Unix form using ASCII LF
(linefeed), the Windows form using the ASCII sequence CR LF (return
followed by linefeed), or the old Macintosh form using the ASCII CR
(return) character.  All of these forms can be used equally, regardless
of platform.

  When embedding Python, source code strings should be passed to Python
APIs using the standard C conventions for newline characters (the `\n'
character, representing ASCII LF, is the line terminator).

4.2.1.3 Comments
................

A comment starts with a hash character (`#') that is not part of a
string literal, and ends at the end of the physical line.  A comment
signifies the end of the logical line unless the implicit line joining
rules are invoked. Comments are ignored by the syntax; they are not
tokens.

4.2.1.4 Encoding declarations
.............................

If a comment in the first or second line of the Python script matches
the regular expression `coding[=:]\s*([-\w.]+)', this comment is
processed as an encoding declaration; the first group of this
expression names the encoding of the source code file. The recommended
forms of this expression are

    # -*- coding: <encoding-name> -*-

which is recognized also by GNU Emacs, and

    # vim:fileencoding=<encoding-name>

which is recognized by Bram Moolenaar's VIM. In addition, if the first
bytes of the file are the UTF-8 byte-order mark (`'\xef\xbb\xbf''), the
declared file encoding is UTF-8 (this is supported, among others, by
Microsoft's *notepad*).

  If an encoding is declared, the encoding name must be recognized by
Python. The encoding is used for all lexical analysis, in particular to
find the end of a string, and to interpret the contents of Unicode
literals. String literals are converted to Unicode for syntactical
analysis, then converted back to their original encoding before
interpretation starts. The encoding declaration must appear on a line
of its own.

4.2.1.5 Explicit line joining
.............................

Two or more physical lines may be joined into logical lines using
backslash characters (`\'), as follows: when a physical line ends in a
backslash that is not part of a string literal or comment, it is joined
with the following forming a single logical line, deleting the
backslash and the following end-of-line character.  For example:

    if 1900 < year < 2100 and 1 <= month <= 12 \
       and 1 <= day <= 31 and 0 <= hour < 24 \
       and 0 <= minute < 60 and 0 <= second < 60:   # Looks like a valid date
            return 1

A line ending in a backslash cannot carry a comment.  A backslash does
not continue a comment.  A backslash does not continue a token except
for string literals (i.e., tokens other than string literals cannot be
split across physical lines using a backslash).  A backslash is illegal
elsewhere on a line outside a string literal.

4.2.1.6 Implicit line joining
.............................

Expressions in parentheses, square brackets or curly braces can be
split over more than one physical line without using backslashes. For
example:

    month_names = ['Januari', 'Februari', 'Maart',      # These are the
                   'April',   'Mei',      'Juni',       # Dutch names
                   'Juli',    'Augustus', 'September',  # for the months
                   'Oktober', 'November', 'December']   # of the year

Implicitly continued lines can carry comments.  The indentation of the
continuation lines is not important.  Blank continuation lines are
allowed.  There is no NEWLINE token between implicit continuation
lines.  Implicitly continued lines can also occur within triple-quoted
strings (see below); in that case they cannot carry comments.

4.2.1.7 Blank lines
...................

A logical line that contains only spaces, tabs, formfeeds and possibly a
comment, is ignored (i.e., no NEWLINE token is generated).  During
interactive input of statements, handling of a blank line may differ
depending on the implementation of the read-eval-print loop.  In the
standard implementation, an entirely blank logical line (i.e. one
containing not even whitespace or a comment) terminates a multi-line
statement.

4.2.1.8 Indentation
...................

Leading whitespace (spaces and tabs) at the beginning of a logical line
is used to compute the indentation level of the line, which in turn is
used to determine the grouping of statements.

  First, tabs are replaced (from left to right) by one to eight spaces
such that the total number of characters up to and including the
replacement is a multiple of eight (this is intended to be the same
rule as used by Unix).  The total number of spaces preceding the first
non-blank character then determines the line's indentation.
Indentation cannot be split over multiple physical lines using
backslashes; the whitespace up to the first backslash determines the
indentation.

  *Cross-platform compatibility note:* because of the nature of text
editors on non-UNIX platforms, it is unwise to use a mixture of spaces
and tabs for the indentation in a single source file.  It should also
be noted that different platforms may explicitly limit the maximum
indentation level.

  A formfeed character may be present at the start of the line; it will
be ignored for the indentation calculations above.  Formfeed characters
occurring elsewhere in the leading whitespace have an undefined effect
(for instance, they may reset the space count to zero).

  The indentation levels of consecutive lines are used to generate
INDENT and DEDENT tokens, using a stack, as follows.

  Before the first line of the file is read, a single zero is pushed on
the stack; this will never be popped off again.  The numbers pushed on
the stack will always be strictly increasing from bottom to top.  At
the beginning of each logical line, the line's indentation level is
compared to the top of the stack.  If it is equal, nothing happens. If
it is larger, it is pushed on the stack, and one INDENT token is
generated.  If it is smaller, it _must_ be one of the numbers occurring
on the stack; all numbers on the stack that are larger are popped off,
and for each number popped off a DEDENT token is generated.  At the end
of the file, a DEDENT token is generated for each number remaining on
the stack that is larger than zero.

  Here is an example of a correctly (though confusingly) indented piece
of Python code:

    def perm(l):
            # Compute the list of all permutations of l
        if len(l) <= 1:
                      return [l]
        r = []
        for i in range(len(l)):
                 s = l[:i] + l[i+1:]
                 p = perm(s)
                 for x in p:
                  r.append(l[i:i+1] + x)
        return r

The following example shows various indentation errors:

     def perm(l):                       # error: first line indented
    for i in range(len(l)):             # error: not indented
        s = l[:i] + l[i+1:]
            p = perm(l[:i] + l[i+1:])   # error: unexpected indent
            for x in p:
                    r.append(l[i:i+1] + x)
                return r                # error: inconsistent dedent

(Actually, the first three errors are detected by the parser; only the
last error is found by the lexical analyzer -- the indentation of
`return r' does not match a level popped off the stack.)

4.2.1.9 Whitespace between tokens
.................................

Except at the beginning of a logical line or in string literals, the
whitespace characters space, tab and formfeed can be used
interchangeably to separate tokens.  Whitespace is needed between two
tokens only if their concatenation could otherwise be interpreted as a
different token (e.g., ab is one token, but a b is two tokens).

4.2.2 Other tokens
------------------

Besides NEWLINE, INDENT and DEDENT, the following categories of tokens
exist: _identifiers_, _keywords_, _literals_, _operators_, and
_delimiters_. Whitespace characters (other than line terminators,
discussed earlier) are not tokens, but serve to delimit tokens. Where
ambiguity exists, a token comprises the longest possible string that
forms a legal token, when read from left to right.

4.2.3 Identifiers and keywords
------------------------------

Identifiers (also referred to as _names_) are described by the
following lexical definitions:

    identifier ::= (letter|"_") (letter | digit | "_")*
    letter     ::= lowercase | uppercase
    lowercase  ::= "a"..."z"
    uppercase  ::= "A"..."Z"
    digit      ::= "0"..."9"

Identifiers are unlimited in length.  Case is significant.

4.2.3.1 Keywords
................

The following identifiers are used as reserved words, or _keywords_ of
the language, and cannot be used as ordinary identifiers.  They must be
spelled exactly as written here:

    and       del       from      not       while
    as        elif      global    or        with
    assert    else      if        pass      yield
    break     except    import    print
    class     exec      in        raise
    continue  finally   is        return
    def       for       lambda    try

Changed in version 2.4: None became a constant and is now
recognized by the compiler as a name for the built-in object
None.  Although it is not a keyword, you cannot assign a different
object to it.

  Changed in version 2.5: Both as and with are
only recognized when the `with_statement' future feature has been
enabled. It will always be enabled in Python 2.6.  See section
The with statement for details.  Note that using as.
and with as identifiers will always issue a warning, even
when the `with_statement' future directive is not in effect.

4.2.3.2 Reserved classes of identifiers
.......................................

Certain classes of identifiers (besides keywords) have special
meanings.  These classes are identified by the patterns of leading and
trailing underscore characters:

`_*'
     Not imported by `from module import *'.  The special identifier
     `_' is used in the interactive interpreter to store the result of
     the last evaluation; it is stored in the __builtin__.
     module.  When not in interactive mode, `_' has no special meaning
     and is not defined. See section The import statement.

          Note: The name `_' is often used in conjunction with
          internationalization; refer to the documentation for the
          gettext module for more information on this
          convention.

`__*__'
     System-defined names. These names are defined by the interpreter
     and its implementation (including the standard library).  Current
     system names are discussed in the Special method names.
     section and elsewhere.  More will likely be defined in future
     versions of Python.  _Any_ use of `__*__' names, in any context,
     that does not follow explicitly documented use, is subject to
     breakage without warning.

`__*'
     Class-private names.  Names in this category, when used within the
     context of a class definition, are re-written to use a mangled
     form to help avoid name clashes between "private" attributes of
     base and derived classes. See section Identifiers (Names).

4.2.4 Literals
--------------

Literals are notations for constant values of some built-in types.

4.2.4.1 String literals
.......................

String literals are described by the following lexical definitions:

    stringliteral   ::= [stringprefix](shortstring | longstring)
    stringprefix    ::= "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR"
                     | "b" | "B" | "br" | "Br" | "bR" | "BR"
    shortstring     ::= "'" shortstringitem* "'" | '"' shortstringitem* '"'
    longstring      ::= "'''" longstringitem* "'''"
                   | '"""' longstringitem* '"""'
    shortstringitem ::= shortstringchar | escapeseq
    longstringitem  ::= longstringchar | escapeseq
    shortstringchar ::= <any source character except "\" or newline or the quote>
    longstringchar  ::= <any source character except "\">
    escapeseq       ::= "\" <any ASCII character>

One syntactic restriction not indicated by these productions is that
whitespace is not allowed between the stringprefix and the
rest of the string literal. The source character set is defined by the
encoding declaration; it is ASCII if no encoding declaration is given
in the source file; see section Encoding declarations.

  In plain English: String literals can be enclosed in matching single
quotes (`'') or double quotes (`"').  They can also be enclosed in
matching groups of three single or double quotes (these are generally
referred to as _triple-quoted strings_).  The backslash (`\') character
is used to escape characters that otherwise have a special meaning,
such as newline, backslash itself, or the quote character.  String
literals may optionally be prefixed with a letter `'r'' or `'R''; such
strings are called _raw strings_ and use different rules for
interpreting backslash escape sequences.  A prefix of `'u'' or `'U''
makes the string a Unicode string.  Unicode strings use the Unicode
character set as defined by the Unicode Consortium and ISO 10646.  Some
additional escape sequences, described below, are available in Unicode
strings.  A prefix of `'b'' or `'B'' is ignored in Python 2; it
indicates that the literal should become a bytes literal in Python 3
(e.g. when code is automatically converted with 2to3).  A `'u'' or
`'b'' prefix may be followed by an `'r'' prefix.

  In triple-quoted strings, unescaped newlines and quotes are allowed
(and are retained), except that three unescaped quotes in a row
terminate the string.  (A "quote" is the character used to open the
string, i.e. either `'' or `"'.)

  Unless an `'r'' or `'R'' prefix is present, escape sequences in
strings are interpreted according to rules similar to those used by
Standard C.  The recognized escape sequences are:

Escape Sequence       Meaning                               Notes
------------------------------------------------------------------------
`\newline'            Ignored
`\\'                  Backslash (`\')
`\''                  Single quote (`'')
`\"'                  Double quote (`"')
`\a'                  ASCII Bell (BEL)
`\b'                  ASCII Backspace (BS)
`\f'                  ASCII Formfeed (FF)
`\n'                  ASCII Linefeed (LF)
`\N{name}'            Character named _name_ in the
                      Unicode database (Unicode only)
`\r'                  ASCII Carriage Return (CR)
`\t'                  ASCII Horizontal Tab (TAB)
`\uxxxx'              Character with 16-bit hex value       (1)
                      _xxxx_ (Unicode only)
`\Uxxxxxxxx'          Character with 32-bit hex value       (2)
                      _xxxxxxxx_ (Unicode only)
`\v'                  ASCII Vertical Tab (VT)
`\ooo'                Character with octal value _ooo_      (3,5)
`\xhh'                Character with hex value _hh_         (4,5)

  Notes:

  1. Individual code units which form parts of a surrogate pair can be
     encoded using this escape sequence.

  2. Any Unicode character can be encoded this way, but characters
     outside the Basic Multilingual Plane (BMP) will be encoded using a
     surrogate pair if Python is compiled to use 16-bit code units (the
     default).  Individual code units which form parts of a surrogate
     pair can be encoded using this escape sequence.

  3. As in Standard C, up to three octal digits are accepted.

  4. Unlike in Standard C, exactly two hex digits are required.

  5. In a string literal, hexadecimal and octal escapes denote the byte
     with the given value; it is not necessary that the byte encodes a
     character in the source character set. In a Unicode literal, these
     escapes denote a Unicode character with the given value.

  Unlike Standard C, all unrecognized escape sequences are left in the
string unchanged, i.e., _the backslash is left in the string_.  (This
behavior is useful when debugging: if an escape sequence is mistyped,
the resulting output is more easily recognized as broken.)  It is also
important to note that the escape sequences marked as "(Unicode only)"
in the table above fall into the category of unrecognized escapes for
non-Unicode string literals.

  When an `'r'' or `'R'' prefix is present, a character following a
backslash is included in the string without change, and _all
backslashes are left in the string_.  For example, the string literal
`r"\n"' consists of two characters: a backslash and a lowercase `'n''.
String quotes can be escaped with a backslash, but the backslash
remains in the string; for example, `r"\""' is a valid string literal
consisting of two characters: a backslash and a double quote; `r"\"' is
not a valid string literal (even a raw string cannot end in an odd
number of backslashes).  Specifically, _a raw string cannot end in a
single backslash_ (since the backslash would escape the following quote
character).  Note also that a single backslash followed by a newline is
interpreted as those two characters as part of the string, _not_ as a
line continuation.

  When an `'r'' or `'R'' prefix is used in conjunction with a `'u'' or
`'U'' prefix, then the `\uXXXX' and `\UXXXXXXXX' escape sequences are
processed while  _all other backslashes are left in the string_. For
example, the string literal `ur"\u0062\n"' consists of three Unicode
characters: 'LATIN SMALL LETTER B', 'REVERSE SOLIDUS', and 'LATIN SMALL
LETTER N'. Backslashes can be escaped with a preceding backslash;
however, both remain in the string.  As a result, `\uXXXX' escape
sequences are only recognized when there are an odd number of
backslashes.

4.2.4.2 String literal concatenation
....................................

Multiple adjacent string literals (delimited by whitespace), possibly
using different quoting conventions, are allowed, and their meaning is
the same as their concatenation.  Thus, `"hello" 'world'' is equivalent
to `"helloworld"'.  This feature can be used to reduce the number of
backslashes needed, to split long strings conveniently across long
lines, or even to add comments to parts of strings, for example:

    re.compile("[A-Za-z_]"       # letter or underscore
               "[A-Za-z0-9_]*"   # letter, digit or underscore
              )

Note that this feature is defined at the syntactical level, but
implemented at compile time.  The '+' operator must be used to
concatenate string expressions at run time.  Also note that literal
concatenation can use different quoting styles for each component (even
mixing raw strings and triple quoted strings).

4.2.4.3 Numeric literals
........................

There are four types of numeric literals: plain integers, long integers,
floating point numbers, and imaginary numbers.  There are no complex
literals (complex numbers can be formed by adding a real number and an
imaginary number).

  Note that numeric literals do not include a sign; a phrase like `-1'
is actually an expression composed of the unary operator '`-'' and the
literal `1'.

4.2.4.4 Integer and long integer literals
.........................................

Integer and long integer literals are described by the following lexical
definitions:

    longinteger    ::= integer ("l" | "L")
    integer        ::= decimalinteger | octinteger | hexinteger | bininteger
    decimalinteger ::= nonzerodigit digit* | "0"
    octinteger     ::= "0" ("o" | "O") octdigit+ | "0" octdigit+
    hexinteger     ::= "0" ("x" | "X") hexdigit+
    bininteger     ::= "0" ("b" | "B") bindigit+
    nonzerodigit   ::= "1"..."9"
    octdigit       ::= "0"..."7"
    bindigit       ::= "0" | "1"
    hexdigit       ::= digit | "a"..."f" | "A"..."F"

Although both lower case `'l'' and upper case `'L'' are allowed as
suffix for long integers, it is strongly recommended to always use
`'L'', since the letter `'l'' looks too much like the digit `'1''.

  Plain integer literals that are above the largest representable plain
integer (e.g., 2147483647 when using 32-bit arithmetic) are accepted as
if they were long integers instead. (1)  There is no limit for long
integer literals apart from what can be stored in available memory.

  Some examples of plain integer literals (first row) and long integer
literals (second and third rows):

    7     2147483647                        0177
    3L    79228162514264337593543950336L    0377L   0x100000000L
          79228162514264337593543950336             0xdeadbeef

4.2.4.5 Floating point literals
...............................

Floating point literals are described by the following lexical
definitions:

    floatnumber   ::= pointfloat | exponentfloat
    pointfloat    ::= [intpart] fraction | intpart "."
    exponentfloat ::= (intpart | pointfloat) exponent
    intpart       ::= digit+
    fraction      ::= "." digit+
    exponent      ::= ("e" | "E") ["+" | "-"] digit+

Note that the integer and exponent parts of floating point numbers can
look like octal integers, but are interpreted using radix 10.  For
example, `077e010' is legal, and denotes the same number as `77e10'.
The allowed range of floating point literals is
implementation-dependent. Some examples of floating point literals:

    3.14    10.    .001    1e100    3.14e-10    0e0

Note that numeric literals do not include a sign; a phrase like `-1' is
actually an expression composed of the unary operator `-' and the
literal `1'.

4.2.4.6 Imaginary literals
..........................

Imaginary literals are described by the following lexical definitions:

    imagnumber ::= (floatnumber | intpart) ("j" | "J")

An imaginary literal yields a complex number with a real part of 0.0.
Complex numbers are represented as a pair of floating point numbers and
have the same restrictions on their range.  To create a complex number
with a nonzero real part, add a floating point number to it, e.g.,
`(3+4j)'.  Some examples of imaginary literals:

    3.14j   10.j    10j     .001j   1e100j  3.14e-10j

4.2.5 Operators
---------------

The following tokens are operators:

    +       -       *       **      /       //      %
    <<      >>      &       |       ^       ~
    <       >       <=      >=      ==      !=      <>

The comparison operators `<>' and `!=' are alternate spellings of the
same operator.  `!=' is the preferred spelling; `<>' is obsolescent.

4.2.6 Delimiters
----------------

The following tokens serve as delimiters in the grammar:

    (       )       [       ]       {       }      @
    ,       :       .       `       =       ;
    +=      -=      *=      /=      //=     %=
    &=      |=      ^=      >>=     <<=     **=

The period can also occur in floating-point and imaginary literals.  A
sequence of three periods has a special meaning as an ellipsis in
slices. The second half of the list, the augmented assignment
operators, serve lexically as delimiters, but also perform an operation.

  The following printing ASCII characters have special meaning as part
of other tokens or are otherwise significant to the lexical analyzer:

    '       "       #       \

The following printing ASCII characters are not used in Python.  Their
occurrence outside string literals and comments is an unconditional
error:

    $       ?

Footnotes
.........

     (1) In versions of Python prior to 2.4, octal and hexadecimal
     literals in the range just above the largest representable plain
     integer but below the largest unsigned 32-bit number (on a machine
     using 32-bit arithmetic), 4294967296, were taken as the negative
     plain integer obtained by subtracting 4294967296 from their
     unsigned value.

4.3 Data model
==============

4.3.1 Objects, values and types
-------------------------------

_Objects_ are Python's abstraction for data.  All data in a Python
program is represented by objects or by relations between objects. (In
a sense, and in conformance to Von Neumann's model of a "stored program
computer," code is also represented by objects.)

  Every object has an identity, a type and a value.  An object's
_identity_ never changes once it has been created; you may think of it
as the object's address in memory.  The 'is.' operator
compares the identity of two objects; the id() function
returns an integer representing its identity (currently implemented as
its address). An object's _type_ is also unchangeable. (1) An object's
type determines the operations that the object supports (e.g., "does it
have a length?") and also defines the possible values for objects of
that type.  The type() function returns an object's type
(which is an object itself).  The _value_ of some objects can change.
Objects whose value can change are said to be _mutable_; objects whose
value is unchangeable once they are created are called _immutable_.
(The value of an immutable container object that contains a reference
to a mutable object can change when the latter's value is changed;
however the container is still considered immutable, because the
collection of objects it contains cannot be changed.  So, immutability
is not strictly the same as having an unchangeable value, it is more
subtle.) An object's mutability is determined by its type; for
instance, numbers, strings and tuples are immutable, while dictionaries
and lists are mutable.

  Objects are never explicitly destroyed; however, when they become
unreachable they may be garbage-collected.  An implementation is
allowed to postpone garbage collection or omit it altogether -- it is a
matter of implementation quality how garbage collection is implemented,
as long as no objects are collected that are still reachable.

*CPython implementation detail:* CPython currently uses a
reference-counting scheme with (optional) delayed detection of
cyclically linked garbage, which collects most objects as soon as they
become unreachable, but is not guaranteed to collect garbage containing
circular references.  See the documentation of the gc.
module for information on controlling the collection of cyclic garbage.
Other implementations act differently and CPython may change.

  Note that the use of the implementation's tracing or debugging
facilities may keep objects alive that would normally be collectable.
Also note that catching an exception with a 'try...See
except.' statement may keep objects alive.

  Some objects contain references to "external" resources such as open
files or windows.  It is understood that these resources are freed when
the object is garbage-collected, but since garbage collection is not
guaranteed to happen, such objects also provide an explicit way to
release the external resource, usually a `close()' method. Programs are
strongly recommended to explicitly close such objects.  The 'try...
See finally.' statement provides a convenient way to do
this.

  Some objects contain references to other objects; these are called
_containers_.  Examples of containers are tuples, lists and
dictionaries.  The references are part of a container's value.  In most
cases, when we talk about the value of a container, we imply the
values, not the identities of the contained objects; however, when we
talk about the mutability of a container, only the identities of the
immediately contained objects are implied.  So, if an immutable
container (like a tuple) contains a reference to a mutable object, its
value changes if that mutable object is changed.

  Types affect almost all aspects of object behavior.  Even the
importance of object identity is affected in some sense: for immutable
types, operations that compute new values may actually return a
reference to any existing object with the same type and value, while
for mutable objects this is not allowed.  E.g., after `a = 1; b = 1',
`a' and `b' may or may not refer to the same object with the value one,
depending on the implementation, but after `c = []; d = []', `c' and
`d' are guaranteed to refer to two different, unique, newly created
empty lists. (Note that `c = d = []' assigns the same object to both
`c' and `d'.)

4.3.2 The standard type hierarchy
---------------------------------

Below is a list of the types that are built into Python.  Extension
modules (written in C, Java, or other languages, depending on the
implementation) can define additional types.  Future versions of Python
may add types to the type hierarchy (e.g., rational numbers,
efficiently stored arrays of integers, etc.).

  Some of the type descriptions below contain a paragraph listing
'special attributes.'  These are attributes that provide access to the
implementation and are not intended for general use.  Their definition
may change in the future.

None
     This type has a single value.  There is a single object with this
     value. This object is accessed through the built-in name `None'.
     It is used to signify the absence of a value in many situations,
     e.g., it is returned from functions that don't explicitly return
     anything. Its truth value is false.

NotImplemented
     This type has a single value.  There is a single object with this
     value. This object is accessed through the built-in name
     `NotImplemented'. Numeric methods and rich comparison methods may
     return this value if they do not implement the operation for the
     operands provided.  (The interpreter will then try the reflected
     operation, or some other fallback, depending on the operator.)  Its
     truth value is true.

Ellipsis
     This type has a single value.  There is a single object with this
     value. This object is accessed through the built-in name
     `Ellipsis'. It is used to indicate the presence of the `...'
     syntax in a slice.  Its truth value is true.

See numbers.Number.
     These are created by numeric literals and returned as results by
     arithmetic operators and arithmetic built-in functions.  Numeric
     objects are immutable; once created their value never changes.
     Python numbers are of course strongly related to mathematical
     numbers, but subject to the limitations of numerical
     representation in computers.

     Python distinguishes between integers, floating point numbers, and
     complex numbers:

    numbers.Integral.
          These represent elements from the mathematical set of
          integers (positive and negative).

          There are three types of integers:

         Plain integers
               These represent numbers in the range -2147483648 through
               2147483647.  (The range may be larger on machines with a
               larger natural word size, but not smaller.)  When the
               result of an operation would fall outside this range,
               the result is normally returned as a long integer (in
               some cases, the exception OverflowError is
               raised instead).  For the purpose of shift and mask
               operations, integers are assumed to have a binary, 2's
               complement notation using 32 or more bits, and hiding no
               bits from the user (i.e., all 4294967296 different bit
               patterns correspond to different values).

         Long integers
               These represent numbers in an unlimited range, subject
               to available (virtual) memory only.  For the purpose of
               shift and mask operations, a binary representation is
               assumed, and negative numbers are represented in a
               variant of 2's complement which gives the illusion of an
               infinite string of sign bits extending to the left.

         Booleans
               These represent the truth values False and True.  The
               two objects representing the values False and True are
               the only Boolean objects.  The Boolean type is a subtype
               of plain integers, and Boolean values behave like the
               values 0 and 1, respectively, in almost all contexts,
               the exception being that when converted to a string, the
               strings `"False"' or `"True"' are returned, respectively.

          The rules for integer representation are intended to give the
          most meaningful interpretation of shift and mask operations
          involving negative integers and the least surprises when
          switching between the plain and long integer domains.  Any
          operation, if it yields a result in the plain integer domain,
          will yield the same result in the long integer domain or when
          using mixed operands.  The switch between domains is
          transparent to the programmer.

    See numbers.Real. (see float)
          These represent machine-level double precision floating point
          numbers. You are at the mercy of the underlying machine
          architecture (and C or Java implementation) for the accepted
          range and handling of overflow. Python does not support
          single-precision floating point numbers; the savings in
          processor and memory usage that are usually the reason for
          using these is dwarfed by the overhead of using objects in
          Python, so there is no reason to complicate the language with
          two kinds of floating point numbers.

    See numbers.Complex.
          These represent complex numbers as a pair of machine-level
          double precision floating point numbers.  The same caveats
          apply as for floating point numbers.  The real and imaginary
          parts of a complex number `z' can be retrieved through the
          read-only attributes `z.real' and `z.imag'.

Sequences
     These represent finite ordered sets indexed by non-negative
     numbers. The built-in function len() returns the
     number of items of a sequence. When the length of a sequence is
     _n_, the index set contains the numbers 0, 1, ..., _n_-1.  Item
     _i_ of sequence _a_ is selected by `a[i]'.

     Sequences also support slicing: `a[i:j]' selects all items with
     index _k_ such that _i_ `<=' _k_ `<' _j_.  When used as an
     expression, a slice is a sequence of the same type.  This implies
     that the index set is renumbered so that it starts at 0.

     Some sequences also support "extended slicing" with a third "step"
     parameter: `a[i:j:k]' selects all items of _a_ with index _x_
     where `x = i + n*k', _n_ `>=' `0' and _i_ `<=' _x_ `<' _j_.

     Sequences are distinguished according to their mutability:

    Immutable sequences
          An object of an immutable sequence type cannot change once it
          is created.  (If the object contains references to other
          objects, these other objects may be mutable and may be
          changed; however, the collection of objects directly
          referenced by an immutable object cannot change.)

          The following types are immutable sequences:

         Strings
               The items of a string are characters.  There is no
               separate character type; a character is represented by a
               string of one item. Characters represent (at least)
               8-bit bytes.  The built-in functions chr().
               and ord() convert between characters and
               nonnegative integers representing the byte values.  Bytes
               with the values 0-127 usually represent the
               corresponding ASCII values, but the interpretation of
               values is up to the program.  The string data type is
               also used to represent arrays of bytes, e.g., to hold
               data read from a file.

               (On systems whose native character set is not ASCII,
               strings may use EBCDIC in their internal representation,
               provided the functions chr() and
               ord() implement a mapping between ASCII and
               EBCDIC, and string comparison preserves the ASCII order.
               Or perhaps someone can propose a better rule?)

         Unicode
               The items of a Unicode object are Unicode code units.  A
               Unicode code unit is represented by a Unicode object of
               one item and can hold either a 16-bit or 32-bit value
               representing a Unicode ordinal (the maximum value for
               the ordinal is given in `sys.maxunicode', and depends on
               how Python is configured at compile time).  Surrogate
               pairs may be present in the Unicode object, and will be
               reported as two separate items.  The built-in functions
               unichr() and ord() convert
               between code units and nonnegative integers representing
               the Unicode ordinals as defined in the Unicode Standard
               3.0. Conversion from and to other encodings are possible
               through the Unicode method `encode()' and the built-in
               function unicode().

         Tuples
               The items of a tuple are arbitrary Python objects.
               Tuples of two or more items are formed by
               comma-separated lists of expressions.  A tuple of one
               item (a 'singleton') can be formed by affixing a comma
               to an expression (an expression by itself does not
               create a tuple, since parentheses must be usable for
               grouping of expressions).  An empty tuple can be formed
               by an empty pair of parentheses.

    Mutable sequences
          Mutable sequences can be changed after they are created.  The
          subscription and slicing notations can be used as the target
          of assignment and del.  (delete) statements.

          There are currently two intrinsic mutable sequence types:

         Lists
               The items of a list are arbitrary Python objects.  Lists
               are formed by placing a comma-separated list of
               expressions in square brackets. (Note that there are no
               special cases needed to form lists of length 0 or 1.)

         Byte Arrays
               A bytearray object is a mutable array. They are created
               by the built-in bytearray() constructor.
               Aside from being mutable (and hence unhashable), byte
               arrays otherwise provide the same interface and
               functionality as immutable bytes objects.

          The extension module array provides an additional
          example of a mutable sequence type.

Set types
     These represent unordered, finite sets of unique, immutable
     objects. As such, they cannot be indexed by any subscript.
     However, they can be iterated over, and the built-in function
     len() returns the number of items in a set. Common
     uses for sets are fast membership testing, removing duplicates
     from a sequence, and computing mathematical operations such as
     intersection, union, difference, and symmetric difference.

     For set elements, the same immutability rules apply as for
     dictionary keys. Note that numeric types obey the normal rules for
     numeric comparison: if two numbers compare equal (e.g., `1' and
     `1.0'), only one of them can be contained in a set.

     There are currently two intrinsic set types:

    Sets
          These represent a mutable set. They are created by the
          built-in set().  constructor and can be modified
          afterwards by several methods, such as `add()'.

    Frozen sets
          These represent an immutable set.  They are created by the
          built-in frozenset() constructor.  As a frozenset
          is immutable and hashable, it can be used again as
          an element of another set, or as a dictionary key.

Mappings
     These represent finite sets of objects indexed by arbitrary index
     sets. The subscript notation `a[k]' selects the item indexed by
     `k' from the mapping `a'; this can be used in expressions and as
     the target of assignments or del statements. The
     built-in function len() returns the number of items in
     a mapping.

     There is currently a single intrinsic mapping type:

    Dictionaries
          These represent finite sets of objects indexed by nearly
          arbitrary values.  The only types of values not acceptable as
          keys are values containing lists or dictionaries or other
          mutable types that are compared by value rather than by
          object identity, the reason being that the efficient
          implementation of dictionaries requires a key's hash value to
          remain constant. Numeric types used for keys obey the normal
          rules for numeric comparison: if two numbers compare equal
          (e.g., `1' and `1.0') then they can be used interchangeably
          to index the same dictionary entry.

          Dictionaries are mutable; they can be created by the `{...}'
          notation (see section Dictionary displays).

          The extension modules dbm, gdbm, and
          bsddb provide additional examples of mapping types.

Callable types
     These are the types to which the function call operation (see
     section Calls) can be applied:

    User-defined functions
          A user-defined function object is created by a function
          definition (see section Function definitions).
          It should be called with an argument list containing the same
          number of items as the function's formal parameter list.

          Special attributes:

          Attribute                   Meaning
          --------------------------------------------------------------------------------
          `func_doc'                  The function's documentation        Writable
                                      string, or `None' if unavailable
          `__doc__'                   Another way of spelling `func_doc'  Writable
          `func_name'                 The function's name                 Writable
          `__name__'                  Another way of spelling `func_name' Writable
          `__module__'                The name of the module the          Writable
                                      function was defined in, or `None'
                                      if unavailable.
          `func_defaults'             A tuple containing default          Writable
                                      argument values for those
                                      arguments that have defaults, or
                                      `None' if no arguments have a
                                      default value
          `func_code'                 The code object representing the    Writable
                                      compiled function body.
          `func_globals'              A reference to the dictionary that  Read-only
                                      holds the function's global
                                      variables -- the global namespace
                                      of the module in which the function
                                      was defined.
          `func_dict'                 The namespace supporting arbitrary  Writable
                                      function attributes.
          `func_closure'              `None' or a tuple of cells that     Read-only
                                      contain bindings for the
                                      function's free variables.

          Most of the attributes labelled "Writable" check the type of
          the assigned value.

          Changed in version 2.4: `func_name' is now writable.

          Function objects also support getting and setting arbitrary
          attributes, which can be used, for example, to attach
          metadata to functions.  Regular attribute dot-notation is
          used to get and set such attributes. _Note that the current
          implementation only supports function attributes on
          user-defined functions.  Function attributes on built-in
          functions may be supported in the future._

          Additional information about a function's definition can be
          retrieved from its code object; see the description of
          internal types below.

    User-defined methods
          A user-defined method object combines a class, a class
          instance (or `None') and any callable object (normally a
          user-defined function).

          Special read-only attributes: `im_self' is the class instance
          object, `im_func' is the function object; `im_class' is the
          class of `im_self' for bound methods or the class that asked
          for the method for unbound methods; `__doc__' is the method's
          documentation (same as `im_func.__doc__'); `__name__' is the
          method name (same as `im_func.__name__'); `__module__' is the
          name of the module the method was defined in, or `None' if
          unavailable.

          Changed in version 2.2: `im_self' used to refer to the class
          that defined the method.

          Changed in version 2.6: For 3.0 forward-compatibility,
          `im_func' is also available as `__func__', and `im_self' as
          `__self__'.

          Methods also support accessing (but not setting) the
          arbitrary function attributes on the underlying function
          object.

          User-defined method objects may be created when getting an
          attribute of a class (perhaps via an instance of that class),
          if that attribute is a user-defined function object, an
          unbound user-defined method object, or a class method object.
          When the attribute is a user-defined method object, a new
          method object is only created if the class from which it is
          being retrieved is the same as, or a derived class of, the
          class stored in the original method object; otherwise, the
          original method object is used as it is.

          When a user-defined method object is created by retrieving a
          user-defined function object from a class, its `im_self'
          attribute is `None' and the method object is said to be
          unbound. When one is created by retrieving a user-defined
          function object from a class via one of its instances, its
          `im_self' attribute is the instance, and the method object is
          said to be bound. In either case, the new method's `im_class'
          attribute is the class from which the retrieval takes place,
          and its `im_func' attribute is the original function object.

          When a user-defined method object is created by retrieving
          another method object from a class or instance, the behaviour
          is the same as for a function object, except that the
          `im_func' attribute of the new instance is not the original
          method object but its `im_func' attribute.

          When a user-defined method object is created by retrieving a
          class method object from a class or instance, its `im_self'
          attribute is the class itself (the same as the `im_class'
          attribute), and its `im_func' attribute is the function
          object underlying the class method.

          When an unbound user-defined method object is called, the
          underlying function (`im_func') is called, with the
          restriction that the first argument must be an instance of
          the proper class (`im_class') or of a derived class thereof.

          When a bound user-defined method object is called, the
          underlying function (`im_func') is called, inserting the
          class instance (`im_self') in front of the argument list.
          For instance, when `C' is a class which contains a definition
          for a function `f()', and `x' is an instance of `C', calling
          `x.f(1)' is equivalent to calling `C.f(x, 1)'.

          When a user-defined method object is derived from a class
          method object, the "class instance" stored in `im_self' will
          actually be the class itself, so that calling either `x.f(1)'
          or `C.f(1)' is equivalent to calling `f(C,1)' where `f' is
          the underlying function.

          Note that the transformation from function object to (unbound
          or bound) method object happens each time the attribute is
          retrieved from the class or instance.  In some cases, a
          fruitful optimization is to assign the attribute to a local
          variable and call that local variable. Also notice that this
          transformation only happens for user-defined functions; other
          callable objects (and all non-callable objects) are retrieved
          without transformation.  It is also important to note that
          user-defined functions which are attributes of a class
          instance are not converted to bound methods; this _only_
          happens when the function is an attribute of the class.

    Generator functions
          A function or method which uses the yield.
          statement (see section The yield statement) is
          called a _generator function_.  Such a function, when called,
          always returns an iterator object which can be used to
          execute the body of the function:  calling the iterator's
          next() method will cause the function to execute
          until it provides a value using the yield.
          statement.  When the function executes a return.
          statement or falls off the end, a StopIteration.
          exception is raised and the iterator will have reached the
          end of the set of values to be returned.

    Built-in functions
          A built-in function object is a wrapper around a C function.
          Examples of built-in functions are len() and
          math.sin(). (see math is a standard
          built-in module). The number and type of the arguments are
          determined by the C function. Special read-only attributes:
          `__doc__' is the function's documentation string, or `None' if
          unavailable; `__name__' is the function's name; `__self__' is
          set to `None' (but see the next item); `__module__' is the
          name of the module the function was defined in or `None' if
          unavailable.

    Built-in methods
          This is really a different disguise of a built-in function,
          this time containing an object passed to the C function as an
          implicit extra argument.  An example of a built-in method is
          `alist.append()', assuming _alist_ is a list object. In this
          case, the special read-only attribute `__self__' is set to
          the object denoted by _list_.

    Class Types
          Class types, or "new-style classes," are callable.  These
          objects normally act as factories for new instances of
          themselves, but variations are possible for class types that
          override __new__().  The arguments of the call are
          passed to __new__() and, in the typical case, to
          __init__() to initialize the new instance.

    Classic Classes
          Class objects are described below.  When a class object is
          called, a new class instance (also described below) is
          created and returned.  This implies a call to the class's
          __init__() method if it has one.  Any arguments
          are passed on to the __init__() method.  If there
          is no __init__() method, the class must be called
          without arguments.

    Class instances
          Class instances are described below.  Class instances are
          callable only when the class has a __call__().
          method; `x(arguments)' is a shorthand for
          `x.__call__(arguments)'.

Modules
     Modules are imported by the import statement (see
     section The import statement). A module object has a
     namespace implemented by a dictionary object (this is the
     dictionary referenced by the func_globals attribute of functions
     defined in the module).  Attribute references are translated to
     lookups in this dictionary, e.g., `m.x' is equivalent to
     `m.__dict__["x"]'. A module object does not contain the code
     object used to initialize the module (since it isn't needed once
     the initialization is done).

     Attribute assignment updates the module's namespace dictionary,
     e.g., `m.x = 1' is equivalent to `m.__dict__["x"] = 1'.

     Special read-only attribute: `__dict__' is the module's namespace
     as a dictionary object.

     *CPython implementation detail:* Because of the way CPython clears
     module dictionaries, the module dictionary will be cleared when
     the module falls out of scope even if the dictionary still has
     live references.  To avoid this, copy the dictionary or keep the
     module around while using its dictionary directly.

     Predefined (writable) attributes: `__name__' is the module's name;
     `__doc__' is the module's documentation string, or `None' if
     unavailable; `__file__' is the pathname of the file from which the
     module was loaded, if it was loaded from a file. The `__file__'
     attribute is not present for C modules that are statically linked
     into the interpreter; for extension modules loaded dynamically
     from a shared library, it is the pathname of the shared library
     file.

Classes
     Both class types (new-style classes) and class objects
     (old-style/classic classes) are typically created by class
     definitions (see section Class definitions).  A class
     has a namespace implemented by a dictionary object.  Class
     attribute references are translated to lookups in this dictionary,
     e.g., `C.x' is translated to `C.__dict__["x"]' (although for
     new-style classes in particular there are a number of hooks which
     allow for other means of locating attributes). When the attribute
     name is not found there, the attribute search continues in the
     base classes.  For old-style classes, the search is depth-first,
     left-to-right in the order of occurrence in the base class list.
     New-style classes use the more complex C3 method resolution order
     which behaves correctly even in the presence of 'diamond'
     inheritance structures where there are multiple inheritance paths
     leading back to a common ancestor. Additional details on the C3
     MRO used by new-style classes can be found in the documentation
     accompanying the 2.3 release at
     <http://www.python.org/download/releases/2.3/mro/>.

     When a class attribute reference (for class `C', say) would yield a
     user-defined function object or an unbound user-defined method
     object whose associated class is either `C' or one of its base
     classes, it is transformed into an unbound user-defined method
     object whose `im_class' attribute is `C'. When it would yield a
     class method object, it is transformed into a bound user-defined
     method object whose `im_class' and `im_self' attributes are both
     `C'.  When it would yield a static method object, it is
     transformed into the object wrapped by the static method object.
     See section Implementing Descriptors for another way
     in which attributes retrieved from a class may differ from those
     actually contained in its `__dict__' (note that only new-style
     classes support descriptors).

     Class attribute assignments update the class's dictionary, never
     the dictionary of a base class.

     A class object can be called (see above) to yield a class instance
     (see below).

     Special attributes: `__name__' is the class name; `__module__' is
     the module name in which the class was defined; `__dict__' is the
     dictionary containing the class's namespace; `__bases__' is a tuple
     (possibly empty or a singleton) containing the base classes, in
     the order of their occurrence in the base class list; `__doc__' is
     the class's documentation string, or None if undefined.

Class instances
     A class instance is created by calling a class object (see above).
     A class instance has a namespace implemented as a dictionary which
     is the first place in which attribute references are searched.
     When an attribute is not found there, and the instance's class has
     an attribute by that name, the search continues with the class
     attributes.  If a class attribute is found that is a user-defined
     function object or an unbound user-defined method object whose
     associated class is the class (call it `C') of the instance for
     which the attribute reference was initiated or one of its bases,
     it is transformed into a bound user-defined method object whose
     `im_class' attribute is `C' and whose `im_self' attribute is the
     instance. Static method and class method objects are also
     transformed, as if they had been retrieved from class `C'; see
     above under "Classes". See section Implementing Descriptors
     for another way in which attributes of a class retrieved via
     its instances may differ from the objects actually stored in the
     class's `__dict__'. If no class attribute is found, and the
     object's class has a __getattr__().  method, that is
     called to satisfy the lookup.

     Attribute assignments and deletions update the instance's
     dictionary, never a class's dictionary.  If the class has a
     __setattr__() or __delattr__() method, this is
     called instead of updating the instance dictionary directly.

     Class instances can pretend to be numbers, sequences, or mappings
     if they have methods with certain special names.  See section
     Special method names.

     Special attributes: `__dict__' is the attribute dictionary;
     `__class__' is the instance's class.

Files
     A file object represents an open file.  File objects are created
     by the open() built-in function, and also by
     os.popen(), os.fdopen(), and the `makefile()'
     method of socket objects (and perhaps by other functions or
     methods provided by extension modules).  The objects `sys.stdin',
     `sys.stdout' and `sys.stderr' are initialized to file objects
     corresponding to the interpreter's standard input, output and
     error streams.  See File Objects for complete
     documentation of file objects.

Internal types
     A few types used internally by the interpreter are exposed to the
     user. Their definitions may change with future versions of the
     interpreter, but they are mentioned here for completeness.

    Code objects
          Code objects represent _byte-compiled_ executable Python
          code, or bytecode.  The difference between a code
          object and a function object is that the function object
          contains an explicit reference to the function's globals (the
          module in which it was defined), while a code object contains
          no context; also the default argument values are stored in
          the function object, not in the code object (because they
          represent values calculated at run-time).  Unlike function
          objects, code objects are immutable and contain no references
          (directly or indirectly) to mutable objects.

          Special read-only attributes: `co_name' gives the function
          name; `co_argcount' is the number of positional arguments
          (including arguments with default values); `co_nlocals' is
          the number of local variables used by the function (including
          arguments); `co_varnames' is a tuple containing the names of
          the local variables (starting with the argument names);
          `co_cellvars' is a tuple containing the names of local
          variables that are referenced by nested functions;
          `co_freevars' is a tuple containing the names of free
          variables; `co_code' is a string representing the sequence of
          bytecode instructions; `co_consts' is a tuple containing the
          literals used by the bytecode; `co_names' is a tuple
          containing the names used by the bytecode; `co_filename' is
          the filename from which the code was compiled;
          `co_firstlineno' is the first line number of the function;
          `co_lnotab' is a string encoding the mapping from bytecode
          offsets to line numbers (for details see the source code of
          the interpreter); `co_stacksize' is the required stack size
          (including local variables); `co_flags' is an integer
          encoding a number of flags for the interpreter.

          The following flag bits are defined for `co_flags': bit
          `0x04' is set if the function uses the `*arguments' syntax to
          accept an arbitrary number of positional arguments; bit
          `0x08' is set if the function uses the `**keywords' syntax to
          accept arbitrary keyword arguments; bit `0x20' is set if the
          function is a generator.

          Future feature declarations (`from __future__ import
          division') also use bits in `co_flags' to indicate whether a
          code object was compiled with a particular feature enabled:
          bit `0x2000' is set if the function was compiled with future
          division enabled; bits `0x10' and `0x1000' were used in
          earlier versions of Python.

          Other bits in `co_flags' are reserved for internal use.

          If a code object represents a function, the first item in
          `co_consts' is the documentation string of the function, or
          `None' if undefined.

    Frame objects
          Frame objects represent execution frames.  They may occur in
          traceback objects (see below).

          Special read-only attributes: `f_back' is to the previous
          stack frame (towards the caller), or `None' if this is the
          bottom stack frame; `f_code' is the code object being
          executed in this frame; `f_locals' is the dictionary used to
          look up local variables; `f_globals' is used for global
          variables; `f_builtins' is used for built-in (intrinsic)
          names; `f_restricted' is a flag indicating whether the
          function is executing in restricted execution mode; `f_lasti'
          gives the precise instruction (this is an index into the
          bytecode string of the code object).

          Special writable attributes: `f_trace', if not `None', is a
          function called at the start of each source code line (this
          is used by the debugger); `f_exc_type', `f_exc_value',
          `f_exc_traceback' represent the last exception raised in the
          parent frame provided another exception was ever raised in
          the current frame (in all other cases they are None);
          `f_lineno' is the current line number of the frame -- writing
          to this from within a trace function jumps to the given line
          (only for the bottom-most frame).  A debugger can implement a
          Jump command (aka Set Next Statement) by writing to f_lineno.

    Traceback objects
          Traceback objects represent a stack trace of an exception.  A
          traceback object is created when an exception occurs.  When
          the search for an exception handler unwinds the execution
          stack, at each unwound level a traceback object is inserted
          in front of the current traceback.  When an exception handler
          is entered, the stack trace is made available to the program.
          (See section The try statement) It is accessible
          as `sys.exc_traceback', and also as the third item of the
          tuple returned by `sys.exc_info()'.  The latter is the
          preferred interface, since it works correctly when the
          program is using multiple threads. When the program contains
          no suitable handler, the stack trace is written (nicely
          formatted) to the standard error stream; if the interpreter
          is interactive, it is also made available to the user as
          `sys.last_traceback'.

          Special read-only attributes: `tb_next' is the next level in
          the stack trace (towards the frame where the exception
          occurred), or `None' if there is no next level; `tb_frame'
          points to the execution frame of the current level;
          `tb_lineno' gives the line number where the exception
          occurred; `tb_lasti' indicates the precise instruction.  The
          line number and last instruction in the traceback may differ
          from the line number of its frame object if the exception
          occurred in a try statement with no matching
          except clause or with a finally clause.

    Slice objects
          Slice objects are used to represent slices when _extended
          slice syntax_ is used.  This is a slice using two colons, or
          multiple slices or ellipses separated by commas, e.g.,
          `a[i:j:step]', `a[i:j, k:l]', or `a[..., i:j]'.  They are
          also created by the built-in slice() function.

          Special read-only attributes: `start' is the lower bound;
          `stop' is the upper bound; `step' is the step value; each is
          `None' if omitted.  These attributes can have any type.

          Slice objects support one method:

           -- Method: slice.indices (self, length)
               This method takes a single integer argument _length_ and
               computes information about the extended slice that the
               slice object would describe if applied to a sequence of
               _length_ items.  It returns a tuple of three integers;
               respectively these are the _start_ and _stop_ indices
               and the _step_ or stride length of the slice. Missing or
               out-of-bounds indices are handled in a manner consistent
               with regular slices.

               New in version 2.3.

    Static method objects
          Static method objects provide a way of defeating the
          transformation of function objects to method objects
          described above. A static method object is a wrapper around
          any other object, usually a user-defined method object. When
          a static method object is retrieved from a class or a class
          instance, the object actually returned is the wrapped object,
          which is not subject to any further transformation. Static
          method objects are not themselves callable, although the
          objects they wrap usually are. Static method objects are
          created by the built-in staticmethod().
          constructor.

    Class method objects
          A class method object, like a static method object, is a
          wrapper around another object that alters the way in which
          that object is retrieved from classes and class instances.
          The behaviour of class method objects upon such retrieval is
          described above, under "User-defined methods". Class method
          objects are created by the built-in classmethod().
          constructor.

4.3.3 New-style and classic classes
-----------------------------------

Classes and instances come in two flavors: old-style (or classic) and
new-style.

  Up to Python 2.1, old-style classes were the only flavour available
to the user.  The concept of (old-style) class is unrelated to the
concept of type: if _x_ is an instance of an old-style class, then
`x.__class__' designates the class of _x_, but `type(x)' is always
`<type 'instance'>'.  This reflects the fact that all old-style
instances, independently of their class, are implemented with a single
built-in type, called `instance'.

  New-style classes were introduced in Python 2.2 to unify classes and
types.  A new-style class is neither more nor less than a user-defined
type.  If _x_ is an instance of a new-style class, then `type(x)' is
typically the same as `x.__class__' (although this is not guaranteed -
a new-style class instance is permitted to override the value returned
for `x.__class__').

  The major motivation for introducing new-style classes is to provide
a unified object model with a full meta-model.  It also has a number of
practical benefits, like the ability to subclass most built-in types,
or the introduction of "descriptors", which enable computed properties.

  For compatibility reasons, classes are still old-style by default.
New-style classes are created by specifying another new-style class
(i.e. a type) as a parent class, or the "top-level type" object
if no other parent is needed.  The behaviour of new-style classes
differs from that of old-style classes in a number of important details
in addition to what type().  returns.  Some of these changes
are fundamental to the new object model, like the way special methods
are invoked.  Others are "fixes" that could not be implemented before
for compatibility concerns, like the method resolution order in case of
multiple inheritance.

  While this manual aims to provide comprehensive coverage of Python's
class mechanics, it may still be lacking in some areas when it comes to
its coverage of new-style classes. Please see
<http://www.python.org/doc/newstyle/> for sources of additional
information.

  Old-style classes are removed in Python 3.0, leaving only the
semantics of new-style classes.

4.3.4 Special method names
--------------------------

A class can implement certain operations that are invoked by special
syntax (such as arithmetic operations or subscripting and slicing) by
defining methods with special names. This is Python's approach to
_operator overloading_, allowing classes to define their own behavior
with respect to language operators.  For instance, if a class defines a
method named __getitem__(), and `x' is an instance of this
class, then `x[i]' is roughly equivalent to `x.__getitem__(i)' for
old-style classes and `type(x).__getitem__(x, i)' for new-style
classes.  Except where mentioned, attempts to execute an operation
raise an exception when no appropriate method is defined (typically
AttributeError or TypeError).

  When implementing a class that emulates any built-in type, it is
important that the emulation only be implemented to the degree that it
makes sense for the object being modelled.  For example, some sequences
may work well with retrieval of individual elements, but extracting a
slice may not make sense.  (One example of this is the `NodeList'
interface in the W3C's Document Object Model.)

4.3.4.1 Basic customization
...........................

 -- Method: object.__new__ (cls[, ...])
     Called to create a new instance of class _cls_.  __new__()
     is a static method (special-cased so you need not declare it
     as such) that takes the class of which an instance was requested
     as its first argument.  The remaining arguments are those passed
     to the object constructor expression (the call to the class).  The
     return value of __new__() should be the new object
     instance (usually an instance of _cls_).

     Typical implementations create a new instance of the class by
     invoking the superclass's __new__() method using
     `super(currentclass, cls).__new__(cls[, ...])' with appropriate
     arguments and then modifying the newly-created instance as
     necessary before returning it.

     If __new__() returns an instance of _cls_, then the
     new instance's __init__() method will be invoked like
     `__init__(self[, ...])', where _self_ is the new instance and the
     remaining arguments are the same as were passed to
     __new__().

     If __new__() does not return an instance of _cls_,
     then the new instance's __init__() method will not be
     invoked.

     __new__() is intended mainly to allow subclasses of
     immutable types (like int, str, or tuple) to customize instance
     creation.  It is also commonly overridden in custom metaclasses in
     order to customize class creation.

 -- Method: object.__init__ (self[, ...])
     Called when the instance is created.  The arguments are those
     passed to the class constructor expression.  If a base class has
     an __init__() method, the derived class's
     __init__() method, if any, must explicitly call it to ensure
     proper initialization of the base class part of the instance; for
     example: `BaseClass.__init__(self, [args...])'.  As a special
     constraint on constructors, no value may be returned; doing so
     will cause a TypeError.  to be raised at runtime.

 -- Method: object.__del__ (self)
     Called when the instance is about to be destroyed.  This is also
     called a destructor.  If a base class has a __del__().
     method, the derived class's __del__() method, if any,
     must explicitly call it to ensure proper deletion of the base
     class part of the instance.  Note that it is possible (though not
     recommended!) for the __del__() method to postpone
     destruction of the instance by creating a new reference to it.  It
     may then be called at a later time when this new reference is
     deleted.  It is not guaranteed that __del__() methods
     are called for objects that still exist when the interpreter exits.

          Note: `del x' doesn't directly call `x.__del__()' -- the
          former decrements the reference count for `x' by one, and the
          latter is only called when `x''s reference count reaches
          zero.  Some common situations that may prevent the reference
          count of an object from going to zero include: circular
          references between objects (e.g., a doubly-linked list or a
          tree data structure with parent and child pointers); a
          reference to the object on the stack frame of a function that
          caught an exception (the traceback stored in
          `sys.exc_traceback' keeps the stack frame alive); or a
          reference to the object on the stack frame that raised an
          unhandled exception in interactive mode (the traceback stored
          in `sys.last_traceback' keeps the stack frame alive).  The
          first situation can only be remedied by explicitly breaking
          the cycles; the latter two situations can be resolved by
          storing `None' in `sys.exc_traceback' or
          `sys.last_traceback'.  Circular references which are garbage
          are detected when the option cycle detector is enabled (it's
          on by default), but can only be cleaned up if there are no
          Python-level __del__().  methods involved. Refer
          to the documentation for the gc module for more
          information about how __del__() methods are
          handled by the cycle detector, particularly the description
          of the `garbage' value.

          Warning: Due to the precarious circumstances under which
          __del__() methods are invoked, exceptions that
          occur during their execution are ignored, and a warning is
          printed to `sys.stderr' instead.  Also, when __del__()
          is invoked in response to a module being deleted (e.g.,
          when execution of the program is done), other globals
          referenced by the __del__() method may already
          have been deleted or in the process of being torn down (e.g.
          the import machinery shutting down).  For this reason, see
          __del__() methods should do the absolute minimum needed
          to maintain external invariants.  Starting with version 1.5,
          Python guarantees that globals whose name begins with a
          single underscore are deleted from their module before other
          globals are deleted; if no other references to such globals
          exist, this may help in assuring that imported modules are
          still available at the time when the __del__().
          method is called.

 -- Method: object.__repr__ (self)
     Called by the repr() built-in function and by string
     conversions (reverse quotes) to compute the "official" string
     representation of an object.  If at all possible, this should look
     like a valid Python expression that could be used to recreate an
     object with the same value (given an appropriate environment).  If
     this is not possible, a string of the form `<...some useful
     description...>' should be returned.  The return value must be a
     string object. If a class defines __repr__() but not
     __str__(), then __repr__() is also used
     when an "informal" string representation of instances of that
     class is required.

     This is typically used for debugging, so it is important that the
     representation is information-rich and unambiguous.

 -- Method: object.__str__ (self)
     Called by the str() built-in function and by the
     print.  statement to compute the "informal" string
     representation of an object.  This differs from __repr__()
     in that it does not have to be a valid Python expression: a
     more convenient or concise representation may be used instead.
     The return value must be a string object.

 -- Method: object.__lt__ (self, other)
 -- Method: object.__le__ (self, other)
 -- Method: object.__eq__ (self, other)
 -- Method: object.__ne__ (self, other)
 -- Method: object.__gt__ (self, other)
 -- Method: object.__ge__ (self, other)
     New in version 2.1.

     These are the so-called "rich comparison" methods, and are called
     for comparison operators in preference to __cmp__().
     below. The correspondence between operator symbols and method
     names is as follows: `x<y' calls `x.__lt__(y)', `x<=y' calls
     `x.__le__(y)', `x==y' calls `x.__eq__(y)', `x!=y' and `x<>y' call
     `x.__ne__(y)', `x>y' calls `x.__gt__(y)', and `x>=y' calls
     `x.__ge__(y)'.

     A rich comparison method may return the singleton `NotImplemented'
     if it does not implement the operation for a given pair of
     arguments. By convention, `False' and `True' are returned for a
     successful comparison. However, these methods can return any
     value, so if the comparison operator is used in a Boolean context
     (e.g., in the condition of an `if' statement), Python will call
     bool() on the value to determine if the result is true
     or false.

     There are no implied relationships among the comparison operators.
     The truth of `x==y' does not imply that `x!=y' is false.
     Accordingly, when defining __eq__(), one should also
     define __ne__() so that the operators will behave as
     expected.  See the paragraph on __hash__() for some
     important notes on creating hashable objects which
     support custom comparison operations and are usable as dictionary
     keys.

     There are no swapped-argument versions of these methods (to be
     used when the left argument does not support the operation but the
     right argument does); rather, see __lt__() and
     __gt__() are each other's reflection, see __le__().
     and __ge__() are each other's reflection, and
     __eq__() and __ne__() are their own reflection.

     Arguments to rich comparison methods are never coerced.

     To automatically generate ordering operations from a single root
     operation, see functools.total_ordering().

 -- Method: object.__cmp__ (self, other)
     Called by comparison operations if rich comparison (see above) is
     not defined.  Should return a negative integer if `self < other',
     zero if `self == other', a positive integer if `self > other'.  If
     no __cmp__(), __eq__() or __ne__()
     operation is defined, class instances are compared by object
     identity ("address").  See also the description of
     __hash__() for some important notes on creating
     hashable objects which support custom comparison operations
     and are usable as dictionary keys. (Note: the restriction that
     exceptions are not propagated by __cmp__() has been
     removed since Python 1.5.)

 -- Method: object.__rcmp__ (self, other)
     Changed in version 2.1: No longer supported.

 -- Method: object.__hash__ (self)
     Called by built-in function hash() and for operations
     on members of hashed collections including set,
     frozenset, and dict.  See __hash__().
     should return an integer.  The only required property is that
     objects which compare equal have the same hash value; it is
     advised to somehow mix together (e.g. using exclusive or) the hash
     values for the components of the object that also play a part in
     comparison of objects.

     If a class does not define a __cmp__() or
     __eq__() method it should not define a __hash__()
     operation either; if it defines __cmp__() or
     __eq__() but not __hash__(), its instances
     will not be usable in hashed collections.  If a class defines
     mutable objects and implements a __cmp__() or
     __eq__() method, it should not implement __hash__(),
     since hashable collection implementations require that a
     object's hash value is immutable (if the object's hash value
     changes, it will be in the wrong hash bucket).

     User-defined classes have __cmp__() and
     __hash__() methods by default; with them, all objects
     compare unequal (except with themselves) and `x.__hash__()'
     returns `id(x)'.

     Classes which inherit a __hash__() method from a
     parent class but change the meaning of __cmp__() or
     __eq__() such that the hash value returned is no
     longer appropriate (e.g. by switching to a value-based concept of
     equality instead of the default identity based equality) can
     explicitly flag themselves as being unhashable by setting
     `__hash__ = None' in the class definition. Doing so means that not
     only will instances of the class raise an appropriate
     TypeError when a program attempts to retrieve their hash
     value, but they will also be correctly identified as unhashable
     when checking `isinstance(obj, collections.Hashable)' (unlike
     classes which define their own __hash__() to
     explicitly raise TypeError).

     Changed in version 2.5: __hash__() may now also return
     a long integer object; the 32-bit integer is then derived from the
     hash of that object.

     Changed in version 2.6: __hash__ may now be set to
     None to explicitly flag instances of a class as
     unhashable.

 -- Method: object.__nonzero__ (self)
     Called to implement truth value testing and the built-in operation
     `bool()'; should return `False' or `True', or their integer
     equivalents `0' or `1'.  When this method is not defined, see
     __len__() is called, if it is defined, and the object is
     considered true if its result is nonzero.  If a class defines
     neither __len__() nor __nonzero__(), all its
     instances are considered true.

 -- Method: object.__unicode__ (self)
     Called to implement unicode() built-in; should return
     a Unicode object.  When this method is not defined, string
     conversion is attempted, and the result of string conversion is
     converted to Unicode using the system default encoding.

4.3.4.2 Customizing attribute access
....................................

The following methods can be defined to customize the meaning of
attribute access (use of, assignment to, or deletion of `x.name') for
class instances.

 -- Method: object.__getattr__ (self, name)
     Called when an attribute lookup has not found the attribute in the
     usual places (i.e. it is not an instance attribute nor is it found
     in the class tree for `self').  `name' is the attribute name. This
     method should return the (computed) attribute value or raise an
     AttributeError exception.

     Note that if the attribute is found through the normal mechanism,
     see __getattr__() is not called.  (This is an intentional
     asymmetry between __getattr__() and
     __setattr__()) This is done both for efficiency reasons and
     because otherwise __getattr__() would have no way to
     access other attributes of the instance.  Note that at least for
     instance variables, you can fake total control by not inserting
     any values in the instance attribute dictionary (but instead
     inserting them in another object).  See the
     __getattribute__() method below for a way to actually get
     total control in new-style classes.

 -- Method: object.__setattr__ (self, name, value)
     Called when an attribute assignment is attempted.  This is called
     instead of the normal mechanism (i.e. store the value in the
     instance dictionary).  _name_ is the attribute name, _value_ is
     the value to be assigned to it.

     If __setattr__() wants to assign to an instance
     attribute, it should not simply execute `self.name = value' --
     this would cause a recursive call to itself.  Instead, it should
     insert the value in the dictionary of instance attributes, e.g.,
     `self.__dict__[name] = value'.  For new-style classes, rather than
     accessing the instance dictionary, it should call the base class
     method with the same name, for example, `object.__setattr__(self,
     name, value)'.

 -- Method: object.__delattr__ (self, name)
     Like __setattr__() but for attribute deletion instead
     of assignment.  This should only be implemented if `del obj.name'
     is meaningful for the object.

4.3.4.3 More attribute access for new-style classes
...................................................

The following methods only apply to new-style classes.

 -- Method: object.__getattribute__ (self, name)
     Called unconditionally to implement attribute accesses for
     instances of the class. If the class also defines
     __getattr__(), the latter will not be called unless
     __getattribute__() either calls it explicitly or raises an
     AttributeError. This method should return the
     (computed) attribute value or raise an AttributeError.
     exception. In order to avoid infinite recursion in this method,
     its implementation should always call the base class method with
     the same name to access any attributes it needs, for example,
     `object.__getattribute__(self, name)'.

          Note: This method may still be bypassed when looking up
          special methods as the result of implicit invocation via
          language syntax or built-in functions.  See Special method lookup for new-style classes.

4.3.4.4 Implementing Descriptors
................................

The following methods only apply when an instance of the class
containing the method (a so-called _descriptor_ class) appears in the
class dictionary of another new-style class, known as the _owner_
class. In the examples below, "the attribute" refers to the attribute
whose name is the key of the property in the owner class' `__dict__'.
Descriptors can only be implemented as new-style classes themselves.

 -- Method: object.__get__ (self, instance, owner)
     Called to get the attribute of the owner class (class attribute
     access) or of an instance of that class (instance attribute
     access). _owner_ is always the owner class, while _instance_ is
     the instance that the attribute was accessed through, or `None'
     when the attribute is accessed through the _owner_.  This method
     should return the (computed) attribute value or raise an
     AttributeError.  exception.

 -- Method: object.__set__ (self, instance, value)
     Called to set the attribute on an instance _instance_ of the owner
     class to a new value, _value_.

 -- Method: object.__delete__ (self, instance)
     Called to delete the attribute on an instance _instance_ of the
     owner class.

4.3.4.5 Invoking Descriptors
............................

In general, a descriptor is an object attribute with "binding
behavior", one whose attribute access has been overridden by methods in
the descriptor protocol:  __get__(), __set__(),
and __delete__(). If any of those methods are defined for an
object, it is said to be a descriptor.

  The default behavior for attribute access is to get, set, or delete
the attribute from an object's dictionary. For instance, `a.x' has a
lookup chain starting with `a.__dict__['x']', then
`type(a).__dict__['x']', and continuing through the base classes of
`type(a)' excluding metaclasses.

  However, if the looked-up value is an object defining one of the
descriptor methods, then Python may override the default behavior and
invoke the descriptor method instead.  Where this occurs in the
precedence chain depends on which descriptor methods were defined and
how they were called.  Note that descriptors are only invoked for new
style objects or classes (ones that subclass object() or
type()).

  The starting point for descriptor invocation is a binding, `a.x'. How
the arguments are assembled depends on `a':

Direct Call
     The simplest and least common call is when user code directly
     invokes a descriptor method:    `x.__get__(a)'.

Instance Binding
     If binding to a new-style object instance, `a.x' is transformed
     into the call: `type(a).__dict__['x'].__get__(a, type(a))'.

Class Binding
     If binding to a new-style class, `A.x' is transformed into the
     call: `A.__dict__['x'].__get__(None, A)'.

Super Binding
     If `a' is an instance of super, then the binding
     `super(B, obj).m()' searches `obj.__class__.__mro__' for the base
     class `A' immediately preceding `B' and then invokes the
     descriptor with the call: `A.__dict__['m'].__get__(obj, A)'.

  For instance bindings, the precedence of descriptor invocation
depends on the which descriptor methods are defined.  A descriptor can
define any combination of __get__(), __set__().
and __delete__().  If it does not define __get__(),
then accessing the attribute will return the descriptor object
itself unless there is a value in the object's instance dictionary.  If
the descriptor defines __set__() and/or __delete__(),
it is a data descriptor; if it defines neither, it is a non-data
descriptor.  Normally, data descriptors define both __get__()
and __set__(), while non-data descriptors have just the
__get__() method.  Data descriptors with __set__()
and __get__() defined always override a redefinition
in an instance dictionary.  In contrast, non-data descriptors can be
overridden by instances.

  Python methods (including staticmethod() and
classmethod()) are implemented as non-data descriptors.
Accordingly, instances can redefine and override methods.  This allows
individual instances to acquire behaviors that differ from other
instances of the same class.

  The property() function is implemented as a data
descriptor. Accordingly, instances cannot override the behavior of a
property.

4.3.4.6 __slots__
.................

By default, instances of both old and new-style classes have a
dictionary for attribute storage.  This wastes space for objects having
very few instance variables.  The space consumption can become acute
when creating large numbers of instances.

  The default can be overridden by defining ___slots___ in a new-style
class definition.  The ___slots___ declaration takes a sequence of
instance variables and reserves just enough space in each instance to
hold a value for each variable.  Space is saved because ___dict___ is
not created for each instance.

 -- Data: __slots__
     This class variable can be assigned a string, iterable, or
     sequence of strings with variable names used by instances.  If
     defined in a new-style class, ___slots___ reserves space for the
     declared variables and prevents the automatic creation of
     ___dict___ and ___weakref___ for each instance.

     New in version 2.2.

  Notes on using ___slots___

   * When inheriting from a class without ___slots___, the ___dict___
     attribute of that class will always be accessible, so a
     ___slots___ definition in the subclass is meaningless.

   * Without a ___dict___ variable, instances cannot be assigned new
     variables not listed in the ___slots___ definition.  Attempts to
     assign to an unlisted variable name raises AttributeError.
     If dynamic assignment of new variables is desired, then add
     `'__dict__'' to the sequence of strings in the ___slots___
     declaration.

     Changed in version 2.3: Previously, adding `'__dict__'' to the
     ___slots___ declaration would not enable the assignment of new
     attributes not specifically listed in the sequence of instance
     variable names.

   * Without a ___weakref___ variable for each instance, classes
     defining ___slots___ do not support weak references to its
     instances. If weak reference support is needed, then add
     `'__weakref__'' to the sequence of strings in the ___slots___
     declaration.

     Changed in version 2.3: Previously, adding `'__weakref__'' to the
     ___slots___ declaration would not enable support for weak
     references.

   * ___slots___ are implemented at the class level by creating
     descriptors (see Implementing Descriptors) for each
     variable name.  As a result, class attributes cannot be used to
     set default values for instance variables defined by ___slots___;
     otherwise, the class attribute would overwrite the descriptor
     assignment.

   * The action of a ___slots___ declaration is limited to the class
     where it is defined.  As a result, subclasses will have a
     ___dict___ unless they also define ___slots___ (which must only
     contain names of any _additional_ slots).

   * If a class defines a slot also defined in a base class, the
     instance variable defined by the base class slot is inaccessible
     (except by retrieving its descriptor directly from the base
     class). This renders the meaning of the program undefined.  In the
     future, a check may be added to prevent this.

   * Nonempty ___slots___ does not work for classes derived from
     "variable-length" built-in types such as long,
     str and tuple.

   * Any non-string iterable may be assigned to ___slots___. Mappings
     may also be used; however, in the future, special meaning may be
     assigned to the values corresponding to each key.

   * ___class___ assignment works only if both classes have the same
     ___slots___.

     Changed in version 2.6: Previously, ___class___ assignment raised
     an error if either new or old class had ___slots___.

4.3.4.7 Customizing class creation
..................................

By default, new-style classes are constructed using type().
A class definition is read into a separate namespace and the value of
class name is bound to the result of `type(name, bases, dict)'.

  When the class definition is read, if ___metaclass___ is defined then
the callable assigned to it will be called instead of type().
This allows classes or functions to be written which monitor or
alter the class creation process:

   * Modifying the class dictionary prior to the class being created.

   * Returning an instance of another class - essentially performing
     the role of a factory function.

  These steps will have to be performed in the metaclass's
__new__() method - `type.__new__()' can then be called from this
method to create a class with different properties.  This example adds
a new element to the class dictionary before creating the class:

    class metacls(type):
        def __new__(mcs, name, bases, dict):
            dict['foo'] = 'metacls was here'
            return type.__new__(mcs, name, bases, dict)

You can of course also override other class methods (or add new
methods); for example defining a custom __call__() method
in the metaclass allows custom behavior when the class is called, e.g.
not always creating a new instance.

 -- Data: __metaclass__
     This variable can be any callable accepting arguments for `name',
     `bases', and `dict'.  Upon class creation, the callable is used
     instead of the built-in type().

     New in version 2.2.

  The appropriate metaclass is determined by the following precedence
rules:

   * If `dict['__metaclass__']' exists, it is used.

   * Otherwise, if there is at least one base class, its metaclass is
     used (this looks for a ___class___ attribute first and if not
     found, uses its type).

   * Otherwise, if a global variable named __metaclass__ exists, it is
     used.

   * Otherwise, the old-style, classic metaclass (types.ClassType) is
     used.

  The potential uses for metaclasses are boundless. Some ideas that
have been explored including logging, interface checking, automatic
delegation, automatic property creation, proxies, frameworks, and
automatic resource locking/synchronization.

4.3.4.8 Customizing instance and subclass checks
................................................

New in version 2.6.

  The following methods are used to override the default behavior of the
isinstance() and issubclass() built-in
functions.

  In particular, the metaclass abc.ABCMeta implements these
methods in order to allow the addition of Abstract Base Classes (ABCs)
as "virtual base classes" to any class or type (including built-in
types), including other ABCs.

 -- Method: class.__instancecheck__ (self, instance)
     Return true if _instance_ should be considered a (direct or
     indirect) instance of _class_. If defined, called to implement
     `isinstance(instance, class)'.

 -- Method: class.__subclasscheck__ (self, subclass)
     Return true if _subclass_ should be considered a (direct or
     indirect) subclass of _class_.  If defined, called to implement
     `issubclass(subclass, class)'.

  Note that these methods are looked up on the type (metaclass) of a
class.  They cannot be defined as class methods in the actual class.
This is consistent with the lookup of special methods that are called
on instances, only in this case the instance is itself a class.

PEP 3119 (http://www.python.org/dev/peps/pep-3119) - Introducing Abstract Base Classes
     Includes the specification for customizing isinstance()
     and issubclass() behavior through
     `__instancecheck__()' and `__subclasscheck__()', with motivation
     for this functionality in the context of adding Abstract Base
     Classes (see the abc module) to the language.

4.3.4.9 Emulating callable objects
..................................

 -- Method: object.__call__ (self[, args...])
     Called when the instance is "called" as a function; if this method
     is defined, `x(arg1, arg2, ...)' is a shorthand for
     `x.__call__(arg1, arg2, ...)'.

4.3.4.10 Emulating container types
..................................

The following methods can be defined to implement container objects.
Containers usually are sequences (such as lists or tuples) or mappings
(like dictionaries), but can represent other containers as well.  The
first set of methods is used either to emulate a sequence or to emulate
a mapping; the difference is that for a sequence, the allowable keys
should be the integers _k_ for which `0 <= k < N' where _N_ is the
length of the sequence, or slice objects, which define a range of
items. (For backwards compatibility, the method __getslice__().
(see below) can also be defined to handle simple, but not
extended slices.) It is also recommended that mappings provide the
methods `keys()', `values()', `items()', `has_key()', `get()',
`clear()', `setdefault()', `iterkeys()', `itervalues()', `iteritems()',
`pop()', `popitem()', see copy(), and `update()' behaving similar
to those for Python's standard dictionary objects.  The UserDict
module provides a `DictMixin' class to help create those methods
from a base set of __getitem__(), __setitem__(),
__delitem__(), and `keys()'. Mutable sequences should
provide methods `append()', `count()', `index()', `extend()',
`insert()', `pop()', `remove()', `reverse()' and `sort()', like Python
standard list objects.  Finally, sequence types should implement
addition (meaning concatenation) and multiplication (meaning
repetition) by defining the methods __add__(),
__radd__(), __iadd__(), __mul__(),
__rmul__() and __imul__() described below; they
should not define __coerce__() or other numerical
operators.  It is recommended that both mappings and sequences
implement the __contains__() method to allow efficient use
of the `in' operator; for mappings, `in' should be equivalent of
`has_key()'; for sequences, it should search through the values.  It is
further recommended that both mappings and sequences implement the
__iter__() method to allow efficient iteration through the
container; for mappings, see __iter__() should be the same as
`iterkeys()'; for sequences, it should iterate through the values.

 -- Method: object.__len__ (self)
     Called to implement the built-in function len().
     Should return the length of the object, an integer `>=' 0.  Also,
     an object that doesn't define a __nonzero__() method
     and whose __len__() method returns zero is considered
     to be false in a Boolean context.

 -- Method: object.__getitem__ (self, key)
     Called to implement evaluation of `self[key]'. For sequence types,
     the accepted keys should be integers and slice objects.  Note that
     the special interpretation of negative indexes (if the class
     wishes to emulate a sequence type) is up to the
     __getitem__() method. If _key_ is of an inappropriate type,
     see TypeError may be raised; if of a value outside the set
     of indexes for the sequence (after any special interpretation of
     negative values), see IndexError should be raised. For
     mapping types, if _key_ is missing (not in the container), see
     KeyError should be raised.

          Note: for loops expect that an IndexError
          will be raised for illegal indexes to allow proper
          detection of the end of the sequence.

 -- Method: object.__setitem__ (self, key, value)
     Called to implement assignment to `self[key]'.  Same note as for
     __getitem__().  This should only be implemented for
     mappings if the objects support changes to the values for keys, or
     if new keys can be added, or for sequences if elements can be
     replaced.  The same exceptions should be raised for improper _key_
     values as for the __getitem__() method.

 -- Method: object.__delitem__ (self, key)
     Called to implement deletion of `self[key]'.  Same note as for
     __getitem__().  This should only be implemented for
     mappings if the objects support removal of keys, or for sequences
     if elements can be removed from the sequence.  The same exceptions
     should be raised for improper _key_ values as for the
     __getitem__() method.

 -- Method: object.__iter__ (self)
     This method is called when an iterator is required for a
     container. This method should return a new iterator object that
     can iterate over all the objects in the container.  For mappings,
     it should iterate over the keys of the container, and should also
     be made available as the method `iterkeys()'.

     Iterator objects also need to implement this method; they are
     required to return themselves.  For more information on iterator
     objects, see Iterator Types.

 -- Method: object.__reversed__ (self)
     Called (if present) by the reversed() built-in to
     implement reverse iteration.  It should return a new iterator
     object that iterates over all the objects in the container in
     reverse order.

     If the __reversed__() method is not provided, the
     reversed().  built-in will fall back to using the
     sequence protocol (see __len__() and __getitem__()).
     Objects that support the sequence protocol should only
     provide __reversed__() if they can provide an
     implementation that is more efficient than the one provided by
     reversed().

     New in version 2.6.

  The membership test operators (see in and `not in') are
normally implemented as an iteration through a sequence.  However,
container objects can supply the following special method with a more
efficient implementation, which also does not require the object be a
sequence.

 -- Method: object.__contains__ (self, item)
     Called to implement membership test operators.  Should return true
     if _item_ is in _self_, false otherwise.  For mapping objects,
     this should consider the keys of the mapping rather than the
     values or the key-item pairs.

     For objects that don't define __contains__(), the
     membership test first tries iteration via __iter__(),
     then the old sequence iteration protocol via __getitem__(),
     see this section in the language reference.

4.3.4.11 Additional methods for emulation of sequence types
...........................................................

The following optional methods can be defined to further emulate
sequence objects.  Immutable sequences methods should at most only
define __getslice__().; mutable sequences might define all
three methods.

 -- Method: object.__getslice__ (self, i, j)
     Deprecated since version 2.0: Support slice objects as parameters
     to the __getitem__() method.  (However, built-in types
     in CPython currently still implement __getslice__().
     Therefore, you have to override it in derived classes when
     implementing slicing.)

     Called to implement evaluation of `self[i:j]'. The returned object
     should be of the same type as _self_.  Note that missing _i_ or
     _j_ in the slice expression are replaced by zero or `sys.maxint',
     respectively.  If negative indexes are used in the slice, the
     length of the sequence is added to that index. If the instance
     does not implement the __len__() method, an
     AttributeError is raised. No guarantee is made that indexes
     adjusted this way are not still negative.  Indexes which are
     greater than the length of the sequence are not modified. If no
     __getslice__() is found, a slice object is created
     instead, and passed to __getitem__() instead.

 -- Method: object.__setslice__ (self, i, j, sequence)
     Called to implement assignment to `self[i:j]'. Same notes for _i_
     and _j_ as for __getslice__().

     This method is deprecated. If no __setslice__() is
     found, or for extended slicing of the form `self[i:j:k]', a slice
     object is created, and passed to __setitem__(), instead
     of __setslice__() being called.

 -- Method: object.__delslice__ (self, i, j)
     Called to implement deletion of `self[i:j]'. Same notes for _i_
     and _j_ as for __getslice__(). This method is
     deprecated. If no __delslice__() is found, or for
     extended slicing of the form `self[i:j:k]', a slice object is
     created, and passed to __delitem__(), instead of
     __delslice__().  being called.

  Notice that these methods are only invoked when a single slice with a
single colon is used, and the slice method is available.  For slice
operations involving extended slice notation, or in absence of the
slice methods, see __getitem__(), __setitem__() or
__delitem__() is called with a slice object as argument.

  The following example demonstrate how to make your program or module
compatible with earlier versions of Python (assuming that methods
__getitem__(), __setitem__() and __delitem__()
support slice objects as arguments):

    class MyClass:
        ...
        def __getitem__(self, index):
            ...
        def __setitem__(self, index, value):
            ...
        def __delitem__(self, index):
            ...

        if sys.version_info < (2, 0):
            # They won't be defined if version is at least 2.0 final

            def __getslice__(self, i, j):
                return self[max(0, i):max(0, j):]
            def __setslice__(self, i, j, seq):
                self[max(0, i):max(0, j):] = seq
            def __delslice__(self, i, j):
                del self[max(0, i):max(0, j):]
        ...

Note the calls to max().; these are necessary because of the
handling of negative indices before the `__*slice__()' methods are
called.  When negative indexes are used, the `__*item__()' methods
receive them as provided, but the `__*slice__()' methods get a "cooked"
form of the index values.  For each negative index value, the length of
the sequence is added to the index before calling the method (which may
still result in a negative index); this is the customary handling of
negative indexes by the built-in sequence types, and the `__*item__()'
methods are expected to do this as well.  However, since they should
already be doing that, negative indexes cannot be passed in; they must
be constrained to the bounds of the sequence before being passed to the
`__*item__()' methods. Calling `max(0, i)' conveniently returns the
proper value.

4.3.4.12 Emulating numeric types
................................

The following methods can be defined to emulate numeric objects. Methods
corresponding to operations that are not supported by the particular
kind of number implemented (e.g., bitwise operations for non-integral
numbers) should be left undefined.

 -- Method: object.__add__ (self, other)
 -- Method: object.__sub__ (self, other)
 -- Method: object.__mul__ (self, other)
 -- Method: object.__floordiv__ (self, other)
 -- Method: object.__mod__ (self, other)
 -- Method: object.__divmod__ (self, other)
 -- Method: object.__pow__ (self, other[, modulo])
 -- Method: object.__lshift__ (self, other)
 -- Method: object.__rshift__ (self, other)
 -- Method: object.__and__ (self, other)
 -- Method: object.__xor__ (self, other)
 -- Method: object.__or__ (self, other)
     These methods are called to implement the binary arithmetic
     operations (`+', `-', `*', `//', `%', see divmod(),
     pow(), `**', `<<', `>>', `&', `^', `|').  For instance, to
     evaluate the expression `x + y', where _x_ is an instance of a
     class that has an __add__().  method, `x.__add__(y)' is
     called.  The __divmod__() method should be the
     equivalent to using __floordiv__() and
     __mod__().; it should not be related to __truediv__().
     (described below).  Note that __pow__().  should
     be defined to accept an optional third argument if the ternary
     version of the built-in pow() function is to be
     supported.

     If one of those methods does not support the operation with the
     supplied arguments, it should return `NotImplemented'.

 -- Method: object.__div__ (self, other)
 -- Method: object.__truediv__ (self, other)
     The division operator (`/') is implemented by these methods.  The
     __truediv__() method is used when
     `__future__.division' is in effect, otherwise __div__()
     is used.  If only one of these two methods is defined, the
     object will not support division in the alternate context; see
     TypeError.  will be raised instead.

 -- Method: object.__radd__ (self, other)
 -- Method: object.__rsub__ (self, other)
 -- Method: object.__rmul__ (self, other)
 -- Method: object.__rdiv__ (self, other)
 -- Method: object.__rtruediv__ (self, other)
 -- Method: object.__rfloordiv__ (self, other)
 -- Method: object.__rmod__ (self, other)
 -- Method: object.__rdivmod__ (self, other)
 -- Method: object.__rpow__ (self, other)
 -- Method: object.__rlshift__ (self, other)
 -- Method: object.__rrshift__ (self, other)
 -- Method: object.__rand__ (self, other)
 -- Method: object.__rxor__ (self, other)
 -- Method: object.__ror__ (self, other)
     These methods are called to implement the binary arithmetic
     operations (`+', `-', `*', `/', `%', see divmod(),
     pow(), `**', `<<', `>>', `&', `^', `|') with reflected
     (swapped) operands.  These functions are only called if the left
     operand does not support the corresponding operation and the
     operands are of different types. (2) For instance, to evaluate the
     expression `x - y', where _y_ is an instance of a class that has an
     __rsub__() method, `y.__rsub__(x)' is called if
     `x.__sub__(y)' returns _NotImplemented_.

     Note that ternary pow() will not try calling
     __rpow__(). (the coercion rules would become too complicated).

          Note: If the right operand's type is a subclass of the left
          operand's type and that subclass provides the reflected
          method for the operation, this method will be called before
          the left operand's non-reflected method.  This behavior allows
          subclasses to override their ancestors' operations.

 -- Method: object.__iadd__ (self, other)
 -- Method: object.__isub__ (self, other)
 -- Method: object.__imul__ (self, other)
 -- Method: object.__idiv__ (self, other)
 -- Method: object.__itruediv__ (self, other)
 -- Method: object.__ifloordiv__ (self, other)
 -- Method: object.__imod__ (self, other)
 -- Method: object.__ipow__ (self, other[, modulo])
 -- Method: object.__ilshift__ (self, other)
 -- Method: object.__irshift__ (self, other)
 -- Method: object.__iand__ (self, other)
 -- Method: object.__ixor__ (self, other)
 -- Method: object.__ior__ (self, other)
     These methods are called to implement the augmented arithmetic
     assignments (`+=', `-=', `*=', `/=', `//=', `%=', `**=', `<<=',
     `>>=', `&=', `^=', `|=').  These methods should attempt to do the
     operation in-place (modifying _self_) and return the result (which
     could be, but does not have to be, _self_).  If a specific method
     is not defined, the augmented assignment falls back to the normal
     methods.  For instance, to execute the statement `x += y', where
     _x_ is an instance of a class that has an __iadd__().
     method, `x.__iadd__(y)' is called.  If _x_ is an instance of a
     class that does not define a __iadd__() method,
     `x.__add__(y)' and `y.__radd__(x)' are considered, as with the
     evaluation of `x + y'.

 -- Method: object.__neg__ (self)
 -- Method: object.__pos__ (self)
 -- Method: object.__abs__ (self)
 -- Method: object.__invert__ (self)
     Called to implement the unary arithmetic operations (`-', `+',
     see abs().  and `~').

 -- Method: object.__complex__ (self)
 -- Method: object.__int__ (self)
 -- Method: object.__long__ (self)
 -- Method: object.__float__ (self)
     Called to implement the built-in functions complex(),
     int(), long(), and float().
     Should return a value of the appropriate type.

 -- Method: object.__oct__ (self)
 -- Method: object.__hex__ (self)
     Called to implement the built-in functions oct() and
     hex().  Should return a string value.

 -- Method: object.__index__ (self)
     Called to implement operator.index().  Also called
     whenever Python needs an integer object (such as in slicing).
     Must return an integer (int or long).

     New in version 2.5.

 -- Method: object.__coerce__ (self, other)
     Called to implement "mixed-mode" numeric arithmetic.  Should
     either return a 2-tuple containing _self_ and _other_ converted to
     a common numeric type, or `None' if conversion is impossible.
     When the common type would be the type of `other', it is
     sufficient to return `None', since the interpreter will also ask
     the other object to attempt a coercion (but sometimes, if the
     implementation of the other type cannot be changed, it is useful
     to do the conversion to the other type here).  A return value of
     `NotImplemented' is equivalent to returning `None'.

4.3.4.13 Coercion rules
.......................

This section used to document the rules for coercion.  As the language
has evolved, the coercion rules have become hard to document precisely;
documenting what one version of one particular implementation does is
undesirable.  Instead, here are some informal guidelines regarding
coercion.  In Python 3.0, coercion will not be supported.

   * If the left operand of a % operator is a string or Unicode object,
     no coercion takes place and the string formatting operation is
     invoked instead.

   * It is no longer recommended to define a coercion operation.
     Mixed-mode operations on types that don't define coercion pass the
     original arguments to the operation.

   * New-style classes (those derived from object) never
     invoke the __coerce__() method in response to a binary
     operator; the only time __coerce__() is invoked is
     when the built-in function coerce() is called.

   * For most intents and purposes, an operator that returns
     `NotImplemented' is treated the same as one that is not
     implemented at all.

   * Below, `__op__()' and `__rop__()' are used to signify the generic
     method names corresponding to an operator; `__iop__()' is used for
     the corresponding in-place operator.  For example, for the
     operator '`+'', see __add__() and __radd__().
     are used for the left and right variant of the binary operator,
     and __iadd__() for the in-place variant.

   * For objects _x_ and _y_, first `x.__op__(y)' is tried.  If this is
     not implemented or returns `NotImplemented', `y.__rop__(x)' is
     tried.  If this is also not implemented or returns
     `NotImplemented', a TypeError.  exception is raised.
     But see the following exception:

   * Exception to the previous item: if the left operand is an instance
     of a built-in type or a new-style class, and the right operand is
     an instance of a proper subclass of that type or class and
     overrides the base's `__rop__()' method, the right operand's
     `__rop__()' method is tried _before_ the left operand's `__op__()'
     method.

     This is done so that a subclass can completely override binary
     operators.  Otherwise, the left operand's `__op__()' method would
     always accept the right operand: when an instance of a given class
     is expected, an instance of a subclass of that class is always
     acceptable.

   * When either operand type defines a coercion, this coercion is
     called before that type's `__op__()' or `__rop__()' method is
     called, but no sooner.  If the coercion returns an object of a
     different type for the operand whose coercion is invoked, part of
     the process is redone using the new object.

   * When an in-place operator (like '`+='') is used, if the left
     operand implements `__iop__()', it is invoked without any
     coercion.  When the operation falls back to `__op__()' and/or
     `__rop__()', the normal coercion rules apply.

   * In `x + y', if _x_ is a sequence that implements sequence
     concatenation, sequence concatenation is invoked.

   * In `x * y', if one operator is a sequence that implements sequence
     repetition, and the other is an integer (see int or
     long), sequence repetition is invoked.

   * Rich comparisons (implemented by methods __eq__() and
     so on) never use coercion.  Three-way comparison (implemented by
     __cmp__()) does use coercion under the same conditions
     as other binary operations use it.

   * In the current implementation, the built-in numeric types
     int, long, float, and complex
     do not use coercion.  All these types implement a
     __coerce__() method, for use by the built-in coerce()
     function.

     Changed in version 2.7.

4.3.4.14 With Statement Context Managers
........................................

New in version 2.5.

  A _context manager_ is an object that defines the runtime context to
be established when executing a with statement. The context
manager handles the entry into, and the exit from, the desired runtime
context for the execution of the block of code.  Context managers are
normally invoked using the with statement (described in
section The with statement), but can also be used by
directly invoking their methods.

  Typical uses of context managers include saving and restoring various
kinds of global state, locking and unlocking resources, closing opened
files, etc.

  For more information on context managers, see Context Manager Types.

 -- Method: object.__enter__ (self)
     Enter the runtime context related to this object. The with
     statement will bind this method's return value to the
     target(s) specified in the as clause of the statement,
     if any.

 -- Method: object.__exit__ (self, exc_type, exc_value, traceback)
     Exit the runtime context related to this object. The parameters
     describe the exception that caused the context to be exited. If
     the context was exited without an exception, all three arguments
     will be None.

     If an exception is supplied, and the method wishes to suppress the
     exception (i.e., prevent it from being propagated), it should
     return a true value.  Otherwise, the exception will be processed
     normally upon exit from this method.

     Note that __exit__() methods should not reraise the
     passed-in exception; this is the caller's responsibility.

PEP 0343 (http://www.python.org/dev/peps/pep-0343) - The "with" statement
     The specification, background, and examples for the Python
     with.  statement.

4.3.4.15 Special method lookup for old-style classes
....................................................

For old-style classes, special methods are always looked up in exactly
the same way as any other method or attribute. This is the case
regardless of whether the method is being looked up explicitly as in
`x.__getitem__(i)' or implicitly as in `x[i]'.

  This behaviour means that special methods may exhibit different
behaviour for different instances of a single old-style class if the
appropriate special attributes are set differently:

    >>> class C:
    ...     pass
    ...
    >>> c1 = C()
    >>> c2 = C()
    >>> c1.__len__ = lambda: 5
    >>> c2.__len__ = lambda: 9
    >>> len(c1)
    5
    >>> len(c2)
    9

4.3.4.16 Special method lookup for new-style classes
....................................................

For new-style classes, implicit invocations of special methods are only
guaranteed to work correctly if defined on an object's type, not in the
object's instance dictionary.  That behaviour is the reason why the
following code raises an exception (unlike the equivalent example with
old-style classes):

    >>> class C(object):
    ...     pass
    ...
    >>> c = C()
    >>> c.__len__ = lambda: 5
    >>> len(c)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: object of type 'C' has no len()

The rationale behind this behaviour lies with a number of special
methods such as __hash__() and __repr__() that
are implemented by all objects, including type objects. If the implicit
lookup of these methods used the conventional lookup process, they
would fail when invoked on the type object itself:

    >>> 1 .__hash__() == hash(1)
    True
    >>> int.__hash__() == hash(int)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: descriptor '__hash__' of 'int' object needs an argument

Incorrectly attempting to invoke an unbound method of a class in this
way is sometimes referred to as 'metaclass confusion', and is avoided
by bypassing the instance when looking up special methods:

    >>> type(1).__hash__(1) == hash(1)
    True
    >>> type(int).__hash__(int) == hash(int)
    True

In addition to bypassing any instance attributes in the interest of
correctness, implicit special method lookup generally also bypasses the
__getattribute__() method even of the object's metaclass:

    >>> class Meta(type):
    ...    def __getattribute__(*args):
    ...       print "Metaclass getattribute invoked"
    ...       return type.__getattribute__(*args)
    ...
    >>> class C(object):
    ...     __metaclass__ = Meta
    ...     def __len__(self):
    ...         return 10
    ...     def __getattribute__(*args):
    ...         print "Class getattribute invoked"
    ...         return object.__getattribute__(*args)
    ...
    >>> c = C()
    >>> c.__len__()                 # Explicit lookup via instance
    Class getattribute invoked
    10
    >>> type(c).__len__(c)          # Explicit lookup via type
    Metaclass getattribute invoked
    10
    >>> len(c)                      # Implicit lookup
    10

Bypassing the __getattribute__() machinery in this fashion
provides significant scope for speed optimisations within the
interpreter, at the cost of some flexibility in the handling of special
methods (the special method _must_ be set on the class object itself in
order to be consistently invoked by the interpreter).

Footnotes
.........

     (1) It _is_ possible in some cases to change an object's type,
     under certain controlled conditions. It generally isn't a good
     idea though, since it can lead to some very strange behaviour if
     it is handled incorrectly.

     (2) For operands of the same type, it is assumed that if the
     non-reflected method (such as __add__()) fails the
     operation is not supported, which is why the reflected method is
     not called.

4.4 Execution model
===================

4.4.1 Naming and binding
------------------------

_Names_ refer to objects.  Names are introduced by name binding
operations.  Each occurrence of a name in the program text refers to
the _binding_ of that name established in the innermost function block
containing the use.

  A _block_ is a piece of Python program text that is executed as a
unit.  The following are blocks: a module, a function body, and a class
definition.  Each command typed interactively is a block.  A script
file (a file given as standard input to the interpreter or specified on
the interpreter command line the first argument) is a code block.  A
script command (a command specified on the interpreter command line
with the '*-c*' option) is a code block.  The file read by the built-in
function execfile() is a code block.  The string argument
passed to the built-in function eval() and to the
exec.  statement is a code block. The expression read and
evaluated by the built-in function input() is a code block.

  A code block is executed in an _execution frame_.  A frame contains
some administrative information (used for debugging) and determines
where and how execution continues after the code block's execution has
completed.

  A _scope_ defines the visibility of a name within a block.  If a local
variable is defined in a block, its scope includes that block.  If the
definition occurs in a function block, the scope extends to any blocks
contained within the defining one, unless a contained block introduces
a different binding for the name.  The scope of names defined in a
class block is limited to the class block; it does not extend to the
code blocks of methods - this includes generator expressions since they
are implemented using a function scope.  This means that the following
will fail:

    class A:
        a = 42
        b = list(a + i for i in range(10))

When a name is used in a code block, it is resolved using the nearest
enclosing scope.  The set of all such scopes visible to a code block is
called the block's _environment_.

  If a name is bound in a block, it is a local variable of that block.
If a name is bound at the module level, it is a global variable.  (The
variables of the module code block are local and global.)  If a
variable is used in a code block but not defined there, it is a _free
variable_.

  When a name is not found at all, a NameError exception is
raised.  If the name refers to a local variable that has not been
bound, a UnboundLocalError exception is raised.
UnboundLocalError is a subclass of NameError.

  The following constructs bind names: formal parameters to functions,
see import statements, class and function definitions (these
bind the class or function name in the defining block), and targets
that are identifiers if occurring in an assignment, see for.
loop header, in the second position of an except clause
header or after as in a with statement.  The
import statement of the form `from ... import *' binds all
names defined in the imported module, except those beginning with an
underscore.  This form may only be used at the module level.

  A target occurring in a del statement is also considered
bound for this purpose (though the actual semantics are to unbind the
name).  It is illegal to unbind a name that is referenced by an
enclosing scope; the compiler will report a SyntaxError.

  Each assignment or import statement occurs within a block defined by
a class or function definition or at the module level (the top-level
code block).

  If a name binding operation occurs anywhere within a code block, all
uses of the name within the block are treated as references to the
current block.  This can lead to errors when a name is used within a
block before it is bound. This rule is subtle.  Python lacks
declarations and allows name binding operations to occur anywhere
within a code block.  The local variables of a code block can be
determined by scanning the entire text of the block for name binding
operations.

  If the global statement occurs within a block, all uses of the name
specified in the statement refer to the binding of that name in the
top-level namespace.  Names are resolved in the top-level namespace by
searching the global namespace, i.e. the namespace of the module
containing the code block, and the builtins namespace, the namespace of
the module __builtin__.  The global namespace is searched
first.  If the name is not found there, the builtins namespace is
searched.  The global statement must precede all uses of the name.

  The builtins namespace associated with the execution of a code block
is actually found by looking up the name `__builtins__' in its global
namespace; this should be a dictionary or a module (in the latter case
the module's dictionary is used).  By default, when in the
__main__ module, `__builtins__' is the built-in module
__builtin__. (note: no 's'); when in any other module,
`__builtins__' is an alias for the dictionary of the __builtin__
module itself.  `__builtins__' can be set to a user-created
dictionary to create a weak form of restricted execution.

*CPython implementation detail:* Users should not touch `__builtins__';
it is strictly an implementation detail.  Users wanting to override
values in the builtins namespace should import the
__builtin__. (no 's') module and modify its attributes appropriately.

  The namespace for a module is automatically created the first time a
module is imported.  The main module for a script is always called
__main__.

  The global statement has the same scope as a name binding operation
in the same block.  If the nearest enclosing scope for a free variable
contains a global statement, the free variable is treated as a global.

  A class definition is an executable statement that may use and define
names.  These references follow the normal rules for name resolution.
The namespace of the class definition becomes the attribute dictionary
of the class.  Names defined at the class scope are not visible in
methods.

4.4.1.1 Interaction with dynamic features
.........................................

There are several cases where Python statements are illegal when used in
conjunction with nested scopes that contain free variables.

  If a variable is referenced in an enclosing scope, it is illegal to
delete the name.  An error will be reported at compile time.

  If the wild card form of import -- `import *' -- is used in a
function and the function contains or is a nested block with free
variables, the compiler will raise a SyntaxError.

  If exec is used in a function and the function contains
or is a nested block with free variables, the compiler will raise a
SyntaxError.  unless the exec explicitly specifies the local
namespace for the exec.  (In other words, `exec obj' would
be illegal, but `exec obj in ns' would be legal.)

  The eval(), execfile(), and input().
functions and the exec statement do not have access to the
full environment for resolving names.  Names may be resolved in the
local and global namespaces of the caller.  Free variables are not
resolved in the nearest enclosing namespace, but in the global
namespace. (1) The exec statement and the eval()
and execfile() functions have optional arguments to
override the global and local namespace.  If only one namespace is
specified, it is used for both.

4.4.2 Exceptions
----------------

Exceptions are a means of breaking out of the normal flow of control of
a code block in order to handle errors or other exceptional conditions.
An exception is _raised_ at the point where the error is detected; it
may be _handled_ by the surrounding code block or by any code block
that directly or indirectly invoked the code block where the error
occurred.

  The Python interpreter raises an exception when it detects a run-time
error (such as division by zero).  A Python program can also explicitly
raise an exception with the raise statement. Exception
handlers are specified with the try... See except.
statement.  The finally.  clause of such a statement can be
used to specify cleanup code which does not handle the exception, but
is executed whether an exception occurred or not in the preceding code.

  Python uses the "termination" model of error handling: an exception
handler can find out what happened and continue execution at an outer
level, but it cannot repair the cause of the error and retry the
failing operation (except by re-entering the offending piece of code
from the top).

  When an exception is not handled at all, the interpreter terminates
execution of the program, or returns to its interactive main loop.  In
either case, it prints a stack backtrace, except when the exception is
SystemExit.

  Exceptions are identified by class instances.  The except.
clause is selected depending on the class of the instance: it must
reference the class of the instance or a base class thereof.  The
instance can be received by the handler and can carry additional
information about the exceptional condition.

  Exceptions can also be identified by strings, in which case the
except clause is selected by object identity.  An arbitrary value
can be raised along with the identifying string which can be passed to
the handler.

     Note: Messages to exceptions are not part of the Python API.
     Their contents may change from one version of Python to the next
     without warning and should not be relied on by code which will run
     under multiple versions of the interpreter.

  See also the description of the try statement in section
The try statement.  and raise statement in
section The raise statement.

Footnotes
.........

     (1) This limitation occurs because the code that is executed by
     these operations is not available at the time the module is
     compiled.

4.5 Expressions
===============

This chapter explains the meaning of the elements of expressions in
Python.

  *Syntax Notes:* In this and the following chapters, extended BNF
notation will be used to describe syntax, not lexical analysis.  When
(one alternative of) a syntax rule has the form

    name ::= othername

and no semantics are given, the semantics of this form of `name' are
the same as for `othername'.

4.5.1 Arithmetic conversions
----------------------------

When a description of an arithmetic operator below uses the phrase "the
numeric arguments are converted to a common type," the arguments are
coerced using the coercion rules listed at  Coercion rules.
If both arguments are standard numeric types, the following coercions
are applied:

   * If either argument is a complex number, the other is converted to
     complex;

   * otherwise, if either argument is a floating point number, the
     other is converted to floating point;

   * otherwise, if either argument is a long integer, the other is
     converted to long integer;

   * otherwise, both must be plain integers and no conversion is
     necessary.

  Some additional rules apply for certain operators (e.g., a string
left argument to the '%' operator). Extensions can define their own
coercions.

4.5.2 Atoms
-----------

Atoms are the most basic elements of expressions.  The simplest atoms
are identifiers or literals.  Forms enclosed in reverse quotes or in
parentheses, brackets or braces are also categorized syntactically as
atoms.  The syntax for atoms is:

    atom      ::= identifier | literal | enclosure
    enclosure ::= parenth_form | list_display
                  | generator_expression | dict_display | set_display
                  | string_conversion | yield_atom

4.5.2.1 Identifiers (Names)
...........................

An identifier occurring as an atom is a name.  See section
Identifiers and keywords.  for lexical definition and section
Naming and binding for documentation of naming and binding.

  When the name is bound to an object, evaluation of the atom yields
that object.  When a name is not bound, an attempt to evaluate it
raises a NameError.  exception.

  *Private name mangling:* When an identifier that textually occurs in
a class definition begins with two or more underscore characters and
does not end in two or more underscores, it is considered a _private
name_ of that class.  Private names are transformed to a longer form
before code is generated for them.  The transformation inserts the
class name in front of the name, with leading underscores removed, and
a single underscore inserted in front of the class name.  For example,
the identifier `__spam' occurring in a class named `Ham' will be
transformed to `_Ham__spam'.  This transformation is independent of the
syntactical context in which the identifier is used.  If the
transformed name is extremely long (longer than 255 characters),
implementation defined truncation may happen.  If the class name
consists only of underscores, no transformation is done.

4.5.2.2 Literals
................

Python supports string literals and various numeric literals:

    literal ::= stringliteral | integer | longinteger
                | floatnumber | imagnumber

Evaluation of a literal yields an object of the given type (string,
integer, long integer, floating point number, complex number) with the
given value.  The value may be approximated in the case of floating
point and imaginary (complex) literals.  See section Literals
for details.

  All literals correspond to immutable data types, and hence the
object's identity is less important than its value.  Multiple
evaluations of literals with the same value (either the same occurrence
in the program text or a different occurrence) may obtain the same
object or a different object with the same value.

4.5.2.3 Parenthesized forms
...........................

A parenthesized form is an optional expression list enclosed in
parentheses:

    parenth_form ::= "(" [expression_list] ")"

A parenthesized expression list yields whatever that expression list
yields: if the list contains at least one comma, it yields a tuple;
otherwise, it yields the single expression that makes up the expression
list.

  An empty pair of parentheses yields an empty tuple object.  Since
tuples are immutable, the rules for literals apply (i.e., two
occurrences of the empty tuple may or may not yield the same object).

  Note that tuples are not formed by the parentheses, but rather by use
of the comma operator.  The exception is the empty tuple, for which
parentheses _are_ required -- allowing unparenthesized "nothing" in
expressions would cause ambiguities and allow common typos to pass
uncaught.

4.5.2.4 List displays
.....................

A list display is a possibly empty series of expressions enclosed in
square brackets:

    list_display        ::= "[" [expression_list | list_comprehension] "]"
    list_comprehension  ::= expression list_for
    list_for            ::= "for" target_list "in" old_expression_list [list_iter]
    old_expression_list ::= old_expression [("," old_expression)+ [","]]
    old_expression      ::= or_test | old_lambda_form
    list_iter           ::= list_for | list_if
    list_if             ::= "if" old_expression [list_iter]

A list display yields a new list object.  Its contents are specified by
providing either a list of expressions or a list comprehension.  When a
comma-separated list of expressions is supplied, its elements are
evaluated from left to right and placed into the list object in that
order.  When a list comprehension is supplied, it consists of a single
expression followed by at least one for clause and zero or
more for or if.  clauses.  In this case, the
elements of the new list are those that would be produced by
considering each of the for or if clauses a
block, nesting from left to right, and evaluating the expression to
produce a list element each time the innermost block is reached (1).

4.5.2.5 Displays for sets and dictionaries
..........................................

For constructing a set or a dictionary Python provides special syntax
called "displays", each of them in two flavors:

   * either the container contents are listed explicitly, or

   * they are computed via a set of looping and filtering instructions,
     called a _comprehension_.

  Common syntax elements for comprehensions are:

    comprehension ::= expression comp_for
    comp_for      ::= "for" target_list "in" or_test [comp_iter]
    comp_iter     ::= comp_for | comp_if
    comp_if       ::= "if" expression_nocond [comp_iter]

The comprehension consists of a single expression followed by at least
one for clause and zero or more for or
if clauses.  In this case, the elements of the new container are
those that would be produced by considering each of the for.
or if clauses a block, nesting from left to right, and
evaluating the expression to produce an element each time the innermost
block is reached.

  Note that the comprehension is executed in a separate scope, so names
assigned to in the target list don't "leak" in the enclosing scope.

4.5.2.6 Generator expressions
.............................

A generator expression is a compact generator notation in parentheses:

    generator_expression ::= "(" expression comp_for ")"

A generator expression yields a new generator object.  Its syntax is
the same as for comprehensions, except that it is enclosed in
parentheses instead of brackets or curly braces.

  Variables used in the generator expression are evaluated lazily when
the `__next__()' method is called for generator object (in the same
fashion as normal generators).  However, the leftmost for.
clause is immediately evaluated, so that an error produced by it can be
seen before any other possible error in the code that handles the
generator expression.  Subsequent for clauses cannot be
evaluated immediately since they may depend on the previous for
loop. For example: `(x*y for x in range(10) for y in bar(x))'.

  The parentheses can be omitted on calls with only one argument.  See
section Calls for the detail.

4.5.2.7 Dictionary displays
...........................

A dictionary display is a possibly empty series of key/datum pairs
enclosed in curly braces:

    dict_display       ::= "{" [key_datum_list | dict_comprehension] "}"
    key_datum_list     ::= key_datum ("," key_datum)* [","]
    key_datum          ::= expression ":" expression
    dict_comprehension ::= expression ":" expression comp_for

A dictionary display yields a new dictionary object.

  If a comma-separated sequence of key/datum pairs is given, they are
evaluated from left to right to define the entries of the dictionary:
each key object is used as a key into the dictionary to store the
corresponding datum.  This means that you can specify the same key
multiple times in the key/datum list, and the final dictionary's value
for that key will be the last one given.

  A dict comprehension, in contrast to list and set comprehensions,
needs two expressions separated with a colon followed by the usual
"for" and "if" clauses.  When the comprehension is run, the resulting
key and value elements are inserted in the new dictionary in the order
they are produced.

  Restrictions on the types of the key values are listed earlier in
section The standard type hierarchy.  (To summarize, the key
type should be hashable, which excludes all mutable
objects.)  Clashes between duplicate keys are not detected; the last
datum (textually rightmost in the display) stored for a given key value
prevails.

4.5.2.8 Set displays
....................

A set display is denoted by curly braces and distinguishable from
dictionary displays by the lack of colons separating keys and values:

    set_display ::= "{" (expression_list | comprehension) "}"

A set display yields a new mutable set object, the contents being
specified by either a sequence of expressions or a comprehension.  When
a comma-separated list of expressions is supplied, its elements are
evaluated from left to right and added to the set object.  When a
comprehension is supplied, the set is constructed from the elements
resulting from the comprehension.

  An empty set cannot be constructed with `{}'; this literal constructs
an empty dictionary.

4.5.2.9 String conversions
..........................

A string conversion is an expression list enclosed in reverse (a.k.a.
backward) quotes:

    string_conversion ::= "'" expression_list "'"

A string conversion evaluates the contained expression list and
converts the resulting object into a string according to rules specific
to its type.

  If the object is a string, a number, `None', or a tuple, list or
dictionary containing only objects whose type is one of these, the
resulting string is a valid Python expression which can be passed to
the built-in function eval() to yield an expression with
the same value (or an approximation, if floating point numbers are
involved).

  (In particular, converting a string adds quotes around it and
converts "funny" characters to escape sequences that are safe to print.)

  Recursive objects (for example, lists or dictionaries that contain a
reference to themselves, directly or indirectly) use `...' to indicate
a recursive reference, and the result cannot be passed to eval()
to get an equal value (see SyntaxError will be raised
instead).

  The built-in function repr() performs exactly the same
conversion in its argument as enclosing it in parentheses and reverse
quotes does.  The built-in function str() performs a
similar but more user-friendly conversion.

4.5.2.10 Yield expressions
..........................

    yield_atom       ::= "(" yield_expression ")"
    yield_expression ::= "yield" [expression_list]

New in version 2.5.

  The yield expression is only used when defining a
generator function, and can only be used in the body of a function
definition. Using a yield expression in a function
definition is sufficient to cause that definition to create a generator
function instead of a normal function.

  When a generator function is called, it returns an iterator known as a
generator.  That generator then controls the execution of a generator
function.  The execution starts when one of the generator's methods is
called.  At that time, the execution proceeds to the first yield
expression, where it is suspended again, returning the value of
expression_list to generator's caller.  By suspended we
mean that all local state is retained, including the current bindings
of local variables, the instruction pointer, and the internal
evaluation stack.  When the execution is resumed by calling one of the
generator's methods, the function can proceed exactly as if the
yield expression was just another external call. The value of the
yield expression after resuming depends on the method which
resumed the execution.

  All of this makes generator functions quite similar to coroutines;
they yield multiple times, they have more than one entry point and
their execution can be suspended.  The only difference is that a
generator function cannot control where should the execution continue
after it yields; the control is always transferred to the generator's
caller.

  The following generator's methods can be used to control the
execution of a generator function:

 -- Method: generator.next ()
     Starts the execution of a generator function or resumes it at the
     last executed yield expression.  When a generator
     function is resumed with a next() method, the current
     yield expression always evaluates to None.
     The execution then continues to the next yield.
     expression, where the generator is suspended again, and the value
     of the expression_list is returned to next().'
     s caller. If the generator exits without yielding another
     value, a StopIteration exception is raised.

 -- Method: generator.send (value)
     Resumes the execution and "sends" a value into the generator
     function.  The `value' argument becomes the result of the current
     yield.  expression.  The send() method
     returns the next value yielded by the generator, or raises
     StopIteration if the generator exits without yielding
     another value. When send() is called to start the
     generator, it must be called with None as the
     argument, because there is no yield expression that
     could receive the value.

 -- Method: generator.throw (type[, value[, traceback]])
     Raises an exception of type `type' at the point where generator
     was paused, and returns the next value yielded by the generator
     function.  If the generator exits without yielding another value,
     a StopIteration exception is raised.  If the generator
     function does not catch the passed-in exception, or raises a
     different exception, then that exception propagates to the caller.

 -- Method: generator.close ()
     Raises a GeneratorExit at the point where the
     generator function was paused.  If the generator function then
     raises StopIteration. (by exiting normally, or due to
     already being closed) or GeneratorExit. (by not
     catching the exception), close returns to its caller.  If the
     generator yields a value, a RuntimeError is raised.
     If the generator raises any other exception, it is propagated to
     the caller.  close() does nothing if the generator has
     already exited due to an exception or normal exit.

  Here is a simple example that demonstrates the behavior of generators
and generator functions:

    >>> def echo(value=None):
    ...     print "Execution starts when 'next()' is called for the first time."
    ...     try:
    ...         while True:
    ...             try:
    ...                 value = (yield value)
    ...             except Exception, e:
    ...                 value = e
    ...     finally:
    ...         print "Don't forget to clean up when 'close()' is called."
    ...
    >>> generator = echo(1)
    >>> print generator.next()
    Execution starts when 'next()' is called for the first time.
    1
    >>> print generator.next()
    None
    >>> print generator.send(2)
    2
    >>> generator.throw(TypeError, "spam")
    TypeError('spam',)
    >>> generator.close()
    Don't forget to clean up when 'close()' is called.

PEP 0342 (http://www.python.org/dev/peps/pep-0342) - Coroutines via Enhanced Generators
     The proposal to enhance the API and syntax of generators, making
     them usable as simple coroutines.

4.5.3 Primaries
---------------

Primaries represent the most tightly bound operations of the language.
Their syntax is:

    primary ::= atom | attributeref | subscription | slicing | call

4.5.3.1 Attribute references
............................

An attribute reference is a primary followed by a period and a name:

    attributeref ::= primary "." identifier

The primary must evaluate to an object of a type that supports attribute
references, e.g., a module, list, or an instance.  This object is then
asked to produce the attribute whose name is the identifier.  If this
attribute is not available, the exception AttributeError is
raised. Otherwise, the type and value of the object produced is
determined by the object.  Multiple evaluations of the same attribute
reference may yield different objects.

4.5.3.2 Subscriptions
.....................

A subscription selects an item of a sequence (string, tuple or list) or
mapping (dictionary) object:

    subscription ::= primary "[" expression_list "]"

The primary must evaluate to an object of a sequence or mapping type.

  If the primary is a mapping, the expression list must evaluate to an
object whose value is one of the keys of the mapping, and the
subscription selects the value in the mapping that corresponds to that
key.  (The expression list is a tuple except if it has exactly one
item.)

  If the primary is a sequence, the expression (list) must evaluate to
a plain integer.  If this value is negative, the length of the sequence
is added to it (so that, e.g., `x[-1]' selects the last item of `x'.)
The resulting value must be a nonnegative integer less than the number
of items in the sequence, and the subscription selects the item whose
index is that value (counting from zero).

  A string's items are characters.  A character is not a separate data
type but a string of exactly one character.

4.5.3.3 Slicings
................

A slicing selects a range of items in a sequence object (e.g., a
string, tuple or list).  Slicings may be used as expressions or as
targets in assignment or del statements.  The syntax for a
slicing:

    slicing          ::= simple_slicing | extended_slicing
    simple_slicing   ::= primary "[" short_slice "]"
    extended_slicing ::= primary "[" slice_list "]"
    slice_list       ::= slice_item ("," slice_item)* [","]
    slice_item       ::= expression | proper_slice | ellipsis
    proper_slice     ::= short_slice | long_slice
    short_slice      ::= [lower_bound] ":" [upper_bound]
    long_slice       ::= short_slice ":" [stride]
    lower_bound      ::= expression
    upper_bound      ::= expression
    stride           ::= expression
    ellipsis         ::= "..."

There is ambiguity in the formal syntax here: anything that looks like
an expression list also looks like a slice list, so any subscription
can be interpreted as a slicing.  Rather than further complicating the
syntax, this is disambiguated by defining that in this case the
interpretation as a subscription takes priority over the interpretation
as a slicing (this is the case if the slice list contains no proper
slice nor ellipses).  Similarly, when the slice list has exactly one
short slice and no trailing comma, the interpretation as a simple
slicing takes priority over that as an extended slicing.

  The semantics for a simple slicing are as follows.  The primary must
evaluate to a sequence object.  The lower and upper bound expressions,
if present, must evaluate to plain integers; defaults are zero and the
`sys.maxint', respectively.  If either bound is negative, the
sequence's length is added to it.  The slicing now selects all items
with index _k_ such that `i <= k < j' where _i_ and _j_ are the
specified lower and upper bounds.  This may be an empty sequence.  It
is not an error if _i_ or _j_ lie outside the range of valid indexes
(such items don't exist so they aren't selected).

  The semantics for an extended slicing are as follows.  The primary
must evaluate to a mapping object, and it is indexed with a key that is
constructed from the slice list, as follows.  If the slice list
contains at least one comma, the key is a tuple containing the
conversion of the slice items; otherwise, the conversion of the lone
slice item is the key.  The conversion of a slice item that is an
expression is that expression.  The conversion of an ellipsis slice
item is the built-in `Ellipsis' object.  The conversion of a proper
slice is a slice object (see section The standard type hierarchy)
whose `start', `stop' and `step' attributes are the values of the
expressions given as lower bound, upper bound and stride, respectively,
substituting `None' for missing expressions.

4.5.3.4 Calls
.............

A call calls a callable object (e.g., a function) with a possibly empty
series of arguments:

    call                 ::= primary "(" [argument_list [","]
             | expression genexpr_for] ")"
    argument_list        ::= positional_arguments ["," keyword_arguments]
                        ["," "*" expression] ["," keyword_arguments]
                        ["," "**" expression]
                      | keyword_arguments ["," "*" expression]
                        ["," "**" expression]
                      | "*" expression ["," "*" expression] ["," "**" expression]
                      | "**" expression
    positional_arguments ::= expression ("," expression)*
    keyword_arguments    ::= keyword_item ("," keyword_item)*
    keyword_item         ::= identifier "=" expression

A trailing comma may be present after the positional and keyword
arguments but does not affect the semantics.

  The primary must evaluate to a callable object (user-defined
functions, built-in functions, methods of built-in objects, class
objects, methods of class instances, and certain class instances
themselves are callable; extensions may define additional callable
object types).  All argument expressions are evaluated before the call
is attempted.  Please refer to section Function definitions.
for the syntax of formal parameter lists.

  If keyword arguments are present, they are first converted to
positional arguments, as follows.  First, a list of unfilled slots is
created for the formal parameters.  If there are N positional
arguments, they are placed in the first N slots.  Next, for each
keyword argument, the identifier is used to determine the corresponding
slot (if the identifier is the same as the first formal parameter name,
the first slot is used, and so on).  If the slot is already filled, a
TypeError exception is raised. Otherwise, the value of the
argument is placed in the slot, filling it (even if the expression is
`None', it fills the slot).  When all arguments have been processed,
the slots that are still unfilled are filled with the corresponding
default value from the function definition.  (Default values are
calculated, once, when the function is defined; thus, a mutable object
such as a list or dictionary used as default value will be shared by
all calls that don't specify an argument value for the corresponding
slot; this should usually be avoided.)  If there are any unfilled slots
for which no default value is specified, a TypeError.
exception is raised.  Otherwise, the list of filled slots is used as
the argument list for the call.

*CPython implementation detail:* An implementation may provide built-in
functions whose positional parameters do not have names, even if they
are 'named' for the purpose of documentation, and which therefore
cannot be supplied by keyword.  In CPython, this is the case for
functions implemented in C that use PyArg_ParseTuple() to
parse their arguments.

  If there are more positional arguments than there are formal
parameter slots, a TypeError exception is raised, unless a
formal parameter using the syntax `*identifier' is present; in this
case, that formal parameter receives a tuple containing the excess
positional arguments (or an empty tuple if there were no excess
positional arguments).

  If any keyword argument does not correspond to a formal parameter
name, a TypeError exception is raised, unless a formal
parameter using the syntax `**identifier' is present; in this case,
that formal parameter receives a dictionary containing the excess
keyword arguments (using the keywords as keys and the argument values
as corresponding values), or a (new) empty dictionary if there were no
excess keyword arguments.

  If the syntax `*expression' appears in the function call,
`expression' must evaluate to a sequence.  Elements from this sequence
are treated as if they were additional positional arguments; if there
are positional arguments _x1_,..., _xN_, and `expression' evaluates to
a sequence _y1_, ..., _yM_, this is equivalent to a call with M+N
positional arguments _x1_, ..., _xN_, _y1_, ..., _yM_.

  A consequence of this is that although the `*expression' syntax may
appear _after_ some keyword arguments, it is processed _before_ the
keyword arguments (and the `**expression' argument, if any - see
below).  So:

    >>> def f(a, b):
    ...  print a, b
    ...
    >>> f(b=1, *(2,))
    2 1
    >>> f(a=1, *(2,))
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: f() got multiple values for keyword argument 'a'
    >>> f(1, *(2,))
    1 2

It is unusual for both keyword arguments and the `*expression' syntax
to be used in the same call, so in practice this confusion does not
arise.

  If the syntax `**expression' appears in the function call,
`expression' must evaluate to a mapping, the contents of which are
treated as additional keyword arguments.  In the case of a keyword
appearing in both `expression' and as an explicit keyword argument, a
TypeError exception is raised.

  Formal parameters using the syntax `*identifier' or `**identifier'
cannot be used as positional argument slots or as keyword argument
names.  Formal parameters using the syntax `(sublist)' cannot be used
as keyword argument names; the outermost sublist corresponds to a
single unnamed argument slot, and the argument value is assigned to the
sublist using the usual tuple assignment rules after all other
parameter processing is done.

  A call always returns some value, possibly `None', unless it raises an
exception.  How this value is computed depends on the type of the
callable object.

  If it is--

a user-defined function:
     The code block for the function is executed, passing it the
     argument list.  The first thing the code block will do is bind the
     formal parameters to the arguments; this is described in section
     Function definitions.  When the code block executes a
     return statement, this specifies the return value of
     the function call.

a built-in function or method:
     The result is up to the interpreter; see Built-in Functions
     for the descriptions of built-in functions and methods.

a class object:
     A new instance of that class is returned.

a class instance method:
     The corresponding user-defined function is called, with an
     argument list that is one longer than the argument list of the
     call: the instance becomes the first argument.

a class instance:
     The class must define a __call__() method; the effect
     is then the same as if that method was called.

4.5.4 The power operator
------------------------

The power operator binds more tightly than unary operators on its left;
it binds less tightly than unary operators on its right.  The syntax is:

    power ::= primary ["**" u_expr]

Thus, in an unparenthesized sequence of power and unary operators, the
operators are evaluated from right to left (this does not constrain the
evaluation order for the operands): `-1**2' results in `-1'.

  The power operator has the same semantics as the built-in
pow() function, when called with two arguments: it yields its
left argument raised to the power of its right argument.  The numeric
arguments are first converted to a common type.  The result type is
that of the arguments after coercion.

  With mixed operand types, the coercion rules for binary arithmetic
operators apply. For int and long int operands, the result has the same
type as the operands (after coercion) unless the second argument is
negative; in that case, all arguments are converted to float and a
float result is delivered. For example, `10**2' returns `100', but
`10**-2' returns `0.01'. (This last feature was added in Python 2.2. In
Python 2.1 and before, if both arguments were of integer types and the
second argument was negative, an exception was raised).

  Raising `0.0' to a negative power results in a
ZeroDivisionError.  Raising a negative number to a fractional
power results in a ValueError.

4.5.5 Unary arithmetic and bitwise operations
---------------------------------------------

All unary arithmetic and bitwise operations have the same priority:

    u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr

The unary `-' (minus) operator yields the negation of its numeric
argument.

  The unary `+' (plus) operator yields its numeric argument unchanged.

  The unary `~' (invert) operator yields the bitwise inversion of its
plain or long integer argument.  The bitwise inversion of `x' is
defined as `-(x+1)'.  It only applies to integral numbers.

  In all three cases, if the argument does not have the proper type, a
TypeError exception is raised.

4.5.6 Binary arithmetic operations
----------------------------------

The binary arithmetic operations have the conventional priority levels.
Note that some of these operations also apply to certain non-numeric
types.  Apart from the power operator, there are only two levels, one
for multiplicative operators and one for additive operators:

    m_expr ::= u_expr | m_expr "*" u_expr | m_expr "//" u_expr | m_expr "/" u_expr
               | m_expr "%" u_expr
    a_expr ::= m_expr | a_expr "+" m_expr | a_expr "-" m_expr

The `*' (multiplication) operator yields the product of its arguments.
The arguments must either both be numbers, or one argument must be an
integer (plain or long) and the other must be a sequence. In the former
case, the numbers are converted to a common type and then multiplied
together.  In the latter case, sequence repetition is performed; a
negative repetition factor yields an empty sequence.

  The `/' (division) and `//' (floor division) operators yield the
quotient of their arguments.  The numeric arguments are first converted
to a common type.  Plain or long integer division yields an integer of
the same type; the result is that of mathematical division with the
'floor' function applied to the result.  Division by zero raises the
ZeroDivisionError exception.

  The `%' (modulo) operator yields the remainder from the division of
the first argument by the second.  The numeric arguments are first
converted to a common type.  A zero right argument raises the
ZeroDivisionError exception.  The arguments may be floating point
numbers, e.g., `3.14%0.7' equals `0.34' (since `3.14' equals `4*0.7 +
0.34'.)  The modulo operator always yields a result with the same sign
as its second operand (or zero); the absolute value of the result is
strictly smaller than the absolute value of the second operand (2).

  The integer division and modulo operators are connected by the
following identity: `x == (x/y)*y + (x%y)'.  Integer division and
modulo are also connected with the built-in function divmod().:
`divmod(x, y) == (x/y, x%y)'.  These identities don't hold for
floating point numbers; there similar identities hold approximately
where `x/y' is replaced by `floor(x/y)' or `floor(x/y) - 1' (3).

  In addition to performing the modulo operation on numbers, the `%'
operator is also overloaded by string and unicode objects to perform
string formatting (also known as interpolation). The syntax for string
formatting is described in the Python Library Reference, section
String Formatting Operations.

  Deprecated since version 2.3: The floor division operator, the modulo
operator, and the divmod().  function are no longer defined
for complex numbers.  Instead, convert to a floating point number using
the abs() function if appropriate.

  The `+' (addition) operator yields the sum of its arguments. The
arguments must either both be numbers or both sequences of the same
type.  In the former case, the numbers are converted to a common type
and then added together.  In the latter case, the sequences are
concatenated.

  The `-' (subtraction) operator yields the difference of its
arguments.  The numeric arguments are first converted to a common type.

4.5.7 Shifting operations
-------------------------

The shifting operations have lower priority than the arithmetic
operations:

    shift_expr ::= a_expr | shift_expr ( "<<" | ">>" ) a_expr

These operators accept plain or long integers as arguments.  The
arguments are converted to a common type.  They shift the first
argument to the left or right by the number of bits given by the second
argument.

  A right shift by _n_ bits is defined as division by `pow(2, n)'.  A
left shift by _n_ bits is defined as multiplication with `pow(2, n)'.
Negative shift counts raise a ValueError exception.

     Note: In the current implementation, the right-hand operand is
     required to be at most sys.maxsize.  If the right-hand
     operand is larger than sys.maxsize an
     OverflowError exception is raised.

4.5.8 Binary bitwise operations
-------------------------------

Each of the three bitwise operations has a different priority level:

    and_expr ::= shift_expr | and_expr "&" shift_expr
    xor_expr ::= and_expr | xor_expr "^" and_expr
    or_expr  ::= xor_expr | or_expr "|" xor_expr

The `&' operator yields the bitwise AND of its arguments, which must be
plain or long integers.  The arguments are converted to a common type.

  The `^' operator yields the bitwise XOR (exclusive OR) of its
arguments, which must be plain or long integers.  The arguments are
converted to a common type.

  The `|' operator yields the bitwise (inclusive) OR of its arguments,
which must be plain or long integers.  The arguments are converted to a
common type.

4.5.9 Comparisons
-----------------

Unlike C, all comparison operations in Python have the same priority,
which is lower than that of any arithmetic, shifting or bitwise
operation.  Also unlike C, expressions like `a < b < c' have the
interpretation that is conventional in mathematics:

    comparison    ::= or_expr ( comp_operator or_expr )*
    comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "<>" | "!="
                      | "is" ["not"] | ["not"] "in"

Comparisons yield boolean values: `True' or `False'.

  Comparisons can be chained arbitrarily, e.g., `x < y <= z' is
equivalent to `x < y and y <= z', except that `y' is evaluated only
once (but in both cases `z' is not evaluated at all when `x < y' is
found to be false).

  Formally, if _a_, _b_, _c_, ..., _y_, _z_ are expressions and _op1_,
_op2_, ..., _opN_ are comparison operators, then `a op1 b op2 c ... y
opN z' is equivalent to `a op1 b and b op2 c and ... y opN z', except
that each expression is evaluated at most once.

  Note that `a op1 b op2 c' doesn't imply any kind of comparison
between _a_ and _c_, so that, e.g., `x < y > z' is perfectly legal
(though perhaps not pretty).

  The forms `<>' and `!=' are equivalent; for consistency with C, `!='
is preferred; where `!=' is mentioned below `<>' is also accepted.  The
`<>' spelling is considered obsolescent.

  The operators `<', `>', `==', `>=', `<=', and `!=' compare the values
of two objects.  The objects need not have the same type. If both are
numbers, they are converted to a common type.  Otherwise, objects of
different types _always_ compare unequal, and are ordered consistently
but arbitrarily.  You can control comparison behavior of objects of
non-built-in types by defining a `__cmp__' method or rich comparison
methods like `__gt__', described in section Special method names.

  (This unusual definition of comparison was used to simplify the
definition of operations like sorting and the in and `not
in' operators.  In the future, the comparison rules for objects of
different types are likely to change.)

  Comparison of objects of the same type depends on the type:

   * Numbers are compared arithmetically.

   * Strings are compared lexicographically using the numeric
     equivalents (the result of the built-in function ord())
     of their characters.  Unicode and 8-bit strings are fully
     interoperable in this behavior. (4)

   * Tuples and lists are compared lexicographically using comparison of
     corresponding elements.  This means that to compare equal, each
     element must compare equal and the two sequences must be of the
     same type and have the same length.

     If not equal, the sequences are ordered the same as their first
     differing elements.  For example, `cmp([1,2,x], [1,2,y])' returns
     the same as `cmp(x,y)'.  If the corresponding element does not
     exist, the shorter sequence is ordered first (for example, `[1,2]
     < [1,2,3]').

   * Mappings (dictionaries) compare equal if and only if their sorted
     (key, value) lists compare equal. (5) Outcomes other than equality
     are resolved consistently, but are not otherwise defined. (6)

   * Most other objects of built-in types compare unequal unless they
     are the same object; the choice whether one object is considered
     smaller or larger than another one is made arbitrarily but
     consistently within one execution of a program.

  The operators in and `not in' test for collection
membership.  `x in s' evaluates to true if _x_ is a member of the
collection _s_, and false otherwise.  `x not in s' returns the negation
of `x in s'.  The collection membership test has traditionally been
bound to sequences; an object is a member of a collection if the
collection is a sequence and contains an element equal to that object.
However, it make sense for many other object types to support
membership tests without being a sequence.  In particular, dictionaries
(for keys) and sets support membership testing.

  For the list and tuple types, `x in y' is true if and only if there
exists an index _i_ such that `x == y[i]' is true.

  For the Unicode and string types, `x in y' is true if and only if _x_
is a substring of _y_.  An equivalent test is `y.find(x) != -1'.  Note,
_x_ and _y_ need not be the same type; consequently, `u'ab' in 'abc''
will return `True'. Empty strings are always considered to be a
substring of any other string, so `"" in "abc"' will return `True'.

  Changed in version 2.3: Previously, _x_ was required to be a string
of length `1'.

  For user-defined classes which define the __contains__().
method, `x in y' is true if and only if `y.__contains__(x)' is true.

  For user-defined classes which do not define __contains__()
but do define __iter__(), `x in y' is true if some
value `z' with `x == z' is produced while iterating over `y'.  If an
exception is raised during the iteration, it is as if in.
raised that exception.

  Lastly, the old-style iteration protocol is tried: if a class defines
__getitem__(), `x in y' is true if and only if there is a
non-negative integer index _i_ such that `x == y[i]', and all lower
integer indices do not raise IndexError exception. (If any
other exception is raised, it is as if in raised that
exception).

  The operator `not in' is defined to have the inverse true value of
in.

  The operators is and `is not' test for object identity: `x
is y' is true if and only if _x_ and _y_ are the same object.  `x is
not y' yields the inverse truth value. (7)

4.5.10 Boolean operations
-------------------------

    or_test  ::= and_test | or_test "or" and_test
    and_test ::= not_test | and_test "and" not_test
    not_test ::= comparison | "not" not_test

In the context of Boolean operations, and also when expressions are
used by control flow statements, the following values are interpreted
as false: `False', `None', numeric zero of all types, and empty strings
and containers (including strings, tuples, lists, dictionaries, sets
and frozensets).  All other values are interpreted as true.  (See the
__nonzero__().  special method for a way to change this.)

  The operator not yields `True' if its argument is false,
`False' otherwise.

  The expression `x and y' first evaluates _x_; if _x_ is false, its
value is returned; otherwise, _y_ is evaluated and the resulting value
is returned.

  The expression `x or y' first evaluates _x_; if _x_ is true, its
value is returned; otherwise, _y_ is evaluated and the resulting value
is returned.

  (Note that neither and nor or restrict the
value and type they return to `False' and `True', but rather return the
last evaluated argument. This is sometimes useful, e.g., if `s' is a
string that should be replaced by a default value if it is empty, the
expression `s or 'foo'' yields the desired value.  Because not
has to invent a value anyway, it does not bother to return a value
of the same type as its argument, so e.g., `not 'foo'' yields `False',
not `'''.)

4.5.11 Conditional Expressions
------------------------------

New in version 2.5.

    conditional_expression ::= or_test ["if" or_test "else" expression]
    expression             ::= conditional_expression | lambda_form

Conditional expressions (sometimes called a "ternary operator") have
the lowest priority of all Python operations.

  The expression `x if C else y' first evaluates the condition, _C_
(_not_ _x_); if _C_ is true, _x_ is evaluated and its value is
returned; otherwise, _y_ is evaluated and its value is returned.

  See PEP 308 (http://www.python.org/dev/peps/pep-0308) for more
details about conditional expressions.

4.5.12 Lambdas
--------------

    lambda_form     ::= "lambda" [parameter_list]: expression
    old_lambda_form ::= "lambda" [parameter_list]: old_expression

Lambda forms (lambda expressions) have the same syntactic position as
expressions.  They are a shorthand to create anonymous functions; the
expression `lambda arguments: expression' yields a function object.
The unnamed object behaves like a function object defined with

    def name(arguments):
        return expression

See section Function definitions for the syntax of
parameter lists.  Note that functions created with lambda forms cannot
contain statements.

4.5.13 Expression lists
-----------------------

    expression_list ::= expression ( "," expression )* [","]

An expression list containing at least one comma yields a tuple.  The
length of the tuple is the number of expressions in the list.  The
expressions are evaluated from left to right.

  The trailing comma is required only to create a single tuple (a.k.a. a
_singleton_); it is optional in all other cases.  A single expression
without a trailing comma doesn't create a tuple, but rather yields the
value of that expression. (To create an empty tuple, use an empty pair
of parentheses: `()'.)

4.5.14 Evaluation order
-----------------------

Python evaluates expressions from left to right. Notice that while
evaluating an assignment, the right-hand side is evaluated before the
left-hand side.

  In the following lines, expressions will be evaluated in the
arithmetic order of their suffixes:

    expr1, expr2, expr3, expr4
    (expr1, expr2, expr3, expr4)
    {expr1: expr2, expr3: expr4}
    expr1 + expr2 * (expr3 - expr4)
    expr1(expr2, expr3, *expr4, **expr5)
    expr3, expr4 = expr1, expr2

4.5.15 Summary
--------------

The following table summarizes the operator precedences in Python, from
lowest precedence (least binding) to highest precedence (most binding).
Operators in the same box have the same precedence.  Unless the syntax
is explicitly given, operators are binary.  Operators in the same box
group left to right (except for comparisons, including tests, which all
have the same precedence and chain from left to right -- see section
Comparisons. -- and exponentiation, which groups from right
to left).

Operator                                            Description
----------------------------------------------------------------------------------------------
see lambda.                                  Lambda expression
if. - see else.                   Conditional expression
or.                                      Boolean OR
and.                                     Boolean AND
not. _x_                                 Boolean NOT
in, not. See in,       Comparisons, including membership tests
is, `is not', `<', `<=', `>', `>=',      and identity tests,
`<>', `!=', `=='
`|'                                                 Bitwise OR
`^'                                                 Bitwise XOR
`&'                                                 Bitwise AND
`<<', `>>'                                          Shifts
`+', `-'                                            Addition and subtraction
`*', `/', `//', `%'                                 Multiplication, division, remainder (8)
`+x', `-x', `~x'                                    Positive, negative, bitwise NOT
`**'                                                Exponentiation (9)
`x[index]', `x[index:index]', `x(arguments...)',    Subscription, slicing, call, attribute
`x.attribute'                                       reference
`(expressions...)', `[expressions...]',             Binding or tuple display, list display,
`{key:datum...}', ``expressions...`'                dictionary display, string conversion

Footnotes
.........

     (1) In Python 2.3 and later releases, a list comprehension "leaks"
     the control variables of each `for' it contains into the
     containing scope.  However, this behavior is deprecated, and
     relying on it will not work in Python 3.0

     (2) While `abs(x%y) < abs(y)' is true mathematically, for floats
     it may not be true numerically due to roundoff.  For example, and
     assuming a platform on which a Python float is an IEEE 754
     double-precision number, in order that `-1e-100 % 1e100' have the
     same sign as `1e100', the computed result is `-1e-100 + 1e100',
     which is numerically exactly equal to `1e100'.  Function `fmod()'
     in the math module returns a result whose sign matches
     the sign of the first argument instead, and so returns `-1e-100'
     in this case. Which approach is more appropriate depends on the
     application.

     (3) If x is very close to an exact integer multiple of y, it's
     possible for `floor(x/y)' to be one larger than `(x-x%y)/y' due to
     rounding.  In such cases, Python returns the latter result, in
     order to preserve that `divmod(x,y)[0] * y + x % y' be very close
     to `x'.

     (4) While comparisons between unicode strings make sense at the
     byte level, they may be counter-intuitive to users. For example,
     the strings `u"\u00C7"' and `u"\u0043\u0327"' compare differently,
     even though they both represent the same unicode character (LATIN
     CAPITAL LETTER C WITH CEDILLA). To compare strings in a human
     recognizable way, compare using unicodedata.normalize().

     (5) The implementation computes this efficiently, without
     constructing lists or sorting.

     (6) Earlier versions of Python used lexicographic comparison of
     the sorted (key, value) lists, but this was very expensive for the
     common case of comparing for equality.  An even earlier version of
     Python compared dictionaries by identity only, but this caused
     surprises because people expected to be able to test a dictionary
     for emptiness by comparing it to `{}'.

     (7) Due to automatic garbage-collection, free lists, and the
     dynamic nature of descriptors, you may notice seemingly unusual
     behaviour in certain uses of the is operator, like
     those involving comparisons between instance methods, or
     constants.  Check their documentation for more info.

     (8) The `%' is also used for string formatting; the same
     precedence applies.

     (9) The power operator `**' binds less tightly than an arithmetic
     or bitwise unary operator on its right, that is, `2**-1' is `0.5'.

4.6 Simple statements
=====================

Simple statements are comprised within a single logical line. Several
simple statements may occur on a single line separated by semicolons.
The syntax for simple statements is:

    simple_stmt ::= expression_stmt
                    | assert_stmt
                    | assignment_stmt
                    | augmented_assignment_stmt
                    | pass_stmt
                    | del_stmt
                    | print_stmt
                    | return_stmt
                    | yield_stmt
                    | raise_stmt
                    | break_stmt
                    | continue_stmt
                    | import_stmt
                    | global_stmt
                    | exec_stmt

4.6.1 Expression statements
---------------------------

Expression statements are used (mostly interactively) to compute and
write a value, or (usually) to call a procedure (a function that
returns no meaningful result; in Python, procedures return the value
`None').  Other uses of expression statements are allowed and
occasionally useful.  The syntax for an expression statement is:

    expression_stmt ::= expression_list

An expression statement evaluates the expression list (which may be a
single expression).

  In interactive mode, if the value is not `None', it is converted to a
string using the built-in repr() function and the resulting
string is written to standard output (see section The print statement)
on a line by itself.  (Expression statements yielding
`None' are not written, so that procedure calls do not cause any
output.)

4.6.2 Assignment statements
---------------------------

Assignment statements are used to (re)bind names to values and to modify
attributes or items of mutable objects:

    assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)
    target_list     ::= target ("," target)* [","]
    target          ::= identifier
               | "(" target_list ")"
               | "[" target_list "]"
               | attributeref
               | subscription
               | slicing

(See section Primaries for the syntax definitions for the
last three symbols.)

  An assignment statement evaluates the expression list (remember that
this can be a single expression or a comma-separated list, the latter
yielding a tuple) and assigns the single resulting object to each of
the target lists, from left to right.

  Assignment is defined recursively depending on the form of the target
(list).  When a target is part of a mutable object (an attribute
reference, subscription or slicing), the mutable object must ultimately
perform the assignment and decide about its validity, and may raise an
exception if the assignment is unacceptable.  The rules observed by
various types and the exceptions raised are given with the definition
of the object types (see section The standard type hierarchy).

  Assignment of an object to a target list is recursively defined as
follows.

   * If the target list is a single target: The object is assigned to
     that target.

   * If the target list is a comma-separated list of targets: The
     object must be an iterable with the same number of items as there
     are targets in the target list, and the items are assigned, from
     left to right, to the corresponding targets.  (This rule is
     relaxed as of Python 1.5; in earlier versions, the object had to
     be a tuple.  Since strings are sequences, an assignment like `a, b
     = "xy"' is now legal as long as the string has the right length.)

  Assignment of an object to a single target is recursively defined as
follows.

   * If the target is an identifier (name):

        * If the name does not occur in a global statement
          in the current code block: the name is bound to the object in
          the current local namespace.

        * Otherwise: the name is bound to the object in the current
          global namespace.

     The name is rebound if it was already bound.  This may cause the
     reference count for the object previously bound to the name to
     reach zero, causing the object to be deallocated and its
     destructor (if it has one) to be called.

   * If the target is a target list enclosed in parentheses or in
     square brackets: The object must be an iterable with the same
     number of items as there are targets in the target list, and its
     items are assigned, from left to right, to the corresponding
     targets.

   * If the target is an attribute reference: The primary expression in
     the reference is evaluated.  It should yield an object with
     assignable attributes; if this is not the case, see TypeError
     is raised.  That object is then asked to assign the assigned
     object to the given attribute; if it cannot perform the
     assignment, it raises an exception (usually but not necessarily
     AttributeError).

     Note: If the object is a class instance and the attribute
     reference occurs on both sides of the assignment operator, the RHS
     expression, `a.x' can access either an instance attribute or (if
     no instance attribute exists) a class attribute.  The LHS target
     `a.x' is always set as an instance attribute, creating it if
     necessary.  Thus, the two occurrences of `a.x' do not necessarily
     refer to the same attribute: if the RHS expression refers to a
     class attribute, the LHS creates a new instance attribute as the
     target of the assignment:

         class Cls:
             x = 3             # class variable
         inst = Cls()
         inst.x = inst.x + 1   # writes inst.x as 4 leaving Cls.x as 3

     This description does not necessarily apply to descriptor
     attributes, such as properties created with property().

   * If the target is a subscription: The primary expression in the
     reference is evaluated.  It should yield either a mutable sequence
     object (such as a list) or a mapping object (such as a
     dictionary). Next, the subscript expression is evaluated.

     If the primary is a mutable sequence object (such as a list), the
     subscript must yield a plain integer.  If it is negative, the
     sequence's length is added to it.  The resulting value must be a
     nonnegative integer less than the sequence's length, and the
     sequence is asked to assign the assigned object to its item with
     that index.  If the index is out of range, see IndexError.
     is raised (assignment to a subscripted sequence cannot add new
     items to a list).

     If the primary is a mapping object (such as a dictionary), the
     subscript must have a type compatible with the mapping's key type,
     and the mapping is then asked to create a key/datum pair which
     maps the subscript to the assigned object.  This can either
     replace an existing key/value pair with the same key value, or
     insert a new key/value pair (if no key with the same value
     existed).

   * If the target is a slicing: The primary expression in the
     reference is evaluated.  It should yield a mutable sequence object
     (such as a list).  The assigned object should be a sequence object
     of the same type.  Next, the lower and upper bound expressions are
     evaluated, insofar they are present; defaults are zero and the
     sequence's length.  The bounds should evaluate to (small)
     integers.  If either bound is negative, the sequence's length is
     added to it.  The resulting bounds are clipped to lie between zero
     and the sequence's length, inclusive.  Finally, the sequence
     object is asked to replace the slice with the items of the
     assigned sequence.  The length of the slice may be different from
     the length of the assigned sequence, thus changing the length of
     the target sequence, if the object allows it.

*CPython implementation detail:* In the current implementation, the
syntax for targets is taken to be the same as for expressions, and
invalid syntax is rejected during the code generation phase, causing
less detailed error messages.

  WARNING: Although the definition of assignment implies that overlaps
between the left-hand side and the right-hand side are 'safe' (for
example `a, b = b, a' swaps two variables), overlaps _within_ the
collection of assigned-to variables are not safe!  For instance, the
following program prints `[0, 2]':

    x = [0, 1]
    i = 0
    i, x[i] = 1, 2
    print x

4.6.2.1 Augmented assignment statements
.......................................

Augmented assignment is the combination, in a single statement, of a
binary operation and an assignment statement:

    augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)
    augtarget                 ::= identifier | attributeref | subscription | slicing
    augop                     ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="
              | ">>=" | "<<=" | "&=" | "^=" | "|="

(See section Primaries for the syntax definitions for the
last three symbols.)

  An augmented assignment evaluates the target (which, unlike normal
assignment statements, cannot be an unpacking) and the expression list,
performs the binary operation specific to the type of assignment on the
two operands, and assigns the result to the original target.  The
target is only evaluated once.

  An augmented assignment expression like `x += 1' can be rewritten as
`x = x + 1' to achieve a similar, but not exactly equal effect. In the
augmented version, `x' is only evaluated once. Also, when possible, the
actual operation is performed _in-place_, meaning that rather than
creating a new object and assigning that to the target, the old object
is modified instead.

  With the exception of assigning to tuples and multiple targets in a
single statement, the assignment done by augmented assignment
statements is handled the same way as normal assignments. Similarly,
with the exception of the possible _in-place_ behavior, the binary
operation performed by augmented assignment is the same as the normal
binary operations.

  For targets which are attribute references, the same caveat about class and instance attributes
applies as for regular
assignments.

4.6.3 The `assert' statement
----------------------------

Assert statements are a convenient way to insert debugging assertions
into a program:

    assert_stmt ::= "assert" expression ["," expression]

The simple form, `assert expression', is equivalent to

    if __debug__:
       if not expression: raise AssertionError

The extended form, `assert expression1, expression2', is equivalent to

    if __debug__:
       if not expression1: raise AssertionError(expression2)

These equivalences assume that __debug__ and
AssertionError refer to the built-in variables with those names.
In the current implementation, the built-in variable __debug__
is `True' under normal circumstances, `False' when optimization is
requested (command line option -O).  The current code generator emits
no code for an assert statement when optimization is requested at
compile time.  Note that it is unnecessary to include the source code
for the expression that failed in the error message; it will be
displayed as part of the stack trace.

  Assignments to __debug__ are illegal.  The value for the
built-in variable is determined when the interpreter starts.

4.6.4 The `pass' statement
--------------------------

    pass_stmt ::= "pass"

pass is a null operation -- when it is executed, nothing
happens.  It is useful as a placeholder when a statement is required
syntactically, but no code needs to be executed, for example:

    def f(arg): pass    # a function that does nothing (yet)

    class C: pass       # a class with no methods (yet)

4.6.5 The `del' statement
-------------------------

    del_stmt ::= "del" target_list

Deletion is recursively defined very similar to the way assignment is
defined.  Rather that spelling it out in full details, here are some
hints.

  Deletion of a target list recursively deletes each target, from left
to right.

  Deletion of a name removes the binding of that name  from the local
or global namespace, depending on whether the name occurs in a
global statement in the same code block.  If the name is unbound,
a NameError exception will be raised.

  It is illegal to delete a name from the local namespace if it occurs
as a free variable in a nested block.

  Deletion of attribute references, subscriptions and slicings is
passed to the primary object involved; deletion of a slicing is in
general equivalent to assignment of an empty slice of the right type
(but even this is determined by the sliced object).

4.6.6 The `print' statement
---------------------------

    print_stmt ::= "print" ([expression ("," expression)* [","]]
                   | ">>" expression [("," expression)+ [","]])

print evaluates each expression in turn and writes the
resulting object to standard output (see below).  If an object is not a
string, it is first converted to a string using the rules for string
conversions.  The (resulting or original) string is then written.  A
space is written before each object is (converted and) written, unless
the output system believes it is positioned at the beginning of a line.
This is the case (1) when no characters have yet been written to
standard output, (2) when the last character written to standard output
is a whitespace character except `' '', or (3) when the last write
operation on standard output was not a print statement.
(In some cases it may be functional to write an empty string to
standard output for this reason.)

     Note: Objects which act like file objects but which are not the
     built-in file objects often do not properly emulate this aspect of
     the file object's behavior, so it is best not to rely on this.

  A `'\n'' character is written at the end, unless the print.
statement ends with a comma.  This is the only action if the statement
contains just the keyword print.

  Standard output is defined as the file object named `stdout' in the
built-in module sys.  If no such object exists, or if it
does not have a `write()' method, a RuntimeError exception
is raised.

  print also has an extended form, defined by the second
portion of the syntax described above. This form is sometimes referred
to as "print.  chevron." In this form, the first expression
after the `>>' must evaluate to a "file-like" object, specifically an
object that has a `write()' method as described above.  With this
extended form, the subsequent expressions are printed to this file
object.  If the first expression evaluates to `None', then `sys.stdout'
is used as the file for output.

4.6.7 The `return' statement
----------------------------

    return_stmt ::= "return" [expression_list]

return may only occur syntactically nested in a function
definition, not within a nested class definition.

  If an expression list is present, it is evaluated, else `None' is
substituted.

  return leaves the current function call with the
expression list (or `None') as return value.

  When return passes control out of a try.
statement with a finally clause, that finally.
clause is executed before really leaving the function.

  In a generator function, the return statement is not
allowed to include an expression_list.  In that context, a
bare return.  indicates that the generator is done and will
cause StopIteration to be raised.

4.6.8 The `yield' statement
---------------------------

    yield_stmt ::= yield_expression

The yield statement is only used when defining a generator
function, and is only used in the body of the generator function. Using
a yield.  statement in a function definition is sufficient
to cause that definition to create a generator function instead of a
normal function.

  When a generator function is called, it returns an iterator known as
a generator iterator, or more commonly, a generator.  The body of the
generator function is executed by calling the generator's next()
method repeatedly until it raises an exception.

  When a yield statement is executed, the state of the
generator is frozen and the value of expression_list is
returned to next().'s caller.  By "frozen" we mean that all
local state is retained, including the current bindings of local
variables, the instruction pointer, and the internal evaluation stack:
enough information is saved so that the next time next().
is invoked, the function can proceed exactly as if the yield.
statement were just another external call.

  As of Python version 2.5, the yield statement is now
allowed in the try clause of a try...
finally construct.  If the generator is not resumed before it is
finalized (by reaching a zero reference count or by being garbage
collected), the generator-iterator's `close()' method will be called,
allowing any pending finally.  clauses to execute.

     Note: In Python 2.2, the yield statement was only
     allowed when the `generators' feature has been enabled.  This
     `__future__' import statement was used to enable the feature:

         from __future__ import generators

PEP 0255 (http://www.python.org/dev/peps/pep-0255) - Simple Generators
     The proposal for adding generators and the yield.
     statement to Python.

PEP 0342 (http://www.python.org/dev/peps/pep-0342) - Coroutines via Enhanced Generators
     The proposal that, among other generator enhancements, proposed
     allowing yield to appear inside a try...
     finally block.

4.6.9 The `raise' statement
---------------------------

    raise_stmt ::= "raise" [expression ["," expression ["," expression]]]

If no expressions are present, see raise re-raises the last
exception that was active in the current scope.  If no exception is
active in the current scope, a TypeError exception is
raised indicating that this is an error (if running under IDLE, a
Queue.Empty exception is raised instead).

  Otherwise, see raise evaluates the expressions to get three
objects, using `None' as the value of omitted expressions.  The first
two objects are used to determine the _type_ and _value_ of the
exception.

  If the first object is an instance, the type of the exception is the
class of the instance, the instance itself is the value, and the second
object must be `None'.

  If the first object is a class, it becomes the type of the exception.
The second object is used to determine the exception value: If it is an
instance of the class, the instance becomes the exception value. If the
second object is a tuple, it is used as the argument list for the class
constructor; if it is `None', an empty argument list is used, and any
other object is treated as a single argument to the constructor.  The
instance so created by calling the constructor is used as the exception
value.

  If a third object is present and not `None', it must be a traceback
object (see section The standard type hierarchy), and it is
substituted instead of the current location as the place where the
exception occurred.  If the third object is present and not a traceback
object or `None', a TypeError exception is raised.  The
three-expression form of raise is useful to re-raise an
exception transparently in an except clause, but raise with
no expressions should be preferred if the exception to be re-raised was
the most recently active exception in the current scope.

  Additional information on exceptions can be found in section
Exceptions, and information about handling exceptions is in
section The try statement.

4.6.10 The `break' statement
----------------------------

    break_stmt ::= "break"

break may only occur syntactically nested in a for
or while loop, but not nested in a function or class
definition within that loop.

  It terminates the nearest enclosing loop, skipping the optional
else.  clause if the loop has one.

  If a for loop is terminated by break, the loop
control target keeps its current value.

  When break passes control out of a try.
statement with a finally clause, that finally.
clause is executed before really leaving the loop.

4.6.11 The `continue' statement
-------------------------------

    continue_stmt ::= "continue"

continue may only occur syntactically nested in a
for or while loop, but not nested in a function or
class definition or finally clause within that loop.  It
continues with the next cycle of the nearest enclosing loop.

  When continue passes control out of a try.
statement with a finally clause, that finally.
clause is executed before really starting the next loop cycle.

4.6.12 The `import' statement
-----------------------------

    import_stmt     ::= "import" module ["as" name] ( "," module ["as" name] )*
                    | "from" relative_module "import" identifier ["as" name]
                    ( "," identifier ["as" name] )*
                    | "from" relative_module "import" "(" identifier ["as" name]
                    ( "," identifier ["as" name] )* [","] ")"
                    | "from" module "import" "*"
    module          ::= (identifier ".")* identifier
    relative_module ::= "."* module | "."+
    name            ::= identifier

Import statements are executed in two steps: (1) find a module, and
initialize it if necessary; (2) define a name or names in the local
namespace (of the scope where the import statement occurs).
The statement comes in two forms differing on whether it uses the
from keyword. The first form (without from) repeats
these steps for each identifier in the list.  The form with from
performs step (1) once, and then performs step (2) repeatedly.

  To understand how step (1) occurs, one must first understand how
Python handles hierarchical naming of modules. To help organize modules
and provide a hierarchy in naming, Python has a concept of packages. A
package can contain other packages and modules while modules cannot
contain other modules or packages. From a file system perspective,
packages are directories and modules are files. The original
specification for packages
(http://www.python.org/doc/essays/packages.html)is still available to
read, although minor details have changed since the writing of that
document.

  Once the name of the module is known (unless otherwise specified, the
term "module" will refer to both packages and modules), searching for
the module or package can begin. The first place checked is
sys.modules, the cache of all modules that have been imported
previously. If the module is found there then it is used in step (2) of
import.

  If the module is not found in the cache, then sys.meta_path
is searched (the specification for sys.meta_path can
be found in PEP 302 (http://www.python.org/dev/peps/pep-0302)).  The
object is a list of finder objects which are queried in
order as to whether they know how to load the module by calling their
`find_module()' method with the name of the module. If the module
happens to be contained within a package (as denoted by the existence
of a dot in the name), then a second argument to `find_module()' is
given as the value of the `__path__' attribute from the parent package
(everything up to the last dot in the name of the module being
imported). If a finder can find the module it returns a loader.
(discussed later) or returns `None'.

  If none of the finders on sys.meta_path are able to find
the module then some implicitly defined finders are queried.
Implementations of Python vary in what implicit meta path finders are
defined. The one they all do define, though, is one that handles
sys.path_hooks, sys.path_importer_cache, and
sys.path.

  The implicit finder searches for the requested module in the "paths"
specified in one of two places ("paths" do not have to be file system
paths). If the module being imported is supposed to be contained within
a package then the second argument passed to `find_module()',
`__path__' on the parent package, is used as the source of paths. If
the module is not contained in a package then sys.path is
used as the source of paths.

  Once the source of paths is chosen it is iterated over to find a
finder that can handle that path. The dict at
sys.path_importer_cache caches finders for paths and is checked
for a finder. If the path does not have a finder cached then
sys.path_hooks is searched by calling each object in the list
with a single argument of the path, returning a finder or raises
ImportError. If a finder is returned then it is cached in
sys.path_importer_cache and then used for that path entry. If no
finder can be found but the path exists then a value of `None' is
stored in sys.path_importer_cache to signify that an
implicit, file-based finder that handles modules stored as individual
files should be used for that path. If the path does not exist then a
finder which always returns `None' is placed in the cache for the path.

  If no finder can find the module then ImportError is
raised. Otherwise some finder returned a loader whose `load_module()'
method is called with the name of the module to load (see PEP 302
(http://www.python.org/dev/peps/pep-0302) for the original definition of
loaders). A loader has several responsibilities to perform on a module
it loads. First, if the module already exists in sys.modules.
(a possibility if the loader is called outside of the import
machinery) then it is to use that module for initialization and not a
new module. But if the module does not exist in sys.modules.
then it is to be added to that dict before initialization begins. If an
error occurs during loading of the module and it was added to
sys.modules it is to be removed from the dict. If an error occurs
but the module was already in sys.modules it is left in the
dict.

  The loader must set several attributes on the module. `__name__' is
to be set to the name of the module. `__file__' is to be the "path" to
the file unless the module is built-in (and thus listed in
sys.builtin_module_names) in which case the attribute is not set.
If what is being imported is a package then `__path__' is to be set to a
list of paths to be searched when looking for modules and packages
contained within the package being imported. `__package__' is optional
but should be set to the name of package that contains the module or
package (the empty string is used for module not contained in a
package). `__loader__' is also optional but should be set to the loader
object that is loading the module.

  If an error occurs during loading then the loader raises
ImportError if some other exception is not already being
propagated. Otherwise the loader returns the module that was loaded and
initialized.

  When step (1) finishes without raising an exception, step (2) can
begin.

  The first form of import statement binds the module name
in the local namespace to the module object, and then goes on to import
the next identifier, if any.  If the module name is followed by
as, the name following as is used as the local name
for the module.

  The from form does not bind the module name: it goes
through the list of identifiers, looks each one of them up in the
module found in step (1), and binds the name in the local namespace to
the object thus found.  As with the first form of import, an
alternate local name can be supplied by specifying "as.
localname".  If a name is not found, see ImportError is raised.
If the list of identifiers is replaced by a star (`'*''), all public
names defined in the module are bound in the local namespace of the
import statement..

  The _public names_ defined by a module are determined by checking the
module's namespace for a variable named `__all__'; if defined, it must
be a sequence of strings which are names defined or imported by that
module.  The names given in `__all__' are all considered public and are
required to exist.  If `__all__' is not defined, the set of public
names includes all names found in the module's namespace which do not
begin with an underscore character (`'_'').  `__all__' should contain
the entire public API. It is intended to avoid accidentally exporting
items that are not part of the API (such as library modules which were
imported and used within the module).

  The from form with `*' may only occur in a module scope.
If the wild card form of import -- `import *' -- is used in a function
and the function contains or is a nested block with free variables, the
compiler will raise a SyntaxError.

  When specifying what module to import you do not have to specify the
absolute name of the module. When a module or package is contained
within another package it is possible to make a relative import within
the same top package without having to mention the package name. By
using leading dots in the specified module or package after from
you can specify how high to traverse up the current package
hierarchy without specifying exact names. One leading dot means the
current package where the module making the import exists. Two dots
means up one package level. Three dots is up two levels, etc.  So if
you execute `from . import mod' from a module in the `pkg' package then
you will end up importing `pkg.mod'. If you execute `from ..subpkg2
import mod' from within `pkg.subpkg1' you will import `pkg.subpkg2.mod'.
The specification for relative imports is contained within PEP 328
(http://www.python.org/dev/peps/pep-0328).

  importlib.import_module() is provided to support
applications that determine which modules need to be loaded dynamically.

4.6.12.1 Future statements
..........................

A _future statement_ is a directive to the compiler that a particular
module should be compiled using syntax or semantics that will be
available in a specified future release of Python.  The future
statement is intended to ease migration to future versions of Python
that introduce incompatible changes to the language.  It allows use of
the new features on a per-module basis before the release in which the
feature becomes standard.

    future_statement ::= "from" "__future__" "import" feature ["as" name]
                         ("," feature ["as" name])*
                         | "from" "__future__" "import" "(" feature ["as" name]
                         ("," feature ["as" name])* [","] ")"
    feature          ::= identifier
    name             ::= identifier

A future statement must appear near the top of the module.  The only
lines that can appear before a future statement are:

   * the module docstring (if any),

   * comments,

   * blank lines, and

   * other future statements.

  The features recognized by Python 2.6 are `unicode_literals',
`print_function', `absolute_import', `division', `generators',
`nested_scopes' and `with_statement'.  `generators', `with_statement',
`nested_scopes' are redundant in Python version 2.6 and above because
they are always enabled.

  A future statement is recognized and treated specially at compile
time: Changes to the semantics of core constructs are often implemented
by generating different code.  It may even be the case that a new
feature introduces new incompatible syntax (such as a new reserved
word), in which case the compiler may need to parse the module
differently.  Such decisions cannot be pushed off until runtime.

  For any given release, the compiler knows which feature names have
been defined, and raises a compile-time error if a future statement
contains a feature not known to it.

  The direct runtime semantics are the same as for any import
statement: there is a standard module __future__, described
later, and it will be imported in the usual way at the time the future
statement is executed.

  The interesting runtime semantics depend on the specific feature
enabled by the future statement.

  Note that there is nothing special about the statement:

    import __future__ [as name]

That is not a future statement; it's an ordinary import statement with
no special semantics or syntax restrictions.

  Code compiled by an exec statement or calls to the
built-in functions compile() and execfile().
that occur in a module `M' containing a future statement will, by
default, use the new  syntax or semantics associated with the future
statement.  This can, starting with Python 2.2 be controlled by
optional arguments to compile(). -- see the documentation of
that function for details.

  A future statement typed at an interactive interpreter prompt will
take effect for the rest of the interpreter session.  If an interpreter
is started with the -i option, is passed a script name to
execute, and the script includes a future statement, it will be in
effect in the interactive session started after the script is executed.

PEP 236 (http://www.python.org/dev/peps/pep-0236) - Back to the __future__
     The original proposal for the __future__ mechanism.

4.6.13 The `global' statement
-----------------------------

    global_stmt ::= "global" identifier ("," identifier)*

The global statement is a declaration which holds for the
entire current code block.  It means that the listed identifiers are to
be interpreted as globals.  It would be impossible to assign to a
global variable without global, although free variables may
refer to globals without being declared global.

  Names listed in a global statement must not be used in
the same code block textually preceding that global.
statement.

  Names listed in a global statement must not be defined as
formal parameters or in a for loop control target, see
class.  definition, function definition, or import.
statement.

*CPython implementation detail:* The current implementation does not
enforce the latter two restrictions, but programs should not abuse this
freedom, as future implementations may enforce them or silently change
the meaning of the program.

  *Programmer's note:* the global is a directive to the
parser.  It applies only to code parsed at the same time as the
global statement.  In particular, a global statement
contained in an exec.  statement does not affect the code
block _containing_ the exec.  statement, and code contained
in an exec statement is unaffected by global.
statements in the code containing the exec.  statement.  The
same applies to the eval(), execfile() and
compile() functions.

4.6.14 The `exec' statement
---------------------------

    exec_stmt ::= "exec" or_expr ["in" expression ["," expression]]

This statement supports dynamic execution of Python code.  The first
expression should evaluate to either a string, an open file object, or
a code object.  If it is a string, the string is parsed as a suite of
Python statements which is then executed (unless a syntax error
occurs). (1)  If it is an open file, the file is parsed until EOF and
executed.  If it is a code object, it is simply executed.  In all
cases, the code that's executed is expected to be valid as file input
(see section File input).  Be aware that the return
and yield statements may not be used outside of
function definitions even within the context of code passed to the
exec statement.

  In all cases, if the optional parts are omitted, the code is executed
in the current scope.  If only the first expression after in
is specified, it should be a dictionary, which will be used for
both the global and the local variables.  If two expressions are given,
they are used for the global and local variables, respectively. If
provided, _locals_ can be any mapping object.

  Changed in version 2.4: Formerly, _locals_ was required to be a
dictionary.

  As a side effect, an implementation may insert additional keys into
the dictionaries given besides those corresponding to variable names
set by the executed code.  For example, the current implementation may
add a reference to the dictionary of the built-in module
__builtin__ under the key `__builtins__' (!).

  *Programmer's hints:* dynamic evaluation of expressions is supported
by the built-in function eval().  The built-in functions
globals() and locals() return the current
global and local dictionary, respectively, which may be useful to pass
around for use by exec.

Footnotes
.........

     (1) Note that the parser only accepts the Unix-style end of line
     convention.  If you are reading the code from a file, make sure to
     use universal newline mode to convert Windows or Mac-style
     newlines.

4.7 Compound statements
=======================

Compound statements contain (groups of) other statements; they affect
or control the execution of those other statements in some way.  In
general, compound statements span multiple lines, although in simple
incarnations a whole compound statement may be contained in one line.

  The if, while and for statements
implement traditional control flow constructs.  See try.
specifies exception handlers and/or cleanup code for a group of
statements.  Function and class definitions are also syntactically
compound statements.

  Compound statements consist of one or more 'clauses.'  A clause
consists of a header and a 'suite.'  The clause headers of a particular
compound statement are all at the same indentation level. Each clause
header begins with a uniquely identifying keyword and ends with a
colon.  A suite is a group of statements controlled by a clause.  A
suite can be one or more semicolon-separated simple statements on the
same line as the header, following the header's colon, or it can be one
or more indented statements on subsequent lines.  Only the latter form
of suite can contain nested compound statements; the following is
illegal, mostly because it wouldn't be clear to which if.
clause a following else clause would belong:

    if test1: if test2: print x

Also note that the semicolon binds tighter than the colon in this
context, so that in the following example, either all or none of the
print.  statements are executed:

    if x < y < z: print x; print y; print z

Summarizing:

    compound_stmt ::= if_stmt
                      | while_stmt
                      | for_stmt
                      | try_stmt
                      | with_stmt
                      | funcdef
                      | classdef
                      | decorated
    suite         ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
    statement     ::= stmt_list NEWLINE | compound_stmt
    stmt_list     ::= simple_stmt (";" simple_stmt)* [";"]

Note that statements always end in a `NEWLINE' possibly followed by a
`DEDENT'. Also note that optional continuation clauses always begin
with a keyword that cannot start a statement, thus there are no
ambiguities (the 'dangling else.' problem is solved in
Python by requiring nested if statements to be indented).

  The formatting of the grammar rules in the following sections places
each clause on a separate line for clarity.

4.7.1 The `if' statement
------------------------

The if statement is used for conditional execution:

    if_stmt ::= "if" expression ":" suite
                ( "elif" expression ":" suite )*
                ["else" ":" suite]

It selects exactly one of the suites by evaluating the expressions one
by one until one is found to be true (see section Boolean operations
for the definition of true and false); then that suite
is executed (and no other part of the if statement is
executed or evaluated).  If all expressions are false, the suite of the
else clause, if present, is executed.

4.7.2 The `while' statement
---------------------------

The while statement is used for repeated execution as long
as an expression is true:

    while_stmt ::= "while" expression ":" suite
                   ["else" ":" suite]

This repeatedly tests the expression and, if it is true, executes the
first suite; if the expression is false (which may be the first time it
is tested) the suite of the else clause, if present, is
executed and the loop terminates.

  A break statement executed in the first suite terminates
the loop without executing the else clause's suite.  A
continue.  statement executed in the first suite skips the
rest of the suite and goes back to testing the expression.

4.7.3 The `for' statement
-------------------------

The for statement is used to iterate over the elements of a
sequence (such as a string, tuple or list) or other iterable object:

    for_stmt ::= "for" target_list "in" expression_list ":" suite
                 ["else" ":" suite]

The expression list is evaluated once; it should yield an iterable
object.  An iterator is created for the result of the
`expression_list'.  The suite is then executed once for each item
provided by the iterator, in the order of ascending indices.  Each item
in turn is assigned to the target list using the standard rules for
assignments, and then the suite is executed.  When the items are
exhausted (which is immediately when the sequence is empty), the suite
in the else clause, if present, is executed, and the loop
terminates.

  A break statement executed in the first suite terminates
the loop without executing the else clause's suite.  A
continue.  statement executed in the first suite skips the
rest of the suite and continues with the next item, or with the
else clause if there was no next item.

  The suite may assign to the variable(s) in the target list; this does
not affect the next item assigned to it.

  The target list is not deleted when the loop is finished, but if the
sequence is empty, it will not have been assigned to at all by the
loop.  Hint: the built-in function range() returns a
sequence of integers suitable to emulate the effect of Pascal's `for i
:= a to b do'; e.g., `range(3)' returns the list `[0, 1, 2]'.

     Note: There is a subtlety when the sequence is being modified by
     the loop (this can only occur for mutable sequences, i.e. lists).
     An internal counter is used to keep track of which item is used
     next, and this is incremented on each iteration.  When this
     counter has reached the length of the sequence the loop
     terminates.  This means that if the suite deletes the current (or
     a previous) item from the sequence, the next item will be skipped
     (since it gets the index of the current item which has already
     been treated).  Likewise, if the suite inserts an item in the
     sequence before the current item, the current item will be treated
     again the next time through the loop. This can lead to nasty bugs
     that can be avoided by making a temporary copy using a slice of
     the whole sequence, e.g.,

         for x in a[:]:
             if x < 0: a.remove(x)

4.7.4 The `try' statement
-------------------------

The try statement specifies exception handlers and/or
cleanup code for a group of statements:

    try_stmt  ::= try1_stmt | try2_stmt
    try1_stmt ::= "try" ":" suite
                  ("except" [expression [("as" | ",") target]] ":" suite)+
                  ["else" ":" suite]
                  ["finally" ":" suite]
    try2_stmt ::= "try" ":" suite
                  "finally" ":" suite

Changed in version 2.5: In previous versions of Python, see try...
See except...finally did not work. See try...
except had to be nested in try...See
finally.

  The except clause(s) specify one or more exception
handlers. When no exception occurs in the try clause, no
exception handler is executed.  When an exception occurs in the
try suite, a search for an exception handler is started.  This
search inspects the except clauses in turn until one is found that
matches the exception.  An expression-less except clause, if present,
must be last; it matches any exception.  For an except clause with an
expression, that expression is evaluated, and the clause matches the
exception if the resulting object is "compatible" with the exception.
An object is compatible with an exception if it is the class or a base
class of the exception object, a tuple containing an item compatible
with the exception, or, in the (deprecated) case of string exceptions,
is the raised string itself (note that the object identities must
match, i.e. it must be the same string object, not just a string with
the same value).

  If no except clause matches the exception, the search for an
exception handler continues in the surrounding code and on the
invocation stack.  (1)

  If the evaluation of an expression in the header of an except clause
raises an exception, the original search for a handler is canceled and
a search starts for the new exception in the surrounding code and on
the call stack (it is treated as if the entire try.
statement raised the exception).

  When a matching except clause is found, the exception is assigned to
the target specified in that except clause, if present, and the except
clause's suite is executed.  All except clauses must have an executable
block.  When the end of this block is reached, execution continues
normally after the entire try statement.  (This means that if two
nested handlers exist for the same exception, and the exception occurs
in the try clause of the inner handler, the outer handler will not
handle the exception.)

  Before an except clause's suite is executed, details about the
exception are assigned to three variables in the sys.
module: `sys.exc_type' receives the object identifying the exception;
`sys.exc_value' receives the exception's parameter; `sys.exc_traceback'
receives a traceback object (see section The standard type hierarchy)
identifying the point in the program where the
exception occurred. These details are also available through the
sys.exc_info().  function, which returns a tuple `(exc_type,
exc_value, exc_traceback)'.  Use of the corresponding variables is
deprecated in favor of this function, since their use is unsafe in a
threaded program.  As of Python 1.5, the variables are restored to
their previous values (before the call) when returning from a function
that handled an exception.

  The optional else clause is executed if and when control
flows off the end of the try clause. (2) Exceptions in the
else.  clause are not handled by the preceding except
clauses.

  If finally is present, it specifies a 'cleanup' handler.
The try clause is executed, including any except
and else clauses.  If an exception occurs in any of
the clauses and is not handled, the exception is temporarily saved. The
finally clause is executed.  If there is a saved exception,
it is re-raised at the end of the finally clause. If the
finally clause raises another exception or executes a
return or break statement, the saved exception is
lost.  The exception information is not available to the program during
execution of the finally clause.

  When a return, break or continue.
statement is executed in the try suite of a try...
See finally.  statement, the finally clause is
also executed 'on the way out.' A continue statement is
illegal in the finally clause. (The reason is a problem
with the current implementation -- this restriction may be lifted in
the future).

  Additional information on exceptions can be found in section
Exceptions, and information on using the raise.
statement to generate exceptions may be found in section The raise statement.

4.7.5 The `with' statement
--------------------------

New in version 2.5.

  The with statement is used to wrap the execution of a
block with methods defined by a context manager (see section With Statement Context Managers).
This allows common try...
See except...finally usage patterns to be
encapsulated for convenient reuse.

    with_stmt ::= "with" with_item ("," with_item)* ":" suite
    with_item ::= expression ["as" target]

The execution of the with statement with one "item"
proceeds as follows:

  1. The context expression (the expression given in the
     with_item) is evaluated to obtain a context manager.

  2. The context manager's __exit__() is loaded for later
     use.

  3. The context manager's __enter__() method is invoked.

  4. If a target was included in the with statement, the
     return value from __enter__() is assigned to it.

          Note: The with statement guarantees that if the
          __enter__() method returns without an error, then
          __exit__() will always be called. Thus, if an
          error occurs during the assignment to the target list, it
          will be treated the same as an error occurring within the
          suite would be. See step 6 below.

  5. The suite is executed.

  6. The context manager's __exit__() method is invoked. If
     an exception caused the suite to be exited, its type, value, and
     traceback are passed as arguments to __exit__().
     Otherwise, three None arguments are supplied.

     If the suite was exited due to an exception, and the return value
     from the __exit__() method was false, the exception is
     reraised. If the return value was true, the exception is
     suppressed, and execution continues with the statement following
     the with statement.

     If the suite was exited for any reason other than an exception,
     the return value from __exit__() is ignored, and
     execution proceeds at the normal location for the kind of exit
     that was taken.

  With more than one item, the context managers are processed as if
multiple with statements were nested:

    with A() as a, B() as b:
        suite

is equivalent to

    with A() as a:
        with B() as b:
            suite

     Note: In Python 2.5, the with statement is only
     allowed when the `with_statement' feature has been enabled.  It is
     always enabled in Python 2.6.

  Changed in version 2.7: Support for multiple context expressions.

PEP 0343 (http://www.python.org/dev/peps/pep-0343) - The "with" statement
     The specification, background, and examples for the Python
     with.  statement.

4.7.6 Function definitions
--------------------------

A function definition defines a user-defined function object (see
section The standard type hierarchy):

    decorated      ::= decorators (classdef | funcdef)
    decorators     ::= decorator+
    decorator      ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
    funcdef        ::= "def" funcname "(" [parameter_list] ")" ":" suite
    dotted_name    ::= identifier ("." identifier)*
    parameter_list ::= (defparameter ",")*
                       (  "*" identifier [, "**" identifier]
                       | "**" identifier
                       | defparameter [","] )
    defparameter   ::= parameter ["=" expression]
    sublist        ::= parameter ("," parameter)* [","]
    parameter      ::= identifier | "(" sublist ")"
    funcname       ::= identifier

A function definition is an executable statement.  Its execution binds
the function name in the current local namespace to a function object
(a wrapper around the executable code for the function).  This function
object contains a reference to the current global namespace as the
global namespace to be used when the function is called.

  The function definition does not execute the function body; this gets
executed only when the function is called. (3)

  A function definition may be wrapped by one or more decorator
expressions.  Decorator expressions are evaluated when the
function is defined, in the scope that contains the function
definition.  The result must be a callable, which is invoked with the
function object as the only argument. The returned value is bound to
the function name instead of the function object.  Multiple decorators
are applied in nested fashion. For example, the following code:

    @f1(arg)
    @f2
    def func(): pass

is equivalent to:

    def func(): pass
    func = f1(arg)(f2(func))

When one or more top-level parameters have the form _parameter_ `='
_expression_, the function is said to have "default parameter values."
For a parameter with a default value, the corresponding argument may be
omitted from a call, in which case the parameter's default value is
substituted.  If a parameter has a default value, all following
parameters must also have a default value -- this is a syntactic
restriction that is not expressed by the grammar.

  *Default parameter values are evaluated when the function definition
is executed.*  This means that the expression is evaluated once, when
the function is defined, and that that same "pre-computed" value is
used for each call.  This is especially important to understand when a
default parameter is a mutable object, such as a list or a dictionary:
if the function modifies the object (e.g. by appending an item to a
list), the default value is in effect modified.  This is generally not
what was intended.  A way around this  is to use `None' as the default,
and explicitly test for it in the body of the function, e.g.:

    def whats_on_the_telly(penguin=None):
        if penguin is None:
            penguin = []
        penguin.append("property of the zoo")
        return penguin

Function call semantics are described in more detail in section
Calls. A function call always assigns values to all parameters
mentioned in the parameter list, either from position arguments, from
keyword arguments, or from default values.  If the form "`*identifier'"
is present, it is initialized to a tuple receiving any excess
positional parameters, defaulting to the empty tuple.  If the form
"`**identifier'" is present, it is initialized to a new dictionary
receiving any excess keyword arguments, defaulting to a new empty
dictionary.

  It is also possible to create anonymous functions (functions not
bound to a name), for immediate use in expressions.  This uses lambda
forms, described in section Lambdas.  Note that the lambda
form is merely a shorthand for a simplified function definition; a
function defined in a "def."  statement can be passed around
or assigned to another name just like a function defined by a lambda
form.  The "def." form is actually more powerful since it
allows the execution of multiple statements.

  *Programmer's note:* Functions are first-class objects.  A "`def'"
form executed inside a function definition defines a local function
that can be returned or passed around.  Free variables used in the
nested function can access the local variables of the function
containing the def.  See section Naming and binding for
details.

4.7.7 Class definitions
-----------------------

A class definition defines a class object (see section The standard type hierarchy):

    classdef    ::= "class" classname [inheritance] ":" suite
    inheritance ::= "(" [expression_list] ")"
    classname   ::= identifier

A class definition is an executable statement.  It first evaluates the
inheritance list, if present.  Each item in the inheritance list should
evaluate to a class object or class type which allows subclassing.  The
class's suite is then executed in a new execution frame (see section
Naming and binding), using a newly created local namespace
and the original global namespace. (Usually, the suite contains only
function definitions.)  When the class's suite finishes execution, its
execution frame is discarded but its local namespace is saved. (4) A
class object is then created using the inheritance list for the base
classes and the saved local namespace for the attribute dictionary.  The
class name is bound to this class object in the original local
namespace.

  *Programmer's note:* Variables defined in the class definition are
class variables; they are shared by all instances.  To create instance
variables, they can be set in a method with `self.name = value'.  Both
class and instance variables are accessible through the notation
"`self.name'", and an instance variable hides a class variable with the
same name when accessed in this way.  Class variables can be used as
defaults for instance variables, but using mutable values there can
lead to unexpected results.  For new-style class.es,
descriptors can be used to create instance variables with different
implementation details.

  Class definitions, like function definitions, may be wrapped by one
or more decorator expressions.  The evaluation rules for
the decorator expressions are the same as for functions.  The result
must be a class object, which is then bound to the class name.

Footnotes
.........

     (1) The exception is propagated to the invocation stack only if
     there is no finally clause that negates the exception.

     (2) Currently, control "flows off the end" except in the case of
     an exception or the execution of a return,
     continue, or break.  statement.

     (3) A string literal appearing as the first statement in the
     function body is transformed into the function's `__doc__'
     attribute and therefore the function's docstring.

     (4) A string literal appearing as the first statement in the class
     body is transformed into the namespace's `__doc__' item and
     therefore the class's docstring.

4.8 Top-level components
========================

The Python interpreter can get its input from a number of sources: from
a script passed to it as standard input or as program argument, typed
in interactively, from a module source file, etc.  This chapter gives
the syntax used in these cases.

4.8.1 Complete Python programs
------------------------------

While a language specification need not prescribe how the language
interpreter is invoked, it is useful to have a notion of a complete
Python program.  A complete Python program is executed in a minimally
initialized environment: all built-in and standard modules are
available, but none have been initialized, except for sys.
(various system services), see __builtin__. (built-in functions,
exceptions and `None') and __main__.  The latter is used to
provide the local and global namespace for execution of the complete
program.

  The syntax for a complete Python program is that for file input,
described in the next section.

  The interpreter may also be invoked in interactive mode; in this
case, it does not read and execute a complete program but reads and
executes one statement (possibly compound) at a time.  The initial
environment is identical to that of a complete program; each statement
is executed in the namespace of __main__.

  Under Unix, a complete program can be passed to the interpreter in
three forms: with the -c. _string_ command line option, as a
file passed as the first command line argument, or as standard input.
If the file or standard input is a tty device, the interpreter enters
interactive mode; otherwise, it executes the file as a complete program.

4.8.2 File input
----------------

All input read from non-interactive files has the same form:

    file_input ::= (NEWLINE | statement)*

This syntax is used in the following situations:

   * when parsing a complete Python program (from a file or from a
     string);

   * when parsing a module;

   * when parsing a string passed to the exec statement;

4.8.3 Interactive input
-----------------------

Input in interactive mode is parsed using the following grammar:

    interactive_input ::= [stmt_list] NEWLINE | compound_stmt NEWLINE

Note that a (top-level) compound statement must be followed by a blank
line in interactive mode; this is needed to help the parser detect the
end of the input.

4.8.4 Expression input
----------------------

There are two forms of expression input.  Both ignore leading
whitespace. The string argument to eval() must have the
following form:

    eval_input ::= expression_list NEWLINE*

The input line read by input() must have the following form:

    input_input ::= expression_list NEWLINE

Note: to read 'raw' input line without interpretation, you can use the
built-in function raw_input() or the readline().
method of file objects.

4.9 Full Grammar specification
==============================

This is the full Python grammar, as it is read by the parser generator
and used to parse Python source files:

    # Grammar for Python

    # Note:  Changing the grammar specified in this file will most likely
    #        require corresponding changes in the parser module
    #        (../Modules/parsermodule.c).  If you can't make the changes to
    #        that module yourself, please co-ordinate the required changes
    #        with someone who can; ask around on python-dev for help.  Fred
    #        Drake <fdrake@acm.org> will probably be listening there.

    # NOTE WELL: You should also follow all the steps listed in PEP 306,
    # "How to Change Python's Grammar"

    # Start symbols for the grammar:
    #       single_input is a single interactive statement;
    #       file_input is a module or sequence of commands read from an input file;
    #       eval_input is the input for the eval() and input() functions.
    # NB: compound_stmt in single_input is followed by extra NEWLINE!
    single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
    file_input: (NEWLINE | stmt)* ENDMARKER
    eval_input: testlist NEWLINE* ENDMARKER

    decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
    decorators: decorator+
    decorated: decorators (classdef | funcdef)
    funcdef: 'def' NAME parameters ':' suite
    parameters: '(' [varargslist] ')'
    varargslist: ((fpdef ['=' test] ',')*
                  ('*' NAME [',' '**' NAME] | '**' NAME) |
                  fpdef ['=' test] (',' fpdef ['=' test])* [','])
    fpdef: NAME | '(' fplist ')'
    fplist: fpdef (',' fpdef)* [',']

    stmt: simple_stmt | compound_stmt
    simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
    small_stmt: (expr_stmt | print_stmt  | del_stmt | pass_stmt | flow_stmt |
                 import_stmt | global_stmt | exec_stmt | assert_stmt)
    expr_stmt: testlist (augassign (yield_expr|testlist) |
                         ('=' (yield_expr|testlist))*)
    augassign: ('+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' |
                '<<=' | '>>=' | '**=' | '//=')
    # For normal assignments, additional restrictions enforced by the interpreter
    print_stmt: 'print' ( [ test (',' test)* [','] ] |
                          '>>' test [ (',' test)+ [','] ] )
    del_stmt: 'del' exprlist
    pass_stmt: 'pass'
    flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt
    break_stmt: 'break'
    continue_stmt: 'continue'
    return_stmt: 'return' [testlist]
    yield_stmt: yield_expr
    raise_stmt: 'raise' [test [',' test [',' test]]]
    import_stmt: import_name | import_from
    import_name: 'import' dotted_as_names
    import_from: ('from' ('.'* dotted_name | '.'+)
                  'import' ('*' | '(' import_as_names ')' | import_as_names))
    import_as_name: NAME ['as' NAME]
    dotted_as_name: dotted_name ['as' NAME]
    import_as_names: import_as_name (',' import_as_name)* [',']
    dotted_as_names: dotted_as_name (',' dotted_as_name)*
    dotted_name: NAME ('.' NAME)*
    global_stmt: 'global' NAME (',' NAME)*
    exec_stmt: 'exec' expr ['in' test [',' test]]
    assert_stmt: 'assert' test [',' test]

    compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
    if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
    while_stmt: 'while' test ':' suite ['else' ':' suite]
    for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
    try_stmt: ('try' ':' suite
               ((except_clause ':' suite)+
                ['else' ':' suite]
                ['finally' ':' suite] |
               'finally' ':' suite))
    with_stmt: 'with' with_item (',' with_item)*  ':' suite
    with_item: test ['as' expr]
    # NB compile.c makes sure that the default except clause is last
    except_clause: 'except' [test [('as' | ',') test]]
    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

    # Backward compatibility cruft to support:
    # [ x for x in lambda: True, lambda: False if x() ]
    # even while also allowing:
    # lambda x: 5 if x else 2
    # (But not a mix of the two)
    testlist_safe: old_test [(',' old_test)+ [',']]
    old_test: or_test | old_lambdef
    old_lambdef: 'lambda' [varargslist] ':' old_test

    test: or_test ['if' or_test 'else' test] | lambdef
    or_test: and_test ('or' and_test)*
    and_test: not_test ('and' not_test)*
    not_test: 'not' not_test | comparison
    comparison: expr (comp_op expr)*
    comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'
    expr: xor_expr ('|' xor_expr)*
    xor_expr: and_expr ('^' and_expr)*
    and_expr: shift_expr ('&' shift_expr)*
    shift_expr: arith_expr (('<<'|'>>') arith_expr)*
    arith_expr: term (('+'|'-') term)*
    term: factor (('*'|'/'|'%'|'//') factor)*
    factor: ('+'|'-'|'~') factor | power
    power: atom trailer* ['**' factor]
    atom: ('(' [yield_expr|testlist_comp] ')' |
           '[' [listmaker] ']' |
           '{' [dictorsetmaker] '}' |
           '`' testlist1 '`' |
           NAME | NUMBER | STRING+)
    listmaker: test ( list_for | (',' test)* [','] )
    testlist_comp: test ( comp_for | (',' test)* [','] )
    lambdef: 'lambda' [varargslist] ':' test
    trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
    subscriptlist: subscript (',' subscript)* [',']
    subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]
    sliceop: ':' [test]
    exprlist: expr (',' expr)* [',']
    testlist: test (',' test)* [',']
    dictmaker: test ':' test (',' test ':' test)* [',']
    dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
                      (test (comp_for | (',' test)* [','])) )

    classdef: 'class' NAME ['(' [testlist] ')'] ':' suite

    arglist: (argument ',')* (argument [',']
                             |'*' test (',' argument)* [',' '**' test]
                             |'**' test)
    # The reason that keywords are test nodes instead of NAME is that using NAME
    # results in an ambiguity. ast.c makes sure it's a NAME.
    argument: test [comp_for] | test '=' test

    list_iter: list_for | list_if
    list_for: 'for' exprlist 'in' testlist_safe [list_iter]
    list_if: 'if' old_test [list_iter]

    comp_iter: comp_for | comp_if
    comp_for: 'for' exprlist 'in' or_test [comp_iter]
    comp_if: 'if' old_test [comp_iter]

    testlist1: test (',' test)*

    # not used in grammar, but may appear in "node" passed from Parser to Compiler
    encoding_decl: NAME

    yield_expr: 'yield' [testlist]

5 The Python Standard Library
*****************************

     Release: 2.7

     Date: February 08, 2011

  While The Python Language Reference describes the exact
syntax and semantics of the Python language, this library reference
manual describes the standard library that is distributed with Python.
It also describes some of the optional components that are commonly
included in Python distributions.

  Python's standard library is very extensive, offering a wide range of
facilities as indicated by the long table of contents listed below. The
library contains built-in modules (written in C) that provide access to
system functionality such as file I/O that would otherwise be
inaccessible to Python programmers, as well as modules written in Python
that provide standardized solutions for many problems that occur in
everyday programming. Some of these modules are explicitly designed to
encourage and enhance the portability of Python programs by abstracting
away platform-specifics into platform-neutral APIs.

  The Python installers for the Windows platform usually includes the
entire standard library and often also include many additional
components. For Unix-like operating systems Python is normally provided
as a collection of packages, so it may be necessary to use the packaging
tools provided with the operating system to obtain some or all of the
optional components.

  In addition to the standard library, there is a growing collection of
several thousand components (from individual programs and modules to
packages and entire application development frameworks), available from
the Python Package Index (http://pypi.python.org/pypi).

5.1 Introduction
================

The "Python library" contains several different kinds of components.

  It contains data types that would normally be considered part of the
"core" of a language, such as numbers and lists.  For these types, the
Python language core defines the form of literals and places some
constraints on their semantics, but does not fully define the
semantics.  (On the other hand, the language core does define syntactic
properties like the spelling and priorities of operators.)

  The library also contains built-in functions and exceptions --
objects that can be used by all Python code without the need of an
import statement.  Some of these are defined by the core
language, but many are not essential for the core semantics and are
only described here.

  The bulk of the library, however, consists of a collection of
modules. There are many ways to dissect this collection.  Some modules
are written in C and built in to the Python interpreter; others are
written in Python and imported in source form.  Some modules provide
interfaces that are highly specific to Python, like printing a stack
trace; some provide interfaces that are specific to particular
operating systems, such as access to specific hardware; others provide
interfaces that are specific to a particular application domain, like
the World Wide Web. Some modules are available in all versions and
ports of Python; others are only available when the underlying system
supports or requires them; yet others are available only when a
particular configuration option was chosen at the time when Python was
compiled and installed.

  This manual is organized "from the inside out:" it first describes
the built-in data types, then the built-in functions and exceptions,
and finally the modules, grouped in chapters of related modules.  The
ordering of the chapters as well as the ordering of the modules within
each chapter is roughly from most relevant to least important.

  This means that if you start reading this manual from the start, and
skip to the next chapter when you get bored, you will get a reasonable
overview of the available modules and application areas that are
supported by the Python library.  Of course, you don't _have_ to read
it like a novel -- you can also browse the table of contents (in front
of the manual), or look for a specific function, module or term in the
index (in the back).  And finally, if you enjoy learning about random
subjects, you choose a random page number (see module random)
and read a section or two.  Regardless of the order in which you
read the sections of this manual, it helps to start with chapter
Built-in Functions, as the remainder of the manual assumes
familiarity with this material.

  Let the show begin!

5.2 Built-in Functions
======================

The Python interpreter has a number of functions built into it that are
always available.  They are listed here in alphabetical order.

                                              Built-in Functions
--------------------------------------------------------------------------------------------------------------------
see abs().       See divmod().  See input().    See open().    See staticmethod().
See all().       See enumerate().      See ord().     See str().
                        418.
any().       See eval().    See isinstance().     See sum().
                                              31b.
See basestring().   See super().
44b.                    427.                  31c.
See bin().       See file().    See iter().     See property().
                                                                     476.
bool().      See filter().  See len().      See range().   See type().
See bytearray(). See float().   See list().     See raw_input().
                                                                     4e3.
See callable().  See format().  See locals().   See reduce().  See unicode().
See chr().       See frozenset().     See reload().  See vars().
                        36d.
See classmethod(). See map().      See repr().    See xrange().
3ed.
See cmp().       See globals(). See max().      See reversed().
                                                                     3ef.
See compile().   See hasattr(). See memoryview().   See __import__().
                                              1be.
See complex().   See hash().    See min().      See set().     See apply().
See delattr().   See help().    See next().     See setattr(). See buffer().
See dict().      See hex().     See object().   See slice().   See coerce().
See dir().       See id().      See oct().      See sorted().  See intern().

 -- Function: abs (x)
     Return the absolute value of a number.  The argument may be a
     plain or long integer or a floating point number.  If the argument
     is a complex number, its magnitude is returned.

 -- Function: all (iterable)
     Return True if all elements of the _iterable_ are true (or if the
     iterable is empty).  Equivalent to:

         def all(iterable):
             for element in iterable:
                 if not element:
                     return False
             return True

     New in version 2.5.

 -- Function: any (iterable)
     Return True if any element of the _iterable_ is true.  If the
     iterable is empty, return False.  Equivalent to:

         def any(iterable):
             for element in iterable:
                 if element:
                     return True
             return False

     New in version 2.5.

 -- Function: basestring ()
     This abstract type is the superclass for str and
     unicode. It cannot be called or instantiated, but it can be
     used to test whether an object is an instance of str.
     or unicode. `isinstance(obj, basestring)' is equivalent
     to `isinstance(obj, (str, unicode))'.

     New in version 2.3.

 -- Function: bin (x)
     Convert an integer number to a binary string. The result is a
     valid Python expression.  If _x_ is not a Python int.
     object, it has to define an __index__() method that
     returns an integer.

     New in version 2.6.

 -- Function: bool ([x])
     Convert a value to a Boolean, using the standard truth testing
     procedure.  If _x_ is false or omitted, this returns False.;
     otherwise it returns True. bool is
     also a class, which is a subclass of int. Class
     bool cannot be subclassed further.  Its only instances are
     False and True.

     New in version 2.2.1.

     Changed in version 2.3: If no argument is given, this function
     returns False.

 -- Function: bytearray ([source[, encoding[, errors]]])
     Return a new array of bytes.  The bytearray type is a
     mutable sequence of integers in the range 0 <= x < 256.  It has
     most of the usual methods of mutable sequences, described in
     Mutable Sequence Types, as well as most methods that the
     str type has, see String Methods.

     The optional _source_ parameter can be used to initialize the
     array in a few different ways:

        * If it is a _string_, you must also give the _encoding_ (and
          optionally, _errors_) parameters; see bytearray().
          then converts the string to bytes using str.encode().

        * If it is an _integer_, the array will have that size and will
          be initialized with null bytes.

        * If it is an object conforming to the _buffer_ interface, a
          read-only buffer of the object will be used to initialize the
          bytes array.

        * If it is an _iterable_, it must be an iterable of integers in
          the range `0 <= x < 256', which are used as the initial
          contents of the array.

     Without an argument, an array of size 0 is created.

 -- Function: callable (object)
     Return True if the _object_ argument appears callable,
     see False if not.  If this returns true, it is still
     possible that a call fails, but if it is false, calling _object_
     will never succeed.  Note that classes are callable (calling a
     class returns a new instance); class instances are callable if
     they have a __call__() method.

 -- Function: chr (i)
     Return a string of one character whose ASCII code is the integer
     _i_.  For example, `chr(97)' returns the string `'a''. This is the
     inverse of ord().  The argument must be in the range
     [0..255], inclusive; see ValueError will be raised if _i_
     is outside that range. See also unichr().

 -- Function: classmethod (function)
     Return a class method for _function_.

     A class method receives the class as implicit first argument, just
     like an instance method receives the instance. To declare a class
     method, use this idiom:

         class C:
             @classmethod
             def f(cls, arg1, arg2, ...): ...

     The `@classmethod' form is a function decorator. - see
     the description of function definitions in Function definitions
     for details.

     It can be called either on the class (such as `C.f()') or on an
     instance (such as `C().f()').  The instance is ignored except for
     its class. If a class method is called for a derived class, the
     derived class object is passed as the implied first argument.

     Class methods are different than C++ or Java static methods. If
     you want those, see staticmethod() in this section.

     For more information on class methods, consult the documentation
     on the standard type hierarchy in The standard type hierarchy.

     New in version 2.2.

     Changed in version 2.4: Function decorator syntax added.

 -- Function: cmp (x, y)
     Compare the two objects _x_ and _y_ and return an integer
     according to the outcome.  The return value is negative if `x <
     y', zero if `x == y' and strictly positive if `x > y'.

 -- Function: compile (source, filename, mode[, flags[, dont_inherit]])
     Compile the _source_ into a code or AST object.  Code objects can
     be executed by an exec statement or evaluated by a
     call to eval().  _source_ can either be a string or an
     AST object.  Refer to the ast.  module documentation for
     information on how to work with AST objects.

     The _filename_ argument should give the file from which the code
     was read; pass some recognizable value if it wasn't read from a
     file (`'<string>'' is commonly used).

     The _mode_ argument specifies what kind of code must be compiled;
     it can be `'exec'' if _source_ consists of a sequence of
     statements, `'eval'' if it consists of a single expression, or
     `'single'' if it consists of a single interactive statement (in
     the latter case, expression statements that evaluate to something
     other than `None' will be printed).

     The optional arguments _flags_ and _dont_inherit_ control which
     future statements (see PEP 236
     (http://www.python.org/dev/peps/pep-0236)) affect the compilation
     of _source_.  If neither is present (or both are zero) the code is
     compiled with those future statements that are in effect in the
     code that is calling compile.  If the _flags_ argument is given
     and _dont_inherit_ is not (or is zero) then the future statements
     specified by the _flags_ argument are used in addition to those
     that would be used anyway. If _dont_inherit_ is a non-zero integer
     then the _flags_ argument is it - the future statements in effect
     around the call to compile are ignored.

     Future statements are specified by bits which can be bitwise ORed
     together to specify multiple statements.  The bitfield required to
     specify a given feature can be found as the `compiler_flag'
     attribute on the `_Feature' instance in the __future__.
     module.

     This function raises SyntaxError if the compiled
     source is invalid, and TypeError if the source
     contains null bytes.

          Note: When compiling a string with multi-line code in
          `'single'' or `'eval'' mode, input must be terminated by at
          least one newline character.  This is to facilitate detection
          of incomplete and complete statements in the code.
          module.

     Changed in version 2.3: The _flags_ and _dont_inherit_ arguments
     were added.

     Changed in version 2.6: Support for compiling AST objects.

     Changed in version 2.7: Allowed use of Windows and Mac newlines.
     Also input in `'exec'' mode does not have to end in a newline
     anymore.

 -- Function: complex ([real[, imag]])
     Create a complex number with the value _real_ + _imag_*j or
     convert a string or number to a complex number.  If the first
     parameter is a string, it will be interpreted as a complex number
     and the function must be called without a second parameter.  The
     second parameter can never be a string. Each argument may be any
     numeric type (including complex). If _imag_ is omitted, it
     defaults to zero and the function serves as a numeric conversion
     function like int(), long() and
     float().  If both arguments are omitted, returns `0j'.

     The complex type is described in Numeric Types -- int, float, long, complex.

 -- Function: delattr (object, name)
     This is a relative of setattr().  The arguments are an
     object and a string.  The string must be the name of one of the
     object's attributes.  The function deletes the named attribute,
     provided the object allows it.  For example, `delattr(x,
     'foobar')' is equivalent to `del x.foobar'.

 -- Function: dict ([arg])
     Create a new data dictionary, optionally with items taken from
     _arg_.  The dictionary type is described in Mapping Types -- dict.

     For other containers see the built in list, set,
     and tuple classes, and the collections.
     module.

 -- Function: dir ([object])
     Without arguments, return the list of names in the current local
     scope.  With an argument, attempt to return a list of valid
     attributes for that object.

     If the object has a method named `__dir__()', this method will be
     called and must return the list of attributes. This allows objects
     that implement a custom __getattr__() or
     __getattribute__() function to customize the way
     dir() reports their attributes.

     If the object does not provide `__dir__()', the function tries its
     best to gather information from the object's `__dict__' attribute,
     if defined, and from its type object.  The resulting list is not
     necessarily complete, and may be inaccurate when the object has a
     custom __getattr__().

     The default dir() mechanism behaves differently with
     different types of objects, as it attempts to produce the most
     relevant, rather than complete, information:

        * If the object is a module object, the list contains the names
          of the module's attributes.

        * If the object is a type or class object, the list contains
          the names of its attributes, and recursively of the
          attributes of its bases.

        * Otherwise, the list contains the object's attributes' names,
          the names of its class's attributes, and recursively of the
          attributes of its class's base classes.

     The resulting list is sorted alphabetically.  For example:

         >>> import struct
         >>> dir()   # doctest: +SKIP
         ['__builtins__', '__doc__', '__name__', 'struct']
         >>> dir(struct)   # doctest: +NORMALIZE_WHITESPACE
         ['Struct', '__builtins__', '__doc__', '__file__', '__name__',
          '__package__', '_clearcache', 'calcsize', 'error', 'pack', 'pack_into',
          'unpack', 'unpack_from']
         >>> class Foo(object):
         ...     def __dir__(self):
         ...         return ["kan", "ga", "roo"]
         ...
         >>> f = Foo()
         >>> dir(f)
         ['ga', 'kan', 'roo']

          Note: Because dir() is supplied primarily as a
          convenience for use at an interactive prompt, it tries to
          supply an interesting set of names more than it tries to
          supply a rigorously or consistently defined set of names, and
          its detailed behavior may change across releases.  For
          example, metaclass attributes are not in the result list when
          the argument is a class.

 -- Function: divmod (a, b)
     Take two (non complex) numbers as arguments and return a pair of
     numbers consisting of their quotient and remainder when using long
     division.  With mixed operand types, the rules for binary
     arithmetic operators apply.  For plain and long integers, the
     result is the same as `(a // b, a % b)'. For floating point
     numbers the result is `(q, a % b)', where _q_ is usually
     `math.floor(a / b)' but may be 1 less than that.  In any case `q *
     b + a % b' is very close to _a_, if `a % b' is non-zero it has the
     same sign as _b_, and `0 <= abs(a % b) < abs(b)'.

     Changed in version 2.3: Using divmod() with complex
     numbers is deprecated.

 -- Function: enumerate (sequence[, start=0])
     Return an enumerate object. _sequence_ must be a sequence, an
     iterator, or some other object which supports
     iteration.  The `next()' method of the iterator returned by
     enumerate() returns a tuple containing a count (from _start_
     which defaults to 0) and the corresponding value obtained from
     iterating over _iterable_.  enumerate() is useful for
     obtaining an indexed series: `(0, seq[0])', `(1, seq[1])', `(2,
     seq[2])', .... For example:

         >>> for i, season in enumerate(['Spring', 'Summer', 'Fall', 'Winter']):
         ...     print i, season
         0 Spring
         1 Summer
         2 Fall
         3 Winter

     New in version 2.3.

     New in version 2.6: The _start_ parameter.

 -- Function: eval (expression[, globals[, locals]])
     The arguments are a string and optional globals and locals.  If
     provided, _globals_ must be a dictionary.  If provided, _locals_
     can be any mapping object.

     Changed in version 2.4: formerly _locals_ was required to be a
     dictionary.

     The _expression_ argument is parsed and evaluated as a Python
     expression (technically speaking, a condition list) using the
     _globals_ and _locals_ dictionaries as global and local namespace.
     If the _globals_ dictionary is present and lacks '__builtins__',
     the current globals are copied into _globals_ before _expression_
     is parsed.  This means that _expression_ normally has full access
     to the standard __builtin__ module and restricted
     environments are propagated.  If the _locals_ dictionary is
     omitted it defaults to the _globals_ dictionary.  If both
     dictionaries are omitted, the expression is executed in the
     environment where eval() is called.  The return value
     is the result of the evaluated expression. Syntax errors are
     reported as exceptions.  Example:

         >>> x = 1
         >>> print eval('x+1')
         2

     This function can also be used to execute arbitrary code objects
     (such as those created by compile()).  In this case
     pass a code object instead of a string.  If the code object has
     been compiled with `'exec'' as the _mode_ argument, see eval().'
     s return value will be `None'.

     Hints: dynamic execution of statements is supported by the
     exec.  statement.  Execution of statements from a file is
     supported by the execfile() function.  The
     globals() and locals() functions returns the
     current global and local dictionary, respectively, which may be
     useful to pass around for use by eval() or
     execfile().

     See ast.literal_eval() for a function that can safely
     evaluate strings with expressions containing only literals.

 -- Function: execfile (filename[, globals[, locals]])
     This function is similar to the exec statement, but
     parses a file instead of a string.  It is different from the
     import statement in that it does not use the module
     administration -- it reads the file unconditionally and does not
     create a new module. (1)

     The arguments are a file name and two optional dictionaries.  The
     file is parsed and evaluated as a sequence of Python statements
     (similarly to a module) using the _globals_ and _locals_
     dictionaries as global and local namespace. If provided, _locals_
     can be any mapping object.

     Changed in version 2.4: formerly _locals_ was required to be a
     dictionary.

     If the _locals_ dictionary is omitted it defaults to the _globals_
     dictionary.  If both dictionaries are omitted, the expression is
     executed in the environment where execfile() is
     called.  The return value is `None'.

          Note: The default _locals_ act as described for function
          locals() below: modifications to the default
          _locals_ dictionary should not be attempted.  Pass an
          explicit _locals_ dictionary if you need to see effects of
          the code on _locals_ after function execfile().
          returns.  execfile() cannot be used reliably to
          modify a function's locals.

 -- Function: file (filename[, mode[, bufsize]])
     Constructor function for the file type, described
     further in section File Objects.  The constructor's
     arguments are the same as those of the open() built-in
     function described below.

     When opening a file, it's preferable to use open().
     instead of  invoking this constructor directly.  See file.
     is more suited to type testing (for example, writing
     `isinstance(f, file)').

     New in version 2.2.

 -- Function: filter (function, iterable)
     Construct a list from those elements of _iterable_ for which
     _function_ returns true.  _iterable_ may be either a sequence, a
     container which supports iteration, or an iterator.  If _iterable_
     is a string or a tuple, the result also has that type; otherwise
     it is always a list.  If _function_ is `None', the identity
     function is assumed, that is, all elements of _iterable_ that are
     false are removed.

     Note that `filter(function, iterable)' is equivalent to `[item for
     item in iterable if function(item)]' if function is not `None' and
     `[item for item in iterable if item]' if function is `None'.

     See itertools.ifilter() and
     itertools.ifilterfalse() for iterator versions of this
     function, including a variation that filters for elements where
     the _function_ returns false.

 -- Function: float ([x])
     Convert a string or a number to floating point.  If the argument
     is a string, it must contain a possibly signed decimal or floating
     point number, possibly embedded in whitespace. The argument may
     also be [+|-]nan or [+|-]inf.  Otherwise, the argument may be a
     plain or long integer or a floating point number, and a floating
     point number with the same value (within Python's floating point
     precision) is returned.  If no argument is given, returns `0.0'.

          Note: When passing in a string, values for NaN and Infinity
          may be returned, depending on the underlying C library.
          Float accepts the strings nan, inf and -inf for NaN and
          positive or negative infinity. The case and a leading + are
          ignored as well as a leading - is ignored for NaN. Float
          always represents NaN and infinity as nan, inf or -inf.

     The float type is described in Numeric Types -- int, float, long, complex.

 -- Function: format (value[, format_spec])
     Convert a _value_ to a "formatted" representation, as controlled by
     _format_spec_.  The interpretation of _format_spec_ will depend on
     the type of the _value_ argument, however there is a standard
     formatting syntax that is used by most built-in types:
     Format Specification Mini-Language.

          Note: `format(value, format_spec)' merely calls
          `value.__format__(format_spec)'.

     New in version 2.6.

 -- Function: frozenset ([iterable])
     Return a frozenset object, optionally with elements taken from
     _iterable_.  The frozenset type is described in Set Types -- set, frozenset.

     For other containers see the built in dict, list,
     and tuple classes, and the collections.
     module.

     New in version 2.4.

 -- Function: getattr (object, name[, default])
     Return the value of the named attribute of _object_.  _name_ must
     be a string.  If the string is the name of one of the object's
     attributes, the result is the value of that attribute.  For
     example, `getattr(x, 'foobar')' is equivalent to `x.foobar'.  If
     the named attribute does not exist, _default_ is returned if
     provided, otherwise AttributeError is raised.

 -- Function: globals ()
     Return a dictionary representing the current global symbol table.
     This is always the dictionary of the current module (inside a
     function or method, this is the module where it is defined, not
     the module from which it is called).

 -- Function: hasattr (object, name)
     The arguments are an object and a string.  The result is `True' if
     the string is the name of one of the object's attributes, `False'
     if not. (This is implemented by calling `getattr(object, name)'
     and seeing whether it raises an exception or not.)

 -- Function: hash (object)
     Return the hash value of the object (if it has one).  Hash values
     are integers.  They are used to quickly compare dictionary keys
     during a dictionary lookup.  Numeric values that compare equal
     have the same hash value (even if they are of different types, as
     is the case for 1 and 1.0).

 -- Function: help ([object])
     Invoke the built-in help system.  (This function is intended for
     interactive use.)  If no argument is given, the interactive help
     system starts on the interpreter console.  If the argument is a
     string, then the string is looked up as the name of a module,
     function, class, method, keyword, or documentation topic, and a
     help page is printed on the console.  If the argument is any other
     kind of object, a help page on the object is generated.

     This function is added to the built-in namespace by the
     site module.

     New in version 2.2.

 -- Function: hex (x)
     Convert an integer number (of any size) to a hexadecimal string.
     The result is a valid Python expression.

          Note: To obtain a hexadecimal string representation for a
          float, use the float.hex() method.

     Changed in version 2.4: Formerly only returned an unsigned literal.

 -- Function: id (object)
     Return the "identity" of an object.  This is an integer (or long
     integer) which is guaranteed to be unique and constant for this
     object during its lifetime.  Two objects with non-overlapping
     lifetimes may have the same id().  value.

     *CPython implementation detail:* This is the address of the object.

 -- Function: input ([prompt])
     Equivalent to `eval(raw_input(prompt))'.

          Warning: This function is not safe from user errors!  It
          expects a valid Python expression as input; if the input is
          not syntactically valid, a SyntaxError will be
          raised. Other exceptions may be raised if there is an error
          during evaluation.  (On the other hand, sometimes this is
          exactly what you need when writing a quick script for expert
          use.)

     If the readline module was loaded, then input()
     will use it to provide elaborate line editing and history
     features.

     Consider using the raw_input() function for general
     input from users.

 -- Function: int ([x[, base]])
     Convert a string or number to a plain integer.  If the argument is
     a string, it must contain a possibly signed decimal number
     representable as a Python integer, possibly embedded in
     whitespace.  The _base_ parameter gives the base for the
     conversion (which is 10 by default) and may be any integer in the
     range [2, 36], or zero.  If _base_ is zero, the proper radix is
     determined based on the contents of string; the interpretation is
     the same as for integer literals.  (See Numeric literals)
     If _base_ is specified and _x_ is not a string, see
     TypeError is raised. Otherwise, the argument may be a plain
     or long integer or a floating point number.  Conversion of floating
     point numbers to integers truncates (towards zero).  If the
     argument is outside the integer range a long object will be
     returned instead.  If no arguments are given, returns `0'.

     The integer type is described in Numeric Types -- int, float, long, complex.

 -- Function: isinstance (object, classinfo)
     Return true if the _object_ argument is an instance of the
     _classinfo_ argument, or of a (direct or indirect) subclass
     thereof.  Also return true if _classinfo_ is a type object
     (new-style class) and _object_ is an object of that type or of a
     (direct or indirect) subclass thereof.  If _object_ is not a class
     instance or an object of the given type, the function always
     returns false.  If _classinfo_ is neither a class object nor a
     type object, it may be a tuple of class or type objects, or may
     recursively contain other such tuples (other sequence types are
     not accepted).  If _classinfo_ is not a class, type, or tuple of
     classes, types, and such tuples, a TypeError exception
     is raised.

     Changed in version 2.2: Support for a tuple of type information
     was added.

 -- Function: issubclass (class, classinfo)
     Return true if _class_ is a subclass (direct or indirect) of
     _classinfo_.  A class is considered a subclass of itself.
     _classinfo_ may be a tuple of class objects, in which case every
     entry in _classinfo_ will be checked. In any other case, a
     TypeError exception is raised.

     Changed in version 2.3: Support for a tuple of type information
     was added.

 -- Function: iter (o[, sentinel])
     Return an iterator object.  The first argument is
     interpreted very differently depending on the presence of the
     second argument. Without a second argument, _o_ must be a
     collection object which supports the iteration protocol (the
     __iter__() method), or it must support the sequence protocol
     (the __getitem__() method with integer arguments
     starting at `0').  If it does not support either of those
     protocols, see TypeError is raised. If the second
     argument, _sentinel_, is given, then _o_ must be a callable
     object.  The iterator created in this case will call _o_ with no
     arguments for each call to its next() method; if the
     value returned is equal to _sentinel_, see StopIteration.
     will be raised, otherwise the value will be returned.

     One useful application of the second form of iter() is
     to read lines of a file until a certain line is reached.  The
     following example reads a file until `"STOP"' is reached:

         with open("mydata.txt") as fp:
             for line in iter(fp.readline, "STOP"):
                 process_line(line)

     New in version 2.2.

 -- Function: len (s)
     Return the length (the number of items) of an object.  The
     argument may be a sequence (string, tuple or list) or a mapping
     (dictionary).

 -- Function: list ([iterable])
     Return a list whose items are the same and in the same order as
     _iterable_'s items.  _iterable_ may be either a sequence, a
     container that supports iteration, or an iterator object.  If
     _iterable_ is already a list, a copy is made and returned, similar
     to `iterable[:]'.  For instance, `list('abc')' returns `['a', 'b',
     'c']' and `list( (1, 2, 3) )' returns `[1, 2, 3]'.  If no argument
     is given, returns a new empty list, `[]'.

     list is a mutable sequence type, as documented in
     Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange.
     For other containers see the built in
     dict, set, and tuple classes, and the
     collections module.

 -- Function: locals ()
     Update and return a dictionary representing the current local
     symbol table.  Free variables are returned by locals().
     when it is called in function blocks, but not in class blocks.

          Note: The contents of this dictionary should not be modified;
          changes may not affect the values of local and free variables
          used by the interpreter.

 -- Function: long ([x[, base]])
     Convert a string or number to a long integer.  If the argument is
     a string, it must contain a possibly signed number of arbitrary
     size, possibly embedded in whitespace. The _base_ argument is
     interpreted in the same way as for int(), and may only
     be given when _x_ is a string. Otherwise, the argument may be a
     plain or long integer or a floating point number, and a long
     integer with the same value is returned.    Conversion of floating
     point numbers to integers truncates (towards zero).  If no
     arguments are given, returns `0L'.

     The long type is described in Numeric Types -- int, float, long, complex.

 -- Function: map (function, iterable, ...)
     Apply _function_ to every item of _iterable_ and return a list of
     the results.  If additional _iterable_ arguments are passed,
     _function_ must take that many arguments and is applied to the
     items from all iterables in parallel.  If one iterable is shorter
     than another it is assumed to be extended with `None' items.  If
     _function_ is `None', the identity function is assumed; if there
     are multiple arguments, see map() returns a list
     consisting of tuples containing the corresponding items from all
     iterables (a kind of transpose operation).  The _iterable_
     arguments may be a sequence  or any iterable object; the result is
     always a list.

 -- Function: max (iterable[, args...][key])
     With a single argument _iterable_, return the largest item of a
     non-empty iterable (such as a string, tuple or list).  With more
     than one argument, return the largest of the arguments.

     The optional _key_ argument specifies a one-argument ordering
     function like that used for `list.sort()'.  The _key_ argument, if
     supplied, must be in keyword form (for example,
     `max(a,b,c,key=func)').

     Changed in version 2.5: Added support for the optional _key_
     argument.

 -- Function: memoryview (obj)
     Return a "memory view" object created from the given argument.  See
     memoryview type for more information.

 -- Function: min (iterable[, args...][key])
     With a single argument _iterable_, return the smallest item of a
     non-empty iterable (such as a string, tuple or list).  With more
     than one argument, return the smallest of the arguments.

     The optional _key_ argument specifies a one-argument ordering
     function like that used for `list.sort()'.  The _key_ argument, if
     supplied, must be in keyword form (for example,
     `min(a,b,c,key=func)').

     Changed in version 2.5: Added support for the optional _key_
     argument.

 -- Function: next (iterator[, default])
     Retrieve the next item from the _iterator_ by calling its
     next() method.  If _default_ is given, it is returned if the
     iterator is exhausted, otherwise StopIteration is
     raised.

     New in version 2.6.

 -- Function: object ()
     Return a new featureless object.  object is a base for
     all new style classes.  It has the methods that are common to all
     instances of new style classes.

     New in version 2.2.

     Changed in version 2.3: This function does not accept any
     arguments. Formerly, it accepted arguments but ignored them.

 -- Function: oct (x)
     Convert an integer number (of any size) to an octal string.  The
     result is a valid Python expression.

     Changed in version 2.4: Formerly only returned an unsigned literal.

 -- Function: open (filename[, mode[, bufsize]])
     Open a file, returning an object of the file type
     described in section File Objects.  If the file cannot
     be opened, see IOError is raised.  When opening a file,
     it's preferable to use open() instead of invoking the
     file constructor directly.

     The first two arguments are the same as for `stdio''s `fopen()':
     _filename_ is the file name to be opened, and _mode_ is a string
     indicating how the file is to be opened.

     The most commonly-used values of _mode_ are `'r'' for reading,
     `'w'' for writing (truncating the file if it already exists), and
     `'a'' for appending (which on _some_ Unix systems means that _all_
     writes append to the end of the file regardless of the current
     seek position).  If _mode_ is omitted, it defaults to `'r''.  The
     default is to use text mode, which may convert `'\n'' characters
     to a platform-specific representation on writing and back on
     reading.  Thus, when opening a binary file, you should append
     `'b'' to the _mode_ value to open the file in binary mode, which
     will improve portability.  (Appending `'b'' is useful even on
     systems that don't treat binary and text files differently, where
     it serves as documentation.)  See below for more possible values
     of _mode_.

     The optional _bufsize_ argument specifies the file's desired
     buffer size: 0 means unbuffered, 1 means line buffered, any other
     positive value means use a buffer of (approximately) that size.  A
     negative _bufsize_ means to use the system default, which is
     usually line buffered for tty devices and fully buffered for other
     files.  If omitted, the system default is used. (2)

     Modes `'r+'', `'w+'' and `'a+'' open the file for updating (note
     that `'w+'' truncates the file).  Append `'b'' to the mode to open
     the file in binary mode, on systems that differentiate between
     binary and text files; on systems that don't have this
     distinction, adding the `'b'' has no effect.

     In addition to the standard `fopen()' values _mode_ may be `'U'' or
     `'rU''.  Python is usually built with universal newline support;
     supplying `'U'' opens the file as a text file, but lines may be
     terminated by any of the following: the Unix end-of-line
     convention `'\n'',  the Macintosh convention `'\r'', or the
     Windows convention `'\r\n''. All of these external representations
     are seen as `'\n'' by the Python program. If Python is built
     without universal newline support a _mode_ with `'U'' is the same
     as normal text mode.  Note that file objects so opened also have
     an attribute called `newlines' which has a value of `None' (if no
     newlines have yet been seen), `'\n'', `'\r'', `'\r\n'', or a tuple
     containing all the newline types seen.

     Python enforces that the mode, after stripping `'U'', begins with
     `'r'', `'w'' or `'a''.

     Python provides many file handling modules including
     fileinput, os, os.path, tempfile,
     and shutil.

     Changed in version 2.5: Restriction on first letter of mode string
     introduced.

 -- Function: ord (c)
     Given a string of length one, return an integer representing the
     Unicode code point of the character when the argument is a unicode
     object, or the value of the byte when the argument is an 8-bit
     string. For example, `ord('a')' returns the integer `97',
     `ord(u'\u2020')' returns `8224'.  This is the inverse of
     chr() for 8-bit strings and of unichr() for
     unicode objects.  If a unicode argument is given and Python was
     built with UCS2 Unicode, then the character's code point must be
     in the range [0..65535] inclusive; otherwise the string length is
     two, and a TypeError will be raised.

 -- Function: pow (x, y[, z])
     Return _x_ to the power _y_; if _z_ is present, return _x_ to the
     power _y_, modulo _z_ (computed more efficiently than `pow(x, y) %
     z'). The two-argument form `pow(x, y)' is equivalent to using the
     power operator: `x**y'.

     The arguments must have numeric types.  With mixed operand types,
     the coercion rules for binary arithmetic operators apply.  For int
     and long int operands, the result has the same type as the
     operands (after coercion) unless the second argument is negative;
     in that case, all arguments are converted to float and a float
     result is delivered.  For example, `10**2' returns `100', but
     `10**-2' returns `0.01'.  (This last feature was added in Python
     2.2.  In Python 2.1 and before, if both arguments were of integer
     types and the second argument was negative, an exception was
     raised.) If the second argument is negative, the third argument
     must be omitted. If _z_ is present, _x_ and _y_ must be of integer
     types, and _y_ must be non-negative.  (This restriction was added
     in Python 2.2.  In Python 2.1 and before, floating 3-argument
     `pow()' returned platform-dependent results depending on
     floating-point rounding accidents.)

 -- Function: print ([object, ...][, sep=' '][, end='\n'][,
          file=sys.stdout])
     Print _object_(s) to the stream _file_, separated by _sep_ and
     followed by _end_.  _sep_, _end_ and _file_, if present, must be
     given as keyword arguments.

     All non-keyword arguments are converted to strings like
     str() does and written to the stream, separated by _sep_ and
     followed by _end_.  Both _sep_ and _end_ must be strings; they can
     also be `None', which means to use the default values.  If no
     _object_ is given, see print() will just write _end_.

     The _file_ argument must be an object with a `write(string)'
     method; if it is not present or `None', see sys.stdout.
     will be used.

          Note: This function is not normally available as a built-in
          since the name `print' is recognized as the print.
          statement.  To disable the statement and use the
          print() function, use this future statement at the top
          of your module:

              from __future__ import print_function

     New in version 2.6.

 -- Function: property ([fget[, fset[, fdel[, doc]]]])
     Return a property attribute for new-style class.es
     (classes that derive from object).

     _fget_ is a function for getting an attribute value, likewise
     _fset_ is a function for setting, and _fdel_ a function for
     del'ing, an attribute.  Typical use is to define a managed
     attribute `x':

         class C(object):
             def __init__(self):
                 self._x = None

             def getx(self):
                 return self._x
             def setx(self, value):
                 self._x = value
             def delx(self):
                 del self._x
             x = property(getx, setx, delx, "I'm the 'x' property.")

     If then _c_ is an instance of _C_, `c.x' will invoke the getter,
     `c.x = value' will invoke the setter and `del c.x' the deleter.

     If given, _doc_ will be the docstring of the property attribute.
     Otherwise, the property will copy _fget_'s docstring (if it
     exists).  This makes it possible to create read-only properties
     easily using property() as a decorator.:

         class Parrot(object):
             def __init__(self):
                 self._voltage = 100000

             @property
             def voltage(self):
                 """Get the current voltage."""
                 return self._voltage

     turns the `voltage()' method into a "getter" for a read-only
     attribute with the same name.

     A property object has `getter', `setter', and `deleter' methods
     usable as decorators that create a copy of the property with the
     corresponding accessor function set to the decorated function.
     This is best explained with an example:

         class C(object):
             def __init__(self):
                 self._x = None

             @property
             def x(self):
                 """I'm the 'x' property."""
                 return self._x

             @x.setter
             def x(self, value):
                 self._x = value

             @x.deleter
             def x(self):
                 del self._x

     This code is exactly equivalent to the first example.  Be sure to
     give the additional functions the same name as the original
     property (`x' in this case.)

     The returned property also has the attributes `fget', `fset', and
     `fdel' corresponding to the constructor arguments.

     New in version 2.2.

     Changed in version 2.5: Use _fget_'s docstring if no _doc_ given.

     Changed in version 2.6: The `getter', `setter', and `deleter'
     attributes were added.

 -- Function: range ([start], stop[, step])
     This is a versatile function to create lists containing arithmetic
     progressions.  It is most often used in for loops.
     The arguments must be plain integers.  If the _step_ argument is
     omitted, it defaults to `1'.  If the _start_ argument is omitted,
     it defaults to `0'.  The full form returns a list of plain
     integers `[start, start + step, start + 2 * step, ...]'.  If _step_
     is positive, the last element is the largest `start + i * step'
     less than _stop_; if _step_ is negative, the last element is the
     smallest `start + i * step' greater than _stop_.  _step_ must not
     be zero (or else ValueError.  is raised).  Example:

         >>> range(10)
         [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
         >>> range(1, 11)
         [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
         >>> range(0, 30, 5)
         [0, 5, 10, 15, 20, 25]
         >>> range(0, 10, 3)
         [0, 3, 6, 9]
         >>> range(0, -10, -1)
         [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
         >>> range(0)
         []
         >>> range(1, 0)
         []

 -- Function: raw_input ([prompt])
     If the _prompt_ argument is present, it is written to standard
     output without a trailing newline.  The function then reads a line
     from input, converts it to a string (stripping a trailing
     newline), and returns that. When EOF is read, see EOFError.
     is raised. Example:

         >>> s = raw_input('--> ')
         --> Monty Python's Flying Circus
         >>> s
         "Monty Python's Flying Circus"

     If the readline module was loaded, then
     raw_input() will use it to provide elaborate line editing
     and history features.

 -- Function: reduce (function, iterable[, initializer])
     Apply _function_ of two arguments cumulatively to the items of
     _iterable_, from left to right, so as to reduce the iterable to a
     single value.  For example, `reduce(lambda x, y: x+y, [1, 2, 3, 4,
     5])' calculates `((((1+2)+3)+4)+5)'.  The left argument, _x_, is
     the accumulated value and the right argument, _y_, is the update
     value from the _iterable_.  If the optional _initializer_ is
     present, it is placed before the items of the iterable in the
     calculation, and serves as a default when the iterable is empty.
     If _initializer_ is not given and _iterable_ contains only one
     item, the first item is returned.

 -- Function: reload (module)
     Reload a previously imported _module_.  The argument must be a
     module object, so it must have been successfully imported before.
     This is useful if you have edited the module source file using an
     external editor and want to try out the new version without
     leaving the Python interpreter.  The return value is the module
     object (the same as the _module_ argument).

     When `reload(module)' is executed:

        * Python modules' code is recompiled and the module-level code
          reexecuted, defining a new set of objects which are bound to
          names in the module's dictionary.  The `init' function of
          extension modules is not called a second time.

        * As with all other objects in Python the old objects are only
          reclaimed after their reference counts drop to zero.

        * The names in the module namespace are updated to point to any
          new or changed objects.

        * Other references to the old objects (such as names external
          to the module) are not rebound to refer to the new objects
          and must be updated in each namespace where they occur if
          that is desired.

     There are a number of other caveats:

     If a module is syntactically correct but its initialization fails,
     the first import statement for it does not bind its
     name locally, but does store a (partially initialized) module
     object in `sys.modules'.  To reload the module you must first
     import it again (this will bind the name to the
     partially initialized module object) before you can
     reload() it.

     When a module is reloaded, its dictionary (containing the module's
     global variables) is retained.  Redefinitions of names will
     override the old definitions, so this is generally not a problem.
     If the new version of a module does not define a name that was
     defined by the old version, the old definition remains.  This
     feature can be used to the module's advantage if it maintains a
     global table or cache of objects -- with a try.
     statement it can test for the table's presence and skip its
     initialization if desired:

         try:
             cache
         except NameError:
             cache = {}

     It is legal though generally not very useful to reload built-in or
     dynamically loaded modules, except for sys,
     __main__ and __builtin__.  In many cases, however,
     extension modules are not designed to be initialized more than
     once, and may fail in arbitrary ways when reloaded.

     If a module imports objects from another module using from...
     See import..., calling reload() for
     the other module does not redefine the objects imported from it --
     one way around this is to re-execute the from.
     statement, another is to use import and qualified
     names (_module_.*name*) instead.

     If a module instantiates instances of a class, reloading the
     module that defines the class does not affect the method
     definitions of the instances -- they continue to use the old class
     definition.  The same is true for derived classes.

 -- Function: repr (object)
     Return a string containing a printable representation of an
     object.  This is the same value yielded by conversions (reverse
     quotes).  It is sometimes useful to be able to access this
     operation as an ordinary function.  For many types, this function
     makes an attempt to return a string that would yield an object
     with the same value when passed to eval(), otherwise the
     representation is a string enclosed in angle brackets that
     contains the name of the type of the object together with
     additional information often including the name and address of the
     object.  A class can control what this function returns for its
     instances by defining a __repr__() method.

 -- Function: reversed (seq)
     Return a reverse iterator.  _seq_ must be an object
     which has a __reversed__() method or supports the
     sequence protocol (the __len__() method and the
     __getitem__() method with integer arguments starting at `0').

     New in version 2.4.

     Changed in version 2.6: Added the possibility to write a custom
     __reversed__() method.

 -- Function: round (x[, n])
     Return the floating point value _x_ rounded to _n_ digits after
     the decimal point.  If _n_ is omitted, it defaults to zero. The
     result is a floating point number.  Values are rounded to the
     closest multiple of 10 to the power minus _n_; if two multiples
     are equally close, rounding is done away from 0 (so. for example,
     `round(0.5)' is `1.0' and `round(-0.5)' is `-1.0').

          Note: The behavior of round() for floats can be
          surprising: for example, `round(2.675, 2)' gives `2.67'
          instead of the expected `2.68'.  This is not a bug: it's a
          result of the fact that most decimal fractions can't be
          represented exactly as a float.  See Floating Point Arithmetic; Issues and Limitations
          for more information.

 -- Function: set ([iterable])
     Return a new set, optionally with elements taken from _iterable_.
     The set type is described in Set Types -- set, frozenset.

     For other containers see the built in dict, list,
     and tuple classes, and the collections.
     module.

     New in version 2.4.

 -- Function: setattr (object, name, value)
     This is the counterpart of getattr().  The arguments
     are an object, a string and an arbitrary value.  The string may
     name an existing attribute or a new attribute.  The function
     assigns the value to the attribute, provided the object allows it.
     For example, `setattr(x, 'foobar', 123)' is equivalent to
     `x.foobar = 123'.

 -- Function: slice ([start], stop[, step])
     Return a slice object representing the set of indices
     specified by `range(start, stop, step)'.  The _start_ and _step_
     arguments default to `None'.  Slice objects have read-only data
     attributes `start', `stop' and `step' which merely return the
     argument values (or their default).  They have no other explicit
     functionality; however they are used by Numerical Python and other
     third party extensions.  Slice objects are also generated when
     extended indexing syntax is used.  For example:
     `a[start:stop:step]' or `a[start:stop, i]'.  See
     itertools.islice().  for an alternate version that returns an
     iterator.

 -- Function: sorted (iterable[, cmp[, key[, reverse]]])
     Return a new sorted list from the items in _iterable_.

     The optional arguments _cmp_, _key_, and _reverse_ have the same
     meaning as those for the `list.sort()' method (described in section
     Mutable Sequence Types).

     _cmp_ specifies a custom comparison function of two arguments
     (iterable elements) which should return a negative, zero or
     positive number depending on whether the first argument is
     considered smaller than, equal to, or larger than the second
     argument: `cmp=lambda x,y: cmp(x.lower(), y.lower())'.  The default
     value is `None'.

     _key_ specifies a function of one argument that is used to extract
     a comparison key from each list element: `key=str.lower'.  The
     default value is `None' (compare the elements directly).

     _reverse_ is a boolean value.  If set to `True', then the list
     elements are sorted as if each comparison were reversed.

     In general, the _key_ and _reverse_ conversion processes are much
     faster than specifying an equivalent _cmp_ function.  This is
     because _cmp_ is called multiple times for each list element while
     _key_ and _reverse_ touch each element only once.  Use
     functools.cmp_to_key() to convert an old-style _cmp_
     function to a _key_ function.

     For sorting examples and a brief sorting tutorial, see Sorting
     HowTo (http://wiki.python.org/moin/HowTo/Sorting/).

     New in version 2.4.

 -- Function: staticmethod (function)
     Return a static method for _function_.

     A static method does not receive an implicit first argument. To
     declare a static method, use this idiom:

         class C:
             @staticmethod
             def f(arg1, arg2, ...): ...

     The `@staticmethod' form is a function decorator. - see
     the description of function definitions in Function definitions
     for details.

     It can be called either on the class (such as `C.f()') or on an
     instance (such as `C().f()').  The instance is ignored except for
     its class.

     Static methods in Python are similar to those found in Java or
     C++. For a more advanced concept, see classmethod() in
     this section.

     For more information on static methods, consult the documentation
     on the standard type hierarchy in The standard type hierarchy.

     New in version 2.2.

     Changed in version 2.4: Function decorator syntax added.

 -- Function: str ([object])
     Return a string containing a nicely printable representation of an
     object.  For strings, this returns the string itself.  The
     difference with `repr(object)' is that `str(object)' does not
     always attempt to return a string that is acceptable to
     eval().; its goal is to return a printable string.  If no
     argument is given, returns the empty string, `'''.

     For more information on strings see Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange
     which
     describes sequence functionality (strings are sequences), and also
     the string-specific methods described in the String Methods
     section. To output formatted strings use template strings or
     the `%' operator described in the String Formatting Operations
     section. In addition see the String Services.
     section. See also unicode().

 -- Function: sum (iterable[, start])
     Sums _start_ and the items of an _iterable_ from left to right and
     returns the total.  _start_ defaults to `0'. The _iterable_'s
     items are normally numbers, and the start value is not allowed to
     be a string.

     For some use cases, there are good alternatives to sum().
     The preferred, fast way to concatenate a sequence of strings
     is by calling `''.join(sequence)'.  To add floating point values
     with extended precision, see math.fsum().  To
     concatenate a series of iterables, consider using
     itertools.chain().

     New in version 2.3.

 -- Function: super (type[, object-or-type])
     Return a proxy object that delegates method calls to a parent or
     sibling class of _type_.  This is useful for accessing inherited
     methods that have been overridden in a class. The search order is
     same as that used by getattr() except that the _type_
     itself is skipped.

     The `__mro__' attribute of the _type_ lists the method resolution
     search order used by both getattr() and super().
     The attribute is dynamic and can change whenever the
     inheritance hierarchy is updated.

     If the second argument is omitted, the super object returned is
     unbound.  If the second argument is an object, `isinstance(obj,
     type)' must be true.  If the second argument is a type,
     `issubclass(type2, type)' must be true (this is useful for
     classmethods).

          Note: super() only works for new-style class.
          es.

     There are two typical use cases for _super_.  In a class hierarchy
     with single inheritance, _super_ can be used to refer to parent
     classes without naming them explicitly, thus making the code more
     maintainable.  This use closely parallels the use of _super_ in
     other programming languages.

     The second use case is to support cooperative multiple inheritance
     in a dynamic execution environment.  This use case is unique to
     Python and is not found in statically compiled languages or
     languages that only support single inheritance.  This makes it
     possible to implement "diamond diagrams" where multiple base
     classes implement the same method.  Good design dictates that this
     method have the same calling signature in every case (because the
     order of calls is determined at runtime, because that order adapts
     to changes in the class hierarchy, and because that order can
     include sibling classes that are unknown prior to runtime).

     For both use cases, a typical superclass call looks like this:

         class C(B):
             def method(self, arg):
                 super(C, self).method(arg)

     Note that super() is implemented as part of the
     binding process for explicit dotted attribute lookups such as
     `super().__getitem__(name)'.  It does so by implementing its own
     __getattribute__() method for searching classes in a
     predictable order that supports cooperative multiple inheritance.
     Accordingly, see super() is undefined for implicit lookups
     using statements or operators such as `super()[name]'.

     Also note that super() is not limited to use inside
     methods.  The two argument form specifies the arguments exactly
     and makes the appropriate references.

     New in version 2.2.

 -- Function: tuple ([iterable])
     Return a tuple whose items are the same and in the same order as
     _iterable_'s items.  _iterable_ may be a sequence, a container
     that supports iteration, or an iterator object. If _iterable_ is
     already a tuple, it is returned unchanged.  For instance,
     `tuple('abc')' returns `('a', 'b', 'c')' and `tuple([1, 2, 3])'
     returns `(1, 2, 3)'.  If no argument is given, returns a new empty
     tuple, `()'.

     tuple is an immutable sequence type, as documented in
     Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange.
     For other containers see the built in
     dict, list, and set classes, and the
     collections module.

 -- Function: type (object)
     Return the type of an _object_.  The return value is a type
     object.  The isinstance() built-in function is
     recommended for testing the type of an object.

     With three arguments, see type() functions as a
     constructor as detailed below.

 -- Function: type (name, bases, dict)
     Return a new type object.  This is essentially a dynamic form of
     the class statement. The _name_ string is the class
     name and becomes the `__name__' attribute; the _bases_ tuple
     itemizes the base classes and becomes the `__bases__' attribute;
     and the _dict_ dictionary is the namespace containing definitions
     for class body and becomes the `__dict__' attribute.  For example,
     the following two statements create identical type.
     objects:

         >>> class X(object):
         ...     a = 1
         ...
         >>> X = type('X', (object,), dict(a=1))

     New in version 2.2.

 -- Function: unichr (i)
     Return the Unicode string of one character whose Unicode code is
     the integer _i_.  For example, `unichr(97)' returns the string
     `u'a''.  This is the inverse of ord() for Unicode
     strings.  The valid range for the argument depends how Python was
     configured - it may be either UCS2 [0..0xFFFF] or UCS4
     [0..0x10FFFF]. ValueError is raised otherwise. For
     ASCII and 8-bit strings see chr().

     New in version 2.0.

 -- Function: unicode ([object[, encoding[, errors]]])
     Return the Unicode string version of _object_ using one of the
     following modes:

     If _encoding_ and/or _errors_ are given, `unicode()' will decode
     the object which can either be an 8-bit string or a character
     buffer using the codec for _encoding_. The _encoding_ parameter is
     a string giving the name of an encoding; if the encoding is not
     known, see LookupError is raised. Error handling is done
     according to _errors_; this specifies the treatment of characters
     which are invalid in the input encoding.  If _errors_ is
     `'strict'' (the default), a ValueError is raised on
     errors, while a value of `'ignore'' causes errors to be silently
     ignored, and a value of `'replace'' causes the official Unicode
     replacement character, `U+FFFD', to be used to replace input
     characters which cannot be decoded.  See also the codecs
     module.

     If no optional parameters are given, `unicode()' will mimic the
     behaviour of `str()' except that it returns Unicode strings
     instead of 8-bit strings. More precisely, if _object_ is a Unicode
     string or subclass it will return that Unicode string without any
     additional decoding applied.

     For objects which provide a __unicode__() method, it
     will call this method without arguments to create a Unicode
     string. For all other objects, the 8-bit string version or
     representation is requested and then converted to a Unicode string
     using the codec for the default encoding in `'strict'' mode.

     For more information on Unicode strings see Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange.
     which describes sequence functionality (Unicode strings are
     sequences), and also the string-specific methods described in the
     String Methods section. To output formatted strings
     use template strings or the `%' operator described in the
     String Formatting Operations section. In addition see the
     String Services section. See also str().

     New in version 2.0.

     Changed in version 2.2: Support for __unicode__().
     added.

 -- Function: vars ([object])
     Without an argument, act like locals().

     With a module, class or class instance object as argument (or
     anything else that has a `__dict__' attribute), return that
     attribute.

          Note: The returned dictionary should not be modified: the
          effects on the corresponding symbol table are undefined. (3)

 -- Function: xrange ([start], stop[, step])
     This function is very similar to range(), but returns
     an "xrange object" instead of a list.  This is an opaque sequence
     type which yields the same values as the corresponding list,
     without actually storing them all simultaneously.  The advantage
     of xrange() over range() is minimal (since
     xrange() still has to create the values when asked for
     them) except when a very large range is used on a memory-starved
     machine or when all of the range's elements are never used (such
     as when the loop is usually terminated with break).

     *CPython implementation detail:* xrange() is intended
     to be simple and fast.  Implementations may impose restrictions to
     achieve this.  The C implementation of Python restricts all
     arguments to native C longs ("short" Python integers), and also
     requires that the number of elements fit in a native C long.  If a
     larger range is needed, an alternate version can be crafted using
     the itertools module: `islice(count(start, step),
     (stop-start+step-1)//step)'.

 -- Function: zip ([iterable, ...])
     This function returns a list of tuples, where the _i_-th tuple
     contains the _i_-th element from each of the argument sequences or
     iterables. The returned list is truncated in length to the length
     of the shortest argument sequence.  When there are multiple
     arguments which are all of the same length, see zip().  is
     similar to map() with an initial argument of `None'.
     With a single sequence argument, it returns a list of 1-tuples.
     With no arguments, it returns an empty list.

     The left-to-right evaluation order of the iterables is guaranteed.
     This makes possible an idiom for clustering a data series into
     n-length groups using `zip(*[iter(s)]*n)'.

     zip() in conjunction with the `*' operator can be used
     to unzip a list:

         >>> x = [1, 2, 3]
         >>> y = [4, 5, 6]
         >>> zipped = zip(x, y)
         >>> zipped
         [(1, 4), (2, 5), (3, 6)]
         >>> x2, y2 = zip(*zipped)
         >>> x == list(x2) and y == list(y2)
         True

     New in version 2.0.

     Changed in version 2.4: Formerly, see zip() required at
     least one argument and `zip()' raised a TypeError.
     instead of returning an empty list.

 -- Function: __import__ (name[, globals[, locals[, fromlist[,
          level]]]])
          Note: This is an advanced function that is not needed in
          everyday Python programming.

     This function is invoked by the import statement.  It
     can be replaced (by importing the __builtin__ module and
     assigning to `__builtin__.__import__') in order to change
     semantics of the import statement, but nowadays it is
     usually simpler to use import hooks (see PEP 302
     (http://www.python.org/dev/peps/pep-0302)).  Direct use of
     __import__() is rare, except in cases where you want to
     import a module whose name is only known at runtime.

     The function imports the module _name_, potentially using the
     given _globals_ and _locals_ to determine how to interpret the
     name in a package context.  The _fromlist_ gives the names of
     objects or submodules that should be imported from the module
     given by _name_.  The standard implementation does not use its
     _locals_ argument at all, and uses its _globals_ only to determine
     the package context of the import statement.

     _level_ specifies whether to use absolute or relative imports.
     The default is `-1' which indicates both absolute and relative
     imports will be attempted.  `0' means only perform absolute
     imports.  Positive values for _level_ indicate the number of
     parent directories to search relative to the directory of the
     module calling __import__().

     When the _name_ variable is of the form `package.module',
     normally, the top-level package (the name up till the first dot)
     is returned, _not_ the module named by _name_.  However, when a
     non-empty _fromlist_ argument is given, the module named by _name_
     is returned.

     For example, the statement `import spam' results in bytecode
     resembling the following code:

         spam = __import__('spam', globals(), locals(), [], -1)

     The statement `import spam.ham' results in this call:

         spam = __import__('spam.ham', globals(), locals(), [], -1)

     Note how __import__() returns the toplevel module here
     because this is the object that is bound to a name by the
     import statement.

     On the other hand, the statement `from spam.ham import eggs,
     sausage as saus' results in

         _temp = __import__('spam.ham', globals(), locals(), ['eggs', 'sausage'], -1)
         eggs = _temp.eggs
         saus = _temp.sausage

     Here, the `spam.ham' module is returned from __import__().
     From this object, the names to import are retrieved and
     assigned to their respective names.

     If you simply want to import a module (potentially within a
     package) by name, you can call __import__() and then
     look it up in sys.modules.:

         >>> import sys
         >>> name = 'foo.bar.baz'
         >>> __import__(name)
         <module 'foo' from ...>
         >>> baz = sys.modules[name]
         >>> baz
         <module 'foo.bar.baz' from ...>

     Changed in version 2.5: The level parameter was added.

     Changed in version 2.5: Keyword support for parameters was added.

5.3 Non-essential Built-in Functions
====================================

There are several built-in functions that are no longer essential to
learn, know or use in modern Python programming.  They have been kept
here to maintain backwards compatibility with programs written for
older versions of Python.

  Python programmers, trainers, students and book writers should feel
free to bypass these functions without concerns about missing something
important.

 -- Function: apply (function, args[, keywords])
     The _function_ argument must be a callable object (a user-defined
     or built-in function or method, or a class object) and the _args_
     argument must be a sequence.  The _function_ is called with _args_
     as the argument list; the number of arguments is the length of the
     tuple. If the optional _keywords_ argument is present, it must be
     a dictionary whose keys are strings.  It specifies keyword
     arguments to be added to the end of the argument list. Calling
     apply() is different from just calling
     `function(args)', since in that case there is always exactly one
     argument.  The use of apply() is equivalent to
     `function(*args, **keywords)'.

     Deprecated since version 2.3: Use the extended call syntax with
     `*args' and `**keywords' instead.

 -- Function: buffer (object[, offset[, size]])
     The _object_ argument must be an object that supports the buffer
     call interface (such as strings, arrays, and buffers).  A new
     buffer object will be created which references the _object_
     argument. The buffer object will be a slice from the beginning of
     _object_ (or from the specified _offset_). The slice will extend
     to the end of _object_ (or will have a length given by the _size_
     argument).

 -- Function: coerce (x, y)
     Return a tuple consisting of the two numeric arguments converted
     to a common type, using the same rules as used by arithmetic
     operations. If coercion is not possible, raise TypeError.

 -- Function: intern (string)
     Enter _string_ in the table of "interned" strings and return the
     interned string - which is _string_ itself or a copy. Interning
     strings is useful to gain a little performance on dictionary
     lookup - if the keys in a dictionary are interned, and the lookup
     key is interned, the key comparisons (after hashing) can be done
     by a pointer compare instead of a string compare.  Normally, the
     names used in Python programs are automatically interned, and the
     dictionaries used to hold module, class or instance attributes
     have interned keys.

     Changed in version 2.3: Interned strings are not immortal (like
     they used to be in Python 2.2 and before); you must keep a
     reference to the return value of intern() around to
     benefit from it.

Footnotes
.........

     (1) It is used relatively rarely so does not warrant being made
     into a statement.

     (2) Specifying a buffer size currently has no effect on systems
     that don't have `setvbuf()'.  The interface to specify the buffer
     size is not done using a method that calls `setvbuf()', because
     that may dump core when called after any I/O has been performed,
     and there's no reliable way to determine whether this is the case.

     (3) In the current implementation, local variable bindings cannot
     normally be affected this way, but variables retrieved from other
     scopes (such as modules) can be.  This may change.

5.4 Built-in Constants
======================

A small number of constants live in the built-in namespace.  They are:

 -- Data: False
     The false value of the bool type.

     New in version 2.3.

 -- Data: True
     The true value of the bool type.

     New in version 2.3.

 -- Data: None
     The sole value of types.NoneType.  `None' is frequently
     used to represent the absence of a value, as when default
     arguments are not passed to a function.

     Changed in version 2.4: Assignments to `None' are illegal and
     raise a SyntaxError.

 -- Data: NotImplemented
     Special value which can be returned by the "rich comparison"
     special methods (see __eq__(), __lt__(), and
     friends), to indicate that the comparison is not implemented with
     respect to the other type.

 -- Data: Ellipsis
     Special value used in conjunction with extended slicing syntax.

 -- Data: __debug__
     This constant is true if Python was not started with an -O
     option.  See also the assert statement.

     Note: The names None and __debug__ cannot
     be reassigned (assignments to them, even as an attribute name,
     raise SyntaxError), so they can be considered "true"
     constants.

     Changed in version 2.7: Assignments to `__debug__' as an attribute
     became illegal.

5.4.1 Constants added by the `site' module
------------------------------------------

The site module (which is imported automatically during
startup, except if the -S command-line option is given)
adds several constants to the built-in namespace.  They are useful for
the interactive interpreter shell and should not be used in programs.

 -- Data: quit ([code=None])
 -- Data: exit ([code=None])
     Objects that when printed, print a message like "Use quit() or
     Ctrl-D (i.e. EOF) to exit", and when called, raise
     SystemExit with the specified exit code.

 -- Data: copyright
 -- Data: license
 -- Data: credits
     Objects that when printed, print a message like "Type license() to
     see the full license text", and when called, display the
     corresponding text in a pager-like fashion (one screen at a time).

5.5 Built-in Types
==================

The following sections describe the standard types that are built into
the interpreter.

     Note: Historically (until release 2.2), Python's built-in types
     have differed from user-defined types because it was not possible
     to use the built-in types as the basis for object-oriented
     inheritance. This limitation no longer exists.

  The principal built-in types are numerics, sequences, mappings,
files, classes, instances and exceptions.

  Some operations are supported by several object types; in particular,
practically all objects can be compared, tested for truth value, and
converted to a string (with the repr() function or the
slightly different str() function).  The latter function is
implicitly used when an object is written by the print().
function.

5.5.1 Truth Value Testing
-------------------------

Any object can be tested for truth value, for use in an if.
or while condition or as operand of the Boolean operations
below. The following values are considered false:

   * `None'

   * `False'

   * zero of any numeric type, for example, `0', `0L', `0.0', `0j'.

   * any empty sequence, for example, `''', `()', `[]'.

   * any empty mapping, for example, `{}'.

   * instances of user-defined classes, if the class defines a
     __nonzero__().  or __len__() method, when that
     method returns the integer zero or bool value `False'.
     (1)

  All other values are considered true -- so objects of many types are
always true.

  Operations and built-in functions that have a Boolean result always
return `0' or `False' for false and `1' or `True' for true, unless
otherwise stated.  (Important exception: the Boolean operations `or'
and `and' always return one of their operands.)

5.5.2 Boolean Operations -- `and', `or', `not'
----------------------------------------------

These are the Boolean operations, ordered by ascending priority:

Operation         Result                                Notes
--------------------------------------------------------------------
`x or y'          if _x_ is false, then _y_, else _x_   (1)
`x and y'         if _x_ is false, then _x_, else _y_   (2)
`not x'           if _x_ is false, then `True', else    (3)
                  `False'

  Notes:

  1. This is a short-circuit operator, so it only evaluates the second
     argument if the first one is False.

  2. This is a short-circuit operator, so it only evaluates the second
     argument if the first one is True.

  3. `not' has a lower priority than non-Boolean operators, so `not a
     == b' is interpreted as `not (a == b)', and `a == not b' is a
     syntax error.

5.5.3 Comparisons
-----------------

Comparison operations are supported by all objects.  They all have the
same priority (which is higher than that of the Boolean operations).
Comparisons can be chained arbitrarily; for example, `x < y <= z' is
equivalent to `x < y and y <= z', except that _y_ is evaluated only
once (but in both cases _z_ is not evaluated at all when `x < y' is
found to be false).

  This table summarizes the comparison operations:

Operation        Meaning                       Notes
-----------------------------------------------------------
`<'              strictly less than
`<='             less than or equal
`>'              strictly greater than
`>='             greater than or equal
`=='             equal
`!='             not equal                     (1)
`is'             object identity
`is not'         negated object identity

  Notes:

  1. `!=' can also be written `<>', but this is an obsolete usage kept
     for backwards compatibility only. New code should always use `!='.

  Objects of different types, except different numeric types and
different string types, never compare equal; such objects are ordered
consistently but arbitrarily (so that sorting a heterogeneous array
yields a consistent result).  Furthermore, some types (for example,
file objects) support only a degenerate notion of comparison where any
two objects of that type are unequal.  Again, such objects are ordered
arbitrarily but consistently. The `<', `<=', `>' and `>=' operators
will raise a TypeError exception when any operand is a
complex number.

  Instances of a class normally compare as non-equal unless the class
defines the __cmp__() method.  Refer to Basic customization)
for information on the use of this method to
effect object comparisons.

*CPython implementation detail:* Objects of different types except
numbers are ordered by their type names; objects of the same types that
don't support proper comparison are ordered by their address.

  Two more operations with the same syntactic priority, `in' and `not
in', are supported only by sequence types (below).

5.5.4 Numeric Types -- `int', `float', `long', `complex'
--------------------------------------------------------

There are four distinct numeric types: _plain integers_, _long
integers_, _floating point numbers_, and _complex numbers_. In
addition, Booleans are a subtype of plain integers. Plain integers
(also just called _integers_) are implemented using `long' in C, which
gives them at least 32 bits of precision (`sys.maxint' is always set to
the maximum plain integer value for the current platform, the minimum
value is `-sys.maxint - 1').  Long integers have unlimited precision.
Floating point numbers are usually implemented using `double' in C;
information about the precision and internal representation of floating
point numbers for the machine on which your program is running is
available in sys.float_info.  Complex numbers have a real
and imaginary part, which are each a floating point number.  To extract
these parts from a complex number _z_, use `z.real' and `z.imag'. (The
standard library includes additional numeric types, see fractions
that hold rationals, and decimal that hold
floating-point numbers with user-definable precision.)

  Numbers are created by numeric literals or as the result of built-in
functions and operators.  Unadorned integer literals (including binary,
hex, and octal numbers) yield plain integers unless the value they
denote is too large to be represented as a plain integer, in which case
they yield a long integer.  Integer literals with an `'L'' or `'l''
suffix yield long integers (`'L'' is preferred because `1l' looks too
much like eleven!).  Numeric literals containing a decimal point or an
exponent sign yield floating point numbers.  Appending `'j'' or `'J''
to a numeric literal yields a complex number with a zero real part. A
complex numeric literal is the sum of a real and an imaginary part.

  Python fully supports mixed arithmetic: when a binary arithmetic
operator has operands of different numeric types, the operand with the
"narrower" type is widened to that of the other, where plain integer is
narrower than long integer is narrower than floating point is narrower
than complex. Comparisons between numbers of mixed type use the same
rule. (2) The constructors int(), long(),
float(), and complex() can be used to produce numbers
of a specific type.

  All built-in numeric types support the following operations. See
The power operator and later sections for the operators'
priorities.

Operation                Result                                Notes
----------------------------------------------------------------------------
`x + y'                  sum of _x_ and _y_
`x - y'                  difference of _x_ and _y_
`x * y'                  product of _x_ and _y_
`x / y'                  quotient of _x_ and _y_               (1)
`x // y'                 (floored) quotient of _x_ and _y_     (4)(5)
`x % y'                  remainder of `x / y'                  (4)
`-x'                     _x_ negated
`+x'                     _x_ unchanged
`abs(x)'                 absolute value or magnitude of _x_    (3)
`int(x)'                 _x_ converted to integer              (2)
`long(x)'                _x_ converted to long integer         (2)
`float(x)'               _x_ converted to floating point       (6)
`complex(re,im)'         a complex number with real part
                         _re_, imaginary part _im_.  _im_
                         defaults to zero.
`c.conjugate()'          conjugate of the complex number _c_.
                         (Identity on real numbers)
`divmod(x, y)'           the pair `(x // y, x % y)'            (3)(4)
`pow(x, y)'              _x_ to the power _y_                  (3)(7)
`x ** y'                 _x_ to the power _y_                  (7)

  Notes:

  1. For (plain or long) integer division, the result is an integer.
     The result is always rounded towards minus infinity: 1/2 is 0,
     (-1)/2 is -1, 1/(-2) is -1, and (-1)/(-2) is 0.  Note that the
     result is a long integer if either operand is a long integer,
     regardless of the numeric value.

  2. Conversion from floats using int() or long()
     truncates toward zero like the related function, see
     math.trunc().  Use the function math.floor() to
     round downward and math.ceil() to round upward.

  3. See Built-in Functions for a full description.

  4. Complex floor division operator, modulo operator, and
     divmod().

     Deprecated since version 2.3: Instead convert to float using
     abs() if appropriate.

  5. Also referred to as integer division.  The resultant value is a
     whole integer, though the result's type is not necessarily int.

  6. float also accepts the strings "nan" and "inf" with an optional
     prefix "+" or "-" for Not a Number (NaN) and positive or negative
     infinity.

     New in version 2.6.

  7. Python defines `pow(0, 0)' and `0 ** 0' to be `1', as is common for
     programming languages.

  All numbers.Real types (see int, long,
and float) also include the following operations:

Operation                Result                                   Notes
-------------------------------------------------------------------------------
`math.trunc(x)'          _x_ truncated to Integral
`round(x[, n])'          _x_ rounded to n digits, rounding half
                         to even. If n is omitted, it defaults
                         to 0.
`math.floor(x)'          the greatest integral float <= _x_
`math.ceil(x)'           the least integral float >= _x_

5.5.4.1 Bit-string Operations on Integer Types
..............................................

Plain and long integer types support additional operations that make
sense only for bit-strings.  Negative numbers are treated as their 2's
complement value (for long integers, this assumes a sufficiently large
number of bits that no overflow occurs during the operation).

  The priorities of the binary bitwise operations are all lower than
the numeric operations and higher than the comparisons; the unary
operation `~' has the same priority as the other unary numeric
operations (`+' and `-').

  This table lists the bit-string operations sorted in ascending
priority:

Operation        Result                               Notes
---------------------------------------------------------------------
`x | y'          bitwise _or_ of _x_ and _y_
`x ^ y'          bitwise _exclusive or_ of _x_ and
                 _y_
`x & y'          bitwise _and_ of _x_ and _y_
`x << n'         _x_ shifted left by _n_ bits         (1)(2)
`x >> n'         _x_ shifted right by _n_ bits        (1)(3)
`~x'             the bits of _x_ inverted

  Notes:

  1. Negative shift counts are illegal and cause a ValueError
     to be raised.

  2. A left shift by _n_ bits is equivalent to multiplication by
     `pow(2, n)'.  A long integer is returned if the result exceeds the
     range of plain integers.

  3. A right shift by _n_ bits is equivalent to division by `pow(2, n)'.

5.5.4.2 Additional Methods on Integer Types
...........................................

 -- Method: int.bit_length ()

 -- Method: long.bit_length ()
     Return the number of bits necessary to represent an integer in
     binary, excluding the sign and leading zeros:

         >>> n = -37
         >>> bin(n)
         '-0b100101'
         >>> n.bit_length()
         6

     More precisely, if `x' is nonzero, then `x.bit_length()' is the
     unique positive integer `k' such that `2**(k-1) <= abs(x) < 2**k'.
     Equivalently, when `abs(x)' is small enough to have a correctly
     rounded logarithm, then `k = 1 + int(log(abs(x), 2))'.  If `x' is
     zero, then `x.bit_length()' returns `0'.

     Equivalent to:

         def bit_length(self):
             s = bin(self)       # binary representation:  bin(-37) --> '-0b100101'
             s = s.lstrip('-0b') # remove leading zeros and minus sign
             return len(s)       # len('100101') --> 6

     New in version 2.7.

5.5.4.3 Additional Methods on Float
...................................

The float type has some additional methods.

 -- Method: float.as_integer_ratio ()
     Return a pair of integers whose ratio is exactly equal to the
     original float and with a positive denominator.  Raises
     OverflowError on infinities and a ValueError on
     NaNs.

     New in version 2.6.

 -- Method: float.is_integer ()
     Return `True' if the float instance is finite with integral value,
     and `False' otherwise:

         >>> (-2.0).is_integer()
         True
         >>> (3.2).is_integer()
         False

     New in version 2.6.

  Two methods support conversion to and from hexadecimal strings.
Since Python's floats are stored internally as binary numbers,
converting a float to or from a _decimal_ string usually involves a
small rounding error.  In contrast, hexadecimal strings allow exact
representation and specification of floating-point numbers.  This can
be useful when debugging, and in numerical work.

 -- Method: float.hex ()
     Return a representation of a floating-point number as a hexadecimal
     string.  For finite floating-point numbers, this representation
     will always include a leading `0x' and a trailing `p' and exponent.

     New in version 2.6.

 -- Method: float.fromhex (s)
     Class method to return the float represented by a hexadecimal
     string _s_.  The string _s_ may have leading and trailing
     whitespace.

     New in version 2.6.

  Note that float.hex() is an instance method, while
float.fromhex() is a class method.

  A hexadecimal string takes the form:

    [sign] ['0x'] integer ['.' fraction] ['p' exponent]

where the optional `sign' may by either `+' or `-', `integer' and
`fraction' are strings of hexadecimal digits, and `exponent' is a
decimal integer with an optional leading sign.  Case is not
significant, and there must be at least one hexadecimal digit in either
the integer or the fraction.  This syntax is similar to the syntax
specified in section 6.4.4.2 of the C99 standard, and also to the
syntax used in Java 1.5 onwards.  In particular, the output of
float.hex() is usable as a hexadecimal floating-point literal in
C or Java code, and hexadecimal strings produced by C's `%a' format
character or Java's `Double.toHexString' are accepted by
float.fromhex().

  Note that the exponent is written in decimal rather than hexadecimal,
and that it gives the power of 2 by which to multiply the coefficient.
For example, the hexadecimal string `0x3.a7p10' represents the
floating-point number `(3 + 10./16 + 7./16**2) * 2.0**10', or `3740.0':

    >>> float.fromhex('0x3.a7p10')
    3740.0

Applying the reverse conversion to `3740.0' gives a different
hexadecimal string representing the same number:

    >>> float.hex(3740.0)
    '0x1.d380000000000p+11'

5.5.5 Iterator Types
--------------------

New in version 2.2.

  Python supports a concept of iteration over containers.  This is
implemented using two distinct methods; these are used to allow
user-defined classes to support iteration.  Sequences, described below
in more detail, always support the iteration methods.

  One method needs to be defined for container objects to provide
iteration support:

 -- Method: container.__iter__ ()
     Return an iterator object.  The object is required to support the
     iterator protocol described below.  If a container supports
     different types of iteration, additional methods can be provided
     to specifically request iterators for those iteration types.  (An
     example of an object supporting multiple forms of iteration would
     be a tree structure which supports both breadth-first and
     depth-first traversal.)  This method corresponds to the `tp_iter'
     slot of the type structure for Python objects in the Python/C API.

  The iterator objects themselves are required to support the following
two methods, which together form the _iterator protocol_:

 -- Method: iterator.__iter__ ()
     Return the iterator object itself.  This is required to allow both
     containers and iterators to be used with the for and
     in statements.  This method corresponds to the
     `tp_iter' slot of the type structure for Python objects in the
     Python/C API.

 -- Method: iterator.next ()
     Return the next item from the container.  If there are no further
     items, raise the StopIteration exception.  This method
     corresponds to the `tp_iternext' slot of the type structure for
     Python objects in the Python/C API.

  Python defines several iterator objects to support iteration over
general and specific sequence types, dictionaries, and other more
specialized forms.  The specific types are not important beyond their
implementation of the iterator protocol.

  The intention of the protocol is that once an iterator's
next() method raises StopIteration, it will continue
to do so on subsequent calls.  Implementations that do not obey this
property are deemed broken.  (This constraint was added in Python 2.3;
in Python 2.2, various iterators are broken according to this rule.)

5.5.5.1 Generator Types
.......................

Python's generator.s provide a convenient way to implement
the iterator protocol.  If a container object's __iter__().
method is implemented as a generator, it will automatically return an
iterator object (technically, a generator object) supplying the
__iter__() and next() methods.  More information
about generators can be found in the documentation for the yield expression.

5.5.6 Sequence Types -- `str', `unicode', `list', `tuple', `bytearray', `buffer', `xrange'
------------------------------------------------------------------------------------------

There are seven sequence types: strings, Unicode strings, lists, tuples,
bytearrays, buffers, and xrange objects.

  For other containers see the built in dict and set
classes, and the collections module.

  String literals are written in single or double quotes: `'xyzzy'',
`"frobozz"'.  See String literals for more about string
literals.  Unicode strings are much like strings, but are specified in
the syntax using a preceding `'u'' character: `u'abc'', `u"def"'. In
addition to the functionality described here, there are also
string-specific methods described in the String Methods.
section. Lists are constructed with square brackets, separating items
with commas: `[a, b, c]'.  Tuples are constructed by the comma operator
(not within square brackets), with or without enclosing parentheses,
but an empty tuple must have the enclosing parentheses, such as `a, b,
c' or `()'.  A single item tuple must have a trailing comma, such as
`(d,)'.

  Bytearray objects are created with the built-in function
bytearray().

  Buffer objects are not directly supported by Python syntax, but can
be created by calling the built-in function buffer().  They
don't support concatenation or repetition.

  Objects of type xrange are similar to buffers in that there is no
specific syntax to create them, but they are created using the
xrange() function.  They don't support slicing, concatenation or
repetition, and using `in', `not in', see min() or max()
on them is inefficient.

  Most sequence types support the following operations.  The `in' and
`not in' operations have the same priorities as the comparison
operations.  The `+' and `*' operations have the same priority as the
corresponding numeric operations.  (3) Additional methods are provided
for Mutable Sequence Types.

  This table lists the sequence operations sorted in ascending priority
(operations in the same box have the same priority).  In the table, _s_
and _t_ are sequences of the same type; _n_, _i_ and _j_ are integers:

Operation              Result                               Notes
---------------------------------------------------------------------------
`x in s'               `True' if an item of _s_ is equal    (1)
                       to _x_, else `False'
`x not in s'           `False' if an item of _s_ is equal   (1)
                       to _x_, else `True'
`s + t'                the concatenation of _s_ and _t_     (6)
`s * n, n * s'         _n_ shallow copies of _s_            (2)
                       concatenated
`s[i]'                 _i_'th item of _s_, origin 0         (3)
`s[i:j]'               slice of _s_ from _i_ to _j_         (3)(4)
`s[i:j:k]'             slice of _s_ from _i_ to _j_ with    (3)(5)
                       step _k_
`len(s)'               length of _s_
`min(s)'               smallest item of _s_
`max(s)'               largest item of _s_
`s.index(i)'           index of the first occurence of _i_
                       in _s_
`s.count(i)'           total number of occurences of _i_
                       in _s_

  Sequence types also support comparisons. In particular, tuples and
lists are compared lexicographically by comparing corresponding
elements. This means that to compare equal, every element must compare
equal and the two sequences must be of the same type and have the same
length. (For full details see Comparisons in the language
reference.)

  Notes:

  1. When _s_ is a string or Unicode string object the `in' and `not in'
     operations act like a substring test.  In Python versions before
     2.3, _x_ had to be a string of length 1. In Python 2.3 and beyond,
     _x_ may be a string of any length.

  2. Values of _n_ less than `0' are treated as `0' (which yields an
     empty sequence of the same type as _s_).  Note also that the
     copies are shallow; nested structures are not copied.  This often
     haunts new Python programmers; consider:

         >>> lists = [[]] * 3
         >>> lists
         [[], [], []]
         >>> lists[0].append(3)
         >>> lists
         [[3], [3], [3]]

     What has happened is that `[[]]' is a one-element list containing
     an empty list, so all three elements of `[[]] * 3' are (pointers
     to) this single empty list.  Modifying any of the elements of
     `lists' modifies this single list.  You can create a list of
     different lists this way:

         >>> lists = [[] for i in range(3)]
         >>> lists[0].append(3)
         >>> lists[1].append(5)
         >>> lists[2].append(7)
         >>> lists
         [[3], [5], [7]]

  3. If _i_ or _j_ is negative, the index is relative to the end of the
     string: `len(s) + i' or `len(s) + j' is substituted.  But note
     that `-0' is still `0'.

  4. The slice of _s_ from _i_ to _j_ is defined as the sequence of
     items with index _k_ such that `i <= k < j'.  If _i_ or _j_ is
     greater than `len(s)', use `len(s)'.  If _i_ is omitted or `None',
     use `0'.  If _j_ is omitted or `None', use `len(s)'.  If _i_ is
     greater than or equal to _j_, the slice is empty.

  5. The slice of _s_ from _i_ to _j_ with step _k_ is defined as the
     sequence of items with index  `x = i + n*k' such that `0 <= n <
     (j-i)/k'.  In other words, the indices are `i', `i+k', `i+2*k',
     `i+3*k' and so on, stopping when _j_ is reached (but never
     including _j_).  If _i_ or _j_ is greater than `len(s)', use
     `len(s)'.  If _i_ or _j_ are omitted or `None', they become "end"
     values (which end depends on the sign of _k_).  Note, _k_ cannot
     be zero.  If _k_ is `None', it is treated like `1'.

  6. *CPython implementation detail:* If _s_ and _t_ are both strings,
     some Python implementations such as CPython can usually perform an
     in-place optimization for assignments of the form `s = s + t' or
     `s += t'.  When applicable, this optimization makes quadratic
     run-time much less likely.  This optimization is both version and
     implementation dependent.  For performance sensitive code, it is
     preferable to use the str.join() method which assures
     consistent linear concatenation performance across versions and
     implementations.

     Changed in version 2.4: Formerly, string concatenation never
     occurred in-place.

5.5.6.1 String Methods
......................

Below are listed the string methods which both 8-bit strings and
Unicode objects support.  Some of them are also available on
bytearray.  objects.

  In addition, Python's strings support the sequence type methods
described in the Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange
section. To output formatted strings
use template strings or the `%' operator described in the String Formatting Operations
section. Also, see the re.
module for string functions based on regular expressions.

 -- Method: str.capitalize ()
     Return a copy of the string with its first character capitalized
     and the rest lowercased.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.center (width[, fillchar])
     Return centered in a string of length _width_. Padding is done
     using the specified _fillchar_ (default is a space).

     Changed in version 2.4: Support for the _fillchar_ argument.

 -- Method: str.count (sub[, start[, end]])
     Return the number of non-overlapping occurrences of substring
     _sub_ in the range [_start_, _end_].  Optional arguments _start_
     and _end_ are interpreted as in slice notation.

 -- Method: str.decode ([encoding[, errors]])
     Decodes the string using the codec registered for _encoding_.
     _encoding_ defaults to the default string encoding.  _errors_ may
     be given to set a different error handling scheme.  The default is
     `'strict'', meaning that encoding errors raise UnicodeError.
     Other possible values are `'ignore'', `'replace'' and any
     other name registered via codecs.register_error(), see
     section Codec Base Classes.

     New in version 2.2.

     Changed in version 2.3: Support for other error handling schemes
     added.

     Changed in version 2.7: Support for keyword arguments added.

 -- Method: str.encode ([encoding[, errors]])
     Return an encoded version of the string.  Default encoding is the
     current default string encoding.  _errors_ may be given to set a
     different error handling scheme.  The default for _errors_ is
     `'strict'', meaning that encoding errors raise a
     UnicodeError.  Other possible values are `'ignore'',
     `'replace'', `'xmlcharrefreplace'', `'backslashreplace'' and any
     other name registered via codecs.register_error(), see
     section Codec Base Classes. For a list of possible
     encodings, see section Standard Encodings.

     New in version 2.0.

     Changed in version 2.3: Support for `'xmlcharrefreplace'' and
     `'backslashreplace'' and other error handling schemes added.

     Changed in version 2.7: Support for keyword arguments added.

 -- Method: str.endswith (suffix[, start[, end]])
     Return `True' if the string ends with the specified _suffix_,
     otherwise return `False'.  _suffix_ can also be a tuple of
     suffixes to look for.  With optional _start_, test beginning at
     that position.  With optional _end_, stop comparing at that
     position.

     Changed in version 2.5: Accept tuples as _suffix_.

 -- Method: str.expandtabs ([tabsize])
     Return a copy of the string where all tab characters are replaced
     by one or more spaces, depending on the current column and the
     given tab size.  The column number is reset to zero after each
     newline occurring in the string.  If _tabsize_ is not given, a tab
     size of `8' characters is assumed.  This doesn't understand other
     non-printing characters or escape sequences.

 -- Method: str.find (sub[, start[, end]])
     Return the lowest index in the string where substring _sub_ is
     found, such that _sub_ is contained in the slice `s[start:end]'.
     Optional arguments _start_ and _end_ are interpreted as in slice
     notation.  Return `-1' if _sub_ is not found.

 -- Method: str.format (*args, **kwargs)
     Perform a string formatting operation.  The string on which this
     method is called can contain literal text or replacement fields
     delimited by braces `{}'.  Each replacement field contains either
     the numeric index of a positional argument, or the name of a
     keyword argument.  Returns a copy of the string where each
     replacement field is replaced with the string value of the
     corresponding argument.

         >>> "The sum of 1 + 2 is {0}".format(1+2)
         'The sum of 1 + 2 is 3'

     See Format String Syntax for a description of the
     various formatting options that can be specified in format strings.

     This method of string formatting is the new standard in Python
     3.0, and should be preferred to the `%' formatting described in
     String Formatting Operations in new code.

     New in version 2.6.

 -- Method: str.index (sub[, start[, end]])
     Like find(), but raise ValueError when the
     substring is not found.

 -- Method: str.isalnum ()
     Return true if all characters in the string are alphanumeric and
     there is at least one character, false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.isalpha ()
     Return true if all characters in the string are alphabetic and
     there is at least one character, false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.isdigit ()
     Return true if all characters in the string are digits and there
     is at least one character, false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.islower ()
     Return true if all cased characters in the string are lowercase
     and there is at least one cased character, false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.isspace ()
     Return true if there are only whitespace characters in the string
     and there is at least one character, false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.istitle ()
     Return true if the string is a titlecased string and there is at
     least one character, for example uppercase characters may only
     follow uncased characters and lowercase characters only cased
     ones.  Return false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.isupper ()
     Return true if all cased characters in the string are uppercase
     and there is at least one cased character, false otherwise.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.join (iterable)
     Return a string which is the concatenation of the strings in the
     iterable. _iterable_.  The separator between elements
     is the string providing this method.

 -- Method: str.ljust (width[, fillchar])
     Return the string left justified in a string of length _width_.
     Padding is done using the specified _fillchar_ (default is a
     space).  The original string is returned if _width_ is less than
     `len(s)'.

     Changed in version 2.4: Support for the _fillchar_ argument.

 -- Method: str.lower ()
     Return a copy of the string converted to lowercase.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.lstrip ([chars])
     Return a copy of the string with leading characters removed.  The
     _chars_ argument is a string specifying the set of characters to
     be removed.  If omitted or `None', the _chars_ argument defaults
     to removing whitespace.  The _chars_ argument is not a prefix;
     rather, all combinations of its values are stripped:

         >>> '   spacious   '.lstrip()
         'spacious   '
         >>> 'www.example.com'.lstrip('cmowz.')
         'example.com'

     Changed in version 2.2.2: Support for the _chars_ argument.

 -- Method: str.partition (sep)
     Split the string at the first occurrence of _sep_, and return a
     3-tuple containing the part before the separator, the separator
     itself, and the part after the separator.  If the separator is not
     found, return a 3-tuple containing the string itself, followed by
     two empty strings.

     New in version 2.5.

 -- Method: str.replace (old, new[, count])
     Return a copy of the string with all occurrences of substring
     _old_ replaced by _new_.  If the optional argument _count_ is
     given, only the first _count_ occurrences are replaced.

 -- Method: str.rfind (sub[, start[, end]])
     Return the highest index in the string where substring _sub_ is
     found, such that _sub_ is contained within `s[start:end]'.
     Optional arguments _start_ and _end_ are interpreted as in slice
     notation.  Return `-1' on failure.

 -- Method: str.rindex (sub[, start[, end]])
     Like rfind() but raises ValueError when
     the substring _sub_ is not found.

 -- Method: str.rjust (width[, fillchar])
     Return the string right justified in a string of length _width_.
     Padding is done using the specified _fillchar_ (default is a
     space). The original string is returned if _width_ is less than
     `len(s)'.

     Changed in version 2.4: Support for the _fillchar_ argument.

 -- Method: str.rpartition (sep)
     Split the string at the last occurrence of _sep_, and return a
     3-tuple containing the part before the separator, the separator
     itself, and the part after the separator.  If the separator is not
     found, return a 3-tuple containing two empty strings, followed by
     the string itself.

     New in version 2.5.

 -- Method: str.rsplit ([sep[, maxsplit]])
     Return a list of the words in the string, using _sep_ as the
     delimiter string.  If _maxsplit_ is given, at most _maxsplit_
     splits are done, the _rightmost_ ones.  If _sep_ is not specified
     or `None', any whitespace string is a separator.  Except for
     splitting from the right, see rsplit() behaves like
     split() which is described in detail below.

     New in version 2.4.

 -- Method: str.rstrip ([chars])
     Return a copy of the string with trailing characters removed.  The
     _chars_ argument is a string specifying the set of characters to
     be removed.  If omitted or `None', the _chars_ argument defaults
     to removing whitespace.  The _chars_ argument is not a suffix;
     rather, all combinations of its values are stripped:

         >>> '   spacious   '.rstrip()
         '   spacious'
         >>> 'mississippi'.rstrip('ipz')
         'mississ'

     Changed in version 2.2.2: Support for the _chars_ argument.

 -- Method: str.split ([sep[, maxsplit]])
     Return a list of the words in the string, using _sep_ as the
     delimiter string.  If _maxsplit_ is given, at most _maxsplit_
     splits are done (thus, the list will have at most `maxsplit+1'
     elements).  If _maxsplit_ is not specified, then there is no limit
     on the number of splits (all possible splits are made).

     If _sep_ is given, consecutive delimiters are not grouped together
     and are deemed to delimit empty strings (for example,
     `'1,,2'.split(',')' returns `['1', '', '2']').  The _sep_ argument
     may consist of multiple characters (for example,
     `'1<>2<>3'.split('<>')' returns `['1', '2', '3']').  Splitting an
     empty string with a specified separator returns `['']'.

     If _sep_ is not specified or is `None', a different splitting
     algorithm is applied: runs of consecutive whitespace are regarded
     as a single separator, and the result will contain no empty
     strings at the start or end if the string has leading or trailing
     whitespace.  Consequently, splitting an empty string or a string
     consisting of just whitespace with a `None' separator returns `[]'.

     For example, `' 1  2   3  '.split()' returns `['1', '2', '3']', and
     `'  1  2   3  '.split(None, 1)' returns `['1', '2   3  ']'.

 -- Method: str.splitlines ([keepends])
     Return a list of the lines in the string, breaking at line
     boundaries.  Line breaks are not included in the resulting list
     unless _keepends_ is given and true.

 -- Method: str.startswith (prefix[, start[, end]])
     Return `True' if string starts with the _prefix_, otherwise return
     `False'.  _prefix_ can also be a tuple of prefixes to look for.
     With optional _start_, test string beginning at that position.
     With optional _end_, stop comparing string at that position.

     Changed in version 2.5: Accept tuples as _prefix_.

 -- Method: str.strip ([chars])
     Return a copy of the string with the leading and trailing
     characters removed.  The _chars_ argument is a string specifying
     the set of characters to be removed.  If omitted or `None', the
     _chars_ argument defaults to removing whitespace.  The _chars_
     argument is not a prefix or suffix; rather, all combinations of its
     values are stripped:

         >>> '   spacious   '.strip()
         'spacious'
         >>> 'www.example.com'.strip('cmowz.')
         'example'

     Changed in version 2.2.2: Support for the _chars_ argument.

 -- Method: str.swapcase ()
     Return a copy of the string with uppercase characters converted to
     lowercase and vice versa.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.title ()
     Return a titlecased version of the string where words start with
     an uppercase character and the remaining characters are lowercase.

     The algorithm uses a simple language-independent definition of a
     word as groups of consecutive letters.  The definition works in
     many contexts but it means that apostrophes in contractions and
     possessives form word boundaries, which may not be the desired
     result:

         >>> "they're bill's friends from the UK".title()
         "They'Re Bill'S Friends From The Uk"

     A workaround for apostrophes can be constructed using regular
     expressions:

         >>> import re
         >>> def titlecase(s):
                 return re.sub(r"[A-Za-z]+('[A-Za-z]+)?",
                               lambda mo: mo.group(0)[0].upper() +
                                          mo.group(0)[1:].lower(),
                               s)

         >>> titlecase("they're bill's friends.")
         "They're Bill's Friends."

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.translate (table[, deletechars])
     Return a copy of the string where all characters occurring in the
     optional argument _deletechars_ are removed, and the remaining
     characters have been mapped through the given translation table,
     which must be a string of length 256.

     You can use the maketrans() helper function in the
     string.  module to create a translation table. For
     string objects, set the _table_ argument to `None' for
     translations that only delete characters:

         >>> 'read this short text'.translate(None, 'aeiou')
         'rd ths shrt txt'

     New in version 2.6: Support for a `None' _table_ argument.

     For Unicode objects, the translate() method does not
     accept the optional _deletechars_ argument.  Instead, it returns a
     copy of the _s_ where all characters have been mapped through the
     given translation table which must be a mapping of Unicode
     ordinals to Unicode ordinals, Unicode strings or `None'.  Unmapped
     characters are left untouched. Characters mapped to `None' are
     deleted.  Note, a more flexible approach is to create a custom
     character mapping codec using the codecs module (see
     `encodings.cp1251' for an example).

 -- Method: str.upper ()
     Return a copy of the string converted to uppercase.

     For 8-bit strings, this method is locale-dependent.

 -- Method: str.zfill (width)
     Return the numeric string left filled with zeros in a string of
     length _width_.  A sign prefix is handled correctly.  The original
     string is returned if _width_ is less than `len(s)'.

     New in version 2.2.2.

  The following methods are present only on unicode objects:

 -- Method: unicode.isnumeric ()
     Return `True' if there are only numeric characters in S, `False'
     otherwise. Numeric characters include digit characters, and all
     characters that have the Unicode numeric value property, e.g.
     U+2155, VULGAR FRACTION ONE FIFTH.

 -- Method: unicode.isdecimal ()
     Return `True' if there are only decimal characters in S, `False'
     otherwise. Decimal characters include digit characters, and all
     characters that that can be used to form decimal-radix numbers,
     e.g. U+0660, ARABIC-INDIC DIGIT ZERO.

5.5.6.2 String Formatting Operations
....................................

String and Unicode objects have one unique built-in operation: the `%'
operator (modulo).  This is also known as the string _formatting_ or
_interpolation_ operator.  Given `format % values' (where _format_ is a
string or Unicode object), `%' conversion specifications in _format_
are replaced with zero or more elements of _values_.  The effect is
similar to the using `sprintf()' in the C language.  If _format_ is a
Unicode object, or if any of the objects being converted using the `%s'
conversion are Unicode objects, the result will also be a Unicode
object.

  If _format_ requires a single argument, _values_ may be a single
non-tuple object. (4)  Otherwise, _values_ must be a tuple with exactly
the number of items specified by the format string, or a single mapping
object (for example, a dictionary).

  A conversion specifier contains two or more characters and has the
following components, which must occur in this order:

  1. The `'%'' character, which marks the start of the specifier.

  2. Mapping key (optional), consisting of a parenthesised sequence of
     characters (for example, `(somename)').

  3. Conversion flags (optional), which affect the result of some
     conversion types.

  4. Minimum field width (optional).  If specified as an `'*''
     (asterisk), the actual width is read from the next element of the
     tuple in _values_, and the object to convert comes after the
     minimum field width and optional precision.

  5. Precision (optional), given as a `'.'' (dot) followed by the
     precision.  If specified as `'*'' (an asterisk), the actual width
     is read from the next element of the tuple in _values_, and the
     value to convert comes after the precision.

  6. Length modifier (optional).

  7. Conversion type.

  When the right argument is a dictionary (or other mapping type), then
the formats in the string _must_ include a parenthesised mapping key
into that dictionary inserted immediately after the `'%'' character.
The mapping key selects the value to be formatted from the mapping.
For example:

    >>> print '%(language)s has %(number)03d quote types.' % \
    ...       {"language": "Python", "number": 2}
    Python has 002 quote types.

In this case no `*' specifiers may occur in a format (since they
require a sequential parameter list).

  The conversion flag characters are:

Flag          Meaning
----------------------------------------------------------------------------------------
`'#''         The value conversion will use the "alternate form" (where defined below).
`'0''         The conversion will be zero padded for numeric values.
`'-''         The converted value is left adjusted (overrides the `'0'' conversion if
              both are given).
`' ''         (a space) A blank should be left before a positive number (or empty
              string) produced by a signed conversion.
`'+''         A sign character (`'+'' or `'-'') will precede the conversion (overrides
              a "space" flag).

  A length modifier (`h', `l', or `L') may be present, but is ignored
as it is not necessary for Python - so e.g. `%ld' is identical to `%d'.

  The conversion types are:

Conversion       Meaning                                                   Notes
---------------------------------------------------------------------------------------
`'d''            Signed integer decimal.
`'i''            Signed integer decimal.
`'o''            Signed octal value.                                       (1)
`'u''            Obsolete type - it is identical to `'d''.                 (7)
`'x''            Signed hexadecimal (lowercase).                           (2)
`'X''            Signed hexadecimal (uppercase).                           (2)
`'e''            Floating point exponential format (lowercase).            (3)
`'E''            Floating point exponential format (uppercase).            (3)
`'f''            Floating point decimal format.                            (3)
`'F''            Floating point decimal format.                            (3)
`'g''            Floating point format. Uses lowercase exponential format  (4)
                 if exponent is less than -4 or not less than precision,
                 decimal format otherwise.
`'G''            Floating point format. Uses uppercase exponential format  (4)
                 if exponent is less than -4 or not less than precision,
                 decimal format otherwise.
`'c''            Single character (accepts integer or single character
                 string).
`'r''            String (converts any Python object using repr() (5)).

`'s''            String (converts any Python object using str() (6)).

`'%''            No argument is converted, results in a `'%'' character
                 in the result.

  Notes:

  1. The alternate form causes a leading zero (`'0'') to be inserted
     between left-hand padding and the formatting of the number if the
     leading character of the result is not already a zero.

  2. The alternate form causes a leading `'0x'' or `'0X'' (depending on
     whether the `'x'' or `'X'' format was used) to be inserted between
     left-hand padding and the formatting of the number if the leading
     character of the result is not already a zero.

  3. The alternate form causes the result to always contain a decimal
     point, even if no digits follow it.

     The precision determines the number of digits after the decimal
     point and defaults to 6.

  4. The alternate form causes the result to always contain a decimal
     point, and trailing zeroes are not removed as they would otherwise
     be.

     The precision determines the number of significant digits before
     and after the decimal point and defaults to 6.

  5. The `%r' conversion was added in Python 2.0.

     The precision determines the maximal number of characters used.

  6. If the object or format provided is a unicode string,
     the resulting string will also be unicode.

     The precision determines the maximal number of characters used.

  7. See PEP 237 (http://www.python.org/dev/peps/pep-0237).

  Since Python strings have an explicit length, `%s' conversions do not
assume that `'\0'' is the end of the string.

  Changed in version 2.7: `%f' conversions for numbers whose absolute
value is over 1e50 are no longer replaced by `%g' conversions.

  Additional string operations are defined in standard modules
string and re.

5.5.6.3 XRange Type
...................

The xrange type is an immutable sequence which is commonly
used for looping.  The advantage of the xrange type is that
an xrange.  object will always take the same amount of
memory, no matter the size of the range it represents.  There are no
consistent performance advantages.

  XRange objects have very little behavior: they only support indexing,
iteration, and the len() function.

5.5.6.4 Mutable Sequence Types
..............................

List and bytearray objects support additional operations
that allow in-place modification of the object. Other mutable sequence
types (when added to the language) should also support these
operations. Strings and tuples are immutable sequence types: such
objects cannot be modified once created.  The following operations are
defined on mutable sequence types (where _x_ is an arbitrary object):

Operation                          Result                               Notes
--------------------------------------------------------------------------------------------------
`s[i] = x'                         item _i_ of _s_ is replaced by _x_
`s[i:j] = t'                       slice of _s_ from _i_ to _j_ is
                                   replaced by the contents of the
                                   iterable _t_
`del s[i:j]'                       same as `s[i:j] = []'
`s[i:j:k] = t'                     the elements of `s[i:j:k]' are       (1)
                                   replaced by those of _t_
`del s[i:j:k]'                     removes the elements of `s[i:j:k]'
                                   from the list
`s.append(x)'                      same as `s[len(s):len(s)] = [x]'     (2)
`s.extend(x)'                      same as `s[len(s):len(s)] = x'       (3)
`s.count(x)'                       return number of _i_'s for which
                                   `s[i] == x'
`s.index(x[, i[, j]])'             return smallest _k_ such that `s[k]  (4)
                                   == x' and `i <= k < j'
`s.insert(i, x)'                   same as `s[i:i] = [x]'               (5)
`s.pop([i])'                       same as `x = s[i]; del s[i]; return  (6)
                                   x'
`s.remove(x)'                      same as `del s[s.index(x)]'          (4)
`s.reverse()'                      reverses the items of _s_ in place   (7)
`s.sort([cmp[, key[, reverse]]])'  sort the items of _s_ in place       (7)(8)(9)(10)

  Notes:

  1. _t_ must have the same length as the slice it is  replacing.

  2. The C implementation of Python has historically accepted multiple
     parameters and implicitly joined them into a tuple; this no longer
     works in Python 2.0.  Use of this misfeature has been deprecated
     since Python 1.4.

  3. _x_ can be any iterable object.

  4. Raises ValueError when _x_ is not found in _s_. When a
     negative index is passed as the second or third parameter to the
     `index()' method, the list length is added, as for slice indices.
     If it is still negative, it is truncated to zero, as for slice
     indices.

     Changed in version 2.3: Previously, `index()' didn't have
     arguments for specifying start and stop positions.

  5. When a negative index is passed as the first parameter to the
     `insert()' method, the list length is added, as for slice indices.
     If it is still negative, it is truncated to zero, as for slice
     indices.

     Changed in version 2.3: Previously, all negative indices were
     truncated to zero.

  6. The `pop()' method is only supported by the list and array types.
     The optional argument _i_ defaults to `-1', so that by default the
     last item is removed and returned.

  7. The `sort()' and `reverse()' methods modify the list in place for
     economy of space when sorting or reversing a large list.  To
     remind you that they operate by side effect, they don't return the
     sorted or reversed list.

  8. The `sort()' method takes optional arguments for controlling the
     comparisons.

     _cmp_ specifies a custom comparison function of two arguments
     (list items) which should return a negative, zero or positive
     number depending on whether the first argument is considered
     smaller than, equal to, or larger than the second argument:
     `cmp=lambda x,y: cmp(x.lower(), y.lower())'.  The default value is
     `None'.

     _key_ specifies a function of one argument that is used to extract
     a comparison key from each list element: `key=str.lower'.  The
     default value is `None'.

     _reverse_ is a boolean value.  If set to `True', then the list
     elements are sorted as if each comparison were reversed.

     In general, the _key_ and _reverse_ conversion processes are much
     faster than specifying an equivalent _cmp_ function.  This is
     because _cmp_ is called multiple times for each list element while
     _key_ and _reverse_ touch each element only once.  Use
     functools.cmp_to_key() to convert an old-style _cmp_
     function to a _key_ function.

     Changed in version 2.3: Support for `None' as an equivalent to
     omitting _cmp_ was added.

     Changed in version 2.4: Support for _key_ and _reverse_ was added.

  9. Starting with Python 2.3, the `sort()' method is guaranteed to be
     stable.  A sort is stable if it guarantees not to change the
     relative order of elements that compare equal -- this is helpful
     for sorting in multiple passes (for example, sort by department,
     then by salary grade).

 10. *CPython implementation detail:* While a list is being sorted, the
     effect of attempting to mutate, or even inspect, the list is
     undefined.  The C implementation of Python 2.3 and newer makes the
     list appear empty for the duration, and raises ValueError
     if it can detect that the list has been mutated during a sort.

5.5.7 Set Types -- `set', `frozenset'
-------------------------------------

A _set_ object is an unordered collection of distinct hashable
objects.  Common uses include membership testing, removing
duplicates from a sequence, and computing mathematical operations such
as intersection, union, difference, and symmetric difference.  (For
other containers see the built in dict, list, and
tuple classes, and the collections module.)

  New in version 2.4.

  Like other collections, sets support `x in set', `len(set)', and `for
x in set'.  Being an unordered collection, sets do not record element
position or order of insertion.  Accordingly, sets do not support
indexing, slicing, or other sequence-like behavior.

  There are currently two built-in set types, see set and
frozenset.  The set type is mutable -- the contents
can be changed using methods like `add()' and `remove()'.  Since it is
mutable, it has no hash value and cannot be used as either a dictionary
key or as an element of another set.  The frozenset type is
immutable and hashable. -- its contents cannot be altered
after it is created; it can therefore be used as a dictionary key or as
an element of another set.

  As of Python 2.7, non-empty sets (not frozensets) can be created by
placing a comma-separated list of elements within braces, for example:
`{'jack', 'sjoerd'}', in addition to the set constructor.

  The constructors for both classes work the same:

 -- Class: set ([iterable])
 -- Class: frozenset ([iterable])
     Return a new set or frozenset object whose elements are taken from
     _iterable_.  The elements of a set must be hashable.  To represent
     sets of sets, the inner sets must be frozenset.
     objects.  If _iterable_ is not specified, a new empty set is
     returned.

     Instances of set and frozenset provide the
     following operations:

      -- Description: len(s)
          Return the cardinality of set _s_.

      -- Description: x in s
          Test _x_ for membership in _s_.

      -- Description: x not in s
          Test _x_ for non-membership in _s_.

      -- Method: isdisjoint (other)
          Return True if the set has no elements in common with
          _other_.  Sets are disjoint if and only if their intersection
          is the empty set.

          New in version 2.6.

      -- Method: issubset (other)
      -- Method: set <= other
          Test whether every element in the set is in _other_.

      -- Method: set < other
          Test whether the set is a true subset of _other_, that is,
          `set <= other and set != other'.

      -- Method: issuperset (other)
      -- Method: set >= other
          Test whether every element in _other_ is in the set.

      -- Method: set > other
          Test whether the set is a true superset of _other_, that is,
          `set >= other and set != other'.

      -- Method: union (other, ...)
      -- Method: set | other | ...
          Return a new set with elements from the set and all others.

          Changed in version 2.6: Accepts multiple input iterables.

      -- Method: intersection (other, ...)
      -- Method: set & other & ...
          Return a new set with elements common to the set and all
          others.

          Changed in version 2.6: Accepts multiple input iterables.

      -- Method: difference (other, ...)
      -- Method: set - other - ...
          Return a new set with elements in the set that are not in the
          others.

          Changed in version 2.6: Accepts multiple input iterables.

      -- Method: symmetric_difference (other)
      -- Method: set ^ other
          Return a new set with elements in either the set or _other_
          but not both.

      -- Method: copy ()
          Return a new set with a shallow copy of _s_.

     Note, the non-operator versions of union(),
     intersection(), difference(), and
     symmetric_difference(), issubset(), and
     issuperset() methods will accept any iterable as an
     argument.  In contrast, their operator based counterparts require
     their arguments to be sets.  This precludes error-prone
     constructions like `set('abc') & 'cbs'' in favor of the more
     readable `set('abc').intersection('cbs')'.

     Both set and frozenset support set to set
     comparisons. Two sets are equal if and only if every element of
     each set is contained in the other (each is a subset of the
     other). A set is less than another set if and only if the first
     set is a proper subset of the second set (is a subset, but is not
     equal). A set is greater than another set if and only if the first
     set is a proper superset of the second set (is a superset, but is
     not equal).

     Instances of set are compared to instances of
     frozenset.  based on their members.  For example, `set('abc')
     == frozenset('abc')' returns `True' and so does `set('abc') in
     set([frozenset('abc')])'.

     The subset and equality comparisons do not generalize to a
     complete ordering function.  For example, any two disjoint sets
     are not equal and are not subsets of each other, so _all_ of the
     following return `False': `a<b', `a==b', or `a>b'. Accordingly,
     sets do not implement the __cmp__().  method.

     Since sets only define partial ordering (subset relationships),
     the output of the `list.sort()' method is undefined for lists of
     sets.

     Set elements, like dictionary keys, must be hashable.

     Binary operations that mix set instances with
     frozenset.  return the type of the first operand.  For
     example: `frozenset('ab') | set('bc')' returns an instance of
     frozenset.

     The following table lists operations available for set.
     that do not apply to immutable instances of frozenset.:

      -- Method: update (other, ...)
      -- Method: set |= other | ...
          Update the set, adding elements from all others.

          Changed in version 2.6: Accepts multiple input iterables.

      -- Method: intersection_update (other, ...)
      -- Method: set &= other & ...
          Update the set, keeping only elements found in it and all
          others.

          Changed in version 2.6: Accepts multiple input iterables.

      -- Method: difference_update (other, ...)
      -- Method: set -= other | ...
          Update the set, removing elements found in others.

          Changed in version 2.6: Accepts multiple input iterables.

      -- Method: symmetric_difference_update (other)
      -- Method: set ^= other
          Update the set, keeping only elements found in either set,
          but not in both.

      -- Method: add (elem)
          Add element _elem_ to the set.

      -- Method: remove (elem)
          Remove element _elem_ from the set.  Raises KeyError
          if _elem_ is not contained in the set.

      -- Method: discard (elem)
          Remove element _elem_ from the set if it is present.

      -- Method: pop ()
          Remove and return an arbitrary element from the set.  Raises
          KeyError if the set is empty.

      -- Method: clear ()
          Remove all elements from the set.

     Note, the non-operator versions of the update(),
     intersection_update(), difference_update(), and
     symmetric_difference_update() methods will accept any
     iterable as an argument.

     Note, the _elem_ argument to the __contains__(),
     remove(), and discard() methods may be a set.  To
     support searching for an equivalent frozenset, the _elem_ set is
     temporarily mutated during the search and then restored.  During
     the search, the _elem_ set should not be read or mutated since it
     does not have a meaningful value.

See Comparison to the built-in set types.
     Differences between the sets module and the built-in
     set types.

5.5.8 Mapping Types -- `dict'
-----------------------------

A _mapping_ object maps hashable values to arbitrary
objects.  Mappings are mutable objects.  There is currently only one
standard mapping type, the _dictionary_.  (For other containers see the
built in list, set, and tuple.
classes, and the collections module.)

  A dictionary's keys are _almost_ arbitrary values.  Values that are
not hashable, that is, values containing lists, dictionaries
or other mutable types (that are compared by value rather than by
object identity) may not be used as keys.  Numeric types used for keys
obey the normal rules for numeric comparison: if two numbers compare
equal (such as `1' and `1.0') then they can be used interchangeably to
index the same dictionary entry.  (Note however, that since computers
store floating-point numbers as approximations it is usually unwise to
use them as dictionary keys.)

  Dictionaries can be created by placing a comma-separated list of
`key: value' pairs within braces, for example: `{'jack': 4098,
'sjoerd': 4127}' or `{4098: 'jack', 4127: 'sjoerd'}', or by the
dict constructor.

 -- Class: dict ([arg])
     Return a new dictionary initialized from an optional positional
     argument or from a set of keyword arguments. If no arguments are
     given, return a new empty dictionary. If the positional argument
     _arg_ is a mapping object, return a dictionary mapping the same
     keys to the same values as does the mapping object.  Otherwise the
     positional argument must be a sequence, a container that supports
     iteration, or an iterator object.  The elements of the argument
     must each also be of one of those kinds, and each must in turn
     contain exactly two objects.  The first is used as a key in the
     new dictionary, and the second as the key's value.  If a given key
     is seen more than once, the last value associated with it is
     retained in the new dictionary.

     If keyword arguments are given, the keywords themselves with their
     associated values are added as items to the dictionary. If a key
     is specified both in the positional argument and as a keyword
     argument, the value associated with the keyword is retained in the
     dictionary. For example, these all return a dictionary equal to
     `{"one": 1, "two": 2}':

        * `dict(one=1, two=2)'

        * `dict({'one': 1, 'two': 2})'

        * `dict(zip(('one', 'two'), (1, 2)))'

        * `dict([['two', 2], ['one', 1]])'

     The first example only works for keys that are valid Python
     identifiers; the others work with any valid keys.

     New in version 2.2.

     Changed in version 2.3: Support for building a dictionary from
     keyword arguments added.

     These are the operations that dictionaries support (and therefore,
     custom mapping types should support too):

      -- Description: len(d)
          Return the number of items in the dictionary _d_.

      -- Description: d[key]
          Return the item of _d_ with key _key_.  Raises a
          KeyError if _key_ is not in the map.

          New in version 2.5: If a subclass of dict defines a method
          `__missing__()', if the key _key_ is not present, the
          `d[key]' operation calls that method with the key _key_ as
          argument.  The `d[key]' operation then returns or raises
          whatever is returned or raised by the `__missing__(key)' call
          if the key is not present. No other operations or methods
          invoke `__missing__()'. If `__missing__()' is not defined,
          see KeyError is raised.  `__missing__()' must be a
          method; it cannot be an instance variable. For an example, see
          collections.defaultdict.

      -- Description: d[key] = value
          Set `d[key]' to _value_.

      -- Description: del d[key]
          Remove `d[key]' from _d_.  Raises a KeyError if
          _key_ is not in the map.

      -- Description: key in d
          Return `True' if _d_ has a key _key_, else `False'.

          New in version 2.2.

      -- Description: key not in d
          Equivalent to `not key in d'.

          New in version 2.2.

      -- Description: iter(d)
          Return an iterator over the keys of the dictionary.  This is
          a shortcut for iterkeys().

      -- Method: clear ()
          Remove all items from the dictionary.

      -- Method: copy ()
          Return a shallow copy of the dictionary.

      -- Method: fromkeys (seq[, value])
          Create a new dictionary with keys from _seq_ and values set
          to _value_.

          fromkeys() is a class method that returns a new
          dictionary. _value_ defaults to `None'.

          New in version 2.3.

      -- Method: get (key[, default])
          Return the value for _key_ if _key_ is in the dictionary,
          else _default_.  If _default_ is not given, it defaults to
          `None', so that this method never raises a KeyError.

      -- Method: has_key (key)
          Test for the presence of _key_ in the dictionary.
          has_key() is deprecated in favor of `key in d'.

      -- Method: items ()
          Return a copy of the dictionary's list of `(key, value)'
          pairs.

          *CPython implementation detail:* Keys and values are listed
          in an arbitrary order which is non-random, varies across
          Python implementations, and depends on the dictionary's
          history of insertions and deletions.

          If items(), keys(), values(),
          see iteritems(), iterkeys(), and
          itervalues() are called with no intervening
          modifications to the dictionary, the lists will directly
          correspond.  This allows the creation of `(value, key)' pairs
          using zip().: `pairs = zip(d.values(), d.keys())'.
          The same relationship holds for the iterkeys().
          and itervalues() methods: `pairs =
          zip(d.itervalues(), d.iterkeys())' provides the same value for
          `pairs'. Another way to create the same list is `pairs = [(v,
          k) for (k, v) in d.iteritems()]'.

      -- Method: iteritems ()
          Return an iterator over the dictionary's `(key, value)'
          pairs.  See the note for dict.items().

          Using iteritems() while adding or deleting
          entries in the dictionary may raise a RuntimeError
          or fail to iterate over all entries.

          New in version 2.2.

      -- Method: iterkeys ()
          Return an iterator over the dictionary's keys.  See the note
          for dict.items().

          Using iterkeys() while adding or deleting entries
          in the dictionary may raise a RuntimeError or
          fail to iterate over all entries.

          New in version 2.2.

      -- Method: itervalues ()
          Return an iterator over the dictionary's values.  See the
          note for dict.items().

          Using itervalues() while adding or deleting
          entries in the dictionary may raise a RuntimeError
          or fail to iterate over all entries.

          New in version 2.2.

      -- Method: keys ()
          Return a copy of the dictionary's list of keys.  See the note
          for dict.items().

      -- Method: pop (key[, default])
          If _key_ is in the dictionary, remove it and return its
          value, else return _default_.  If _default_ is not given and
          _key_ is not in the dictionary, a KeyError is
          raised.

          New in version 2.3.

      -- Method: popitem ()
          Remove and return an arbitrary `(key, value)' pair from the
          dictionary.

          popitem() is useful to destructively iterate over
          a dictionary, as often used in set algorithms.  If the
          dictionary is empty, calling popitem() raises a
          KeyError.

      -- Method: setdefault (key[, default])
          If _key_ is in the dictionary, return its value.  If not,
          insert _key_ with a value of _default_ and return _default_.
          _default_ defaults to `None'.

      -- Method: update ([other])
          Update the dictionary with the key/value pairs from _other_,
          overwriting existing keys.  Return `None'.

          update() accepts either another dictionary object
          or an iterable of key/value pairs (as tuples or other
          iterables of length two).  If keyword arguments are
          specified, the dictionary is then updated with those
          key/value pairs: `d.update(red=1, blue=2)'.

          Changed in version 2.4: Allowed the argument to be an
          iterable of key/value pairs and allowed keyword arguments.

      -- Method: values ()
          Return a copy of the dictionary's list of values.  See the
          note for dict.items().

      -- Method: viewitems ()
          Return a new view of the dictionary's items (`(key, value)'
          pairs).  See below for documentation of view objects.

          New in version 2.7.

      -- Method: viewkeys ()
          Return a new view of the dictionary's keys.  See below for
          documentation of view objects.

          New in version 2.7.

      -- Method: viewvalues ()
          Return a new view of the dictionary's values.  See below for
          documentation of view objects.

          New in version 2.7.

5.5.8.1 Dictionary view objects
...............................

The objects returned by dict.viewkeys(),
dict.viewvalues() and dict.viewitems() are _view
objects_.  They provide a dynamic view on the dictionary's entries,
which means that when the dictionary changes, the view reflects these
changes.

  Dictionary views can be iterated over to yield their respective data,
and support membership tests:

 -- Description: len(dictview)
     Return the number of entries in the dictionary.

 -- Description: iter(dictview)
     Return an iterator over the keys, values or items (represented as
     tuples of `(key, value)') in the dictionary.

     Keys and values are iterated over in an arbitrary order which is
     non-random, varies across Python implementations, and depends on
     the dictionary's history of insertions and deletions. If keys,
     values and items views are iterated over with no intervening
     modifications to the dictionary, the order of items will directly
     correspond.  This allows the creation of `(value, key)' pairs
     using zip().: `pairs = zip(d.values(), d.keys())'.
     Another way to create the same list is `pairs = [(v, k) for (k, v)
     in d.items()]'.

     Iterating views while adding or deleting entries in the dictionary
     may raise a RuntimeError or fail to iterate over all
     entries.

 -- Description: x in dictview
     Return `True' if _x_ is in the underlying dictionary's keys,
     values or items (in the latter case, _x_ should be a `(key,
     value)' tuple).

  Keys views are set-like since their entries are unique and hashable.
If all values are hashable, so that (key, value) pairs are unique and
hashable, then the items view is also set-like.  (Values views are not
treated as set-like since the entries are generally not unique.)  Then
these set operations are available ("other" refers either to another
view or a set):

 -- Description: dictview & other
     Return the intersection of the dictview and the other object as a
     new set.

 -- Description: dictview | other
     Return the union of the dictview and the other object as a new set.

 -- Description: dictview - other
     Return the difference between the dictview and the other object
     (all elements in _dictview_ that aren't in _other_) as a new set.

 -- Description: dictview ^ other
     Return the symmetric difference (all elements either in _dictview_
     or _other_, but not in both) of the dictview and the other object
     as a new set.

  An example of dictionary view usage:

    >>> dishes = {'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500}
    >>> keys = dishes.viewkeys()
    >>> values = dishes.viewvalues()

    >>> # iteration
    >>> n = 0
    >>> for val in values:
    ...     n += val
    >>> print(n)
    504

    >>> # keys and values are iterated over in the same order
    >>> list(keys)
    ['eggs', 'bacon', 'sausage', 'spam']
    >>> list(values)
    [2, 1, 1, 500]

    >>> # view objects are dynamic and reflect dict changes
    >>> del dishes['eggs']
    >>> del dishes['sausage']
    >>> list(keys)
    ['spam', 'bacon']

    >>> # set operations
    >>> keys & {'eggs', 'bacon', 'salad'}
    {'bacon'}

5.5.9 File Objects
------------------

File objects are implemented using C's `stdio' package and can be
created with the built-in open() function.  File objects
are also returned by some other built-in functions and methods, such as
os.popen() and os.fdopen() and the `makefile()'
method of socket objects. Temporary files can be created using the
tempfile module, and high-level file operations such as
copying, moving, and deleting files and directories can be achieved
with the shutil module.

  When a file operation fails for an I/O-related reason, the exception
IOError is raised.  This includes situations where the
operation is not defined for some reason, like `seek()' on a tty device
or writing a file opened for reading.

  Files have the following methods:

 -- Method: file.close ()
     Close the file.  A closed file cannot be read or written any more.
     Any operation which requires that the file be open will raise a
     ValueError after the file has been closed.  Calling
     close() more than once is allowed.

     As of Python 2.5, you can avoid having to call this method
     explicitly if you use the with statement.  For
     example, the following code will automatically close _f_ when the
     with block is exited:

         from __future__ import with_statement # This isn't required in Python 2.6

         with open("hello.txt") as f:
             for line in f:
                 print line

     In older versions of Python, you would have needed to do this to
     get the same effect:

         f = open("hello.txt")
         try:
             for line in f:
                 print line
         finally:
             f.close()

          Note: Not all "file-like" types in Python support use as a
          context manager for the with statement.  If your
          code is intended to work with any file-like object, you can
          use the function contextlib.closing() instead of
          using the object directly.

 -- Method: file.flush ()
     Flush the internal buffer, like `stdio''s `fflush()'.  This may be
     a no-op on some file-like objects.

          Note: flush() does not necessarily write the
          file's data to disk.  Use flush() followed by
          os.fsync() to ensure this behavior.

 -- Method: file.fileno ()
     Return the integer "file descriptor" that is used by the underlying
     implementation to request I/O operations from the operating
     system.  This can be useful for other, lower level interfaces that
     use file descriptors, such as the fcntl module or
     os.read() and friends.

          Note: File-like objects which do not have a real file
          descriptor should _not_ provide this method!

 -- Method: file.isatty ()
     Return `True' if the file is connected to a tty(-like) device,
     else `False'.

          Note: If a file-like object is not associated with a real
          file, this method should _not_ be implemented.

 -- Method: file.next ()
     A file object is its own iterator, for example `iter(f)' returns
     _f_ (unless _f_ is closed).  When a file is used as an iterator,
     typically in a for loop (for example, `for line in f:
     print line'), the next() method is called repeatedly.
     This method returns the next input line, or raises
     StopIteration when EOF is hit when the file is open for
     reading (behavior is undefined when the file is open for writing).
     In order to make a for loop the most efficient way of
     looping over the lines of a file (a very common operation), the
     next() method uses a hidden read-ahead buffer.  As a
     consequence of using a read-ahead buffer, combining next().
     with other file methods (like readline()) does
     not work right.  However, using seek() to reposition
     the file to an absolute position will flush the read-ahead buffer.

     New in version 2.3.

 -- Method: file.read ([size])
     Read at most _size_ bytes from the file (less if the read hits EOF
     before obtaining _size_ bytes).  If the _size_ argument is
     negative or omitted, read all data until EOF is reached.  The
     bytes are returned as a string object.  An empty string is
     returned when EOF is encountered immediately.  (For certain files,
     like ttys, it makes sense to continue reading after an EOF is
     hit.)  Note that this method may call the underlying C function
     `fread()' more than once in an effort to acquire as close to
     _size_ bytes as possible. Also note that when in non-blocking
     mode, less data than was requested may be returned, even if no
     _size_ parameter was given.

          Note: This function is simply a wrapper for the underlying
          `fread()' C function, and will behave the same in corner
          cases, such as whether the EOF value is cached.

 -- Method: file.readline ([size])
     Read one entire line from the file.  A trailing newline character
     is kept in the string (but may be absent when a file ends with an
     incomplete line). (5)  If the _size_ argument is present and
     non-negative, it is a maximum byte count (including the trailing
     newline) and an incomplete line may be returned. An empty string
     is returned _only_ when EOF is encountered immediately.

          Note: Unlike `stdio''s `fgets()', the returned string
          contains null characters (`'\0'') if they occurred in the
          input.

 -- Method: file.readlines ([sizehint])
     Read until EOF using readline() and return a list
     containing the lines thus read.  If the optional _sizehint_
     argument is present, instead of reading up to EOF, whole lines
     totalling approximately _sizehint_ bytes (possibly after rounding
     up to an internal buffer size) are read.  Objects implementing a
     file-like interface may choose to ignore _sizehint_ if it cannot
     be implemented, or cannot be implemented efficiently.

 -- Method: file.xreadlines ()
     This method returns the same thing as `iter(f)'.

     New in version 2.1.

     Deprecated since version 2.3: Use `for line in file' instead.

 -- Method: file.seek (offset[, whence])
     Set the file's current position, like `stdio''s `fseek()'. The
     _whence_ argument is optional and defaults to  `os.SEEK_SET' or
     `0' (absolute file positioning); other values are `os.SEEK_CUR' or
     `1' (seek relative to the current position) and `os.SEEK_END' or
     `2'  (seek relative to the file's end).  There is no return value.

     For example, `f.seek(2, os.SEEK_CUR)' advances the position by two
     and `f.seek(-3, os.SEEK_END)' sets the position to the third to
     last.

     Note that if the file is opened for appending (mode `'a'' or
     `'a+''), any seek() operations will be undone at the
     next write.  If the file is only opened for writing in append mode
     (mode `'a''), this method is essentially a no-op, but it remains
     useful for files opened in append mode with reading enabled (mode
     `'a+'').  If the file is opened in text mode (without `'b''), only
     offsets returned by tell() are legal.  Use of other
     offsets causes undefined behavior.

     Note that not all file objects are seekable.

     Changed in version 2.6: Passing float values as offset has been
     deprecated.

 -- Method: file.tell ()
     Return the file's current position, like `stdio''s `ftell()'.

          Note: On Windows, see tell() can return illegal
          values (after an `fgets()') when reading files with
          Unix-style line-endings. Use binary mode (`'rb'') to
          circumvent this problem.

 -- Method: file.truncate ([size])
     Truncate the file's size.  If the optional _size_ argument is
     present, the file is truncated to (at most) that size.  The size
     defaults to the current position.  The current file position is
     not changed.  Note that if a specified size exceeds the file's
     current size, the result is platform-dependent:  possibilities
     include that the file may remain unchanged, increase to the
     specified size as if zero-filled, or increase to the specified
     size with undefined new content.  Availability:  Windows, many
     Unix variants.

 -- Method: file.write (str)
     Write a string to the file.  There is no return value.  Due to
     buffering, the string may not actually show up in the file until
     the flush() or close() method is called.

 -- Method: file.writelines (sequence)
     Write a sequence of strings to the file.  The sequence can be any
     iterable object producing strings, typically a list of strings.
     There is no return value.  (The name is intended to match
     readlines().; writelines() does not add line
     separators.)

  Files support the iterator protocol.  Each iteration returns the same
result as `file.readline()', and iteration ends when the
readline() method returns an empty string.

  File objects also offer a number of other interesting attributes.
These are not required for file-like objects, but should be implemented
if they make sense for the particular object.

 -- Attribute: file.closed
     bool indicating the current state of the file object.  This is a
     read-only attribute; the close() method changes the
     value. It may not be available on all file-like objects.

 -- Attribute: file.encoding
     The encoding that this file uses. When Unicode strings are written
     to a file, they will be converted to byte strings using this
     encoding. In addition, when the file is connected to a terminal,
     the attribute gives the encoding that the terminal is likely to
     use (that  information might be incorrect if the user has
     misconfigured the  terminal). The attribute is read-only and may
     not be present on all file-like objects. It may also be `None', in
     which case the file uses the system default encoding for
     converting Unicode strings.

     New in version 2.3.

 -- Attribute: file.errors
     The Unicode error handler used along with the encoding.

     New in version 2.6.

 -- Attribute: file.mode
     The I/O mode for the file.  If the file was created using the
     open().  built-in function, this will be the value of
     the _mode_ parameter.  This is a read-only attribute and may not
     be present on all file-like objects.

 -- Attribute: file.name
     If the file object was created using open(), the name
     of the file.  Otherwise, some string that indicates the source of
     the file object, of the form `<...>'.  This is a read-only
     attribute and may not be present on all file-like objects.

 -- Attribute: file.newlines
     If Python was built with the `--with-universal-newlines' option to
     *configure* (the default) this read-only attribute exists, and for
     files opened in universal newline read mode it keeps track of the
     types of newlines encountered while reading the file. The values
     it can take are `'\r'', `'\n'', `'\r\n'', `None' (unknown, no
     newlines read yet) or a tuple containing all the newline types
     seen, to indicate that multiple newline conventions were
     encountered. For files not opened in universal newline read mode
     the value of this attribute will be `None'.

 -- Attribute: file.softspace
     Boolean that indicates whether a space character needs to be
     printed before another value when using the print.
     statement. Classes that are trying to simulate a file object
     should also have a writable softspace.  attribute,
     which should be initialized to zero.  This will be automatic for
     most classes implemented in Python (care may be needed for objects
     that override attribute access); types implemented in C will have
     to provide a writable softspace attribute.

          Note: This attribute is not used to control the print
          statement, but to allow the implementation of
          print to keep track of its internal state.

5.5.10 memoryview type
----------------------

New in version 2.7.

  memoryview objects allow Python code to access the
internal data of an object that supports the buffer protocol without
copying.  Memory is generally interpreted as simple bytes.

 -- Class: memoryview (obj)
     Create a memoryview that references _obj_.  _obj_ must
     support the buffer protocol.  Builtin objects that support the
     buffer protocol include str and bytearray.
     (but not unicode).

     A memoryview has the notion of an _element_, which is
     the atomic memory unit handled by the originating object _obj_.
     For many simple types such as str and bytearray,
     an element is a single byte, but other third-party types may
     expose larger elements.

     `len(view)' returns the total number of elements in the memoryview,
     _view_.  The itemsize attribute will give you the
     number of bytes in a single element.

     A memoryview supports slicing to expose its data.
     Taking a single index will return a single element as a str
     object.  Full slicing will result in a subview:

         >>> v = memoryview('abcefg')
         >>> v[1]
         'b'
         >>> v[-1]
         'g'
         >>> v[1:4]
         <memory at 0x77ab28>
         >>> v[1:4].tobytes()
         'bce'

     If the object the memoryview is over supports changing its data,
     the memoryview supports slice assignment:

         >>> data = bytearray('abcefg')
         >>> v = memoryview(data)
         >>> v.readonly
         False
         >>> v[0] = 'z'
         >>> data
         bytearray(b'zbcefg')
         >>> v[1:4] = '123'
         >>> data
         bytearray(b'z123fg')
         >>> v[2] = 'spam'
         Traceback (most recent call last):
           File "<stdin>", line 1, in <module>
         ValueError: cannot modify size of memoryview object

     Notice how the size of the memoryview object cannot be changed.

     memoryview has two methods:

      -- Method: tobytes ()
          Return the data in the buffer as a bytestring (an object of
          class str).

              >>> m = memoryview("abc")
              >>> m.tobytes()
              'abc'

      -- Method: tolist ()
          Return the data in the buffer as a list of integers.

              >>> memoryview("abc").tolist()
              [97, 98, 99]

     There are also several readonly attributes available:

      -- Attribute: format
          A string containing the format (in struct module
          style) for each element in the view.  This defaults to `'B'',
          a simple bytestring.

      -- Attribute: itemsize
          The size in bytes of each element of the memoryview.

      -- Attribute: shape
          A tuple of integers the length of ndim giving the
          shape of the memory as a N-dimensional array.

      -- Attribute: ndim
          An integer indicating how many dimensions of a
          multi-dimensional array the memory represents.

      -- Attribute: strides
          A tuple of integers the length of ndim giving the
          size in bytes to access each element for each dimension of
          the array.

5.5.11 Context Manager Types
----------------------------

New in version 2.5.

  Python's with statement supports the concept of a runtime
context defined by a context manager.  This is implemented using two
separate methods that allow user-defined classes to define a runtime
context that is entered before the statement body is executed and
exited when the statement ends.

  The _context management protocol_ consists of a pair of methods that
need to be provided for a context manager object to define a runtime
context:

 -- Method: contextmanager.__enter__ ()
     Enter the runtime context and return either this object or another
     object related to the runtime context. The value returned by this
     method is bound to the identifier in the as clause of
     with statements using this context manager.

     An example of a context manager that returns itself is a file
     object. File objects return themselves from __enter__() to allow
     open() to be used as the context expression in a
     with statement.

     An example of a context manager that returns a related object is
     the one returned by decimal.localcontext(). These
     managers set the active decimal context to a copy of the original
     decimal context and then return the copy. This allows changes to
     be made to the current decimal context in the body of the
     with statement without affecting code outside the
     with statement.

 -- Method: contextmanager.__exit__ (exc_type, exc_val, exc_tb)
     Exit the runtime context and return a Boolean flag indicating if
     any exception that occurred should be suppressed. If an exception
     occurred while executing the body of the with.
     statement, the arguments contain the exception type, value and
     traceback information. Otherwise, all three arguments are `None'.

     Returning a true value from this method will cause the with
     statement to suppress the exception and continue execution
     with the statement immediately following the with.
     statement. Otherwise the exception continues propagating after
     this method has finished executing. Exceptions that occur during
     execution of this method will replace any exception that occurred
     in the body of the with statement.

     The exception passed in should never be reraised explicitly -
     instead, this method should return a false value to indicate that
     the method completed successfully and does not want to suppress
     the raised exception. This allows context management code (such as
     `contextlib.nested') to easily detect whether or not an
     __exit__() method has actually failed.

  Python defines several context managers to support easy thread
synchronisation, prompt closure of files or other objects, and simpler
manipulation of the active decimal arithmetic context. The specific
types are not treated specially beyond their implementation of the
context management protocol. See the contextlib module for
some examples.

  Python's generator.s and the `contextlib.contextmanager'
decorator.  provide a convenient way to implement these
protocols.  If a generator function is decorated with the
`contextlib.contextmanager' decorator, it will return a context manager
implementing the necessary __enter__() and
__exit__() methods, rather than the iterator produced by an
undecorated generator function.

  Note that there is no specific slot for any of these methods in the
type structure for Python objects in the Python/C API. Extension types
wanting to define these methods must provide them as a normal Python
accessible method.  Compared to the overhead of setting up the runtime
context, the overhead of a single class dictionary lookup is negligible.

5.5.12 Other Built-in Types
---------------------------

The interpreter supports several other kinds of objects. Most of these
support only one or two operations.

5.5.12.1 Modules
................

The only special operation on a module is attribute access: `m.name',
where _m_ is a module and _name_ accesses a name defined in _m_'s
symbol table.  Module attributes can be assigned to.  (Note that the
import.  statement is not, strictly speaking, an operation
on a module object; `import foo' does not require a module object named
_foo_ to exist, rather it requires an (external) _definition_ for a
module named _foo_ somewhere.)

  A special member of every module is `__dict__'. This is the dictionary
containing the module's symbol table. Modifying this dictionary will
actually change the module's symbol table, but direct assignment to the
`__dict__' attribute is not possible (you can write `m.__dict__['a'] =
1', which defines `m.a' to be `1', but you can't write `m.__dict__ =
{}').  Modifying `__dict__' directly is not recommended.

  Modules built into the interpreter are written like this: `<module
'sys' (built-in)>'.  If loaded from a file, they are written as
`<module 'os' from '/usr/local/lib/pythonX.Y/os.pyc'>'.

5.5.12.2 Classes and Class Instances
....................................

See Objects, values and types and Class definitions
for these.

5.5.12.3 Functions
..................

Function objects are created by function definitions.  The only
operation on a function object is to call it: `func(argument-list)'.

  There are really two flavors of function objects: built-in functions
and user-defined functions.  Both support the same operation (to call
the function), but the implementation is different, hence the different
object types.

  See Function definitions for more information.

5.5.12.4 Methods
................

Methods are functions that are called using the attribute notation.
There are two flavors: built-in methods (such as `append()' on lists)
and class instance methods.  Built-in methods are described with the
types that support them.

  The implementation adds two special read-only attributes to class
instance methods: `m.im_self' is the object on which the method
operates, and `m.im_func' is the function implementing the method.
Calling `m(arg-1, arg-2, ..., arg-n)' is completely equivalent to
calling `m.im_func(m.im_self, arg-1, arg-2, ..., arg-n)'.

  Class instance methods are either _bound_ or _unbound_, referring to
whether the method was accessed through an instance or a class,
respectively.  When a method is unbound, its `im_self' attribute will
be `None' and if called, an explicit `self' object must be passed as
the first argument.  In this case, `self' must be an instance of the
unbound method's class (or a subclass of that class), otherwise a
TypeError is raised.

  Like function objects, methods objects support getting arbitrary
attributes.  However, since method attributes are actually stored on
the underlying function object (`meth.im_func'), setting method
attributes on either bound or unbound methods is disallowed.
Attempting to set a method attribute results in a TypeError.
being raised.  In order to set a method attribute, you need to
explicitly set it on the underlying function object:

    class C:
        def method(self):
            pass

    c = C()
    c.method.im_func.whoami = 'my name is c'

See The standard type hierarchy for more information.

5.5.12.5 Code Objects
.....................

Code objects are used by the implementation to represent
"pseudo-compiled" executable Python code such as a function body. They
differ from function objects because they don't contain a reference to
their global execution environment.  Code objects are returned by the
built-in compile() function and can be extracted from
function objects through their `func_code' attribute. See also the
code module.

  A code object can be executed or evaluated by passing it (instead of
a source string) to the exec statement or the built-in
eval() function.

  See The standard type hierarchy for more information.

5.5.12.6 Type Objects
.....................

Type objects represent the various object types.  An object's type is
accessed by the built-in function type().  There are no
special operations on types.  The standard module types.
defines names for all standard built-in types.

  Types are written like this: `<type 'int'>'.

5.5.12.7 The Null Object
........................

This object is returned by functions that don't explicitly return a
value.  It supports no special operations.  There is exactly one null
object, named `None' (a built-in name).

  It is written as `None'.

5.5.12.8 The Ellipsis Object
............................

This object is used by extended slice notation (see Slicings).
It supports no special operations.  There is exactly one
ellipsis object, named Ellipsis. (a built-in name).

  It is written as `Ellipsis'.

5.5.12.9 Boolean Values
.......................

Boolean values are the two constant objects `False' and `True'.  They
are used to represent truth values (although other values can also be
considered false or true).  In numeric contexts (for example when used
as the argument to an arithmetic operator), they behave like the
integers 0 and 1, respectively.  The built-in function bool()
can be used to cast any value to a Boolean, if the value can be
interpreted as a truth value (see section Truth Value Testing above).

  They are written as `False' and `True', respectively.

5.5.12.10 Internal Objects
..........................

See The standard type hierarchy for this information.  It
describes stack frame objects, traceback objects, and slice objects.

5.5.13 Special Attributes
-------------------------

The implementation adds a few special read-only attributes to several
object types, where they are relevant.  Some of these are not reported
by the dir() built-in function.

 -- Attribute: object.__dict__
     A dictionary or other mapping object used to store an object's
     (writable) attributes.

 -- Attribute: object.__methods__
     Deprecated since version 2.2: Use the built-in function
     dir() to get a list of an object's attributes.  This
     attribute is no longer available.

 -- Attribute: object.__members__
     Deprecated since version 2.2: Use the built-in function
     dir() to get a list of an object's attributes.  This
     attribute is no longer available.

 -- Attribute: instance.__class__
     The class to which a class instance belongs.

 -- Attribute: class.__bases__
     The tuple of base classes of a class object.

 -- Attribute: class.__name__
     The name of the class or type.

  The following attributes are only supported by new-style class.
es.

 -- Attribute: class.__mro__
     This attribute is a tuple of classes that are considered when
     looking for base classes during method resolution.

 -- Method: class.mro ()
     This method can be overridden by a metaclass to customize the
     method resolution order for its instances.  It is called at class
     instantiation, and its result is stored in __mro__.

 -- Method: class.__subclasses__ ()
     Each new-style class keeps a list of weak references to its
     immediate subclasses.  This method returns a list of all those
     references still alive.  Example:

         >>> int.__subclasses__()
         [<type 'bool'>]

Footnotes
.........

     (1) Additional information on these special methods may be found
     in the Python Reference Manual (see Basic customization).

     (2) As a consequence, the list `[1, 2]' is considered equal to
     `[1.0, 2.0]', and similarly for tuples.

     (3) They must have since the parser can't tell the type of the
     operands.

     (4) To format only a tuple you should therefore provide a
     singleton tuple whose only element is the tuple to be formatted.

     (5) The advantage of leaving the newline on is that returning an
     empty string is then an unambiguous EOF indication.  It is also
     possible (in cases where it might matter, for example, if you want
     to make an exact copy of a file while scanning its lines) to tell
     whether the last line of a file ended in a newline or not (yes
     this happens!).

5.6 Built-in Exceptions
=======================

Exceptions should be class objects.   The exceptions are defined in the
module exceptions.  This module never needs to be imported
explicitly: the exceptions are provided in the built-in namespace as
well as the exceptions module.

  For class exceptions, in a try statement with an
except.  clause that mentions a particular class, that clause also
handles any exception classes derived from that class (but not
exception classes from which _it_ is derived).  Two exception classes
that are not related via subclassing are never equivalent, even if they
have the same name.

  The built-in exceptions listed below can be generated by the
interpreter or built-in functions.  Except where mentioned, they have
an "associated value" indicating the detailed cause of the error. This
may be a string or a tuple containing several items of information
(e.g., an error code and a string explaining the code). The associated
value is the second argument to the raise statement.  If
the exception class is derived from the standard root class
BaseException, the associated value is present as the exception
instance's `args' attribute.

  User code can raise built-in exceptions.  This can be used to test an
exception handler or to report an error condition "just like" the
situation in which the interpreter raises the same exception; but
beware that there is nothing to prevent user code from raising an
inappropriate error.

  The built-in exception classes can be sub-classed to define new
exceptions; programmers are encouraged to at least derive new
exceptions from the Exception class and not
BaseException.  More information on defining exceptions is
available in the Python Tutorial under User-defined Exceptions.

  The following exceptions are only used as base classes for other
exceptions.

 -- Exception: BaseException
     The base class for all built-in exceptions.  It is not meant to be
     directly inherited by user-defined classes (for that use
     Exception).  If str() or unicode().
     is called on an instance of this class, the representation of the
     argument(s) to the instance are returned or the empty string when
     there were no arguments.  All arguments are  stored in `args' as a
     tuple.

     New in version 2.5.

 -- Exception: Exception
     All built-in, non-system-exiting exceptions are derived from this
     class.  All user-defined exceptions should also be derived from
     this class.

     Changed in version 2.5: Changed to inherit from
     BaseException.

 -- Exception: StandardError
     The base class for all built-in exceptions except
     StopIteration, GeneratorExit,
     KeyboardInterrupt and SystemExit.
     StandardError itself is derived from Exception.

 -- Exception: ArithmeticError
     The base class for those built-in exceptions that are raised for
     various arithmetic errors: OverflowError,
     ZeroDivisionError, FloatingPointError.

 -- Exception: LookupError
     The base class for the exceptions that are raised when a key or
     index used on a mapping or sequence is invalid: IndexError,
     see KeyError.  This can be raised directly by
     codecs.lookup().

 -- Exception: EnvironmentError
     The base class for exceptions that can occur outside the Python
     system: IOError, OSError.  When exceptions
     of this type are created with a 2-tuple, the first item is
     available on the instance's errno attribute (it is
     assumed to be an error number), and the second item is available
     on the `strerror' attribute (it is usually the associated error
     message).  The tuple itself is also available on the `args'
     attribute.

     New in version 1.5.2.

     When an EnvironmentError exception is instantiated
     with a 3-tuple, the first two items are available as above, while
     the third item is available on the `filename' attribute.  However,
     for backwards compatibility, the `args' attribute contains only a
     2-tuple of the first two constructor arguments.

     The `filename' attribute is `None' when this exception is created
     with other than 3 arguments.  The errno and `strerror'
     attributes are also `None' when the instance was created with
     other than 2 or 3 arguments.  In this last case, `args' contains
     the verbatim constructor arguments as a tuple.

  The following exceptions are the exceptions that are actually raised.

 -- Exception: AssertionError
     Raised when an assert statement fails.

 -- Exception: AttributeError
     Raised when an attribute reference (see Attribute references)
     or assignment fails.  (When an object does not
     support attribute references or attribute assignments at all,
     see TypeError is raised.)

 -- Exception: EOFError
     Raised when one of the built-in functions (see input() or
     raw_input())  hits an end-of-file condition (EOF)
     without reading any data. (N.B.: the file.read() and
     file.readline() methods return an empty string when
     they hit EOF.)

 -- Exception: FloatingPointError
     Raised when a floating point operation fails.  This exception is
     always defined, but can only be raised when Python is configured
     with the `--with-fpectl' option, or the `WANT_SIGFPE_HANDLER'
     symbol is defined in the `pyconfig.h' file.

 -- Exception: GeneratorExit
     Raise when a generator.'s `close()' method is called.
     It directly inherits from BaseException instead of
     StandardError since it is technically not an error.

     New in version 2.5.

     Changed in version 2.6: Changed to inherit from
     BaseException.

 -- Exception: IOError
     Raised when an I/O operation (such as a print.
     statement, the built-in open() function or a method of
     a file object) fails for an I/O-related reason, e.g., "file not
     found" or "disk full".

     This class is derived from EnvironmentError.  See the
     discussion above for more information on exception instance
     attributes.

     Changed in version 2.6: Changed socket.error to use
     this as a base class.

 -- Exception: ImportError
     Raised when an import statement fails to find the
     module definition or when a `from ... import' fails to find a name
     that is to be imported.

 -- Exception: IndexError
     Raised when a sequence subscript is out of range.  (Slice indices
     are silently truncated to fall in the allowed range; if an index
     is not a plain integer, see TypeError is raised.)

 -- Exception: KeyError
     Raised when a mapping (dictionary) key is not found in the set of
     existing keys.

 -- Exception: KeyboardInterrupt
     Raised when the user hits the interrupt key (normally `Control-C'
     or `Delete').  During execution, a check for interrupts is made
     regularly.  Interrupts typed when a built-in function
     input() or raw_input() is waiting for input also
     raise this exception. The exception inherits from
     BaseException so as to not be accidentally caught by code
     that catches Exception and thus prevent the
     interpreter from exiting.

     Changed in version 2.5: Changed to inherit from
     BaseException.

 -- Exception: MemoryError
     Raised when an operation runs out of memory but the situation may
     still be rescued (by deleting some objects).  The associated value
     is a string indicating what kind of (internal) operation ran out
     of memory. Note that because of the underlying memory management
     architecture (C's `malloc()' function), the interpreter may not
     always be able to completely recover from this situation; it
     nevertheless raises an exception so that a stack traceback can be
     printed, in case a run-away program was the cause.

 -- Exception: NameError
     Raised when a local or global name is not found.  This applies
     only to unqualified names.  The associated value is an error
     message that includes the name that could not be found.

 -- Exception: NotImplementedError
     This exception is derived from RuntimeError.  In user
     defined base classes, abstract methods should raise this exception
     when they require derived classes to override the method.

     New in version 1.5.2.

 -- Exception: OSError
     This exception is derived from EnvironmentError.  It is
     raised when a function returns a system-related error (not for
     illegal argument types or other incidental errors).  The
     errno attribute is a numeric error code from `errno', and the
     `strerror' attribute is the corresponding string, as would be
     printed by the C function `perror()'.  See the module errno,
     which contains names for the error codes defined by the
     underlying operating system.

     For exceptions that involve a file system path (such as `chdir()'
     or `unlink()'), the exception instance will contain a third
     attribute, `filename', which is the file name passed to the
     function.

     New in version 1.5.2.

 -- Exception: OverflowError
     Raised when the result of an arithmetic operation is too large to
     be represented.  This cannot occur for long integers (which would
     rather raise MemoryError than give up) and for most
     operations with plain integers, which return a long integer
     instead.  Because of the lack of standardization of floating point
     exception handling in C, most floating point operations also
     aren't checked.

 -- Exception: ReferenceError
     This exception is raised when a weak reference proxy, created by
     the weakref.proxy() function, is used to access an
     attribute of the referent after it has been garbage collected. For
     more information on weak references, see the weakref.
     module.

     New in version 2.2: Previously known as the
     weakref.ReferenceError exception.

 -- Exception: RuntimeError
     Raised when an error is detected that doesn't fall in any of the
     other categories.  The associated value is a string indicating
     what precisely went wrong.  (This exception is mostly a relic from
     a previous version of the interpreter; it is not used very much
     any more.)

 -- Exception: StopIteration
     Raised by an iterator.'s next() method to
     signal that there are no further values.  This is derived from
     Exception rather than StandardError, since
     this is not considered an error in its normal application.

     New in version 2.2.

 -- Exception: SyntaxError
     Raised when the parser encounters a syntax error.  This may occur
     in an import statement, in an exec.
     statement, in a call to the built-in function eval().
     or input(), or when reading the initial script or
     standard input (also interactively).

     Instances of this class have attributes `filename', `lineno',
     `offset' and `text' for easier access to the details.  See
     str().  of the exception instance returns only the message.

 -- Exception: SystemError
     Raised when the interpreter finds an internal error, but the
     situation does not look so serious to cause it to abandon all
     hope. The associated value is a string indicating what went wrong
     (in low-level terms).

     You should report this to the author or maintainer of your Python
     interpreter.  Be sure to report the version of the Python
     interpreter (`sys.version'; it is also printed at the start of an
     interactive Python session), the exact error message (the
     exception's associated value) and if possible the source of the
     program that triggered the error.

 -- Exception: SystemExit
     This exception is raised by the sys.exit() function.
     When it is not handled, the Python interpreter exits; no stack
     traceback is printed.  If the associated value is a plain integer,
     it specifies the system exit status (passed to C's `exit()'
     function); if it is `None', the exit status is zero; if it has
     another type (such as a string), the object's value is printed and
     the exit status is one.

     Instances have an attribute code which is set to the
     proposed exit status or error message (defaulting to `None').
     Also, this exception derives directly from BaseException
     and not StandardError, since it is not technically
     an error.

     A call to sys.exit() is translated into an exception
     so that clean-up handlers (see finally clauses of
     try statements) can be executed, and so that a debugger can
     execute a script without running the risk of losing control.  The
     os._exit() function can be used if it is absolutely
     positively necessary to exit immediately (for example, in the child
     process after a call to `fork()').

     The exception inherits from BaseException instead of
     StandardError.  or Exception so that it is
     not accidentally caught by code that catches Exception.
     This allows the exception to properly propagate up and cause the
     interpreter to exit.

     Changed in version 2.5: Changed to inherit from
     BaseException.

 -- Exception: TypeError
     Raised when an operation or function is applied to an object of
     inappropriate type.  The associated value is a string giving
     details about the type mismatch.

 -- Exception: UnboundLocalError
     Raised when a reference is made to a local variable in a function
     or method, but no value has been bound to that variable.  This is
     a subclass of NameError.

     New in version 2.0.

 -- Exception: UnicodeError
     Raised when a Unicode-related encoding or decoding error occurs.
     It is a subclass of ValueError.

     New in version 2.0.

 -- Exception: UnicodeEncodeError
     Raised when a Unicode-related error occurs during encoding.  It is
     a subclass of UnicodeError.

     New in version 2.3.

 -- Exception: UnicodeDecodeError
     Raised when a Unicode-related error occurs during decoding.  It is
     a subclass of UnicodeError.

     New in version 2.3.

 -- Exception: UnicodeTranslateError
     Raised when a Unicode-related error occurs during translating.  It
     is a subclass of UnicodeError.

     New in version 2.3.

 -- Exception: ValueError
     Raised when a built-in operation or function receives an argument
     that has the right type but an inappropriate value, and the
     situation is not described by a more precise exception such as
     IndexError.

 -- Exception: VMSError
     Only available on VMS.  Raised when a VMS-specific error occurs.

 -- Exception: WindowsError
     Raised when a Windows-specific error occurs or when the error
     number does not correspond to an `errno' value.  The `winerror' and
     `strerror' values are created from the return values of the
     `GetLastError()' and `FormatMessage()' functions from the Windows
     Platform API. The errno value maps the `winerror' value
     to corresponding `errno.h' values. This is a subclass of
     OSError.

     New in version 2.0.

     Changed in version 2.5: Previous versions put the `GetLastError()'
     codes into errno.

 -- Exception: ZeroDivisionError
     Raised when the second argument of a division or modulo operation
     is zero.  The associated value is a string indicating the type of
     the operands and the operation.

  The following exceptions are used as warning categories; see the
warnings.  module for more information.

 -- Exception: Warning
     Base class for warning categories.

 -- Exception: UserWarning
     Base class for warnings generated by user code.

 -- Exception: DeprecationWarning
     Base class for warnings about deprecated features.

 -- Exception: PendingDeprecationWarning
     Base class for warnings about features which will be deprecated in
     the future.

 -- Exception: SyntaxWarning
     Base class for warnings about dubious syntax

 -- Exception: RuntimeWarning
     Base class for warnings about dubious runtime behavior.

 -- Exception: FutureWarning
     Base class for warnings about constructs that will change
     semantically in the future.

 -- Exception: ImportWarning
     Base class for warnings about probable mistakes in module imports.

     New in version 2.5.

 -- Exception: UnicodeWarning
     Base class for warnings related to Unicode.

     New in version 2.5.

5.6.1 Exception hierarchy
-------------------------

The class hierarchy for built-in exceptions is:

    BaseException
     +-- SystemExit
     +-- KeyboardInterrupt
     +-- GeneratorExit
     +-- Exception
          +-- StopIteration
          +-- StandardError
          |    +-- BufferError
          |    +-- ArithmeticError
          |    |    +-- FloatingPointError
          |    |    +-- OverflowError
          |    |    +-- ZeroDivisionError
          |    +-- AssertionError
          |    +-- AttributeError
          |    +-- EnvironmentError
          |    |    +-- IOError
          |    |    +-- OSError
          |    |         +-- WindowsError (Windows)
          |    |         +-- VMSError (VMS)
          |    +-- EOFError
          |    +-- ImportError
          |    +-- LookupError
          |    |    +-- IndexError
          |    |    +-- KeyError
          |    +-- MemoryError
          |    +-- NameError
          |    |    +-- UnboundLocalError
          |    +-- ReferenceError
          |    +-- RuntimeError
          |    |    +-- NotImplementedError
          |    +-- SyntaxError
          |    |    +-- IndentationError
          |    |         +-- TabError
          |    +-- SystemError
          |    +-- TypeError
          |    +-- ValueError
          |         +-- UnicodeError
          |              +-- UnicodeDecodeError
          |              +-- UnicodeEncodeError
          |              +-- UnicodeTranslateError
          +-- Warning
               +-- DeprecationWarning
               +-- PendingDeprecationWarning
               +-- RuntimeWarning
               +-- SyntaxWarning
               +-- UserWarning
               +-- FutureWarning
    	   +-- ImportWarning
    	   +-- UnicodeWarning
    	   +-- BytesWarning

5.7 String Services
===================

The modules described in this chapter provide a wide range of string
manipulation operations.

  In addition, Python's built-in string classes support the sequence
type methods described in the Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange
section, and also the
string-specific methods described in the String Methods.
section.  To output formatted strings use template strings or the `%'
operator described in the String Formatting Operations.
section. Also, see the re module for string functions based
on regular expressions.

5.7.1 `string' -- Common string operations
------------------------------------------

The string module contains a number of useful constants and
classes, as well as some deprecated legacy functions that are also
available as methods on strings. In addition, Python's built-in string
classes support the sequence type methods described in the
Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange
section, and also the string-specific methods described in the
String Methods section. To output formatted strings use
template strings or the `%' operator described in the String Formatting Operations
section. Also, see the re.
module for string functions based on regular expressions.

  Latest version of the string module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/string.py?view=markup)

5.7.1.1 String constants
........................

The constants defined in this module are:

 -- Data: string.ascii_letters
     The concatenation of the ascii_lowercase and
     ascii_uppercase.  constants described below.  This value is
     not locale-dependent.

 -- Data: string.ascii_lowercase
     The lowercase letters `'abcdefghijklmnopqrstuvwxyz''.  This value
     is not locale-dependent and will not change.

 -- Data: string.ascii_uppercase
     The uppercase letters `'ABCDEFGHIJKLMNOPQRSTUVWXYZ''.  This value
     is not locale-dependent and will not change.

 -- Data: string.digits
     The string `'0123456789''.

 -- Data: string.hexdigits
     The string `'0123456789abcdefABCDEF''.

 -- Data: string.letters
     The concatenation of the strings lowercase and
     uppercase.  described below.  The specific value is
     locale-dependent, and will be updated when
     locale.setlocale() is called.

 -- Data: string.lowercase
     A string containing all the characters that are considered
     lowercase letters.  On most systems this is the string
     `'abcdefghijklmnopqrstuvwxyz''.  The specific value is
     locale-dependent, and will be updated when
     locale.setlocale() is called.

 -- Data: string.octdigits
     The string `'01234567''.

 -- Data: string.punctuation
     String of ASCII characters which are considered punctuation
     characters in the `C' locale.

 -- Data: string.printable
     String of characters which are considered printable.  This is a
     combination of digits, letters,
     punctuation, and whitespace.

 -- Data: string.uppercase
     A string containing all the characters that are considered
     uppercase letters.  On most systems this is the string
     `'ABCDEFGHIJKLMNOPQRSTUVWXYZ''.  The specific value is
     locale-dependent, and will be updated when
     locale.setlocale() is called.

 -- Data: string.whitespace
     A string containing all characters that are considered whitespace.
     On most systems this includes the characters space, tab, linefeed,
     return, formfeed, and vertical tab.

5.7.1.2 String Formatting
.........................

New in version 2.6.

  The built-in str and unicode classes provide the ability to do
complex variable substitutions and value formatting via the
str.format() method described in PEP 3101
(http://www.python.org/dev/peps/pep-3101).  The Formatter.
class in the string module allows you to create and
customize your own string formatting behaviors using the same
implementation as the built-in format() method.

 -- Class: string.Formatter
     The Formatter class has the following public methods:

      -- Method: format (format_string, *args, *kwargs)
          format() is the primary API method.  It takes a
          format template string, and an arbitrary set of positional
          and keyword argument.  format() is just a wrapper
          that calls vformat().

      -- Method: vformat (format_string, args, kwargs)
          This function does the actual work of formatting.  It is
          exposed as a separate function for cases where you want to
          pass in a predefined dictionary of arguments, rather than
          unpacking and repacking the dictionary as individual
          arguments using the `*args' and `**kwds' syntax.
          vformat() does the work of breaking up the format
          template string into character data and replacement fields.
          It calls the various methods described below.

     In addition, the Formatter defines a number of methods
     that are intended to be replaced by subclasses:

      -- Method: parse (format_string)
          Loop over the format_string and return an iterable of tuples
          (_literal_text_, _field_name_, _format_spec_, _conversion_).
          This is used by vformat() to break the string
          into either literal text, or replacement fields.

          The values in the tuple conceptually represent a span of
          literal text followed by a single replacement field.  If
          there is no literal text (which can happen if two replacement
          fields occur consecutively), then _literal_text_ will be a
          zero-length string.  If there is no replacement field, then
          the values of _field_name_, _format_spec_ and _conversion_
          will be `None'.

      -- Method: get_field (field_name, args, kwargs)
          Given _field_name_ as returned by parse(). (see
          above), convert it to an object to be formatted.  Returns a
          tuple (obj, used_key).  The default version takes strings of
          the form defined in PEP 3101
          (http://www.python.org/dev/peps/pep-3101), such as "0[name]"
          or "label.title".  _args_ and _kwargs_ are as passed in to
          vformat().  The return value _used_key_ has the
          same meaning as the _key_ parameter to get_value().

      -- Method: get_value (key, args, kwargs)
          Retrieve a given field value.  The _key_ argument will be
          either an integer or a string.  If it is an integer, it
          represents the index of the positional argument in _args_; if
          it is a string, then it represents a named argument in
          _kwargs_.

          The _args_ parameter is set to the list of positional
          arguments to vformat(), and the _kwargs_ parameter
          is set to the dictionary of keyword arguments.

          For compound field names, these functions are only called for
          the first component of the field name; Subsequent components
          are handled through normal attribute and indexing operations.

          So for example, the field expression '0.name' would cause
          get_value() to be called with a _key_ argument of
          0.  The `name' attribute will be looked up after
          get_value() returns by calling the built-in
          getattr() function.

          If the index or keyword refers to an item that does not
          exist, then an IndexError or KeyError.
          should be raised.

      -- Method: check_unused_args (used_args, args, kwargs)
          Implement checking for unused arguments if desired.  The
          arguments to this function is the set of all argument keys
          that were actually referred to in the format string (integers
          for positional arguments, and strings for named arguments),
          and a reference to the _args_ and _kwargs_ that was passed to
          vformat.  The set of unused args can be calculated from these
          parameters.  check_unused_args() is assumed to
          raise an exception if the check fails.

      -- Method: format_field (value, format_spec)
          format_field() simply calls the global
          format() built-in.  The method is provided so that
          subclasses can override it.

      -- Method: convert_field (value, conversion)
          Converts the value (returned by get_field())
          given a conversion type (as in the tuple returned by the
          parse() method).  The default version understands
          'r' (repr) and 's' (str) conversion types.

5.7.1.3 Format String Syntax
............................

The str.format() method and the Formatter class
share the same syntax for format strings (although in the case of
Formatter, subclasses can define their own format string syntax).

  Format strings contain "replacement fields" surrounded by curly
braces `{}'.  Anything that is not contained in braces is considered
literal text, which is copied unchanged to the output.  If you need to
include a brace character in the literal text, it can be escaped by
doubling: `{{' and `}}'.

  The grammar for a replacement field is as follows:

         replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"
         field_name        ::= arg_name ("." attribute_name | "[" element_index "]")*
         arg_name          ::= [identifier | integer]
         attribute_name    ::= identifier
         element_index     ::= integer | index_string
         index_string      ::= <any source character except "]"> +
         conversion        ::= "r" | "s"
         format_spec       ::= <described in the next section>

  In less formal terms, the replacement field can start with a
_field_name_ that specifies the object whose value is to be formatted
and inserted into the output instead of the replacement field.  The
_field_name_ is optionally followed by a  _conversion_ field, which is
preceded by an exclamation point `'!'', and a _format_spec_, which is
preceded by a colon `':''.  These specify a non-default format for the
replacement value.

  See also the Format Specification Mini-Language section.

  The _field_name_ itself begins with an _arg_name_ that is either
either a number or a keyword.  If it's a number, it refers to a
positional argument, and if it's a keyword, it refers to a named
keyword argument.  If the numerical arg_names in a format string are 0,
1, 2, ... in sequence, they can all be omitted (not just some) and the
numbers 0, 1, 2, ... will be automatically inserted in that order.  The
_arg_name_ can be followed by any number of index or attribute
expressions. An expression of the form `'.name'' selects the named
attribute using getattr(), while an expression of the form
`'[index]'' does an index lookup using __getitem__().

  Changed in version 2.7: The positional argument specifiers can be
omitted, so `'{} {}'' is equivalent to `'{0} {1}''.

  Some simple format string examples:

    "First, thou shalt count to {0}" # References first positional argument
    "Bring me a {}"                  # Implicitly references the first positional argument
    "From {} to {}"                  # Same as "From {0} to {1}"
    "My quest is {name}"             # References keyword argument 'name'
    "Weight in tons {0.weight}"      # 'weight' attribute of first positional arg
    "Units destroyed: {players[0]}"  # First element of keyword argument 'players'.

The _conversion_ field causes a type coercion before formatting.
Normally, the job of formatting a value is done by the `__format__()'
method of the value itself.  However, in some cases it is desirable to
force a type to be formatted as a string, overriding its own definition
of formatting.  By converting the value to a string before calling
`__format__()', the normal formatting logic is bypassed.

  Two conversion flags are currently supported: `'!s'' which calls
str().  on the value, and `'!r'' which calls repr().

  Some examples:

    "Harold's a clever {0!s}"        # Calls str() on the argument first
    "Bring out the holy {name!r}"    # Calls repr() on the argument first

The _format_spec_ field contains a specification of how the value
should be presented, including such details as field width, alignment,
padding, decimal precision and so on.  Each value type can define its
own "formatting mini-language" or interpretation of the _format_spec_.

  Most built-in types support a common formatting mini-language, which
is described in the next section.

  A _format_spec_ field can also include nested replacement fields
within it.  These nested replacement fields can contain only a field
name; conversion flags and format specifications are not allowed.  The
replacement fields within the format_spec are substituted before the
_format_spec_ string is interpreted.  This allows the formatting of a
value to be dynamically specified.

  See the Format examples section for some examples.

5.7.1.4 Format Specification Mini-Language
..........................................

"Format specifications" are used within replacement fields contained
within a format string to define how individual values are presented
(see Format String Syntax).  They can also be passed
directly to the built-in format() function.  Each
formattable type may define how the format specification is to be
interpreted.

  Most built-in types implement the following options for format
specifications, although some of the formatting options are only
supported by the numeric types.

  A general convention is that an empty format string (`""') produces
the same result as if you had called str() on the value. A
non-empty format string typically modifies the result.

  The general form of a _standard format specifier_ is:

    format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]
    fill        ::= <a character other than '}'>
    align       ::= "<" | ">" | "=" | "^"
    sign        ::= "+" | "-" | " "
    width       ::= integer
    precision   ::= integer
    type        ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"

The _fill_ character can be any character other than '{' or '}'.  The
presence of a fill character is signaled by the character following it,
which must be one of the alignment options.  If the second character of
_format_spec_ is not a valid alignment option, then it is assumed that
both the fill character and the alignment option are absent.

  The meaning of the various alignment options is as follows:

     Option        Meaning
     -----------------------------------------------------------------------------
     `'<''         Forces the field to be left-aligned within the available
                   space (this is the default).
     `'>''         Forces the field to be right-aligned within the available
                   space.
     `'=''         Forces the padding to be placed after the sign (if any) but
                   before the digits.  This is used for printing fields in the
                   form '+000000120'. This alignment option is only valid for
                   numeric types.
     `'^''         Forces the field to be centered within the available space.

  Note that unless a minimum field width is defined, the field width
will always be the same size as the data to fill it, so that the
alignment option has no meaning in this case.

  The _sign_ option is only valid for number types, and can be one of
the following:

     Option        Meaning
     -----------------------------------------------------------------------------
     `'+''         indicates that a sign should be used for both positive as
                   well as negative numbers.
     `'-''         indicates that a sign should be used only for negative
                   numbers (this is the default behavior).
     space         indicates that a leading space should be used on positive
                   numbers, and a minus sign on negative numbers.

  The `'#'' option is only valid for integers, and only for binary,
octal, or hexadecimal output.  If present, it specifies that the output
will be prefixed by `'0b'', `'0o'', or `'0x'', respectively.

  The `','' option signals the use of a comma for a thousands separator.
For a locale aware separator, use the `'n'' integer presentation type
instead.

  Changed in version 2.7: Added the `','' option (see also PEP 378
(http://www.python.org/dev/peps/pep-0378)).

  _width_ is a decimal integer defining the minimum field width.  If not
specified, then the field width will be determined by the content.

  If the _width_ field is preceded by a zero (`'0'') character, this
enables zero-padding.  This is equivalent to an _alignment_ type of
`'='' and a _fill_ character of `'0''.

  The _precision_ is a decimal number indicating how many digits should
be displayed after the decimal point for a floating point value
formatted with `'f'' and `'F'', or before and after the decimal point
for a floating point value formatted with `'g'' or `'G''.  For
non-number types the field indicates the maximum field size - in other
words, how many characters will be used from the field content. The
_precision_ is not allowed for integer values.

  Finally, the _type_ determines how the data should be presented.

  The available string presentation types are:

     Type          Meaning
     -----------------------------------------------------------------------------
     `'s''         String format. This is the default type for strings and may
                   be omitted.
     None          The same as `'s''.

  The available integer presentation types are:

     Type          Meaning
     -----------------------------------------------------------------------------
     `'b''         Binary format. Outputs the number in base 2.
     `'c''         Character. Converts the integer to the corresponding unicode
                   character before printing.
     `'d''         Decimal Integer. Outputs the number in base 10.
     `'o''         Octal format. Outputs the number in base 8.
     `'x''         Hex format. Outputs the number in base 16, using lower- case
                   letters for the digits above 9.
     `'X''         Hex format. Outputs the number in base 16, using upper- case
                   letters for the digits above 9.
     `'n''         Number. This is the same as `'d'', except that it uses the
                   current locale setting to insert the appropriate number
                   separator characters.
     None          The same as `'d''.

  In addition to the above presentation types, integers can be formatted
with the floating point presentation types listed below (except `'n''
and None). When doing so, see float() is used to convert the
integer to a floating point number before formatting.

  The available presentation types for floating point and decimal
values are:

     Type          Meaning
     -----------------------------------------------------------------------------
     `'e''         Exponent notation. Prints the number in scientific notation
                   using the letter 'e' to indicate the exponent.
     `'E''         Exponent notation. Same as `'e'' except it uses an upper case
                   'E' as the separator character.
     `'f''         Fixed point. Displays the number as a fixed-point number.
     `'F''         Fixed point. Same as `'f''.
     `'g''         General format.  For a given precision `p >= 1', this rounds
                   the number to `p' significant digits and then formats the
                   result in either fixed-point format or in scientific
                   notation, depending on its magnitude.

                   The precise rules are as follows: suppose that the result
                   formatted with presentation type `'e'' and precision `p-1'
                   would have exponent `exp'.  Then if `-4 <= exp < p', the
                   number is formatted with presentation type `'f'' and precision
                   `p-1-exp'.  Otherwise, the number is formatted with
                   presentation type `'e'' and precision `p-1'.  In both cases
                   insignificant trailing zeros are removed from the
                   significand, and the decimal point is also removed if there
                   are no remaining digits following it.

                   Positive and negative infinity, positive and negative zero,
                   and nans, are formatted as `inf', `-inf', `0', `-0' and `nan'
                   respectively, regardless of the precision.

                   A precision of `0' is treated as equivalent to a precision of
                   `1'.
     `'G''         General format. Same as `'g'' except switches to `'E'' if the
                   number gets too large. The representations of infinity and
                   NaN are uppercased, too.
     `'n''         Number. This is the same as `'g'', except that it uses the
                   current locale setting to insert the appropriate number
                   separator characters.
     `'%''         Percentage. Multiplies the number by 100 and displays in
                   fixed (`'f'') format, followed by a percent sign.
     None          The same as `'g''.

5.7.1.5 Format examples
.......................

This section contains examples of the new format syntax and comparison
with the old `%'-formatting.

  In most of the cases the syntax is similar to the old `%'-formatting,
with the addition of the `{}' and with `:' used instead of `%'.  For
example, `'%03.2f'' can be translated to `'{:03.2f}''.

  The new format syntax also supports new and different options, shown
in the follow examples.

  Accessing arguments by position:

    >>> '{0}, {1}, {2}'.format('a', 'b', 'c')
    'a, b, c'
    >>> '{}, {}, {}'.format('a', 'b', 'c')  # 2.7+ only
    'a, b, c'
    >>> '{2}, {1}, {0}'.format('a', 'b', 'c')
    'c, b, a'
    >>> '{2}, {1}, {0}'.format(*'abc')      # unpacking argument sequence
    'c, b, a'
    >>> '{0}{1}{0}'.format('abra', 'cad')   # arguments' indices can be repeated
    'abracadabra'

Accessing arguments by name:

    >>> 'Coordinates: {latitude}, {longitude}'.format(latitude='37.24N', longitude='-115.81W')
    'Coordinates: 37.24N, -115.81W'
    >>> coord = {'latitude': '37.24N', 'longitude': '-115.81W'}
    >>> 'Coordinates: {latitude}, {longitude}'.format(**coord)
    'Coordinates: 37.24N, -115.81W'

Accessing arguments' attributes:

    >>> c = 3-5j
    >>> ('The complex number {0} is formed from the real part {0.real} '
    ...  'and the imaginary part {0.imag}.').format(c)
    'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.'
    >>> class Point(object):
    ...     def __init__(self, x, y):
    ...         self.x, self.y = x, y
    ...     def __str__(self):
    ...         return 'Point({self.x}, {self.y})'.format(self=self)
    ...
    >>> str(Point(4, 2))
    'Point(4, 2)'

Accessing arguments' items:

    >>> coord = (3, 5)
    >>> 'X: {0[0]};  Y: {0[1]}'.format(coord)
    'X: 3;  Y: 5'

Replacing `%s' and `%r':

    >>> "repr() shows quotes: {!r}; str() doesn't: {!s}".format('test1', 'test2')
    "repr() shows quotes: 'test1'; str() doesn't: test2"

Aligning the text and specifying a width:

    >>> '{:<30}'.format('left aligned')
    'left aligned                  '
    >>> '{:>30}'.format('right aligned')
    '                 right aligned'
    >>> '{:^30}'.format('centered')
    '           centered           '
    >>> '{:*^30}'.format('centered')  # use '*' as a fill char
    '***********centered***********'

Replacing `%+f', `%-f', and `% f' and specifying a sign:

    >>> '{:+f}; {:+f}'.format(3.14, -3.14)  # show it always
    '+3.140000; -3.140000'
    >>> '{: f}; {: f}'.format(3.14, -3.14)  # show a space for positive numbers
    ' 3.140000; -3.140000'
    >>> '{:-f}; {:-f}'.format(3.14, -3.14)  # show only the minus -- same as '{:f}; {:f}'
    '3.140000; -3.140000'

Replacing `%x' and `%o' and converting the value to different bases:

    >>> # format also supports binary numbers
    >>> "int: {0:d};  hex: {0:x};  oct: {0:o};  bin: {0:b}".format(42)
    'int: 42;  hex: 2a;  oct: 52;  bin: 101010'
    >>> # with 0x, 0o, or 0b as prefix:
    >>> "int: {0:d};  hex: {0:#x};  oct: {0:#o};  bin: {0:#b}".format(42)
    'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010'

Using the comma as a thousands separator:

    >>> '{:,}'.format(1234567890)
    '1,234,567,890'

Expressing a percentage:

    >>> points = 19.5
    >>> total = 22
    >>> 'Correct answers: {:.2%}.'.format(points/total)
    'Correct answers: 88.64%'

Using type-specific formatting:

    >>> import datetime
    >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)
    >>> '{:%Y-%m-%d %H:%M:%S}'.format(d)
    '2010-07-04 12:15:58'

Nesting arguments and more complex examples:

    >>> for align, text in zip('<^>', ['left', 'center', 'right']):
    ...     '{0:{align}{fill}16}'.format(text, fill=align, align=align)
    ...
    'left<<<<<<<<<<<<'
    '^^^^^center^^^^^'
    '>>>>>>>>>>>right'
    >>>
    >>> octets = [192, 168, 0, 1]
    >>> '{:02X}{:02X}{:02X}{:02X}'.format(*octets)
    'C0A80001'
    >>> int(_, 16)
    3232235521
    >>>
    >>> width = 5
    >>> for num in range(5,12):
    ...     for base in 'dXob':
    ...         print '{0:{width}{base}}'.format(num, base=base, width=width),
    ...     print
    ...
        5     5     5   101
        6     6     6   110
        7     7     7   111
        8     8    10  1000
        9     9    11  1001
       10     A    12  1010
       11     B    13  1011

5.7.1.6 Template strings
........................

New in version 2.4.

  Templates provide simpler string substitutions as described in PEP
292 (http://www.python.org/dev/peps/pep-0292).  Instead of the normal
`%'-based substitutions, Templates support `$'-based substitutions,
using the following rules:

   * `$$' is an escape; it is replaced with a single `$'.

   * `$identifier' names a substitution placeholder matching a mapping
     key of `"identifier"'.  By default, `"identifier"' must spell a
     Python identifier.  The first non-identifier character after the
     `$' character terminates this placeholder specification.

   * `${identifier}' is equivalent to `$identifier'.  It is required
     when valid identifier characters follow the placeholder but are
     not part of the placeholder, such as `"${noun}ification"'.

  Any other appearance of `$' in the string will result in a
ValueError.  being raised.

  The string module provides a Template class
that implements these rules.  The methods of Template are:

 -- Class: string.Template (template)
     The constructor takes a single argument which is the template
     string.

      -- Method: substitute (mapping[, **kws])
          Performs the template substitution, returning a new string.
          _mapping_ is any dictionary-like object with keys that match
          the placeholders in the template.  Alternatively, you can
          provide keyword arguments, where the keywords are the
          placeholders.  When both _mapping_ and _kws_ are given and
          there are duplicates, the placeholders from _kws_ take
          precedence.

      -- Method: safe_substitute (mapping[, **kws])
          Like substitute(), except that if placeholders are
          missing from _mapping_ and _kws_, instead of raising a
          KeyError exception, the original placeholder will
          appear in the resulting string intact.  Also, unlike with
          substitute(), any other appearances of the `$' will
          simply return `$' instead of raising ValueError.

          While other exceptions may still occur, this method is called
          "safe" because substitutions always tries to return a usable
          string instead of raising an exception.  In another sense,
          see safe_substitute() may be anything other than
          safe, since it will silently ignore malformed templates
          containing dangling delimiters, unmatched braces, or
          placeholders that are not valid Python identifiers.

     Template instances also provide one public data
     attribute:

      -- Attribute: template
          This is the object passed to the constructor's _template_
          argument.  In general, you shouldn't change it, but read-only
          access is not enforced.

  Here is an example of how to use a Template:

    >>> from string import Template
    >>> s = Template('$who likes $what')
    >>> s.substitute(who='tim', what='kung pao')
    'tim likes kung pao'
    >>> d = dict(who='tim')
    >>> Template('Give $who $100').substitute(d)
    Traceback (most recent call last):
    [...]
    ValueError: Invalid placeholder in string: line 1, col 10
    >>> Template('$who likes $what').substitute(d)
    Traceback (most recent call last):
    [...]
    KeyError: 'what'
    >>> Template('$who likes $what').safe_substitute(d)
    'tim likes $what'

Advanced usage: you can derive subclasses of Template to
customize the placeholder syntax, delimiter character, or the entire
regular expression used to parse template strings.  To do this, you can
override these class attributes:

   * _delimiter_ - This is the literal string describing a placeholder
     introducing delimiter.  The default value `$'.  Note that this
     should _not_ be a regular expression, as the implementation will
     call re.escape() on this string as needed.

   * _idpattern_ - This is the regular expression describing the
     pattern for non-braced placeholders (the braces will be added
     automatically as appropriate).  The default value is the regular
     expression `[_a-z][_a-z0-9]*'.

  Alternatively, you can provide the entire regular expression pattern
by overriding the class attribute _pattern_.  If you do this, the value
must be a regular expression object with four named capturing groups.
The capturing groups correspond to the rules given above, along with
the invalid placeholder rule:

   * _escaped_ - This group matches the escape sequence, e.g. `$$', in
     the default pattern.

   * _named_ - This group matches the unbraced placeholder name; it
     should not include the delimiter in capturing group.

   * _braced_ - This group matches the brace enclosed placeholder name;
     it should not include either the delimiter or braces in the
     capturing group.

   * _invalid_ - This group matches any other delimiter pattern
     (usually a single delimiter), and it should appear last in the
     regular expression.

5.7.1.7 String functions
........................

The following functions are available to operate on string and Unicode
objects.  They are not available as string methods.

 -- Function: string.capwords (s[, sep])
     Split the argument into words using str.split(),
     capitalize each word using str.capitalize(), and join
     the capitalized words using str.join().  If the
     optional second argument _sep_ is absent or `None', runs of
     whitespace characters are replaced by a single space and leading
     and trailing whitespace are removed, otherwise _sep_ is used to
     split and join the words.

 -- Function: string.maketrans (from, to)
     Return a translation table suitable for passing to
     translate(), that will map each character in _from_ into the
     character at the same position in _to_; _from_ and _to_ must have
     the same length.

          Note: Don't use strings derived from lowercase.
          and uppercase as arguments; in some locales,
          these don't have the same length.  For case conversions,
          always use str.lower() and str.upper().

5.7.1.8 Deprecated string functions
...................................

The following list of functions are also defined as methods of string
and Unicode objects; see section String Methods for more
information on those.  You should consider these functions as
deprecated, although they will not be removed until Python 3.0.  The
functions defined in this module are:

 -- Function: string.atof (s)
     Deprecated since version 2.0: Use the float() built-in
     function.

     Convert a string to a floating point number.  The string must have
     the standard syntax for a floating point literal in Python,
     optionally preceded by a sign (`+' or `-').  Note that this
     behaves identical to the built-in function float().
     when passed a string.

          Note: When passing in a string, values for NaN and Infinity
          may be returned, depending on the underlying C library.  The
          specific set of strings accepted which cause these values to
          be returned depends entirely on the C library and is known to
          vary.

 -- Function: string.atoi (s[, base])
     Deprecated since version 2.0: Use the int() built-in
     function.

     Convert string _s_ to an integer in the given _base_.  The string
     must consist of one or more digits, optionally preceded by a sign
     (`+' or `-').  The _base_ defaults to 10.  If it is 0, a default
     base is chosen depending on the leading characters of the string
     (after stripping the sign): `0x' or `0X' means 16, `0' means 8,
     anything else means 10.  If _base_ is 16, a leading `0x' or `0X'
     is always accepted, though not required.  This behaves identically
     to the built-in function int() when passed a string.
     (Also note: for a more flexible interpretation of numeric
     literals, use the built-in function eval())

 -- Function: string.atol (s[, base])
     Deprecated since version 2.0: Use the long() built-in
     function.

     Convert string _s_ to a long integer in the given _base_. The
     string must consist of one or more digits, optionally preceded by
     a sign (`+' or `-').  The _base_ argument has the same meaning as
     for atoi().  A trailing `l' or `L' is not allowed,
     except if the base is 0.  Note that when invoked without _base_ or
     with _base_ set to 10, this behaves identical to the built-in
     function long() when passed a string.

 -- Function: string.capitalize (word)
     Return a copy of _word_ with only its first character capitalized.

 -- Function: string.expandtabs (s[, tabsize])
     Expand tabs in a string replacing them by one or more spaces,
     depending on the current column and the given tab size.  The
     column number is reset to zero after each newline occurring in the
     string. This doesn't understand other non-printing characters or
     escape sequences.  The tab size defaults to 8.

 -- Function: string.find (s, sub[, start[, end]])
     Return the lowest index in _s_ where the substring _sub_ is found
     such that _sub_ is wholly contained in `s[start:end]'.  Return
     `-1' on failure.  Defaults for _start_ and _end_ and
     interpretation of negative values is the same as for slices.

 -- Function: string.rfind (s, sub[, start[, end]])
     Like find() but find the highest index.

 -- Function: string.index (s, sub[, start[, end]])
     Like find() but raise ValueError when the
     substring is not found.

 -- Function: string.rindex (s, sub[, start[, end]])
     Like rfind() but raise ValueError when the
     substring is not found.

 -- Function: string.count (s, sub[, start[, end]])
     Return the number of (non-overlapping) occurrences of substring
     _sub_ in string `s[start:end]'. Defaults for _start_ and _end_ and
     interpretation of negative values are the same as for slices.

 -- Function: string.lower (s)
     Return a copy of _s_, but with upper case letters converted to
     lower case.

 -- Function: string.split (s[, sep[, maxsplit]])
     Return a list of the words of the string _s_.  If the optional
     second argument _sep_ is absent or `None', the words are separated
     by arbitrary strings of whitespace characters (space, tab,
     newline, return, formfeed).  If the second argument _sep_ is
     present and not `None', it specifies a string to be used as the
     word separator.  The returned list will then have one more item
     than the number of non-overlapping occurrences of the separator in
     the string.  The optional third argument _maxsplit_ defaults to 0.
     If it is nonzero, at most _maxsplit_ number of splits occur, and
     the remainder of the string is returned as the final element of
     the list (thus, the list will have at most `maxsplit+1' elements).

     The behavior of split on an empty string depends on the value of
     _sep_. If _sep_ is not specified, or specified as `None', the
     result will be an empty list.  If _sep_ is specified as any
     string, the result will be a list containing one element which is
     an empty string.

 -- Function: string.rsplit (s[, sep[, maxsplit]])
     Return a list of the words of the string _s_, scanning _s_ from
     the end.  To all intents and purposes, the resulting list of words
     is the same as returned by split(), except when the
     optional third argument _maxsplit_ is explicitly specified and
     nonzero.  When _maxsplit_ is nonzero, at most _maxsplit_ number of
     splits - the _rightmost_ ones - occur, and the remainder of the
     string is returned as the first element of the list (thus, the
     list will have at most `maxsplit+1' elements).

     New in version 2.4.

 -- Function: string.splitfields (s[, sep[, maxsplit]])
     This function behaves identically to split().  (In the
     past, see split().  was only used with one argument, while
     splitfields() was only used with two arguments.)

 -- Function: string.join (words[, sep])
     Concatenate a list or tuple of words with intervening occurrences
     of  _sep_.  The default value for _sep_ is a single space
     character.  It is always true that `string.join(string.split(s,
     sep), sep)' equals _s_.

 -- Function: string.joinfields (words[, sep])
     This function behaves identically to join().  (In the
     past,  see join().  was only used with one argument, while
     joinfields() was only used with two arguments.) Note
     that there is no joinfields() method on string objects;
     use the join() method instead.

 -- Function: string.lstrip (s[, chars])
     Return a copy of the string with leading characters removed.  If
     _chars_ is omitted or `None', whitespace characters are removed.
     If given and not `None', _chars_ must be a string; the characters
     in the string will be stripped from the beginning of the string
     this method is called on.

     Changed in version 2.2.3: The _chars_ parameter was added.  The
     _chars_ parameter cannot be passed in earlier 2.2 versions.

 -- Function: string.rstrip (s[, chars])
     Return a copy of the string with trailing characters removed.  If
     _chars_ is omitted or `None', whitespace characters are removed.
     If given and not `None', _chars_ must be a string; the characters
     in the string will be stripped from the end of the string this
     method is called on.

     Changed in version 2.2.3: The _chars_ parameter was added.  The
     _chars_ parameter cannot be passed in earlier 2.2 versions.

 -- Function: string.strip (s[, chars])
     Return a copy of the string with leading and trailing characters
     removed.  If _chars_ is omitted or `None', whitespace characters
     are removed.  If given and not `None', _chars_ must be a string;
     the characters in the string will be stripped from the both ends
     of the string this method is called on.

     Changed in version 2.2.3: The _chars_ parameter was added.  The
     _chars_ parameter cannot be passed in earlier 2.2 versions.

 -- Function: string.swapcase (s)
     Return a copy of _s_, but with lower case letters converted to
     upper case and vice versa.

 -- Function: string.translate (s, table[, deletechars])
     Delete all characters from _s_ that are in _deletechars_ (if
     present), and then translate the characters using _table_, which
     must be a 256-character string giving the translation for each
     character value, indexed by its ordinal.  If _table_ is `None',
     then only the character deletion step is performed.

 -- Function: string.upper (s)
     Return a copy of _s_, but with lower case letters converted to
     upper case.

 -- Function: string.ljust (s, width[, fillchar])
 -- Function: string.rjust (s, width[, fillchar])
 -- Function: string.center (s, width[, fillchar])
     These functions respectively left-justify, right-justify and
     center a string in a field of given width.  They return a string
     that is at least _width_ characters wide, created by padding the
     string _s_ with the character _fillchar_ (default is a space)
     until the given width on the right, left or both sides.  The
     string is never truncated.

 -- Function: string.zfill (s, width)
     Pad a numeric string on the left with zero digits until the given
     width is reached.  Strings starting with a sign are handled
     correctly.

 -- Function: string.replace (str, old, new[, maxreplace])
     Return a copy of string _str_ with all occurrences of substring
     _old_ replaced by _new_.  If the optional argument _maxreplace_ is
     given, the first _maxreplace_ occurrences are replaced.

5.7.2 `re' -- Regular expression operations
-------------------------------------------

This module provides regular expression matching operations similar to
those found in Perl. Both patterns and strings to be searched can be
Unicode strings as well as 8-bit strings.

  Regular expressions use the backslash character (`'\'') to indicate
special forms or to allow special characters to be used without invoking
their special meaning.  This collides with Python's usage of the same
character for the same purpose in string literals; for example, to match
a literal backslash, one might have to write `'\\\\'' as the pattern
string, because the regular expression must be `\\', and each backslash
must be expressed as `\\' inside a regular Python string literal.

  The solution is to use Python's raw string notation for regular
expression patterns; backslashes are not handled in any special way in
a string literal prefixed with `'r''.  So `r"\n"' is a two-character
string containing `'\'' and `'n'', while `"\n"' is a one-character
string containing a newline.  Usually patterns will be expressed in
Python code using this raw string notation.

  It is important to note that most regular expression operations are
available as module-level functions and RegexObject.
methods.  The functions are shortcuts that don't require you to compile
a regex object first, but miss some fine-tuning parameters.

Mastering Regular Expressions
     Book on regular expressions by Jeffrey Friedl, published by
     O'Reilly.  The second edition of the book no longer covers Python
     at all, but the first edition covered writing good regular
     expression patterns in great detail.

5.7.2.1 Regular Expression Syntax
.................................

A regular expression (or RE) specifies a set of strings that matches
it; the functions in this module let you check if a particular string
matches a given regular expression (or if a given regular expression
matches a particular string, which comes down to the same thing).

  Regular expressions can be concatenated to form new regular
expressions; if _A_ and _B_ are both regular expressions, then _AB_ is
also a regular expression.  In general, if a string _p_ matches _A_ and
another string _q_ matches _B_, the string _pq_ will match AB.  This
holds unless _A_ or _B_ contain low precedence operations; boundary
conditions between _A_ and _B_; or have numbered group references.
Thus, complex expressions can easily be constructed from simpler
primitive expressions like the ones described here.  For details of the
theory and implementation of regular expressions, consult the Friedl
book referenced above, or almost any textbook about compiler
construction.

  A brief explanation of the format of regular expressions follows.
For further information and a gentler presentation, consult the
Regular Expression HOWTO.

  Regular expressions can contain both special and ordinary characters.
Most ordinary characters, like `'A'', `'a'', or `'0'', are the simplest
regular expressions; they simply match themselves.  You can concatenate
ordinary characters, so `last' matches the string `'last''.  (In the
rest of this section, we'll write RE's in `this special style', usually
without quotes, and strings to be matched `'in single quotes''.)

  Some characters, like `'|'' or `'('', are special. Special characters
either stand for classes of ordinary characters, or affect how the
regular expressions around them are interpreted. Regular expression
pattern strings may not contain null bytes, but can specify the null
byte using the `\number' notation, e.g., `'\x00''.

  The special characters are:

`'.''
     (Dot.)  In the default mode, this matches any character except a
     newline.  If the DOTALL flag has been specified, this
     matches any character including a newline.

`'^''
     (Caret.)  Matches the start of the string, and in MULTILINE
     mode also matches immediately after each newline.

`'$''
     Matches the end of the string or just before the newline at the
     end of the string, and in MULTILINE mode also matches
     before a newline.  `foo' matches both 'foo' and 'foobar', while
     the regular expression `foo$' matches only 'foo'.  More
     interestingly, searching for `foo.$' in `'foo1\nfoo2\n'' matches
     'foo2' normally, but 'foo1' in MULTILINE mode;
     searching for a single `$' in `'foo\n'' will find two (empty)
     matches: one just before the newline, and one at the end of the
     string.

`'*''
     Causes the resulting RE to match 0 or more repetitions of the
     preceding RE, as many repetitions as are possible.  `ab*' will
     match 'a', 'ab', or 'a' followed by any number of 'b's.

`'+''
     Causes the resulting RE to match 1 or more repetitions of the
     preceding RE.  `ab+' will match 'a' followed by any non-zero
     number of 'b's; it will not match just 'a'.

`'?''
     Causes the resulting RE to match 0 or 1 repetitions of the
     preceding RE.  `ab?' will match either 'a' or 'ab'.

`*?', `+?', `??'
     The `'*'', `'+'', and `'?'' qualifiers are all _greedy_; they match
     as much text as possible.  Sometimes this behaviour isn't desired;
     if the RE `<.*>' is matched against `'<H1>title</H1>'', it will
     match the entire string, and not just `'<H1>''.  Adding `'?''
     after the qualifier makes it perform the match in _non-greedy_ or
     _minimal_ fashion; as _few_ characters as possible will be
     matched.  Using `.*?' in the previous expression will match only
     `'<H1>''.

`{m}'
     Specifies that exactly _m_ copies of the previous RE should be
     matched; fewer matches cause the entire RE not to match.  For
     example, `a{6}' will match exactly six `'a'' characters, but not
     five.

`{m,n}'
     Causes the resulting RE to match from _m_ to _n_ repetitions of
     the preceding RE, attempting to match as many repetitions as
     possible.  For example, `a{3,5}' will match from 3 to 5 `'a''
     characters.  Omitting _m_ specifies a lower bound of zero,  and
     omitting _n_ specifies an infinite upper bound.  As an example,
     `a{4,}b' will match `aaaab' or a thousand `'a'' characters
     followed by a `b', but not `aaab'. The comma may not be omitted or
     the modifier would be confused with the previously described form.

`{m,n}?'
     Causes the resulting RE to match from _m_ to _n_ repetitions of
     the preceding RE, attempting to match as _few_ repetitions as
     possible.  This is the non-greedy version of the previous
     qualifier.  For example, on the 6-character string `'aaaaaa'',
     `a{3,5}' will match 5 `'a'' characters, while `a{3,5}?' will only
     match 3 characters.

`'\''
     Either escapes special characters (permitting you to match
     characters like `'*'', `'?'', and so forth), or signals a special
     sequence; special sequences are discussed below.

     If you're not using a raw string to express the pattern, remember
     that Python also uses the backslash as an escape sequence in
     string literals; if the escape sequence isn't recognized by
     Python's parser, the backslash and subsequent character are
     included in the resulting string.  However, if Python would
     recognize the resulting sequence, the backslash should be repeated
     twice.  This is complicated and hard to understand, so it's highly
     recommended that you use raw strings for all but the simplest
     expressions.

`[]'
     Used to indicate a set of characters.  Characters can be listed
     individually, or a range of characters can be indicated by giving
     two characters and separating them by a `'-''.  Special characters
     are not active inside sets.  For example, `[akm$]' will match any
     of the characters `'a'', `'k'', `'m'', or `'$''; `[a-z]' will
     match any lowercase letter, and `[a-zA-Z0-9]' matches any letter
     or digit.  Character classes such as `\w' or `\S' (defined below)
     are also acceptable inside a range, although the characters they
     match depends on whether LOCALE.  or  UNICODE
     mode is in force.  If you want to include a `']'' or a `'-''
     inside a set, precede it with a backslash, or place it as the
     first character.  The pattern `[]]' will match `']'', for example.

     You can match the characters not within a range by _complementing_
     the set.  This is indicated by including a `'^'' as the first
     character of the set; `'^'' elsewhere will simply match the `'^''
     character.  For example, `[^5]' will match any character except
     `'5'', and `[^^]' will match any character except `'^''.

     Note that inside `[]' the special forms and special characters lose
     their meanings and only the syntaxes described here are valid. For
     example, `+', `*', `(', `)', and so on are treated as literals
     inside `[]', and backreferences cannot be used inside `[]'.

`'|''
     `A|B', where A and B can be arbitrary REs, creates a regular
     expression that will match either A or B.  An arbitrary number of
     REs can be separated by the `'|'' in this way.  This can be used
     inside groups (see below) as well.  As the target string is
     scanned, REs separated by `'|'' are tried from left to right. When
     one pattern completely matches, that branch is accepted. This means
     that once `A' matches, `B' will not be tested further, even if it
     would produce a longer overall match.  In other words, the `'|''
     operator is never greedy.  To match a literal `'|'', use `\|', or
     enclose it inside a character class, as in `[|]'.

`(...)'
     Matches whatever regular expression is inside the parentheses, and
     indicates the start and end of a group; the contents of a group
     can be retrieved after a match has been performed, and can be
     matched later in the string with the `\number' special sequence,
     described below.  To match the literals `'('' or `')'', use `\('
     or `\)', or enclose them inside a character class: `[(] [)]'.

`(?...)'
     This is an extension notation (a `'?'' following a `'('' is not
     meaningful otherwise).  The first character after the `'?''
     determines what the meaning and further syntax of the construct
     is. Extensions usually do not create a new group; `(?P<name>...)'
     is the only exception to this rule. Following are the currently
     supported extensions.

`(?iLmsux)'
     (One or more letters from the set `'i'', `'L'', `'m'', `'s'',
     `'u'', `'x''.)  The group matches the empty string; the letters
     set the corresponding flags: re.I. (ignore case), see
     re.L. (locale dependent), see re.M. (multi-line),
     see re.S. (dot matches all), see re.U. (Unicode
     dependent), and re.X. (verbose), for the entire regular
     expression. (The flags are described in Module Contents)
     This is useful if you wish to include the flags as part of
     the regular expression, instead of passing a _flag_ argument to the
     re.compile() function.

     Note that the `(?x)' flag changes how the expression is parsed. It
     should be used first in the expression string, or after one or
     more whitespace characters.  If there are non-whitespace
     characters before the flag, the results are undefined.

`(?:...)'
     A non-capturing version of regular parentheses.  Matches whatever
     regular expression is inside the parentheses, but the substring
     matched by the group _cannot_ be retrieved after performing a
     match or referenced later in the pattern.

`(?P<name>...)'
     Similar to regular parentheses, but the substring matched by the
     group is accessible within the rest of the regular expression via
     the symbolic group name _name_.  Group names must be valid Python
     identifiers, and each group name must be defined only once within
     a regular expression.  A symbolic group is also a numbered group,
     just as if the group were not named.  So the group named `id' in
     the example below can also be referenced as the numbered group `1'.

     For example, if the pattern is `(?P<id>[a-zA-Z_]\w*)', the group
     can be referenced by its name in arguments to methods of match
     objects, such as `m.group('id')' or `m.end('id')', and also by
     name in the regular expression itself (using `(?P=id)') and
     replacement text given to `.sub()' (using `\g<id>').

`(?P=name)'
     Matches whatever text was matched by the earlier group named
     _name_.

`(?#...)'
     A comment; the contents of the parentheses are simply ignored.

`(?=...)'
     Matches if `...' matches next, but doesn't consume any of the
     string.  This is called a lookahead assertion.  For example,
     `Isaac (?=Asimov)' will match `'Isaac '' only if it's followed by
     `'Asimov''.

`(?!...)'
     Matches if `...' doesn't match next.  This is a negative lookahead
     assertion.  For example, `Isaac (?!Asimov)' will match `'Isaac ''
     only if it's _not_ followed by `'Asimov''.

`(?<=...)'
     Matches if the current position in the string is preceded by a
     match for `...' that ends at the current position.  This is called
     a _positive lookbehind assertion_. `(?<=abc)def' will find a match
     in `abcdef', since the lookbehind will back up 3 characters and
     check if the contained pattern matches.  The contained pattern
     must only match strings of some fixed length, meaning that `abc'
     or `a|b' are allowed, but `a*' and `a{3,4}' are not.  Note that
     patterns which start with positive lookbehind assertions will
     never match at the beginning of the string being searched; you
     will most likely want to use the search() function
     rather than the match() function:

         >>> import re
         >>> m = re.search('(?<=abc)def', 'abcdef')
         >>> m.group(0)
         'def'

     This example looks for a word following a hyphen:

         >>> m = re.search('(?<=-)\w+', 'spam-egg')
         >>> m.group(0)
         'egg'

`(?<!...)'
     Matches if the current position in the string is not preceded by a
     match for `...'.  This is called a _negative lookbehind
     assertion_.  Similar to positive lookbehind assertions, the
     contained pattern must only match strings of some fixed length.
     Patterns which start with negative lookbehind assertions may match
     at the beginning of the string being searched.

`(?(id/name)yes-pattern|no-pattern)'
     Will try to match with `yes-pattern' if the group with given _id_
     or _name_ exists, and with `no-pattern' if it doesn't.
     `no-pattern' is optional and can be omitted. For example,
     `(<)?(\w+@\w+(?:\.\w+)+)(?(1)>)' is a poor email matching pattern,
     which will match with `'<user@host.com>'' as well as
     `'user@host.com'', but not with `'<user@host.com''.

     New in version 2.4.

  The special sequences consist of `'\'' and a character from the list
below.  If the ordinary character is not on the list, then the
resulting RE will match the second character.  For example, `\$'
matches the character `'$''.

`\number'
     Matches the contents of the group of the same number.  Groups are
     numbered starting from 1.  For example, `(.+) \1' matches `'the
     the'' or `'55 55'', but not `'the end'' (note the space after the
     group).  This special sequence can only be used to match one of
     the first 99 groups.  If the first digit of _number_ is 0, or
     _number_ is 3 octal digits long, it will not be interpreted as a
     group match, but as the character with octal value _number_.
     Inside the `'['' and `']'' of a character class, all numeric
     escapes are treated as characters.

`\A'
     Matches only at the start of the string.

`\b'
     Matches the empty string, but only at the beginning or end of a
     word.  A word is defined as a sequence of alphanumeric or
     underscore characters, so the end of a word is indicated by
     whitespace or a non-alphanumeric, non-underscore character.  Note
     that  `\b' is defined as the boundary between `\w' and `\W', so the
     precise set of characters deemed to be alphanumeric depends on the
     values of the `UNICODE' and `LOCALE' flags.  Inside a character
     range, `\b' represents the backspace character, for compatibility
     with Python's string literals.

`\B'
     Matches the empty string, but only when it is _not_ at the
     beginning or end of a word.  This is just the opposite of `\b', so
     is also subject to the settings of `LOCALE' and `UNICODE'.

`\d'
     When the UNICODE flag is not specified, matches any
     decimal digit; this is equivalent to the set `[0-9]'.  With
     UNICODE, it will match whatever is classified as a decimal
     digit in the Unicode character properties database.

`\D'
     When the UNICODE flag is not specified, matches any
     non-digit character; this is equivalent to the set  `[^0-9]'.
     With UNICODE, it will match  anything other than
     character marked as digits in the Unicode character  properties
     database.

`\s'
     When the LOCALE and UNICODE flags are not
     specified, matches any whitespace character; this is equivalent to
     the set `[ \t\n\r\f\v]'. With LOCALE, it will match
     this set plus whatever characters are defined as space for the
     current locale. If UNICODE is set, this will match the
     characters `[ \t\n\r\f\v]' plus whatever is classified as space in
     the Unicode character properties database.

`\S'
     When the LOCALE and UNICODE flags are not
     specified, matches any non-whitespace character; this is
     equivalent to the set `[^ \t\n\r\f\v]' With LOCALE, it
     will match any character not in this set, and not defined as space
     in the current locale. If UNICODE is set, this will
     match anything other than `[ \t\n\r\f\v]' and characters marked as
     space in the Unicode character properties database.

`\w'
     When the LOCALE and UNICODE flags are not
     specified, matches any alphanumeric character and the underscore;
     this is equivalent to the set `[a-zA-Z0-9_]'.  With LOCALE,
     it will match the set `[0-9_]' plus whatever characters are
     defined as alphanumeric for the current locale.  If UNICODE
     is set, this will match the characters `[0-9_]' plus whatever
     is classified as alphanumeric in the Unicode character properties
     database.

`\W'
     When the LOCALE and UNICODE flags are not
     specified, matches any non-alphanumeric character; this is
     equivalent to the set `[^a-zA-Z0-9_]'.  With LOCALE, it
     will match any character not in the set `[0-9_]', and not defined
     as alphanumeric for the current locale. If UNICODE is
     set, this will match anything other than `[0-9_]' and characters
     marked as alphanumeric in the Unicode character properties
     database.

`\Z'
     Matches only at the end of the string.

  Most of the standard escapes supported by Python string literals are
also accepted by the regular expression parser:

    \a      \b      \f      \n
    \r      \t      \v      \x
    \\

Octal escapes are included in a limited form: If the first digit is a
0, or if there are three octal digits, it is considered an octal
escape. Otherwise, it is a group reference.  As for string literals,
octal escapes are always at most three digits in length.

5.7.2.2 Matching vs Searching
.............................

Python offers two different primitive operations based on regular
expressions: *match* checks for a match only at the beginning of the
string, while *search* checks for a match anywhere in the string (this
is what Perl does by default).

  Note that match may differ from search even when using a regular
expression beginning with `'^'': `'^'' matches only at the start of the
string, or in MULTILINE mode also immediately following a
newline.  The "match" operation succeeds only if the pattern matches at
the start of the string regardless of mode, or at the starting position
given by the optional _pos_ argument regardless of whether a newline
precedes it.

    >>> re.match("c", "abcdef")  # No match
    >>> re.search("c", "abcdef") # Match
    <_sre.SRE_Match object at ...>

5.7.2.3 Module Contents
.......................

The module defines several functions, constants, and an exception. Some
of the functions are simplified versions of the full featured methods
for compiled regular expressions.  Most non-trivial applications always
use the compiled form.

 -- Function: re.compile (pattern[, flags])
     Compile a regular expression pattern into a regular expression
     object, which can be used for matching using its match()
     and search() methods, described below.

     The expression's behaviour can be modified by specifying a _flags_
     value.  Values can be any of the following variables, combined
     using bitwise OR (the `|' operator).

     The sequence

         prog = re.compile(pattern)
         result = prog.match(string)

     is equivalent to

         result = re.match(pattern, string)

     but using re.compile() and saving the resulting
     regular expression object for reuse is more efficient when the
     expression will be used several times in a single program.

          Note: The compiled versions of the most recent patterns
          passed to re.match(), re.search() or
          re.compile() are cached, so programs that use
          only a few regular expressions at a time needn't worry about
          compiling regular expressions.

 -- Data: re.I
 -- Data: re.IGNORECASE
     Perform case-insensitive matching; expressions like `[A-Z]' will
     match lowercase letters, too.  This is not affected by the current
     locale.

 -- Data: re.L
 -- Data: re.LOCALE
     Make `\w', `\W', `\b', `\B', `\s' and `\S' dependent on the
     current locale.

 -- Data: re.M
 -- Data: re.MULTILINE
     When specified, the pattern character `'^'' matches at the
     beginning of the string and at the beginning of each line
     (immediately following each newline); and the pattern character
     `'$'' matches at the end of the string and at the end of each line
     (immediately preceding each newline).  By default, `'^'' matches
     only at the beginning of the string, and `'$'' only at the end of
     the string and immediately before the newline (if any) at the end
     of the string.

 -- Data: re.S
 -- Data: re.DOTALL
     Make the `'.'' special character match any character at all,
     including a newline; without this flag, `'.'' will match anything
     _except_ a newline.

 -- Data: re.U
 -- Data: re.UNICODE
     Make `\w', `\W', `\b', `\B', `\d', `\D', `\s' and `\S' dependent
     on the Unicode character properties database.

     New in version 2.0.

 -- Data: re.X
 -- Data: re.VERBOSE
     This flag allows you to write regular expressions that look nicer.
     Whitespace within the pattern is ignored, except when in a
     character class or preceded by an unescaped backslash, and, when a
     line contains a `'#'' neither in a character class or preceded by
     an unescaped backslash, all characters from the leftmost such
     `'#'' through the end of the line are ignored.

     That means that the two following regular expression objects that
     match a decimal number are functionally equal:

         a = re.compile(r"""\d +  # the integral part
                            \.    # the decimal point
                            \d *  # some fractional digits""", re.X)
         b = re.compile(r"\d+\.\d*")

 -- Function: re.search (pattern, string[, flags])
     Scan through _string_ looking for a location where the regular
     expression _pattern_ produces a match, and return a corresponding
     MatchObject.  instance. Return `None' if no position in
     the string matches the pattern; note that this is different from
     finding a zero-length match at some point in the string.

 -- Function: re.match (pattern, string[, flags])
     If zero or more characters at the beginning of _string_ match the
     regular expression _pattern_, return a corresponding
     MatchObject instance.  Return `None' if the string does not
     match the pattern; note that this is different from a zero-length
     match.

          Note: If you want to locate a match anywhere in _string_, use
          search().  instead.

 -- Function: re.split (pattern, string[, maxsplit=0, flags=0])
     Split _string_ by the occurrences of _pattern_.  If capturing
     parentheses are used in _pattern_, then the text of all groups in
     the pattern are also returned as part of the resulting list. If
     _maxsplit_ is nonzero, at most _maxsplit_ splits occur, and the
     remainder of the string is returned as the final element of the
     list.  (Incompatibility note: in the original Python 1.5 release,
     _maxsplit_ was ignored.  This has been fixed in later releases.)

         >>> re.split('\W+', 'Words, words, words.')
         ['Words', 'words', 'words', '']
         >>> re.split('(\W+)', 'Words, words, words.')
         ['Words', ', ', 'words', ', ', 'words', '.', '']
         >>> re.split('\W+', 'Words, words, words.', 1)
         ['Words', 'words, words.']
         >>> re.split('[a-f]+', '0a3B9', flags=re.IGNORECASE)
         ['0', '3', '9']

     If there are capturing groups in the separator and it matches at
     the start of the string, the result will start with an empty
     string.  The same holds for the end of the string:

         >>> re.split('(\W+)', '...words, words...')
         ['', '...', 'words', ', ', 'words', '...', '']

     That way, separator components are always found at the same
     relative indices within the result list (e.g., if there's one
     capturing group in the separator, the 0th, the 2nd and so forth).

     Note that _split_ will never split a string on an empty pattern
     match.  For example:

         >>> re.split('x*', 'foo')
         ['foo']
         >>> re.split("(?m)^$", "foo\n\nbar\n")
         ['foo\n\nbar\n']

     Changed in version 2.7: Added the optional flags argument.

 -- Function: re.findall (pattern, string[, flags])
     Return all non-overlapping matches of _pattern_ in _string_, as a
     list of strings.  The _string_ is scanned left-to-right, and
     matches are returned in the order found.  If one or more groups
     are present in the pattern, return a list of groups; this will be
     a list of tuples if the pattern has more than one group.  Empty
     matches are included in the result unless they touch the beginning
     of another match.

     New in version 1.5.2.

     Changed in version 2.4: Added the optional flags argument.

 -- Function: re.finditer (pattern, string[, flags])
     Return an iterator yielding MatchObject.
     instances over all non-overlapping matches for the RE _pattern_ in
     _string_.  The _string_ is scanned left-to-right, and matches are
     returned in the order found.  Empty matches are included in the
     result unless they touch the beginning of another match.

     New in version 2.2.

     Changed in version 2.4: Added the optional flags argument.

 -- Function: re.sub (pattern, repl, string[, count, flags])
     Return the string obtained by replacing the leftmost
     non-overlapping occurrences of _pattern_ in _string_ by the
     replacement _repl_.  If the pattern isn't found, _string_ is
     returned unchanged.  _repl_ can be a string or a function; if it is
     a string, any backslash escapes in it are processed.  That is,
     `\n' is converted to a single newline character, `\r' is converted
     to a linefeed, and so forth.  Unknown escapes such as `\j' are
     left alone.  Backreferences, such as `\6', are replaced with the
     substring matched by group 6 in the pattern.  For example:

         >>> re.sub(r'def\s+([a-zA-Z_][a-zA-Z_0-9]*)\s*\(\s*\):',
         ...        r'static PyObject*\npy_\1(void)\n{',
         ...        'def myfunc():')
         'static PyObject*\npy_myfunc(void)\n{'

     If _repl_ is a function, it is called for every non-overlapping
     occurrence of _pattern_.  The function takes a single match object
     argument, and returns the replacement string.  For example:

         >>> def dashrepl(matchobj):
         ...     if matchobj.group(0) == '-': return ' '
         ...     else: return '-'
         >>> re.sub('-{1,2}', dashrepl, 'pro----gram-files')
         'pro--gram files'
         >>> re.sub(r'\sAND\s', ' & ', 'Baked Beans And Spam', flags=re.IGNORECASE)
         'Baked Beans & Spam'

     The pattern may be a string or an RE object.

     The optional argument _count_ is the maximum number of pattern
     occurrences to be replaced; _count_ must be a non-negative
     integer.  If omitted or zero, all occurrences will be replaced.
     Empty matches for the pattern are replaced only when not adjacent
     to a previous match, so `sub('x*', '-', 'abc')' returns
     `'-a-b-c-''.

     In addition to character escapes and backreferences as described
     above, `\g<name>' will use the substring matched by the group
     named `name', as defined by the `(?P<name>...)' syntax.
     `\g<number>' uses the corresponding group number; `\g<2>' is
     therefore equivalent to `\2', but isn't ambiguous in a replacement
     such as `\g<2>0'.  `\20' would be interpreted as a reference to
     group 20, not a reference to group 2 followed by the literal
     character `'0''.  The backreference `\g<0>' substitutes in the
     entire substring matched by the RE.

     Changed in version 2.7: Added the optional flags argument.

 -- Function: re.subn (pattern, repl, string[, count, flags])
     Perform the same operation as sub(), but return a tuple
     `(new_string, number_of_subs_made)'.

     Changed in version 2.7: Added the optional flags argument.

 -- Function: re.escape (string)
     Return _string_ with all non-alphanumerics backslashed; this is
     useful if you want to match an arbitrary literal string that may
     have regular expression metacharacters in it.

 -- Function: re.purge ()
     Clear the regular expression cache.

 -- Exception: re.error
     Exception raised when a string passed to one of the functions here
     is not a valid regular expression (for example, it might contain
     unmatched parentheses) or when some other error occurs during
     compilation or matching.  It is never an error if a string
     contains no match for a pattern.

5.7.2.4 Regular Expression Objects
..................................

 -- Class: re.RegexObject
     The RegexObject class supports the following methods
     and attributes:

      -- Method: search (string[, pos[, endpos]])
          Scan through _string_ looking for a location where this
          regular expression produces a match, and return a
          corresponding MatchObject instance.  Return
          `None' if no position in the string matches the pattern; note
          that this is different from finding a zero-length match at
          some point in the string.

          The optional second parameter _pos_ gives an index in the
          string where the search is to start; it defaults to `0'.
          This is not completely equivalent to slicing the string; the
          `'^'' pattern character matches at the real beginning of the
          string and at positions just after a newline, but not
          necessarily at the index where the search is to start.

          The optional parameter _endpos_ limits how far the string
          will be searched; it will be as if the string is _endpos_
          characters long, so only the characters from _pos_ to `endpos
          - 1' will be searched for a match.  If _endpos_ is less than
          _pos_, no match will be found, otherwise, if _rx_ is a
          compiled regular expression object, `rx.search(string, 0,
          50)' is equivalent to `rx.search(string[:50], 0)'.

              >>> pattern = re.compile("d")
              >>> pattern.search("dog")     # Match at index 0
              <_sre.SRE_Match object at ...>
              >>> pattern.search("dog", 1)  # No match; search doesn't include the "d"

      -- Method: match (string[, pos[, endpos]])
          If zero or more characters at the _beginning_ of _string_
          match this regular expression, return a corresponding
          MatchObject instance.  Return `None' if the string does
          not match the pattern; note that this is different from a
          zero-length match.

          The optional _pos_ and _endpos_ parameters have the same
          meaning as for the search() method.

               Note: If you want to locate a match anywhere in
               _string_, use search() instead.

              >>> pattern = re.compile("o")
              >>> pattern.match("dog")      # No match as "o" is not at the start of "dog".
              >>> pattern.match("dog", 1)   # Match as "o" is the 2nd character of "dog".
              <_sre.SRE_Match object at ...>

      -- Method: split (string[, maxsplit=0])
          Identical to the split() function, using the
          compiled pattern.

      -- Method: findall (string[, pos[, endpos]])
          Similar to the findall() function, using the
          compiled pattern, but also accepts optional _pos_ and
          _endpos_ parameters that limit the search region like for
          match().

      -- Method: finditer (string[, pos[, endpos]])
          Similar to the finditer() function, using the
          compiled pattern, but also accepts optional _pos_ and
          _endpos_ parameters that limit the search region like for
          match().

      -- Method: sub (repl, string[, count=0])
          Identical to the sub() function, using the
          compiled pattern.

      -- Method: subn (repl, string[, count=0])
          Identical to the subn() function, using the
          compiled pattern.

      -- Attribute: flags
          The flags argument used when the RE object was compiled, or
          `0' if no flags were provided.

      -- Attribute: groups
          The number of capturing groups in the pattern.

      -- Attribute: groupindex
          A dictionary mapping any symbolic group names defined by
          `(?P<id>)' to group numbers.  The dictionary is empty if no
          symbolic groups were used in the pattern.

      -- Attribute: pattern
          The pattern string from which the RE object was compiled.

5.7.2.5 Match Objects
.....................

 -- Class: re.MatchObject
     Match Objects always have a boolean value of True, so
     that you can test whether e.g. see match() resulted in a
     match with a simple if statement.  They support the following
     methods and attributes:

      -- Method: expand (template)
          Return the string obtained by doing backslash substitution on
          the template string _template_, as done by the sub()
          method.  Escapes such as `\n' are converted to the
          appropriate characters, and numeric backreferences (`\1',
          `\2') and named backreferences (`\g<1>', `\g<name>') are
          replaced by the contents of the corresponding group.

      -- Method: group ([group1, ...])
          Returns one or more subgroups of the match.  If there is a
          single argument, the result is a single string; if there are
          multiple arguments, the result is a tuple with one item per
          argument. Without arguments, _group1_ defaults to zero (the
          whole match is returned). If a _groupN_ argument is zero, the
          corresponding return value is the entire matching string; if
          it is in the inclusive range [1..99], it is the string
          matching the corresponding parenthesized group.  If a group
          number is negative or larger than the number of groups
          defined in the pattern, an IndexError exception
          is raised. If a group is contained in a part of the pattern
          that did not match, the corresponding result is `None'.  If a
          group is contained in a part of the pattern that matched
          multiple times, the last match is returned.

              >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
              >>> m.group(0)       # The entire match
              'Isaac Newton'
              >>> m.group(1)       # The first parenthesized subgroup.
              'Isaac'
              >>> m.group(2)       # The second parenthesized subgroup.
              'Newton'
              >>> m.group(1, 2)    # Multiple arguments give us a tuple.
              ('Isaac', 'Newton')

          If the regular expression uses the `(?P<name>...)' syntax,
          the _groupN_ arguments may also be strings identifying groups
          by their group name.  If a string argument is not used as a
          group name in the pattern, an IndexError.
          exception is raised.

          A moderately complicated example:

              >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
              >>> m.group('first_name')
              'Malcolm'
              >>> m.group('last_name')
              'Reynolds'

          Named groups can also be referred to by their index:

              >>> m.group(1)
              'Malcolm'
              >>> m.group(2)
              'Reynolds'

          If a group matches multiple times, only the last match is
          accessible:

              >>> m = re.match(r"(..)+", "a1b2c3")  # Matches 3 times.
              >>> m.group(1)                        # Returns only the last match.
              'c3'

      -- Method: groups ([default])
          Return a tuple containing all the subgroups of the match,
          from 1 up to however many groups are in the pattern.  The
          _default_ argument is used for groups that did not
          participate in the match; it defaults to `None'.
          (Incompatibility note: in the original Python 1.5 release, if
          the tuple was one element long, a string would be returned
          instead.  In later versions (from 1.5.1 on), a singleton
          tuple is returned in such cases.)

          For example:

              >>> m = re.match(r"(\d+)\.(\d+)", "24.1632")
              >>> m.groups()
              ('24', '1632')

          If we make the decimal place and everything after it
          optional, not all groups might participate in the match.
          These groups will default to `None' unless the _default_
          argument is given:

              >>> m = re.match(r"(\d+)\.?(\d+)?", "24")
              >>> m.groups()      # Second group defaults to None.
              ('24', None)
              >>> m.groups('0')   # Now, the second group defaults to '0'.
              ('24', '0')

      -- Method: groupdict ([default])
          Return a dictionary containing all the _named_ subgroups of
          the match, keyed by the subgroup name.  The _default_
          argument is used for groups that did not participate in the
          match; it defaults to `None'.  For example:

              >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
              >>> m.groupdict()
              {'first_name': 'Malcolm', 'last_name': 'Reynolds'}

      -- Method: start ([group])
      -- Method: end ([group])
          Return the indices of the start and end of the substring
          matched by _group_; _group_ defaults to zero (meaning the
          whole matched substring). Return `-1' if _group_ exists but
          did not contribute to the match.  For a match object _m_, and
          a group _g_ that did contribute to the match, the substring
          matched by group _g_ (equivalent to `m.group(g)') is

              m.string[m.start(g):m.end(g)]

          Note that `m.start(group)' will equal `m.end(group)' if
          _group_ matched a null string.  For example, after `m =
          re.search('b(c?)', 'cba')', `m.start(0)' is 1, `m.end(0)' is
          2, `m.start(1)' and `m.end(1)' are both 2, and `m.start(2)'
          raises an IndexError exception.

          An example that will remove _remove_this_ from email
          addresses:

              >>> email = "tony@tiremove_thisger.net"
              >>> m = re.search("remove_this", email)
              >>> email[:m.start()] + email[m.end():]
              'tony@tiger.net'

      -- Method: span ([group])
          For MatchObject. _m_, return the 2-tuple
          `(m.start(group), m.end(group))'. Note that if _group_ did
          not contribute to the match, this is `(-1, -1)'.  _group_
          defaults to zero, the entire match.

      -- Attribute: pos
          The value of _pos_ which was passed to the search()
          or match() method of the RegexObject.
          This is the index into the string at which the RE
          engine started looking for a match.

      -- Attribute: endpos
          The value of _endpos_ which was passed to the search()
          or match() method of the RegexObject.
          This is the index into the string beyond which the RE
          engine will not go.

      -- Attribute: lastindex
          The integer index of the last matched capturing group, or
          `None' if no group was matched at all. For example, the
          expressions `(a)b', `((a)(b))', and `((ab))' will have
          `lastindex == 1' if applied to the string `'ab'', while the
          expression `(a)(b)' will have `lastindex == 2', if applied to
          the same string.

      -- Attribute: lastgroup
          The name of the last matched capturing group, or `None' if
          the group didn't have a name, or if no group was matched at
          all.

      -- Attribute: re
          The regular expression object whose match() or
          search() method produced this MatchObject.
          instance.

      -- Attribute: string
          The string passed to match() or search().

5.7.2.6 Examples
................

5.7.2.7 Checking For a Pair
...........................

In this example, we'll use the following helper function to display
match objects a little more gracefully:

    def displaymatch(match):
        if match is None:
            return None
        return '<Match: %r, groups=%r>' % (match.group(), match.groups())

Suppose you are writing a poker program where a player's hand is
represented as a 5-character string with each character representing a
card, "a" for ace, "k" for king, "q" for queen, j for jack, "0" for 10,
and "1" through "9" representing the card with that value.

  To see if a given string is a valid hand, one could do the following:

    >>> valid = re.compile(r"[0-9akqj]{5}$")
    >>> displaymatch(valid.match("ak05q"))  # Valid.
    "<Match: 'ak05q', groups=()>"
    >>> displaymatch(valid.match("ak05e"))  # Invalid.
    >>> displaymatch(valid.match("ak0"))    # Invalid.
    >>> displaymatch(valid.match("727ak"))  # Valid.
    "<Match: '727ak', groups=()>"

That last hand, `"727ak"', contained a pair, or two of the same valued
cards.  To match this with a regular expression, one could use
backreferences as such:

    >>> pair = re.compile(r".*(.).*\1")
    >>> displaymatch(pair.match("717ak"))     # Pair of 7s.
    "<Match: '717', groups=('7',)>"
    >>> displaymatch(pair.match("718ak"))     # No pairs.
    >>> displaymatch(pair.match("354aa"))     # Pair of aces.
    "<Match: '354aa', groups=('a',)>"

To find out what card the pair consists of, one could use the
group() method of MatchObject in the following manner:

    >>> pair.match("717ak").group(1)
    '7'

    # Error because re.match() returns None, which doesn't have a group() method:
    >>> pair.match("718ak").group(1)
    Traceback (most recent call last):
      File "<pyshell#23>", line 1, in <module>
        re.match(r".*(.).*\1", "718ak").group(1)
    AttributeError: 'NoneType' object has no attribute 'group'

    >>> pair.match("354aa").group(1)
    'a'

5.7.2.8 Simulating scanf()
..........................

Python does not currently have an equivalent to `scanf()'.  Regular
expressions are generally more powerful, though also more verbose, than
`scanf()' format strings.  The table below offers some more-or-less
equivalent mappings between `scanf()' format tokens and regular
expressions.

`scanf()' Token                      Regular Expression
---------------------------------------------------------------------------------------
`%c'                                 `.'
`%5c'                                `.{5}'
`%d'                                 `[-+]?\d+'
`%e', `%E', `%f', `%g'               `[-+]?(\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?'
`%i'                                 `[-+]?(0[xX][\dA-Fa-f]+|0[0-7]*|\d+)'
`%o'                                 `0[0-7]*'
`%s'                                 `\S+'
`%u'                                 `\d+'
`%x', `%X'                           `0[xX][\dA-Fa-f]+'

  To extract the filename and numbers from a string like

    /usr/sbin/sendmail - 0 errors, 4 warnings

you would use a `scanf()' format like

    %s - %d errors, %d warnings

The equivalent regular expression would be

    (\S+) - (\d+) errors, (\d+) warnings

5.7.2.9 Avoiding recursion
..........................

If you create regular expressions that require the engine to perform a
lot of recursion, you may encounter a RuntimeError.
exception with the message `maximum recursion limit' exceeded. For
example,

    >>> s = 'Begin ' + 1000*'a very long string ' + 'end'
    >>> re.match('Begin (\w| )*? end', s).end()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "/usr/local/lib/python2.5/re.py", line 132, in match
        return _compile(pattern, flags).match(string)
    RuntimeError: maximum recursion limit exceeded

You can often restructure your regular expression to avoid recursion.

  Starting with Python 2.3, simple uses of the `*?' pattern are
special-cased to avoid recursion.  Thus, the above regular expression
can avoid recursion by being recast as `Begin [a-zA-Z0-9_ ]*?end'.  As
a further benefit, such regular expressions will run faster than their
recursive equivalents.

5.7.2.10 search() vs. match()
.............................

In a nutshell, see match() only attempts to match a pattern at
the beginning of a string where search() will match a
pattern anywhere in a string.  For example:

    >>> re.match("o", "dog")  # No match as "o" is not the first letter of "dog".
    >>> re.search("o", "dog") # Match as search() looks everywhere in the string.
    <_sre.SRE_Match object at ...>

     Note: The following applies only to regular expression objects
     like those created with `re.compile("pattern")', not the
     primitives `re.match(pattern, string)' or `re.search(pattern,
     string)'.

  match() has an optional second parameter that gives an
index in the string where the search is to start:

    >>> pattern = re.compile("o")
    >>> pattern.match("dog")      # No match as "o" is not at the start of "dog."

    # Equivalent to the above expression as 0 is the default starting index:
    >>> pattern.match("dog", 0)

    # Match as "o" is the 2nd character of "dog" (index 0 is the first):
    >>> pattern.match("dog", 1)
    <_sre.SRE_Match object at ...>
    >>> pattern.match("dog", 2)   # No match as "o" is not the 3rd character of "dog."

5.7.2.11 Making a Phonebook
...........................

split() splits a string into a list delimited by the passed
pattern.  The method is invaluable for converting textual data into
data structures that can be easily read and modified by Python as
demonstrated in the following example that creates a phonebook.

  First, here is the input.  Normally it may come from a file, here we
are using triple-quoted string syntax:

    >>> input = """Ross McFluff: 834.345.1254 155 Elm Street
    ...
    ... Ronald Heathmore: 892.345.3428 436 Finley Avenue
    ... Frank Burger: 925.541.7625 662 South Dogwood Way
    ...
    ...
    ... Heather Albrecht: 548.326.4584 919 Park Place"""

The entries are separated by one or more newlines. Now we convert the
string into a list with each nonempty line having its own entry:

    >>> entries = re.split("\n+", input)
    >>> entries
    ['Ross McFluff: 834.345.1254 155 Elm Street',
    'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
    'Frank Burger: 925.541.7625 662 South Dogwood Way',
    'Heather Albrecht: 548.326.4584 919 Park Place']

Finally, split each entry into a list with first name, last name,
telephone number, and address.  We use the `maxsplit' parameter of
split().  because the address has spaces, our splitting
pattern, in it:

    >>> [re.split(":? ", entry, 3) for entry in entries]
    [['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
    ['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
    ['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
    ['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]

The `:?' pattern matches the colon after the last name, so that it does
not occur in the result list.  With a `maxsplit' of `4', we could
separate the house number from the street name:

    >>> [re.split(":? ", entry, 4) for entry in entries]
    [['Ross', 'McFluff', '834.345.1254', '155', 'Elm Street'],
    ['Ronald', 'Heathmore', '892.345.3428', '436', 'Finley Avenue'],
    ['Frank', 'Burger', '925.541.7625', '662', 'South Dogwood Way'],
    ['Heather', 'Albrecht', '548.326.4584', '919', 'Park Place']]

5.7.2.12 Text Munging
.....................

sub() replaces every occurrence of a pattern with a string
or the result of a function.  This example demonstrates using
sub() with a function to "munge" text, or randomize the order of
all the characters in each word of a sentence except for the first and
last characters:

    >>> def repl(m):
    ...   inner_word = list(m.group(2))
    ...   random.shuffle(inner_word)
    ...   return m.group(1) + "".join(inner_word) + m.group(3)
    >>> text = "Professor Abdolmalek, please report your absences promptly."
    >>> re.sub(r"(\w)(\w+)(\w)", repl, text)
    'Poefsrosr Aealmlobdk, pslaee reorpt your abnseces plmrptoy.'
    >>> re.sub(r"(\w)(\w+)(\w)", repl, text)
    'Pofsroser Aodlambelk, plasee reoprt yuor asnebces potlmrpy.'

5.7.2.13 Finding all Adverbs
............................

findall() matches _all_ occurrences of a pattern, not just
the first one as search() does.  For example, if one was a
writer and wanted to find all of the adverbs in some text, he or she
might use findall() in the following manner:

    >>> text = "He was carefully disguised but captured quickly by police."
    >>> re.findall(r"\w+ly", text)
    ['carefully', 'quickly']

5.7.2.14 Finding all Adverbs and their Positions
................................................

If one wants more information about all matches of a pattern than the
matched text, see finditer() is useful as it provides instances
of MatchObject instead of strings.  Continuing with the
previous example, if one was a writer who wanted to find all of the
adverbs _and their positions_ in some text, he or she would use
finditer() in the following manner:

    >>> text = "He was carefully disguised but captured quickly by police."
    >>> for m in re.finditer(r"\w+ly", text):
    ...     print '%02d-%02d: %s' % (m.start(), m.end(), m.group(0))
    07-16: carefully
    40-47: quickly

5.7.2.15 Raw String Notation
............................

Raw string notation (`r"text"') keeps regular expressions sane.
Without it, every backslash (`'\'') in a regular expression would have
to be prefixed with another one to escape it.  For example, the two
following lines of code are functionally identical:

    >>> re.match(r"\W(.)\1\W", " ff ")
    <_sre.SRE_Match object at ...>
    >>> re.match("\\W(.)\\1\\W", " ff ")
    <_sre.SRE_Match object at ...>

When one wants to match a literal backslash, it must be escaped in the
regular expression.  With raw string notation, this means `r"\\"'.
Without raw string notation, one must use `"\\\\"', making the
following lines of code functionally identical:

    >>> re.match(r"\\", r"\\")
    <_sre.SRE_Match object at ...>
    >>> re.match("\\\\", r"\\")
    <_sre.SRE_Match object at ...>

5.7.3 `struct' -- Interpret strings as packed binary data
---------------------------------------------------------

This module performs conversions between Python values and C structs
represented as Python strings.  This can be used in handling binary
data stored in files or from network connections, among other sources.
It uses Format Strings as compact descriptions of the
layout of the C structs and the intended conversion to/from Python
values.

     Note: By default, the result of packing a given C struct includes
     pad bytes in order to maintain proper alignment for the C types
     involved; similarly, alignment is taken into account when
     unpacking.  This behavior is chosen so that the bytes of a packed
     struct correspond exactly to the layout in memory of the
     corresponding C struct.  To handle platform-independent data
     formats or omit implicit pad bytes, use `standard' size and
     alignment instead of `native' size and alignment: see Byte Order, Size, and Alignment
     for details.

5.7.3.1 Functions and Exceptions
................................

The module defines the following exception and functions:

 -- Exception: struct.error
     Exception raised on various occasions; argument is a string
     describing what is wrong.

 -- Function: struct.pack (fmt, v1, v2, ...)
     Return a string containing the values `v1, v2, ...' packed
     according to the given format.  The arguments must match the
     values required by the format exactly.

 -- Function: struct.pack_into (fmt, buffer, offset, v1, v2, ...)
     Pack the values `v1, v2, ...' according to the given format, write
     the packed bytes into the writable _buffer_ starting at _offset_.
     Note that the offset is a required argument.

     New in version 2.5.

 -- Function: struct.unpack (fmt, string)
     Unpack the string (presumably packed by `pack(fmt, ...)')
     according to the given format.  The result is a tuple even if it
     contains exactly one item.  The string must contain exactly the
     amount of data required by the format (`len(string)' must equal
     `calcsize(fmt)').

 -- Function: struct.unpack_from (fmt, buffer[, offset=0])
     Unpack the _buffer_ according to the given format. The result is a
     tuple even if it contains exactly one item. The _buffer_ must
     contain at least the amount of data required by the format
     (`len(buffer[offset:])' must be at least `calcsize(fmt)').

     New in version 2.5.

 -- Function: struct.calcsize (fmt)
     Return the size of the struct (and hence of the string)
     corresponding to the given format.

5.7.3.2 Format Strings
......................

Format strings are the mechanism used to specify the expected layout
when packing and unpacking data.  They are built up from Format Characters,
which specify the type of data being packed/unpacked.
In addition, there are special characters for controlling the
Byte Order, Size, and Alignment.

5.7.3.3 Byte Order, Size, and Alignment
.......................................

By default, C types are represented in the machine's native format and
byte order, and properly aligned by skipping pad bytes if necessary
(according to the rules used by the C compiler).

  Alternatively, the first character of the format string can be used
to indicate the byte order, size and alignment of the packed data,
according to the following table:

Character       Byte order                   Size           Alignment
----------------------------------------------------------------------------
`@'             native                       native         native
`='             native                       standard       none
`<'             little-endian                standard       none
`>'             big-endian                   standard       none
`!'             network (= big-endian)       standard       none

  If the first character is not one of these, `'@'' is assumed.

  Native byte order is big-endian or little-endian, depending on the
host system. For example, Intel x86 and AMD64 (x86-64) are
little-endian; Motorola 68000 and PowerPC G5 are big-endian; ARM and
Intel Itanium feature switchable endianness (bi-endian). Use
`sys.byteorder' to check the endianness of your system.

  Native size and alignment are determined using the C compiler's
`sizeof' expression.  This is always combined with native byte order.

  Standard size depends only on the format character;  see the table in
the Format Characters section.

  Note the difference between `'@'' and `'='': both use native byte
order, but the size and alignment of the latter is standardized.

  The form `'!'' is available for those poor souls who claim they can't
remember whether network byte order is big-endian or little-endian.

  There is no way to indicate non-native byte order (force
byte-swapping); use the appropriate choice of `'<'' or `'>''.

  Notes:

  1. Padding is only automatically added between successive structure
     members.  No padding is added at the beginning or the end of the
     encoded struct.

  2. No padding is added when using non-native size and alignment, e.g.
     with '<', '>', '=', and '!'.

  3. To align the end of a structure to the alignment requirement of a
     particular type, end the format with the code for that type with a
     repeat count of zero.  See Examples.

5.7.3.4 Format Characters
.........................

Format characters have the following meaning; the conversion between C
and Python values should be obvious given their types.  The 'Standard
size' column refers to the size of the packed value in bytes when using
standard size; that is, when the format string starts with one of
`'<'', `'>'', `'!'' or `'=''.  When using native size, the size of the
packed value is platform-dependent.

Format       C Type                        Python type              Standard size        Notes
----------------------------------------------------------------------------------------------------------
`x'          pad byte                      no value
`c'          `char'                        string of length 1       1
`b'          `signed char'                 integer                  1                    (3)
`B'          `unsigned char'               integer                  1                    (3)
`?'          `_Bool'                       bool                     1                    (1)
`h'          `short'                       integer                  2                    (3)
`H'          `unsigned short'              integer                  2                    (3)
`i'          `int'                         integer                  4                    (3)
`I'          `unsigned int'                integer                  4                    (3)
`l'          `long'                        integer                  4                    (3)
`L'          `unsigned long'               integer                  4                    (3)
`q'          `long long'                   integer                  8                    (2), (3)
`Q'          `unsigned long long'          integer                  8                    (2), (3)
`f'          `float'                       float                    4                    (4)
`d'          `double'                      float                    8                    (4)
`s'          `char[]'                      string
`p'          `char[]'                      string
`P'          `void *'                      integer                                       (5), (3)

  Notes:

  1. The `'?'' conversion code corresponds to the `_Bool' type defined
     by C99. If this type is not available, it is simulated using a
     `char'. In standard mode, it is always represented by one byte.

     New in version 2.6.

  2. The `'q'' and `'Q'' conversion codes are available in native mode
     only if the platform C compiler supports C `long long', or, on
     Windows, `__int64'.  They are always available in standard modes.

     New in version 2.2.

  3. When attempting to pack a non-integer using any of the integer
     conversion codes, if the non-integer has a __index__().
     method then that method is called to convert the argument to an
     integer before packing.  If no __index__() method
     exists, or the call to __index__() raises
     TypeError, then the __int__() method is tried.
     However, the use of __int__() is deprecated, and will
     raise DeprecationWarning.

     Changed in version 2.7: Use of the __index__() method
     for non-integers is new in 2.7.

     Changed in version 2.7: Prior to version 2.7, not all integer
     conversion codes would use the __int__() method to
     convert, and DeprecationWarning was raised only for
     float arguments.

  4. For the `'f'' and `'d'' conversion codes, the packed
     representation uses the IEEE 754 binary32 (for `'f'') or binary64
     (for `'d'') format, regardless of the floating-point format used
     by the platform.

  5. The `'P'' format character is only available for the native byte
     ordering (selected as the default or with the `'@'' byte order
     character). The byte order character `'='' chooses to use little-
     or big-endian ordering based on the host system. The struct module
     does not interpret this as native ordering, so the `'P'' format is
     not available.

  A format character may be preceded by an integral repeat count.  For
example, the format string `'4h'' means exactly the same as `'hhhh''.

  Whitespace characters between formats are ignored; a count and its
format must not contain whitespace though.

  For the `'s'' format character, the count is interpreted as the size
of the string, not a repeat count like for the other format characters;
for example, `'10s'' means a single 10-byte string, while `'10c'' means
10 characters.  For packing, the string is truncated or padded with
null bytes as appropriate to make it fit. For unpacking, the resulting
string always has exactly the specified number of bytes.  As a special
case, `'0s'' means a single, empty string (while `'0c'' means 0
characters).

  The `'p'' format character encodes a "Pascal string", meaning a short
variable-length string stored in a _fixed number of bytes_, given by
the count.  The first byte stored is the length of the string, or 255,
whichever is smaller.  The bytes of the string follow.  If the string
passed in to pack() is too long (longer than the count
minus 1), only the leading `count-1' bytes of the string are stored.
If the string is shorter than `count-1', it is padded with null bytes
so that exactly count bytes in all are used.  Note that for
unpack(), the `'p'' format character consumes count bytes, but
that the string returned can never contain more than 255 characters.

  For the `'P'' format character, the return value is a Python integer
or long integer, depending on the size needed to hold a pointer when it
has been cast to an integer type.  A _NULL_ pointer will always be
returned as the Python integer `0'. When packing pointer-sized values,
Python integer or long integer objects may be used.  For example, the
Alpha and Merced processors use 64-bit pointer values, meaning a Python
long integer will be used to hold the pointer; other platforms use
32-bit pointers and will use a Python integer.

  For the `'?'' format character, the return value is either
True or False. When packing, the truth value of the
argument object is used.  Either 0 or 1 in the native or standard bool
representation will be packed, and any non-zero value will be True when
unpacking.

5.7.3.5 Examples
................

     Note: All examples assume a native byte order, size, and alignment
     with a big-endian machine.

A basic example of packing/unpacking three integers:

    >>> from struct import *
    >>> pack('hhl', 1, 2, 3)
    '\x00\x01\x00\x02\x00\x00\x00\x03'
    >>> unpack('hhl', '\x00\x01\x00\x02\x00\x00\x00\x03')
    (1, 2, 3)
    >>> calcsize('hhl')
    8

Unpacked fields can be named by assigning them to variables or by
wrapping the result in a named tuple:

    >>> record = 'raymond   \x32\x12\x08\x01\x08'
    >>> name, serialnum, school, gradelevel = unpack('<10sHHb', record)

    >>> from collections import namedtuple
    >>> Student = namedtuple('Student', 'name serialnum school gradelevel')
    >>> Student._make(unpack('<10sHHb', s))
    Student(name='raymond   ', serialnum=4658, school=264, gradelevel=8)

The ordering of format characters may have an impact on size since the
padding needed to satisfy alignment requirements is different:

    >>> pack('ci', '*', 0x12131415)
    '*\x00\x00\x00\x12\x13\x14\x15'
    >>> pack('ic', 0x12131415, '*')
    '\x12\x13\x14\x15*'
    >>> calcsize('ci')
    8
    >>> calcsize('ic')
    5

The following format `'llh0l'' specifies two pad bytes at the end,
assuming longs are aligned on 4-byte boundaries:

    >>> pack('llh0l', 1, 2, 3)
    '\x00\x00\x00\x01\x00\x00\x00\x02\x00\x03\x00\x00'

This only works when native size and alignment are in effect; standard
size and alignment does not enforce any alignment.

Module array.
     Packed binary storage of homogeneous data.

Module xdrlib.
     Packing and unpacking of XDR data.

5.7.3.6 Classes
...............

The struct module also defines the following type:

 -- Class: struct.Struct (format)
     Return a new Struct object which writes and reads binary data
     according to the format string _format_.  Creating a Struct object
     once and calling its methods is more efficient than calling the
     struct functions with the same format since the format
     string only needs to be compiled once.

     New in version 2.5.

     Compiled Struct objects support the following methods and
     attributes:

      -- Method: pack (v1, v2, ...)
          Identical to the pack() function, using the
          compiled format.  (`len(result)' will equal `self.size'.)

      -- Method: pack_into (buffer, offset, v1, v2, ...)
          Identical to the pack_into() function, using the
          compiled format.

      -- Method: unpack (string)
          Identical to the unpack() function, using the
          compiled format.  (`len(string)' must equal `self.size').

      -- Method: unpack_from (buffer[, offset=0])
          Identical to the unpack_from() function, using
          the compiled format.  (`len(buffer[offset:])' must be at
          least `self.size').

      -- Attribute: format
          The format string used to construct this Struct object.

      -- Attribute: size
          The calculated size of the struct (and hence of the string)
          corresponding to format.

5.7.4 `difflib' -- Helpers for computing deltas
-----------------------------------------------

New in version 2.1.

  This module provides classes and functions for comparing sequences. It
can be used for example, for comparing files, and can produce difference
information in various formats, including HTML and context and unified
diffs. For comparing directories and files, see also, the
filecmp module.

 -- Class: difflib.SequenceMatcher
     This is a flexible class for comparing pairs of sequences of any
     type, so long as the sequence elements are hashable.
     The basic algorithm predates, and is a little fancier than, an
     algorithm published in the late 1980's by Ratcliff and Obershelp
     under the hyperbolic name "gestalt pattern matching."  The idea is
     to find the longest contiguous matching subsequence that contains
     no "junk" elements (the Ratcliff and Obershelp algorithm doesn't
     address junk).  The same idea is then applied recursively to the
     pieces of the sequences to the left and to the right of the
     matching subsequence.  This does not yield minimal edit sequences,
     but does tend to yield matches that "look right" to people.

     *Timing:* The basic Ratcliff-Obershelp algorithm is cubic time in
     the worst case and quadratic time in the expected case.
     SequenceMatcher is quadratic time for the worst case and has
     expected-case behavior dependent in a complicated way on how many
     elements the sequences have in common; best case time is linear.

     *Automatic junk heuristic:* SequenceMatcher supports a
     heuristic that automatically treats certain sequence items as
     junk. The heuristic counts how many times each individual item
     appears in the sequence. If an item's duplicates (after the first
     one) account for more than 1% of the sequence and the sequence is
     at least 200 items long, this item is marked as "popular" and is
     treated as junk for the purpose of sequence matching. This
     heuristic can be turned off by setting the `autojunk' argument to
     `False' when creating the SequenceMatcher.

     New in version 2.7.1: The _autojunk_ parameter.

 -- Class: difflib.Differ
     This is a class for comparing sequences of lines of text, and
     producing human-readable differences or deltas.  Differ uses
     SequenceMatcher.  both to compare sequences of lines, and to
     compare sequences of characters within similar (near-matching)
     lines.

     Each line of a Differ delta begins with a two-letter
     code:

     Code           Meaning
     ---------------------------------------------------------------
     `'- ''         line unique to sequence 1
     `'+ ''         line unique to sequence 2
     `'  ''         line common to both sequences
     `'? ''         line not present in either input sequence

     Lines beginning with '`?'' attempt to guide the eye to intraline
     differences, and were not present in either input sequence. These
     lines can be confusing if the sequences contain tab characters.

 -- Class: difflib.HtmlDiff
     This class can be used to create an HTML table (or a complete HTML
     file containing the table) showing a side by side, line by line
     comparison of text with inter-line and intra-line change
     highlights.  The table can be generated in either full or
     contextual difference mode.

     The constructor for this class is:

      -- Function: __init__ ([tabsize][, wrapcolumn][, linejunk][,
               charjunk])
          Initializes instance of HtmlDiff.

          _tabsize_ is an optional keyword argument to specify tab stop
          spacing and defaults to `8'.

          _wrapcolumn_ is an optional keyword to specify column number
          where lines are broken and wrapped, defaults to `None' where
          lines are not wrapped.

          _linejunk_ and _charjunk_ are optional keyword arguments
          passed into `ndiff()' (used by HtmlDiff to
          generate the side by side HTML differences).  See `ndiff()'
          documentation for argument default values and descriptions.

     The following methods are public:

      -- Function: make_file (fromlines, tolines [, fromdesc][,
               todesc][, context][, numlines])
          Compares _fromlines_ and _tolines_ (lists of strings) and
          returns a string which is a complete HTML file containing a
          table showing line by line differences with inter-line and
          intra-line changes highlighted.

          _fromdesc_ and _todesc_ are optional keyword arguments to
          specify from/to file column header strings (both default to
          an empty string).

          _context_ and _numlines_ are both optional keyword arguments.
          Set _context_ to `True' when contextual differences are to be
          shown, else the default is `False' to show the full files.
          _numlines_ defaults to `5'.  When _context_ is `True'
          _numlines_ controls the number of context lines which
          surround the difference highlights.  When _context_ is
          `False' _numlines_ controls the number of lines which are
          shown before a difference highlight when using the "next"
          hyperlinks (setting to zero would cause the "next" hyperlinks
          to place the next difference highlight at the top of the
          browser without any leading context).

      -- Function: make_table (fromlines, tolines [, fromdesc][,
               todesc][, context][, numlines])
          Compares _fromlines_ and _tolines_ (lists of strings) and
          returns a string which is a complete HTML table showing line
          by line differences with inter-line and intra-line changes
          highlighted.

          The arguments for this method are the same as those for the
          make_file().  method.

     `Tools/scripts/diff.py' is a command-line front-end to this class
     and contains a good example of its use.

     New in version 2.4.

 -- Function: difflib.context_diff (a, b[, fromfile][, tofile][,
          fromfiledate][, tofiledate][, n][, lineterm])
     Compare _a_ and _b_ (lists of strings); return a delta (a
     generator.  generating the delta lines) in context diff
     format.

     Context diffs are a compact way of showing just the lines that
     have changed plus a few lines of context.  The changes are shown
     in a before/after style.  The number of context lines is set by
     _n_ which defaults to three.

     By default, the diff control lines (those with `***' or `---') are
     created with a trailing newline.  This is helpful so that inputs
     created from file.readlines() result in diffs that are
     suitable for use with file.writelines() since both the
     inputs and outputs have trailing newlines.

     For inputs that do not have trailing newlines, set the _lineterm_
     argument to `""' so that the output will be uniformly newline free.

     The context diff format normally has a header for filenames and
     modification times.  Any or all of these may be specified using
     strings for _fromfile_, _tofile_, _fromfiledate_, and
     _tofiledate_.  The modification times are normally expressed in
     the ISO 8601 format. If not specified, the strings default to
     blanks.

         >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n']
         >>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n']
         >>> for line in context_diff(s1, s2, fromfile='before.py', tofile='after.py'):
         ...     sys.stdout.write(line)  # doctest: +NORMALIZE_WHITESPACE
         *** before.py
         --- after.py
         ***************
         *** 1,4 ****
         ! bacon
         ! eggs
         ! ham
           guido
         --- 1,4 ----
         ! python
         ! eggy
         ! hamster
           guido

     See A command-line interface to difflib for a more
     detailed example.

     New in version 2.3.

 -- Function: difflib.get_close_matches (word, possibilities[, n][,
          cutoff])
     Return a list of the best "good enough" matches.  _word_ is a
     sequence for which close matches are desired (typically a string),
     and _possibilities_ is a list of sequences against which to match
     _word_ (typically a list of strings).

     Optional argument _n_ (default `3') is the maximum number of close
     matches to return; _n_ must be greater than `0'.

     Optional argument _cutoff_ (default `0.6') is a float in the range
     [0, 1].  Possibilities that don't score at least that similar to
     _word_ are ignored.

     The best (no more than _n_) matches among the possibilities are
     returned in a list, sorted by similarity score, most similar first.

         >>> get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'])
         ['apple', 'ape']
         >>> import keyword
         >>> get_close_matches('wheel', keyword.kwlist)
         ['while']
         >>> get_close_matches('apple', keyword.kwlist)
         []
         >>> get_close_matches('accept', keyword.kwlist)
         ['except']

 -- Function: difflib.ndiff (a, b[, linejunk][, charjunk])
     Compare _a_ and _b_ (lists of strings); return a Differ.
     -style delta (a generator generating the delta
     lines).

     Optional keyword parameters _linejunk_ and _charjunk_ are for
     filter functions (or `None'):

     _linejunk_: A function that accepts a single string argument, and
     returns true if the string is junk, or false if not. The default
     is (`None'), starting with Python 2.3.  Before then, the default
     was the module-level function IS_LINE_JUNK(), which
     filters out lines without visible characters, except for at most
     one pound character (`'#''). As of Python 2.3, the underlying
     SequenceMatcher class does a dynamic analysis of which
     lines are so frequent as to constitute noise, and this usually
     works better than the pre-2.3 default.

     _charjunk_: A function that accepts a character (a string of
     length 1), and returns if the character is junk, or false if not.
     The default is module-level function IS_CHARACTER_JUNK(),
     which filters out whitespace characters (a blank or tab;
     note: bad idea to include newline in this!).

     `Tools/scripts/ndiff.py' is a command-line front-end to this
     function.

         >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(1),
         ...              'ore\ntree\nemu\n'.splitlines(1))
         >>> print ''.join(diff),
         - one
         ?  ^
         + ore
         ?  ^
         - two
         - three
         ?  -
         + tree
         + emu

 -- Function: difflib.restore (sequence, which)
     Return one of the two sequences that generated a delta.

     Given a _sequence_ produced by Differ.compare() or
     ndiff(), extract lines originating from file 1 or 2
     (parameter _which_), stripping off line prefixes.

     Example:

         >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(1),
         ...              'ore\ntree\nemu\n'.splitlines(1))
         >>> diff = list(diff) # materialize the generated delta into a list
         >>> print ''.join(restore(diff, 1)),
         one
         two
         three
         >>> print ''.join(restore(diff, 2)),
         ore
         tree
         emu

 -- Function: difflib.unified_diff (a, b[, fromfile][, tofile][,
          fromfiledate][, tofiledate][, n][, lineterm])
     Compare _a_ and _b_ (lists of strings); return a delta (a
     generator.  generating the delta lines) in unified diff
     format.

     Unified diffs are a compact way of showing just the lines that
     have changed plus a few lines of context.  The changes are shown
     in a inline style (instead of separate before/after blocks).  The
     number of context lines is set by _n_ which defaults to three.

     By default, the diff control lines (those with `---', `+++', or
     `@@') are created with a trailing newline.  This is helpful so
     that inputs created from file.readlines() result in
     diffs that are suitable for use with file.writelines().
     since both the inputs and outputs have trailing newlines.

     For inputs that do not have trailing newlines, set the _lineterm_
     argument to `""' so that the output will be uniformly newline free.

     The context diff format normally has a header for filenames and
     modification times.  Any or all of these may be specified using
     strings for _fromfile_, _tofile_, _fromfiledate_, and
     _tofiledate_.  The modification times are normally expressed in
     the ISO 8601 format. If not specified, the strings default to
     blanks.

         >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n']
         >>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n']
         >>> for line in unified_diff(s1, s2, fromfile='before.py', tofile='after.py'):
         ...     sys.stdout.write(line)   # doctest: +NORMALIZE_WHITESPACE
         --- before.py
         +++ after.py
         @@ -1,4 +1,4 @@
         -bacon
         -eggs
         -ham
         +python
         +eggy
         +hamster
          guido

     See A command-line interface to difflib for a more
     detailed example.

     New in version 2.3.

 -- Function: difflib.IS_LINE_JUNK (line)
     Return true for ignorable lines.  The line _line_ is ignorable if
     _line_ is blank or contains a single `'#'', otherwise it is not
     ignorable.  Used as a default for parameter _linejunk_ in
     ndiff() before Python 2.3.

 -- Function: difflib.IS_CHARACTER_JUNK (ch)
     Return true for ignorable characters.  The character _ch_ is
     ignorable if _ch_ is a space or tab, otherwise it is not
     ignorable.  Used as a default for parameter _charjunk_ in
     ndiff().

Pattern Matching: The Gestalt Approach (http://www.ddj.com/184407970?pgno=5)
     Discussion of a similar algorithm by John W. Ratcliff and D. E.
     Metzener. This was published in Dr. Dobb's Journal
     (http://www.ddj.com/)in July, 1988.

5.7.4.1 SequenceMatcher Objects
...............................

The SequenceMatcher class has this constructor:

 -- Class: difflib.SequenceMatcher ([isjunk[, a[, b[, autojunk=True]]]])
     Optional argument _isjunk_ must be `None' (the default) or a
     one-argument function that takes a sequence element and returns
     true if and only if the element is "junk" and should be ignored.
     Passing `None' for _isjunk_ is equivalent to passing `lambda x:
     0'; in other words, no elements are ignored.  For example, pass:

         lambda x: x in " \t"

     if you're comparing lines as sequences of characters, and don't
     want to synch up on blanks or hard tabs.

     The optional arguments _a_ and _b_ are sequences to be compared;
     both default to empty strings.  The elements of both sequences
     must be hashable.

     The optional argument _autojunk_ can be used to disable the
     automatic junk heuristic.

     New in version 2.7.1: The _autojunk_ parameter.

     SequenceMatcher objects have the following methods:

      -- Method: set_seqs (a, b)
          Set the two sequences to be compared.

     SequenceMatcher computes and caches detailed
     information about the second sequence, so if you want to compare
     one sequence against many sequences, use set_seq2() to
     set the commonly used sequence once and call set_seq1()
     repeatedly, once for each of the other sequences.

      -- Method: set_seq1 (a)
          Set the first sequence to be compared.  The second sequence
          to be compared is not changed.

      -- Method: set_seq2 (b)
          Set the second sequence to be compared.  The first sequence
          to be compared is not changed.

      -- Method: find_longest_match (alo, ahi, blo, bhi)
          Find longest matching block in `a[alo:ahi]' and `b[blo:bhi]'.

          If _isjunk_ was omitted or `None', see
          find_longest_match() returns `(i, j, k)' such that
          `a[i:i+k]' is equal to `b[j:j+k]', where `alo <= i <= i+k <=
          ahi' and `blo <= j <= j+k <= bhi'. For all `(i', j', k')'
          meeting those conditions, the additional conditions `k >=
          k'', `i <= i'', and if `i == i'', `j <= j'' are also met. In
          other words, of all maximal matching blocks, return one that
          starts earliest in _a_, and of all those maximal matching
          blocks that start earliest in _a_, return the one that starts
          earliest in _b_.

              >>> s = SequenceMatcher(None, " abcd", "abcd abcd")
              >>> s.find_longest_match(0, 5, 0, 9)
              Match(a=0, b=4, size=5)

          If _isjunk_ was provided, first the longest matching block is
          determined as above, but with the additional restriction that
          no junk element appears in the block.  Then that block is
          extended as far as possible by matching (only) junk elements
          on both sides. So the resulting block never matches on junk
          except as identical junk happens to be adjacent to an
          interesting match.

          Here's the same example as before, but considering blanks to
          be junk. That prevents `' abcd'' from matching the `' abcd''
          at the tail end of the second sequence directly.  Instead
          only the `'abcd'' can match, and matches the leftmost
          `'abcd'' in the second sequence:

              >>> s = SequenceMatcher(lambda x: x==" ", " abcd", "abcd abcd")
              >>> s.find_longest_match(0, 5, 0, 9)
              Match(a=1, b=0, size=4)

          If no blocks match, this returns `(alo, blo, 0)'.

          Changed in version 2.6: This method returns a named tuple.
          `Match(a, b, size)'.

      -- Method: get_matching_blocks ()
          Return list of triples describing matching subsequences. Each
          triple is of the form `(i, j, n)', and means that `a[i:i+n]
          == b[j:j+n]'.  The triples are monotonically increasing in
          _i_ and _j_.

          The last triple is a dummy, and has the value `(len(a),
          len(b), 0)'.  It is the only triple with `n == 0'.  If `(i,
          j, n)' and `(i', j', n')' are adjacent triples in the list,
          and the second is not the last triple in the list, then `i+n
          != i'' or `j+n != j''; in other words, adjacent triples
          always describe non-adjacent equal blocks.

          Changed in version 2.5: The guarantee that adjacent triples
          always describe non-adjacent blocks was implemented.

              >>> s = SequenceMatcher(None, "abxcd", "abcd")
              >>> s.get_matching_blocks()
              [Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)]

      -- Method: get_opcodes ()
          Return list of 5-tuples describing how to turn _a_ into _b_.
          Each tuple is of the form `(tag, i1, i2, j1, j2)'.  The first
          tuple has `i1 == j1 == 0', and remaining tuples have _i1_
          equal to the _i2_ from the preceding tuple, and, likewise,
          _j1_ equal to the previous _j2_.

          The _tag_ values are strings, with these meanings:

          Value               Meaning
          ----------------------------------------------------------------------
          `'replace''         `a[i1:i2]' should be replaced by `b[j1:j2]'.
          `'delete''          `a[i1:i2]' should be deleted.  Note that `j1 ==
                              j2' in this case.
          `'insert''          `b[j1:j2]' should be inserted at `a[i1:i1]'.
                              Note that `i1 == i2' in this case.
          `'equal''           `a[i1:i2] == b[j1:j2]' (the sub-sequences are
                              equal).

          For example:

              >>> a = "qabxcd"
              >>> b = "abycdf"
              >>> s = SequenceMatcher(None, a, b)
              >>> for tag, i1, i2, j1, j2 in s.get_opcodes():
              ...    print ("%7s a[%d:%d] (%s) b[%d:%d] (%s)" %
              ...           (tag, i1, i2, a[i1:i2], j1, j2, b[j1:j2]))
               delete a[0:1] (q) b[0:0] ()
                equal a[1:3] (ab) b[0:2] (ab)
              replace a[3:4] (x) b[2:3] (y)
                equal a[4:6] (cd) b[3:5] (cd)
               insert a[6:6] () b[5:6] (f)

      -- Method: get_grouped_opcodes ([n])
          Return a generator of groups with up to _n_ lines
          of context.

          Starting with the groups returned by get_opcodes(),
          this method splits out smaller change clusters and
          eliminates intervening ranges which have no changes.

          The groups are returned in the same format as
          get_opcodes().

          New in version 2.3.

      -- Method: ratio ()
          Return a measure of the sequences' similarity as a float in
          the range [0, 1].

          Where T is the total number of elements in both sequences,
          and M is the number of matches, this is 2.0*M / T. Note that
          this is `1.0' if the sequences are identical, and `0.0' if
          they have nothing in common.

          This is expensive to compute if get_matching_blocks()
          or get_opcodes() hasn't already been called,
          in which case you may want to try quick_ratio().
          or real_quick_ratio() first to get an upper bound.

      -- Method: quick_ratio ()
          Return an upper bound on ratio() relatively
          quickly.

      -- Method: real_quick_ratio ()
          Return an upper bound on ratio() very quickly.

  The three methods that return the ratio of matching to total
characters can give different results due to differing levels of
approximation, although `quick_ratio()' and `real_quick_ratio()' are
always at least as large as `ratio()':

    >>> s = SequenceMatcher(None, "abcd", "bcde")
    >>> s.ratio()
    0.75
    >>> s.quick_ratio()
    0.75
    >>> s.real_quick_ratio()
    1.0

5.7.4.2 SequenceMatcher Examples
................................

This example compares two strings, considering blanks to be "junk:"

    >>> s = SequenceMatcher(lambda x: x == " ",
    ...                     "private Thread currentThread;",
    ...                     "private volatile Thread currentThread;")

`ratio()' returns a float in [0, 1], measuring the similarity of the
sequences.  As a rule of thumb, a `ratio()' value over 0.6 means the
sequences are close matches:

    >>> print round(s.ratio(), 3)
    0.866

If you're only interested in where the sequences match,
`get_matching_blocks()' is handy:

    >>> for block in s.get_matching_blocks():
    ...     print "a[%d] and b[%d] match for %d elements" % block
    a[0] and b[0] match for 8 elements
    a[8] and b[17] match for 21 elements
    a[29] and b[38] match for 0 elements

Note that the last tuple returned by `get_matching_blocks()' is always a
dummy, `(len(a), len(b), 0)', and this is the only case in which the
last tuple element (number of elements matched) is `0'.

  If you want to know how to change the first sequence into the second,
use `get_opcodes()':

    >>> for opcode in s.get_opcodes():
    ...     print "%6s a[%d:%d] b[%d:%d]" % opcode
     equal a[0:8] b[0:8]
    insert a[8:8] b[8:17]
     equal a[8:29] b[17:38]

   * The get_close_matches() function in this module which
     shows how simple code building on SequenceMatcher can
     be used to do useful work.

   * Simple version control recipe
     (http://code.activestate.com/recipes/576729/)for a small
     application built with SequenceMatcher.

5.7.4.3 Differ Objects
......................

Note that Differ.-generated deltas make no claim to be
*minimal* diffs. To the contrary, minimal diffs are often
counter-intuitive, because they synch up anywhere possible, sometimes
accidental matches 100 pages apart.  Restricting synch points to
contiguous matches preserves some notion of locality, at the occasional
cost of producing a longer diff.

  The Differ class has this constructor:

 -- Class: difflib.Differ ([linejunk[, charjunk]])
     Optional keyword parameters _linejunk_ and _charjunk_ are for
     filter functions (or `None'):

     _linejunk_: A function that accepts a single string argument, and
     returns true if the string is junk.  The default is `None',
     meaning that no line is considered junk.

     _charjunk_: A function that accepts a single character argument (a
     string of length 1), and returns true if the character is junk.
     The default is `None', meaning that no character is considered
     junk.

     Differ objects are used (deltas generated) via a
     single method:

      -- Method: compare (a, b)
          Compare two sequences of lines, and generate the delta (a
          sequence of lines).

          Each sequence must contain individual single-line strings
          ending with newlines.  Such sequences can be obtained from
          the `readlines()' method of file-like objects.  The delta
          generated also consists of newline-terminated strings, ready
          to be printed as-is via the `writelines()' method of a
          file-like object.

5.7.4.4 Differ Example
......................

This example compares two texts. First we set up the texts, sequences of
individual single-line strings ending with newlines (such sequences can
also be obtained from the `readlines()' method of file-like objects):

    >>> text1 = '''  1. Beautiful is better than ugly.
    ...   2. Explicit is better than implicit.
    ...   3. Simple is better than complex.
    ...   4. Complex is better than complicated.
    ... '''.splitlines(1)
    >>> len(text1)
    4
    >>> text1[0][-1]
    '\n'
    >>> text2 = '''  1. Beautiful is better than ugly.
    ...   3.   Simple is better than complex.
    ...   4. Complicated is better than complex.
    ...   5. Flat is better than nested.
    ... '''.splitlines(1)

Next we instantiate a Differ object:

    >>> d = Differ()

Note that when instantiating a Differ object we may pass
functions to filter out line and character "junk."  See the
Differ() constructor for details.

  Finally, we compare the two:

    >>> result = list(d.compare(text1, text2))

`result' is a list of strings, so let's pretty-print it:

    >>> from pprint import pprint
    >>> pprint(result)
    ['    1. Beautiful is better than ugly.\n',
     '-   2. Explicit is better than implicit.\n',
     '-   3. Simple is better than complex.\n',
     '+   3.   Simple is better than complex.\n',
     '?     ++\n',
     '-   4. Complex is better than complicated.\n',
     '?            ^                     ---- ^\n',
     '+   4. Complicated is better than complex.\n',
     '?           ++++ ^                      ^\n',
     '+   5. Flat is better than nested.\n']

As a single multi-line string it looks like this:

    >>> import sys
    >>> sys.stdout.writelines(result)
        1. Beautiful is better than ugly.
    -   2. Explicit is better than implicit.
    -   3. Simple is better than complex.
    +   3.   Simple is better than complex.
    ?     ++
    -   4. Complex is better than complicated.
    ?            ^                     ---- ^
    +   4. Complicated is better than complex.
    ?           ++++ ^                      ^
    +   5. Flat is better than nested.

5.7.4.5 A command-line interface to difflib
...........................................

This example shows how to use difflib to create a `diff'-like utility.
It is also contained in the Python source distribution, as
`Tools/scripts/diff.py'.

    """ Command line interface to difflib.py providing diffs in four formats:

    * ndiff:    lists every line and highlights interline changes.
    * context:  highlights clusters of changes in a before/after format.
    * unified:  highlights clusters of changes in an inline format.
    * html:     generates side by side comparison with change highlights.

    """

    import sys, os, time, difflib, optparse

    def main():
         # Configure the option parser
        usage = "usage: %prog [options] fromfile tofile"
        parser = optparse.OptionParser(usage)
        parser.add_option("-c", action="store_true", default=False,
                          help='Produce a context format diff (default)')
        parser.add_option("-u", action="store_true", default=False,
                          help='Produce a unified format diff')
        hlp = 'Produce HTML side by side diff (can use -c and -l in conjunction)'
        parser.add_option("-m", action="store_true", default=False, help=hlp)
        parser.add_option("-n", action="store_true", default=False,
                          help='Produce a ndiff format diff')
        parser.add_option("-l", "--lines", type="int", default=3,
                          help='Set number of context lines (default 3)')
        (options, args) = parser.parse_args()

        if len(args) == 0:
            parser.print_help()
            sys.exit(1)
        if len(args) != 2:
            parser.error("need to specify both a fromfile and tofile")

        n = options.lines
        fromfile, tofile = args # as specified in the usage string

        # we're passing these as arguments to the diff function
        fromdate = time.ctime(os.stat(fromfile).st_mtime)
        todate = time.ctime(os.stat(tofile).st_mtime)
        fromlines = open(fromfile, 'U').readlines()
        tolines = open(tofile, 'U').readlines()

        if options.u:
            diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile,
                                        fromdate, todate, n=n)
        elif options.n:
            diff = difflib.ndiff(fromlines, tolines)
        elif options.m:
            diff = difflib.HtmlDiff().make_file(fromlines, tolines, fromfile,
                                                tofile, context=options.c,
                                                numlines=n)
        else:
            diff = difflib.context_diff(fromlines, tolines, fromfile, tofile,
                                        fromdate, todate, n=n)

        # we're using writelines because diff is a generator
        sys.stdout.writelines(diff)

    if __name__ == '__main__':
        main()

5.7.5 `StringIO' -- Read and write strings as files
---------------------------------------------------

This module implements a file-like class, see StringIO, that
reads and writes a string buffer (also known as _memory files_).  See
the description of file objects for operations (section File Objects).
(For standard strings, see str and
unicode)

 -- Class: StringIO.StringIO ([buffer])
     When a StringIO object is created, it can be
     initialized to an existing string by passing the string to the
     constructor. If no string is given, the StringIO will
     start empty. In both cases, the initial file position starts at
     zero.

     The StringIO object can accept either Unicode or 8-bit
     strings, but mixing the two may take some care.  If both are used,
     8-bit strings that cannot be interpreted as 7-bit ASCII (that use
     the 8th bit) will cause a UnicodeError to be raised
     when getvalue() is called.

  The following methods of StringIO objects require special
mention:

 -- Method: StringIO.getvalue ()
     Retrieve the entire contents of the "file" at any time before the
     StringIO object's close() method is
     called.  See the note above for information about mixing Unicode
     and 8-bit strings; such mixing can cause this method to raise
     UnicodeError.

 -- Method: StringIO.close ()
     Free the memory buffer.  Attempting to do further operations with
     a closed StringIO object will raise a
     ValueError.

  Example usage:

    import StringIO

    output = StringIO.StringIO()
    output.write('First line.\n')
    print >>output, 'Second line.'

    # Retrieve file contents -- this will be
    # 'First line.\nSecond line.\n'
    contents = output.getvalue()

    # Close object and discard memory buffer --
    # .getvalue() will now raise an exception.
    output.close()

5.7.6 `cStringIO' -- Faster version of `StringIO'
-------------------------------------------------

The module cStringIO provides an interface similar to that
of the StringIO module.  Heavy use of
StringIO.StringIO objects can be made more efficient by using the
function StringIO() from this module instead.

 -- Function: cStringIO.StringIO ([s])
     Return a StringIO-like stream for reading or writing.

     Since this is a factory function which returns objects of built-in
     types, there's no way to build your own version using subclassing.
     It's not possible to set attributes on it.  Use the original
     StringIO module in those cases.

     Unlike the StringIO module, this module is not able to
     accept Unicode strings that cannot be encoded as plain ASCII
     strings.  Calling StringIO() with a Unicode string
     parameter populates the object with the buffer representation of
     the Unicode string instead of encoding the string.

     Another difference from the StringIO module is that
     calling StringIO() with a string parameter creates a
     read-only object. Unlike an object created without a string
     parameter, it does not have write methods.  These objects are not
     generally visible.  They turn up in tracebacks as `StringI' and
     `StringO'.

  The following data objects are provided as well:

 -- Data: cStringIO.InputType
     The type object of the objects created by calling
     StringIO() with a string parameter.

 -- Data: cStringIO.OutputType
     The type object of the objects returned by calling
     StringIO() with no parameters.

  There is a C API to the module as well; refer to the module source
for  more information.

  Example usage:

    import cStringIO

    output = cStringIO.StringIO()
    output.write('First line.\n')
    print >>output, 'Second line.'

    # Retrieve file contents -- this will be
    # 'First line.\nSecond line.\n'
    contents = output.getvalue()

    # Close object and discard memory buffer --
    # .getvalue() will now raise an exception.
    output.close()

5.7.7 `textwrap' -- Text wrapping and filling
---------------------------------------------

New in version 2.3.

  The textwrap module provides two convenience functions,
see wrap() and fill(), as well as TextWrapper,
the class that does all the work, and a utility function
dedent().  If you're just wrapping or filling one or two  text
strings, the convenience functions should be good enough; otherwise,
you should use an instance of TextWrapper for efficiency.

  Latest version of the textwrap module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/textwrap.py?view=markup)

 -- Function: textwrap.wrap (text[, width[, ...]])
     Wraps the single paragraph in _text_ (a string) so every line is
     at most _width_ characters long.  Returns a list of output lines,
     without final newlines.

     Optional keyword arguments correspond to the instance attributes of
     TextWrapper, documented below.  _width_ defaults to
     `70'.

 -- Function: textwrap.fill (text[, width[, ...]])
     Wraps the single paragraph in _text_, and returns a single string
     containing the wrapped paragraph.  fill() is shorthand
     for

         "\n".join(wrap(text, ...))

     In particular, see fill() accepts exactly the same keyword
     arguments as wrap().

  Both wrap() and fill() work by creating a
TextWrapper.  instance and calling a single method on it.
That instance is not reused, so for applications that wrap/fill many
text strings, it will be more efficient for you to create your own
TextWrapper object.

  Text is preferably wrapped on whitespaces and right after the hyphens
in hyphenated words; only then will long words be broken if necessary,
unless TextWrapper.break_long_words is set to false.

  An additional utility function, see dedent(), is provided to
remove indentation from strings that have unwanted whitespace to the
left of the text.

 -- Function: textwrap.dedent (text)
     Remove any common leading whitespace from every line in _text_.

     This can be used to make triple-quoted strings line up with the
     left edge of the display, while still presenting them in the
     source code in indented form.

     Note that tabs and spaces are both treated as whitespace, but they
     are not equal: the lines `"  hello"' and `"\thello"' are
     considered to have no common leading whitespace.  (This behaviour
     is new in Python 2.5; older versions of this module incorrectly
     expanded tabs before searching for common leading whitespace.)

     For example:

         def test():
             # end first line with \ to avoid the empty line!
             s = '''\
             hello
               world
             '''
             print repr(s)          # prints '    hello\n      world\n    '
             print repr(dedent(s))  # prints 'hello\n  world\n'

 -- Class: textwrap.TextWrapper (...)
     The TextWrapper constructor accepts a number of
     optional keyword arguments.  Each argument corresponds to one
     instance attribute, so for example

         wrapper = TextWrapper(initial_indent="* ")

     is the same as

         wrapper = TextWrapper()
         wrapper.initial_indent = "* "

     You can re-use the same TextWrapper object many times,
     and you can change any of its options through direct assignment to
     instance attributes between uses.

     The TextWrapper instance attributes (and keyword
     arguments to the constructor) are as follows:

      -- Attribute: width
          (default: `70') The maximum length of wrapped lines.  As long
          as there are no individual words in the input text longer
          than width, TextWrapper guarantees
          that no output line will be longer than width.
          characters.

      -- Attribute: expand_tabs
          (default: `True') If true, then all tab characters in _text_
          will be expanded to spaces using the `expandtabs()' method of
          _text_.

      -- Attribute: replace_whitespace
          (default: `True') If true, each whitespace character (as
          defined by `string.whitespace') remaining after tab expansion
          will be replaced by a single space.

               Note: If expand_tabs is false and
               replace_whitespace is true, each tab character
               will be replaced by a single space, which is _not_ the
               same as tab expansion.

               Note: If replace_whitespace is false,
               newlines may appear in the middle of a line and cause
               strange output. For this reason, text should be split
               into paragraphs (using str.splitlines() or
               similar) which are wrapped separately.

      -- Attribute: drop_whitespace
          (default: `True') If true, whitespace that, after wrapping,
          happens to end up at the beginning or end of a line is
          dropped (leading whitespace in the first line is always
          preserved, though).

          New in version 2.6: Whitespace was always dropped in earlier
          versions.

      -- Attribute: initial_indent
          (default: `''') String that will be prepended to the first
          line of wrapped output.  Counts towards the length of the
          first line.

      -- Attribute: subsequent_indent
          (default: `''') String that will be prepended to all lines of
          wrapped output except the first.  Counts towards the length
          of each line except the first.

      -- Attribute: fix_sentence_endings
          (default: `False') If true, see TextWrapper attempts
          to detect sentence endings and ensure that sentences are
          always separated by exactly two spaces.  This is generally
          desired for text in a monospaced font.  However, the sentence
          detection algorithm is imperfect: it assumes that a sentence
          ending consists of a lowercase letter followed by one of
          `'.'', `'!'', or `'?'', possibly followed by one of `'"'' or
          `"'"', followed by a space.  One problem with this is
          algorithm is that it is unable to detect the difference
          between "Dr." in

              [...] Dr. Frankenstein's monster [...]

          and "Spot." in

              [...] See Spot. See Spot run [...]

          fix_sentence_endings is false by default.

          Since the sentence detection algorithm relies on
          `string.lowercase' for the definition of "lowercase letter,"
          and a convention of using two spaces after a period to
          separate sentences on the same line, it is specific to
          English-language texts.

      -- Attribute: break_long_words
          (default: `True') If true, then words longer than
          width will be broken in order to ensure that no lines
          are longer than width.  If it is false, long words
          will not be broken, and some lines may be longer than
          width.  (Long words will be put on a line by themselves,
          in order to minimize the amount by which width is
          exceeded.)

      -- Attribute: break_on_hyphens
          (default: `True') If true, wrapping will occur preferably on
          whitespaces and right after hyphens in compound words, as it
          is customary in English.  If false, only whitespaces will be
          considered as potentially good places for line breaks, but
          you need to set break_long_words to false if you
          want truly insecable words.  Default behaviour in previous
          versions was to always allow breaking hyphenated words.

          New in version 2.6.

     TextWrapper also provides two public methods,
     analogous to the module-level convenience functions:

      -- Method: wrap (text)
          Wraps the single paragraph in _text_ (a string) so every line
          is at most width characters long.  All wrapping
          options are taken from instance attributes of the
          TextWrapper instance. Returns a list of output lines,
          without final newlines.

      -- Method: fill (text)
          Wraps the single paragraph in _text_, and returns a single
          string containing the wrapped paragraph.

5.7.8 `codecs' -- Codec registry and base classes
-------------------------------------------------

This module defines base classes for standard Python codecs (encoders
and decoders) and provides access to the internal Python codec registry
which manages the codec and error handling lookup process.

  It defines the following functions:

 -- Function: codecs.register (search_function)
     Register a codec search function. Search functions are expected to
     take one argument, the encoding name in all lower case letters,
     and return a `CodecInfo' object having the following attributes:

        * `name' The name of the encoding;

        * `encode' The stateless encoding function;

        * `decode' The stateless decoding function;

        * `incrementalencoder' An incremental encoder class or factory
          function;

        * `incrementaldecoder' An incremental decoder class or factory
          function;

        * `streamwriter' A stream writer class or factory function;

        * `streamreader' A stream reader class or factory function.

     The various functions or classes take the following arguments:

     _encode_ and _decode_: These must be functions or methods which
     have the same interface as the `encode()'/`decode()' methods of
     Codec instances (see Codec Interface). The functions/methods are
     expected to work in a stateless mode.

     _incrementalencoder_ and _incrementaldecoder_: These have to be
     factory functions providing the following interface:

          `factory(errors='strict')'

     The factory functions must return objects providing the interfaces
     defined by the base classes IncrementalEncoder and
     IncrementalDecoder, respectively. Incremental codecs
     can maintain state.

     _streamreader_ and _streamwriter_: These have to be factory
     functions providing the following interface:

          `factory(stream, errors='strict')'

     The factory functions must return objects providing the interfaces
     defined by the base classes StreamWriter and
     StreamReader, respectively.  Stream codecs can maintain state.

     Possible values for errors are

        * `'strict'': raise an exception in case of an encoding error

        * `'replace'': replace malformed data with a suitable
          replacement marker, such as `'?'' or `'\ufffd''

        * `'ignore'': ignore malformed data and continue without
          further notice

        * `'xmlcharrefreplace'': replace with the appropriate XML
          character reference (for encoding only)

        * `'backslashreplace'': replace with backslashed escape
          sequences (for encoding only)

     as well as any other error handling name defined via
     register_error().

     In case a search function cannot find a given encoding, it should
     return `None'.

 -- Function: codecs.lookup (encoding)
     Looks up the codec info in the Python codec registry and returns a
     `CodecInfo' object as defined above.

     Encodings are first looked up in the registry's cache. If not
     found, the list of registered search functions is scanned. If no
     `CodecInfo' object is found, a LookupError is raised.
     Otherwise, the `CodecInfo' object is stored in the cache and
     returned to the caller.

  To simplify access to the various codecs, the module provides these
additional functions which use lookup() for the codec
lookup:

 -- Function: codecs.getencoder (encoding)
     Look up the codec for the given encoding and return its encoder
     function.

     Raises a LookupError in case the encoding cannot be
     found.

 -- Function: codecs.getdecoder (encoding)
     Look up the codec for the given encoding and return its decoder
     function.

     Raises a LookupError in case the encoding cannot be
     found.

 -- Function: codecs.getincrementalencoder (encoding)
     Look up the codec for the given encoding and return its
     incremental encoder class or factory function.

     Raises a LookupError in case the encoding cannot be
     found or the codec doesn't support an incremental encoder.

     New in version 2.5.

 -- Function: codecs.getincrementaldecoder (encoding)
     Look up the codec for the given encoding and return its
     incremental decoder class or factory function.

     Raises a LookupError in case the encoding cannot be
     found or the codec doesn't support an incremental decoder.

     New in version 2.5.

 -- Function: codecs.getreader (encoding)
     Look up the codec for the given encoding and return its
     StreamReader class or factory function.

     Raises a LookupError in case the encoding cannot be
     found.

 -- Function: codecs.getwriter (encoding)
     Look up the codec for the given encoding and return its
     StreamWriter class or factory function.

     Raises a LookupError in case the encoding cannot be
     found.

 -- Function: codecs.register_error (name, error_handler)
     Register the error handling function _error_handler_ under the
     name _name_.  _error_handler_ will be called during encoding and
     decoding in case of an error, when _name_ is specified as the
     errors parameter.

     For encoding _error_handler_ will be called with a
     UnicodeEncodeError.  instance, which contains information
     about the location of the error. The error handler must either
     raise this or a different exception or return a tuple with a
     replacement for the unencodable part of the input and a position
     where encoding should continue. The encoder will encode the
     replacement and continue encoding the original input at the
     specified position. Negative position values will be treated as
     being relative to the end of the input string. If the resulting
     position is out of bound an IndexError will be raised.

     Decoding and translating works similar, except
     UnicodeDecodeError or UnicodeTranslateError will
     be passed to the handler and that the replacement from the error
     handler will be put into the output directly.

 -- Function: codecs.lookup_error (name)
     Return the error handler previously registered under the name
     _name_.

     Raises a LookupError in case the handler cannot be
     found.

 -- Function: codecs.strict_errors (exception)
     Implements the `strict' error handling: each encoding or decoding
     error raises a UnicodeError.

 -- Function: codecs.replace_errors (exception)
     Implements the `replace' error handling: malformed data is
     replaced with a suitable replacement character such as `'?'' in
     bytestrings and `'\ufffd'' in Unicode strings.

 -- Function: codecs.ignore_errors (exception)
     Implements the `ignore' error handling: malformed data is ignored
     and encoding or decoding is continued without further notice.

 -- Function: codecs.xmlcharrefreplace_errors (exception)
     Implements the `xmlcharrefreplace' error handling (for encoding
     only): the unencodable character is replaced by an appropriate XML
     character reference.

 -- Function: codecs.backslashreplace_errors (exception)
     Implements the `backslashreplace' error handling (for encoding
     only): the unencodable character is replaced by a backslashed
     escape sequence.

  To simplify working with encoded files or stream, the module also
defines these utility functions:

 -- Function: codecs.open (filename, mode[, encoding[, errors[,
          buffering]]])
     Open an encoded file using the given _mode_ and return a wrapped
     version providing transparent encoding/decoding.  The default file
     mode is `'r'' meaning to open the file in read mode.

          Note: The wrapped version will only accept the object format
          defined by the codecs, i.e. Unicode objects for most built-in
          codecs.  Output is also codec-dependent and will usually be
          Unicode as well.

          Note: Files are always opened in binary mode, even if no
          binary mode was specified.  This is done to avoid data loss
          due to encodings using 8-bit values.  This means that no
          automatic conversion of `'\n'' is done on reading and writing.

     _encoding_ specifies the encoding which is to be used for the file.

     _errors_ may be given to define the error handling. It defaults to
     `'strict'' which causes a ValueError to be raised in
     case an encoding error occurs.

     _buffering_ has the same meaning as for the built-in open()
     function.  It defaults to line buffered.

 -- Function: codecs.EncodedFile (file, input[, output[, errors]])
     Return a wrapped version of file which provides transparent
     encoding translation.

     Strings written to the wrapped file are interpreted according to
     the given _input_ encoding and then written to the original file
     as strings using the _output_ encoding. The intermediate encoding
     will usually be Unicode but depends on the specified codecs.

     If _output_ is not given, it defaults to _input_.

     _errors_ may be given to define the error handling. It defaults to
     `'strict'', which causes ValueError to be raised in
     case an encoding error occurs.

 -- Function: codecs.iterencode (iterable, encoding[, errors])
     Uses an incremental encoder to iteratively encode the input
     provided by _iterable_. This function is a generator.
     _errors_ (as well as any other keyword argument) is passed through
     to the incremental encoder.

     New in version 2.5.

 -- Function: codecs.iterdecode (iterable, encoding[, errors])
     Uses an incremental decoder to iteratively decode the input
     provided by _iterable_. This function is a generator.
     _errors_ (as well as any other keyword argument) is passed through
     to the incremental decoder.

     New in version 2.5.

  The module also provides the following constants which are useful for
reading and writing to platform dependent files:

 -- Data: codecs.BOM
 -- Data: codecs.BOM_BE
 -- Data: codecs.BOM_LE
 -- Data: codecs.BOM_UTF8
 -- Data: codecs.BOM_UTF16
 -- Data: codecs.BOM_UTF16_BE
 -- Data: codecs.BOM_UTF16_LE
 -- Data: codecs.BOM_UTF32
 -- Data: codecs.BOM_UTF32_BE
 -- Data: codecs.BOM_UTF32_LE
     These constants define various encodings of the Unicode byte order
     mark (BOM) used in UTF-16 and UTF-32 data streams to indicate the
     byte order used in the stream or file and in UTF-8 as a Unicode
     signature. BOM_UTF16 is either BOM_UTF16_BE
     or BOM_UTF16_LE depending on the platform's
     native byte order, see BOM is an alias for
     BOM_UTF16, BOM_LE for BOM_UTF16_LE.
     and BOM_BE for BOM_UTF16_BE. The others
     represent the BOM in UTF-8 and UTF-32 encodings.

5.7.8.1 Codec Base Classes
..........................

The codecs module defines a set of base classes which define
the interface and can also be used to easily write your own codecs for
use in Python.

  Each codec has to define four interfaces to make it usable as codec
in Python: stateless encoder, stateless decoder, stream reader and
stream writer. The stream reader and writers typically reuse the
stateless encoder/decoder to implement the file protocols.

  The `Codec' class defines the interface for stateless
encoders/decoders.

  To simplify and standardize error handling, the `encode()' and
`decode()' methods may implement different error handling schemes by
providing the _errors_ string argument.  The following string values
are defined and implemented by all standard Python codecs:

Value                         Meaning
----------------------------------------------------------------------------------
`'strict''                    Raise UnicodeError. (or a subclass);
                              this is the default.
`'ignore''                    Ignore the character and continue with the next.
`'replace''                   Replace with a suitable replacement character;
                              Python will use the official U+FFFD REPLACEMENT
                              CHARACTER for the built-in Unicode codecs on
                              decoding and '?' on encoding.
`'xmlcharrefreplace''         Replace with the appropriate XML character
                              reference (only for encoding).
`'backslashreplace''          Replace with backslashed escape sequences (only
                              for encoding).

  The set of allowed values can be extended via register_error().

5.7.8.2 Codec Objects
.....................

The `Codec' class defines these methods which also define the function
interfaces of the stateless encoder and decoder:

 -- Method: Codec.encode (input[, errors])
     Encodes the object _input_ and returns a tuple (output object,
     length consumed).  While codecs are not restricted to use with
     Unicode, in a Unicode context, encoding converts a Unicode object
     to a plain string using a particular character set encoding (e.g.,
     `cp1252' or `iso-8859-1').

     _errors_ defines the error handling to apply. It defaults to
     `'strict'' handling.

     The method may not store state in the `Codec' instance. Use
     `StreamCodec' for codecs which have to keep state in order to make
     encoding/decoding efficient.

     The encoder must be able to handle zero length input and return an
     empty object of the output object type in this situation.

 -- Method: Codec.decode (input[, errors])
     Decodes the object _input_ and returns a tuple (output object,
     length consumed).  In a Unicode context, decoding converts a plain
     string encoded using a particular character set encoding to a
     Unicode object.

     _input_ must be an object which provides the `bf_getreadbuf'
     buffer slot.  Python strings, buffer objects and memory mapped
     files are examples of objects providing this slot.

     _errors_ defines the error handling to apply. It defaults to
     `'strict'' handling.

     The method may not store state in the `Codec' instance. Use
     `StreamCodec' for codecs which have to keep state in order to make
     encoding/decoding efficient.

     The decoder must be able to handle zero length input and return an
     empty object of the output object type in this situation.

  The IncrementalEncoder and IncrementalDecoder.
classes provide the basic interface for incremental encoding and
decoding. Encoding/decoding the input isn't done with one call to the
stateless encoder/decoder function, but with multiple calls to the
`encode()'/`decode()' method of the incremental encoder/decoder. The
incremental encoder/decoder keeps track of the encoding/decoding
process during method calls.

  The joined output of calls to the `encode()'/`decode()' method is the
same as if all the single inputs were joined into one, and this input
was encoded/decoded with the stateless encoder/decoder.

5.7.8.3 IncrementalEncoder Objects
..................................

New in version 2.5.

  The IncrementalEncoder class is used for encoding an
input in multiple steps. It defines the following methods which every
incremental encoder must define in order to be compatible with the
Python codec registry.

 -- Class: codecs.IncrementalEncoder ([errors])
     Constructor for an IncrementalEncoder instance.

     All incremental encoders must provide this constructor interface.
     They are free to add additional keyword arguments, but only the
     ones defined here are used by the Python codec registry.

     The IncrementalEncoder may implement different error
     handling schemes by providing the _errors_ keyword argument. These
     parameters are predefined:

        * `'strict'' Raise ValueError. (or a subclass); this
          is the default.

        * `'ignore'' Ignore the character and continue with the next.

        * `'replace'' Replace with a suitable replacement character

        * `'xmlcharrefreplace'' Replace with the appropriate XML
          character reference

        * `'backslashreplace'' Replace with backslashed escape
          sequences.

     The _errors_ argument will be assigned to an attribute of the same
     name.  Assigning to this attribute makes it possible to switch
     between different error handling strategies during the lifetime of
     the IncrementalEncoder.  object.

     The set of allowed values for the _errors_ argument can be
     extended with register_error().

      -- Method: encode (object[, final])
          Encodes _object_ (taking the current state of the encoder
          into account) and returns the resulting encoded object. If
          this is the last call to encode(). _final_ must be
          true (the default is false).

      -- Method: reset ()
          Reset the encoder to the initial state.

5.7.8.4 IncrementalDecoder Objects
..................................

The IncrementalDecoder class is used for decoding an input
in multiple steps. It defines the following methods which every
incremental decoder must define in order to be compatible with the
Python codec registry.

 -- Class: codecs.IncrementalDecoder ([errors])
     Constructor for an IncrementalDecoder instance.

     All incremental decoders must provide this constructor interface.
     They are free to add additional keyword arguments, but only the
     ones defined here are used by the Python codec registry.

     The IncrementalDecoder may implement different error
     handling schemes by providing the _errors_ keyword argument. These
     parameters are predefined:

        * `'strict'' Raise ValueError. (or a subclass); this
          is the default.

        * `'ignore'' Ignore the character and continue with the next.

        * `'replace'' Replace with a suitable replacement character.

     The _errors_ argument will be assigned to an attribute of the same
     name.  Assigning to this attribute makes it possible to switch
     between different error handling strategies during the lifetime of
     the IncrementalDecoder.  object.

     The set of allowed values for the _errors_ argument can be
     extended with register_error().

      -- Method: decode (object[, final])
          Decodes _object_ (taking the current state of the decoder
          into account) and returns the resulting decoded object. If
          this is the last call to decode(). _final_ must be
          true (the default is false). If _final_ is true the decoder
          must decode the input completely and must flush all buffers.
          If this isn't possible (e.g. because of incomplete byte
          sequences at the end of the input) it must initiate error
          handling just like in the stateless case (which might raise
          an exception).

      -- Method: reset ()
          Reset the decoder to the initial state.

  The StreamWriter and StreamReader classes
provide generic working interfaces which can be used to implement new
encoding submodules very easily. See `encodings.utf_8' for an example
of how this is done.

5.7.8.5 StreamWriter Objects
............................

The StreamWriter class is a subclass of `Codec' and defines
the following methods which every stream writer must define in order to
be compatible with the Python codec registry.

 -- Class: codecs.StreamWriter (stream[, errors])
     Constructor for a StreamWriter instance.

     All stream writers must provide this constructor interface. They
     are free to add additional keyword arguments, but only the ones
     defined here are used by the Python codec registry.

     _stream_ must be a file-like object open for writing binary data.

     The StreamWriter may implement different error
     handling schemes by providing the _errors_ keyword argument. These
     parameters are predefined:

        * `'strict'' Raise ValueError. (or a subclass); this
          is the default.

        * `'ignore'' Ignore the character and continue with the next.

        * `'replace'' Replace with a suitable replacement character

        * `'xmlcharrefreplace'' Replace with the appropriate XML
          character reference

        * `'backslashreplace'' Replace with backslashed escape
          sequences.

     The _errors_ argument will be assigned to an attribute of the same
     name.  Assigning to this attribute makes it possible to switch
     between different error handling strategies during the lifetime of
     the StreamWriter object.

     The set of allowed values for the _errors_ argument can be
     extended with register_error().

      -- Method: write (object)
          Writes the object's contents encoded to the stream.

      -- Method: writelines (list)
          Writes the concatenated list of strings to the stream
          (possibly by reusing the write() method).

      -- Method: reset ()
          Flushes and resets the codec buffers used for keeping state.

          Calling this method should ensure that the data on the output
          is put into a clean state that allows appending of new fresh
          data without having to rescan the whole stream to recover
          state.

  In addition to the above methods, the StreamWriter must
also inherit all other methods and attributes from the underlying
stream.

5.7.8.6 StreamReader Objects
............................

The StreamReader class is a subclass of `Codec' and defines
the following methods which every stream reader must define in order to
be compatible with the Python codec registry.

 -- Class: codecs.StreamReader (stream[, errors])
     Constructor for a StreamReader instance.

     All stream readers must provide this constructor interface. They
     are free to add additional keyword arguments, but only the ones
     defined here are used by the Python codec registry.

     _stream_ must be a file-like object open for reading (binary) data.

     The StreamReader may implement different error
     handling schemes by providing the _errors_ keyword argument. These
     parameters are defined:

        * `'strict'' Raise ValueError. (or a subclass); this
          is the default.

        * `'ignore'' Ignore the character and continue with the next.

        * `'replace'' Replace with a suitable replacement character.

     The _errors_ argument will be assigned to an attribute of the same
     name.  Assigning to this attribute makes it possible to switch
     between different error handling strategies during the lifetime of
     the StreamReader object.

     The set of allowed values for the _errors_ argument can be
     extended with register_error().

      -- Method: read ([size[, chars[, firstline]]])
          Decodes data from the stream and returns the resulting object.

          _chars_ indicates the number of characters to read from the
          stream. read() will never return more than
          _chars_ characters, but it might return less, if there are
          not enough characters available.

          _size_ indicates the approximate maximum number of bytes to
          read from the stream for decoding purposes. The decoder can
          modify this setting as appropriate. The default value -1
          indicates to read and decode as much as possible.  _size_ is
          intended to prevent having to decode huge files in one step.

          _firstline_ indicates that it would be sufficient to only
          return the first line, if there are decoding errors on later
          lines.

          The method should use a greedy read strategy meaning that it
          should read as much data as is allowed within the definition
          of the encoding and the given size, e.g.  if optional
          encoding endings or state markers are available on the
          stream, these should be read too.

          Changed in version 2.4: _chars_ argument added.

          Changed in version 2.4.2: _firstline_ argument added.

      -- Method: readline ([size[, keepends]])
          Read one line from the input stream and return the decoded
          data.

          _size_, if given, is passed as size argument to the stream's
          readline() method.

          If _keepends_ is false line-endings will be stripped from the
          lines returned.

          Changed in version 2.4: _keepends_ argument added.

      -- Method: readlines ([sizehint[, keepends]])
          Read all lines available on the input stream and return them
          as a list of lines.

          Line-endings are implemented using the codec's decoder method
          and are included in the list entries if _keepends_ is true.

          _sizehint_, if given, is passed as the _size_ argument to the
          stream's read() method.

      -- Method: reset ()
          Resets the codec buffers used for keeping state.

          Note that no stream repositioning should take place.  This
          method is primarily intended to be able to recover from
          decoding errors.

  In addition to the above methods, the StreamReader must
also inherit all other methods and attributes from the underlying
stream.

  The next two base classes are included for convenience. They are not
needed by the codec registry, but may provide useful in practice.

5.7.8.7 StreamReaderWriter Objects
..................................

The StreamReaderWriter allows wrapping streams which work
in both read and write modes.

  The design is such that one can use the factory functions returned by
the lookup() function to construct the instance.

 -- Class: codecs.StreamReaderWriter (stream, Reader, Writer, errors)
     Creates a StreamReaderWriter instance. _stream_ must
     be a file-like object. _Reader_ and _Writer_ must be factory
     functions or classes providing the StreamReader and
     StreamWriter interface resp. Error handling is done in
     the same way as defined for the stream readers and writers.

  StreamReaderWriter instances define the combined
interfaces of StreamReader and StreamWriter.
classes. They inherit all other methods and attributes from the
underlying stream.

5.7.8.8 StreamRecoder Objects
.............................

The StreamRecoder provide a frontend - backend view of
encoding data which is sometimes useful when dealing with different
encoding environments.

  The design is such that one can use the factory functions returned by
the lookup() function to construct the instance.

 -- Class: codecs.StreamRecoder (stream, encode, decode, Reader,
          Writer, errors)
     Creates a StreamRecoder instance which implements a
     two-way conversion: _encode_ and _decode_ work on the frontend
     (the input to `read()' and output of `write()') while _Reader_ and
     _Writer_ work on the backend (reading and writing to the stream).

     You can use these objects to do transparent direct recodings from
     e.g. Latin-1 to UTF-8 and back.

     _stream_ must be a file-like object.

     _encode_, _decode_ must adhere to the `Codec' interface. _Reader_,
     _Writer_ must be factory functions or classes providing objects of
     the StreamReader and StreamWriter.
     interface respectively.

     _encode_ and _decode_ are needed for the frontend translation,
     _Reader_ and _Writer_ for the backend translation.  The
     intermediate format used is determined by the two sets of codecs,
     e.g. the Unicode codecs will use Unicode as the intermediate
     encoding.

     Error handling is done in the same way as defined for the stream
     readers and writers.

  StreamRecoder instances define the combined interfaces of
StreamReader and StreamWriter classes. They
inherit all other methods and attributes from the underlying stream.

5.7.8.9 Encodings and Unicode
.............................

Unicode strings are stored internally as sequences of codepoints (to be
precise as Py_UNICODE arrays). Depending on the way Python
is compiled (either via `--enable-unicode=ucs2' or
`--enable-unicode=ucs4', with the former being the default)
Py_UNICODE is either a 16-bit or 32-bit data type. Once a Unicode
object is used outside of CPU and memory, CPU endianness and how these
arrays are stored as bytes become an issue.  Transforming a unicode
object into a sequence of bytes is called encoding and recreating the
unicode object from the sequence of bytes is known as decoding.  There
are many different methods for how this transformation can be done
(these methods are also called encodings). The simplest method is to
map the codepoints 0-255 to the bytes `0x0'-`0xff'. This means that a
unicode object that contains codepoints above `U+00FF' can't be encoded
with this method (which is called `'latin-1'' or `'iso-8859-1'').
`unicode.encode()' will raise a UnicodeEncodeError that
looks like this: `UnicodeEncodeError: 'latin-1' codec can't encode
character u'\u1234' in position 3: ordinal not in range(256)'.

  There's another group of encodings (the so called charmap encodings)
that choose a different subset of all unicode code points and how these
codepoints are mapped to the bytes `0x0'-`0xff'. To see how this is
done simply open e.g. `encodings/cp1252.py' (which is an encoding that
is used primarily on Windows). There's a string constant with 256
characters that shows you which character is mapped to which byte value.

  All of these encodings can only encode 256 of the 65536 (or 1114111)
codepoints defined in unicode. A simple and straightforward way that
can store each Unicode code point, is to store each codepoint as two
consecutive bytes. There are two possibilities: Store the bytes in big
endian or in little endian order. These two encodings are called
UTF-16-BE and UTF-16-LE respectively. Their disadvantage is that if
e.g. you use UTF-16-BE on a little endian machine you will always have
to swap bytes on encoding and decoding. UTF-16 avoids this problem:
Bytes will always be in natural endianness. When these bytes are read
by a CPU with a different endianness, then bytes have to be swapped
though. To be able to detect the endianness of a UTF-16 byte sequence,
there's the so called BOM (the "Byte Order Mark"). This is the Unicode
character `U+FEFF'.  This character will be prepended to every UTF-16
byte sequence. The byte swapped version of this character (`0xFFFE') is
an illegal character that may not appear in a Unicode text. So when the
first character in an UTF-16 byte sequence appears to be a `U+FFFE' the
bytes have to be swapped on decoding.  Unfortunately upto Unicode 4.0
the character `U+FEFF' had a second purpose as a `ZERO WIDTH NO-BREAK
SPACE': A character that has no width and doesn't allow a word to be
split. It can e.g. be used to give hints to a ligature algorithm.  With
Unicode 4.0 using `U+FEFF' as a `ZERO WIDTH NO-BREAK SPACE' has been
deprecated (with `U+2060' (`WORD JOINER') assuming this role).
Nevertheless Unicode software still must be able to handle `U+FEFF' in
both roles: As a BOM it's a device to determine the storage layout of
the encoded bytes, and vanishes once the byte sequence has been decoded
into a Unicode string; as a `ZERO WIDTH NO-BREAK SPACE' it's a normal
character that will be decoded like any other.

  There's another encoding that is able to encoding the full range of
Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means
there are no issues with byte order in UTF-8. Each byte in a UTF-8 byte
sequence consists of two parts: Marker bits (the most significant bits)
and payload bits. The marker bits are a sequence of zero to six 1 bits
followed by a 0 bit. Unicode characters are encoded like this (with x
being payload bits, which when concatenated give the Unicode character):

Range                                   Encoding
-------------------------------------------------------------------------------------------
`U-00000000' ... `U-0000007F'           0xxxxxxx
`U-00000080' ... `U-000007FF'           110xxxxx 10xxxxxx
`U-00000800' ... `U-0000FFFF'           1110xxxx 10xxxxxx 10xxxxxx
`U-00010000' ... `U-001FFFFF'           11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
`U-00200000' ... `U-03FFFFFF'           111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
`U-04000000' ... `U-7FFFFFFF'           1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
                                        10xxxxxx

  The least significant bit of the Unicode character is the rightmost x
bit.

  As UTF-8 is an 8-bit encoding no BOM is required and any `U+FEFF'
character in the decoded Unicode string (even if it's the first
character) is treated as a `ZERO WIDTH NO-BREAK SPACE'.

  Without external information it's impossible to reliably determine
which encoding was used for encoding a Unicode string. Each charmap
encoding can decode any random byte sequence. However that's not
possible with UTF-8, as UTF-8 byte sequences have a structure that
doesn't allow arbitrary byte sequences. To increase the reliability
with which a UTF-8 encoding can be detected, Microsoft invented a
variant of UTF-8 (that Python 2.5 calls `"utf-8-sig"') for its Notepad
program: Before any of the Unicode characters is written to the file, a
UTF-8 encoded BOM (which looks like this as a byte sequence: `0xef',
`0xbb', `0xbf') is written. As it's rather improbable that any charmap
encoded file starts with these byte values (which would e.g.  map to

         LATIN SMALL LETTER I WITH DIAERESIS
         RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
         INVERTED QUESTION MARK

  in iso-8859-1), this increases the probability that a utf-8-sig
encoding can be correctly guessed from the byte sequence. So here the
BOM is not used to be able to determine the byte order used for
generating the byte sequence, but as a signature that helps in guessing
the encoding. On encoding the utf-8-sig codec will write `0xef',
`0xbb', `0xbf' as the first three bytes to the file. On decoding
utf-8-sig will skip those three bytes if they appear as the first three
bytes in the file.

5.7.8.10 Standard Encodings
...........................

Python comes with a number of codecs built-in, either implemented as C
functions or with dictionaries as mapping tables. The following table
lists the codecs by name, together with a few common aliases, and the
languages for which the encoding is likely used. Neither the list of
aliases nor the list of languages is meant to be exhaustive. Notice
that spelling alternatives that only differ in case or use a hyphen
instead of an underscore are also valid aliases; therefore, e.g.
`'utf-8'' is a valid alias for the `'utf_8'' codec.

  Many of the character sets support the same languages. They vary in
individual characters (e.g. whether the EURO SIGN is supported or not),
and in the assignment of characters to code positions. For the European
languages in particular, the following variants typically exist:

   * an ISO 8859 codeset

   * a Microsoft Windows code page, which is typically derived from a
     8859 codeset, but replaces control characters with additional
     graphic characters

   * an IBM EBCDIC code page

   * an IBM PC code page, which is ASCII compatible

Codec                 Aliases                              Languages
------------------------------------------------------------------------------------------------
ascii                 646, us-ascii                        English
big5                  big5-tw, csbig5                      Traditional Chinese
big5hkscs             big5-hkscs, hkscs                    Traditional Chinese
cp037                 IBM037, IBM039                       English
cp424                 EBCDIC-CP-HE, IBM424                 Hebrew
cp437                 437, IBM437                          English
cp500                 EBCDIC-CP-BE, EBCDIC-CP-CH, IBM500   Western Europe
cp720                                                      Arabic
cp737                                                      Greek
cp775                 IBM775                               Baltic languages
cp850                 850, IBM850                          Western Europe
cp852                 852, IBM852                          Central and Eastern Europe
cp855                 855, IBM855                          Bulgarian, Byelorussian,
                                                           Macedonian, Russian, Serbian
cp856                                                      Hebrew
cp857                 857, IBM857                          Turkish
cp858                 858, IBM858                          Western Europe
cp860                 860, IBM860                          Portuguese
cp861                 861, CP-IS, IBM861                   Icelandic
cp862                 862, IBM862                          Hebrew
cp863                 863, IBM863                          Canadian
cp864                 IBM864                               Arabic
cp865                 865, IBM865                          Danish, Norwegian
cp866                 866, IBM866                          Russian
cp869                 869, CP-GR, IBM869                   Greek
cp874                                                      Thai
cp875                                                      Greek
cp932                 932, ms932, mskanji, ms-kanji        Japanese
cp949                 949, ms949, uhc                      Korean
cp950                 950, ms950                           Traditional Chinese
cp1006                                                     Urdu
cp1026                ibm1026                              Turkish
cp1140                ibm1140                              Western Europe
cp1250                windows-1250                         Central and Eastern Europe
cp1251                windows-1251                         Bulgarian, Byelorussian,
                                                           Macedonian, Russian, Serbian
cp1252                windows-1252                         Western Europe
cp1253                windows-1253                         Greek
cp1254                windows-1254                         Turkish
cp1255                windows-1255                         Hebrew
cp1256                windows-1256                         Arabic
cp1257                windows-1257                         Baltic languages
cp1258                windows-1258                         Vietnamese
euc_jp                eucjp, ujis, u-jis                   Japanese
euc_jis_2004          jisx0213, eucjis2004                 Japanese
euc_jisx0213          eucjisx0213                          Japanese
euc_kr                euckr, korean, ksc5601, ks_c-5601,   Korean
                      ks_c-5601-1987, ksx1001, ks_x-1001
gb2312                chinese, csiso58gb231280, euc- cn,   Simplified Chinese
                      euccn, eucgb2312-cn, gb2312-1980,
                      gb2312-80, iso- ir-58
gbk                   936, cp936, ms936                    Unified Chinese
gb18030               gb18030-2000                         Unified Chinese
hz                    hzgb, hz-gb, hz-gb-2312              Simplified Chinese
iso2022_jp            csiso2022jp, iso2022jp, iso-2022-jp  Japanese
iso2022_jp_1          iso2022jp-1, iso-2022-jp-1           Japanese
iso2022_jp_2          iso2022jp-2, iso-2022-jp-2           Japanese, Korean, Simplified
                                                           Chinese, Western Europe, Greek
iso2022_jp_2004       iso2022jp-2004, iso-2022-jp-2004     Japanese
iso2022_jp_3          iso2022jp-3, iso-2022-jp-3           Japanese
iso2022_jp_ext        iso2022jp-ext, iso-2022-jp-ext       Japanese
iso2022_kr            csiso2022kr, iso2022kr, iso-2022-kr  Korean
latin_1               iso-8859-1, iso8859-1, 8859, cp819,  West Europe
                      latin, latin1, L1
iso8859_2             iso-8859-2, latin2, L2               Central and Eastern Europe
iso8859_3             iso-8859-3, latin3, L3               Esperanto, Maltese
iso8859_4             iso-8859-4, latin4, L4               Baltic languages
iso8859_5             iso-8859-5, cyrillic                 Bulgarian, Byelorussian,
                                                           Macedonian, Russian, Serbian
iso8859_6             iso-8859-6, arabic                   Arabic
iso8859_7             iso-8859-7, greek, greek8            Greek
iso8859_8             iso-8859-8, hebrew                   Hebrew
iso8859_9             iso-8859-9, latin5, L5               Turkish
iso8859_10            iso-8859-10, latin6, L6              Nordic languages
iso8859_13            iso-8859-13, latin7, L7              Baltic languages
iso8859_14            iso-8859-14, latin8, L8              Celtic languages
iso8859_15            iso-8859-15, latin9, L9              Western Europe
iso8859_16            iso-8859-16, latin10, L10            South-Eastern Europe
johab                 cp1361, ms1361                       Korean
koi8_r                                                     Russian
koi8_u                                                     Ukrainian
mac_cyrillic          maccyrillic                          Bulgarian, Byelorussian,
                                                           Macedonian, Russian, Serbian
mac_greek             macgreek                             Greek
mac_iceland           maciceland                           Icelandic
mac_latin2            maclatin2, maccentraleurope          Central and Eastern Europe
mac_roman             macroman                             Western Europe
mac_turkish           macturkish                           Turkish
ptcp154               csptcp154, pt154, cp154,             Kazakh
                      cyrillic-asian
shift_jis             csshiftjis, shiftjis, sjis, s_jis    Japanese
shift_jis_2004        shiftjis2004, sjis_2004, sjis2004    Japanese
shift_jisx0213        shiftjisx0213, sjisx0213, s_jisx0213 Japanese
utf_32                U32, utf32                           all languages
utf_32_be             UTF-32BE                             all languages
utf_32_le             UTF-32LE                             all languages
utf_16                U16, utf16                           all languages
utf_16_be             UTF-16BE                             all languages (BMP only)
utf_16_le             UTF-16LE                             all languages (BMP only)
utf_7                 U7, unicode-1-1-utf-7                all languages
utf_8                 U8, UTF, utf8                        all languages
utf_8_sig                                                  all languages

  A number of codecs are specific to Python, so their codec names have
no meaning outside Python. Some of them don't convert from Unicode
strings to byte strings, but instead use the property of the Python
codecs machinery that any bijective function with one argument can be
considered as an encoding.

  For the codecs listed below, the result in the "encoding" direction
is always a byte string. The result of the "decoding" direction is
listed as operand type in the table.

Codec                    Aliases                         Operand type         Purpose
--------------------------------------------------------------------------------------------------------------
base64_codec             base64, base-64                 byte string          Convert operand to MIME base64
bz2_codec                bz2                             byte string          Compress the operand using bz2
hex_codec                hex                             byte string          Convert operand to hexadecimal
                                                                              representation, with two
                                                                              digits per byte
idna                                                     Unicode string       Implements RFC 3490
                                                                              (http://tools.ietf.org/html/rfc3490.html),
                                                                              see also encodings.idna.
mbcs                     dbcs                            Unicode string       Windows only: Encode operand
                                                                              according to the ANSI codepage
                                                                              (CP_ACP)
palmos                                                   Unicode string       Encoding of PalmOS 3.5
punycode                                                 Unicode string       Implements RFC 3492
                                                                              (http://tools.ietf.org/html/rfc3492.html)
quopri_codec             quopri, quoted-printable,       byte string          Convert operand to MIME quoted
                         quotedprintable                                      printable
raw_unicode_escape                                       Unicode string       Produce a string that is
                                                                              suitable as raw Unicode
                                                                              literal in Python source code
rot_13                   rot13                           Unicode string       Returns the Caesar-cypher
                                                                              encryption of the operand
string_escape                                            byte string          Produce a string that is
                                                                              suitable as string literal in
                                                                              Python source code
undefined                                                any                  Raise an exception for all
                                                                              conversions. Can be used as
                                                                              the system encoding if no
                                                                              automatic coercion.
                                                                              between byte and Unicode
                                                                              strings is desired.
unicode_escape                                           Unicode string       Produce a string that is
                                                                              suitable as Unicode literal in
                                                                              Python source code
unicode_internal                                         Unicode string       Return the internal
                                                                              representation of the operand
uu_codec                 uu                              byte string          Convert the operand using
                                                                              uuencode
zlib_codec               zip, zlib                       byte string          Compress the operand using gzip

  New in version 2.3: The `idna' and `punycode' encodings.

5.7.8.11 `encodings.idna' -- Internationalized Domain Names in Applications
...........................................................................

New in version 2.3.

  This module implements RFC 3490
(http://tools.ietf.org/html/rfc3490.html) (Internationalized Domain
Names in Applications) and RFC 3492
(http://tools.ietf.org/html/rfc3492.html) (Nameprep: A Stringprep
Profile for Internationalized Domain Names (IDN)). It builds upon the
`punycode' encoding and stringprep.

  These RFCs together define a protocol to support non-ASCII characters
in domain names. A domain name containing non-ASCII characters (such as
`www.Alliancefrançaise.nu') is converted into an ASCII-compatible
encoding (ACE, such as `www.xn--alliancefranaise-npb.nu'). The ACE form
of the domain name is then used in all places where arbitrary
characters are not allowed by the protocol, such as DNS queries, HTTP
`Host' fields, and so on. This conversion is carried out in the
application; if possible invisible to the user: The application should
transparently convert Unicode domain labels to IDNA on the wire, and
convert back ACE labels to Unicode before presenting them to the user.

  Python supports this conversion in several ways: The `idna' codec
allows to convert between Unicode and the ACE. Furthermore, the
socket module transparently converts Unicode host names to ACE,
so that applications need not be concerned about converting host names
themselves when they pass them to the socket module. On top of that,
modules that have host names as function parameters, such as
httplib and ftplib, accept Unicode host names (see
httplib then also transparently sends an IDNA hostname in the
`Host' field if it sends that field at all).

  When receiving host names from the wire (such as in reverse name
lookup), no automatic conversion to Unicode is performed: Applications
wishing to present such host names to the user should decode them to
Unicode.

  The module encodings.idna also implements the nameprep
procedure, which performs certain normalizations on host names, to
achieve case-insensitivity of international domain names, and to unify
similar characters. The nameprep functions can be used directly if
desired.

 -- Function: encodings.idna.nameprep (label)
     Return the nameprepped version of _label_. The implementation
     currently assumes query strings, so `AllowUnassigned' is true.

 -- Function: encodings.idna.ToASCII (label)
     Convert a label to ASCII, as specified in RFC 3490
     (http://tools.ietf.org/html/rfc3490.html). `UseSTD3ASCIIRules' is
     assumed to be false.

 -- Function: encodings.idna.ToUnicode (label)
     Convert a label to Unicode, as specified in RFC 3490
     (http://tools.ietf.org/html/rfc3490.html).

5.7.8.12 `encodings.utf_8_sig' -- UTF-8 codec with BOM signature
................................................................

New in version 2.5.

  This module implements a variant of the UTF-8 codec: On encoding a
UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the
stateful encoder this is only done once (on the first write to the byte
stream).  For decoding an optional UTF-8 encoded BOM at the start of
the data will be skipped.

5.7.9 `unicodedata' -- Unicode Database
---------------------------------------

This module provides access to the Unicode Character Database which
defines character properties for all Unicode characters. The data in
this database is based on the `UnicodeData.txt' file version 5.2.0
which is publicly available from <ftp://ftp.unicode.org/>.

  The module uses the same names and symbols as defined by the
UnicodeData File Format 5.2.0 (see
<http://www.unicode.org/reports/tr44/tr44-4.html>).  It defines the
following functions:

 -- Function: unicodedata.lookup (name)
     Look up character by name.  If a character with the given name is
     found, return the corresponding Unicode character.  If not found,
     see KeyError is raised.

 -- Function: unicodedata.name (unichr[, default])
     Returns the name assigned to the Unicode character _unichr_ as a
     string. If no name is defined, _default_ is returned, or, if not
     given, see ValueError is raised.

 -- Function: unicodedata.decimal (unichr[, default])
     Returns the decimal value assigned to the Unicode character
     _unichr_ as integer.  If no such value is defined, _default_ is
     returned, or, if not given, see ValueError is raised.

 -- Function: unicodedata.digit (unichr[, default])
     Returns the digit value assigned to the Unicode character _unichr_
     as integer.  If no such value is defined, _default_ is returned,
     or, if not given, see ValueError is raised.

 -- Function: unicodedata.numeric (unichr[, default])
     Returns the numeric value assigned to the Unicode character
     _unichr_ as float.  If no such value is defined, _default_ is
     returned, or, if not given, see ValueError is raised.

 -- Function: unicodedata.category (unichr)
     Returns the general category assigned to the Unicode character
     _unichr_ as string.

 -- Function: unicodedata.bidirectional (unichr)
     Returns the bidirectional category assigned to the Unicode
     character _unichr_ as string. If no such value is defined, an
     empty string is returned.

 -- Function: unicodedata.combining (unichr)
     Returns the canonical combining class assigned to the Unicode
     character _unichr_ as integer. Returns `0' if no combining class
     is defined.

 -- Function: unicodedata.east_asian_width (unichr)
     Returns the east asian width assigned to the Unicode character
     _unichr_ as string.

     New in version 2.4.

 -- Function: unicodedata.mirrored (unichr)
     Returns the mirrored property assigned to the Unicode character
     _unichr_ as integer. Returns `1' if the character has been
     identified as a "mirrored" character in bidirectional text, `0'
     otherwise.

 -- Function: unicodedata.decomposition (unichr)
     Returns the character decomposition mapping assigned to the
     Unicode character _unichr_ as string. An empty string is returned
     in case no such mapping is defined.

 -- Function: unicodedata.normalize (form, unistr)
     Return the normal form _form_ for the Unicode string _unistr_.
     Valid values for _form_ are 'NFC', 'NFKC', 'NFD', and 'NFKD'.

     The Unicode standard defines various normalization forms of a
     Unicode string, based on the definition of canonical equivalence
     and compatibility equivalence.  In Unicode, several characters can
     be expressed in various way. For example, the character U+00C7
     (LATIN CAPITAL LETTER C WITH CEDILLA) can also be expressed as the
     sequence U+0327 (COMBINING CEDILLA) U+0043 (LATIN CAPITAL LETTER
     C).

     For each character, there are two normal forms: normal form C and
     normal form D.  Normal form D (NFD) is also known as canonical
     decomposition, and translates each character into its decomposed
     form. Normal form C (NFC) first applies a canonical decomposition,
     then composes pre-combined characters again.

     In addition to these two forms, there are two additional normal
     forms based on compatibility equivalence. In Unicode, certain
     characters are supported which normally would be unified with
     other characters. For example, U+2160 (ROMAN NUMERAL ONE) is
     really the same thing as U+0049 (LATIN CAPITAL LETTER I).
     However, it is supported in Unicode for compatibility with
     existing character sets (e.g. gb2312).

     The normal form KD (NFKD) will apply the compatibility
     decomposition, i.e.  replace all compatibility characters with
     their equivalents. The normal form KC (NFKC) first applies the
     compatibility decomposition, followed by the canonical composition.

     Even if two unicode strings are normalized and look the same to a
     human reader, if one has combining characters and the other
     doesn't, they may not compare equal.

     New in version 2.3.

  In addition, the module exposes the following constant:

 -- Data: unicodedata.unidata_version
     The version of the Unicode database used in this module.

     New in version 2.3.

 -- Data: unicodedata.ucd_3_2_0
     This is an object that has the same methods as the entire module,
     but uses the Unicode database version 3.2 instead, for
     applications that require this specific version of the Unicode
     database (such as IDNA).

     New in version 2.5.

  Examples:

    >>> import unicodedata
    >>> unicodedata.lookup('LEFT CURLY BRACKET')
    u'{'
    >>> unicodedata.name(u'/')
    'SOLIDUS'
    >>> unicodedata.decimal(u'9')
    9
    >>> unicodedata.decimal(u'a')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: not a decimal
    >>> unicodedata.category(u'A')  # 'L'etter, 'u'ppercase
    'Lu'
    >>> unicodedata.bidirectional(u'\u0660') # 'A'rabic, 'N'umber
    'AN'

5.7.10 `stringprep' -- Internet String Preparation
--------------------------------------------------

New in version 2.3.

  When identifying things (such as host names) in the internet, it is
often necessary to compare such identifications for "equality". Exactly
how this comparison is executed may depend on the application domain,
e.g. whether it should be case-insensitive or not. It may be also
necessary to restrict the possible identifications, to allow only
identifications consisting of "printable" characters.

  RFC 3454 (http://tools.ietf.org/html/rfc3454.html) defines a
procedure for "preparing" Unicode strings in internet protocols. Before
passing strings onto the wire, they are processed with the preparation
procedure, after which they have a certain normalized form. The RFC
defines a set of tables, which can be combined into profiles. Each
profile must define which tables it uses, and what other optional parts
of the `stringprep' procedure are part of the profile. One example of a
`stringprep' profile is `nameprep', which is used for internationalized
domain names.

  The module stringprep only exposes the tables from RFC
3454. As these tables would be very large to represent them as
dictionaries or lists, the module uses the Unicode character database
internally. The module source code itself was generated using the
`mkstringprep.py' utility.

  As a result, these tables are exposed as functions, not as data
structures.  There are two kinds of tables in the RFC: sets and
mappings. For a set, see stringprep provides the
"characteristic function", i.e. a function that returns true if the
parameter is part of the set. For mappings, it provides the mapping
function: given the key, it returns the associated value. Below is a
list of all functions available in the module.

 -- Function: stringprep.in_table_a1 (code)
     Determine whether _code_ is in tableA.1 (Unassigned code points in
     Unicode 3.2).

 -- Function: stringprep.in_table_b1 (code)
     Determine whether _code_ is in tableB.1 (Commonly mapped to
     nothing).

 -- Function: stringprep.map_table_b2 (code)
     Return the mapped value for _code_ according to tableB.2 (Mapping
     for case-folding used with NFKC).

 -- Function: stringprep.map_table_b3 (code)
     Return the mapped value for _code_ according to tableB.3 (Mapping
     for case-folding used with no normalization).

 -- Function: stringprep.in_table_c11 (code)
     Determine whether _code_ is in tableC.1.1  (ASCII space
     characters).

 -- Function: stringprep.in_table_c12 (code)
     Determine whether _code_ is in tableC.1.2  (Non-ASCII space
     characters).

 -- Function: stringprep.in_table_c11_c12 (code)
     Determine whether _code_ is in tableC.1  (Space characters, union
     of C.1.1 and C.1.2).

 -- Function: stringprep.in_table_c21 (code)
     Determine whether _code_ is in tableC.2.1  (ASCII control
     characters).

 -- Function: stringprep.in_table_c22 (code)
     Determine whether _code_ is in tableC.2.2  (Non-ASCII control
     characters).

 -- Function: stringprep.in_table_c21_c22 (code)
     Determine whether _code_ is in tableC.2  (Control characters,
     union of C.2.1 and C.2.2).

 -- Function: stringprep.in_table_c3 (code)
     Determine whether _code_ is in tableC.3  (Private use).

 -- Function: stringprep.in_table_c4 (code)
     Determine whether _code_ is in tableC.4  (Non-character code
     points).

 -- Function: stringprep.in_table_c5 (code)
     Determine whether _code_ is in tableC.5  (Surrogate codes).

 -- Function: stringprep.in_table_c6 (code)
     Determine whether _code_ is in tableC.6  (Inappropriate for plain
     text).

 -- Function: stringprep.in_table_c7 (code)
     Determine whether _code_ is in tableC.7  (Inappropriate for
     canonical representation).

 -- Function: stringprep.in_table_c8 (code)
     Determine whether _code_ is in tableC.8  (Change display
     properties or are deprecated).

 -- Function: stringprep.in_table_c9 (code)
     Determine whether _code_ is in tableC.9  (Tagging characters).

 -- Function: stringprep.in_table_d1 (code)
     Determine whether _code_ is in tableD.1  (Characters with
     bidirectional property "R" or "AL").

 -- Function: stringprep.in_table_d2 (code)
     Determine whether _code_ is in tableD.2  (Characters with
     bidirectional property "L").

5.7.11 `fpformat' -- Floating point conversions
-----------------------------------------------

Deprecated since version 2.6: The fpformat module has been
removed in Python 3.0.

  The fpformat module defines functions for dealing with
floating point numbers representations in 100% pure Python.

     Note: This module is unnecessary: everything here can be done
     using the `%' string interpolation operator described in the
     String Formatting Operations section.

  The fpformat module defines the following functions and an
exception:

 -- Function: fpformat.fix (x, digs)
     Format _x_ as `[-]ddd.ddd' with _digs_ digits after the point and
     at least one digit before. If `digs <= 0', the decimal point is
     suppressed.

     _x_ can be either a number or a string that looks like one. _digs_
     is an integer.

     Return value is a string.

 -- Function: fpformat.sci (x, digs)
     Format _x_ as `[-]d.dddE[+-]ddd' with _digs_ digits after the
     point and exactly one digit before. If `digs <= 0', one digit is
     kept and the point is suppressed.

     _x_ can be either a real number, or a string that looks like one.
     _digs_ is an integer.

     Return value is a string.

 -- Exception: fpformat.NotANumber
     Exception raised when a string passed to fix() or
     sci() as the _x_ parameter does not look like a
     number. This is a subclass of ValueError.  when the
     standard exceptions are strings.  The exception value is the
     improperly formatted string that caused the exception to be raised.

  Example:

    >>> import fpformat
    >>> fpformat.fix(1.23, 1)
    '1.2'

5.8 Data Types
==============

The modules described in this chapter provide a variety of specialized
data types such as dates and times, fixed-type arrays, heap queues,
synchronized queues, and sets.

  Python also provides some built-in data types, in particular, see
dict, list, set. (which along with
frozenset, replaces the deprecated sets module), and
tuple. The str class can be used to handle
binary data and 8-bit text, and the unicode class to handle
Unicode text.

  The following modules are documented in this chapter:

5.8.1 `datetime' -- Basic date and time types
---------------------------------------------

New in version 2.3.

  The datetime module supplies classes for manipulating
dates and times in both simple and complex ways.  While date and time
arithmetic is supported, the focus of the implementation is on
efficient member extraction for output formatting and manipulation. For
related functionality, see also the time and
calendar modules.

  There are two kinds of date and time objects: "naive" and "aware".
This distinction refers to whether the object has any notion of time
zone, daylight saving time, or other kind of algorithmic or political
time adjustment.  Whether a naive datetime object represents
Coordinated Universal Time (UTC), local time, or time in some other
timezone is purely up to the program, just like it's up to the program
whether a particular number represents metres, miles, or mass.  Naive
datetime objects are easy to understand and to work with, at
the cost of ignoring some aspects of reality.

  For applications requiring more, see datetime and time
objects have an optional time zone information member, see
tzinfo, that can contain an instance of a subclass of the abstract
tzinfo class.  These tzinfo objects capture
information about the offset from UTC time, the time zone name, and
whether Daylight Saving Time is in effect.  Note that no concrete
tzinfo classes are supplied by the datetime module.
Supporting timezones at whatever level of detail is required is up to
the application.  The rules for time adjustment across the world are
more political than rational, and there is no standard suitable for
every application.

  The datetime module exports the following constants:

 -- Data: datetime.MINYEAR
     The smallest year number allowed in a date or
     datetime object.  MINYEAR is `1'.

 -- Data: datetime.MAXYEAR
     The largest year number allowed in a date or
     datetime object.  MAXYEAR is `9999'.

Module calendar.
     General calendar related functions.

Module time.
     Time access and conversions.

5.8.1.1 Available Types
.......................

 -- Class: datetime.date
     An idealized naive date, assuming the current Gregorian calendar
     always was, and always will be, in effect. Attributes: `year',
     `month', and `day'.

 -- Class: datetime.time
     An idealized time, independent of any particular day, assuming
     that every day has exactly 24*60*60 seconds (there is no notion of
     "leap seconds" here).  Attributes: `hour', `minute', `second',
     `microsecond', and tzinfo.

 -- Class: datetime.datetime
     A combination of a date and a time. Attributes: `year', `month',
     `day', `hour', `minute', `second', `microsecond', and
     tzinfo.

 -- Class: datetime.timedelta
     A duration expressing the difference between two date,
     time, or datetime instances to microsecond
     resolution.

 -- Class: datetime.tzinfo
     An abstract base class for time zone information objects.  These
     are used by the datetime and time classes
     to provide a customizable notion of time adjustment (for example,
     to account for time zone and/or daylight saving time).

  Objects of these types are immutable.

  Objects of the date type are always naive.

  An object _d_ of type time or datetime may be
naive or aware.  _d_ is aware if `d.tzinfo' is not `None' and
`d.tzinfo.utcoffset(d)' does not return `None'.  If `d.tzinfo' is
`None', or if `d.tzinfo' is not `None' but `d.tzinfo.utcoffset(d)'
returns `None', _d_ is naive.

  The distinction between naive and aware doesn't apply to
timedelta.  objects.

  Subclass relationships:

    object
        timedelta
        tzinfo
        time
        date
            datetime

5.8.1.2 `timedelta' Objects
...........................

A timedelta object represents a duration, the difference
between two dates or times.

 -- Class: datetime.timedelta ([days[, seconds[, microseconds[,
          milliseconds[, minutes[, hours[, weeks]]]]]]])
     All arguments are optional and default to `0'.  Arguments may be
     ints, longs, or floats, and may be positive or negative.

     Only _days_, _seconds_ and _microseconds_ are stored internally.
     Arguments are converted to those units:

        * A millisecond is converted to 1000 microseconds.

        * A minute is converted to 60 seconds.

        * An hour is converted to 3600 seconds.

        * A week is converted to 7 days.

     and days, seconds and microseconds are then normalized so that the
     representation is unique, with

        * `0 <= microseconds < 1000000'

        * `0 <= seconds < 3600*24' (the number of seconds in one day)

        * `-999999999 <= days <= 999999999'

     If any argument is a float and there are fractional microseconds,
     the fractional microseconds left over from all arguments are
     combined and their sum is rounded to the nearest microsecond.  If
     no argument is a float, the conversion and normalization processes
     are exact (no information is lost).

     If the normalized value of days lies outside the indicated range,
     see OverflowError is raised.

     Note that normalization of negative values may be surprising at
     first. For example,

         >>> from datetime import timedelta
         >>> d = timedelta(microseconds=-1)
         >>> (d.days, d.seconds, d.microseconds)
         (-1, 86399, 999999)

  Class attributes are:

 -- Attribute: timedelta.min
     The most negative timedelta object,
     `timedelta(-999999999)'.

 -- Attribute: timedelta.max
     The most positive timedelta object,
     `timedelta(days=999999999, hours=23, minutes=59, seconds=59,
     microseconds=999999)'.

 -- Attribute: timedelta.resolution
     The smallest possible difference between non-equal
     timedelta objects, `timedelta(microseconds=1)'.

  Note that, because of normalization, `timedelta.max' >
`-timedelta.min'.  `-timedelta.max' is not representable as a
timedelta object.

  Instance attributes (read-only):

Attribute              Value
------------------------------------------------------------------------
`days'                 Between -999999999 and 999999999 inclusive
`seconds'              Between 0 and 86399 inclusive
`microseconds'         Between 0 and 999999 inclusive

  Supported operations:

Operation                            Result
-----------------------------------------------------------------------------------------
`t1 = t2 + t3'                       Sum of _t2_ and _t3_. Afterwards _t1_-_t2_ == _t3_
                                     and _t1_-_t3_ == _t2_ are true. (1)
`t1 = t2 - t3'                       Difference of _t2_ and _t3_. Afterwards _t1_ ==
                                     _t2_ - _t3_ and _t2_ == _t1_ + _t3_ are true. (1)
`t1 = t2 * i or t1 = i * t2'         Delta multiplied by an integer or long.
                                     Afterwards _t1_ // i == _t2_ is true, provided `i
                                     != 0'.
                                     In general, _t1_ * i == _t1_ * (i-1) + _t1_ is
                                     true. (1)
`t1 = t2 // i'                       The floor is computed and the remainder (if any)
                                     is thrown away. (3)
`+t1'                                Returns a timedelta object with the
                                     same value. (2)
`-t1'                                equivalent to timedelta.(-_t1.days_,
                                     -_t1.seconds_, -_t1.microseconds_), and to _t1_*
                                     -1. (1)(4)
`abs(t)'                             equivalent to +_t_ when `t.days >= 0', and to -_t_
                                     when `t.days < 0'. (2)
`str(t)'                             Returns a string in the form `[D day[s],
                                     ][H]H:MM:SS[.UUUUUU]', where D is negative for
                                     negative `t'. (5)
`repr(t)'                            Returns a string in the form
                                     `datetime.timedelta(D[, S[, U]])', where D is
                                     negative for negative `t'. (5)

  Notes:

  1. This is exact, but may overflow.

  2. This is exact, and cannot overflow.

  3. Division by 0 raises ZeroDivisionError.

  4. -_timedelta.max_ is not representable as a timedelta.
     object.

  5. String representations of timedelta objects are
     normalized similarly to their internal representation.  This leads
     to somewhat unusual results for negative timedeltas.  For example:

         >>> timedelta(hours=-5)
         datetime.timedelta(-1, 68400)
         >>> print(_)
         -1 day, 19:00:00

  In addition to the operations listed above timedelta.
objects support certain additions and subtractions with date
and datetime.  objects (see below).

  Comparisons of timedelta objects are supported with the
timedelta object representing the smaller duration
considered to be the smaller timedelta. In order to stop mixed-type
comparisons from falling back to the default comparison by object
address, when a timedelta object is compared to an object
of a different type, see TypeError is raised unless the
comparison is `==' or `!='.  The latter cases return False.
or True, respectively.

  timedelta objects are hashable. (usable as
dictionary keys), support efficient pickling, and in Boolean contexts,
a timedelta object is considered to be true if and only if
it isn't equal to `timedelta(0)'.

  Instance methods:

 -- Method: timedelta.total_seconds ()
     Return the total number of seconds contained in the duration.
     Equivalent to `(td.microseconds + (td.seconds + td.days * 24 *
     3600) * 10**6) / 10**6' computed with true division enabled.

     Note that for very large time intervals (greater than 270 years on
     most platforms) this method will lose microsecond accuracy.

     New in version 2.7.

  Example usage:

    >>> from datetime import timedelta
    >>> year = timedelta(days=365)
    >>> another_year = timedelta(weeks=40, days=84, hours=23,
    ...                          minutes=50, seconds=600)  # adds up to 365 days
    >>> year.total_seconds()
    31536000.0
    >>> year == another_year
    True
    >>> ten_years = 10 * year
    >>> ten_years, ten_years.days // 365
    (datetime.timedelta(3650), 10)
    >>> nine_years = ten_years - year
    >>> nine_years, nine_years.days // 365
    (datetime.timedelta(3285), 9)
    >>> three_years = nine_years // 3;
    >>> three_years, three_years.days // 365
    (datetime.timedelta(1095), 3)
    >>> abs(three_years - ten_years) == 2 * three_years + year
    True

5.8.1.3 `date' Objects
......................

A date object represents a date (year, month and day) in an
idealized calendar, the current Gregorian calendar indefinitely
extended in both directions.  January 1 of year 1 is called day number
1, January 2 of year 1 is called day number 2, and so on.  This matches
the definition of the "proleptic Gregorian" calendar in Dershowitz and
Reingold's book Calendrical Calculations, where it's the base calendar
for all computations.  See the book for algorithms for converting
between proleptic Gregorian ordinals and many other calendar systems.

 -- Class: datetime.date (year, month, day)
     All arguments are required.  Arguments may be ints or longs, in
     the following ranges:

        * `MINYEAR <= year <= MAXYEAR'

        * `1 <= month <= 12'

        * `1 <= day <= number of days in the given month and year'

     If an argument outside those ranges is given, see ValueError
     is raised.

  Other constructors, all class methods:

 -- Class Method: date.today ()
     Return the current local date.  This is equivalent to
     `date.fromtimestamp(time.time())'.

 -- Class Method: date.fromtimestamp (timestamp)
     Return the local date corresponding to the POSIX timestamp, such
     as is returned by time.time().  This may raise
     ValueError, if the timestamp is out of the range of values
     supported by the platform C `localtime()' function.  It's common
     for this to be restricted to years from 1970 through 2038.  Note
     that on non-POSIX systems that include leap seconds in their
     notion of a timestamp, leap seconds are ignored by
     fromtimestamp().

 -- Class Method: date.fromordinal (ordinal)
     Return the date corresponding to the proleptic Gregorian ordinal,
     where January 1 of year 1 has ordinal 1.  See ValueError.
     is raised unless `1 <= ordinal <= date.max.toordinal()'. For any
     date _d_, `date.fromordinal(d.toordinal()) == d'.

  Class attributes:

 -- Attribute: date.min
     The earliest representable date, `date(MINYEAR, 1, 1)'.

 -- Attribute: date.max
     The latest representable date, `date(MAXYEAR, 12, 31)'.

 -- Attribute: date.resolution
     The smallest possible difference between non-equal date objects,
     `timedelta(days=1)'.

  Instance attributes (read-only):

 -- Attribute: date.year
     Between MINYEAR and MAXYEAR inclusive.

 -- Attribute: date.month
     Between 1 and 12 inclusive.

 -- Attribute: date.day
     Between 1 and the number of days in the given month of the given
     year.

  Supported operations:

Operation                           Result
---------------------------------------------------------------------------------------
`date2 = date1 + timedelta'         _date2_ is `timedelta.days' days removed from
                                    _date1_.  (1)
`date2 = date1 - timedelta'         Computes _date2_ such that `date2 + timedelta ==
                                    date1'. (2)
`timedelta = date1 - date2'         (3)
`date1 < date2'                     _date1_ is considered less than _date2_ when
                                    _date1_ precedes _date2_ in time. (4)

  Notes:

  1. _date2_ is moved forward in time if `timedelta.days > 0', or
     backward if `timedelta.days < 0'.  Afterward `date2 - date1 ==
     timedelta.days'.  `timedelta.seconds' and `timedelta.microseconds'
     are ignored.  OverflowError is raised if `date2.year'
     would be smaller than MINYEAR or larger than
     MAXYEAR.

  2. This isn't quite equivalent to date1 + (-timedelta), because
     -timedelta in isolation can overflow in cases where date1 -
     timedelta does not.  `timedelta.seconds' and
     `timedelta.microseconds' are ignored.

  3. This is exact, and cannot overflow.  timedelta.seconds and
     timedelta.microseconds are 0, and date2 + timedelta == date1 after.

  4. In other words, `date1 < date2' if and only if `date1.toordinal() <
     date2.toordinal()'. In order to stop comparison from falling back
     to the default scheme of comparing object addresses, date
     comparison normally raises TypeError if the other
     comparand isn't also a date object.  However,
     `NotImplemented' is returned instead if the other comparand has a
     `timetuple()' attribute.  This hook gives other kinds of date
     objects a chance at implementing mixed-type comparison. If not,
     when a date.  object is compared to an object of a
     different type, see TypeError is raised unless the
     comparison is `==' or `!='.  The latter cases return False
     or True, respectively.

  Dates can be used as dictionary keys. In Boolean contexts, all
date.  objects are considered to be true.

  Instance methods:

 -- Method: date.replace (year, month, day)
     Return a date with the same value, except for those members given
     new values by whichever keyword arguments are specified.  For
     example, if `d == date(2002, 12, 31)', then `d.replace(day=26) ==
     date(2002, 12, 26)'.

 -- Method: date.timetuple ()
     Return a time.struct_time such as returned by
     time.localtime().  The hours, minutes and seconds are 0, and
     the DST flag is -1. `d.timetuple()' is equivalent to
     `time.struct_time((d.year, d.month, d.day, 0, 0, 0, d.weekday(),
     yday, -1))', where `yday = d.toordinal() - date(d.year, 1,
     1).toordinal() + 1' is the day number within the current year
     starting with `1' for January 1st.

 -- Method: date.toordinal ()
     Return the proleptic Gregorian ordinal of the date, where January
     1 of year 1 has ordinal 1.  For any date object _d_,
     `date.fromordinal(d.toordinal()) == d'.

 -- Method: date.weekday ()
     Return the day of the week as an integer, where Monday is 0 and
     Sunday is 6.  For example, `date(2002, 12, 4).weekday() == 2', a
     Wednesday. See also isoweekday().

 -- Method: date.isoweekday ()
     Return the day of the week as an integer, where Monday is 1 and
     Sunday is 7.  For example, `date(2002, 12, 4).isoweekday() == 3',
     a Wednesday. See also weekday(), isocalendar().

 -- Method: date.isocalendar ()
     Return a 3-tuple, (ISO year, ISO week number, ISO weekday).

     The ISO calendar is a widely used variant of the Gregorian
     calendar. See
     <http://www.phys.uu.nl/~vgent/calendar/isocalendar.htm> for a good
     explanation.

     The ISO year consists of 52 or 53 full weeks, and where a week
     starts on a Monday and ends on a Sunday.  The first week of an ISO
     year is the first (Gregorian) calendar week of a year containing a
     Thursday. This is called week number 1, and the ISO year of that
     Thursday is the same as its Gregorian year.

     For example, 2004 begins on a Thursday, so the first week of ISO
     year 2004 begins on Monday, 29 Dec 2003 and ends on Sunday, 4 Jan
     2004, so that `date(2003, 12, 29).isocalendar() == (2004, 1, 1)'
     and `date(2004, 1, 4).isocalendar() == (2004, 1, 7)'.

 -- Method: date.isoformat ()
     Return a string representing the date in ISO 8601 format,
     'YYYY-MM-DD'.  For example, `date(2002, 12, 4).isoformat() ==
     '2002-12-04''.

 -- Method: date.__str__ ()
     For a date _d_, `str(d)' is equivalent to `d.isoformat()'.

 -- Method: date.ctime ()
     Return a string representing the date, for example `date(2002, 12,
     4).ctime() == 'Wed Dec 4 00:00:00 2002''. `d.ctime()' is
     equivalent to `time.ctime(time.mktime(d.timetuple()))' on
     platforms where the native C `ctime()' function (which
     time.ctime() invokes, but which date.ctime().
     does not invoke) conforms to the C standard.

 -- Method: date.strftime (format)
     Return a string representing the date, controlled by an explicit
     format string.  Format codes referring to hours, minutes or
     seconds will see 0 values. See section strftime() and strptime() Behavior.

  Example of counting days to an event:

    >>> import time
    >>> from datetime import date
    >>> today = date.today()
    >>> today
    datetime.date(2007, 12, 5)
    >>> today == date.fromtimestamp(time.time())
    True
    >>> my_birthday = date(today.year, 6, 24)
    >>> if my_birthday < today:
    ...     my_birthday = my_birthday.replace(year=today.year + 1)
    >>> my_birthday
    datetime.date(2008, 6, 24)
    >>> time_to_birthday = abs(my_birthday - today)
    >>> time_to_birthday.days
    202

Example of working with date.:

    >>> from datetime import date
    >>> d = date.fromordinal(730920) # 730920th day after 1. 1. 0001
    >>> d
    datetime.date(2002, 3, 11)
    >>> t = d.timetuple()
    >>> for i in t:
    ...     print i
    2002                # year
    3                   # month
    11                  # day
    0
    0
    0
    0                   # weekday (0 = Monday)
    70                  # 70th day in the year
    -1
    >>> ic = d.isocalendar()
    >>> for i in ic:
    ...     print i
    2002                # ISO year
    11                  # ISO week number
    1                   # ISO day number ( 1 = Monday )
    >>> d.isoformat()
    '2002-03-11'
    >>> d.strftime("%d/%m/%y")
    '11/03/02'
    >>> d.strftime("%A %d. %B %Y")
    'Monday 11. March 2002'

5.8.1.4 `datetime' Objects
..........................

A datetime object is a single object containing all the
information from a date object and a time.
object.  Like a date.  object, see datetime assumes
the current Gregorian calendar extended in both directions; like a time
object, see datetime assumes there are exactly 3600*24 seconds
in every day.

  Constructor:

 -- Class: datetime.datetime (year, month, day[, hour[, minute[,
          second[, microsecond[, tzinfo]]]]])
     The year, month and day arguments are required.  _tzinfo_ may be
     `None', or an instance of a tzinfo subclass.  The
     remaining arguments may be ints or longs, in the following ranges:

        * `MINYEAR <= year <= MAXYEAR'

        * `1 <= month <= 12'

        * `1 <= day <= number of days in the given month and year'

        * `0 <= hour < 24'

        * `0 <= minute < 60'

        * `0 <= second < 60'

        * `0 <= microsecond < 1000000'

     If an argument outside those ranges is given, see ValueError
     is raised.

  Other constructors, all class methods:

 -- Class Method: datetime.today ()
     Return the current local datetime, with tzinfo. `None'.
     This is equivalent to `datetime.fromtimestamp(time.time())'. See
     also now(), fromtimestamp().

 -- Class Method: datetime.now ([tz])
     Return the current local date and time.  If optional argument _tz_
     is `None' or not specified, this is like today(), but,
     if possible, supplies more precision than can be gotten from going
     through a time.time() timestamp (for example, this may
     be possible on platforms supplying the C `gettimeofday()'
     function).

     Else _tz_ must be an instance of a class tzinfo.
     subclass, and the current date and time are converted to _tz_'s
     time zone.  In this case the result is equivalent to
     `tz.fromutc(datetime.utcnow().replace(tzinfo=tz))'.  See also
     today(), utcnow().

 -- Class Method: datetime.utcnow ()
     Return the current UTC date and time, with tzinfo.
     `None'. This is like now(), but returns the current UTC
     date and time, as a naive datetime object. See also
     now().

 -- Class Method: datetime.fromtimestamp (timestamp[, tz])
     Return the local date and time corresponding to the POSIX
     timestamp, such as is returned by time.time(). If
     optional argument _tz_ is `None' or not specified, the timestamp
     is converted to the platform's local date and time, and the
     returned datetime object is naive.

     Else _tz_ must be an instance of a class tzinfo.
     subclass, and the timestamp is converted to _tz_'s time zone.  In
     this case the result is equivalent to
     `tz.fromutc(datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz))'.

     fromtimestamp() may raise ValueError, if
     the timestamp is out of the range of values supported by the
     platform C `localtime()' or `gmtime()' functions.  It's common for
     this to be restricted to years in 1970 through 2038. Note that on
     non-POSIX systems that include leap seconds in their notion of a
     timestamp, leap seconds are ignored by fromtimestamp(),
     and then it's possible to have two timestamps differing by a
     second that yield identical datetime objects. See also
     utcfromtimestamp().

 -- Class Method: datetime.utcfromtimestamp (timestamp)
     Return the UTC datetime corresponding to the POSIX
     timestamp, with tzinfo. `None'. This may raise
     ValueError, if the timestamp is out of the range of values
     supported by the platform C `gmtime()' function.  It's common for
     this to be restricted to years in 1970 through 2038. See also
     fromtimestamp().

 -- Class Method: datetime.fromordinal (ordinal)
     Return the datetime corresponding to the proleptic
     Gregorian ordinal, where January 1 of year 1 has ordinal 1.
     ValueError is raised unless `1 <= ordinal <=
     datetime.max.toordinal()'.  The hour, minute, second and
     microsecond of the result are all 0, and tzinfo is
     `None'.

 -- Class Method: datetime.combine (date, time)
     Return a new datetime object whose date members are
     equal to the given date object's, and whose time and
     tzinfo members are equal to the given time.
     object's. For any datetime object _d_, `d ==
     datetime.combine(d.date(), d.timetz())'.  If date is a
     datetime.  object, its time and tzinfo members are
     ignored.

 -- Class Method: datetime.strptime (date_string, format)
     Return a datetime corresponding to _date_string_,
     parsed according to _format_.  This is equivalent to
     `datetime(*(time.strptime(date_string, format)[0:6]))'.
     ValueError is raised if the date_string and format can't be
     parsed by time.strptime() or if it returns a value
     which isn't a time tuple. See section strftime() and strptime() Behavior.

     New in version 2.5.

  Class attributes:

 -- Attribute: datetime.min
     The earliest representable datetime, `datetime(MINYEAR,
     1, 1, tzinfo=None)'.

 -- Attribute: datetime.max
     The latest representable datetime, `datetime(MAXYEAR,
     12, 31, 23, 59, 59, 999999, tzinfo=None)'.

 -- Attribute: datetime.resolution
     The smallest possible difference between non-equal datetime
     objects, `timedelta(microseconds=1)'.

  Instance attributes (read-only):

 -- Attribute: datetime.year
     Between MINYEAR and MAXYEAR inclusive.

 -- Attribute: datetime.month
     Between 1 and 12 inclusive.

 -- Attribute: datetime.day
     Between 1 and the number of days in the given month of the given
     year.

 -- Attribute: datetime.hour
     In `range(24)'.

 -- Attribute: datetime.minute
     In `range(60)'.

 -- Attribute: datetime.second
     In `range(60)'.

 -- Attribute: datetime.microsecond
     In `range(1000000)'.

 -- Attribute: datetime.tzinfo
     The object passed as the _tzinfo_ argument to the datetime
     constructor, or `None' if none was passed.

  Supported operations:

Operation                                   Result
--------------------------------------------------------------------------------
`datetime2 = datetime1 + timedelta'         (1)
`datetime2 = datetime1 - timedelta'         (2)
`timedelta = datetime1 - datetime2'         (3)
`datetime1 < datetime2'                     Compares datetime to
                                            datetime. (4)

  1. datetime2 is a duration of timedelta removed from datetime1,
     moving forward in time if `timedelta.days' > 0, or backward if
     `timedelta.days' < 0.  The result has the same tzinfo.
     member as the input datetime, and datetime2 - datetime1 ==
     timedelta after. OverflowError is raised if
     datetime2.year would be smaller than MINYEAR or larger
     than MAXYEAR. Note that no time zone adjustments are
     done even if the input is an aware object.

  2. Computes the datetime2 such that datetime2 + timedelta ==
     datetime1. As for addition, the result has the same tzinfo
     member as the input datetime, and no time zone adjustments
     are done even if the input is aware. This isn't quite equivalent
     to datetime1 + (-timedelta), because -timedelta in isolation can
     overflow in cases where datetime1 - timedelta does not.

  3. Subtraction of a datetime from a datetime is
     defined only if both operands are naive, or if both are aware.  If
     one is aware and the other is naive, see TypeError is
     raised.

     If both are naive, or both are aware and have the same
     tzinfo member, the tzinfo members are ignored,
     and the result is a timedelta.  object _t_ such that
     `datetime2 + t == datetime1'.  No time zone adjustments are done
     in this case.

     If both are aware and have different tzinfo members,
     `a-b' acts as if _a_ and _b_ were first converted to naive UTC
     datetimes first.  The result is `(a.replace(tzinfo=None) -
     a.utcoffset()) - (b.replace(tzinfo=None) - b.utcoffset())' except
     that the implementation never overflows.

  4. _datetime1_ is considered less than _datetime2_ when _datetime1_
     precedes _datetime2_ in time.

     If one comparand is naive and the other is aware, see TypeError
     is raised.  If both comparands are aware, and have the same
     tzinfo member, the common tzinfo member is
     ignored and the base datetimes are compared.  If both comparands
     are aware and have different tzinfo members, the
     comparands are first adjusted by subtracting their UTC offsets
     (obtained from `self.utcoffset()').

          Note: In order to stop comparison from falling back to the
          default scheme of comparing object addresses, datetime
          comparison normally raises TypeError if the other
          comparand isn't also a datetime object.  However,
          `NotImplemented' is returned instead if the other comparand
          has a `timetuple()' attribute.  This hook gives other kinds
          of date objects a chance at implementing mixed-type
          comparison.  If not, when a datetime.  object is
          compared to an object of a different type, see TypeError
          is raised unless the comparison is `==' or `!='.  The
          latter cases return False or True,
          respectively.

  datetime objects can be used as dictionary keys. In
Boolean contexts, all datetime objects are considered to be
true.

  Instance methods:

 -- Method: datetime.date ()
     Return date object with same year, month and day.

 -- Method: datetime.time ()
     Return time object with same hour, minute, second and
     microsecond.  tzinfo is `None'.  See also method
     timetz().

 -- Method: datetime.timetz ()
     Return time object with same hour, minute, second,
     microsecond, and tzinfo members.  See also method time().

 -- Method: datetime.replace ([year[, month[, day[, hour[, minute[,
          second[, microsecond[, tzinfo]]]]]]]])
     Return a datetime with the same members, except for those members
     given new values by whichever keyword arguments are specified.
     Note that `tzinfo=None' can be specified to create a naive
     datetime from an aware datetime with no conversion of date and
     time members.

 -- Method: datetime.astimezone (tz)
     Return a datetime object with new tzinfo.
     member _tz_, adjusting the date and time members so the result is
     the same UTC time as _self_, but in _tz_'s local time.

     _tz_ must be an instance of a tzinfo subclass, and its
     utcoffset() and dst() methods must not
     return `None'.  _self_ must be aware (`self.tzinfo' must not be
     `None', and `self.utcoffset()' must not return `None').

     If `self.tzinfo' is _tz_, `self.astimezone(tz)' is equal to
     _self_:  no adjustment of date or time members is performed. Else
     the result is local time in time zone _tz_, representing the same
     UTC time as _self_:  after `astz = dt.astimezone(tz)', `astz -
     astz.utcoffset()' will usually have the same date and time members
     as `dt - dt.utcoffset()'. The discussion of class tzinfo
     explains the cases at Daylight Saving Time transition
     boundaries where this cannot be achieved (an issue only if _tz_
     models both standard and daylight time).

     If you merely want to attach a time zone object _tz_ to a datetime
     _dt_ without adjustment of date and time members, use
     `dt.replace(tzinfo=tz)'.  If you merely want to remove the time
     zone object from an aware datetime _dt_ without conversion of date
     and time members, use `dt.replace(tzinfo=None)'.

     Note that the default tzinfo.fromutc() method can be
     overridden in a tzinfo subclass to affect the result
     returned by astimezone().  Ignoring error cases, see
     astimezone() acts like:

         def astimezone(self, tz):
             if self.tzinfo is tz:
                 return self
             # Convert self to UTC, and attach the new time zone object.
             utc = (self - self.utcoffset()).replace(tzinfo=tz)
             # Convert from UTC to tz's local time.
             return tz.fromutc(utc)

 -- Method: datetime.utcoffset ()
     If tzinfo is `None', returns `None', else returns
     `self.tzinfo.utcoffset(self)', and raises an exception if the
     latter doesn't return `None', or a timedelta object
     representing a whole number of minutes with magnitude less than
     one day.

 -- Method: datetime.dst ()
     If tzinfo is `None', returns `None', else returns
     `self.tzinfo.dst(self)', and raises an exception if the latter
     doesn't return `None', or a timedelta object
     representing a whole number of minutes with magnitude less than
     one day.

 -- Method: datetime.tzname ()
     If tzinfo is `None', returns `None', else returns
     `self.tzinfo.tzname(self)', raises an exception if the latter
     doesn't return `None' or a string object,

 -- Method: datetime.timetuple ()
     Return a time.struct_time such as returned by
     time.localtime().  `d.timetuple()' is equivalent to
     `time.struct_time((d.year, d.month, d.day, d.hour, d.minute,
     d.second, d.weekday(), yday, dst))', where `yday = d.toordinal() -
     date(d.year, 1, 1).toordinal() + 1' is the day number within the
     current year starting with `1' for January 1st. The `tm_isdst' flag
     of the result is set according to the dst() method:
     tzinfo is `None' or dst() returns `None',
     `tm_isdst' is set to `-1'; else if dst() returns a
     non-zero value, `tm_isdst' is set to `1'; else `tm_isdst' is set
     to `0'.

 -- Method: datetime.utctimetuple ()
     If datetime instance _d_ is naive, this is the same as
     `d.timetuple()' except that `tm_isdst' is forced to 0 regardless
     of what `d.dst()' returns.  DST is never in effect for a UTC time.

     If _d_ is aware, _d_ is normalized to UTC time, by subtracting
     `d.utcoffset()', and a time.struct_time for the
     normalized time is returned.  `tm_isdst' is forced to 0. Note that
     the result's `tm_year' member may be MINYEAR.-1 or
     MAXYEAR.+1, if _d_.year was `MINYEAR' or `MAXYEAR' and
     UTC adjustment spills over a year boundary.

 -- Method: datetime.toordinal ()
     Return the proleptic Gregorian ordinal of the date.  The same as
     `self.date().toordinal()'.

 -- Method: datetime.weekday ()
     Return the day of the week as an integer, where Monday is 0 and
     Sunday is 6.  The same as `self.date().weekday()'. See also
     isoweekday().

 -- Method: datetime.isoweekday ()
     Return the day of the week as an integer, where Monday is 1 and
     Sunday is 7.  The same as `self.date().isoweekday()'. See also
     weekday(), isocalendar().

 -- Method: datetime.isocalendar ()
     Return a 3-tuple, (ISO year, ISO week number, ISO weekday).  The
     same as `self.date().isocalendar()'.

 -- Method: datetime.isoformat ([sep])
     Return a string representing the date and time in ISO 8601 format,
     YYYY-MM-DDTHH:MM:SS.mmmmmm or, if microsecond is 0,
     YYYY-MM-DDTHH:MM:SS

     If utcoffset() does not return `None', a 6-character
     string is appended, giving the UTC offset in (signed) hours and
     minutes: YYYY-MM-DDTHH:MM:SS.mmmmmm+HH:MM or, if
     microsecond is 0 YYYY-MM-DDTHH:MM:SS+HH:MM

     The optional argument _sep_ (default `'T'') is a one-character
     separator, placed between the date and time portions of the
     result.  For example,

         >>> from datetime import tzinfo, timedelta, datetime
         >>> class TZ(tzinfo):
         ...     def utcoffset(self, dt): return timedelta(minutes=-399)
         ...
         >>> datetime(2002, 12, 25, tzinfo=TZ()).isoformat(' ')
         '2002-12-25 00:00:00-06:39'

 -- Method: datetime.__str__ ()
     For a datetime instance _d_, `str(d)' is equivalent to
     `d.isoformat(' ')'.

 -- Method: datetime.ctime ()
     Return a string representing the date and time, for example
     `datetime(2002, 12, 4, 20, 30, 40).ctime() == 'Wed Dec  4 20:30:40
     2002''. `d.ctime()' is equivalent to
     `time.ctime(time.mktime(d.timetuple()))' on platforms where the
     native C `ctime()' function (which time.ctime().
     invokes, but which datetime.ctime() does not invoke)
     conforms to the C standard.

 -- Method: datetime.strftime (format)
     Return a string representing the date and time, controlled by an
     explicit format string.  See section strftime() and strptime() Behavior.

  Examples of working with datetime objects:

    >>> from datetime import datetime, date, time
    >>> # Using datetime.combine()
    >>> d = date(2005, 7, 14)
    >>> t = time(12, 30)
    >>> datetime.combine(d, t)
    datetime.datetime(2005, 7, 14, 12, 30)
    >>> # Using datetime.now() or datetime.utcnow()
    >>> datetime.now()
    datetime.datetime(2007, 12, 6, 16, 29, 43, 79043)   # GMT +1
    >>> datetime.utcnow()
    datetime.datetime(2007, 12, 6, 15, 29, 43, 79060)
    >>> # Using datetime.strptime()
    >>> dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M")
    >>> dt
    datetime.datetime(2006, 11, 21, 16, 30)
    >>> # Using datetime.timetuple() to get tuple of all attributes
    >>> tt = dt.timetuple()
    >>> for it in tt:
    ...     print it
    ...
    2006    # year
    11      # month
    21      # day
    16      # hour
    30      # minute
    0       # second
    1       # weekday (0 = Monday)
    325     # number of days since 1st January
    -1      # dst - method tzinfo.dst() returned None
    >>> # Date in ISO format
    >>> ic = dt.isocalendar()
    >>> for it in ic:
    ...     print it
    ...
    2006    # ISO year
    47      # ISO week
    2       # ISO weekday
    >>> # Formatting datetime
    >>> dt.strftime("%A, %d. %B %Y %I:%M%p")
    'Tuesday, 21. November 2006 04:30PM'

Using datetime with tzinfo:

    >>> from datetime import timedelta, datetime, tzinfo
    >>> class GMT1(tzinfo):
    ...     def __init__(self):         # DST starts last Sunday in March
    ...         d = datetime(dt.year, 4, 1)   # ends last Sunday in October
    ...         self.dston = d - timedelta(days=d.weekday() + 1)
    ...         d = datetime(dt.year, 11, 1)
    ...         self.dstoff = d - timedelta(days=d.weekday() + 1)
    ...     def utcoffset(self, dt):
    ...         return timedelta(hours=1) + self.dst(dt)
    ...     def dst(self, dt):
    ...         if self.dston <=  dt.replace(tzinfo=None) < self.dstoff:
    ...             return timedelta(hours=1)
    ...         else:
    ...             return timedelta(0)
    ...     def tzname(self,dt):
    ...          return "GMT +1"
    ...
    >>> class GMT2(tzinfo):
    ...     def __init__(self):
    ...         d = datetime(dt.year, 4, 1)
    ...         self.dston = d - timedelta(days=d.weekday() + 1)
    ...         d = datetime(dt.year, 11, 1)
    ...         self.dstoff = d - timedelta(days=d.weekday() + 1)
    ...     def utcoffset(self, dt):
    ...         return timedelta(hours=1) + self.dst(dt)
    ...     def dst(self, dt):
    ...         if self.dston <=  dt.replace(tzinfo=None) < self.dstoff:
    ...             return timedelta(hours=2)
    ...         else:
    ...             return timedelta(0)
    ...     def tzname(self,dt):
    ...         return "GMT +2"
    ...
    >>> gmt1 = GMT1()
    >>> # Daylight Saving Time
    >>> dt1 = datetime(2006, 11, 21, 16, 30, tzinfo=gmt1)
    >>> dt1.dst()
    datetime.timedelta(0)
    >>> dt1.utcoffset()
    datetime.timedelta(0, 3600)
    >>> dt2 = datetime(2006, 6, 14, 13, 0, tzinfo=gmt1)
    >>> dt2.dst()
    datetime.timedelta(0, 3600)
    >>> dt2.utcoffset()
    datetime.timedelta(0, 7200)
    >>> # Convert datetime to another time zone
    >>> dt3 = dt2.astimezone(GMT2())
    >>> dt3     # doctest: +ELLIPSIS
    datetime.datetime(2006, 6, 14, 14, 0, tzinfo=<GMT2 object at 0x...>)
    >>> dt2     # doctest: +ELLIPSIS
    datetime.datetime(2006, 6, 14, 13, 0, tzinfo=<GMT1 object at 0x...>)
    >>> dt2.utctimetuple() == dt3.utctimetuple()
    True

5.8.1.5 `time' Objects
......................

A time object represents a (local) time of day, independent of any
particular day, and subject to adjustment via a tzinfo.
object.

 -- Class: datetime.time (hour[, minute[, second[, microsecond[,
          tzinfo]]]])
     All arguments are optional.  _tzinfo_ may be `None', or an
     instance of a tzinfo subclass.  The remaining
     arguments may be ints or longs, in the following ranges:

        * `0 <= hour < 24'

        * `0 <= minute < 60'

        * `0 <= second < 60'

        * `0 <= microsecond < 1000000'.

     If an argument outside those ranges is given, see ValueError
     is raised.  All default to `0' except _tzinfo_, which
     defaults to None.

  Class attributes:

 -- Attribute: time.min
     The earliest representable time, `time(0, 0, 0, 0)'.

 -- Attribute: time.max
     The latest representable time, `time(23, 59, 59,
     999999)'.

 -- Attribute: time.resolution
     The smallest possible difference between non-equal time
     objects, `timedelta(microseconds=1)', although note that
     arithmetic on time.  objects is not supported.

  Instance attributes (read-only):

 -- Attribute: time.hour
     In `range(24)'.

 -- Attribute: time.minute
     In `range(60)'.

 -- Attribute: time.second
     In `range(60)'.

 -- Attribute: time.microsecond
     In `range(1000000)'.

 -- Attribute: time.tzinfo
     The object passed as the tzinfo argument to the time.
     constructor, or `None' if none was passed.

  Supported operations:

   * comparison of time to time, where _a_ is
     considered less than _b_ when _a_ precedes _b_ in time.  If one
     comparand is naive and the other is aware, see TypeError.
     is raised.  If both comparands are aware, and have the same
     tzinfo member, the common tzinfo member is
     ignored and the base times are compared.  If both comparands are
     aware and have different tzinfo members, the
     comparands are first adjusted by subtracting their UTC offsets
     (obtained from `self.utcoffset()'). In order to stop mixed-type
     comparisons from falling back to the default comparison by object
     address, when a time object is compared to an object
     of a different type, see TypeError is raised unless the
     comparison is `==' or `!='.  The latter cases return False
     or True, respectively.

   * hash, use as dict key

   * efficient pickling

   * in Boolean contexts, a time object is considered to be
     true if and only if, after converting it to minutes and
     subtracting `utcoffset()' (or `0' if that's `None'), the result is
     non-zero.

  Instance methods:

 -- Method: time.replace ([hour[, minute[, second[, microsecond[,
          tzinfo]]]]])
     Return a time with the same value, except for those
     members given new values by whichever keyword arguments are
     specified.  Note that `tzinfo=None' can be specified to create a
     naive time from an aware time, without
     conversion of the time members.

 -- Method: time.isoformat ()
     Return a string representing the time in ISO 8601 format,
     HH:MM:SS.mmmmmm or, if self.microsecond is 0, HH:MM:SS If
     utcoffset() does not return `None', a 6-character string is
     appended, giving the UTC offset in (signed) hours and minutes:
     HH:MM:SS.mmmmmm+HH:MM or, if self.microsecond is 0, HH:MM:SS+HH:MM

 -- Method: time.__str__ ()
     For a time _t_, `str(t)' is equivalent to `t.isoformat()'.

 -- Method: time.strftime (format)
     Return a string representing the time, controlled by an explicit
     format string.  See section strftime() and strptime() Behavior.

 -- Method: time.utcoffset ()
     If tzinfo is `None', returns `None', else returns
     `self.tzinfo.utcoffset(None)', and raises an exception if the
     latter doesn't return `None' or a timedelta object
     representing a whole number of minutes with magnitude less than
     one day.

 -- Method: time.dst ()
     If tzinfo is `None', returns `None', else returns
     `self.tzinfo.dst(None)', and raises an exception if the latter
     doesn't return `None', or a timedelta object
     representing a whole number of minutes with magnitude less than
     one day.

 -- Method: time.tzname ()
     If tzinfo is `None', returns `None', else returns
     `self.tzinfo.tzname(None)', or raises an exception if the latter
     doesn't return `None' or a string object.

  Example:

    >>> from datetime import time, tzinfo
    >>> class GMT1(tzinfo):
    ...     def utcoffset(self, dt):
    ...         return timedelta(hours=1)
    ...     def dst(self, dt):
    ...         return timedelta(0)
    ...     def tzname(self,dt):
    ...         return "Europe/Prague"
    ...
    >>> t = time(12, 10, 30, tzinfo=GMT1())
    >>> t                               # doctest: +ELLIPSIS
    datetime.time(12, 10, 30, tzinfo=<GMT1 object at 0x...>)
    >>> gmt = GMT1()
    >>> t.isoformat()
    '12:10:30+01:00'
    >>> t.dst()
    datetime.timedelta(0)
    >>> t.tzname()
    'Europe/Prague'
    >>> t.strftime("%H:%M:%S %Z")
    '12:10:30 Europe/Prague'

5.8.1.6 `tzinfo' Objects
........................

tzinfo is an abstract base class, meaning that this class
should not be instantiated directly.  You need to derive a concrete
subclass, and (at least) supply implementations of the standard
tzinfo methods needed by the datetime methods you use.
The datetime module does not supply any concrete subclasses
of tzinfo.

  An instance of (a concrete subclass of) tzinfo can be
passed to the constructors for datetime and time.
objects. The latter objects view their members as being in local time,
and the tzinfo object supports methods revealing offset of
local time from UTC, the name of the time zone, and DST offset, all
relative to a date or time object passed to them.

  Special requirement for pickling:  A tzinfo subclass must
have an __init__() method that can be called with no
arguments, else it can be pickled but possibly not unpickled again.
This is a technical requirement that may be relaxed in the future.

  A concrete subclass of tzinfo may need to implement the
following methods.  Exactly which methods are needed depends on the
uses made of aware datetime objects.  If in doubt, simply
implement all of them.

 -- Method: tzinfo.utcoffset (self, dt)
     Return offset of local time from UTC, in minutes east of UTC.  If
     local time is west of UTC, this should be negative.  Note that
     this is intended to be the total offset from UTC; for example, if
     a tzinfo object represents both time zone and DST
     adjustments, see utcoffset() should return their sum.  If
     the UTC offset isn't known, return `None'.  Else the value
     returned must be a timedelta object specifying a whole
     number of minutes in the range -1439 to 1439 inclusive (1440 =
     24*60; the magnitude of the offset must be less than one day).
     Most implementations of utcoffset() will probably look
     like one of these two:

         return CONSTANT                 # fixed-offset class
         return CONSTANT + self.dst(dt)  # daylight-aware class

     If utcoffset() does not return `None', see dst()
     should not return `None' either.

     The default implementation of utcoffset() raises
     NotImplementedError.

 -- Method: tzinfo.dst (self, dt)
     Return the daylight saving time (DST) adjustment, in minutes east
     of UTC, or `None' if DST information isn't known.  Return
     `timedelta(0)' if DST is not in effect. If DST is in effect,
     return the offset as a timedelta object (see
     utcoffset() for details). Note that DST offset, if
     applicable, has already been added to the UTC offset returned by
     utcoffset(), so there's no need to consult dst()
     unless you're interested in obtaining DST info separately.
     For example, see datetime.timetuple() calls its
     tzinfo.  member's dst() method to determine how
     the `tm_isdst' flag should be set, and tzinfo.fromutc()
     calls dst() to account for DST changes when
     crossing time zones.

     An instance _tz_ of a tzinfo subclass that models both
     standard and daylight times must be consistent in this sense:

     `tz.utcoffset(dt) - tz.dst(dt)'

     must return the same result for every datetime. _dt_
     with `dt.tzinfo == tz'  For sane tzinfo subclasses,
     this expression yields the time zone's "standard offset", which
     should not depend on the date or the time, but only on geographic
     location.  The implementation of datetime.astimezone().
     relies on this, but cannot detect violations; it's the programmer's
     responsibility to ensure it.  If a tzinfo subclass
     cannot guarantee this, it may be able to override the default
     implementation of tzinfo.fromutc() to work correctly
     with `astimezone()' regardless.

     Most implementations of dst() will probably look like
     one of these two:

         def dst(self):
             # a fixed-offset class:  doesn't account for DST
             return timedelta(0)

     or

         def dst(self):
             # Code to set dston and dstoff to the time zone's DST
             # transition times based on the input dt.year, and expressed
             # in standard local time.  Then

             if dston <= dt.replace(tzinfo=None) < dstoff:
                 return timedelta(hours=1)
             else:
                 return timedelta(0)

     The default implementation of dst() raises
     NotImplementedError.

 -- Method: tzinfo.tzname (self, dt)
     Return the time zone name corresponding to the datetime.
     object _dt_, as a string. Nothing about string names is defined by
     the datetime module, and there's no requirement that it
     mean anything in particular.  For example, "GMT", "UTC", "-500",
     "-5:00", "EDT", "US/Eastern", "America/New York" are all valid
     replies.  Return `None' if a string name isn't known.  Note that
     this is a method rather than a fixed string primarily because some
     tzinfo.  subclasses will wish to return different names
     depending on the specific value of _dt_ passed, especially if the
     tzinfo class is accounting for daylight time.

     The default implementation of tzname() raises
     NotImplementedError.

  These methods are called by a datetime or time.
object, in response to their methods of the same names.  A
datetime object passes itself as the argument, and a time
object passes `None' as the argument.  A tzinfo.
subclass's methods should therefore be prepared to accept a _dt_
argument of `None', or of class datetime.

  When `None' is passed, it's up to the class designer to decide the
best response.  For example, returning `None' is appropriate if the
class wishes to say that time objects don't participate in the
tzinfo protocols.  It may be more useful for `utcoffset(None)' to
return the standard UTC offset, as there is no other convention for
discovering the standard offset.

  When a datetime object is passed in response to a
datetime.  method, `dt.tzinfo' is the same object as _self_.
tzinfo methods can rely on this, unless user code calls
tzinfo methods directly.  The intent is that the tzinfo
methods interpret _dt_ as being in local time, and not need worry
about objects in other timezones.

  There is one more tzinfo method that a subclass may wish
to override:

 -- Method: tzinfo.fromutc (self, dt)
     This is called from the default datetime.astimezone().
     implementation.  When called from that, `dt.tzinfo' is _self_, and
     _dt_'s date and time members are to be viewed as expressing a UTC
     time.  The purpose of fromutc() is to adjust the date
     and time members, returning an equivalent datetime in _self_'s
     local time.

     Most tzinfo subclasses should be able to inherit the
     default fromutc() implementation without problems.
     It's strong enough to handle fixed-offset time zones, and time
     zones accounting for both standard and daylight time, and the
     latter even if the DST transition times differ in different years.
     An example of a time zone the default fromutc().
     implementation may not handle correctly in all cases is one where
     the standard offset (from UTC) depends on the specific date and
     time passed, which can happen for political reasons. The default
     implementations of `astimezone()' and fromutc() may
     not produce the result you want if the result is one of the hours
     straddling the moment the standard offset changes.

     Skipping code for error cases, the default fromutc().
     implementation acts like:

         def fromutc(self, dt):
             # raise ValueError error if dt.tzinfo is not self
             dtoff = dt.utcoffset()
             dtdst = dt.dst()
             # raise ValueError if dtoff is None or dtdst is None
             delta = dtoff - dtdst  # this is self's standard offset
             if delta:
                 dt += delta   # convert to standard local time
                 dtdst = dt.dst()
                 # raise ValueError if dtdst is None
             if dtdst:
                 return dt + dtdst
             else:
                 return dt

  Example tzinfo classes:

    from datetime import tzinfo, timedelta, datetime

    ZERO = timedelta(0)
    HOUR = timedelta(hours=1)

    # A UTC class.

    class UTC(tzinfo):
        """UTC"""

        def utcoffset(self, dt):
            return ZERO

        def tzname(self, dt):
            return "UTC"

        def dst(self, dt):
            return ZERO

    utc = UTC()

    # A class building tzinfo objects for fixed-offset time zones.
    # Note that FixedOffset(0, "UTC") is a different way to build a
    # UTC tzinfo object.

    class FixedOffset(tzinfo):
        """Fixed offset in minutes east from UTC."""

        def __init__(self, offset, name):
            self.__offset = timedelta(minutes = offset)
            self.__name = name

        def utcoffset(self, dt):
            return self.__offset

        def tzname(self, dt):
            return self.__name

        def dst(self, dt):
            return ZERO

    # A class capturing the platform's idea of local time.

    import time as _time

    STDOFFSET = timedelta(seconds = -_time.timezone)
    if _time.daylight:
        DSTOFFSET = timedelta(seconds = -_time.altzone)
    else:
        DSTOFFSET = STDOFFSET

    DSTDIFF = DSTOFFSET - STDOFFSET

    class LocalTimezone(tzinfo):

        def utcoffset(self, dt):
            if self._isdst(dt):
                return DSTOFFSET
            else:
                return STDOFFSET

        def dst(self, dt):
            if self._isdst(dt):
                return DSTDIFF
            else:
                return ZERO

        def tzname(self, dt):
            return _time.tzname[self._isdst(dt)]

        def _isdst(self, dt):
            tt = (dt.year, dt.month, dt.day,
                  dt.hour, dt.minute, dt.second,
                  dt.weekday(), 0, -1)
            stamp = _time.mktime(tt)
            tt = _time.localtime(stamp)
            return tt.tm_isdst > 0

    Local = LocalTimezone()

    # A complete implementation of current DST rules for major US time zones.

    def first_sunday_on_or_after(dt):
        days_to_go = 6 - dt.weekday()
        if days_to_go:
            dt += timedelta(days_to_go)
        return dt

    # US DST Rules
    #
    # This is a simplified (i.e., wrong for a few cases) set of rules for US
    # DST start and end times. For a complete and up-to-date set of DST rules
    # and timezone definitions, visit the Olson Database (or try pytz):
    # http://www.twinsun.com/tz/tz-link.htm
    # http://sourceforge.net/projects/pytz/ (might not be up-to-date)
    #
    # In the US, since 2007, DST starts at 2am (standard time) on the second
    # Sunday in March, which is the first Sunday on or after Mar 8.
    DSTSTART_2007 = datetime(1, 3, 8, 2)
    # and ends at 2am (DST time; 1am standard time) on the first Sunday of Nov.
    DSTEND_2007 = datetime(1, 11, 1, 1)
    # From 1987 to 2006, DST used to start at 2am (standard time) on the first
    # Sunday in April and to end at 2am (DST time; 1am standard time) on the last
    # Sunday of October, which is the first Sunday on or after Oct 25.
    DSTSTART_1987_2006 = datetime(1, 4, 1, 2)
    DSTEND_1987_2006 = datetime(1, 10, 25, 1)
    # From 1967 to 1986, DST used to start at 2am (standard time) on the last
    # Sunday in April (the one on or after April 24) and to end at 2am (DST time;
    # 1am standard time) on the last Sunday of October, which is the first Sunday
    # on or after Oct 25.
    DSTSTART_1967_1986 = datetime(1, 4, 24, 2)
    DSTEND_1967_1986 = DSTEND_1987_2006

    class USTimeZone(tzinfo):

        def __init__(self, hours, reprname, stdname, dstname):
            self.stdoffset = timedelta(hours=hours)
            self.reprname = reprname
            self.stdname = stdname
            self.dstname = dstname

        def __repr__(self):
            return self.reprname

        def tzname(self, dt):
            if self.dst(dt):
                return self.dstname
            else:
                return self.stdname

        def utcoffset(self, dt):
            return self.stdoffset + self.dst(dt)

        def dst(self, dt):
            if dt is None or dt.tzinfo is None:
                # An exception may be sensible here, in one or both cases.
                # It depends on how you want to treat them.  The default
                # fromutc() implementation (called by the default astimezone()
                # implementation) passes a datetime with dt.tzinfo is self.
                return ZERO
            assert dt.tzinfo is self

            # Find start and end times for US DST. For years before 1967, return
            # ZERO for no DST.
            if 2006 < dt.year:
                dststart, dstend = DSTSTART_2007, DSTEND_2007
            elif 1986 < dt.year < 2007:
                dststart, dstend = DSTSTART_1987_2006, DSTEND_1987_2006
            elif 1966 < dt.year < 1987:
                dststart, dstend = DSTSTART_1967_1986, DSTEND_1967_1986
            else:
                return ZERO

            start = first_sunday_on_or_after(dststart.replace(year=dt.year))
            end = first_sunday_on_or_after(dstend.replace(year=dt.year))

            # Can't compare naive to aware objects, so strip the timezone from
            # dt first.
            if start <= dt.replace(tzinfo=None) < end:
                return HOUR
            else:
                return ZERO

    Eastern  = USTimeZone(-5, "Eastern",  "EST", "EDT")
    Central  = USTimeZone(-6, "Central",  "CST", "CDT")
    Mountain = USTimeZone(-7, "Mountain", "MST", "MDT")
    Pacific  = USTimeZone(-8, "Pacific",  "PST", "PDT")

Note that there are unavoidable subtleties twice per year in a
tzinfo.  subclass accounting for both standard and daylight time,
at the DST transition points.  For concreteness, consider US Eastern
(UTC -0500), where EDT begins the minute after 1:59 (EST) on the second
Sunday in March, and ends the minute after 1:59 (EDT) on the first
Sunday in November:

      UTC   3:MM  4:MM  5:MM  6:MM  7:MM  8:MM
      EST  22:MM 23:MM  0:MM  1:MM  2:MM  3:MM
      EDT  23:MM  0:MM  1:MM  2:MM  3:MM  4:MM

    start  22:MM 23:MM  0:MM  1:MM  3:MM  4:MM

      end  23:MM  0:MM  1:MM  1:MM  2:MM  3:MM

When DST starts (the "start" line), the local wall clock leaps from
1:59 to 3:00.  A wall time of the form 2:MM doesn't really make sense
on that day, so `astimezone(Eastern)' won't deliver a result with `hour
== 2' on the day DST begins.  In order for `astimezone()' to make this
guarantee, the `rzinfo.dst()' method must consider times in the
"missing hour" (2:MM for Eastern) to be in daylight time.

  When DST ends (the "end" line), there's a potentially worse problem:
there's an hour that can't be spelled unambiguously in local wall time:
the last hour of daylight time.  In Eastern, that's times of the form
5:MM UTC on the day daylight time ends.  The local wall clock leaps
from 1:59 (daylight time) back to 1:00 (standard time) again. Local
times of the form 1:MM are ambiguous.  `astimezone()' mimics the local
clock's behavior by mapping two adjacent UTC hours into the same local
hour then.  In the Eastern example, UTC times of the form 5:MM and 6:MM
both map to 1:MM when converted to Eastern.  In order for
`astimezone()' to make this guarantee, the tzinfo.dst().
method must consider times in the "repeated hour" to be in standard
time.  This is easily arranged, as in the example, by expressing DST
switch times in the time zone's standard local time.

  Applications that can't bear such ambiguities should avoid using
hybrid tzinfo subclasses; there are no ambiguities when
using UTC, or any other fixed-offset tzinfo subclass (such
as a class representing only EST (fixed offset -5 hours), or only EDT
(fixed offset -4 hours)).

5.8.1.7 `strftime()' and `strptime()' Behavior
..............................................

See date, datetime, and time objects all
support a `strftime(format)' method, to create a string representing
the time under the control of an explicit format string.  Broadly
speaking, `d.strftime(fmt)' acts like the time module's
`time.strftime(fmt, d.timetuple())' although not all objects support a
`timetuple()' method.

  Conversely, the datetime.strptime() class method creates a
datetime object from a string representing a date and time
and a corresponding format string. `datetime.strptime(date_string,
format)' is equivalent to `datetime(*(time.strptime(date_string,
format)[0:6]))'.

  For time objects, the format codes for year, month, and
day should not be used, as time objects have no such values.  If
they're used anyway, `1900' is substituted for the year, and `1' for
the month and day.

  For date objects, the format codes for hours, minutes,
seconds, and microseconds should not be used, as date.
objects have no such values.  If they're used anyway, `0' is
substituted for them.

  New in version 2.6: time and datetime objects
support a `%f' format code which expands to the number of microseconds
in the object, zero-padded on the left to six places.

  For a naive object, the `%z' and `%Z' format codes are replaced by
empty strings.

  For an aware object:

`%z'
     `utcoffset()' is transformed into a 5-character string of the form
     +HHMM or -HHMM, where HH is a 2-digit string giving the number of
     UTC offset hours, and MM is a 2-digit string giving the number of
     UTC offset minutes.  For example, if `utcoffset()' returns
     `timedelta(hours=-3, minutes=-30)', `%z' is replaced with the
     string `'-0330''.

`%Z'
     If `tzname()' returns `None', `%Z' is replaced by an empty string.
     Otherwise `%Z' is replaced by the returned value, which must be a
     string.

  The full set of format codes supported varies across platforms,
because Python calls the platform C library's `strftime()' function,
and platform variations are common.

  The following is a list of all the format codes that the C standard
(1989 version) requires, and these work on all platforms with a
standard C implementation.  Note that the 1999 version of the C
standard added additional format codes.

  The exact range of years for which `strftime()' works also varies
across platforms.  Regardless of platform, years before 1900 cannot be
used.

Directive       Meaning                              Notes
-----------------------------------------------------------------
`%a'            Locale's abbreviated weekday name.
`%A'            Locale's full weekday name.
`%b'            Locale's abbreviated month name.
`%B'            Locale's full month name.
`%c'            Locale's appropriate date and time
                representation.
`%d'            Day of the month as a decimal
                number [01,31].
`%f'            Microsecond as a decimal number      (1)
                [0,999999], zero-padded on the left
`%H'            Hour (24-hour clock) as a decimal
                number [00,23].
`%I'            Hour (12-hour clock) as a decimal
                number [01,12].
`%j'            Day of the year as a decimal number
                [001,366].
`%m'            Month as a decimal number [01,12].
`%M'            Minute as a decimal number [00,59].
`%p'            Locale's equivalent of either AM or  (2)
                PM.
`%S'            Second as a decimal number [00,61].  (3)
`%U'            Week number of the year (Sunday as   (4)
                the first day of the week) as a
                decimal number [00,53].  All days
                in a new year preceding the first
                Sunday are considered to be in week
                0.
`%w'            Weekday as a decimal number
                [0(Sunday),6].
`%W'            Week number of the year (Monday as   (4)
                the first day of the week) as a
                decimal number [00,53].  All days
                in a new year preceding the first
                Monday are considered to be in week
                0.
`%x'            Locale's appropriate date
                representation.
`%X'            Locale's appropriate time
                representation.
`%y'            Year without century as a decimal
                number [00,99].
`%Y'            Year with century as a decimal
                number.
`%z'            UTC offset in the form +HHMM or      (5)
                -HHMM (empty string if the the
                object is naive).
`%Z'            Time zone name (empty string if the
                object is naive).
`%%'            A literal `'%'' character.

  Notes:

  1. When used with the `strptime()' method, the `%f' directive accepts
     from one to six digits and zero pads on the right.  `%f' is an
     extension to the set of format characters in the C standard (but
     implemented separately in datetime objects, and therefore always
     available).

  2. When used with the `strptime()' method, the `%p' directive only
     affects the output hour field if the `%I' directive is used to
     parse the hour.

  3. The range really is `0' to `61'; according to the Posix standard
     this accounts for leap seconds and the (very rare) double leap
     seconds.  The time module may produce and does accept
     leap seconds since it is based on the Posix standard, but the
     datetime module does not accept leap seconds in
     `strptime()' input nor will it produce them in `strftime()' output.

  4. When used with the `strptime()' method, `%U' and `%W' are only
     used in calculations when the day of the week and the year are
     specified.

  5. For example, if `utcoffset()' returns `timedelta(hours=-3,
     minutes=-30)', `%z' is replaced with the string `'-0330''.

5.8.2 `calendar' -- General calendar-related functions
------------------------------------------------------

This module allows you to output calendars like the Unix *cal* program,
and provides additional useful functions related to the calendar. By
default, these calendars have Monday as the first day of the week, and
Sunday as the last (the European convention). Use
setfirstweekday() to set the first day of the week to Sunday (6)
or to any other weekday.  Parameters that specify dates are given as
integers. For related functionality, see also the datetime.
and time modules.

  Most of these functions and classes rely on the datetime.
module which uses an idealized calendar, the current Gregorian calendar
indefinitely extended in both directions.  This matches the definition
of the "proleptic Gregorian" calendar in Dershowitz and Reingold's book
"Calendrical Calculations", where it's the base calendar for all
computations.

  Latest version of the calendar module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/calendar.py?view=markup)

 -- Class: calendar.Calendar ([firstweekday])
     Creates a Calendar object. _firstweekday_ is an
     integer specifying the first day of the week. `0' is Monday (the
     default), `6' is Sunday.

     A Calendar object provides several methods that can be
     used for preparing the calendar data for formatting. This class
     doesn't do any formatting itself. This is the job of subclasses.

     New in version 2.5.

     Calendar instances have the following methods:

      -- Method: iterweekdays ()
          Return an iterator for the week day numbers that will be used
          for one week.  The first value from the iterator will be the
          same as the value of the firstweekday property.

      -- Method: itermonthdates (year, month)
          Return an iterator for the month _month_ (1-12) in the year
          _year_. This iterator will return all days (as
          datetime.date objects) for the month and all days
          before the start of the month or after the end of the month
          that are required to get a complete week.

      -- Method: itermonthdays2 (year, month)
          Return an iterator for the month _month_ in the year _year_
          similar to itermonthdates(). Days returned will be
          tuples consisting of a day number and a week day number.

      -- Method: itermonthdays (year, month)
          Return an iterator for the month _month_ in the year _year_
          similar to itermonthdates(). Days returned will
          simply be day numbers.

      -- Method: monthdatescalendar (year, month)
          Return a list of the weeks in the month _month_ of the _year_
          as full weeks.  Weeks are lists of seven datetime.date
          objects.

      -- Method: monthdays2calendar (year, month)
          Return a list of the weeks in the month _month_ of the _year_
          as full weeks.  Weeks are lists of seven tuples of day
          numbers and weekday numbers.

      -- Method: monthdayscalendar (year, month)
          Return a list of the weeks in the month _month_ of the _year_
          as full weeks.  Weeks are lists of seven day numbers.

      -- Method: yeardatescalendar (year[, width])
          Return the data for the specified year ready for formatting.
          The return value is a list of month rows. Each month row
          contains up to _width_ months (defaulting to 3). Each month
          contains between 4 and 6 weeks and each week contains 1-7
          days. Days are datetime.date objects.

      -- Method: yeardays2calendar (year[, width])
          Return the data for the specified year ready for formatting
          (similar to yeardatescalendar()). Entries in the
          week lists are tuples of day numbers and weekday numbers. Day
          numbers outside this month are zero.

      -- Method: yeardayscalendar (year[, width])
          Return the data for the specified year ready for formatting
          (similar to yeardatescalendar()). Entries in the
          week lists are day numbers. Day numbers outside this month
          are zero.

 -- Class: calendar.TextCalendar ([firstweekday])
     This class can be used to generate plain text calendars.

     New in version 2.5.

     TextCalendar instances have the following methods:

      -- Method: formatmonth (theyear, themonth[, w[, l]])
          Return a month's calendar in a multi-line string. If _w_ is
          provided, it specifies the width of the date columns, which
          are centered. If _l_ is given, it specifies the number of
          lines that each week will use. Depends on the first weekday
          as specified in the constructor or set by the
          setfirstweekday() method.

      -- Method: prmonth (theyear, themonth[, w[, l]])
          Print a month's calendar as returned by formatmonth().

      -- Method: formatyear (theyear[, w[, l[, c[, m]]]])
          Return a _m_-column calendar for an entire year as a
          multi-line string.  Optional parameters _w_, _l_, and _c_ are
          for date column width, lines per week, and number of spaces
          between month columns, respectively. Depends on the first
          weekday as specified in the constructor or set by the
          setfirstweekday() method.  The earliest year for which
          a calendar can be generated is platform-dependent.

      -- Method: pryear (theyear[, w[, l[, c[, m]]]])
          Print the calendar for an entire year as returned by
          formatyear().

 -- Class: calendar.HTMLCalendar ([firstweekday])
     This class can be used to generate HTML calendars.

     New in version 2.5.

     HTMLCalendar instances have the following methods:

      -- Method: formatmonth (theyear, themonth[, withyear])
          Return a month's calendar as an HTML table. If _withyear_ is
          true the year will be included in the header, otherwise just
          the month name will be used.

      -- Method: formatyear (theyear[, width])
          Return a year's calendar as an HTML table. _width_
          (defaulting to 3) specifies the number of months per row.

      -- Method: formatyearpage (theyear[, width[, css[, encoding]]])
          Return a year's calendar as a complete HTML page. _width_
          (defaulting to 3) specifies the number of months per row.
          _css_ is the name for the cascading style sheet to be used.
          None can be passed if no style sheet should be
          used. _encoding_ specifies the encoding to be used for the
          output (defaulting to the system default encoding).

 -- Class: calendar.LocaleTextCalendar ([firstweekday[, locale]])
     This subclass of TextCalendar can be passed a locale
     name in the constructor and will return month and weekday names in
     the specified locale.  If this locale includes an encoding all
     strings containing month and weekday names will be returned as
     unicode.

     New in version 2.5.

 -- Class: calendar.LocaleHTMLCalendar ([firstweekday[, locale]])
     This subclass of HTMLCalendar can be passed a locale
     name in the constructor and will return month and weekday names in
     the specified locale. If this locale includes an encoding all
     strings containing month and weekday names will be returned as
     unicode.

     New in version 2.5.

     Note: The `formatweekday()' and `formatmonthname()' methods of
     these two classes temporarily change the current locale to the
     given _locale_.  Because the current locale is a process-wide
     setting, they are not thread-safe.

  For simple text calendars this module provides the following
functions.

 -- Function: calendar.setfirstweekday (weekday)
     Sets the weekday (`0' is Monday, `6' is Sunday) to start each
     week. The values `MONDAY', `TUESDAY', `WEDNESDAY', `THURSDAY',
     `FRIDAY', `SATURDAY', and `SUNDAY' are provided for convenience.
     For example, to set the first weekday to Sunday:

         import calendar
         calendar.setfirstweekday(calendar.SUNDAY)

     New in version 2.0.

 -- Function: calendar.firstweekday ()
     Returns the current setting for the weekday to start each week.

     New in version 2.0.

 -- Function: calendar.isleap (year)
     Returns True if _year_ is a leap year, otherwise
     False.

 -- Function: calendar.leapdays (y1, y2)
     Returns the number of leap years in the range from _y1_ to _y2_
     (exclusive), where _y1_ and _y2_ are years.

     Changed in version 2.0: This function didn't work for ranges
     spanning a century change in Python 1.5.2.

 -- Function: calendar.weekday (year, month, day)
     Returns the day of the week (`0' is Monday) for _year_
     (`1970'-...), _month_ (`1'-`12'), _day_ (`1'-`31').

 -- Function: calendar.weekheader (n)
     Return a header containing abbreviated weekday names. _n_
     specifies the width in characters for one weekday.

 -- Function: calendar.monthrange (year, month)
     Returns weekday of first day of the month and number of days in
     month,  for the specified _year_ and _month_.

 -- Function: calendar.monthcalendar (year, month)
     Returns a matrix representing a month's calendar.  Each row
     represents a week; days outside of the month a represented by
     zeros. Each week begins with Monday unless set by
     setfirstweekday().

 -- Function: calendar.prmonth (theyear, themonth[, w[, l]])
     Prints a month's calendar as returned by month().

 -- Function: calendar.month (theyear, themonth[, w[, l]])
     Returns a month's calendar in a multi-line string using the
     `formatmonth()' of the TextCalendar class.

     New in version 2.0.

 -- Function: calendar.prcal (year[, w[, l[c]]])
     Prints the calendar for an entire year as returned by
     calendar().

 -- Function: calendar.calendar (year[, w[, l[c]]])
     Returns a 3-column calendar for an entire year as a multi-line
     string using the `formatyear()' of the TextCalendar.
     class.

     New in version 2.0.

 -- Function: calendar.timegm (tuple)
     An unrelated but handy function that takes a time tuple such as
     returned by the `gmtime()' function in the time.
     module, and returns the corresponding Unix timestamp value,
     assuming an epoch of 1970, and the POSIX encoding.  In fact, see
     time.gmtime() and timegm() are each others'
     inverse.

     New in version 2.0.

  The calendar module exports the following data attributes:

 -- Data: calendar.day_name
     An array that represents the days of the week in the current
     locale.

 -- Data: calendar.day_abbr
     An array that represents the abbreviated days of the week in the
     current locale.

 -- Data: calendar.month_name
     An array that represents the months of the year in the current
     locale.  This follows normal convention of January being month
     number 1, so it has a length of 13 and  `month_name[0]' is the
     empty string.

 -- Data: calendar.month_abbr
     An array that represents the abbreviated months of the year in the
     current locale.  This follows normal convention of January being
     month number 1, so it has a length of 13 and  `month_abbr[0]' is
     the empty string.

Module datetime.
     Object-oriented interface to dates and times with similar
     functionality to the time module.

Module time.
     Low-level time related functions.

5.8.3 `collections' -- High-performance container datatypes
-----------------------------------------------------------

New in version 2.4.

  This module implements specialized container datatypes providing
alternatives to Python's general purpose built-in containers, see
dict, list, set, and tuple.

See namedtuple().  factory function for creating tuple subclasses with named fields         New in version 2.6.
See deque.         list-like container with fast appends and pops on either end             New in version 2.4.
See Counter.       dict subclass for counting hashable objects                              New in version 2.7.
See OrderedDict.   dict subclass that remembers the order entries were added                New in version 2.7.
See defaultdict.   dict subclass that calls a factory function to supply missing values     New in version 2.5.

  In addition to containers, the collections module provides some ABCs
(abstract base classes) that can be used to test whether a class
provides a particular interface, for example, whether it is hashable or
a mapping.

  Latest version of the collections module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/collections.py?view=markup)

5.8.3.1 `Counter' objects
.........................

A counter tool is provided to support convenient and rapid tallies.
For example:

    >>> # Tally occurrences of words in a list
    >>> cnt = Counter()
    >>> for word in ['red', 'blue', 'red', 'green', 'blue', 'blue']:
    ...     cnt[word] += 1
    >>> cnt
    Counter({'blue': 3, 'red': 2, 'green': 1})

    >>> # Find the ten most common words in Hamlet
    >>> import re
    >>> words = re.findall('\w+', open('hamlet.txt').read().lower())
    >>> Counter(words).most_common(10)
    [('the', 1143), ('and', 966), ('to', 762), ('of', 669), ('i', 631),
     ('you', 554),  ('a', 546), ('my', 514), ('hamlet', 471), ('in', 451)]

 -- Class: collections.Counter ([iterable-or-mapping])
     A Counter is a dict subclass for counting
     hashable objects.  It is an unordered collection where elements
     are stored as dictionary keys and their counts are stored as
     dictionary values.  Counts are allowed to be any integer value
     including zero or negative counts.  The Counter.  class
     is similar to bags or multisets in other languages.

     Elements are counted from an _iterable_ or initialized from another
     _mapping_ (or counter):

         >>> c = Counter()                           # a new, empty counter
         >>> c = Counter('gallahad')                 # a new counter from an iterable
         >>> c = Counter({'red': 4, 'blue': 2})      # a new counter from a mapping
         >>> c = Counter(cats=4, dogs=8)             # a new counter from keyword args

     Counter objects have a dictionary interface except that they
     return a zero count for missing items instead of raising a
     KeyError.:

         >>> c = Counter(['eggs', 'ham'])
         >>> c['bacon']                              # count of a missing element is zero
         0

     Setting a count to zero does not remove an element from a counter.
     Use `del' to remove it entirely:

         >>> c['sausage'] = 0                        # counter entry with a zero count
         >>> del c['sausage']                        # del actually removes the entry

     New in version 2.7.

     Counter objects support three methods beyond those available for
     all dictionaries:

      -- Method: elements ()
          Return an iterator over elements repeating each as many times
          as its count.  Elements are returned in arbitrary order.  If
          an element's count is less than one, see elements().
          will ignore it.

              >>> c = Counter(a=4, b=2, c=0, d=-2)
              >>> list(c.elements())
              ['a', 'a', 'a', 'a', 'b', 'b']

      -- Method: most_common ([n])
          Return a list of the _n_ most common elements and their
          counts from the most common to the least.  If _n_ is not
          specified, see most_common().  returns _all_ elements
          in the counter.  Elements with equal counts are ordered
          arbitrarily:

              >>> Counter('abracadabra').most_common(3)
              [('a', 5), ('r', 2), ('b', 2)]

      -- Method: subtract ([iterable-or-mapping])
          Elements are subtracted from an _iterable_ or from another
          _mapping_ (or counter).  Like dict.update() but
          subtracts counts instead of replacing them.  Both inputs and
          outputs may be zero or negative.

              >>> c = Counter(a=4, b=2, c=0, d=-2)
              >>> d = Counter(a=1, b=2, c=3, d=4)
              >>> c.subtract(d)
              Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})

     The usual dictionary methods are available for Counter.
     objects except for two which work differently for counters.

      -- Method: fromkeys (iterable)
          This class method is not implemented for Counter.
          objects.

      -- Method: update ([iterable-or-mapping])
          Elements are counted from an _iterable_ or added-in from
          another _mapping_ (or counter).  Like dict.update()
          but adds counts instead of replacing them.  Also, the
          _iterable_ is expected to be a sequence of elements, not a
          sequence of `(key, value)' pairs.

  Common patterns for working with Counter objects:

    sum(c.values())                 # total of all counts
    c.clear()                       # reset all counts
    list(c)                         # list unique elements
    set(c)                          # convert to a set
    dict(c)                         # convert to a regular dictionary
    c.items()                       # convert to a list of (elem, cnt) pairs
    Counter(dict(list_of_pairs))    # convert from a list of (elem, cnt) pairs
    c.most_common()[:-n:-1]         # n least common elements
    c += Counter()                  # remove zero and negative counts

Several mathematical operations are provided for combining
Counter.  objects to produce multisets (counters that have counts
greater than zero).  Addition and subtraction combine counters by
adding or subtracting the counts of corresponding elements.
Intersection and union return the minimum and maximum of corresponding
counts.  Each operation can accept inputs with signed counts, but the
output will exclude results with counts of zero or less.

    >>> c = Counter(a=3, b=1)
    >>> d = Counter(a=1, b=2)
    >>> c + d                       # add two counters together:  c[x] + d[x]
    Counter({'a': 4, 'b': 3})
    >>> c - d                       # subtract (keeping only positive counts)
    Counter({'a': 2})
    >>> c & d                       # intersection:  min(c[x], d[x])
    Counter({'a': 1, 'b': 1})
    >>> c | d                       # union:  max(c[x], d[x])
    Counter({'a': 3, 'b': 2})

     Note: Counters were primarily designed to work with positive
     integers to represent running counts; however, care was taken to
     not unnecessarily preclude use cases needing other types or
     negative values.  To help with those use cases, this section
     documents the minimum range and type restrictions.

        * The Counter class itself is a dictionary subclass
          with no restrictions on its keys and values.  The values are
          intended to be numbers representing counts, but you _could_
          store anything in the value field.

        * The `most_common()' method requires only that the values be
          orderable.

        * For in-place operations such as `c[key] += 1', the value type
          need only support addition and subtraction.  So fractions,
          floats, and decimals would work and negative values are
          supported.  The same is also true for `update()' and
          `subtract()' which allow negative and zero values for both
          inputs and outputs.

        * The multiset methods are designed only for use cases with
          positive values.  The inputs may be negative or zero, but
          only outputs with positive values are created.  There are no
          type restrictions, but the value type needs to support
          support addition, subtraction, and comparison.

        * The `elements()' method requires integer counts.  It ignores
          zero and negative counts.

   * Counter class (http://code.activestate.com/recipes/576611/)adapted
     for Python 2.5 and an early Bag recipe
     (http://code.activestate.com/recipes/259174/)for Python 2.4.

   * Bag class
     (http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html)in
     Smalltalk.

   * Wikipedia entry for Multisets
     (http://en.wikipedia.org/wiki/Multiset).

   * C++ multisets
     (http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm)tutorial
     with examples.

   * For mathematical operations on multisets and their use cases, see
     _Knuth, Donald. The Art of Computer Programming Volume II, Section
     4.6.3, Exercise 19_.

   * To enumerate all distinct multisets of a given size over a given
     set of elements, see
     itertools.combinations_with_replacement().

          map(Counter, combinations_with_replacement('ABC', 2)) -> AA
          AB AC BB BC CC

5.8.3.2 `deque' objects
.......................

 -- Class: collections.deque ([iterable[, maxlen]])
     Returns a new deque object initialized left-to-right (using
     append()) with data from _iterable_.  If _iterable_ is not
     specified, the new deque is empty.

     Deques are a generalization of stacks and queues (the name is
     pronounced "deck" and is short for "double-ended queue").  Deques
     support thread-safe, memory efficient appends and pops from either
     side of the deque with approximately the same O(1) performance in
     either direction.

     Though list objects support similar operations, they
     are optimized for fast fixed-length operations and incur O(n)
     memory movement costs for `pop(0)' and `insert(0, v)' operations
     which change both the size and position of the underlying data
     representation.

     New in version 2.4.

     If _maxlen_ is not specified or is _None_, deques may grow to an
     arbitrary length.  Otherwise, the deque is bounded to the
     specified maximum length.  Once a bounded length deque is full,
     when new items are added, a corresponding number of items are
     discarded from the opposite end.  Bounded length deques provide
     functionality similar to the `tail' filter in Unix. They are also
     useful for tracking transactions and other pools of data where
     only the most recent activity is of interest.

     Changed in version 2.6: Added _maxlen_ parameter.

     Deque objects support the following methods:

      -- Method: append (x)
          Add _x_ to the right side of the deque.

      -- Method: appendleft (x)
          Add _x_ to the left side of the deque.

      -- Method: clear ()
          Remove all elements from the deque leaving it with length 0.

      -- Method: count (x)
          Count the number of deque elements equal to _x_.

          New in version 2.7.

      -- Method: extend (iterable)
          Extend the right side of the deque by appending elements from
          the iterable argument.

      -- Method: extendleft (iterable)
          Extend the left side of the deque by appending elements from
          _iterable_.  Note, the series of left appends results in
          reversing the order of elements in the iterable argument.

      -- Method: pop ()
          Remove and return an element from the right side of the
          deque. If no elements are present, raises an
          IndexError.

      -- Method: popleft ()
          Remove and return an element from the left side of the deque.
          If no elements are present, raises an IndexError.

      -- Method: remove (value)
          Removed the first occurrence of _value_.  If not found,
          raises a ValueError.

          New in version 2.5.

      -- Method: reverse ()
          Reverse the elements of the deque in-place and then return
          `None'.

          New in version 2.7.

      -- Method: rotate (n)
          Rotate the deque _n_ steps to the right.  If _n_ is negative,
          rotate to the left.  Rotating one step to the right is
          equivalent to: `d.appendleft(d.pop())'.

     Deque objects also provide one read-only attribute:

      -- Attribute: maxlen
          Maximum size of a deque or _None_ if unbounded.

          New in version 2.7.

In addition to the above, deques support iteration, pickling, `len(d)',
`reversed(d)', `copy.copy(d)', `copy.deepcopy(d)', membership testing
with the in operator, and subscript references such as
`d[-1]'.  Indexed access is O(1) at both ends but slows to O(n) in the
middle.  For fast random access, use lists instead.

  Example:

    >>> from collections import deque
    >>> d = deque('ghi')                 # make a new deque with three items
    >>> for elem in d:                   # iterate over the deque's elements
    ...     print elem.upper()
    G
    H
    I

    >>> d.append('j')                    # add a new entry to the right side
    >>> d.appendleft('f')                # add a new entry to the left side
    >>> d                                # show the representation of the deque
    deque(['f', 'g', 'h', 'i', 'j'])

    >>> d.pop()                          # return and remove the rightmost item
    'j'
    >>> d.popleft()                      # return and remove the leftmost item
    'f'
    >>> list(d)                          # list the contents of the deque
    ['g', 'h', 'i']
    >>> d[0]                             # peek at leftmost item
    'g'
    >>> d[-1]                            # peek at rightmost item
    'i'

    >>> list(reversed(d))                # list the contents of a deque in reverse
    ['i', 'h', 'g']
    >>> 'h' in d                         # search the deque
    True
    >>> d.extend('jkl')                  # add multiple elements at once
    >>> d
    deque(['g', 'h', 'i', 'j', 'k', 'l'])
    >>> d.rotate(1)                      # right rotation
    >>> d
    deque(['l', 'g', 'h', 'i', 'j', 'k'])
    >>> d.rotate(-1)                     # left rotation
    >>> d
    deque(['g', 'h', 'i', 'j', 'k', 'l'])

    >>> deque(reversed(d))               # make a new deque in reverse order
    deque(['l', 'k', 'j', 'i', 'h', 'g'])
    >>> d.clear()                        # empty the deque
    >>> d.pop()                          # cannot pop from an empty deque
    Traceback (most recent call last):
      File "<pyshell#6>", line 1, in -toplevel-
        d.pop()
    IndexError: pop from an empty deque

    >>> d.extendleft('abc')              # extendleft() reverses the input order
    >>> d
    deque(['c', 'b', 'a'])

5.8.3.3 `deque' Recipes
.......................

This section shows various approaches to working with deques.

  Bounded length deques provide functionality similar to the `tail'
filter in Unix:

    def tail(filename, n=10):
        'Return the last n lines of a file'
        return deque(open(filename), n)

Another approach to using deques is to maintain a sequence of recently
added elements by appending to the right and popping to the left:

    def moving_average(iterable, n=3):
        # moving_average([40, 30, 50, 46, 39, 44]) --> 40.0 42.0 45.0 43.0
        # http://en.wikipedia.org/wiki/Moving_average
        it = iter(iterable)
        d = deque(itertools.islice(it, n-1))
        d.appendleft(0)
        s = sum(d)
        for elem in it:
            s += elem - d.popleft()
            d.append(elem)
            yield s / float(n)

The `rotate()' method provides a way to implement deque.
slicing and deletion.  For example, a pure Python implementation of
`del d[n]' relies on the `rotate()' method to position elements to be
popped:

    def delete_nth(d, n):
        d.rotate(-n)
        d.popleft()
        d.rotate(n)

To implement deque slicing, use a similar approach applying
`rotate()' to bring a target element to the left side of the deque.
Remove old entries with `popleft()', add new entries with `extend()',
and then reverse the rotation.  With minor variations on that approach,
it is easy to implement Forth style stack manipulations such as `dup',
`drop', `swap', `over', `pick', `rot', and `roll'.

5.8.3.4 `defaultdict' objects
.............................

 -- Class: collections.defaultdict ([default_factory[, ...]])
     Returns a new dictionary-like object.  defaultdict is
     a subclass of the built-in dict class.  It overrides
     one method and adds one writable instance variable.  The remaining
     functionality is the same as for the dict class and is
     not documented here.

     The first argument provides the initial value for the
     default_factory.  attribute; it defaults to `None'. All
     remaining arguments are treated the same as if they were passed to
     the dict constructor, including keyword arguments.

     New in version 2.5.

     defaultdict objects support the following method in
     addition to the standard dict operations:

      -- Method: __missing__ (key)
          If the default_factory attribute is `None', this
          raises a KeyError exception with the _key_ as
          argument.

          If default_factory is not `None', it is called
          without arguments to provide a default value for the given
          _key_, this value is inserted in the dictionary for the
          _key_, and returned.

          If calling default_factory raises an exception
          this exception is propagated unchanged.

          This method is called by the __getitem__() method
          of the dict class when the requested key is not
          found; whatever it returns or raises is then returned or
          raised by __getitem__().

     defaultdict objects support the following instance
     variable:

      -- Attribute: default_factory
          This attribute is used by the __missing__().
          method; it is initialized from the first argument to the
          constructor, if present, or to `None', if absent.

5.8.3.5 `defaultdict' Examples
..............................

Using list as the `default_factory', it is easy to group a
sequence of key-value pairs into a dictionary of lists:

    >>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
    >>> d = defaultdict(list)
    >>> for k, v in s:
    ...     d[k].append(v)
    ...
    >>> d.items()
    [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

When each key is encountered for the first time, it is not already in
the mapping; so an entry is automatically created using the
`default_factory' function which returns an empty list.  The
`list.append()' operation then attaches the value to the new list.
When keys are encountered again, the look-up proceeds normally
(returning the list for that key) and the `list.append()' operation
adds another value to the list. This technique is simpler and faster
than an equivalent technique using dict.setdefault().:

    >>> d = {}
    >>> for k, v in s:
    ...     d.setdefault(k, []).append(v)
    ...
    >>> d.items()
    [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

Setting the `default_factory' to int makes the
defaultdict useful for counting (like a bag or multiset in other
languages):

    >>> s = 'mississippi'
    >>> d = defaultdict(int)
    >>> for k in s:
    ...     d[k] += 1
    ...
    >>> d.items()
    [('i', 4), ('p', 2), ('s', 4), ('m', 1)]

When a letter is first encountered, it is missing from the mapping, so
the `default_factory' function calls int() to supply a
default count of zero.  The increment operation then builds up the
count for each letter.

  The function int() which always returns zero is just a
special case of constant functions.  A faster and more flexible way to
create constant functions is to use itertools.repeat().
which can supply any constant value (not just zero):

    >>> def constant_factory(value):
    ...     return itertools.repeat(value).next
    >>> d = defaultdict(constant_factory('<missing>'))
    >>> d.update(name='John', action='ran')
    >>> '%(name)s %(action)s to %(object)s' % d
    'John ran to <missing>'

Setting the `default_factory' to set makes the
defaultdict useful for building a dictionary of sets:

    >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)]
    >>> d = defaultdict(set)
    >>> for k, v in s:
    ...     d[k].add(v)
    ...
    >>> d.items()
    [('blue', set([2, 4])), ('red', set([1, 3]))]

5.8.3.6 `namedtuple()' Factory Function for Tuples with Named Fields
....................................................................

Named tuples assign meaning to each position in a tuple and allow for
more readable, self-documenting code.  They can be used wherever
regular tuples are used, and they add the ability to access fields by
name instead of position index.

 -- Function: collections.namedtuple (typename, field_names[,
          verbose=False][, rename=False])
     Returns a new tuple subclass named _typename_.  The new subclass
     is used to create tuple-like objects that have fields accessible
     by attribute lookup as well as being indexable and iterable.
     Instances of the subclass also have a helpful docstring (with
     typename and field_names) and a helpful __repr__().
     method which lists the tuple contents in a `name=value' format.

     The _field_names_ are a single string with each fieldname
     separated by whitespace and/or commas, for example `'x y'' or `'x,
     y''.  Alternatively, _field_names_ can be a sequence of strings
     such as `['x', 'y']'.

     Any valid Python identifier may be used for a fieldname except for
     names starting with an underscore.  Valid identifiers consist of
     letters, digits, and underscores but do not start with a digit or
     underscore and cannot be a keyword such as _class_,
     _for_, _return_, _global_, _pass_, _print_, or _raise_.

     If _rename_ is true, invalid fieldnames are automatically replaced
     with positional names.  For example, `['abc', 'def', 'ghi',
     'abc']' is converted to `['abc', '_1', 'ghi', '_3']', eliminating
     the keyword `def' and the duplicate fieldname `abc'.

     If _verbose_ is true, the class definition is printed just before
     being built.

     Named tuple instances do not have per-instance dictionaries, so
     they are lightweight and require no more memory than regular
     tuples.

     New in version 2.6.

     Changed in version 2.7: added support for _rename_.

  Example:

    >>> Point = namedtuple('Point', 'x y', verbose=True)
    class Point(tuple):
            'Point(x, y)'

            __slots__ = ()

            _fields = ('x', 'y')

            def __new__(_cls, x, y):
                'Create a new instance of Point(x, y)'
                return _tuple.__new__(_cls, (x, y))

            @classmethod
            def _make(cls, iterable, new=tuple.__new__, len=len):
                'Make a new Point object from a sequence or iterable'
                result = new(cls, iterable)
                if len(result) != 2:
                    raise TypeError('Expected 2 arguments, got %d' % len(result))
                return result

            def __repr__(self):
                'Return a nicely formatted representation string'
                return 'Point(x=%r, y=%r)' % self

            def _asdict(self):
                'Return a new OrderedDict which maps field names to their values'
                return OrderedDict(zip(self._fields, self))

            def _replace(_self, **kwds):
                'Return a new Point object replacing specified fields with new values'
                result = _self._make(map(kwds.pop, ('x', 'y'), _self))
                if kwds:
                    raise ValueError('Got unexpected field names: %r' % kwds.keys())
                return result

            def __getnewargs__(self):
                'Return self as a plain tuple.   Used by copy and pickle.'
                return tuple(self)

            x = _property(_itemgetter(0), doc='Alias for field number 0')
            y = _property(_itemgetter(1), doc='Alias for field number 1')

    >>> p = Point(11, y=22)     # instantiate with positional or keyword arguments
    >>> p[0] + p[1]             # indexable like the plain tuple (11, 22)
    33
    >>> x, y = p                # unpack like a regular tuple
    >>> x, y
    (11, 22)
    >>> p.x + p.y               # fields also accessible by name
    33
    >>> p                       # readable __repr__ with a name=value style
    Point(x=11, y=22)

Named tuples are especially useful for assigning field names to result
tuples returned by the csv or sqlite3 modules:

    EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')

    import csv
    for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
        print emp.name, emp.title

    import sqlite3
    conn = sqlite3.connect('/companydata')
    cursor = conn.cursor()
    cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
    for emp in map(EmployeeRecord._make, cursor.fetchall()):
        print emp.name, emp.title

In addition to the methods inherited from tuples, named tuples support
three additional methods and one attribute.  To prevent conflicts with
field names, the method and attribute names start with an underscore.

 -- Class Method: somenamedtuple._make (iterable)
     Class method that makes a new instance from an existing sequence
     or iterable.

         >>> t = [11, 22]
         >>> Point._make(t)
         Point(x=11, y=22)

 -- Method: somenamedtuple._asdict ()
     Return a new OrderedDict which maps field names to
     their corresponding values:

         >>> p._asdict()
         OrderedDict([('x', 11), ('y', 22)])

     Changed in version 2.7: Returns an OrderedDict instead
     of a regular dict.

 -- Method: somenamedtuple._replace (kwargs)
     Return a new instance of the named tuple replacing specified
     fields with new values:

         >>> p = Point(x=11, y=22)
         >>> p._replace(x=33)
         Point(x=33, y=22)

         >>> for partnum, record in inventory.items():
         ...     inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now())

 -- Attribute: somenamedtuple._fields
     Tuple of strings listing the field names.  Useful for introspection
     and for creating new named tuple types from existing named tuples.

         >>> p._fields            # view the field names
         ('x', 'y')

         >>> Color = namedtuple('Color', 'red green blue')
         >>> Pixel = namedtuple('Pixel', Point._fields + Color._fields)
         >>> Pixel(11, 22, 128, 255, 0)
         Pixel(x=11, y=22, red=128, green=255, blue=0)

  To retrieve a field whose name is stored in a string, use the
getattr().  function:

    >>> getattr(p, 'x')
    11

To convert a dictionary to a named tuple, use the double-star-operator
(as described in Unpacking Argument Lists):

    >>> d = {'x': 11, 'y': 22}
    >>> Point(**d)
    Point(x=11, y=22)

Since a named tuple is a regular Python class, it is easy to add or
change functionality with a subclass.  Here is how to add a calculated
field and a fixed-width print format:

    >>> class Point(namedtuple('Point', 'x y')):
    ...     __slots__ = ()
    ...     @property
    ...     def hypot(self):
    ...         return (self.x ** 2 + self.y ** 2) ** 0.5
    ...     def __str__(self):
    ...         return 'Point: x=%6.3f  y=%6.3f  hypot=%6.3f' % (self.x, self.y, self.hypot)

    >>> for p in Point(3, 4), Point(14, 5/7.):
    ...     print p
    Point: x= 3.000  y= 4.000  hypot= 5.000
    Point: x=14.000  y= 0.714  hypot=14.018

The subclass shown above sets `__slots__' to an empty tuple.  This helps
keep memory requirements low by preventing the creation of instance
dictionaries.

  Subclassing is not useful for adding new, stored fields.  Instead,
simply create a new named tuple type from the `_fields' attribute:

    >>> Point3D = namedtuple('Point3D', Point._fields + ('z',))

Default values can be implemented by using `_replace()' to customize a
prototype instance:

    >>> Account = namedtuple('Account', 'owner balance transaction_count')
    >>> default_account = Account('<owner name>', 0.0, 0)
    >>> johns_account = default_account._replace(owner='John')

Enumerated constants can be implemented with named tuples, but it is
simpler and more efficient to use a simple class declaration:

    >>> Status = namedtuple('Status', 'open pending closed')._make(range(3))
    >>> Status.open, Status.pending, Status.closed
    (0, 1, 2)
    >>> class Status:
    ...     open, pending, closed = range(3)

Named tuple recipe (http://code.activestate.com/recipes/500261/)adapted
for Python 2.4.

5.8.3.7 `OrderedDict' objects
.............................

Ordered dictionaries are just like regular dictionaries but they
remember the order that items were inserted.  When iterating over an
ordered dictionary, the items are returned in the order their keys were
first added.

 -- Class: collections.OrderedDict ([items])
     Return an instance of a dict subclass, supporting the usual
     dict.  methods.  An _OrderedDict_ is a dict that remembers
     the order that keys were first inserted. If a new entry overwrites
     an existing entry, the original insertion position is left
     unchanged.  Deleting an entry and reinserting it will move it to
     the end.

     New in version 2.7.

 -- Method: OrderedDict.popitem (last=True)
     The popitem() method for ordered dictionaries returns
     and removes a (key, value) pair.  The pairs are returned in LIFO
     order if _last_ is true or FIFO order if false.

  In addition to the usual mapping methods, ordered dictionaries also
support reverse iteration using reversed().

  Equality tests between OrderedDict objects are
order-sensitive and are implemented as
`list(od1.items())==list(od2.items())'.  Equality tests between
OrderedDict objects and other `Mapping' objects are
order-insensitive like regular dictionaries.  This allows
OrderedDict objects to be substituted anywhere a regular
dictionary is used.

  The OrderedDict constructor and `update()' method both
accept keyword arguments, but their order is lost because Python's
function call semantics pass-in keyword arguments using a regular
unordered dictionary.

  Equivalent OrderedDict recipe
(http://code.activestate.com/recipes/576693/)that runs on Python 2.4 or
later.

  Since an ordered dictionary remembers its insertion order, it can be
used in conjuction with sorting to make a sorted dictionary:

    >>> # regular unsorted dictionary
    >>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2}

    >>> # dictionary sorted by key
    >>> OrderedDict(sorted(d.items(), key=lambda t: t[0]))
    OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])

    >>> # dictionary sorted by value
    >>> OrderedDict(sorted(d.items(), key=lambda t: t[1]))
    OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)])

    >>> # dictionary sorted by length of the key string
    >>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0])))
    OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)])

The new sorted dictionaries maintain their sort order when entries are
deleted.  But when new keys are added, the keys are appended to the end
and the sort is not maintained.

  It is also straight-forward to create an ordered dictionary variant
that the remembers the order the keys were _last_ inserted.  If a new
entry overwrites an existing entry, the original insertion position is
changed and moved to the end:

    class LastUpdatedOrderedDict(OrderedDict):
        'Store items is the order the keys were last added'
        def __setitem__(self, key, value):
            if key in self:
                del self[key]
            OrderedDict.__setitem__(self, key, value)

5.8.3.8 ABCs - abstract base classes
....................................

The collections module offers the following ABCs:

ABC                           Inherits                  Abstract Methods           Mixin Methods
--------------------------------------------------------------------------------------------------------------------------------------------
`Container'                                             `__contains__'
`Hashable'                                              `__hash__'
`Iterable'                                              `__iter__'
`Iterator'                    `Iterable'                `next'                     `__iter__'
`Sized'                                                 `__len__'
`Callable'                                              `__call__'
`Sequence'                    `Sized', `Iterable',      `__getitem__'              `__contains__'. `__iter__', `__reversed__'.  `index',
                              `Container'                                          and `count'
`MutableSequence'             `Sequence'                `__setitem__'              Inherited Sequence methods and `append', `reverse',
                                                        `__delitem__', and         `extend', `pop', `remove', and `__iadd__'
                                                        `insert'
`Set'                         `Sized', `Iterable',                                 `__le__', `__lt__', `__eq__', `__ne__', `__gt__',
                              `Container'                                          `__ge__', `__and__', `__or__' `__sub__', `__xor__', and
                                                                                   `isdisjoint'
`MutableSet'                  `Set'                     `add' and `discard'        Inherited Set methods and `clear', `pop', `remove',
                                                                                   `__ior__', `__iand__', `__ixor__', and `__isub__'
`Mapping'                     `Sized', `Iterable',      `__getitem__'              `__contains__', `keys', `items', `values', `get',
                              `Container'                                          `__eq__', and `__ne__'
`MutableMapping'              `Mapping'                 `__setitem__' and          Inherited Mapping methods and `pop', `popitem',
                                                        `__delitem__'              `clear', `update', and `setdefault'
`MappingView'                 `Sized'                                              `__len__'
`KeysView'                    `MappingView', `Set'                                 `__contains__', `__iter__'
`ItemsView'                   `MappingView', `Set'                                 `__contains__', `__iter__'
`ValuesView'                  `MappingView'                                        `__contains__', `__iter__'

  These ABCs allow us to ask classes or instances if they provide
particular functionality, for example:

    size = None
    if isinstance(myvar, collections.Sized):
        size = len(myvar)

Several of the ABCs are also useful as mixins that make it easier to
develop classes supporting container APIs.  For example, to write a
class supporting the full `Set' API, it only necessary to supply the
three underlying abstract methods: __contains__(),
__iter__(), and __len__().  The ABC supplies the
remaining methods such as __and__() and `isdisjoint()'

    class ListBasedSet(collections.Set):
         ''' Alternate set implementation favoring space over speed
             and not requiring the set elements to be hashable. '''
         def __init__(self, iterable):
             self.elements = lst = []
             for value in iterable:
                 if value not in lst:
                     lst.append(value)
         def __iter__(self):
             return iter(self.elements)
         def __contains__(self, value):
             return value in self.elements
         def __len__(self):
             return len(self.elements)

    s1 = ListBasedSet('abcdef')
    s2 = ListBasedSet('defghi')
    overlap = s1 & s2            # The __and__() method is supported automatically

Notes on using `Set' and `MutableSet' as a mixin:

  1. Since some set operations create new sets, the default mixin
     methods need a way to create new instances from an iterable. The
     class constructor is assumed to have a signature in the form
     `ClassName(iterable)'.  That assumption is factored-out to an
     internal classmethod called `_from_iterable()' which calls
     `cls(iterable)' to produce a new set.  If the `Set' mixin is being
     used in a class with a different constructor signature, you will
     need to override `from_iterable()' with a classmethod that can
     construct new instances from an iterable argument.

  2. To override the comparisons (presumably for speed, as the
     semantics are fixed), redefine __le__() and then the
     other operations will automatically follow suit.

  3. The `Set' mixin provides a `_hash()' method to compute a hash value
     for the set; however, see __hash__() is not defined
     because not all sets are hashable or immutable.  To add set
     hashabilty using mixins, inherit from both `Set()' and
     `Hashable()', then define `__hash__ = Set._hash'.

   * Latest version of the Python source code for the collections
     abstract base classes
     (http://svn.python.org/view/python/branches/release27-maint/Lib/_abcoll.py?view=markup)

   * OrderedSet recipe (http://code.activestate.com/recipes/576694/)for
     an example built on `MutableSet'.

   * For more about ABCs, see the abc module and PEP 3119
     (http://www.python.org/dev/peps/pep-3119).

5.8.4 `heapq' -- Heap queue algorithm
-------------------------------------

New in version 2.3.

  This module provides an implementation of the heap queue algorithm,
also known as the priority queue algorithm.

  Latest version of the heapq Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/heapq.py?view=markup)Heaps
are binary trees for which every parent node has a value less than or
equal to any of its children.  This implementation uses arrays for which
`heap[k] <= heap[2*k+1]' and `heap[k] <= heap[2*k+2]' for all _k_,
counting elements from zero.  For the sake of comparison, non-existing
elements are considered to be infinite.  The interesting property of a
heap is that its smallest element is always the root, `heap[0]'.

  The API below differs from textbook heap algorithms in two aspects:
(a) We use zero-based indexing.  This makes the relationship between
the index for a node and the indexes for its children slightly less
obvious, but is more suitable since Python uses zero-based indexing.
(b) Our pop method returns the smallest item, not the largest (called a
"min heap" in textbooks; a "max heap" is more common in texts because
of its suitability for in-place sorting).

  These two make it possible to view the heap as a regular Python list
without surprises: `heap[0]' is the smallest item, and `heap.sort()'
maintains the heap invariant!

  To create a heap, use a list initialized to `[]', or you can
transform a populated list into a heap via function heapify().

  The following functions are provided:

 -- Function: heapq.heappush (heap, item)
     Push the value _item_ onto the _heap_, maintaining the heap
     invariant.

 -- Function: heapq.heappop (heap)
     Pop and return the smallest item from the _heap_, maintaining the
     heap invariant.  If the heap is empty, see IndexError is
     raised.

 -- Function: heapq.heappushpop (heap, item)
     Push _item_ on the heap, then pop and return the smallest item
     from the _heap_.  The combined action runs more efficiently than
     heappush().  followed by a separate call to
     heappop().

     New in version 2.6.

 -- Function: heapq.heapify (x)
     Transform list _x_ into a heap, in-place, in linear time.

 -- Function: heapq.heapreplace (heap, item)
     Pop and return the smallest item from the _heap_, and also push
     the new _item_.  The heap size doesn't change. If the heap is
     empty, see IndexError is raised.

     This one step operation is more efficient than a heappop()
     followed by heappush() and can be more
     appropriate when using a fixed-size heap.  The pop/push
     combination always returns an element from the heap and replaces
     it with _item_.

     The value returned may be larger than the _item_ added.  If that
     isn't desired, consider using heappushpop() instead.
     Its push/pop combination returns the smaller of the two values,
     leaving the larger value on the heap.

  The module also offers three general purpose functions based on heaps.

 -- Function: heapq.merge (*iterables)
     Merge multiple sorted inputs into a single sorted output (for
     example, merge timestamped entries from multiple log files).
     Returns an iterator.  over the sorted values.

     Similar to `sorted(itertools.chain(*iterables))' but returns an
     iterable, does not pull the data into memory all at once, and
     assumes that each of the input streams is already sorted (smallest
     to largest).

     New in version 2.6.

 -- Function: heapq.nlargest (n, iterable[, key])
     Return a list with the _n_ largest elements from the dataset
     defined by _iterable_.  _key_, if provided, specifies a function
     of one argument that is used to extract a comparison key from each
     element in the iterable: `key=str.lower' Equivalent to:
     `sorted(iterable, key=key, reverse=True)[:n]'

     New in version 2.4.

     Changed in version 2.5: Added the optional _key_ argument.

 -- Function: heapq.nsmallest (n, iterable[, key])
     Return a list with the _n_ smallest elements from the dataset
     defined by _iterable_.  _key_, if provided, specifies a function
     of one argument that is used to extract a comparison key from each
     element in the iterable: `key=str.lower' Equivalent to:
     `sorted(iterable, key=key)[:n]'

     New in version 2.4.

     Changed in version 2.5: Added the optional _key_ argument.

  The latter two functions perform best for smaller values of _n_.  For
larger values, it is more efficient to use the sorted().
function.  Also, when `n==1', it is more efficient to use the built-in
min() and max().  functions.

5.8.4.1 Basic Examples
......................

A heapsort (http://en.wikipedia.org/wiki/Heapsort)can be implemented by
pushing all values onto a heap and then popping off the smallest values
one at a time:

    >>> def heapsort(iterable):
    ...     'Equivalent to sorted(iterable)'
    ...     h = []
    ...     for value in iterable:
    ...         heappush(h, value)
    ...     return [heappop(h) for i in range(len(h))]
    ...
    >>> heapsort([1, 3, 5, 7, 9, 2, 4, 6, 8, 0])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Heap elements can be tuples.  This is useful for assigning comparison
values (such as task priorities) alongside the main record being
tracked:

    >>> h = []
    >>> heappush(h, (5, 'write code'))
    >>> heappush(h, (7, 'release product'))
    >>> heappush(h, (1, 'write spec'))
    >>> heappush(h, (3, 'create tests'))
    >>> heappop(h)
    (1, 'write spec')

5.8.4.2 Priority Queue Implementation Notes
...........................................

A priority queue (http://en.wikipedia.org/wiki/Priority_queue)is common
use for a heap, and it presents several implementation challenges:

   * Sort stability:  how do you get two tasks with equal priorities to
     be returned in the order they were originally added?

   * In the future with Python 3, tuple comparison breaks for
     (priority, task) pairs if the priorities are equal and the tasks
     do not have a default comparison order.

   * If the priority of a task changes, how do you move it to a new
     position in the heap?

   * Or if a pending task needs to be deleted, how do you find it and
     remove it from the queue?

  A solution to the first two challenges is to store entries as
3-element list including the priority, an entry count, and the task.
The entry count serves as a tie-breaker so that two tasks with the same
priority are returned in the order they were added. And since no two
entry counts are the same, the tuple comparison will never attempt to
directly compare two tasks.

  The remaining challenges revolve around finding a pending task and
making changes to its priority or removing it entirely.  Finding a task
can be done with a dictionary pointing to an entry in the queue.

  Removing the entry or changing its priority is more difficult because
it would break the heap structure invariants.  So, a possible solution
is to mark an entry as invalid and optionally add a new entry with the
revised priority:

    pq = []                         # the priority queue list
    counter = itertools.count(1)    # unique sequence count
    task_finder = {}                # mapping of tasks to entries
    INVALID = 0                     # mark an entry as deleted

    def add_task(priority, task, count=None):
        if count is None:
            count = next(counter)
        entry = [priority, count, task]
        task_finder[task] = entry
        heappush(pq, entry)

    def get_top_priority():
        while True:
            priority, count, task = heappop(pq)
            del task_finder[task]
            if count is not INVALID:
                return task

    def delete_task(task):
        entry = task_finder[task]
        entry[1] = INVALID

    def reprioritize(priority, task):
        entry = task_finder[task]
        add_task(priority, task, entry[1])
        entry[1] = INVALID

5.8.4.3 Theory
..............

Heaps are arrays for which `a[k] <= a[2*k+1]' and `a[k] <= a[2*k+2]'
for all _k_, counting elements from 0.  For the sake of comparison,
non-existing elements are considered to be infinite.  The interesting
property of a heap is that `a[0]' is always its smallest element.

  The strange invariant above is meant to be an efficient memory
representation for a tournament.  The numbers below are _k_, not `a[k]':

                                   0

                  1                                 2

          3               4                5               6

      7       8       9       10      11      12      13      14

    15 16   17 18   19 20   21 22   23 24   25 26   27 28   29 30

In the tree above, each cell _k_ is topping `2*k+1' and `2*k+2'. In an
usual binary tournament we see in sports, each cell is the winner over
the two cells it tops, and we can trace the winner down the tree to see
all opponents s/he had.  However, in many computer applications of such
tournaments, we do not need to trace the history of a winner. To be
more memory efficient, when a winner is promoted, we try to replace it
by something else at a lower level, and the rule becomes that a cell
and the two cells it tops contain three different items, but the top
cell "wins" over the two topped cells.

  If this heap invariant is protected at all time, index 0 is clearly
the overall winner.  The simplest algorithmic way to remove it and find
the "next" winner is to move some loser (let's say cell 30 in the
diagram above) into the 0 position, and then percolate this new 0 down
the tree, exchanging values, until the invariant is re-established.
This is clearly logarithmic on the total number of items in the tree.
By iterating over all items, you get an O(n log n) sort.

  A nice feature of this sort is that you can efficiently insert new
items while the sort is going on, provided that the inserted items are
not "better" than the last 0'th element you extracted.  This is
especially useful in simulation contexts, where the tree holds all
incoming events, and the "win" condition means the smallest scheduled
time.  When an event schedule other events for execution, they are
scheduled into the future, so they can easily go into the heap.  So, a
heap is a good structure for implementing schedulers (this is what I
used for my MIDI sequencer :-).

  Various structures for implementing schedulers have been extensively
studied, and heaps are good for this, as they are reasonably speedy,
the speed is almost constant, and the worst case is not much different
than the average case.  However, there are other representations which
are more efficient overall, yet the worst cases might be terrible.

  Heaps are also very useful in big disk sorts.  You most probably all
know that a big sort implies producing "runs" (which are pre-sorted
sequences, which size is usually related to the amount of CPU memory),
followed by a merging passes for these runs, which merging is often
very cleverly organised (1). It is very important that the initial sort
produces the longest runs possible.  Tournaments are a good way to
that.  If, using all the memory available to hold a tournament, you
replace and percolate items that happen to fit the current run, you'll
produce runs which are twice the size of the memory for random input,
and much better for input fuzzily ordered.

  Moreover, if you output the 0'th item on disk and get an input which
may not fit in the current tournament (because the value "wins" over
the last output value), it cannot fit in the heap, so the size of the
heap decreases.  The freed memory could be cleverly reused immediately
for progressively building a second heap, which grows at exactly the
same rate the first heap is melting.  When the first heap completely
vanishes, you switch heaps and start a new run.  Clever and quite
effective!

  In a word, heaps are useful memory structures to know.  I use them in
a few applications, and I think it is good to keep a 'heap' module
around. :-)

Footnotes
.........

     (1) The disk balancing algorithms which are current, nowadays, are
     more annoying than clever, and this is a consequence of the
     seeking capabilities of the disks.  On devices which cannot seek,
     like big tape drives, the story was quite different, and one had
     to be very clever to ensure (far in advance) that each tape
     movement will be the most effective possible (that is, will best
     participate at "progressing" the merge).  Some tapes were even
     able to read backwards, and this was also used to avoid the
     rewinding time. Believe me, real good tape sorts were quite
     spectacular to watch! From all times, sorting has always been a
     Great Art! :-)

5.8.5 `bisect' -- Array bisection algorithm
-------------------------------------------

This module provides support for maintaining a list in sorted order
without having to sort the list after each insertion.  For long lists
of items with expensive comparison operations, this can be an
improvement over the more common approach.  The module is called
bisect because it uses a basic bisection algorithm to do its work.
The source code may be most useful as a working example of the
algorithm (the boundary conditions are already right!).

  New in version 2.1.

  Latest version of the bisect module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/bisect.py?view=markup)The
following functions are provided:

 -- Function: bisect.bisect_left (a, x, lo=0, hi=len(a))
     Locate the insertion point for _x_ in _a_ to maintain sorted order.
     The parameters _lo_ and _hi_ may be used to specify a subset of
     the list which should be considered; by default the entire list is
     used.  If _x_ is already present in _a_, the insertion point will
     be before (to the left of) any existing entries.  The return value
     is suitable for use as the first parameter to `list.insert()'
     assuming that _a_ is already sorted.

     The returned insertion point _i_ partitions the array _a_ into two
     halves so that `all(val < x for val in a[lo:i])' for the left side
     and `all(val >= x for val in a[i:hi])' for the right side.

 -- Function: bisect.bisect_right (a, x, lo=0, hi=len(a))
 -- Function: bisect.bisect (a, x, lo=0, hi=len(a))
     Similar to bisect_left(), but returns an insertion
     point which comes after (to the right of) any existing entries of
     _x_ in _a_.

     The returned insertion point _i_ partitions the array _a_ into two
     halves so that `all(val <= x for val in a[lo:i])' for the left
     side and `all(val > x for val in a[i:hi])' for the right side.

 -- Function: bisect.insort_left (a, x, lo=0, hi=len(a))
     Insert _x_ in _a_ in sorted order.  This is equivalent to
     `a.insert(bisect.bisect_left(a, x, lo, hi), x)' assuming that _a_
     is already sorted.  Keep in mind that the O(log n) search is
     dominated by the slow O(n) insertion step.

 -- Function: bisect.insort_right (a, x, lo=0, hi=len(a))
 -- Function: bisect.insort (a, x, lo=0, hi=len(a))
     Similar to insort_left(), but inserting _x_ in _a_
     after any existing entries of _x_.

  SortedCollection recipe
(http://code.activestate.com/recipes/577197-sortedcollection/)that uses
bisect to build a full-featured collection class with straight-forward
search methods and support for a key-function.  The keys are
precomputed to save unnecessary calls to the key function during
searches.

5.8.5.1 Searching Sorted Lists
..............................

The above bisect() functions are useful for finding
insertion points but can be tricky or awkward to use for common
searching tasks. The following five functions show how to transform
them into the standard lookups for sorted lists:

    def index(a, x):
        'Locate the leftmost value exactly equal to x'
        i = bisect_left(a, x)
        if i != len(a) and a[i] == x:
            return i
        raise ValueError

    def find_lt(a, x):
        'Find rightmost value less than x'
        i = bisect_left(a, x)
        if i:
            return a[i-1]
        raise ValueError

    def find_le(a, x):
        'Find rightmost value less than or equal to x'
        i = bisect_right(a, x)
        if i:
            return a[i-1]
        raise ValueError

    def find_gt(a, x):
        'Find leftmost value greater than x'
        i = bisect_right(a, x)
        if i != len(a):
            return a[i]
        raise ValueError

    def find_ge(a, x):
        'Find leftmost item greater than or equal to x'
        i = bisect_left(a, x)
        if i != len(a):
            return a[i]
        raise ValueError

5.8.5.2 Other Examples
......................

The bisect() function can be useful for numeric table
lookups. This example uses bisect() to look up a letter
grade for an exam score (say) based on a set of ordered numeric
breakpoints: 90 and up is an 'A', 80 to 89 is a 'B', and so on:

    >>> def grade(score, breakpoints=[60, 70, 80, 90], grades='FDCBA'):
    ...     i = bisect(breakpoints, score)
    ...     return grades[i]
    ...
    >>> [grade(score) for score in [33, 99, 77, 70, 89, 90, 100]]
    ['F', 'A', 'C', 'C', 'B', 'A', 'A']

Unlike the sorted() function, it does not make sense for
the bisect().  functions to have _key_ or _reversed_
arguments because that would lead to an inefficient design (successive
calls to bisect functions would not "remember" all of the previous key
lookups).

  Instead, it is better to search a list of precomputed keys to find
the index of the record in question:

    >>> data = [('red', 5), ('blue', 1), ('yellow', 8), ('black', 0)]
    >>> data.sort(key=lambda r: r[1])
    >>> keys = [r[1] for r in data]         # precomputed list of keys
    >>> data[bisect_left(keys, 0)]
    ('black', 0)
    >>> data[bisect_left(keys, 1)]
    ('blue', 1)
    >>> data[bisect_left(keys, 5)]
    ('red', 5)
    >>> data[bisect_left(keys, 8)]
    ('yellow', 8)

5.8.6 `array' -- Efficient arrays of numeric values
---------------------------------------------------

This module defines an object type which can compactly represent an
array of basic values: characters, integers, floating point numbers.
Arrays are sequence types and behave very much like lists, except that
the type of objects stored in them is constrained.  The type is
specified at object creation time by using a _type code_, which is a
single character.  The following type codes are defined:

Type code       C Type               Python Type             Minimum size in bytes
-----------------------------------------------------------------------------------------
`'c''           char                 character               1
`'b''           signed char          int                     1
`'B''           unsigned char        int                     1
`'u''           Py_UNICODE           Unicode character       2 (see note)
`'h''           signed short         int                     2
`'H''           unsigned short       int                     2
`'i''           signed int           int                     2
`'I''           unsigned int         long                    2
`'l''           signed long          int                     4
`'L''           unsigned long        long                    4
`'f''           float                float                   4
`'d''           double               float                   8

     Note: The `'u'' typecode corresponds to Python's unicode
     character.  On narrow Unicode builds this is 2-bytes, on wide
     builds this is 4-bytes.

  The actual representation of values is determined by the machine
architecture (strictly speaking, by the C implementation).  The actual
size can be accessed through the `itemsize' attribute.  The values
stored  for `'L'' and `'I'' items will be represented as Python long
integers when retrieved, because Python's plain integer type cannot
represent the full range of C's unsigned (long) integers.

  The module defines the following type:

 -- Class: array.array (typecode[, initializer])
     A new array whose items are restricted by _typecode_, and
     initialized from the optional _initializer_ value, which must be a
     list, string, or iterable over elements of the appropriate type.

     Changed in version 2.4: Formerly, only lists or strings were
     accepted.

     If given a list or string, the initializer is passed to the new
     array's fromlist(), fromstring(), or
     fromunicode() method (see below) to add initial items to the
     array.  Otherwise, the iterable initializer is passed to the
     extend() method.

 -- Data: array.ArrayType
     Obsolete alias for array.

  Array objects support the ordinary sequence operations of indexing,
slicing, concatenation, and multiplication.  When using slice
assignment, the assigned value must be an array object with the same
type code; in all other cases, see TypeError is raised. Array
objects also implement the buffer interface, and may be used wherever
buffer objects are supported.

  The following data items and methods are also supported:

 -- Attribute: array.typecode
     The typecode character used to create the array.

 -- Attribute: array.itemsize
     The length in bytes of one array item in the internal
     representation.

 -- Method: array.append (x)
     Append a new item with value _x_ to the end of the array.

 -- Method: array.buffer_info ()
     Return a tuple `(address, length)' giving the current memory
     address and the length in elements of the buffer used to hold
     array's contents.  The size of the memory buffer in bytes can be
     computed as `array.buffer_info()[1] * array.itemsize'.  This is
     occasionally useful when working with low-level (and inherently
     unsafe) I/O interfaces that require memory addresses, such as
     certain `ioctl()' operations.  The returned numbers are valid as
     long as the array exists and no length-changing operations are
     applied to it.

          Note: When using array objects from code written in C or C++
          (the only way to effectively make use of this information),
          it makes more sense to use the buffer interface supported by
          array objects.  This method is maintained for backward
          compatibility and should be avoided in new code.  The buffer
          interface is documented in Buffers and Memoryview Objects.

 -- Method: array.byteswap ()
     "Byteswap" all items of the array.  This is only supported for
     values which are 1, 2, 4, or 8 bytes in size; for other types of
     values, see RuntimeError is raised.  It is useful when
     reading data from a file written on a machine with a different
     byte order.

 -- Method: array.count (x)
     Return the number of occurrences of _x_ in the array.

 -- Method: array.extend (iterable)
     Append items from _iterable_ to the end of the array.  If
     _iterable_ is another array, it must have _exactly_ the same type
     code; if not, see TypeError will be raised.  If _iterable_
     is not an array, it must be iterable and its elements must be the
     right type to be appended to the array.

     Changed in version 2.4: Formerly, the argument could only be
     another array.

 -- Method: array.fromfile (f, n)
     Read _n_ items (as machine values) from the file object _f_ and
     append them to the end of the array.  If less than _n_ items are
     available, see EOFError is raised, but the items that were
     available are still inserted into the array.  _f_ must be a real
     built-in file object; something else with a read().
     method won't do.

 -- Method: array.fromlist (list)
     Append items from the list.  This is equivalent to `for x in list:
     a.append(x)' except that if there is a type error, the array is
     unchanged.

 -- Method: array.fromstring (s)
     Appends items from the string, interpreting the string as an array
     of machine values (as if it had been read from a file using the
     fromfile() method).

 -- Method: array.fromunicode (s)
     Extends this array with data from the given unicode string.  The
     array must be a type `'u'' array; otherwise a ValueError
     is raised.  Use `array.fromstring(unicodestring.encode(enc))'
     to append Unicode data to an array of some other type.

 -- Method: array.index (x)
     Return the smallest _i_ such that _i_ is the index of the first
     occurrence of _x_ in the array.

 -- Method: array.insert (i, x)
     Insert a new item with value _x_ in the array before position _i_.
     Negative values are treated as being relative to the end of the
     array.

 -- Method: array.pop ([i])
     Removes the item with the index _i_ from the array and returns it.
     The optional argument defaults to `-1', so that by default the
     last item is removed and returned.

 -- Method: array.read (f, n)
     Deprecated since version 1.5.1: Use the fromfile().
     method.

     Read _n_ items (as machine values) from the file object _f_ and
     append them to the end of the array.  If less than _n_ items are
     available, see EOFError is raised, but the items that were
     available are still inserted into the array.  _f_ must be a real
     built-in file object; something else with a read().
     method won't do.

 -- Method: array.remove (x)
     Remove the first occurrence of _x_ from the array.

 -- Method: array.reverse ()
     Reverse the order of the items in the array.

 -- Method: array.tofile (f)
     Write all items (as machine values) to the file object _f_.

 -- Method: array.tolist ()
     Convert the array to an ordinary list with the same items.

 -- Method: array.tostring ()
     Convert the array to an array of machine values and return the
     string representation (the same sequence of bytes that would be
     written to a file by the tofile() method.)

 -- Method: array.tounicode ()
     Convert the array to a unicode string.  The array must be a type
     `'u'' array; otherwise a ValueError is raised. Use
     `array.tostring().decode(enc)' to obtain a unicode string from an
     array of some other type.

 -- Method: array.write (f)
     Deprecated since version 1.5.1: Use the tofile().
     method.

     Write all items (as machine values) to the file object _f_.

  When an array object is printed or converted to a string, it is
represented as `array(typecode, initializer)'.  The _initializer_ is
omitted if the array is empty, otherwise it is a string if the
_typecode_ is `'c'', otherwise it is a list of numbers.  The string is
guaranteed to be able to be converted back to an array with the same
type and value using eval(), so long as the array()
function has been imported using `from array import array'.
Examples:

    array('l')
    array('c', 'hello world')
    array('u', u'hello \u2641')
    array('l', [1, 2, 3, 4, 5])
    array('d', [1.0, 2.0, 3.14])

Module struct.
     Packing and unpacking of heterogeneous binary data.

Module xdrlib.
     Packing and unpacking of External Data Representation (XDR) data
     as used in some remote procedure call systems.

The Numerical Python Manual (http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm)
     The Numeric Python extension (NumPy) defines another array type;
     see <http://numpy.sourceforge.net/> for further information about
     Numerical Python.  (A PDF version of the NumPy manual is available
     at <http://numpy.sourceforge.net/numdoc/numdoc.pdf>).

5.8.7 `sets' -- Unordered collections of unique elements
--------------------------------------------------------

New in version 2.3.

  Deprecated since version 2.6: The built-in `set'/`frozenset' types
replace this module.

  The sets module provides classes for constructing and
manipulating unordered collections of unique elements.  Common uses
include membership testing, removing duplicates from a sequence, and
computing standard math operations on sets such as intersection, union,
difference, and symmetric difference.

  Like other collections, sets support `x in set', `len(set)', and `for
x in set'.  Being an unordered collection, sets do not record element
position or order of insertion.  Accordingly, sets do not support
indexing, slicing, or other sequence-like behavior.

  Most set applications use the Set class which provides
every set method except for __hash__(). For advanced
applications requiring a hash method, the ImmutableSet.
class adds a __hash__() method but omits methods which
alter the contents of the set. Both Set and
ImmutableSet.  derive from `BaseSet', an abstract class useful for
determining whether something is a set: `isinstance(obj, BaseSet)'.

  The set classes are implemented using dictionaries.  Accordingly, the
requirements for set elements are the same as those for dictionary
keys; namely, that the element defines both __eq__() and
__hash__(). As a result, sets cannot contain mutable
elements such as lists or dictionaries. However, they can contain
immutable collections such as tuples or instances of
ImmutableSet.  For convenience in implementing sets of sets, inner
sets are automatically converted to immutable form, for example,
`Set([Set(['dog'])])' is transformed to `Set([ImmutableSet(['dog'])])'.

 -- Class: sets.Set ([iterable])
     Constructs a new empty Set object.  If the optional
     _iterable_ parameter is supplied, updates the set with elements
     obtained from iteration.  All of the elements in _iterable_ should
     be immutable or be transformable to an immutable using the
     protocol described in section Protocol for automatic conversion to immutable.

 -- Class: sets.ImmutableSet ([iterable])
     Constructs a new empty ImmutableSet object.  If the
     optional _iterable_ parameter is supplied, updates the set with
     elements obtained from iteration.  All of the elements in
     _iterable_ should be immutable or be transformable to an immutable
     using the protocol described in section Protocol for automatic conversion to immutable.

     Because ImmutableSet objects provide a
     __hash__() method, they can be used as set elements or as
     dictionary keys.  See ImmutableSet.  objects do not have
     methods for adding or removing elements, so all of the elements
     must be known when the constructor is called.

5.8.7.1 Set Objects
...................

Instances of Set and ImmutableSet both provide
the following operations:

Operation                           Equivalent       Result
-------------------------------------------------------------------------------------------
`len(s)'                                             cardinality of set _s_
`x in s'                                             test _x_ for membership in _s_
`x not in s'                                         test _x_ for non-membership in _s_
`s.issubset(t)'                     `s <= t'         test whether every element in _s_ is
                                                     in _t_
`s.issuperset(t)'                   `s >= t'         test whether every element in _t_ is
                                                     in _s_
`s.union(t)'                        `s | t'          new set with elements from both _s_
                                                     and _t_
`s.intersection(t)'                 `s & t'          new set with elements common to _s_
                                                     and _t_
`s.difference(t)'                   `s - t'          new set with elements in _s_ but not
                                                     in _t_
`s.symmetric_difference(t)'         `s ^ t'          new set with elements in either _s_
                                                     or _t_ but not both
`s.copy()'                                           new set with a shallow copy of _s_

  Note, the non-operator versions of `union()', `intersection()',
`difference()', and `symmetric_difference()' will accept any iterable as
an argument. In contrast, their operator based counterparts require
their arguments to be sets.  This precludes error-prone constructions
like `Set('abc') & 'cbs'' in favor of the more readable
`Set('abc').intersection('cbs')'.

  Changed in version 2.3.1: Formerly all arguments were required to be
sets.

  In addition, both Set and ImmutableSet.
support set to set comparisons.  Two sets are equal if and only if
every element of each set is contained in the other (each is a subset
of the other). A set is less than another set if and only if the first
set is a proper subset of the second set (is a subset, but is not
equal). A set is greater than another set if and only if the first set
is a proper superset of the second set (is a superset, but is not
equal).

  The subset and equality comparisons do not generalize to a complete
ordering function.  For example, any two disjoint sets are not equal
and are not subsets of each other, so _all_ of the following return
`False':  `a<b', `a==b', or `a>b'. Accordingly, sets do not implement
the __cmp__() method.

  Since sets only define partial ordering (subset relationships), the
output of the `list.sort()' method is undefined for lists of sets.

  The following table lists operations available in ImmutableSet
but not found in Set.:

Operation         Result
-----------------------------------------------------
`hash(s)'         returns a hash value for _s_

  The following table lists operations available in Set but
not found in ImmutableSet.:

Operation                                  Equivalent        Result
---------------------------------------------------------------------------------------------------
`s.update(t)'                              _s_ |= _t_        return set _s_ with elements added
                                                             from _t_
`s.intersection_update(t)'                 _s_ &= _t_        return set _s_ keeping only elements
                                                             also found in _t_
`s.difference_update(t)'                   _s_ -= _t_        return set _s_ after removing
                                                             elements found in _t_
`s.symmetric_difference_update(t)'         _s_ ^= _t_        return set _s_ with elements from
                                                             _s_ or _t_ but not both
`s.add(x)'                                                   add element _x_ to set _s_
`s.remove(x)'                                                remove _x_ from set _s_; raises
                                                             KeyError if not present
`s.discard(x)'                                               removes _x_ from set _s_ if present
`s.pop()'                                                    remove and return an arbitrary
                                                             element from _s_; raises
                                                             KeyError if empty
`s.clear()'                                                  remove all elements from set _s_

  Note, the non-operator versions of `update()',
`intersection_update()', `difference_update()', and
`symmetric_difference_update()' will accept any iterable as an argument.

  Changed in version 2.3.1: Formerly all arguments were required to be
sets.

  Also note, the module also includes a `union_update()' method which
is an alias for `update()'.  The method is included for backwards
compatibility.  Programmers should prefer the `update()' method because
it is supported by the built-in set() and
frozenset() types.

5.8.7.2 Example
...............

    >>> from sets import Set
    >>> engineers = Set(['John', 'Jane', 'Jack', 'Janice'])
    >>> programmers = Set(['Jack', 'Sam', 'Susan', 'Janice'])
    >>> managers = Set(['Jane', 'Jack', 'Susan', 'Zack'])
    >>> employees = engineers | programmers | managers           # union
    >>> engineering_management = engineers & managers            # intersection
    >>> fulltime_management = managers - engineers - programmers # difference
    >>> engineers.add('Marvin')                                  # add element
    >>> print engineers # doctest: +SKIP
    Set(['Jane', 'Marvin', 'Janice', 'John', 'Jack'])
    >>> employees.issuperset(engineers)     # superset test
    False
    >>> employees.update(engineers)         # update from another set
    >>> employees.issuperset(engineers)
    True
    >>> for group in [engineers, programmers, managers, employees]: # doctest: +SKIP
    ...     group.discard('Susan')          # unconditionally remove element
    ...     print group
    ...
    Set(['Jane', 'Marvin', 'Janice', 'John', 'Jack'])
    Set(['Janice', 'Jack', 'Sam'])
    Set(['Jane', 'Zack', 'Jack'])
    Set(['Jack', 'Sam', 'Jane', 'Marvin', 'Janice', 'John', 'Zack'])

5.8.7.3 Protocol for automatic conversion to immutable
......................................................

Sets can only contain immutable elements.  For convenience, mutable
Set.  objects are automatically copied to an
ImmutableSet before being added as a set element.

  The mechanism is to always add a hashable element, or if
it is not hashable, the element is checked to see if it has an
`__as_immutable__()' method which returns an immutable equivalent.

  Since Set objects have a `__as_immutable__()' method
returning an instance of ImmutableSet, it is possible to
construct sets of sets.

  A similar mechanism is needed by the __contains__() and
`remove()' methods which need to hash an element to check for
membership in a set.  Those methods check an element for hashability
and, if not, check for a `__as_temporarily_immutable__()' method which
returns the element wrapped by a class that provides temporary methods
for __hash__(), __eq__(), and __ne__().

  The alternate mechanism spares the need to build a separate copy of
the original mutable object.

  Set objects implement the
`__as_temporarily_immutable__()' method which returns the Set
object wrapped by a new class `_TemporarilyImmutableSet'.

  The two mechanisms for adding hashability are normally invisible to
the user; however, a conflict can arise in a multi-threaded environment
where one thread is updating a set while another has temporarily
wrapped it in `_TemporarilyImmutableSet'.  In other words, sets of
mutable sets are not thread-safe.

5.8.7.4 Comparison to the built-in `set' types
..............................................

The built-in set and frozenset types were
designed based on lessons learned from the sets module.
The key differences are:

   * Set and ImmutableSet were renamed to
     set and frozenset.

   * There is no equivalent to `BaseSet'.  Instead, use `isinstance(x,
     (set, frozenset))'.

   * The hash algorithm for the built-ins performs significantly better
     (fewer collisions) for most datasets.

   * The built-in versions have more space efficient pickles.

   * The built-in versions do not have a `union_update()' method.
     Instead, use the `update()' method which is equivalent.

   * The built-in versions do not have a `_repr(sorted=True)' method.
     Instead, use the built-in repr() and sorted()
     functions: `repr(sorted(s))'.

   * The built-in version does not have a protocol for automatic
     conversion to immutable.  Many found this feature to be confusing
     and no one in the community reported having found real uses for it.

5.8.8 `sched' -- Event scheduler
--------------------------------

The sched module defines a class which implements a general
purpose event scheduler:

  Latest version of the sched module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/sched.py?view=markup)

 -- Class: sched.scheduler (timefunc, delayfunc)
     The scheduler class defines a generic interface to
     scheduling events.  It needs two functions to actually deal with
     the "outside world" -- _timefunc_ should be callable without
     arguments, and return  a number (the "time", in any units
     whatsoever).  The _delayfunc_ function should be callable with one
     argument, compatible with the output of _timefunc_, and should
     delay that many time units. _delayfunc_ will also be called with
     the argument `0' after each event is run to allow other threads an
     opportunity to run in multi-threaded applications.

  Example:

    >>> import sched, time
    >>> s = sched.scheduler(time.time, time.sleep)
    >>> def print_time(): print "From print_time", time.time()
    ...
    >>> def print_some_times():
    ...     print time.time()
    ...     s.enter(5, 1, print_time, ())
    ...     s.enter(10, 1, print_time, ())
    ...     s.run()
    ...     print time.time()
    ...
    >>> print_some_times()
    930343690.257
    From print_time 930343695.274
    From print_time 930343700.273
    930343700.276

In multi-threaded environments, the scheduler class has
limitations with respect to thread-safety, inability to insert a new
task before the one currently pending in a running scheduler, and
holding up the main thread until the event queue is empty.  Instead,
the preferred approach is to use the threading.Timer class
instead.

  Example:

    >>> import time
    >>> from threading import Timer
    >>> def print_time():
    ...     print "From print_time", time.time()
    ...
    >>> def print_some_times():
    ...     print time.time()
    ...     Timer(5, print_time, ()).start()
    ...     Timer(10, print_time, ()).start()
    ...     time.sleep(11)  # sleep while time-delay events execute
    ...     print time.time()
    ...
    >>> print_some_times()
    930343690.257
    From print_time 930343695.274
    From print_time 930343700.273
    930343701.301

5.8.8.1 Scheduler Objects
.........................

scheduler instances have the following methods and
attributes:

 -- Method: scheduler.enterabs (time, priority, action, argument)
     Schedule a new event. The _time_ argument should be a numeric type
     compatible with the return value of the _timefunc_ function passed
     to the constructor.  Events scheduled for the same _time_ will be
     executed in the order of their _priority_.

     Executing the event means executing `action(*argument)'.
     _argument_ must be a sequence holding the parameters for _action_.

     Return value is an event which may be used for later cancellation
     of the event (see cancel()).

 -- Method: scheduler.enter (delay, priority, action, argument)
     Schedule an event for _delay_ more time units. Other then the
     relative time, the other arguments, the effect and the return
     value are the same as those for enterabs().

 -- Method: scheduler.cancel (event)
     Remove the event from the queue. If _event_ is not an event
     currently in the queue, this method will raise a ValueError.

 -- Method: scheduler.empty ()
     Return true if the event queue is empty.

 -- Method: scheduler.run ()
     Run all scheduled events. This function will wait  (using the
     `delayfunc()' function passed to the constructor) for the next
     event, then execute it and so on until there are no more scheduled
     events.

     Either _action_ or _delayfunc_ can raise an exception.  In either
     case, the scheduler will maintain a consistent state and propagate
     the exception.  If an exception is raised by _action_, the event
     will not be attempted in future calls to run().

     If a sequence of events takes longer to run than the time
     available before the next event, the scheduler will simply fall
     behind.  No events will be dropped; the calling code is
     responsible for canceling  events which are no longer pertinent.

 -- Attribute: scheduler.queue
     Read-only attribute returning a list of upcoming events in the
     order they will be run.  Each event is shown as a named tuple
     with the following fields:  time, priority, action,
     argument.

     New in version 2.6.

5.8.9 `mutex' -- Mutual exclusion support
-----------------------------------------

Deprecated since version 2.6: The mutex module has been
removed in Python 3.0.

  The mutex module defines a class that allows
mutual-exclusion via acquiring and releasing locks. It does not require
(or imply) threading or multi-tasking, though it could be
useful for those purposes.

  The mutex module defines the following class:

 -- Class: mutex.mutex
     Create a new (unlocked) mutex.

     A mutex has two pieces of state -- a "locked" bit and a queue.
     When the mutex is not locked, the queue is empty. Otherwise, the
     queue contains zero or more `(function, argument)' pairs
     representing functions (or methods) waiting to acquire the lock.
     When the mutex is unlocked while the queue is not empty, the first
     queue entry is removed and its  `function(argument)' pair called,
     implying it now has the lock.

     Of course, no multi-threading is implied - hence the funny
     interface for lock(), where a function is called once
     the lock is acquired.

5.8.9.1 Mutex Objects
.....................

mutex objects have following methods:

 -- Method: mutex.test ()
     Check whether the mutex is locked.

 -- Method: mutex.testandset ()
     "Atomic" test-and-set, grab the lock if it is not set, and return
     `True', otherwise, return `False'.

 -- Method: mutex.lock (function, argument)
     Execute `function(argument)', unless the mutex is locked. In the
     case it is locked, place the function and argument on the queue.
     See unlock() for explanation of when
     `function(argument)' is executed in that case.

 -- Method: mutex.unlock ()
     Unlock the mutex if queue is empty, otherwise execute the first
     element in the queue.

5.8.10 `queue' -- A synchronized queue class
--------------------------------------------

     Note: The Queue module has been renamed to `queue' in
     Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

The Queue module implements multi-producer, multi-consumer
queues.  It is especially useful in threaded programming when
information must be exchanged safely between multiple threads.  The
Queue class in this module implements all the required
locking semantics.  It depends on the availability of thread support in
Python; see the threading.  module.

  Implements three types of queue whose only difference is the order
that the entries are retrieved.  In a FIFO queue, the first tasks added
are the first retrieved. In a LIFO queue, the most recently added entry
is the first retrieved (operating like a stack).  With a priority queue,
the entries are kept sorted (using the heapq module) and the
lowest valued entry is retrieved first.

  Latest version of the queue module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/queue.py?view=markup)The
Queue module defines the following classes and exceptions:

 -- Class: Queue.Queue (maxsize=0)
     Constructor for a FIFO queue.  _maxsize_ is an integer that sets
     the upperbound limit on the number of items that can be placed in
     the queue.  Insertion will block once this size has been reached,
     until queue items are consumed.  If _maxsize_ is less than or
     equal to zero, the queue size is infinite.

 -- Class: Queue.LifoQueue (maxsize=0)
     Constructor for a LIFO queue.  _maxsize_ is an integer that sets
     the upperbound limit on the number of items that can be placed in
     the queue.  Insertion will block once this size has been reached,
     until queue items are consumed.  If _maxsize_ is less than or
     equal to zero, the queue size is infinite.

     New in version 2.6.

 -- Class: Queue.PriorityQueue (maxsize=0)
     Constructor for a priority queue.  _maxsize_ is an integer that
     sets the upperbound limit on the number of items that can be
     placed in the queue.  Insertion will block once this size has been
     reached, until queue items are consumed.  If _maxsize_ is less
     than or equal to zero, the queue size is infinite.

     The lowest valued entries are retrieved first (the lowest valued
     entry is the one returned by `sorted(list(entries))[0]').  A
     typical pattern for entries is a tuple in the form:
     `(priority_number, data)'.

     New in version 2.6.

 -- Exception: Queue.Empty
     Exception raised when non-blocking `get()' (or `get_nowait()') is
     called on a Queue object which is empty.

 -- Exception: Queue.Full
     Exception raised when non-blocking `put()' (or `put_nowait()') is
     called on a Queue object which is full.

  collections.deque is an alternative implementation of
unbounded queues with fast atomic `append()' and `popleft()' operations
that do not require locking.

5.8.10.1 Queue Objects
......................

Queue objects (see Queue, LifoQueue, or
PriorityQueue)  provide the public methods described below.

 -- Method: Queue.qsize ()
     Return the approximate size of the queue.  Note, qsize() > 0
     doesn't guarantee that a subsequent get() will not block, nor will
     qsize() < maxsize guarantee that put() will not block.

 -- Method: Queue.empty ()
     Return `True' if the queue is empty, `False' otherwise.  If empty()
     returns `True' it doesn't guarantee that a subsequent call to put()
     will not block.  Similarly, if empty() returns `False' it doesn't
     guarantee that a subsequent call to get() will not block.

 -- Method: Queue.full ()
     Return `True' if the queue is full, `False' otherwise.  If full()
     returns `True' it doesn't guarantee that a subsequent call to get()
     will not block.  Similarly, if full() returns `False' it doesn't
     guarantee that a subsequent call to put() will not block.

 -- Method: Queue.put (item[, block[, timeout]])
     Put _item_ into the queue. If optional args _block_ is true and
     _timeout_ is None (the default), block if necessary until a free
     slot is available. If _timeout_ is a positive number, it blocks at
     most _timeout_ seconds and raises the Full exception
     if no free slot was available within that time.  Otherwise
     (_block_ is false), put an item on the queue if a free slot is
     immediately available, else raise the Full exception
     (_timeout_ is ignored in that case).

     New in version 2.3: The _timeout_ parameter.

 -- Method: Queue.put_nowait (item)
     Equivalent to `put(item, False)'.

 -- Method: Queue.get ([block[, timeout]])
     Remove and return an item from the queue. If optional args _block_
     is true and _timeout_ is None (the default), block if necessary
     until an item is available.  If _timeout_ is a positive number, it
     blocks at most _timeout_ seconds and raises the Empty.
     exception if no item was available within that time.  Otherwise
     (_block_ is false), return an item if one is immediately available,
     else raise the Empty exception (_timeout_ is ignored
     in that case).

     New in version 2.3: The _timeout_ parameter.

 -- Method: Queue.get_nowait ()
     Equivalent to `get(False)'.

  Two methods are offered to support tracking whether enqueued tasks
have been fully processed by daemon consumer threads.

 -- Method: Queue.task_done ()
     Indicate that a formerly enqueued task is complete.  Used by queue
     consumer threads.  For each get() used to fetch a
     task, a subsequent call to task_done() tells the queue
     that the processing on the task is complete.

     If a join() is currently blocking, it will resume when
     all items have been processed (meaning that a task_done()
     call was received for every item that had been put()
     into the queue).

     Raises a ValueError if called more times than there
     were items placed in the queue.

     New in version 2.5.

 -- Method: Queue.join ()
     Blocks until all items in the queue have been gotten and processed.

     The count of unfinished tasks goes up whenever an item is added to
     the queue.  The count goes down whenever a consumer thread calls
     task_done() to indicate that the item was retrieved
     and all work on it is complete. When the count of unfinished tasks
     drops to zero, see join() unblocks.

     New in version 2.5.

  Example of how to wait for enqueued tasks to be completed:

    def worker():
        while True:
            item = q.get()
            do_work(item)
            q.task_done()

    q = Queue()
    for i in range(num_worker_threads):
         t = Thread(target=worker)
         t.daemon = True
         t.start()

    for item in source():
        q.put(item)

    q.join()       # block until all tasks are done

5.8.11 `weakref' -- Weak references
-----------------------------------

New in version 2.1.

  The weakref module allows the Python programmer to create
_weak references_ to objects.

  In the following, the term _referent_ means the object which is
referred to by a weak reference.

  A weak reference to an object is not enough to keep the object alive:
when the only remaining references to a referent are weak references,
see garbage collection is free to destroy the referent and
reuse its memory for something else.  A primary use for weak references
is to implement caches or mappings holding large objects, where it's
desired that a large object not be kept alive solely because it appears
in a cache or mapping.

  For example, if you have a number of large binary image objects, you
may wish to associate a name with each.  If you used a Python
dictionary to map names to images, or images to names, the image
objects would remain alive just because they appeared as values or keys
in the dictionaries.  The WeakKeyDictionary and
WeakValueDictionary classes supplied by the weakref.
module are an alternative, using weak references to construct mappings
that don't keep objects alive solely because they appear in the mapping
objects.  If, for example, an image object is a value in a
WeakValueDictionary, then when the last remaining references to
that image object are the weak references held by weak mappings,
garbage collection can reclaim the object, and its corresponding
entries in weak mappings are simply deleted.

  WeakKeyDictionary and WeakValueDictionary use
weak references in their implementation, setting up callback functions
on the weak references that notify the weak dictionaries when a key or
value has been reclaimed by garbage collection.  Most programs should
find that using one of these weak dictionary types is all they need -
it's not usually necessary to create your own weak references directly.
The low-level machinery used by the weak dictionary implementations is
exposed by the weakref module for the benefit of advanced
uses.

     Note: Weak references to an object are cleared before the object's
     __del__().  is called, to ensure that the weak
     reference callback (if any) finds the object still alive.

  Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
methods (both bound and unbound), sets, frozensets, file objects, see
generator.s, type objects, `DBcursor' objects from the
bsddb module, sockets, arrays, deques, regular expression pattern
objects, and code objects.

  Changed in version 2.4: Added support for files, sockets, arrays, and
patterns.

  Changed in version 2.7: Added support for thread.lock,
threading.Lock, and code objects.

  Several built-in types such as list and dict.
do not directly support weak references but can add support through
subclassing:

    class Dict(dict):
        pass

    obj = Dict(red=1, green=2, blue=3)   # this object is weak referenceable

*CPython implementation detail:* Other built-in types such as
tuple and long do not support weak references even
when subclassed.

  Extension types can easily be made to support weak references; see
Weak Reference Support.

 -- Class: weakref.ref (object[, callback])
     Return a weak reference to _object_.  The original object can be
     retrieved by calling the reference object if the referent is still
     alive; if the referent is no longer alive, calling the reference
     object will cause None to be returned.  If _callback_
     is provided and not None, and the returned weakref
     object is still alive, the callback will be called when the object
     is about to be finalized; the weak reference object will be passed
     as the only parameter to the callback; the referent will no longer
     be available.

     It is allowable for many weak references to be constructed for the
     same object.  Callbacks registered for each weak reference will be
     called from the most recently registered callback to the oldest
     registered callback.

     Exceptions raised by the callback will be noted on the standard
     error output, but cannot be propagated; they are handled in
     exactly the same way as exceptions raised from an object's
     __del__() method.

     Weak references are hashable if the _object_ is
     hashable.  They will maintain their hash value even after the
     _object_ was deleted.  If hash() is called the first
     time only after the _object_ was deleted, the call will raise
     TypeError.

     Weak references support tests for equality, but not ordering.  If
     the referents are still alive, two references have the same
     equality relationship as their referents (regardless of the
     _callback_).  If either referent has been deleted, the references
     are equal only if the reference objects are the same object.

     Changed in version 2.4: This is now a subclassable type rather
     than a factory function; it derives from object.

 -- Function: weakref.proxy (object[, callback])
     Return a proxy to _object_ which uses a weak reference.  This
     supports use of the proxy in most contexts instead of requiring
     the explicit dereferencing used with weak reference objects.  The
     returned object will have a type of either `ProxyType' or
     `CallableProxyType', depending on whether _object_ is callable.
     Proxy objects are not hashable regardless of the
     referent; this avoids a number of problems related to their
     fundamentally mutable nature, and prevent their use as dictionary
     keys.  _callback_ is the same as the parameter of the same name to
     the ref() function.

 -- Function: weakref.getweakrefcount (object)
     Return the number of weak references and proxies which refer to
     _object_.

 -- Function: weakref.getweakrefs (object)
     Return a list of all weak reference and proxy objects which refer
     to _object_.

 -- Class: weakref.WeakKeyDictionary ([dict])
     Mapping class that references keys weakly.  Entries in the
     dictionary will be discarded when there is no longer a strong
     reference to the key.  This can be used to associate additional
     data with an object owned by other parts of an application without
     adding attributes to those objects.  This can be especially useful
     with objects that override attribute accesses.

          Note: Caution: Because a WeakKeyDictionary is
          built on top of a Python dictionary, it must not change size
          when iterating over it.  This can be difficult to ensure for
          a WeakKeyDictionary because actions performed by
          the program during iteration may cause items in the
          dictionary to vanish "by magic" (as a side effect of garbage
          collection).

  WeakKeyDictionary objects have the following additional
methods.  These expose the internal references directly.  The
references are not guaranteed to be "live" at the time they are used,
so the result of calling the references needs to be checked before
being used.  This can be used to avoid creating references that will
cause the garbage collector to keep the keys around longer than needed.

 -- Method: WeakKeyDictionary.iterkeyrefs ()
     Return an iterator that yields the weak references to
     the keys.

     New in version 2.5.

 -- Method: WeakKeyDictionary.keyrefs ()
     Return a list of weak references to the keys.

     New in version 2.5.

 -- Class: weakref.WeakValueDictionary ([dict])
     Mapping class that references values weakly.  Entries in the
     dictionary will be discarded when no strong reference to the value
     exists any more.

          Note: Caution:  Because a WeakValueDictionary is
          built on top of a Python dictionary, it must not change size
          when iterating over it.  This can be difficult to ensure for
          a WeakValueDictionary because actions performed
          by the program during iteration may cause items in the
          dictionary to vanish "by magic" (as a side effect of garbage
          collection).

  WeakValueDictionary objects have the following additional
methods.  These method have the same issues as the `iterkeyrefs()' and
`keyrefs()' methods of WeakKeyDictionary objects.

 -- Method: WeakValueDictionary.itervaluerefs ()
     Return an iterator that yields the weak references to
     the values.

     New in version 2.5.

 -- Method: WeakValueDictionary.valuerefs ()
     Return a list of weak references to the values.

     New in version 2.5.

 -- Class: weakref.WeakSet ([elements])
     Set class that keeps weak references to its elements.  An element
     will be discarded when no strong reference to it exists any more.

     New in version 2.7.

 -- Data: weakref.ReferenceType
     The type object for weak references objects.

 -- Data: weakref.ProxyType
     The type object for proxies of objects which are not callable.

 -- Data: weakref.CallableProxyType
     The type object for proxies of callable objects.

 -- Data: weakref.ProxyTypes
     Sequence containing all the type objects for proxies.  This can
     make it simpler to test if an object is a proxy without being
     dependent on naming both proxy types.

 -- Exception: weakref.ReferenceError
     Exception raised when a proxy object is used but the underlying
     object has been collected.  This is the same as the standard
     ReferenceError exception.

PEP 0205 (http://www.python.org/dev/peps/pep-0205) - Weak References
     The proposal and rationale for this feature, including links to
     earlier implementations and information about similar features in
     other languages.

5.8.11.1 Weak Reference Objects
...............................

Weak reference objects have no attributes or methods, but do allow the
referent to be obtained, if it still exists, by calling it:

    >>> import weakref
    >>> class Object:
    ...     pass
    ...
    >>> o = Object()
    >>> r = weakref.ref(o)
    >>> o2 = r()
    >>> o is o2
    True

If the referent no longer exists, calling the reference object returns
None.:

    >>> del o, o2
    >>> print r()
    None

Testing that a weak reference object is still live should be done using
the expression `ref() is not None'.  Normally, application code that
needs to use a reference object should follow this pattern:

    # r is a weak reference object
    o = r()
    if o is None:
        # referent has been garbage collected
        print "Object has been deallocated; can't frobnicate."
    else:
        print "Object is still live!"
        o.do_something_useful()

Using a separate test for "liveness" creates race conditions in threaded
applications; another thread can cause a weak reference to become
invalidated before the weak reference is called; the idiom shown above
is safe in threaded applications as well as single-threaded
applications.

  Specialized versions of ref objects can be created
through subclassing.  This is used in the implementation of the
WeakValueDictionary to reduce the memory overhead for each entry
in the mapping.  This may be most useful to associate additional
information with a reference, but could also be used to insert
additional processing on calls to retrieve the referent.

  This example shows how a subclass of ref can be used to
store additional information about an object and affect the value
that's returned when the referent is accessed:

    import weakref

    class ExtendedRef(weakref.ref):
        def __init__(self, ob, callback=None, **annotations):
            super(ExtendedRef, self).__init__(ob, callback)
            self.__counter = 0
            for k, v in annotations.iteritems():
                setattr(self, k, v)

        def __call__(self):
            """Return a pair containing the referent and the number of
            times the reference has been called.
            """
            ob = super(ExtendedRef, self).__call__()
            if ob is not None:
                self.__counter += 1
                ob = (ob, self.__counter)
            return ob

5.8.11.2 Example
................

This simple example shows how an application can use objects IDs to
retrieve objects that it has seen before.  The IDs of the objects can
then be used in other data structures without forcing the objects to
remain alive, but the objects can still be retrieved by ID if they do.

    import weakref

    _id2obj_dict = weakref.WeakValueDictionary()

    def remember(obj):
        oid = id(obj)
        _id2obj_dict[oid] = obj
        return oid

    def id2obj(oid):
        return _id2obj_dict[oid]

5.8.12 `UserDict' -- Class wrapper for dictionary objects
---------------------------------------------------------

The module defines a mixin,  see DictMixin, defining all
dictionary methods for classes that already have a minimum mapping
interface.  This greatly simplifies writing classes that need to be
substitutable for dictionaries (such as the shelve module).

  This module also defines a class, see UserDict, that acts as a
wrapper around dictionary objects.  The need for this class has been
largely supplanted by the ability to subclass directly from dict.
(a feature that became available starting with Python version
2.2).  Prior to the introduction of dict, the
UserDict class was used to create dictionary-like sub-classes
that obtained new behaviors by overriding existing methods or adding
new ones.

  Latest version of the UserDict Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/UserDict.py?view=markup)The
UserDict module defines the UserDict class and
DictMixin.:

 -- Class: UserDict.UserDict ([initialdata])
     Class that simulates a dictionary.  The instance's contents are
     kept in a regular dictionary, which is accessible via the `data'
     attribute of UserDict instances.  If _initialdata_ is
     provided, `data' is initialized with its contents; note that a
     reference to _initialdata_ will not be kept, allowing it be used
     for other purposes.

          Note: For backward compatibility, instances of
          UserDict are not iterable.

 -- Class: UserDict.IterableUserDict ([initialdata])
     Subclass of UserDict that supports direct iteration
     (e.g.  `for key in myDict').

  In addition to supporting the methods and operations of mappings (see
section Mapping Types -- dict), see UserDict and
IterableUserDict instances provide the following attribute:

 -- Attribute: IterableUserDict.data
     A real dictionary used to store the contents of the
     UserDict class.

 -- Class: UserDict.DictMixin
     Mixin defining all dictionary methods for classes that already
     have a minimum dictionary interface including __getitem__(),
     see __setitem__(), __delitem__(), and
     `keys()'.

     This mixin should be used as a superclass.  Adding each of the
     above methods adds progressively more functionality.  For
     instance, defining all but __delitem__() will preclude
     only `pop()' and `popitem()' from the full interface.

     In addition to the four base methods, progressively more
     efficiency comes with defining __contains__(),
     __iter__(), and `iteritems()'.

     Since the mixin has no knowledge of the subclass constructor, it
     does not define __init__() or copy().

     Starting with Python version 2.6, it is recommended to use
     `collections.MutableMapping' instead of DictMixin.

5.8.13 `UserList' -- Class wrapper for list objects
---------------------------------------------------

     Note: This module is available for backward compatibility only.
     If you are writing code that does not need to work with versions
     of Python earlier than Python 2.2, please consider subclassing
     directly from the built-in list type.

This module defines a class that acts as a wrapper around list objects.
It is a useful base class for your own list-like classes, which can
inherit from them and override existing methods or add new ones.  In
this way one can add new behaviors to lists.

  The UserList module defines the UserList.
class:

 -- Class: UserList.UserList ([list])
     Class that simulates a list.  The instance's contents are kept in
     a regular list, which is accessible via the data.
     attribute of UserList.  instances.  The instance's
     contents are initially set to a copy of _list_, defaulting to the
     empty list `[]'.  _list_ can be any iterable, e.g. a real Python
     list or a UserList object.

          Note: The UserList class has been moved to the
          collections.  module in Python 3.0. The 2to3
          tool will automatically adapt imports when converting
          your sources to 3.0.

  In addition to supporting the methods and operations of mutable
sequences (see section Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange),
see UserList instances
provide the following attribute:

 -- Attribute: UserList.data
     A real Python list object used to store the contents of the
     UserList.  class.

  *Subclassing requirements:* Subclasses of UserList are
expect to offer a constructor which can be called with either no
arguments or one argument.  List operations which return a new sequence
attempt to create an instance of the actual implementation class.  To
do so, it assumes that the constructor can be called with a single
parameter, which is a sequence object used as a data source.

  If a derived class does not wish to comply with this requirement, all
of the special methods supported by this class will need to be
overridden; please consult the sources for information about the
methods which need to be provided in that case.

  Changed in version 2.0: Python versions 1.5.2 and 1.6 also required
that the constructor be callable with no parameters, and offer a
mutable `data' attribute.  Earlier versions of Python did not attempt
to create instances of the derived class.

5.8.14 `UserString' -- Class wrapper for string objects
-------------------------------------------------------

     Note: This UserString class from this module is
     available for backward compatibility only.  If you are writing
     code that does not need to work with versions of Python earlier
     than Python 2.2, please consider subclassing directly from the
     built-in str type instead of using UserString.
     (there is no built-in equivalent to MutableString).

This module defines a class that acts as a wrapper around string
objects.  It is a useful base class for your own string-like classes,
which can inherit from them and override existing methods or add new
ones.  In this way one can add new behaviors to strings.

  It should be noted that these classes are highly inefficient compared
to real string or Unicode objects; this is especially the case for
MutableString.

  The UserString module defines the following classes:

 -- Class: UserString.UserString ([sequence])
     Class that simulates a string or a Unicode string object.  The
     instance's content is kept in a regular string or Unicode string
     object, which is accessible via the `data' attribute of
     UserString instances.  The instance's contents are initially
     set to a copy of _sequence_.  _sequence_ can be either a regular
     Python string or Unicode string, an instance of UserString.
     (or a subclass) or an arbitrary sequence which can be
     converted into a string using the built-in str().
     function.

          Note: The UserString class has been moved to the
          collections.  module in Python 3.0. The 2to3
          tool will automatically adapt imports when converting
          your sources to 3.0.

 -- Class: UserString.MutableString ([sequence])
     This class is derived from the UserString above and
     redefines strings to be _mutable_.  Mutable strings can't be used
     as dictionary keys, because dictionaries require _immutable_
     objects as keys.  The main intention of this class is to serve as
     an educational example for inheritance and necessity to remove
     (override) the __hash__() method in order to trap
     attempts to use a mutable object as dictionary key, which would be
     otherwise very error prone and hard to track down.

     Deprecated since version 2.6: The MutableString class
     has been removed in Python 3.0.

  In addition to supporting the methods and operations of string and
Unicode objects (see section String Methods), see
UserString instances provide the following attribute:

 -- Attribute: MutableString.data
     A real Python string or Unicode object used to store the content
     of the UserString class.

5.8.15 `types' -- Names for built-in types
------------------------------------------

This module defines names for some object types that are used by the
standard Python interpreter, but not for the types defined by various
extension modules.  Also, it does not include some of the types that
arise during processing such as the `listiterator' type. It is safe to
use `from types import *' -- the module does not export any names
besides the ones listed here. New names exported by future versions of
this module will all end in `Type'.

  Typical use is for functions that do different things depending on
their argument types, like the following:

    from types import *
    def delete(mylist, item):
        if type(item) is IntType:
           del mylist[item]
        else:
           mylist.remove(item)

Starting in Python 2.2, built-in factory functions such as int()
and str() are also names for the corresponding types.
This is now the preferred way to access the type instead of using the
types module.  Accordingly, the example above should be
written as follows:

    def delete(mylist, item):
        if isinstance(item, int):
           del mylist[item]
        else:
           mylist.remove(item)

The module defines the following names:

 -- Data: types.NoneType
     The type of `None'.

 -- Data: types.TypeType
     The type of type objects (such as returned by type());
     alias of the built-in type.

 -- Data: types.BooleanType
     The type of the bool values `True' and `False'; alias
     of the built-in bool.

     New in version 2.3.

 -- Data: types.IntType
     The type of integers (e.g. `1'); alias of the built-in int.

 -- Data: types.LongType
     The type of long integers (e.g. `1L'); alias of the built-in
     long.

 -- Data: types.FloatType
     The type of floating point numbers (e.g. `1.0'); alias of the
     built-in float.

 -- Data: types.ComplexType
     The type of complex numbers (e.g. `1.0j').  This is not defined if
     Python was built without complex number support.

 -- Data: types.StringType
     The type of character strings (e.g. `'Spam''); alias of the
     built-in str.

 -- Data: types.UnicodeType
     The type of Unicode character strings (e.g. `u'Spam'').  This is
     not defined if Python was built without Unicode support.  It's an
     alias of the built-in unicode.

 -- Data: types.TupleType
     The type of tuples (e.g. `(1, 2, 3, 'Spam')'); alias of the
     built-in tuple.

 -- Data: types.ListType
     The type of lists (e.g. `[0, 1, 2, 3]'); alias of the built-in
     list.

 -- Data: types.DictType
     The type of dictionaries (e.g. `{'Bacon': 1, 'Ham': 0}'); alias of
     the built-in dict.

 -- Data: types.DictionaryType
     An alternate name for `DictType'.

 -- Data: types.FunctionType
 -- Data: types.LambdaType
     The type of user-defined functions and functions created by
     lambda.  expressions.

 -- Data: types.GeneratorType
     The type of generator.-iterator objects, produced by
     calling a generator function.

     New in version 2.2.

 -- Data: types.CodeType
     The type for code objects such as returned by compile().

 -- Data: types.ClassType
     The type of user-defined old-style classes.

 -- Data: types.InstanceType
     The type of instances of user-defined classes.

 -- Data: types.MethodType
     The type of methods of user-defined class instances.

 -- Data: types.UnboundMethodType
     An alternate name for `MethodType'.

 -- Data: types.BuiltinFunctionType
 -- Data: types.BuiltinMethodType
     The type of built-in functions like len() or
     sys.exit(), and methods of built-in classes.  (Here, the term
     "built-in" means "written in C".)

 -- Data: types.ModuleType
     The type of modules.

 -- Data: types.FileType
     The type of open file objects such as `sys.stdout'; alias of the
     built-in file.

 -- Data: types.XRangeType
     The type of range objects returned by xrange().; alias
     of the built-in xrange.

 -- Data: types.SliceType
     The type of objects returned by slice().; alias of the
     built-in slice.

 -- Data: types.EllipsisType
     The type of `Ellipsis'.

 -- Data: types.TracebackType
     The type of traceback objects such as found in `sys.exc_traceback'.

 -- Data: types.FrameType
     The type of frame objects such as found in `tb.tb_frame' if `tb'
     is a traceback object.

 -- Data: types.BufferType
     The type of buffer objects created by the buffer().
     function.

 -- Data: types.DictProxyType
     The type of dict proxies, such as `TypeType.__dict__'.

 -- Data: types.NotImplementedType
     The type of `NotImplemented'

 -- Data: types.GetSetDescriptorType
     The type of objects defined in extension modules with
     `PyGetSetDef', such as `FrameType.f_locals' or
     `array.array.typecode'.  This type is used as descriptor for
     object attributes; it has the same purpose as the property
     type, but for classes defined in extension modules.

     New in version 2.5.

 -- Data: types.MemberDescriptorType
     The type of objects defined in extension modules with
     `PyMemberDef', such as `datetime.timedelta.days'.  This type is
     used as descriptor for simple C data members which use standard
     conversion functions; it has the same purpose as the
     property type, but for classes defined in extension modules.

     *CPython implementation detail:* In other implementations of
     Python, this type may be identical to `GetSetDescriptorType'.

     New in version 2.5.

 -- Data: types.StringTypes
     A sequence containing `StringType' and `UnicodeType' used to
     facilitate easier checking for any string object.  Using this is
     more portable than using a sequence of the two string types
     constructed elsewhere since it only contains `UnicodeType' if it
     has been built in the running version of Python.  For example:
     `isinstance(s, types.StringTypes)'.

     New in version 2.2.

5.8.16 `new' -- Creation of runtime internal objects
----------------------------------------------------

Deprecated since version 2.6: The new module has been
removed in Python 3.0.  Use the types.  module's classes
instead.

  The new module allows an interface to the interpreter
object creation functions. This is for use primarily in marshal-type
functions, when a new object needs to be created "magically" and not by
using the regular creation functions. This module provides a low-level
interface to the interpreter, so care must be exercised when using this
module. It is possible to supply non-sensical arguments which crash the
interpreter when the object is used.

  The new module defines the following functions:

 -- Function: new.instance (class[, dict])
     This function creates an instance of _class_ with dictionary
     _dict_ without calling the __init__() constructor.  If
     _dict_ is omitted or `None', a new, empty dictionary is created
     for the new instance.  Note that there are no guarantees that the
     object will be in a consistent state.

 -- Function: new.instancemethod (function, instance, class)
     This function will return a method object, bound to _instance_, or
     unbound if _instance_ is `None'.  _function_ must be callable.

 -- Function: new.function (code, globals[, name[, argdefs[, closure]]])
     Returns a (Python) function with the given code and globals. If
     _name_ is given, it must be a string or `None'.  If it is a
     string, the function will have the given name, otherwise the
     function name will be taken from `code.co_name'.  If _argdefs_ is
     given, it must be a tuple and will be used to determine the default
     values of parameters.  If _closure_ is given, it must be `None' or
     a tuple of cell objects containing objects to bind to the names in
     `code.co_freevars'.

 -- Function: new.code (argcount, nlocals, stacksize, flags,
          codestring, constants, names, varnames, filename, name,
          firstlineno, lnotab)
     This function is an interface to the PyCode_New(). C
     function.

 -- Function: new.module (name[, doc])
     This function returns a new module object with name _name_. _name_
     must be a string. The optional _doc_ argument can have any type.

 -- Function: new.classobj (name, baseclasses, dict)
     This function returns a new class object, with name _name_,
     derived from _baseclasses_ (which should be a tuple of classes)
     and with namespace _dict_.

5.8.17 `copy' -- Shallow and deep copy operations
-------------------------------------------------

This module provides generic (shallow and deep) copying operations.

  Interface summary:

 -- Function: copy.copy (x)
     Return a shallow copy of _x_.

 -- Function: copy.deepcopy (x)
     Return a deep copy of _x_.

 -- Exception: copy.error
     Raised for module specific errors.

  The difference between shallow and deep copying is only relevant for
compound objects (objects that contain other objects, like lists or
class instances):

   * A _shallow copy_ constructs a new compound object and then (to the
     extent possible) inserts _references_ into it to the objects found
     in the original.

   * A _deep copy_ constructs a new compound object and then,
     recursively, inserts _copies_ into it of the objects found in the
     original.

  Two problems often exist with deep copy operations that don't exist
with shallow copy operations:

   * Recursive objects (compound objects that, directly or indirectly,
     contain a reference to themselves) may cause a recursive loop.

   * Because deep copy copies _everything_ it may copy too much, e.g.,
     administrative data structures that should be shared even between
     copies.

  The deepcopy() function avoids these problems by:

   * keeping a "memo" dictionary of objects already copied during the
     current copying pass; and

   * letting user-defined classes override the copying operation or the
     set of components copied.

  This module does not copy types like module, method, stack trace,
stack frame, file, socket, window, array, or any similar types.  It
does "copy" functions and classes (shallow and deeply), by returning
the original object unchanged; this is compatible with the way these
are treated by the pickle module.

  Shallow copies of dictionaries can be made using dict.copy(),
and of lists by assigning a slice of the entire list, for example,
`copied_list = original_list[:]'.

  Changed in version 2.5: Added copying functions.

  Classes can use the same interfaces to control copying that they use
to control pickling.  See the description of module pickle.
for information on these methods.  The copy module does not
use the copy_reg registration module.

  In order for a class to define its own copy implementation, it can
define special methods `__copy__()' and `__deepcopy__()'.  The former
is called to implement the shallow copy operation; no additional
arguments are passed.  The latter is called to implement the deep copy
operation; it is passed one argument, the memo dictionary.  If the
`__deepcopy__()' implementation needs to make a deep copy of a
component, it should call the deepcopy() function with the
component as first argument and the memo dictionary as second argument.

Module pickle.
     Discussion of the special methods used to support object state
     retrieval and restoration.

5.8.18 `pprint' -- Data pretty printer
--------------------------------------

The pprint module provides a capability to "pretty-print"
arbitrary Python data structures in a form which can be used as input
to the interpreter.  If the formatted structures include objects which
are not fundamental Python types, the representation may not be
loadable.  This may be the case if objects such as files, sockets,
classes, or instances are included, as well as many other built-in
objects which are not representable as Python constants.

  The formatted representation keeps objects on a single line if it
can, and breaks them onto multiple lines if they don't fit within the
allowed width.  Construct PrettyPrinter objects explicitly
if you need to adjust the width constraint.

  Changed in version 2.5: Dictionaries are sorted by key before the
display is computed; before 2.5, a dictionary was sorted only if its
display required more than one line, although that wasn't documented.

  Changed in version 2.6: Added support for set and
frozenset.

  Latest version of the pprint module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/pprint.py?view=markup)The
pprint module defines one class:

 -- Class: pprint.PrettyPrinter (...)
     Construct a PrettyPrinter instance.  This constructor
     understands several keyword parameters.  An output stream may be
     set using the _stream_ keyword; the only method used on the stream
     object is the file protocol's `write()' method.  If not specified,
     the PrettyPrinter adopts `sys.stdout'.  Three
     additional parameters may be used to control the formatted
     representation.  The keywords are _indent_, _depth_, and _width_.
     The amount of indentation added for each recursive level is
     specified by _indent_; the default is one.  Other values can cause
     output to look a little odd, but can make nesting easier to spot.
     The number of levels which may be printed is controlled by
     _depth_; if the data structure being printed is too deep, the next
     contained level is replaced by `...'.  By default, there is no
     constraint on the depth of the objects being formatted.  The
     desired output width is constrained using the _width_ parameter;
     the default is 80 characters.  If a structure cannot be formatted
     within the constrained width, a best effort will be made.

         >>> import pprint
         >>> stuff = ['spam', 'eggs', 'lumberjack', 'knights', 'ni']
         >>> stuff.insert(0, stuff[:])
         >>> pp = pprint.PrettyPrinter(indent=4)
         >>> pp.pprint(stuff)
         [   ['spam', 'eggs', 'lumberjack', 'knights', 'ni'],
             'spam',
             'eggs',
             'lumberjack',
             'knights',
             'ni']
         >>> tup = ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead',
         ... ('parrot', ('fresh fruit',))))))))
         >>> pp = pprint.PrettyPrinter(depth=6)
         >>> pp.pprint(tup)
         ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead', (...)))))))

  The PrettyPrinter class supports several derivative
functions:

 -- Function: pprint.pformat (object[, indent[, width[, depth]]])
     Return the formatted representation of _object_ as a string.
     _indent_, _width_ and _depth_ will be passed to the
     PrettyPrinter constructor as formatting parameters.

     Changed in version 2.4: The parameters _indent_, _width_ and
     _depth_ were added.

 -- Function: pprint.pprint (object[, stream[, indent[, width[,
          depth]]]])
     Prints the formatted representation of _object_ on _stream_,
     followed by a newline.  If _stream_ is omitted, `sys.stdout' is
     used.  This may be used in the interactive interpreter instead of
     a print statement for inspecting values.    _indent_,
     _width_ and _depth_ will be passed to the PrettyPrinter
     constructor as formatting parameters.

         >>> import pprint
         >>> stuff = ['spam', 'eggs', 'lumberjack', 'knights', 'ni']
         >>> stuff.insert(0, stuff)
         >>> pprint.pprint(stuff)
         [<Recursion on list with id=...>,
          'spam',
          'eggs',
          'lumberjack',
          'knights',
          'ni']

     Changed in version 2.4: The parameters _indent_, _width_ and
     _depth_ were added.

 -- Function: pprint.isreadable (object)
     Determine if the formatted representation of _object_ is
     "readable," or can be used to reconstruct the value using
     eval().  This always returns `False' for recursive objects.

         >>> pprint.isreadable(stuff)
         False

 -- Function: pprint.isrecursive (object)
     Determine if _object_ requires a recursive representation.

  One more support function is also defined:

 -- Function: pprint.saferepr (object)
     Return a string representation of _object_, protected against
     recursive data structures.  If the representation of _object_
     exposes a recursive entry, the recursive reference will be
     represented as `<Recursion on typename with id=number>'.  The
     representation is not otherwise formatted.

         >>> pprint.saferepr(stuff)
         "[<Recursion on list with id=...>, 'spam', 'eggs', 'lumberjack', 'knights', 'ni']"

5.8.18.1 PrettyPrinter Objects
..............................

PrettyPrinter instances have the following methods:

 -- Method: PrettyPrinter.pformat (object)
     Return the formatted representation of _object_.  This takes into
     account the options passed to the PrettyPrinter.
     constructor.

 -- Method: PrettyPrinter.pprint (object)
     Print the formatted representation of _object_ on the configured
     stream, followed by a newline.

  The following methods provide the implementations for the
corresponding functions of the same names.  Using these methods on an
instance is slightly more efficient since new PrettyPrinter.
objects don't need to be created.

 -- Method: PrettyPrinter.isreadable (object)
     Determine if the formatted representation of the object is
     "readable," or can be used to reconstruct the value using
     eval().  Note that this returns `False' for recursive
     objects.  If the _depth_ parameter of the PrettyPrinter
     is set and the object is deeper than allowed, this returns
     `False'.

 -- Method: PrettyPrinter.isrecursive (object)
     Determine if the object requires a recursive representation.

  This method is provided as a hook to allow subclasses to modify the
way objects are converted to strings.  The default implementation uses
the internals of the saferepr() implementation.

 -- Method: PrettyPrinter.format (object, context, maxlevels, level)
     Returns three values: the formatted version of _object_ as a
     string, a flag indicating whether the result is readable, and a
     flag indicating whether recursion was detected.  The first
     argument is the object to be presented.  The second is a
     dictionary which contains the id() of objects that are
     part of the current presentation context (direct and indirect
     containers for _object_ that are affecting the presentation) as
     the keys; if an object needs to be presented which is already
     represented in _context_, the third return value should be `True'.
     Recursive calls to the format() method should add
     additional entries for containers to this dictionary.  The third
     argument, _maxlevels_, gives the requested limit to recursion;
     this will be `0' if there is no requested limit.  This argument
     should be passed unmodified to recursive calls. The fourth
     argument, _level_, gives the current level; recursive calls should
     be passed a value less than that of the current call.

     New in version 2.3.

5.8.18.2 pprint Example
.......................

This example demonstrates several uses of the pprint().
function and its parameters.

    >>> import pprint
    >>> tup = ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead',
    ... ('parrot', ('fresh fruit',))))))))
    >>> stuff = ['a' * 10, tup, ['a' * 30, 'b' * 30], ['c' * 20, 'd' * 20]]
    >>> pprint.pprint(stuff)
    ['aaaaaaaaaa',
     ('spam',
      ('eggs',
       ('lumberjack',
        ('knights', ('ni', ('dead', ('parrot', ('fresh fruit',)))))))),
     ['aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'],
     ['cccccccccccccccccccc', 'dddddddddddddddddddd']]
    >>> pprint.pprint(stuff, depth=3)
    ['aaaaaaaaaa',
     ('spam', ('eggs', (...))),
     ['aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'],
     ['cccccccccccccccccccc', 'dddddddddddddddddddd']]
    >>> pprint.pprint(stuff, width=60)
    ['aaaaaaaaaa',
     ('spam',
      ('eggs',
       ('lumberjack',
        ('knights',
         ('ni', ('dead', ('parrot', ('fresh fruit',)))))))),
     ['aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
      'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'],
     ['cccccccccccccccccccc', 'dddddddddddddddddddd']]

5.8.19 `repr' -- Alternate `repr()' implementation
--------------------------------------------------

     Note: The repr module has been renamed to `reprlib' in
     Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

The repr module provides a means for producing object
representations with limits on the size of the resulting strings. This
is used in the Python debugger and may be useful in other contexts as
well.

  Latest version of the repr module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/repr.py?view=markup)This
module provides a class, an instance, and a function:

 -- Class: repr.Repr
     Class which provides formatting services useful in implementing
     functions similar to the built-in repr().; size limits
     for  different object types are added to avoid the generation of
     representations which are excessively long.

 -- Data: repr.aRepr
     This is an instance of Repr which is used to provide
     the repr().  function described below.  Changing the
     attributes of this object will affect the size limits used by
     repr() and the Python debugger.

 -- Function: repr.repr (obj)
     This is the repr() method of `aRepr'.  It returns a
     string similar to that returned by the built-in function of the
     same name, but with limits on most sizes.

5.8.19.1 Repr Objects
.....................

Repr instances provide several members which can be used to
provide size limits for the representations of different object types,
and methods which format specific object types.

 -- Attribute: Repr.maxlevel
     Depth limit on the creation of recursive representations.  The
     default is `6'.

 -- Attribute: Repr.maxdict
 -- Attribute: Repr.maxlist
 -- Attribute: Repr.maxtuple
 -- Attribute: Repr.maxset
 -- Attribute: Repr.maxfrozenset
 -- Attribute: Repr.maxdeque
 -- Attribute: Repr.maxarray
     Limits on the number of entries represented for the named object
     type.  The default is `4' for maxdict, `5' for
     maxarray, and  `6' for the others.

     New in version 2.4: maxset, maxfrozenset,
     and set.

 -- Attribute: Repr.maxlong
     Maximum number of characters in the representation for a long
     integer.  Digits are dropped from the middle.  The default is `40'.

 -- Attribute: Repr.maxstring
     Limit on the number of characters in the representation of the
     string.  Note that the "normal" representation of the string is
     used as the character source: if escape sequences are needed in
     the representation, these may be mangled when the representation
     is shortened.  The default is `30'.

 -- Attribute: Repr.maxother
     This limit is used to control the size of object types for which
     no specific formatting method is available on the Repr.
     object. It is applied in a similar manner as maxstring.
     The default is `20'.

 -- Method: Repr.repr (obj)
     The equivalent to the built-in repr() that uses the
     formatting imposed by the instance.

 -- Method: Repr.repr1 (obj, level)
     Recursive implementation used by repr().  This uses the
     type of _obj_ to determine which formatting method to call,
     passing it _obj_ and _level_.  The type-specific methods should
     call repr1() to perform recursive formatting, with
     `level - 1' for the value of _level_ in the recursive  call.

 -- Method: Repr.repr_TYPE (obj, level)
     Formatting methods for specific types are implemented as methods
     with a name based on the type name.  In the method name, *TYPE* is
     replaced by `string.join(string.split(type(obj).__name__, '_'))'.
     Dispatch to these methods is handled by `repr1()'. Type-specific
     methods which need to recursively format a value should call
     `self.repr1(subobj, level - 1)'.

5.8.19.2 Subclassing Repr Objects
.................................

The use of dynamic dispatching by Repr.repr1() allows
subclasses of Repr to add support for additional built-in
object types or to modify the handling of types already supported. This
example shows how special support for file objects could be added:

    import repr as reprlib
    import sys

    class MyRepr(reprlib.Repr):
        def repr_file(self, obj, level):
            if obj.name in ['<stdin>', '<stdout>', '<stderr>']:
                return obj.name
            else:
                return repr(obj)

    aRepr = MyRepr()
    print aRepr.repr(sys.stdin)          # prints '<stdin>'

5.9 Numeric and Mathematical Modules
====================================

The modules described in this chapter provide numeric and math-related
functions and data types. The numbers module defines an
abstract hierarchy of numeric types. The math and
cmath modules contain various mathematical functions for
floating-point and complex numbers. For users more interested in
decimal accuracy than in speed, the decimal module supports
exact representations of decimal numbers.

  The following modules are documented in this chapter:

5.9.1 `numbers' -- Numeric abstract base classes
------------------------------------------------

New in version 2.6.

  The numbers module ( PEP 3141
(http://www.python.org/dev/peps/pep-3141)) defines a hierarchy of
numeric abstract base classes which progressively define more
operations.  None of the types defined in this module can be
instantiated.

 -- Class: numbers.Number
     The root of the numeric hierarchy. If you just want to check if an
     argument _x_ is a number, without caring what kind, use
     `isinstance(x, Number)'.

5.9.1.1 The numeric tower
.........................

 -- Class: numbers.Complex
     Subclasses of this type describe complex numbers and include the
     operations that work on the built-in complex type.
     These are: conversions to complex and bool,
     real, imag, `+', `-', `*', `/', see abs(),
     see conjugate(), `==', and `!='. All except `-' and
     `!=' are abstract.

      -- Attribute: real
          Abstract. Retrieves the real component of this number.

      -- Attribute: imag
          Abstract. Retrieves the imaginary component of this number.

      -- Method: conjugate ()
          Abstract. Returns the complex conjugate. For example,
          `(1+3j).conjugate() == (1-3j)'.

 -- Class: numbers.Real
     To Complex, Real adds the operations that
     work on real numbers.

     In short, those are: a conversion to float, `trunc()',
     see round(), math.floor(), math.ceil(),
     see divmod(), `//', `%', `<', `<=', `>', and `>='.

     Real also provides defaults for complex(), real,
     see imag, and conjugate().

 -- Class: numbers.Rational
     Subtypes Real and adds numerator and
     denominator properties, which should be in lowest terms.
     With these, it provides a default for float().

      -- Attribute: numerator
          Abstract.

      -- Attribute: denominator
          Abstract.

 -- Class: numbers.Integral
     Subtypes Rational and adds a conversion to int.
     Provides defaults for float(), numerator,
     and denominator, and bit-string operations: `<<',
     `>>', `&', `^', `|', `~'.

5.9.1.2 Notes for type implementors
...................................

Implementors should be careful to make equal numbers equal and hash
them to the same values. This may be subtle if there are two different
extensions of the real numbers. For example, see fractions.Fraction.
implements hash() as follows:

    def __hash__(self):
        if self.denominator == 1:
            # Get integers right.
            return hash(self.numerator)
        # Expensive check, but definitely correct.
        if self == float(self):
            return hash(float(self))
        else:
            # Use tuple's hash to avoid a high collision rate on
            # simple fractions.
            return hash((self.numerator, self.denominator))

5.9.1.3 Adding More Numeric ABCs
................................

There are, of course, more possible ABCs for numbers, and this would be
a poor hierarchy if it precluded the possibility of adding those. You
can add `MyFoo' between Complex and Real with:

    class MyFoo(Complex): ...
    MyFoo.register(Real)

5.9.1.4 Implementing the arithmetic operations
..............................................

We want to implement the arithmetic operations so that mixed-mode
operations either call an implementation whose author knew about the
types of both arguments, or convert both to the nearest built in type
and do the operation there. For subtypes of Integral, this
means that __add__() and __radd__() should be
defined as:

    class MyIntegral(Integral):

        def __add__(self, other):
            if isinstance(other, MyIntegral):
                return do_my_adding_stuff(self, other)
            elif isinstance(other, OtherTypeIKnowAbout):
                return do_my_other_adding_stuff(self, other)
            else:
                return NotImplemented

        def __radd__(self, other):
            if isinstance(other, MyIntegral):
                return do_my_adding_stuff(other, self)
            elif isinstance(other, OtherTypeIKnowAbout):
                return do_my_other_adding_stuff(other, self)
            elif isinstance(other, Integral):
                return int(other) + int(self)
            elif isinstance(other, Real):
                return float(other) + float(self)
            elif isinstance(other, Complex):
                return complex(other) + complex(self)
            else:
                return NotImplemented

There are 5 different cases for a mixed-type operation on subclasses of
Complex. I'll refer to all of the above code that doesn't
refer to `MyIntegral' and `OtherTypeIKnowAbout' as "boilerplate". `a'
will be an instance of `A', which is a subtype of Complex.
(`a : A <: Complex'), and `b : B <: Complex'. I'll consider `a + b':

       1. If `A' defines an __add__() which accepts `b',
          all is well.

       2. If `A' falls back to the boilerplate code, and it were to
          return a value from __add__(), we'd miss the
          possibility that `B' defines a more intelligent
          __radd__(), so the boilerplate should return
          NotImplemented from __add__(). (Or `A' may
          not implement __add__() at all.)

       3. Then `B''s __radd__() gets a chance. If it accepts
          `a', all is well.

       4. If it falls back to the boilerplate, there are no more
          possible methods to try, so this is where the default
          implementation should live.

       5. If `B <: A', Python tries `B.__radd__' before `A.__add__'.
          This is ok, because it was implemented with knowledge of `A',
          so it can handle those instances before delegating to
          Complex.

  If `A <: Complex' and `B <: Real' without sharing any other knowledge,
then the appropriate shared operation is the one involving the built in
complex, and both __radd__() s land there, so
`a+b == b+a'.

  Because most of the operations on any given type will be very similar,
it can be useful to define a helper function which generates the
forward and reverse instances of any given operator. For example, see
fractions.Fraction uses:

    def _operator_fallbacks(monomorphic_operator, fallback_operator):
        def forward(a, b):
            if isinstance(b, (int, long, Fraction)):
                return monomorphic_operator(a, b)
            elif isinstance(b, float):
                return fallback_operator(float(a), b)
            elif isinstance(b, complex):
                return fallback_operator(complex(a), b)
            else:
                return NotImplemented
        forward.__name__ = '__' + fallback_operator.__name__ + '__'
        forward.__doc__ = monomorphic_operator.__doc__

        def reverse(b, a):
            if isinstance(a, Rational):
                # Includes ints.
                return monomorphic_operator(a, b)
            elif isinstance(a, numbers.Real):
                return fallback_operator(float(a), float(b))
            elif isinstance(a, numbers.Complex):
                return fallback_operator(complex(a), complex(b))
            else:
                return NotImplemented
        reverse.__name__ = '__r' + fallback_operator.__name__ + '__'
        reverse.__doc__ = monomorphic_operator.__doc__

        return forward, reverse

    def _add(a, b):
        """a + b"""
        return Fraction(a.numerator * b.denominator +
                        b.numerator * a.denominator,
                        a.denominator * b.denominator)

    __add__, __radd__ = _operator_fallbacks(_add, operator.add)

    # ...

5.9.2 `math' -- Mathematical functions
--------------------------------------

This module is always available.  It provides access to the mathematical
functions defined by the C standard.

  These functions cannot be used with complex numbers; use the
functions of the same name from the cmath module if you
require support for complex numbers.  The distinction between functions
which support complex numbers and those which don't is made since most
users do not want to learn quite as much mathematics as required to
understand complex numbers.  Receiving an exception instead of a
complex result allows earlier detection of the unexpected complex
number used as a parameter, so that the programmer can determine how
and why it was generated in the first place.

  The following functions are provided by this module.  Except when
explicitly noted otherwise, all return values are floats.

5.9.2.1 Number-theoretic and representation functions
.....................................................

 -- Function: math.ceil (x)
     Return the ceiling of _x_ as a float, the smallest integer value
     greater than or equal to _x_.

 -- Function: math.copysign (x, y)
     Return _x_ with the sign of _y_.  On a platform that supports
     signed zeros, `copysign(1.0, -0.0)' returns _-1.0_.

     New in version 2.6.

 -- Function: math.fabs (x)
     Return the absolute value of _x_.

 -- Function: math.factorial (x)
     Return _x_ factorial.  Raises ValueError if _x_ is not
     integral or is negative.

     New in version 2.6.

 -- Function: math.floor (x)
     Return the floor of _x_ as a float, the largest integer value less
     than or equal to _x_.

 -- Function: math.fmod (x, y)
     Return `fmod(x, y)', as defined by the platform C library. Note
     that the Python expression `x % y' may not return the same result.
     The intent of the C standard is that `fmod(x, y)' be exactly
     (mathematically; to infinite precision) equal to `x - n*y' for
     some integer _n_ such that the result has the same sign as _x_ and
     magnitude less than `abs(y)'.  Python's `x % y' returns a result
     with the sign of _y_ instead, and may not be exactly computable
     for float arguments. For example, `fmod(-1e-100, 1e100)' is
     `-1e-100', but the result of Python's `-1e-100 % 1e100' is
     `1e100-1e-100', which cannot be represented exactly as a float,
     and rounds to the surprising `1e100'.  For this reason, function
     fmod() is generally preferred when working with
     floats, while Python's `x % y' is preferred when working with
     integers.

 -- Function: math.frexp (x)
     Return the mantissa and exponent of _x_ as the pair `(m, e)'.  _m_
     is a float and _e_ is an integer such that `x == m * 2**e'
     exactly. If _x_ is zero, returns `(0.0, 0)', otherwise `0.5 <=
     abs(m) < 1'.  This is used to "pick apart" the internal
     representation of a float in a portable way.

 -- Function: math.fsum (iterable)
     Return an accurate floating point sum of values in the iterable.
     Avoids loss of precision by tracking multiple intermediate partial
     sums:

         >>> sum([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1])
         0.9999999999999999
         >>> fsum([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1])
         1.0

     The algorithm's accuracy depends on IEEE-754 arithmetic guarantees
     and the typical case where the rounding mode is half-even.  On
     some non-Windows builds, the underlying C library uses extended
     precision addition and may occasionally double-round an
     intermediate sum causing it to be off in its least significant bit.

     For further discussion and two alternative approaches, see the
     ASPN cookbook recipes for accurate floating point summation
     (http://code.activestate.com/recipes/393090/).

     New in version 2.6.

 -- Function: math.isinf (x)
     Check if the float _x_ is positive or negative infinity.

     New in version 2.6.

 -- Function: math.isnan (x)
     Check if the float _x_ is a NaN (not a number).  For more
     information on NaNs, see the IEEE 754 standards.

     New in version 2.6.

 -- Function: math.ldexp (x, i)
     Return `x * (2**i)'.  This is essentially the inverse of function
     frexp().

 -- Function: math.modf (x)
     Return the fractional and integer parts of _x_.  Both results
     carry the sign of _x_ and are floats.

 -- Function: math.trunc (x)
     Return the `Real' value _x_ truncated to an `Integral' (usually a
     long integer).  Uses the `__trunc__' method.

     New in version 2.6.

  Note that frexp() and modf() have a different
call/return pattern than their C equivalents: they take a single
argument and return a pair of values, rather than returning their
second return value through an 'output parameter' (there is no such
thing in Python).

  For the ceil(), floor(), and modf().
functions, note that _all_ floating-point numbers of sufficiently large
magnitude are exact integers.  Python floats typically carry no more
than 53 bits of precision (the same as the platform C double type), in
which case any float _x_ with `abs(x) >= 2**52' necessarily has no
fractional bits.

5.9.2.2 Power and logarithmic functions
.......................................

 -- Function: math.exp (x)
     Return `e**x'.

 -- Function: math.expm1 (x)
     Return `e**x - 1'.  For small floats _x_, the subtraction in
     `exp(x) - 1' can result in a significant loss of precision; the
     expm1() function provides a way to compute this
     quantity to full precision:

         >>> from math import exp, expm1
         >>> exp(1e-5) - 1  # gives result accurate to 11 places
         1.0000050000069649e-05
         >>> expm1(1e-5)    # result accurate to full precision
         1.0000050000166668e-05

     New in version 2.7.

 -- Function: math.log (x[, base])
     With one argument, return the natural logarithm of _x_ (to base
     _e_).

     With two arguments, return the logarithm of _x_ to the given
     _base_, calculated as `log(x)/log(base)'.

     Changed in version 2.3: _base_ argument added.

 -- Function: math.log1p (x)
     Return the natural logarithm of _1+x_ (base _e_). The result is
     calculated in a way which is accurate for _x_ near zero.

     New in version 2.6.

 -- Function: math.log10 (x)
     Return the base-10 logarithm of _x_.  This is usually more accurate
     than `log(x, 10)'.

 -- Function: math.pow (x, y)
     Return `x' raised to the power `y'.  Exceptional cases follow
     Annex 'F' of the C99 standard as far as possible.  In particular,
     `pow(1.0, x)' and `pow(x, 0.0)' always return `1.0', even when `x'
     is a zero or a NaN.  If both `x' and `y' are finite, `x' is
     negative, and `y' is not an integer then `pow(x, y)' is undefined,
     and raises ValueError.

     Changed in version 2.6: The outcome of `1**nan' and `nan**0' was
     undefined.

 -- Function: math.sqrt (x)
     Return the square root of _x_.

5.9.2.3 Trigonometric functions
...............................

 -- Function: math.acos (x)
     Return the arc cosine of _x_, in radians.

 -- Function: math.asin (x)
     Return the arc sine of _x_, in radians.

 -- Function: math.atan (x)
     Return the arc tangent of _x_, in radians.

 -- Function: math.atan2 (y, x)
     Return `atan(y / x)', in radians. The result is between `-pi' and
     `pi'.  The vector in the plane from the origin to point `(x, y)'
     makes this angle with the positive X axis. The point of
     atan2() is that the signs of both inputs are known to it, so
     it can compute the correct quadrant for the angle.  For example,
     `atan(1)' and `atan2(1, 1)' are both `pi/4', but `atan2(-1, -1)'
     is `-3*pi/4'.

 -- Function: math.cos (x)
     Return the cosine of _x_ radians.

 -- Function: math.hypot (x, y)
     Return the Euclidean norm, `sqrt(x*x + y*y)'. This is the length
     of the vector from the origin to point `(x, y)'.

 -- Function: math.sin (x)
     Return the sine of _x_ radians.

 -- Function: math.tan (x)
     Return the tangent of _x_ radians.

5.9.2.4 Angular conversion
..........................

 -- Function: math.degrees (x)
     Converts angle _x_ from radians to degrees.

 -- Function: math.radians (x)
     Converts angle _x_ from degrees to radians.

5.9.2.5 Hyperbolic functions
............................

 -- Function: math.acosh (x)
     Return the inverse hyperbolic cosine of _x_.

     New in version 2.6.

 -- Function: math.asinh (x)
     Return the inverse hyperbolic sine of _x_.

     New in version 2.6.

 -- Function: math.atanh (x)
     Return the inverse hyperbolic tangent of _x_.

     New in version 2.6.

 -- Function: math.cosh (x)
     Return the hyperbolic cosine of _x_.

 -- Function: math.sinh (x)
     Return the hyperbolic sine of _x_.

 -- Function: math.tanh (x)
     Return the hyperbolic tangent of _x_.

5.9.2.6 Special functions
.........................

 -- Function: math.erf (x)
     Return the error function at _x_.

     New in version 2.7.

 -- Function: math.erfc (x)
     Return the complementary error function at _x_.

     New in version 2.7.

 -- Function: math.gamma (x)
     Return the Gamma function at _x_.

     New in version 2.7.

 -- Function: math.lgamma (x)
     Return the natural logarithm of the absolute value of the Gamma
     function at _x_.

     New in version 2.7.

5.9.2.7 Constants
.................

 -- Data: math.pi
     The mathematical constant π = 3.141592..., to available precision.

 -- Data: math.e
     The mathematical constant e = 2.718281..., to available precision.

*CPython implementation detail:* The math module consists
mostly of thin wrappers around the platform C math library functions.
Behavior in exceptional cases follows Annex F of the C99 standard where
appropriate.  The current implementation will raise ValueError
for invalid operations like `sqrt(-1.0)' or `log(0.0)' (where C99
Annex F recommends signaling invalid operation or divide-by-zero), and
OverflowError for results that overflow (for example,
`exp(1000.0)').  A NaN will not be returned from any of the functions
above unless one or more of the input arguments was a NaN; in that case,
most functions will return a NaN, but (again following C99 Annex F)
there are some exceptions to this rule, for example `pow(float('nan'),
0.0)' or `hypot(float('nan'), float('inf'))'.

Note that Python makes no effort to distinguish signaling NaNs from
quiet NaNs, and behavior for signaling NaNs remains unspecified.
Typical behavior is to treat all NaNs as though they were quiet.

  Changed in version 2.6: Behavior in special cases now aims to follow
C99 Annex F.  In earlier versions of Python the behavior in special
cases was loosely specified.

Module cmath.
     Complex number versions of many of these functions.

5.9.3 `cmath' -- Mathematical functions for complex numbers
-----------------------------------------------------------

This module is always available.  It provides access to mathematical
functions for complex numbers.  The functions in this module accept
integers, floating-point numbers or complex numbers as arguments. They
will also accept any Python object that has either a
__complex__() or a __float__().  method: these methods
are used to convert the object to a complex or floating-point number,
respectively, and the function is then applied to the result of the
conversion.

     Note: On platforms with hardware and system-level support for
     signed zeros, functions involving branch cuts are continuous on
     _both_ sides of the branch cut: the sign of the zero distinguishes
     one side of the branch cut from the other.  On platforms that do
     not support signed zeros the continuity is as specified below.

5.9.3.1 Conversions to and from polar coordinates
.................................................

A Python complex number `z' is stored internally using _rectangular_ or
_Cartesian_ coordinates.  It is completely determined by its _real
part_ `z.real' and its _imaginary part_ `z.imag'.  In other words:

    z == z.real + z.imag*1j

_Polar coordinates_ give an alternative way to represent a complex
number.  In polar coordinates, a complex number _z_ is defined by the
modulus _r_ and the phase angle _phi_. The modulus _r_ is the distance
from _z_ to the origin, while the phase _phi_ is the counterclockwise
angle, measured in radians, from the positive x-axis to the line
segment that joins the origin to _z_.

  The following functions can be used to convert from the native
rectangular coordinates to polar coordinates and back.

 -- Function: cmath.phase (x)
     Return the phase of _x_ (also known as the _argument_ of _x_), as a
     float.  `phase(x)' is equivalent to `math.atan2(x.imag, x.real)'.
     The result lies in the range [-π, π], and the branch cut for this
     operation lies along the negative real axis, continuous from
     above.  On systems with support for signed zeros (which includes
     most systems in current use), this means that the sign of the
     result is the same as the sign of `x.imag', even when `x.imag' is
     zero:

         >>> phase(complex(-1.0, 0.0))
         3.1415926535897931
         >>> phase(complex(-1.0, -0.0))
         -3.1415926535897931

     New in version 2.6.

     Note: The modulus (absolute value) of a complex number _x_ can be
     computed using the built-in abs() function.  There is
     no separate cmath module function for this operation.

 -- Function: cmath.polar (x)
     Return the representation of _x_ in polar coordinates.  Returns a
     pair `(r, phi)' where _r_ is the modulus of _x_ and phi is the
     phase of _x_.  `polar(x)' is equivalent to `(abs(x), phase(x))'.

     New in version 2.6.

 -- Function: cmath.rect (r, phi)
     Return the complex number _x_ with polar coordinates _r_ and _phi_.
     Equivalent to `r * (math.cos(phi) + math.sin(phi)*1j)'.

     New in version 2.6.

5.9.3.2 Power and logarithmic functions
.......................................

 -- Function: cmath.exp (x)
     Return the exponential value `e**x'.

 -- Function: cmath.log (x[, base])
     Returns the logarithm of _x_ to the given _base_. If the _base_ is
     not specified, returns the natural logarithm of _x_. There is one
     branch cut, from 0 along the negative real axis to -∞, continuous
     from above.

     Changed in version 2.4: _base_ argument added.

 -- Function: cmath.log10 (x)
     Return the base-10 logarithm of _x_. This has the same branch cut
     as log().

 -- Function: cmath.sqrt (x)
     Return the square root of _x_. This has the same branch cut as
     log().

5.9.3.3 Trigonometric functions
...............................

 -- Function: cmath.acos (x)
     Return the arc cosine of _x_. There are two branch cuts: One
     extends right from 1 along the real axis to ∞, continuous from
     below. The other extends left from -1 along the real axis to -∞,
     continuous from above.

 -- Function: cmath.asin (x)
     Return the arc sine of _x_. This has the same branch cuts as
     acos().

 -- Function: cmath.atan (x)
     Return the arc tangent of _x_. There are two branch cuts: One
     extends from `1j' along the imaginary axis to `∞j', continuous
     from the right. The other extends from `-1j' along the imaginary
     axis to `-∞j', continuous from the left.

     Changed in version 2.6: direction of continuity of upper cut
     reversed

 -- Function: cmath.cos (x)
     Return the cosine of _x_.

 -- Function: cmath.sin (x)
     Return the sine of _x_.

 -- Function: cmath.tan (x)
     Return the tangent of _x_.

5.9.3.4 Hyperbolic functions
............................

 -- Function: cmath.acosh (x)
     Return the hyperbolic arc cosine of _x_. There is one branch cut,
     extending left from 1 along the real axis to -∞, continuous from
     above.

 -- Function: cmath.asinh (x)
     Return the hyperbolic arc sine of _x_. There are two branch cuts:
     One extends from `1j' along the imaginary axis to `∞j', continuous
     from the right.  The other extends from `-1j' along the imaginary
     axis to `-∞j', continuous from the left.

     Changed in version 2.6: branch cuts moved to match those
     recommended by the C99 standard

 -- Function: cmath.atanh (x)
     Return the hyperbolic arc tangent of _x_. There are two branch
     cuts: One extends from `1' along the real axis to `∞', continuous
     from below. The other extends from `-1' along the real axis to
     `-∞', continuous from above.

     Changed in version 2.6: direction of continuity of right cut
     reversed

 -- Function: cmath.cosh (x)
     Return the hyperbolic cosine of _x_.

 -- Function: cmath.sinh (x)
     Return the hyperbolic sine of _x_.

 -- Function: cmath.tanh (x)
     Return the hyperbolic tangent of _x_.

5.9.3.5 Classification functions
................................

 -- Function: cmath.isinf (x)
     Return _True_ if the real or the imaginary part of x is positive
     or negative infinity.

     New in version 2.6.

 -- Function: cmath.isnan (x)
     Return _True_ if the real or imaginary part of x is not a number
     (NaN).

     New in version 2.6.

5.9.3.6 Constants
.................

 -- Data: cmath.pi
     The mathematical constant _π_, as a float.

 -- Data: cmath.e
     The mathematical constant _e_, as a float.

  Note that the selection of functions is similar, but not identical,
to that in module math.  The reason for having two modules
is that some users aren't interested in complex numbers, and perhaps
don't even know what they are.  They would rather have `math.sqrt(-1)'
raise an exception than return a complex number. Also note that the
functions defined in cmath always return a complex number,
even if the answer can be expressed as a real number (in which case the
complex number has an imaginary part of zero).

  A note on branch cuts: They are curves along which the given function
fails to be continuous.  They are a necessary feature of many complex
functions.  It is assumed that if you need to compute with complex
functions, you will understand about branch cuts.  Consult almost any
(not too elementary) book on complex variables for enlightenment.  For
information of the proper choice of branch cuts for numerical purposes,
a good reference should be the following:

  Kahan, W:  Branch cuts for complex elementary functions; or, Much ado
about nothing's sign bit.  In Iserles, A., and Powell, M. (eds.), The
state of the art in numerical analysis. Clarendon Press (1987)
pp165-211.

5.9.4 `decimal' -- Decimal fixed point and floating point arithmetic
--------------------------------------------------------------------

New in version 2.4.

  The decimal module provides support for decimal floating
point arithmetic.  It offers several advantages over the float
datatype:

   * Decimal "is based on a floating-point model which was designed
     with people in mind, and necessarily has a paramount guiding
     principle - computers must provide an arithmetic that works in the
     same way as the arithmetic that people learn at school." - excerpt
     from the decimal arithmetic specification.

   * Decimal numbers can be represented exactly.  In contrast, numbers
     like `1.1' and `2.2' do not have an exact representations in binary
     floating point.  End users typically would not expect `1.1 + 2.2'
     to display as `3.3000000000000003' as it does with binary floating
     point.

   * The exactness carries over into arithmetic.  In decimal floating
     point, `0.1 + 0.1 + 0.1 - 0.3' is exactly equal to zero.  In
     binary floating point, the result is `5.5511151231257827e-017'.
     While near to zero, the differences prevent reliable equality
     testing and differences can accumulate. For this reason, decimal
     is preferred in accounting applications which have strict equality
     invariants.

   * The decimal module incorporates a notion of significant places so
     that `1.30 + 1.20' is `2.50'.  The trailing zero is kept to
     indicate significance.  This is the customary presentation for
     monetary applications. For multiplication, the "schoolbook"
     approach uses all the figures in the multiplicands.  For instance,
     `1.3 * 1.2' gives `1.56' while `1.30 * 1.20' gives `1.5600'.

   * Unlike hardware based binary floating point, the decimal module
     has a user alterable precision (defaulting to 28 places) which can
     be as large as needed for a given problem:

         >>> from decimal import *
         >>> getcontext().prec = 6
         >>> Decimal(1) / Decimal(7)
         Decimal('0.142857')
         >>> getcontext().prec = 28
         >>> Decimal(1) / Decimal(7)
         Decimal('0.1428571428571428571428571429')

   * Both binary and decimal floating point are implemented in terms of
     published standards.  While the built-in float type exposes only a
     modest portion of its capabilities, the decimal module exposes all
     required parts of the standard.  When needed, the programmer has
     full control over rounding and signal handling.  This includes an
     option to enforce exact arithmetic by using exceptions to block
     any inexact operations.

   * The decimal module was designed to support "without prejudice,
     both exact unrounded decimal arithmetic (sometimes called
     fixed-point arithmetic) and rounded floating-point arithmetic."  -
     excerpt from the decimal arithmetic specification.

  The module design is centered around three concepts:  the decimal
number, the context for arithmetic, and signals.

  A decimal number is immutable.  It has a sign, coefficient digits,
and an exponent.  To preserve significance, the coefficient digits do
not truncate trailing zeros.  Decimals also include special values such
as `Infinity', `-Infinity', and `NaN'.  The standard also
differentiates `-0' from `+0'.

  The context for arithmetic is an environment specifying precision,
rounding rules, limits on exponents, flags indicating the results of
operations, and trap enablers which determine whether signals are
treated as exceptions.  Rounding options include `ROUND_CEILING',
`ROUND_DOWN', `ROUND_FLOOR', `ROUND_HALF_DOWN', `ROUND_HALF_EVEN',
`ROUND_HALF_UP', `ROUND_UP', and `ROUND_05UP'.

  Signals are groups of exceptional conditions arising during the
course of computation.  Depending on the needs of the application,
signals may be ignored, considered as informational, or treated as
exceptions. The signals in the decimal module are: Clamped,
InvalidOperation, DivisionByZero, Inexact,
see Rounded, Subnormal, Overflow, and
Underflow.

  For each signal there is a flag and a trap enabler.  When a signal is
encountered, its flag is set to one, then, if the trap enabler is set
to one, an exception is raised.  Flags are sticky, so the user needs to
reset them before monitoring a calculation.

   * IBM's General Decimal Arithmetic Specification, The General
     Decimal Arithmetic Specification (http://speleotrove.com/decimal/).

   * IEEE standard 854-1987, Unofficial IEEE 854 Text
     (http://754r.ucbtest.org/standards/854.pdf).

5.9.4.1 Quick-start Tutorial
............................

The usual start to using decimals is importing the module, viewing the
current context with getcontext() and, if necessary,
setting new values for precision, rounding, or enabled traps:

    >>> from decimal import *
    >>> getcontext()
    Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999,
            capitals=1, flags=[], traps=[Overflow, DivisionByZero,
            InvalidOperation])

    >>> getcontext().prec = 7       # Set a new precision

Decimal instances can be constructed from integers, strings, floats, or
tuples.  Construction from an integer or a float performs an exact
conversion of the value of that integer or float.  Decimal numbers
include special values such as `NaN' which stands for "Not a number",
positive and negative `Infinity', and `-0'.

    >>> getcontext().prec = 28
    >>> Decimal(10)
    Decimal('10')
    >>> Decimal('3.14')
    Decimal('3.14')
    >>> Decimal(3.14)
    Decimal('3.140000000000000124344978758017532527446746826171875')
    >>> Decimal((0, (3, 1, 4), -2))
    Decimal('3.14')
    >>> Decimal(str(2.0 ** 0.5))
    Decimal('1.41421356237')
    >>> Decimal(2) ** Decimal('0.5')
    Decimal('1.414213562373095048801688724')
    >>> Decimal('NaN')
    Decimal('NaN')
    >>> Decimal('-Infinity')
    Decimal('-Infinity')

The significance of a new Decimal is determined solely by the number of
digits input.  Context precision and rounding only come into play
during arithmetic operations.

    >>> getcontext().prec = 6
    >>> Decimal('3.0')
    Decimal('3.0')
    >>> Decimal('3.1415926535')
    Decimal('3.1415926535')
    >>> Decimal('3.1415926535') + Decimal('2.7182818285')
    Decimal('5.85987')
    >>> getcontext().rounding = ROUND_UP
    >>> Decimal('3.1415926535') + Decimal('2.7182818285')
    Decimal('5.85988')

Decimals interact well with much of the rest of Python.  Here is a
small decimal floating point flying circus:

    >>> data = map(Decimal, '1.34 1.87 3.45 2.35 1.00 0.03 9.25'.split())
    >>> max(data)
    Decimal('9.25')
    >>> min(data)
    Decimal('0.03')
    >>> sorted(data)
    [Decimal('0.03'), Decimal('1.00'), Decimal('1.34'), Decimal('1.87'),
     Decimal('2.35'), Decimal('3.45'), Decimal('9.25')]
    >>> sum(data)
    Decimal('19.29')
    >>> a,b,c = data[:3]
    >>> str(a)
    '1.34'
    >>> float(a)
    1.34
    >>> round(a, 1)     # round() first converts to binary floating point
    1.3
    >>> int(a)
    1
    >>> a * 5
    Decimal('6.70')
    >>> a * b
    Decimal('2.5058')
    >>> c % a
    Decimal('0.77')

And some mathematical functions are also available to Decimal:

    >>> getcontext().prec = 28
    >>> Decimal(2).sqrt()
    Decimal('1.414213562373095048801688724')
    >>> Decimal(1).exp()
    Decimal('2.718281828459045235360287471')
    >>> Decimal('10').ln()
    Decimal('2.302585092994045684017991455')
    >>> Decimal('10').log10()
    Decimal('1')

The `quantize()' method rounds a number to a fixed exponent.  This
method is useful for monetary applications that often round results to
a fixed number of places:

    >>> Decimal('7.325').quantize(Decimal('.01'), rounding=ROUND_DOWN)
    Decimal('7.32')
    >>> Decimal('7.325').quantize(Decimal('1.'), rounding=ROUND_UP)
    Decimal('8')

As shown above, the getcontext() function accesses the
current context and allows the settings to be changed.  This approach
meets the needs of most applications.

  For more advanced work, it may be useful to create alternate contexts
using the Context() constructor.  To make an alternate active, use the
setcontext().  function.

  In accordance with the standard, the `Decimal' module provides two
ready to use standard contexts, see BasicContext and
ExtendedContext. The former is especially useful for debugging
because many of the traps are enabled:

    >>> myothercontext = Context(prec=60, rounding=ROUND_HALF_DOWN)
    >>> setcontext(myothercontext)
    >>> Decimal(1) / Decimal(7)
    Decimal('0.142857142857142857142857142857142857142857142857142857142857')

    >>> ExtendedContext
    Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999,
            capitals=1, flags=[], traps=[])
    >>> setcontext(ExtendedContext)
    >>> Decimal(1) / Decimal(7)
    Decimal('0.142857143')
    >>> Decimal(42) / Decimal(0)
    Decimal('Infinity')

    >>> setcontext(BasicContext)
    >>> Decimal(42) / Decimal(0)
    Traceback (most recent call last):
      File "<pyshell#143>", line 1, in -toplevel-
        Decimal(42) / Decimal(0)
    DivisionByZero: x / 0

Contexts also have signal flags for monitoring exceptional conditions
encountered during computations.  The flags remain set until explicitly
cleared, so it is best to clear the flags before each set of monitored
computations by using the `clear_flags()' method.

    >>> setcontext(ExtendedContext)
    >>> getcontext().clear_flags()
    >>> Decimal(355) / Decimal(113)
    Decimal('3.14159292')
    >>> getcontext()
    Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999,
            capitals=1, flags=[Rounded, Inexact], traps=[])

The _flags_ entry shows that the rational approximation to `Pi' was
rounded (digits beyond the context precision were thrown away) and that
the result is inexact (some of the discarded digits were non-zero).

  Individual traps are set using the dictionary in the `traps' field of
a context:

    >>> setcontext(ExtendedContext)
    >>> Decimal(1) / Decimal(0)
    Decimal('Infinity')
    >>> getcontext().traps[DivisionByZero] = 1
    >>> Decimal(1) / Decimal(0)
    Traceback (most recent call last):
      File "<pyshell#112>", line 1, in -toplevel-
        Decimal(1) / Decimal(0)
    DivisionByZero: x / 0

Most programs adjust the current context only once, at the beginning of
the program.  And, in many applications, data is converted to
Decimal with a single cast inside a loop.  With context set and
decimals created, the bulk of the program manipulates the data no
differently than with other Python numeric types.

5.9.4.2 Decimal objects
.......................

 -- Class: decimal.Decimal ([value[, context]])
     Construct a new Decimal object based from _value_.

     _value_ can be an integer, string, tuple, see float, or
     another Decimal.  object. If no _value_ is given,
     returns `Decimal('0')'.  If _value_ is a string, it should conform
     to the decimal numeric string syntax after leading and trailing
     whitespace characters are removed:

         sign           ::=  '+' | '-'
         digit          ::=  '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
         indicator      ::=  'e' | 'E'
         digits         ::=  digit [digit]...
         decimal-part   ::=  digits '.' [digits] | ['.'] digits
         exponent-part  ::=  indicator [sign] digits
         infinity       ::=  'Infinity' | 'Inf'
         nan            ::=  'NaN' [digits] | 'sNaN' [digits]
         numeric-value  ::=  decimal-part [exponent-part] | infinity
         numeric-string ::=  [sign] numeric-value | [sign] nan

     If _value_ is a unicode string then other Unicode decimal digits
     are also permitted where `digit' appears above.  These include
     decimal digits from various other alphabets (for example,
     Arabic-Indic and Devanāgarī digits) along with the fullwidth digits
     `u'\uff10'' through `u'\uff19''.

     If _value_ is a tuple, it should have three components,
     a sign (`0' for positive or `1' for negative), a tuple.
     of digits, and an integer exponent. For example, `Decimal((0, (1,
     4, 1, 4), -3))' returns `Decimal('1.414')'.

     If _value_ is a float, the binary floating point value
     is losslessly converted to its exact decimal equivalent.  This
     conversion can often require 53 or more digits of precision.  For
     example, `Decimal(float('1.1'))' converts to
     `Decimal('1.100000000000000088817841970012523233890533447265625')'.

     The _context_ precision does not affect how many digits are
     stored. That is determined exclusively by the number of digits in
     _value_. For example, `Decimal('3.00000')' records all five zeros
     even if the context precision is only three.

     The purpose of the _context_ argument is determining what to do if
     _value_ is a malformed string.  If the context traps
     InvalidOperation, an exception is raised; otherwise, the
     constructor returns a new Decimal with the value of `NaN'.

     Once constructed, see Decimal objects are immutable.

     Changed in version 2.6: leading and trailing whitespace characters
     are permitted when creating a Decimal instance from a string.

     Changed in version 2.7: The argument to the constructor is now
     permitted to be a float instance.

     Decimal floating point objects share many properties with the
     other built-in numeric types such as float and
     int.  All of the usual math operations and special methods
     apply.  Likewise, decimal objects can be copied, pickled, printed,
     used as dictionary keys, used as set elements, compared, sorted,
     and coerced to another type (such as float or
     long).

     Decimal objects cannot generally be combined with floats in
     arithmetic operations: an attempt to add a Decimal to a
     float, for example, will raise a TypeError.
     There's one exception to this rule: it's possible to use Python's
     comparison operators to compare a float instance `x'
     with a Decimal instance `y'.  Without this exception,
     comparisons between Decimal and float.
     instances would follow the general rules for comparing objects of
     different types described in the Expressions section
     of the reference manual, leading to confusing results.

     Changed in version 2.7: A comparison between a float.
     instance `x' and a Decimal instance `y' now returns a
     result based on the values of `x' and `y'.  In earlier versions `x
     < y' returned the same (arbitrary) result for any Decimal.
     instance `x' and any float instance `y'.

     In addition to the standard numeric properties, decimal floating
     point objects also have a number of specialized methods:

      -- Method: adjusted ()
          Return the adjusted exponent after shifting out the
          coefficient's rightmost digits until only the lead digit
          remains: `Decimal('321e+5').adjusted()' returns seven.  Used
          for determining the position of the most significant digit
          with respect to the decimal point.

      -- Method: as_tuple ()
          Return a named tuple representation of the number:
          `DecimalTuple(sign, digits, exponent)'.

          Changed in version 2.6: Use a named tuple.

      -- Method: canonical ()
          Return the canonical encoding of the argument.  Currently,
          the encoding of a Decimal instance is always
          canonical, so this operation returns its argument unchanged.

          New in version 2.6.

      -- Method: compare (other[, context])
          Compare the values of two Decimal instances.  This operation
          behaves in the same way as the usual comparison method
          __cmp__(), except that compare() returns a
          Decimal instance rather than an integer, and if either
          operand is a NaN then the result is a NaN:

              a or b is a NaN ==> Decimal('NaN')
              a < b           ==> Decimal('-1')
              a == b          ==> Decimal('0')
              a > b           ==> Decimal('1')

      -- Method: compare_signal (other[, context])
          This operation is identical to the compare().
          method, except that all NaNs signal.  That is, if neither
          operand is a signaling NaN then any quiet NaN operand is
          treated as though it were a signaling NaN.

          New in version 2.6.

      -- Method: compare_total (other)
          Compare two operands using their abstract representation
          rather than their numerical value.  Similar to the
          compare() method, but the result gives a total ordering
          on Decimal instances.  Two Decimal.
          instances with the same numeric value but different
          representations compare unequal in this ordering:

              >>> Decimal('12.0').compare_total(Decimal('12'))
              Decimal('-1')

          Quiet and signaling NaNs are also included in the total
          ordering.  The result of this function is `Decimal('0')' if
          both operands have the same representation, `Decimal('-1')'
          if the first operand is lower in the total order than the
          second, and `Decimal('1')' if the first operand is higher in
          the total order than the second operand.  See the
          specification for details of the total order.

          New in version 2.6.

      -- Method: compare_total_mag (other)
          Compare two operands using their abstract representation
          rather than their value as in compare_total(), but
          ignoring the sign of each operand.  `x.compare_total_mag(y)'
          is equivalent to `x.copy_abs().compare_total(y.copy_abs())'.

          New in version 2.6.

      -- Method: conjugate ()
          Just returns self, this method is only to comply with the
          Decimal Specification.

          New in version 2.6.

      -- Method: copy_abs ()
          Return the absolute value of the argument.  This operation is
          unaffected by the context and is quiet: no flags are changed
          and no rounding is performed.

          New in version 2.6.

      -- Method: copy_negate ()
          Return the negation of the argument.  This operation is
          unaffected by the context and is quiet: no flags are changed
          and no rounding is performed.

          New in version 2.6.

      -- Method: copy_sign (other)
          Return a copy of the first operand with the sign set to be
          the same as the sign of the second operand.  For example:

              >>> Decimal('2.3').copy_sign(Decimal('-1.5'))
              Decimal('-2.3')

          This operation is unaffected by the context and is quiet: no
          flags are changed and no rounding is performed.

          New in version 2.6.

      -- Method: exp ([context])
          Return the value of the (natural) exponential function `e**x'
          at the given number.  The result is correctly rounded using
          the `ROUND_HALF_EVEN' rounding mode.

              >>> Decimal(1).exp()
              Decimal('2.718281828459045235360287471')
              >>> Decimal(321).exp()
              Decimal('2.561702493119680037517373933E+139')

          New in version 2.6.

      -- Method: from_float (f)
          Classmethod that converts a float to a decimal number,
          exactly.

          Note `Decimal.from_float(0.1)' is not the same as
          `Decimal('0.1')'.  Since 0.1 is not exactly representable in
          binary floating point, the value is stored as the nearest
          representable value which is `0x1.999999999999ap-4'.  That
          equivalent value in decimal is
          `0.1000000000000000055511151231257827021181583404541015625'.

               Note: From Python 2.7 onwards, a Decimal.
               instance can also be constructed directly from a
               float.

              >>> Decimal.from_float(0.1)
              Decimal('0.1000000000000000055511151231257827021181583404541015625')
              >>> Decimal.from_float(float('nan'))
              Decimal('NaN')
              >>> Decimal.from_float(float('inf'))
              Decimal('Infinity')
              >>> Decimal.from_float(float('-inf'))
              Decimal('-Infinity')

          New in version 2.7.

      -- Method: fma (other, third[, context])
          Fused multiply-add.  Return self*other+third with no rounding
          of the intermediate product self*other.

              >>> Decimal(2).fma(3, 5)
              Decimal('11')

          New in version 2.6.

      -- Method: is_canonical ()
          Return True if the argument is canonical and
          False.  otherwise.  Currently, a Decimal
          instance is always canonical, so this operation always
          returns True.

          New in version 2.6.

      -- Method: is_finite ()
          Return True if the argument is a finite number,
          and False if the argument is an infinity or a NaN.

          New in version 2.6.

      -- Method: is_infinite ()
          Return True if the argument is either positive or
          negative infinity and False otherwise.

          New in version 2.6.

      -- Method: is_nan ()
          Return True if the argument is a (quiet or
          signaling) NaN and False otherwise.

          New in version 2.6.

      -- Method: is_normal ()
          Return True if the argument is a _normal_ finite
          non-zero number with an adjusted exponent greater than or
          equal to _Emin_.  Return False if the argument is
          zero, subnormal, infinite or a NaN.  Note, the term _normal_
          is used here in a different sense with the normalize()
          method which is used to create canonical values.

          New in version 2.6.

      -- Method: is_qnan ()
          Return True if the argument is a quiet NaN, and
          False otherwise.

          New in version 2.6.

      -- Method: is_signed ()
          Return True if the argument has a negative sign
          and False otherwise.  Note that zeros and NaNs
          can both carry signs.

          New in version 2.6.

      -- Method: is_snan ()
          Return True if the argument is a signaling NaN
          and False.  otherwise.

          New in version 2.6.

      -- Method: is_subnormal ()
          Return True if the argument is subnormal, and
          False.  otherwise. A number is subnormal is if it
          is nonzero, finite, and has an adjusted exponent less than
          _Emin_.

          New in version 2.6.

      -- Method: is_zero ()
          Return True if the argument is a (positive or
          negative) zero and False otherwise.

          New in version 2.6.

      -- Method: ln ([context])
          Return the natural (base e) logarithm of the operand.  The
          result is correctly rounded using the `ROUND_HALF_EVEN'
          rounding mode.

          New in version 2.6.

      -- Method: log10 ([context])
          Return the base ten logarithm of the operand.  The result is
          correctly rounded using the `ROUND_HALF_EVEN' rounding mode.

          New in version 2.6.

      -- Method: logb ([context])
          For a nonzero number, return the adjusted exponent of its
          operand as a Decimal instance.  If the operand is
          a zero then `Decimal('-Infinity')' is returned and the
          DivisionByZero flag is raised.  If the operand is an
          infinity then `Decimal('Infinity')' is returned.

          New in version 2.6.

      -- Method: logical_and (other[, context])
          logical_and() is a logical operation which takes
          two _logical operands_ (see Logical operands).
          The result is the digit-wise `and' of the two operands.

          New in version 2.6.

      -- Method: logical_invert ([context])
          logical_invert() is a logical operation.  The
          result is the digit-wise inversion of the operand.

          New in version 2.6.

      -- Method: logical_or (other[, context])
          logical_or() is a logical operation which takes
          two _logical operands_ (see Logical operands).
          The result is the digit-wise `or' of the two operands.

          New in version 2.6.

      -- Method: logical_xor (other[, context])
          logical_xor() is a logical operation which takes
          two _logical operands_ (see Logical operands).
          The result is the digit-wise exclusive or of the two operands.

          New in version 2.6.

      -- Method: max (other[, context])
          Like `max(self, other)' except that the context rounding rule
          is applied before returning and that `NaN' values are either
          signaled or ignored (depending on the context and whether
          they are signaling or quiet).

      -- Method: max_mag (other[, context])
          Similar to the max() method, but the comparison
          is done using the absolute values of the operands.

          New in version 2.6.

      -- Method: min (other[, context])
          Like `min(self, other)' except that the context rounding rule
          is applied before returning and that `NaN' values are either
          signaled or ignored (depending on the context and whether
          they are signaling or quiet).

      -- Method: min_mag (other[, context])
          Similar to the min() method, but the comparison
          is done using the absolute values of the operands.

          New in version 2.6.

      -- Method: next_minus ([context])
          Return the largest number representable in the given context
          (or in the current thread's context if no context is given)
          that is smaller than the given operand.

          New in version 2.6.

      -- Method: next_plus ([context])
          Return the smallest number representable in the given context
          (or in the current thread's context if no context is given)
          that is larger than the given operand.

          New in version 2.6.

      -- Method: next_toward (other[, context])
          If the two operands are unequal, return the number closest to
          the first operand in the direction of the second operand.  If
          both operands are numerically equal, return a copy of the
          first operand with the sign set to be the same as the sign of
          the second operand.

          New in version 2.6.

      -- Method: normalize ([context])
          Normalize the number by stripping the rightmost trailing
          zeros and converting any result equal to `Decimal('0')' to
          `Decimal('0e0')'. Used for producing canonical values for
          members of an equivalence class. For example,
          `Decimal('32.100')' and `Decimal('0.321000e+2')' both
          normalize to the equivalent value `Decimal('32.1')'.

      -- Method: number_class ([context])
          Return a string describing the _class_ of the operand.  The
          returned value is one of the following ten strings.

             * `"-Infinity"', indicating that the operand is negative
               infinity.

             * `"-Normal"', indicating that the operand is a negative
               normal number.

             * `"-Subnormal"', indicating that the operand is negative
               and subnormal.

             * `"-Zero"', indicating that the operand is a negative
               zero.

             * `"+Zero"', indicating that the operand is a positive
               zero.

             * `"+Subnormal"', indicating that the operand is positive
               and subnormal.

             * `"+Normal"', indicating that the operand is a positive
               normal number.

             * `"+Infinity"', indicating that the operand is positive
               infinity.

             * `"NaN"', indicating that the operand is a quiet NaN (Not
               a Number).

             * `"sNaN"', indicating that the operand is a signaling NaN.

          New in version 2.6.

      -- Method: quantize (exp[, rounding[, context[, watchexp]]])
          Return a value equal to the first operand after rounding and
          having the exponent of the second operand.

              >>> Decimal('1.41421356').quantize(Decimal('1.000'))
              Decimal('1.414')

          Unlike other operations, if the length of the coefficient
          after the quantize operation would be greater than precision,
          then an InvalidOperation is signaled. This
          guarantees that, unless there is an error condition, the
          quantized exponent is always equal to that of the right-hand
          operand.

          Also unlike other operations, quantize never signals
          Underflow, even if the result is subnormal and inexact.

          If the exponent of the second operand is larger than that of
          the first then rounding may be necessary.  In this case, the
          rounding mode is determined by the `rounding' argument if
          given, else by the given `context' argument; if neither
          argument is given the rounding mode of the current thread's
          context is used.

          If _watchexp_ is set (default), then an error is returned
          whenever the resulting exponent is greater than `Emax' or
          less than `Etiny'.

      -- Method: radix ()
          Return `Decimal(10)', the radix (base) in which the
          Decimal.  class does all its arithmetic.  Included for
          compatibility with the specification.

          New in version 2.6.

      -- Method: remainder_near (other[, context])
          Compute the modulo as either a positive or negative value
          depending on which is closest to zero.  For instance,
          `Decimal(10).remainder_near(6)' returns `Decimal('-2')' which
          is closer to zero than `Decimal('4')'.

          If both are equally close, the one chosen will have the same
          sign as _self_.

      -- Method: rotate (other[, context])
          Return the result of rotating the digits of the first operand
          by an amount specified by the second operand.  The second
          operand must be an integer in the range -precision through
          precision.  The absolute value of the second operand gives
          the number of places to rotate.  If the second operand is
          positive then rotation is to the left; otherwise rotation is
          to the right.  The coefficient of the first operand is padded
          on the left with zeros to length precision if necessary.  The
          sign and exponent of the first operand are unchanged.

          New in version 2.6.

      -- Method: same_quantum (other[, context])
          Test whether self and other have the same exponent or whether
          both are `NaN'.

      -- Method: scaleb (other[, context])
          Return the first operand with exponent adjusted by the second.
          Equivalently, return the first operand multiplied by
          `10**other'.  The second operand must be an integer.

          New in version 2.6.

      -- Method: shift (other[, context])
          Return the result of shifting the digits of the first operand
          by an amount specified by the second operand.  The second
          operand must be an integer in the range -precision through
          precision.  The absolute value of the second operand gives
          the number of places to shift.  If the second operand is
          positive then the shift is to the left; otherwise the shift
          is to the right.  Digits shifted into the coefficient are
          zeros.  The sign and exponent of the first operand are
          unchanged.

          New in version 2.6.

      -- Method: sqrt ([context])
          Return the square root of the argument to full precision.

      -- Method: to_eng_string ([context])
          Convert to an engineering-type string.

          Engineering notation has an exponent which is a multiple of
          3, so there are up to 3 digits left of the decimal place.
          For example, converts `Decimal('123E+1')' to
          `Decimal('1.23E+3')'

      -- Method: to_integral ([rounding[, context]])
          Identical to the to_integral_value() method.  The
          `to_integral' name has been kept for compatibility with older
          versions.

      -- Method: to_integral_exact ([rounding[, context]])
          Round to the nearest integer, signaling Inexact or
          Rounded as appropriate if rounding occurs.  The
          rounding mode is determined by the `rounding' parameter if
          given, else by the given `context'.  If neither parameter is
          given then the rounding mode of the current context is used.

          New in version 2.6.

      -- Method: to_integral_value ([rounding[, context]])
          Round to the nearest integer without signaling Inexact
          or Rounded.  If given, applies _rounding_;
          otherwise, uses the rounding method in either the supplied
          _context_ or the current context.

          Changed in version 2.6: renamed from `to_integral' to
          `to_integral_value'.  The old name remains valid for
          compatibility.

5.9.4.3 Logical operands
........................

The `logical_and()', `logical_invert()', `logical_or()', and
`logical_xor()' methods expect their arguments to be _logical
operands_.  A _logical operand_ is a Decimal instance whose
exponent and sign are both zero, and whose digits are all either `0' or
`1'.

5.9.4.4 Context objects
.......................

Contexts are environments for arithmetic operations.  They govern
precision, set rules for rounding, determine which signals are treated
as exceptions, and limit the range for exponents.

  Each thread has its own current context which is accessed or changed
using the getcontext() and setcontext().
functions:

 -- Function: decimal.getcontext ()
     Return the current context for the active thread.

 -- Function: decimal.setcontext (c)
     Set the current context for the active thread to _c_.

  Beginning with Python 2.5, you can also use the with.
statement and the localcontext() function to temporarily
change the active context.

 -- Function: decimal.localcontext ([c])
     Return a context manager that will set the current context for the
     active thread to a copy of _c_ on entry to the with-statement and
     restore the previous context when exiting the with-statement. If
     no context is specified, a copy of the current context is used.

     New in version 2.5.

     For example, the following code sets the current decimal precision
     to 42 places, performs a calculation, and then automatically
     restores the previous context:

         from decimal import localcontext

         with localcontext() as ctx:
             ctx.prec = 42   # Perform a high precision calculation
             s = calculate_something()
         s = +s  # Round the final result back to the default precision

  New contexts can also be created using the Context.
constructor described below. In addition, the module provides three
pre-made contexts:

 -- Class: decimal.BasicContext
     This is a standard context defined by the General Decimal
     Arithmetic Specification.  Precision is set to nine.  Rounding is
     set to `ROUND_HALF_UP'.  All flags are cleared.  All traps are
     enabled (treated as exceptions) except Inexact,
     Rounded, and Subnormal.

     Because many of the traps are enabled, this context is useful for
     debugging.

 -- Class: decimal.ExtendedContext
     This is a standard context defined by the General Decimal
     Arithmetic Specification.  Precision is set to nine.  Rounding is
     set to `ROUND_HALF_EVEN'.  All flags are cleared.  No traps are
     enabled (so that exceptions are not raised during computations).

     Because the traps are disabled, this context is useful for
     applications that prefer to have result value of `NaN' or
     `Infinity' instead of raising exceptions.  This allows an
     application to complete a run in the presence of conditions that
     would otherwise halt the program.

 -- Class: decimal.DefaultContext
     This context is used by the Context constructor as a
     prototype for new contexts.  Changing a field (such a precision)
     has the effect of changing the default for new contexts created by
     the Context constructor.

     This context is most useful in multi-threaded environments.
     Changing one of the fields before threads are started has the
     effect of setting system-wide defaults.  Changing the fields after
     threads have started is not recommended as it would require thread
     synchronization to prevent race conditions.

     In single threaded environments, it is preferable to not use this
     context at all.  Instead, simply create contexts explicitly as
     described below.

     The default values are precision=28, rounding=ROUND_HALF_EVEN, and
     enabled traps for Overflow, InvalidOperation, and DivisionByZero.

  In addition to the three supplied contexts, new contexts can be
created with the Context constructor.

 -- Class: decimal.Context (prec=None, rounding=None, traps=None,
          flags=None, Emin=None, Emax=None, capitals=1)
     Creates a new context.  If a field is not specified or is
     None, the default values are copied from the
     DefaultContext.  If the _flags_ field is not specified or is
     None, all flags are cleared.

     The _prec_ field is a positive integer that sets the precision for
     arithmetic operations in the context.

     The _rounding_ option is one of:

        * `ROUND_CEILING' (towards `Infinity'),

        * `ROUND_DOWN' (towards zero),

        * `ROUND_FLOOR' (towards `-Infinity'),

        * `ROUND_HALF_DOWN' (to nearest with ties going towards zero),

        * `ROUND_HALF_EVEN' (to nearest with ties going to nearest even
          integer),

        * `ROUND_HALF_UP' (to nearest with ties going away from zero),
          or

        * `ROUND_UP' (away from zero).

        * `ROUND_05UP' (away from zero if last digit after rounding
          towards zero would have been 0 or 5; otherwise towards zero)

     The _traps_ and _flags_ fields list any signals to be set.
     Generally, new contexts should only set traps and leave the flags
     clear.

     The _Emin_ and _Emax_ fields are integers specifying the outer
     limits allowable for exponents.

     The _capitals_ field is either `0' or `1' (the default). If set to
     `1', exponents are printed with a capital `E'; otherwise, a
     lowercase `e' is used: `Decimal('6.02e+23')'.

     Changed in version 2.6: The `ROUND_05UP' rounding mode was added.

     The Context class defines several general purpose
     methods as well as a large number of methods for doing arithmetic
     directly in a given context.  In addition, for each of the
     Decimal methods described above (with the exception of the
     `adjusted()' and `as_tuple()' methods) there is a corresponding
     Context method.  For example, for a Context.
     instance `C' and Decimal instance `x', `C.exp(x)' is
     equivalent to `x.exp(context=C)'.  Each Context method
     accepts a Python integer (an instance of int or
     long) anywhere that a Decimal instance is accepted.

      -- Method: clear_flags ()
          Resets all of the flags to `0'.

      -- Method: copy ()
          Return a duplicate of the context.

      -- Method: copy_decimal (num)
          Return a copy of the Decimal instance num.

      -- Method: create_decimal (num)
          Creates a new Decimal instance from _num_ but using _self_ as
          context. Unlike the Decimal constructor, the
          context precision, rounding method, flags, and traps are
          applied to the conversion.

          This is useful because constants are often given to a greater
          precision than is needed by the application.  Another benefit
          is that rounding immediately eliminates unintended effects
          from digits beyond the current precision. In the following
          example, using unrounded inputs means that adding zero to a
          sum can change the result:

              >>> getcontext().prec = 3
              >>> Decimal('3.4445') + Decimal('1.0023')
              Decimal('4.45')
              >>> Decimal('3.4445') + Decimal(0) + Decimal('1.0023')
              Decimal('4.44')

          This method implements the to-number operation of the IBM
          specification.  If the argument is a string, no leading or
          trailing whitespace is permitted.

      -- Method: create_decimal_from_float (f)
          Creates a new Decimal instance from a float _f_ but rounding
          using _self_ as the context.  Unlike the
          Decimal.from_float() class method, the context
          precision, rounding method, flags, and traps are applied to
          the conversion.

              >>> context = Context(prec=5, rounding=ROUND_DOWN)
              >>> context.create_decimal_from_float(math.pi)
              Decimal('3.1415')
              >>> context = Context(prec=5, traps=[Inexact])
              >>> context.create_decimal_from_float(math.pi)
              Traceback (most recent call last):
                  ...
              Inexact: None

          New in version 2.7.

      -- Method: Etiny ()
          Returns a value equal to `Emin - prec + 1' which is the
          minimum exponent value for subnormal results.  When underflow
          occurs, the exponent is set to Etiny.

      -- Method: Etop ()
          Returns a value equal to `Emax - prec + 1'.

     The usual approach to working with decimals is to create
     Decimal.  instances and then apply arithmetic operations
     which take place within the current context for the active thread.
     An alternative approach is to use context methods for calculating
     within a specific context.  The methods are similar to those for
     the Decimal class and are only briefly recounted here.

      -- Method: abs (x)
          Returns the absolute value of _x_.

      -- Method: add (x, y)
          Return the sum of _x_ and _y_.

      -- Method: canonical (x)
          Returns the same Decimal object _x_.

      -- Method: compare (x, y)
          Compares _x_ and _y_ numerically.

      -- Method: compare_signal (x, y)
          Compares the values of the two operands numerically.

      -- Method: compare_total (x, y)
          Compares two operands using their abstract representation.

      -- Method: compare_total_mag (x, y)
          Compares two operands using their abstract representation,
          ignoring sign.

      -- Method: copy_abs (x)
          Returns a copy of _x_ with the sign set to 0.

      -- Method: copy_negate (x)
          Returns a copy of _x_ with the sign inverted.

      -- Method: copy_sign (x, y)
          Copies the sign from _y_ to _x_.

      -- Method: divide (x, y)
          Return _x_ divided by _y_.

      -- Method: divide_int (x, y)
          Return _x_ divided by _y_, truncated to an integer.

      -- Method: divmod (x, y)
          Divides two numbers and returns the integer part of the
          result.

      -- Method: exp (x)
          Returns `e ** x'.

      -- Method: fma (x, y, z)
          Returns _x_ multiplied by _y_, plus _z_.

      -- Method: is_canonical (x)
          Returns True if _x_ is canonical; otherwise returns False.

      -- Method: is_finite (x)
          Returns True if _x_ is finite; otherwise returns False.

      -- Method: is_infinite (x)
          Returns True if _x_ is infinite; otherwise returns False.

      -- Method: is_nan (x)
          Returns True if _x_ is a qNaN or sNaN; otherwise returns
          False.

      -- Method: is_normal (x)
          Returns True if _x_ is a normal number; otherwise returns
          False.

      -- Method: is_qnan (x)
          Returns True if _x_ is a quiet NaN; otherwise returns False.

      -- Method: is_signed (x)
          Returns True if _x_ is negative; otherwise returns False.

      -- Method: is_snan (x)
          Returns True if _x_ is a signaling NaN; otherwise returns
          False.

      -- Method: is_subnormal (x)
          Returns True if _x_ is subnormal; otherwise returns False.

      -- Method: is_zero (x)
          Returns True if _x_ is a zero; otherwise returns False.

      -- Method: ln (x)
          Returns the natural (base e) logarithm of _x_.

      -- Method: log10 (x)
          Returns the base 10 logarithm of _x_.

      -- Method: logb (x)
          Returns the exponent of the magnitude of the operand's MSD.

      -- Method: logical_and (x, y)
          Applies the logical operation _and_ between each operand's
          digits.

      -- Method: logical_invert (x)
          Invert all the digits in _x_.

      -- Method: logical_or (x, y)
          Applies the logical operation _or_ between each operand's
          digits.

      -- Method: logical_xor (x, y)
          Applies the logical operation _xor_ between each operand's
          digits.

      -- Method: max (x, y)
          Compares two values numerically and returns the maximum.

      -- Method: max_mag (x, y)
          Compares the values numerically with their sign ignored.

      -- Method: min (x, y)
          Compares two values numerically and returns the minimum.

      -- Method: min_mag (x, y)
          Compares the values numerically with their sign ignored.

      -- Method: minus (x)
          Minus corresponds to the unary prefix minus operator in
          Python.

      -- Method: multiply (x, y)
          Return the product of _x_ and _y_.

      -- Method: next_minus (x)
          Returns the largest representable number smaller than _x_.

      -- Method: next_plus (x)
          Returns the smallest representable number larger than _x_.

      -- Method: next_toward (x, y)
          Returns the number closest to _x_, in direction towards _y_.

      -- Method: normalize (x)
          Reduces _x_ to its simplest form.

      -- Method: number_class (x)
          Returns an indication of the class of _x_.

      -- Method: plus (x)
          Plus corresponds to the unary prefix plus operator in Python.
          This operation applies the context precision and rounding, so
          it is _not_ an identity operation.

      -- Method: power (x, y[, modulo])
          Return `x' to the power of `y', reduced modulo `modulo' if
          given.

          With two arguments, compute `x**y'.  If `x' is negative then
          `y' must be integral.  The result will be inexact unless `y'
          is integral and the result is finite and can be expressed
          exactly in 'precision' digits.  The result should always be
          correctly rounded, using the rounding mode of the current
          thread's context.

          With three arguments, compute `(x**y) % modulo'.  For the
          three argument form, the following restrictions on the
          arguments hold:

                  - all three arguments must be integral

                  - `y' must be nonnegative

                  - at least one of `x' or `y' must be nonzero

                  - `modulo' must be nonzero and have at most
                    'precision' digits

          The value resulting from `Context.power(x, y, modulo)' is
          equal to the value that would be obtained by computing `(x**y)
          % modulo' with unbounded precision, but is computed more
          efficiently.  The exponent of the result is zero, regardless
          of the exponents of `x', `y' and `modulo'.  The result is
          always exact.

          Changed in version 2.6: `y' may now be nonintegral in `x**y'.
          Stricter requirements for the three-argument version.

      -- Method: quantize (x, y)
          Returns a value equal to _x_ (rounded), having the exponent
          of _y_.

      -- Method: radix ()
          Just returns 10, as this is Decimal, :)

      -- Method: remainder (x, y)
          Returns the remainder from integer division.

          The sign of the result, if non-zero, is the same as that of
          the original dividend.

      -- Method: remainder_near (x, y)
          Returns `x - y * n', where _n_ is the integer nearest the
          exact value of `x / y' (if the result is 0 then its sign will
          be the sign of _x_).

      -- Method: rotate (x, y)
          Returns a rotated copy of _x_, _y_ times.

      -- Method: same_quantum (x, y)
          Returns True if the two operands have the same exponent.

      -- Method: scaleb (x, y)
          Returns the first operand after adding the second value its
          exp.

      -- Method: shift (x, y)
          Returns a shifted copy of _x_, _y_ times.

      -- Method: sqrt (x)
          Square root of a non-negative number to context precision.

      -- Method: subtract (x, y)
          Return the difference between _x_ and _y_.

      -- Method: to_eng_string (x)
          Converts a number to a string, using scientific notation.

      -- Method: to_integral_exact (x)
          Rounds to an integer.

      -- Method: to_sci_string (x)
          Converts a number to a string using scientific notation.

5.9.4.5 Signals
...............

Signals represent conditions that arise during computation. Each
corresponds to one context flag and one context trap enabler.

  The context flag is set whenever the condition is encountered. After
the computation, flags may be checked for informational purposes (for
instance, to determine whether a computation was exact). After checking
the flags, be sure to clear all flags before starting the next
computation.

  If the context's trap enabler is set for the signal, then the
condition causes a Python exception to be raised.  For example, if the
DivisionByZero trap is set, then a DivisionByZero
exception is raised upon encountering the condition.

 -- Class: decimal.Clamped
     Altered an exponent to fit representation constraints.

     Typically, clamping occurs when an exponent falls outside the
     context's `Emin' and `Emax' limits.  If possible, the exponent is
     reduced to fit by adding zeros to the coefficient.

 -- Class: decimal.DecimalException
     Base class for other signals and a subclass of
     ArithmeticError.

 -- Class: decimal.DivisionByZero
     Signals the division of a non-infinite number by zero.

     Can occur with division, modulo division, or when raising a number
     to a negative power.  If this signal is not trapped, returns
     `Infinity' or `-Infinity' with the sign determined by the inputs
     to the calculation.

 -- Class: decimal.Inexact
     Indicates that rounding occurred and the result is not exact.

     Signals when non-zero digits were discarded during rounding. The
     rounded result is returned.  The signal flag or trap is used to
     detect when results are inexact.

 -- Class: decimal.InvalidOperation
     An invalid operation was performed.

     Indicates that an operation was requested that does not make
     sense. If not trapped, returns `NaN'.  Possible causes include:

         Infinity - Infinity
         0 * Infinity
         Infinity / Infinity
         x % 0
         Infinity % x
         x._rescale( non-integer )
         sqrt(-x) and x > 0
         0 ** 0
         x ** (non-integer)
         x ** Infinity

 -- Class: decimal.Overflow
     Numerical overflow.

     Indicates the exponent is larger than `Emax' after rounding has
     occurred.  If not trapped, the result depends on the rounding
     mode, either pulling inward to the largest representable finite
     number or rounding outward to `Infinity'.  In either case, see
     Inexact and Rounded.  are also signaled.

 -- Class: decimal.Rounded
     Rounding occurred though possibly no information was lost.

     Signaled whenever rounding discards digits; even if those digits
     are zero (such as rounding `5.00' to `5.0').  If not trapped,
     returns the result unchanged.  This signal is used to detect loss
     of significant digits.

 -- Class: decimal.Subnormal
     Exponent was lower than `Emin' prior to rounding.

     Occurs when an operation result is subnormal (the exponent is too
     small). If not trapped, returns the result unchanged.

 -- Class: decimal.Underflow
     Numerical underflow with result rounded to zero.

     Occurs when a subnormal result is pushed to zero by rounding.
     See Inexact.  and Subnormal are also signaled.

  The following table summarizes the hierarchy of signals:

    exceptions.ArithmeticError(exceptions.StandardError)
        DecimalException
            Clamped
            DivisionByZero(DecimalException, exceptions.ZeroDivisionError)
            Inexact
                Overflow(Inexact, Rounded)
                Underflow(Inexact, Rounded, Subnormal)
            InvalidOperation
            Rounded
            Subnormal

5.9.4.6 Floating Point Notes
............................

5.9.4.7 Mitigating round-off error with increased precision
...........................................................

The use of decimal floating point eliminates decimal representation
error (making it possible to represent `0.1' exactly); however, some
operations can still incur round-off error when non-zero digits exceed
the fixed precision.

  The effects of round-off error can be amplified by the addition or
subtraction of nearly offsetting quantities resulting in loss of
significance.  Knuth provides two instructive examples where rounded
floating point arithmetic with insufficient precision causes the
breakdown of the associative and distributive properties of addition:

    # Examples from Seminumerical Algorithms, Section 4.2.2.
    >>> from decimal import Decimal, getcontext
    >>> getcontext().prec = 8

    >>> u, v, w = Decimal(11111113), Decimal(-11111111), Decimal('7.51111111')
    >>> (u + v) + w
    Decimal('9.5111111')
    >>> u + (v + w)
    Decimal('10')

    >>> u, v, w = Decimal(20000), Decimal(-6), Decimal('6.0000003')
    >>> (u*v) + (u*w)
    Decimal('0.01')
    >>> u * (v+w)
    Decimal('0.0060000')

The decimal module makes it possible to restore the
identities by expanding the precision sufficiently to avoid loss of
significance:

    >>> getcontext().prec = 20
    >>> u, v, w = Decimal(11111113), Decimal(-11111111), Decimal('7.51111111')
    >>> (u + v) + w
    Decimal('9.51111111')
    >>> u + (v + w)
    Decimal('9.51111111')
    >>>
    >>> u, v, w = Decimal(20000), Decimal(-6), Decimal('6.0000003')
    >>> (u*v) + (u*w)
    Decimal('0.0060000')
    >>> u * (v+w)
    Decimal('0.0060000')

5.9.4.8 Special values
......................

The number system for the decimal module provides special
values including `NaN', `sNaN', `-Infinity', `Infinity', and two zeros,
`+0' and `-0'.

  Infinities can be constructed directly with:  `Decimal('Infinity')'.
Also, they can arise from dividing by zero when the
DivisionByZero signal is not trapped.  Likewise, when the
Overflow signal is not trapped, infinity can result from rounding
beyond the limits of the largest representable number.

  The infinities are signed (affine) and can be used in arithmetic
operations where they get treated as very large, indeterminate numbers.
For instance, adding a constant to infinity gives another infinite
result.

  Some operations are indeterminate and return `NaN', or if the
InvalidOperation signal is trapped, raise an exception.  For
example, `0/0' returns `NaN' which means "not a number".  This variety
of `NaN' is quiet and, once created, will flow through other
computations always resulting in another `NaN'.  This behavior can be
useful for a series of computations that occasionally have missing
inputs -- it allows the calculation to proceed while flagging specific
results as invalid.

  A variant is `sNaN' which signals rather than remaining quiet after
every operation.  This is a useful return value when an invalid result
needs to interrupt a calculation for special handling.

  The behavior of Python's comparison operators can be a little
surprising where a `NaN' is involved.  A test for equality where one of
the operands is a quiet or signaling `NaN' always returns False.
(even when doing `Decimal('NaN')==Decimal('NaN')'), while a test
for inequality always returns True.  An attempt to compare
two Decimals using any of the `<', `<=', `>' or `>=' operators will
raise the InvalidOperation signal if either operand is a
`NaN', and return False if this signal is not trapped.
Note that the General Decimal Arithmetic specification does not specify
the behavior of direct comparisons; these rules for comparisons
involving a `NaN' were taken from the IEEE 854 standard (see Table 3 in
section 5.7).  To ensure strict standards-compliance, use the
`compare()' and `compare-signal()' methods instead.

  The signed zeros can result from calculations that underflow. They
keep the sign that would have resulted if the calculation had been
carried out to greater precision.  Since their magnitude is zero, both
positive and negative zeros are treated as equal and their sign is
informational.

  In addition to the two signed zeros which are distinct yet equal,
there are various representations of zero with differing precisions yet
equivalent in value.  This takes a bit of getting used to.  For an eye
accustomed to normalized floating point representations, it is not
immediately obvious that the following calculation returns a value
equal to zero:

    >>> 1 / Decimal('Infinity')
    Decimal('0E-1000000026')

5.9.4.9 Working with threads
............................

The getcontext() function accesses a different
Context object for each thread.  Having separate thread contexts
means that threads may make changes (such as `getcontext.prec=10')
without interfering with other threads.

  Likewise, the setcontext() function automatically assigns
its target to the current thread.

  If setcontext() has not been called before
getcontext(), then getcontext() will automatically
create a new context for use in the current thread.

  The new context is copied from a prototype context called
_DefaultContext_. To control the defaults so that each thread will use
the same values throughout the application, directly modify the
_DefaultContext_ object. This should be done _before_ any threads are
started so that there won't be a race condition between threads calling
getcontext(). For example:

    # Set applicationwide defaults for all threads about to be launched
    DefaultContext.prec = 12
    DefaultContext.rounding = ROUND_DOWN
    DefaultContext.traps = ExtendedContext.traps.copy()
    DefaultContext.traps[InvalidOperation] = 1
    setcontext(DefaultContext)

    # Afterwards, the threads can be started
    t1.start()
    t2.start()
    t3.start()
     . . .

5.9.4.10 Recipes
................

Here are a few recipes that serve as utility functions and that
demonstrate ways to work with the Decimal class:

    def moneyfmt(value, places=2, curr='', sep=',', dp='.',
                 pos='', neg='-', trailneg=''):
        """Convert Decimal to a money formatted string.

        places:  required number of places after the decimal point
        curr:    optional currency symbol before the sign (may be blank)
        sep:     optional grouping separator (comma, period, space, or blank)
        dp:      decimal point indicator (comma or period)
                 only specify as blank when places is zero
        pos:     optional sign for positive numbers: '+', space or blank
        neg:     optional sign for negative numbers: '-', '(', space or blank
        trailneg:optional trailing minus indicator:  '-', ')', space or blank

        >>> d = Decimal('-1234567.8901')
        >>> moneyfmt(d, curr='$')
        '-$1,234,567.89'
        >>> moneyfmt(d, places=0, sep='.', dp='', neg='', trailneg='-')
        '1.234.568-'
        >>> moneyfmt(d, curr='$', neg='(', trailneg=')')
        '($1,234,567.89)'
        >>> moneyfmt(Decimal(123456789), sep=' ')
        '123 456 789.00'
        >>> moneyfmt(Decimal('-0.02'), neg='<', trailneg='>')
        '<0.02>'

        """
        q = Decimal(10) ** -places      # 2 places --> '0.01'
        sign, digits, exp = value.quantize(q).as_tuple()
        result = []
        digits = map(str, digits)
        build, next = result.append, digits.pop
        if sign:
            build(trailneg)
        for i in range(places):
            build(next() if digits else '0')
        build(dp)
        if not digits:
            build('0')
        i = 0
        while digits:
            build(next())
            i += 1
            if i == 3 and digits:
                i = 0
                build(sep)
        build(curr)
        build(neg if sign else pos)
        return ''.join(reversed(result))

    def pi():
        """Compute Pi to the current precision.

        >>> print pi()
        3.141592653589793238462643383

        """
        getcontext().prec += 2  # extra digits for intermediate steps
        three = Decimal(3)      # substitute "three=3.0" for regular floats
        lasts, t, s, n, na, d, da = 0, three, 3, 1, 0, 0, 24
        while s != lasts:
            lasts = s
            n, na = n+na, na+8
            d, da = d+da, da+32
            t = (t * n) / d
            s += t
        getcontext().prec -= 2
        return +s               # unary plus applies the new precision

    def exp(x):
        """Return e raised to the power of x.  Result type matches input type.

        >>> print exp(Decimal(1))
        2.718281828459045235360287471
        >>> print exp(Decimal(2))
        7.389056098930650227230427461
        >>> print exp(2.0)
        7.38905609893
        >>> print exp(2+0j)
        (7.38905609893+0j)

        """
        getcontext().prec += 2
        i, lasts, s, fact, num = 0, 0, 1, 1, 1
        while s != lasts:
            lasts = s
            i += 1
            fact *= i
            num *= x
            s += num / fact
        getcontext().prec -= 2
        return +s

    def cos(x):
        """Return the cosine of x as measured in radians.

        >>> print cos(Decimal('0.5'))
        0.8775825618903727161162815826
        >>> print cos(0.5)
        0.87758256189
        >>> print cos(0.5+0j)
        (0.87758256189+0j)

        """
        getcontext().prec += 2
        i, lasts, s, fact, num, sign = 0, 0, 1, 1, 1, 1
        while s != lasts:
            lasts = s
            i += 2
            fact *= i * (i-1)
            num *= x * x
            sign *= -1
            s += num / fact * sign
        getcontext().prec -= 2
        return +s

    def sin(x):
        """Return the sine of x as measured in radians.

        >>> print sin(Decimal('0.5'))
        0.4794255386042030002732879352
        >>> print sin(0.5)
        0.479425538604
        >>> print sin(0.5+0j)
        (0.479425538604+0j)

        """
        getcontext().prec += 2
        i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
        while s != lasts:
            lasts = s
            i += 2
            fact *= i * (i-1)
            num *= x * x
            sign *= -1
            s += num / fact * sign
        getcontext().prec -= 2
        return +s

5.9.4.11 Decimal FAQ
....................

Q. It is cumbersome to type `decimal.Decimal('1234.5')'.  Is there a
way to minimize typing when using the interactive interpreter?

  A. Some users abbreviate the constructor to just a single letter:

    >>> D = decimal.Decimal
    >>> D('1.23') + D('3.45')
    Decimal('4.68')

Q. In a fixed-point application with two decimal places, some inputs
have many places and need to be rounded.  Others are not supposed to
have excess digits and need to be validated.  What methods should be
used?

  A. The `quantize()' method rounds to a fixed number of decimal
places. If the Inexact trap is set, it is also useful for
validation:

    >>> TWOPLACES = Decimal(10) ** -2       # same as Decimal('0.01')

    >>> # Round to two places
    >>> Decimal('3.214').quantize(TWOPLACES)
    Decimal('3.21')

    >>> # Validate that a number does not exceed two places
    >>> Decimal('3.21').quantize(TWOPLACES, context=Context(traps=[Inexact]))
    Decimal('3.21')

    >>> Decimal('3.214').quantize(TWOPLACES, context=Context(traps=[Inexact]))
    Traceback (most recent call last):
       ...
    Inexact: None

Q. Once I have valid two place inputs, how do I maintain that invariant
throughout an application?

  A. Some operations like addition, subtraction, and multiplication by
an integer will automatically preserve fixed point.  Others operations,
like division and non-integer multiplication, will change the number of
decimal places and need to be followed-up with a `quantize()' step:

    >>> a = Decimal('102.72')           # Initial fixed-point values
    >>> b = Decimal('3.17')
    >>> a + b                           # Addition preserves fixed-point
    Decimal('105.89')
    >>> a - b
    Decimal('99.55')
    >>> a * 42                          # So does integer multiplication
    Decimal('4314.24')
    >>> (a * b).quantize(TWOPLACES)     # Must quantize non-integer multiplication
    Decimal('325.62')
    >>> (b / a).quantize(TWOPLACES)     # And quantize division
    Decimal('0.03')

In developing fixed-point applications, it is convenient to define
functions to handle the `quantize()' step:

    >>> def mul(x, y, fp=TWOPLACES):
    ...     return (x * y).quantize(fp)
    >>> def div(x, y, fp=TWOPLACES):
    ...     return (x / y).quantize(fp)

    >>> mul(a, b)                       # Automatically preserve fixed-point
    Decimal('325.62')
    >>> div(b, a)
    Decimal('0.03')

Q. There are many ways to express the same value.  The numbers `200',
`200.000', `2E2', and `02E+4' all have the same value at various
precisions. Is there a way to transform them to a single recognizable
canonical value?

  A. The `normalize()' method maps all equivalent values to a single
representative:

    >>> values = map(Decimal, '200 200.000 2E2 .02E+4'.split())
    >>> [v.normalize() for v in values]
    [Decimal('2E+2'), Decimal('2E+2'), Decimal('2E+2'), Decimal('2E+2')]

Q. Some decimal values always print with exponential notation.  Is
there a way to get a non-exponential representation?

  A. For some values, exponential notation is the only way to express
the number of significant places in the coefficient.  For example,
expressing `5.0E+3' as `5000' keeps the value constant but cannot show
the original's two-place significance.

  If an application does not care about tracking significance, it is
easy to remove the exponent and trailing zeros, losing significance,
but keeping the value unchanged:

    def remove_exponent(d):
        '''Remove exponent and trailing zeros.

        >>> remove_exponent(Decimal('5E+3'))
        Decimal('5000')

        '''
        return d.quantize(Decimal(1)) if d == d.to_integral() else d.normalize()

Q. Is there a way to convert a regular float to a Decimal.?

  A. Yes, any binary floating point number can be exactly expressed as a
Decimal though an exact conversion may take more precision than
intuition would suggest:

    >>> Decimal(math.pi)
    Decimal('3.141592653589793115997963468544185161590576171875')

Q. Within a complex calculation, how can I make sure that I haven't
gotten a spurious result because of insufficient precision or rounding
anomalies.

  A. The decimal module makes it easy to test results.  A best practice
is to re-run calculations using greater precision and with various
rounding modes.  Widely differing results indicate insufficient
precision, rounding mode issues, ill-conditioned inputs, or a
numerically unstable algorithm.

  Q. I noticed that context precision is applied to the results of
operations but not to the inputs.  Is there anything to watch out for
when mixing values of different precisions?

  A. Yes.  The principle is that all values are considered to be exact
and so is the arithmetic on those values.  Only the results are
rounded.  The advantage for inputs is that "what you type is what you
get".  A disadvantage is that the results can look odd if you forget
that the inputs haven't been rounded:

    >>> getcontext().prec = 3
    >>> Decimal('3.104') + Decimal('2.104')
    Decimal('5.21')
    >>> Decimal('3.104') + Decimal('0.000') + Decimal('2.104')
    Decimal('5.20')

The solution is either to increase precision or to force rounding of
inputs using the unary plus operation:

    >>> getcontext().prec = 3
    >>> +Decimal('1.23456789')      # unary plus triggers rounding
    Decimal('1.23')

Alternatively, inputs can be rounded upon creation using the
Context.create_decimal() method:

    >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678')
    Decimal('1.2345')

5.9.5 `fractions' -- Rational numbers
-------------------------------------

New in version 2.6.

  The fractions module provides support for rational number
arithmetic.

  A Fraction instance can be constructed from a pair of integers, from
another rational number, or from a string.

 -- Class: fractions.Fraction (numerator=0, denominator=1)
 -- Class: fractions.Fraction (other_fraction)
 -- Class: fractions.Fraction (float)
 -- Class: fractions.Fraction (decimal)
 -- Class: fractions.Fraction (string)
     The first version requires that _numerator_ and _denominator_ are
     instances of numbers.Rational and returns a new
     Fraction instance with value `numerator/denominator'. If
     _denominator_ is `0', it raises a ZeroDivisionError.
     The second version requires that _other_fraction_ is an instance
     of numbers.Rational and returns a Fraction.
     instance with the same value.  The next two versions accept either
     a float or a decimal.Decimal instance, and
     return a Fraction instance with exactly the same
     value.  Note that due to the usual issues with binary
     floating-point (see Floating Point Arithmetic; Issues and Limitations),
     the argument to `Fraction(1.1)' is not exactly
     equal to 11/10, and so `Fraction(1.1)' does _not_ return
     `Fraction(11, 10)' as one might expect.  (But see the
     documentation for the limit_denominator() method
     below.)  The last version of the constructor expects a string or
     unicode instance.  The usual form for this instance is:

         [sign] numerator ['/' denominator]

     where the optional `sign' may be either '+' or '-' and `numerator'
     and `denominator' (if present) are strings of decimal digits.  In
     addition, any string that represents a finite value and is
     accepted by the float constructor is also accepted by
     the Fraction constructor.  In either form the input
     string may also have leading and/or trailing whitespace.  Here are
     some examples:

         >>> from fractions import Fraction
         >>> Fraction(16, -10)
         Fraction(-8, 5)
         >>> Fraction(123)
         Fraction(123, 1)
         >>> Fraction()
         Fraction(0, 1)
         >>> Fraction('3/7')
         Fraction(3, 7)
         [40794 refs]
         >>> Fraction(' -3/7 ')
         Fraction(-3, 7)
         >>> Fraction('1.414213 \t\n')
         Fraction(1414213, 1000000)
         >>> Fraction('-.125')
         Fraction(-1, 8)
         >>> Fraction('7e-6')
         Fraction(7, 1000000)
         >>> Fraction(2.25)
         Fraction(9, 4)
         >>> Fraction(1.1)
         Fraction(2476979795053773, 2251799813685248)
         >>> from decimal import Decimal
         >>> Fraction(Decimal('1.1'))
         Fraction(11, 10)

     The Fraction class inherits from the abstract base
     class numbers.Rational, and implements all of the
     methods and operations from that class.  See Fraction.
     instances are hashable, and should be treated as immutable.  In
     addition, see Fraction has the following methods:

     Changed in version 2.7: The Fraction constructor now
     accepts float and decimal.Decimal.
     instances.

      -- Method: from_float (flt)
          This class method constructs a Fraction.
          representing the exact value of _flt_, which must be a
          float. Beware that `Fraction.from_float(0.3)' is not the
          same value as `Fraction(3, 10)'

               Note: From Python 2.7 onwards, you can also construct a
               Fraction instance directly from a
               float.

      -- Method: from_decimal (dec)
          This class method constructs a Fraction.
          representing the exact value of _dec_, which must be a
          decimal.Decimal.

               Note: From Python 2.7 onwards, you can also construct a
               Fraction instance directly from a
               decimal.Decimal.  instance.

      -- Method: limit_denominator (max_denominator=1000000)
          Finds and returns the closest Fraction to `self'
          that has denominator at most max_denominator.  This method is
          useful for finding rational approximations to a given
          floating-point number:

              >>> from fractions import Fraction
              >>> Fraction('3.1415926535897932').limit_denominator(1000)
              Fraction(355, 113)

          or for recovering a rational number that's represented as a
          float:

              >>> from math import pi, cos
              >>> Fraction(cos(pi/3))
              Fraction(4503599627370497, 9007199254740992)
              >>> Fraction(cos(pi/3)).limit_denominator()
              Fraction(1, 2)
              >>> Fraction(1.1).limit_denominator()
              Fraction(11, 10)

 -- Function: fractions.gcd (a, b)
     Return the greatest common divisor of the integers _a_ and _b_.
     If either _a_ or _b_ is nonzero, then the absolute value of
     `gcd(a, b)' is the largest integer that divides both _a_ and _b_.
     `gcd(a,b)' has the same sign as _b_ if _b_ is nonzero; otherwise
     it takes the sign of _a_.  `gcd(0, 0)' returns `0'.

Module numbers.
     The abstract base classes making up the numeric tower.

5.9.6 `random' -- Generate pseudo-random numbers
------------------------------------------------

This module implements pseudo-random number generators for various
distributions.

  Latest version of the random module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/random.py?view=markup)For
integers, uniform selection from a range. For sequences, uniform
selection of a random element, a function to generate a random
permutation of a list in-place, and a function for random sampling
without replacement.

  On the real line, there are functions to compute uniform, normal
(Gaussian), lognormal, negative exponential, gamma, and beta
distributions. For generating distributions of angles, the von Mises
distribution is available.

  Almost all module functions depend on the basic function
random(), which generates a random float uniformly in the
semi-open range [0.0, 1.0).  Python uses the Mersenne Twister as the
core generator.  It produces 53-bit precision floats and has a period
of 2**19937-1.  The underlying implementation in C is both fast and
threadsafe.  The Mersenne Twister is one of the most extensively tested
random number generators in existence.  However, being completely
deterministic, it is not suitable for all purposes, and is completely
unsuitable for cryptographic purposes.

  The functions supplied by this module are actually bound methods of a
hidden instance of the `random.Random' class.  You can instantiate your
own instances of `Random' to get generators that don't share state.
This is especially useful for multi-threaded programs, creating a
different instance of `Random' for each thread, and using the
jumpahead() method to make it likely that the generated sequences
seen by each thread don't overlap.

  Class `Random' can also be subclassed if you want to use a different
basic generator of your own devising: in that case, override the
random(), seed(), getstate(),
setstate() and jumpahead() methods.  Optionally, a
new generator can supply a getrandbits() method -- this
allows randrange() to produce selections over an
arbitrarily large range.

  New in version 2.4: the getrandbits() method.

  As an example of subclassing, the random module provides
the WichmannHill class that implements an alternative
generator in pure Python.  The class provides a backward compatible way
to reproduce results from earlier versions of Python, which used the
Wichmann-Hill algorithm as the core generator.  Note that this
Wichmann-Hill generator can no longer be recommended: its period is too
short by contemporary standards, and the sequence generated is known to
fail some stringent randomness tests.  See the references below for a
recent variant that repairs these flaws.

  Changed in version 2.3: MersenneTwister replaced Wichmann-Hill as the
default generator.

  The random module also provides the SystemRandom
class which uses the system function os.urandom() to
generate random numbers from sources provided by the operating system.

  Bookkeeping functions:

 -- Function: random.seed ([x])
     Initialize the basic random number generator. Optional argument
     _x_ can be any hashable object. If _x_ is omitted or
     `None', current system time is used; current system time is also
     used to initialize the generator when the module is first
     imported.  If randomness sources are provided by the operating
     system, they are used instead of the system time (see the
     os.urandom() function for details on availability).

     Changed in version 2.4: formerly, operating system resources were
     not used.

 -- Function: random.getstate ()
     Return an object capturing the current internal state of the
     generator.  This object can be passed to setstate() to
     restore the state.

     New in version 2.1.

     Changed in version 2.6: State values produced in Python 2.6 cannot
     be loaded into earlier versions.

 -- Function: random.setstate (state)
     _state_ should have been obtained from a previous call to
     getstate(), and setstate() restores the internal
     state of the generator to what it was at the time
     setstate() was called.

     New in version 2.1.

 -- Function: random.jumpahead (n)
     Change the internal state to one different from and likely far
     away from the current state.  _n_ is a non-negative integer which
     is used to scramble the current state vector.  This is most useful
     in multi-threaded programs, in conjunction with multiple instances
     of the `Random' class: setstate() or seed()
     can be used to force all instances into the same internal
     state, and then jumpahead() can be used to force the
     instances' states far apart.

     New in version 2.1.

     Changed in version 2.3: Instead of jumping to a specific state,
     _n_ steps ahead, `jumpahead(n)' jumps to another state likely to
     be separated by many steps.

 -- Function: random.getrandbits (k)
     Returns a python long int with _k_ random bits. This
     method is supplied with the MersenneTwister generator and some
     other generators may also provide it as an optional part of the
     API. When available, see getrandbits() enables
     randrange() to handle arbitrarily large ranges.

     New in version 2.4.

  Functions for integers:

 -- Function: random.randrange ([start], stop[, step])
     Return a randomly selected element from `range(start, stop,
     step)'.  This is equivalent to `choice(range(start, stop, step))',
     but doesn't actually build a range object.

     New in version 1.5.2.

 -- Function: random.randint (a, b)
     Return a random integer _N_ such that `a <= N <= b'.

  Functions for sequences:

 -- Function: random.choice (seq)
     Return a random element from the non-empty sequence _seq_. If
     _seq_ is empty, raises IndexError.

 -- Function: random.shuffle (x[, random])
     Shuffle the sequence _x_ in place. The optional argument _random_
     is a 0-argument function returning a random float in [0.0, 1.0);
     by default, this is the function random().

     Note that for even rather small `len(x)', the total number of
     permutations of _x_ is larger than the period of most random
     number generators; this implies that most permutations of a long
     sequence can never be generated.

 -- Function: random.sample (population, k)
     Return a _k_ length list of unique elements chosen from the
     population sequence.  Used for random sampling without replacement.

     New in version 2.3.

     Returns a new list containing elements from the population while
     leaving the original population unchanged.  The resulting list is
     in selection order so that all sub-slices will also be valid
     random samples.  This allows raffle winners (the sample) to be
     partitioned into grand prize and second place winners (the
     subslices).

     Members of the population need not be hashable or
     unique.  If the population contains repeats, then each occurrence
     is a possible selection in the sample.

     To choose a sample from a range of integers, use an
     xrange() object as an argument.  This is especially fast and
     space efficient for sampling from a large population:
     `sample(xrange(10000000), 60)'.

  The following functions generate specific real-valued distributions.
Function parameters are named after the corresponding variables in the
distribution's equation, as used in common mathematical practice; most
of these equations can be found in any statistics text.

 -- Function: random.random ()
     Return the next random floating point number in the range [0.0,
     1.0).

 -- Function: random.uniform (a, b)
     Return a random floating point number _N_ such that `a <= N <= b'
     for `a <= b' and `b <= N <= a' for `b < a'.

     The end-point value `b' may or may not be included in the range
     depending on floating-point rounding in the equation `a + (b-a) *
     random()'.

 -- Function: random.triangular (low, high, mode)
     Return a random floating point number _N_ such that `low <= N <=
     high' and with the specified _mode_ between those bounds.  The
     _low_ and _high_ bounds default to zero and one.  The _mode_
     argument defaults to the midpoint between the bounds, giving a
     symmetric distribution.

     New in version 2.6.

 -- Function: random.betavariate (alpha, beta)
     Beta distribution.  Conditions on the parameters are `alpha > 0'
     and `beta > 0'. Returned values range between 0 and 1.

 -- Function: random.expovariate (lambd)
     Exponential distribution.  _lambd_ is 1.0 divided by the desired
     mean.  It should be nonzero.  (The parameter would be called
     "lambda", but that is a reserved word in Python.)  Returned values
     range from 0 to positive infinity if _lambd_ is positive, and from
     negative infinity to 0 if _lambd_ is negative.

 -- Function: random.gammavariate (alpha, beta)
     Gamma distribution.  (_Not_ the gamma function!)  Conditions on the
     parameters are `alpha > 0' and `beta > 0'.

 -- Function: random.gauss (mu, sigma)
     Gaussian distribution.  _mu_ is the mean, and _sigma_ is the
     standard deviation.  This is slightly faster than the
     normalvariate() function defined below.

 -- Function: random.lognormvariate (mu, sigma)
     Log normal distribution.  If you take the natural logarithm of this
     distribution, you'll get a normal distribution with mean _mu_ and
     standard deviation _sigma_.  _mu_ can have any value, and _sigma_
     must be greater than zero.

 -- Function: random.normalvariate (mu, sigma)
     Normal distribution.  _mu_ is the mean, and _sigma_ is the
     standard deviation.

 -- Function: random.vonmisesvariate (mu, kappa)
     _mu_ is the mean angle, expressed in radians between 0 and 2*_pi_,
     and _kappa_ is the concentration parameter, which must be greater
     than or equal to zero.  If _kappa_ is equal to zero, this
     distribution reduces to a uniform random angle over the range 0 to
     2*_pi_.

 -- Function: random.paretovariate (alpha)
     Pareto distribution.  _alpha_ is the shape parameter.

 -- Function: random.weibullvariate (alpha, beta)
     Weibull distribution.  _alpha_ is the scale parameter and _beta_
     is the shape parameter.

  Alternative Generators:

 -- Class: random.WichmannHill ([seed])
     Class that implements the Wichmann-Hill algorithm as the core
     generator. Has all of the same methods as `Random' plus the
     whseed() method described below.  Because this class is
     implemented in pure Python, it is not threadsafe and may require
     locks between calls.  The period of the generator is
     6,953,607,871,644 which is small enough to require care that two
     independent random sequences do not overlap.

 -- Function: random.whseed ([x])
     This is obsolete, supplied for bit-level compatibility with
     versions of Python prior to 2.1. See seed() for
     details.  whseed() does not guarantee that distinct
     integer arguments yield distinct internal states, and can yield no
     more than about 2**24 distinct internal states in all.

 -- Class: random.SystemRandom ([seed])
     Class that uses the os.urandom() function for
     generating random numbers from sources provided by the operating
     system. Not available on all systems.  Does not rely on software
     state and sequences are not reproducible. Accordingly, the
     seed() and jumpahead() methods have no effect
     and are ignored.  The getstate() and setstate()
     methods raise NotImplementedError if called.

     New in version 2.4.

  Examples of basic usage:

    >>> random.random()        # Random float x, 0.0 <= x < 1.0
    0.37444887175646646
    >>> random.uniform(1, 10)  # Random float x, 1.0 <= x < 10.0
    1.1800146073117523
    >>> random.randint(1, 10)  # Integer from 1 to 10, endpoints included
    7
    >>> random.randrange(0, 101, 2)  # Even integer from 0 to 100
    26
    >>> random.choice('abcdefghij')  # Choose a random element
    'c'

    >>> items = [1, 2, 3, 4, 5, 6, 7]
    >>> random.shuffle(items)
    >>> items
    [7, 3, 2, 5, 6, 4, 1]

    >>> random.sample([1, 2, 3, 4, 5],  3)  # Choose 3 elements
    [4, 1, 5]

M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally
equidistributed uniform pseudorandom number generator", ACM
Transactions on Modeling and Computer Simulation Vol. 8, No. 1, January
pp.3-30 1998.

  Wichmann, B. A. & Hill, I. D., "Algorithm AS 183: An efficient and
portable pseudo-random number generator", Applied Statistics 31 (1982)
188-190.

  Complementary-Multiply-with-Carry recipe
(http://code.activestate.com/recipes/576707/)for a compatible
alternative random number generator with a long period and
comparatively simple update operations.

5.9.7 `itertools' -- Functions creating iterators for efficient looping
-----------------------------------------------------------------------

New in version 2.3.

  This module implements a number of iterator building
blocks inspired by constructs from APL, Haskell, and SML.  Each has
been recast in a form suitable for Python.

  The module standardizes a core set of fast, memory efficient tools
that are useful by themselves or in combination.  Together, they form
an "iterator algebra" making it possible to construct specialized tools
succinctly and efficiently in pure Python.

  For instance, SML provides a tabulation tool: `tabulate(f)' which
produces a sequence `f(0), f(1), ...'.  The same effect can be achieved
in Python by combining imap() and count() to
form `imap(f, count())'.

  These tools and their built-in counterparts also work well with the
high-speed functions in the operator module.  For example,
the multiplication operator can be mapped across two vectors to form an
efficient dot-product: `sum(imap(operator.mul, vector1, vector2))'.

  *Infinite Iterators:*

Iterator               Arguments             Results                                               Example
-------------------------------------------------------------------------------------------------------------------------------------------------
see count().    start, [step]         start, start+step, start+2*step, ...                  `count(10) --> 10 11 12 13 14 ...'
See cycle().    p                     p0, p1, ... plast, p0, p1, ...                        `cycle('ABCD') --> A B C D A B C D ...'
See repeat().   elem [,n]             elem, elem, elem, ... endlessly or up to n times      `repeat(10, 3) --> 10 10 10'

  *Iterators terminating on the shortest input sequence:*

Iterator                 Arguments                        Results                                               Example
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
see chain().      p, q, ...                        p0, p1, ... plast, q0, q1, ...                        `chain('ABC', 'DEF') --> A B C D E F'
compress().   data, selectors                  (d[0] if s[0]), (d[1] if s[1]), ...                   `compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F'
dropwhile().  pred, seq                        seq[n], seq[n+1], starting when pred fails            `dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1'
groupby().    iterable[, keyfunc]              sub-iterators grouped by value of keyfunc(v)
ifilter().    pred, seq                        elements of seq where pred(elem) is True              `ifilter(lambda x: x%2, range(10)) --> 1 3 5 7 9'
ifilterfalse(), seq                        elements of seq where pred(elem) is False             `ifilterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8'
7ec.
See islice().     seq, [start,] stop [, step]      elements from seq[start:stop:step]                    `islice('ABCDEFG', 2, None) --> C D E F G'
imap().       func, p, q, ...                  func(p0, q0), func(p1, q1), ...                       `imap(pow, (2,3,10), (5,2,3)) --> 32 9 1000'
starmap().    func, seq                        func(*seq[0]), func(*seq[1]), ...                     `starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000'
tee().        it, n                            it1, it2 , ... itn  splits one iterator into n
takewhile().  pred, seq                        seq[0], seq[1], until pred fails                      `takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4'
izip().       p, q, ...                        (p[0], q[0]), (p[1], q[1]), ...                       `izip('ABCD', 'xy') --> Ax By'
izip_longest(), q, ...                        (p[0], q[0]), (p[1], q[1]), ...                       `izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-'
cac.

  *Combinatoric generators:*

Iterator                                           Arguments                Results
----------------------------------------------------------------------------------------------------------------------------------------------
see product().                              p, q, ... [repeat=1]     cartesian product, equivalent to a nested for-loop
permutations().                         p[, r]                   r-length tuples, all possible orderings, no repeated elements
combinations().                         p, r                     r-length tuples, in sorted order, no repeated elements
combinations_with_replacement().        p, r                     r-length tuples, in sorted order, with repeated elements
`product('ABCD', repeat=2)'                                                 `AA AB AC AD BA BB BC BD CA CB CC CD DA DB DC DD'
`permutations('ABCD', 2)'                                                   `AB AC AD BA BC BD CA CB CD DA DB DC'
`combinations('ABCD', 2)'                                                   `AB AC AD BC BD CD'
`combinations_with_replacement('ABCD', 2)'                                  `AA AB AC AD BB BC BD CC CD DD'

5.9.7.1 Itertool functions
..........................

The following module functions all construct and return iterators. Some
provide streams of infinite length, so they should only be accessed by
functions or loops that truncate the stream.

 -- Function: itertools.chain (*iterables)
     Make an iterator that returns elements from the first iterable
     until it is exhausted, then proceeds to the next iterable, until
     all of the iterables are exhausted.  Used for treating consecutive
     sequences as a single sequence.  Equivalent to:

         def chain(*iterables):
             # chain('ABC', 'DEF') --> A B C D E F
             for it in iterables:
                 for element in it:
                     yield element

 -- Class Method: chain.from_iterable (iterable)
     Alternate constructor for chain().  Gets chained inputs
     from a single iterable argument that is evaluated lazily.
     Equivalent to:

         @classmethod
         def from_iterable(iterables):
             # chain.from_iterable(['ABC', 'DEF']) --> A B C D E F
             for it in iterables:
                 for element in it:
                     yield element

     New in version 2.6.

 -- Function: itertools.combinations (iterable, r)
     Return _r_ length subsequences of elements from the input
     _iterable_.

     Combinations are emitted in lexicographic sort order.  So, if the
     input _iterable_ is sorted, the combination tuples will be produced
     in sorted order.

     Elements are treated as unique based on their position, not on
     their value.  So if the input elements are unique, there will be
     no repeat values in each combination.

     Equivalent to:

         def combinations(iterable, r):
             # combinations('ABCD', 2) --> AB AC AD BC BD CD
             # combinations(range(4), 3) --> 012 013 023 123
             pool = tuple(iterable)
             n = len(pool)
             if r > n:
                 return
             indices = range(r)
             yield tuple(pool[i] for i in indices)
             while True:
                 for i in reversed(range(r)):
                     if indices[i] != i + n - r:
                         break
                 else:
                     return
                 indices[i] += 1
                 for j in range(i+1, r):
                     indices[j] = indices[j-1] + 1
                 yield tuple(pool[i] for i in indices)

     The code for combinations() can be also expressed as a
     subsequence of permutations() after filtering entries
     where the elements are not in sorted order (according to their
     position in the input pool):

         def combinations(iterable, r):
             pool = tuple(iterable)
             n = len(pool)
             for indices in permutations(range(n), r):
                 if sorted(indices) == list(indices):
                     yield tuple(pool[i] for i in indices)

     The number of items returned is `n! / r! / (n-r)!' when `0 <= r <=
     n' or zero when `r > n'.

     New in version 2.6.

 -- Function: itertools.combinations_with_replacement (iterable, r)
     Return _r_ length subsequences of elements from the input
     _iterable_ allowing individual elements to be repeated more than
     once.

     Combinations are emitted in lexicographic sort order.  So, if the
     input _iterable_ is sorted, the combination tuples will be produced
     in sorted order.

     Elements are treated as unique based on their position, not on
     their value.  So if the input elements are unique, the generated
     combinations will also be unique.

     Equivalent to:

         def combinations_with_replacement(iterable, r):
             # combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC
             pool = tuple(iterable)
             n = len(pool)
             if not n and r:
                 return
             indices = [0] * r
             yield tuple(pool[i] for i in indices)
             while True:
                 for i in reversed(range(r)):
                     if indices[i] != n - 1:
                         break
                 else:
                     return
                 indices[i:] = [indices[i] + 1] * (r - i)
                 yield tuple(pool[i] for i in indices)

     The code for combinations_with_replacement() can be
     also expressed as a subsequence of product() after
     filtering entries where the elements are not in sorted order
     (according to their position in the input pool):

         def combinations_with_replacement(iterable, r):
             pool = tuple(iterable)
             n = len(pool)
             for indices in product(range(n), repeat=r):
                 if sorted(indices) == list(indices):
                     yield tuple(pool[i] for i in indices)

     The number of items returned is `(n+r-1)! / r! / (n-1)!' when `n >
     0'.

     New in version 2.7.

 -- Function: itertools.compress (data, selectors)
     Make an iterator that filters elements from _data_ returning only
     those that have a corresponding element in _selectors_ that
     evaluates to `True'.  Stops when either the _data_ or _selectors_
     iterables has been exhausted.  Equivalent to:

         def compress(data, selectors):
             # compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F
             return (d for d, s in izip(data, selectors) if s)

     New in version 2.7.

 -- Function: itertools.count (start=0, step=1)
     Make an iterator that returns evenly spaced values starting with
     _n_. Often used as an argument to imap() to generate
     consecutive data points.  Also, used with izip() to
     add sequence numbers.  Equivalent to:

         def count(start=0, step=1):
             # count(10) --> 10 11 12 13 14 ...
             # count(2.5, 0.5) -> 3.5 3.0 4.5 ...
             n = start
             while True:
                 yield n
                 n += step

     When counting with floating point numbers, better accuracy can
     sometimes be achieved by substituting multiplicative code such as:
     `(start + step * i for i in count())'.

     Changed in version 2.7: added _step_ argument and allowed
     non-integer arguments.

 -- Function: itertools.cycle (iterable)
     Make an iterator returning elements from the iterable and saving a
     copy of each.  When the iterable is exhausted, return elements
     from the saved copy.  Repeats indefinitely.  Equivalent to:

         def cycle(iterable):
             # cycle('ABCD') --> A B C D A B C D A B C D ...
             saved = []
             for element in iterable:
                 yield element
                 saved.append(element)
             while saved:
                 for element in saved:
                       yield element

     Note, this member of the toolkit may require significant auxiliary
     storage (depending on the length of the iterable).

 -- Function: itertools.dropwhile (predicate, iterable)
     Make an iterator that drops elements from the iterable as long as
     the predicate is true; afterwards, returns every element.  Note,
     the iterator does not produce _any_ output until the predicate
     first becomes false, so it may have a lengthy start-up time.
     Equivalent to:

         def dropwhile(predicate, iterable):
             # dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1
             iterable = iter(iterable)
             for x in iterable:
                 if not predicate(x):
                     yield x
                     break
             for x in iterable:
                 yield x

 -- Function: itertools.groupby (iterable[, key])
     Make an iterator that returns consecutive keys and groups from the
     _iterable_.  The _key_ is a function computing a key value for
     each element.  If not specified or is `None', _key_ defaults to an
     identity function and returns the element unchanged.  Generally,
     the iterable needs to already be sorted on the same key function.

     The operation of groupby() is similar to the `uniq'
     filter in Unix.  It generates a break or new group every time the
     value of the key function changes (which is why it is usually
     necessary to have sorted the data using the same key function).
     That behavior differs from SQL's GROUP BY which aggregates common
     elements regardless of their input order.

     The returned group is itself an iterator that shares the
     underlying iterable with groupby().  Because the source
     is shared, when the groupby().  object is advanced, the
     previous group is no longer visible.  So, if that data is needed
     later, it should be stored as a list:

         groups = []
         uniquekeys = []
         data = sorted(data, key=keyfunc)
         for k, g in groupby(data, keyfunc):
             groups.append(list(g))      # Store group iterator as a list
             uniquekeys.append(k)

     groupby() is equivalent to:

         class groupby(object):
             # [k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B
             # [list(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC D
             def __init__(self, iterable, key=None):
                 if key is None:
                     key = lambda x: x
                 self.keyfunc = key
                 self.it = iter(iterable)
                 self.tgtkey = self.currkey = self.currvalue = object()
             def __iter__(self):
                 return self
             def next(self):
                 while self.currkey == self.tgtkey:
                     self.currvalue = next(self.it)    # Exit on StopIteration
                     self.currkey = self.keyfunc(self.currvalue)
                 self.tgtkey = self.currkey
                 return (self.currkey, self._grouper(self.tgtkey))
             def _grouper(self, tgtkey):
                 while self.currkey == tgtkey:
                     yield self.currvalue
                     self.currvalue = next(self.it)    # Exit on StopIteration
                     self.currkey = self.keyfunc(self.currvalue)

     New in version 2.4.

 -- Function: itertools.ifilter (predicate, iterable)
     Make an iterator that filters elements from iterable returning
     only those for which the predicate is `True'. If _predicate_ is
     `None', return the items that are true. Equivalent to:

         def ifilter(predicate, iterable):
             # ifilter(lambda x: x%2, range(10)) --> 1 3 5 7 9
             if predicate is None:
                 predicate = bool
             for x in iterable:
                 if predicate(x):
                     yield x

 -- Function: itertools.ifilterfalse (predicate, iterable)
     Make an iterator that filters elements from iterable returning
     only those for which the predicate is `False'. If _predicate_ is
     `None', return the items that are false. Equivalent to:

         def ifilterfalse(predicate, iterable):
             # ifilterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8
             if predicate is None:
                 predicate = bool
             for x in iterable:
                 if not predicate(x):
                     yield x

 -- Function: itertools.imap (function, *iterables)
     Make an iterator that computes the function using arguments from
     each of the iterables.  If _function_ is set to `None', then
     imap() returns the arguments as a tuple.  Like map()
     but stops when the shortest iterable is exhausted instead of
     filling in `None' for shorter iterables.  The reason for the
     difference is that infinite iterator arguments are typically an
     error for map(). (because the output is fully
     evaluated) but represent a common and useful way of supplying
     arguments to imap(). Equivalent to:

         def imap(function, *iterables):
             # imap(pow, (2,3,10), (5,2,3)) --> 32 9 1000
             iterables = map(iter, iterables)
             while True:
                 args = [next(it) for it in iterables]
                 if function is None:
                     yield tuple(args)
                 else:
                     yield function(*args)

 -- Function: itertools.islice (iterable[, start], stop[, step])
     Make an iterator that returns selected elements from the iterable.
     If _start_ is non-zero, then elements from the iterable are
     skipped until start is reached.  Afterward, elements are returned
     consecutively unless _step_ is set higher than one which results
     in items being skipped.  If _stop_ is `None', then iteration
     continues until the iterator is exhausted, if at all; otherwise,
     it stops at the specified position.  Unlike regular slicing, see
     islice() does not support negative values for _start_,
     _stop_, or _step_.  Can be used to extract related fields from
     data where the internal structure has been flattened (for example,
     a multi-line report may list a name field on every third line).
     Equivalent to:

         def islice(iterable, *args):
             # islice('ABCDEFG', 2) --> A B
             # islice('ABCDEFG', 2, 4) --> C D
             # islice('ABCDEFG', 2, None) --> C D E F G
             # islice('ABCDEFG', 0, None, 2) --> A C E G
             s = slice(*args)
             it = iter(xrange(s.start or 0, s.stop or sys.maxint, s.step or 1))
             nexti = next(it)
             for i, element in enumerate(iterable):
                 if i == nexti:
                     yield element
                     nexti = next(it)

     If _start_ is `None', then iteration starts at zero. If _step_ is
     `None', then the step defaults to one.

     Changed in version 2.5: accept `None' values for default _start_
     and _step_.

 -- Function: itertools.izip (*iterables)
     Make an iterator that aggregates elements from each of the
     iterables. Like zip() except that it returns an
     iterator instead of a list.  Used for lock-step iteration over
     several iterables at a time.  Equivalent to:

         def izip(*iterables):
             # izip('ABCD', 'xy') --> Ax By
             iterables = map(iter, iterables)
             while iterables:
                 yield tuple(map(next, iterables))

     Changed in version 2.4: When no iterables are specified, returns a
     zero length iterator instead of raising a TypeError.
     exception.

     The left-to-right evaluation order of the iterables is guaranteed.
     This makes possible an idiom for clustering a data series into
     n-length groups using `izip(*[iter(s)]*n)'.

     izip() should only be used with unequal length inputs
     when you don't care about trailing, unmatched values from the
     longer iterables.  If those values are important, use
     izip_longest() instead.

 -- Function: itertools.izip_longest (*iterables[, fillvalue])
     Make an iterator that aggregates elements from each of the
     iterables. If the iterables are of uneven length, missing values
     are filled-in with _fillvalue_.  Iteration continues until the
     longest iterable is exhausted.  Equivalent to:

         def izip_longest(*args, **kwds):
             # izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-
             fillvalue = kwds.get('fillvalue')
             def sentinel(counter = ([fillvalue]*(len(args)-1)).pop):
                 yield counter()         # yields the fillvalue, or raises IndexError
             fillers = repeat(fillvalue)
             iters = [chain(it, sentinel(), fillers) for it in args]
             try:
                 for tup in izip(*iters):
                     yield tup
             except IndexError:
                 pass

     If one of the iterables is potentially infinite, then the
     izip_longest() function should be wrapped with something
     that limits the number of calls (for example islice().
     or takewhile()).  If not specified, _fillvalue_
     defaults to `None'.

     New in version 2.6.

 -- Function: itertools.permutations (iterable[, r])
     Return successive _r_ length permutations of elements in the
     _iterable_.

     If _r_ is not specified or is `None', then _r_ defaults to the
     length of the _iterable_ and all possible full-length permutations
     are generated.

     Permutations are emitted in lexicographic sort order.  So, if the
     input _iterable_ is sorted, the permutation tuples will be produced
     in sorted order.

     Elements are treated as unique based on their position, not on
     their value.  So if the input elements are unique, there will be
     no repeat values in each permutation.

     Equivalent to:

         def permutations(iterable, r=None):
             # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
             # permutations(range(3)) --> 012 021 102 120 201 210
             pool = tuple(iterable)
             n = len(pool)
             r = n if r is None else r
             if r > n:
                 return
             indices = range(n)
             cycles = range(n, n-r, -1)
             yield tuple(pool[i] for i in indices[:r])
             while n:
                 for i in reversed(range(r)):
                     cycles[i] -= 1
                     if cycles[i] == 0:
                         indices[i:] = indices[i+1:] + indices[i:i+1]
                         cycles[i] = n - i
                     else:
                         j = cycles[i]
                         indices[i], indices[-j] = indices[-j], indices[i]
                         yield tuple(pool[i] for i in indices[:r])
                         break
                 else:
                     return

     The code for permutations() can be also expressed as a
     subsequence of product(), filtered to exclude entries
     with repeated elements (those from the same position in the input
     pool):

         def permutations(iterable, r=None):
             pool = tuple(iterable)
             n = len(pool)
             r = n if r is None else r
             for indices in product(range(n), repeat=r):
                 if len(set(indices)) == r:
                     yield tuple(pool[i] for i in indices)

     The number of items returned is `n! / (n-r)!' when `0 <= r <= n'
     or zero when `r > n'.

     New in version 2.6.

 -- Function: itertools.product (*iterables[, repeat])
     Cartesian product of input iterables.

     Equivalent to nested for-loops in a generator expression. For
     example, `product(A, B)' returns the same as `((x,y) for x in A
     for y in B)'.

     The nested loops cycle like an odometer with the rightmost element
     advancing on every iteration.  This pattern creates a
     lexicographic ordering so that if the input's iterables are
     sorted, the product tuples are emitted in sorted order.

     To compute the product of an iterable with itself, specify the
     number of repetitions with the optional _repeat_ keyword argument.
     For example, `product(A, repeat=4)' means the same as `product(A,
     A, A, A)'.

     This function is equivalent to the following code, except that the
     actual implementation does not build up intermediate results in
     memory:

         def product(*args, **kwds):
             # product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy
             # product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111
             pools = map(tuple, args) * kwds.get('repeat', 1)
             result = [[]]
             for pool in pools:
                 result = [x+[y] for x in result for y in pool]
             for prod in result:
                 yield tuple(prod)

     New in version 2.6.

 -- Function: itertools.repeat (object[, times])
     Make an iterator that returns _object_ over and over again. Runs
     indefinitely unless the _times_ argument is specified. Used as
     argument to imap() for invariant function parameters.
     Also used with izip() to create constant fields in a
     tuple record.  Equivalent to:

         def repeat(object, times=None):
             # repeat(10, 3) --> 10 10 10
             if times is None:
                 while True:
                     yield object
             else:
                 for i in xrange(times):
                     yield object

 -- Function: itertools.starmap (function, iterable)
     Make an iterator that computes the function using arguments
     obtained from the iterable.  Used instead of imap().
     when argument parameters are already grouped in tuples from a
     single iterable (the data has been "pre-zipped").  The difference
     between imap() and starmap() parallels the
     distinction between `function(a,b)' and `function(*c)'. Equivalent
     to:

         def starmap(function, iterable):
             # starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000
             for args in iterable:
                 yield function(*args)

     Changed in version 2.6: Previously, see starmap() required
     the function arguments to be tuples.  Now, any iterable is allowed.

 -- Function: itertools.takewhile (predicate, iterable)
     Make an iterator that returns elements from the iterable as long
     as the predicate is true.  Equivalent to:

         def takewhile(predicate, iterable):
             # takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4
             for x in iterable:
                 if predicate(x):
                     yield x
                 else:
                     break

 -- Function: itertools.tee (iterable[, n=2])
     Return _n_ independent iterators from a single iterable.
     Equivalent to:

         def tee(iterable, n=2):
             it = iter(iterable)
             deques = [collections.deque() for i in range(n)]
             def gen(mydeque):
                 while True:
                     if not mydeque:             # when the local deque is empty
                         newval = next(it)       # fetch a new value and
                         for d in deques:        # load it to all the deques
                             d.append(newval)
                     yield mydeque.popleft()
             return tuple(gen(d) for d in deques)

     Once tee() has made a split, the original _iterable_
     should not be used anywhere else; otherwise, the _iterable_ could
     get advanced without the tee objects being informed.

     This itertool may require significant auxiliary storage (depending
     on how much temporary data needs to be stored). In general, if one
     iterator uses most or all of the data before another iterator
     starts, it is faster to use list() instead of
     tee().

     New in version 2.4.

5.9.7.2 Recipes
...............

This section shows recipes for creating an extended toolset using the
existing itertools as building blocks.

  The extended tools offer the same high performance as the underlying
toolset.  The superior memory performance is kept by processing
elements one at a time rather than bringing the whole iterable into
memory all at once. Code volume is kept small by linking the tools
together in a functional style which helps eliminate temporary
variables.  High speed is retained by preferring "vectorized" building
blocks over the use of for-loops and generator.s which incur
interpreter overhead.

    def take(n, iterable):
        "Return first n items of the iterable as a list"
        return list(islice(iterable, n))

    def tabulate(function, start=0):
        "Return function(0), function(1), ..."
        return imap(function, count(start))

    def consume(iterator, n):
        "Advance the iterator n-steps ahead. If n is none, consume entirely."
        # Use functions that consume iterators at C speed.
        if n is None:
            # feed the entire iterator into a zero-length deque
            collections.deque(iterator, maxlen=0)
        else:
            # advance to the empty slice starting at position n
            next(islice(iterator, n, n), None)

    def nth(iterable, n, default=None):
        "Returns the nth item or a default value"
        return next(islice(iterable, n, None), default)

    def quantify(iterable, pred=bool):
        "Count how many times the predicate is true"
        return sum(imap(pred, iterable))

    def padnone(iterable):
        """Returns the sequence elements and then returns None indefinitely.

        Useful for emulating the behavior of the built-in map() function.
        """
        return chain(iterable, repeat(None))

    def ncycles(iterable, n):
        "Returns the sequence elements n times"
        return chain.from_iterable(repeat(tuple(iterable), n))

    def dotproduct(vec1, vec2):
        return sum(imap(operator.mul, vec1, vec2))

    def flatten(listOfLists):
        "Flatten one level of nesting"
        return chain.from_iterable(listOfLists)

    def repeatfunc(func, times=None, *args):
        """Repeat calls to func with specified arguments.

        Example:  repeatfunc(random.random)
        """
        if times is None:
            return starmap(func, repeat(args))
        return starmap(func, repeat(args, times))

    def pairwise(iterable):
        "s -> (s0,s1), (s1,s2), (s2, s3), ..."
        a, b = tee(iterable)
        next(b, None)
        return izip(a, b)

    def grouper(n, iterable, fillvalue=None):
        "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
        args = [iter(iterable)] * n
        return izip_longest(fillvalue=fillvalue, *args)

    def roundrobin(*iterables):
        "roundrobin('ABC', 'D', 'EF') --> A D E B F C"
        # Recipe credited to George Sakkis
        pending = len(iterables)
        nexts = cycle(iter(it).next for it in iterables)
        while pending:
            try:
                for next in nexts:
                    yield next()
            except StopIteration:
                pending -= 1
                nexts = cycle(islice(nexts, pending))

    def powerset(iterable):
        "powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
        s = list(iterable)
        return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

    def unique_everseen(iterable, key=None):
        "List unique elements, preserving order. Remember all elements ever seen."
        # unique_everseen('AAAABBBCCDAABBB') --> A B C D
        # unique_everseen('ABBCcAD', str.lower) --> A B C D
        seen = set()
        seen_add = seen.add
        if key is None:
            for element in ifilterfalse(seen.__contains__, iterable):
                seen_add(element)
                yield element
        else:
            for element in iterable:
                k = key(element)
                if k not in seen:
                    seen_add(k)
                    yield element

    def unique_justseen(iterable, key=None):
        "List unique elements, preserving order. Remember only the element just seen."
        # unique_justseen('AAAABBBCCDAABBB') --> A B C D A B
        # unique_justseen('ABBCcAD', str.lower) --> A B C A D
        return imap(next, imap(itemgetter(1), groupby(iterable, key)))

    def iter_except(func, exception, first=None):
        """ Call a function repeatedly until an exception is raised.

        Converts a call-until-exception interface to an iterator interface.
        Like __builtin__.iter(func, sentinel) but uses an exception instead
        of a sentinel to end the loop.

        Examples:
            bsddbiter = iter_except(db.next, bsddb.error, db.first)
            heapiter = iter_except(functools.partial(heappop, h), IndexError)
            dictiter = iter_except(d.popitem, KeyError)
            dequeiter = iter_except(d.popleft, IndexError)
            queueiter = iter_except(q.get_nowait, Queue.Empty)
            setiter = iter_except(s.pop, KeyError)

        """
        try:
            if first is not None:
                yield first()
            while 1:
                yield func()
        except exception:
            pass

    def random_product(*args, **kwds):
        "Random selection from itertools.product(*args, **kwds)"
        pools = map(tuple, args) * kwds.get('repeat', 1)
        return tuple(random.choice(pool) for pool in pools)

    def random_permutation(iterable, r=None):
        "Random selection from itertools.permutations(iterable, r)"
        pool = tuple(iterable)
        r = len(pool) if r is None else r
        return tuple(random.sample(pool, r))

    def random_combination(iterable, r):
        "Random selection from itertools.combinations(iterable, r)"
        pool = tuple(iterable)
        n = len(pool)
        indices = sorted(random.sample(xrange(n), r))
        return tuple(pool[i] for i in indices)

    def random_combination_with_replacement(iterable, r):
        "Random selection from itertools.combinations_with_replacement(iterable, r)"
        pool = tuple(iterable)
        n = len(pool)
        indices = sorted(random.randrange(n) for i in xrange(r))
        return tuple(pool[i] for i in indices)

Note, many of the above recipes can be optimized by replacing global
lookups with local variables defined as default values.  For example,
the _dotproduct_ recipe can be written as:

    def dotproduct(vec1, vec2, sum=sum, imap=imap, mul=operator.mul):
        return sum(imap(mul, vec1, vec2))

5.9.8 `functools' -- Higher order functions and operations on callable objects
------------------------------------------------------------------------------

New in version 2.5.

  The functools module is for higher-order functions:
functions that act on or return other functions. In general, any
callable object can be treated as a function for the purposes of this
module.

  Latest version of the functools Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/functools.py?view=markup)The
functools module defines the following functions:

 -- Function: functools.cmp_to_key (func)
     Transform an old-style comparison function to a key-function.
     Used with tools that accept key functions (such as sorted(),
     see min(), max(), heapq.nlargest(),
     see heapq.nsmallest(), itertools.groupby()).
     This function is primarily used as a transition tool for
     programs being converted to Py3.x where comparison functions are no
     longer supported.

     A compare function is any callable that accept two arguments,
     compares them, and returns a negative number for less-than, zero
     for equality, or a positive number for greater-than.  A key
     function is a callable that accepts one argument and returns
     another value that indicates the position in the desired collation
     sequence.

     Example:

         sorted(iterable, key=cmp_to_key(locale.strcoll))  # locale-aware sort order

     New in version 2.7.

 -- Function: functools.total_ordering (cls)
     Given a class defining one or more rich comparison ordering
     methods, this class decorator supplies the rest.  This simplifies
     the effort involved in specifying all of the possible rich
     comparison operations:

     The class must define one of __lt__(), __le__(),
     see __gt__(), or __ge__().  In addition,
     the class should supply an __eq__() method.

     For example:

         @total_ordering
         class Student:
             def __eq__(self, other):
                 return ((self.lastname.lower(), self.firstname.lower()) ==
                         (other.lastname.lower(), other.firstname.lower()))
             def __lt__(self, other):
                 return ((self.lastname.lower(), self.firstname.lower()) <
                         (other.lastname.lower(), other.firstname.lower()))

     New in version 2.7.

 -- Function: functools.reduce (function, iterable[, initializer])
     This is the same function as reduce().  It is made
     available in this module to allow writing code more
     forward-compatible with Python 3.

     New in version 2.6.

 -- Function: functools.partial (func[,*args][, **keywords])
     Return a new partial object which when called will
     behave like _func_ called with the positional arguments _args_ and
     keyword arguments _keywords_. If more arguments are supplied to
     the call, they are appended to _args_. If additional keyword
     arguments are supplied, they extend and override _keywords_.
     Roughly equivalent to:

         def partial(func, *args, **keywords):
             def newfunc(*fargs, **fkeywords):
                 newkeywords = keywords.copy()
                 newkeywords.update(fkeywords)
                 return func(*(args + fargs), **newkeywords)
             newfunc.func = func
             newfunc.args = args
             newfunc.keywords = keywords
             return newfunc

     The partial() is used for partial function application
     which "freezes" some portion of a function's arguments and/or
     keywords resulting in a new object with a simplified signature.
     For example, see partial() can be used to create a
     callable that behaves like the int() function where
     the _base_ argument defaults to two:

         >>> from functools import partial
         >>> basetwo = partial(int, base=2)
         >>> basetwo.__doc__ = 'Convert base 2 string to an int.'
         >>> basetwo('10010')
         18

 -- Function: functools.update_wrapper (wrapper, wrapped[, assigned][,
          updated])
     Update a _wrapper_ function to look like the _wrapped_ function.
     The optional arguments are tuples to specify which attributes of
     the original function are assigned directly to the matching
     attributes on the wrapper function and which attributes of the
     wrapper function are updated with the corresponding attributes
     from the original function. The default values for these arguments
     are the module level constants _WRAPPER_ASSIGNMENTS_ (which
     assigns to the wrapper function's ___name___, ___module___ and
     ___doc___, the documentation string) and _WRAPPER_UPDATES_ (which
     updates the wrapper function's ___dict___, i.e. the instance
     dictionary).

     The main intended use for this function is in decorator
     functions which wrap the decorated function and return the
     wrapper. If the wrapper function is not updated, the metadata of
     the returned function will reflect the wrapper definition rather
     than the original function definition, which is typically less
     than helpful.

 -- Function: functools.wraps (wrapped[, assigned][, updated])
     This is a convenience function for invoking
     `partial(update_wrapper, wrapped=wrapped, assigned=assigned,
     updated=updated)' as a function decorator when defining a wrapper
     function. For example:

         >>> from functools import wraps
         >>> def my_decorator(f):
         ...     @wraps(f)
         ...     def wrapper(*args, **kwds):
         ...         print 'Calling decorated function'
         ...         return f(*args, **kwds)
         ...     return wrapper
         ...
         >>> @my_decorator
         ... def example():
         ...     """Docstring"""
         ...     print 'Called example function'
         ...
         >>> example()
         Calling decorated function
         Called example function
         >>> example.__name__
         'example'
         >>> example.__doc__
         'Docstring'

     Without the use of this decorator factory, the name of the example
     function would have been `'wrapper'', and the docstring of the
     original `example()' would have been lost.

5.9.8.1 `partial' Objects
.........................

partial objects are callable objects created by
partial(). They have three read-only attributes:

 -- Attribute: partial.func
     A callable object or function.  Calls to the partial.
     object will be forwarded to func with new arguments
     and keywords.

 -- Attribute: partial.args
     The leftmost positional arguments that will be prepended to the
     positional arguments provided to a partial object call.

 -- Attribute: partial.keywords
     The keyword arguments that will be supplied when the
     partial object is called.

  partial objects are like `function' objects in that they
are callable, weak referencable, and can have attributes.  There are
some important differences.  For instance, the `__name__' and `__doc__'
attributes are not created automatically.  Also, see partial.
objects defined in classes behave like static methods and do not
transform into bound methods during instance attribute look-up.

5.9.9 `operator' -- Standard operators as functions
---------------------------------------------------

The operator module exports a set of functions implemented
in C corresponding to the intrinsic operators of Python.  For example,
`operator.add(x, y)' is equivalent to the expression `x+y'.  The
function names are those used for special class methods; variants
without leading and trailing `__' are also provided for convenience.

  The functions fall into categories that perform object comparisons,
logical operations, mathematical operations, sequence operations, and
abstract type tests.

  The object comparison functions are useful for all objects, and are
named after the rich comparison operators they support:

 -- Function: operator.lt (a, b)
 -- Function: operator.le (a, b)
 -- Function: operator.eq (a, b)
 -- Function: operator.ne (a, b)
 -- Function: operator.ge (a, b)
 -- Function: operator.gt (a, b)
 -- Function: operator.__lt__ (a, b)
 -- Function: operator.__le__ (a, b)
 -- Function: operator.__eq__ (a, b)
 -- Function: operator.__ne__ (a, b)
 -- Function: operator.__ge__ (a, b)
 -- Function: operator.__gt__ (a, b)
     Perform "rich comparisons" between _a_ and _b_. Specifically,
     `lt(a, b)' is equivalent to `a < b', `le(a, b)' is equivalent to
     `a <= b', `eq(a, b)' is equivalent to `a == b', `ne(a, b)' is
     equivalent to `a != b', `gt(a, b)' is equivalent to `a > b' and
     `ge(a, b)' is equivalent to `a >= b'.  Note that unlike the
     built-in cmp(), these functions can return any value,
     which may or may not be interpretable as a Boolean value.  See
     Comparisons for more information about rich
     comparisons.

     New in version 2.2.

  The logical operations are also generally applicable to all objects,
and support truth tests, identity tests, and boolean operations:

 -- Function: operator.not_ (obj)
 -- Function: operator.__not__ (obj)
     Return the outcome of not. _obj_.  (Note that there is
     no __not__() method for object instances; only the
     interpreter core defines this operation.  The result is affected
     by the __nonzero__() and __len__().
     methods.)

 -- Function: operator.truth (obj)
     Return True if _obj_ is true, and False.
     otherwise.  This is equivalent to using the bool.
     constructor.

 -- Function: operator.is_ (a, b)
     Return `a is b'.  Tests object identity.

     New in version 2.3.

 -- Function: operator.is_not (a, b)
     Return `a is not b'.  Tests object identity.

     New in version 2.3.

  The mathematical and bitwise operations are the most numerous:

 -- Function: operator.abs (obj)
 -- Function: operator.__abs__ (obj)
     Return the absolute value of _obj_.

 -- Function: operator.add (a, b)
 -- Function: operator.__add__ (a, b)
     Return `a + b', for _a_ and _b_ numbers.

 -- Function: operator.and_ (a, b)
 -- Function: operator.__and__ (a, b)
     Return the bitwise and of _a_ and _b_.

 -- Function: operator.div (a, b)
 -- Function: operator.__div__ (a, b)
     Return `a / b' when `__future__.division' is not in effect.  This
     is also known as "classic" division.

 -- Function: operator.floordiv (a, b)
 -- Function: operator.__floordiv__ (a, b)
     Return `a // b'.

     New in version 2.2.

 -- Function: operator.index (a)
 -- Function: operator.__index__ (a)
     Return _a_ converted to an integer.  Equivalent to `a.__index__()'.

     New in version 2.5.

 -- Function: operator.inv (obj)
 -- Function: operator.invert (obj)
 -- Function: operator.__inv__ (obj)
 -- Function: operator.__invert__ (obj)
     Return the bitwise inverse of the number _obj_.  This is
     equivalent to `~obj'.

     New in version 2.0: The names invert() and
     __invert__().

 -- Function: operator.lshift (a, b)
 -- Function: operator.__lshift__ (a, b)
     Return _a_ shifted left by _b_.

 -- Function: operator.mod (a, b)
 -- Function: operator.__mod__ (a, b)
     Return `a % b'.

 -- Function: operator.mul (a, b)
 -- Function: operator.__mul__ (a, b)
     Return `a * b', for _a_ and _b_ numbers.

 -- Function: operator.neg (obj)
 -- Function: operator.__neg__ (obj)
     Return _obj_ negated (`-obj').

 -- Function: operator.or_ (a, b)
 -- Function: operator.__or__ (a, b)
     Return the bitwise or of _a_ and _b_.

 -- Function: operator.pos (obj)
 -- Function: operator.__pos__ (obj)
     Return _obj_ positive (`+obj').

 -- Function: operator.pow (a, b)
 -- Function: operator.__pow__ (a, b)
     Return `a ** b', for _a_ and _b_ numbers.

     New in version 2.3.

 -- Function: operator.rshift (a, b)
 -- Function: operator.__rshift__ (a, b)
     Return _a_ shifted right by _b_.

 -- Function: operator.sub (a, b)
 -- Function: operator.__sub__ (a, b)
     Return `a - b'.

 -- Function: operator.truediv (a, b)
 -- Function: operator.__truediv__ (a, b)
     Return `a / b' when `__future__.division' is in effect.  This is
     also known as "true" division.

     New in version 2.2.

 -- Function: operator.xor (a, b)
 -- Function: operator.__xor__ (a, b)
     Return the bitwise exclusive or of _a_ and _b_.

  Operations which work with sequences (some of them with mappings too)
include:

 -- Function: operator.concat (a, b)
 -- Function: operator.__concat__ (a, b)
     Return `a + b' for _a_ and _b_ sequences.

 -- Function: operator.contains (a, b)
 -- Function: operator.__contains__ (a, b)
     Return the outcome of the test `b in a'. Note the reversed
     operands.

     New in version 2.0: The name __contains__().

 -- Function: operator.countOf (a, b)
     Return the number of occurrences of _b_ in _a_.

 -- Function: operator.delitem (a, b)
 -- Function: operator.__delitem__ (a, b)
     Remove the value of _a_ at index _b_.

 -- Function: operator.delslice (a, b, c)
 -- Function: operator.__delslice__ (a, b, c)
     Delete the slice of _a_ from index _b_ to index _c-1_.

     Deprecated since version 2.6: This function is removed in Python
     3.x.  Use delitem() with a slice index.

 -- Function: operator.getitem (a, b)
 -- Function: operator.__getitem__ (a, b)
     Return the value of _a_ at index _b_.

 -- Function: operator.getslice (a, b, c)
 -- Function: operator.__getslice__ (a, b, c)
     Return the slice of _a_ from index _b_ to index _c-1_.

     Deprecated since version 2.6: This function is removed in Python
     3.x.  Use getitem() with a slice index.

 -- Function: operator.indexOf (a, b)
     Return the index of the first of occurrence of _b_ in _a_.

 -- Function: operator.repeat (a, b)
 -- Function: operator.__repeat__ (a, b)
     Deprecated since version 2.7: Use __mul__() instead.

     Return `a * b' where _a_ is a sequence and _b_ is an integer.

 -- Function: operator.sequenceIncludes (...)
     Deprecated since version 2.0: Use contains() instead.

     Alias for contains().

 -- Function: operator.setitem (a, b, c)
 -- Function: operator.__setitem__ (a, b, c)
     Set the value of _a_ at index _b_ to _c_.

 -- Function: operator.setslice (a, b, c, v)
 -- Function: operator.__setslice__ (a, b, c, v)
     Set the slice of _a_ from index _b_ to index _c-1_ to the sequence
     _v_.

     Deprecated since version 2.6: This function is removed in Python
     3.x.  Use setitem() with a slice index.

  Example use of operator functions:

    >>> # Elementwise multiplication
    >>> map(mul, [0, 1, 2, 3], [10, 20, 30, 40])
    [0, 20, 60, 120]

    >>> # Dot product
    >>> sum(map(mul, [0, 1, 2, 3], [10, 20, 30, 40]))
    200

Many operations have an "in-place" version.  The following functions
provide a more primitive access to in-place operators than the usual
syntax does; for example, the statement. `x += y' is
equivalent to `x = operator.iadd(x, y)'.  Another way to put it is to
say that `z = operator.iadd(x, y)' is equivalent to the compound
statement `z = x; z += y'.

 -- Function: operator.iadd (a, b)
 -- Function: operator.__iadd__ (a, b)
     `a = iadd(a, b)' is equivalent to `a += b'.

     New in version 2.5.

 -- Function: operator.iand (a, b)
 -- Function: operator.__iand__ (a, b)
     `a = iand(a, b)' is equivalent to `a &= b'.

     New in version 2.5.

 -- Function: operator.iconcat (a, b)
 -- Function: operator.__iconcat__ (a, b)
     `a = iconcat(a, b)' is equivalent to `a += b' for _a_ and _b_
     sequences.

     New in version 2.5.

 -- Function: operator.idiv (a, b)
 -- Function: operator.__idiv__ (a, b)
     `a = idiv(a, b)' is equivalent to `a /= b' when
     `__future__.division' is not in effect.

     New in version 2.5.

 -- Function: operator.ifloordiv (a, b)
 -- Function: operator.__ifloordiv__ (a, b)
     `a = ifloordiv(a, b)' is equivalent to `a //= b'.

     New in version 2.5.

 -- Function: operator.ilshift (a, b)
 -- Function: operator.__ilshift__ (a, b)
     `a = ilshift(a, b)' is equivalent to `a <<= b'.

     New in version 2.5.

 -- Function: operator.imod (a, b)
 -- Function: operator.__imod__ (a, b)
     `a = imod(a, b)' is equivalent to `a %= b'.

     New in version 2.5.

 -- Function: operator.imul (a, b)
 -- Function: operator.__imul__ (a, b)
     `a = imul(a, b)' is equivalent to `a *= b'.

     New in version 2.5.

 -- Function: operator.ior (a, b)
 -- Function: operator.__ior__ (a, b)
     `a = ior(a, b)' is equivalent to `a |= b'.

     New in version 2.5.

 -- Function: operator.ipow (a, b)
 -- Function: operator.__ipow__ (a, b)
     `a = ipow(a, b)' is equivalent to `a **= b'.

     New in version 2.5.

 -- Function: operator.irepeat (a, b)
 -- Function: operator.__irepeat__ (a, b)
     Deprecated since version 2.7: Use __imul__() instead.

     `a = irepeat(a, b)' is equivalent to `a *= b' where _a_ is a
     sequence and _b_ is an integer.

     New in version 2.5.

 -- Function: operator.irshift (a, b)
 -- Function: operator.__irshift__ (a, b)
     `a = irshift(a, b)' is equivalent to `a >>= b'.

     New in version 2.5.

 -- Function: operator.isub (a, b)
 -- Function: operator.__isub__ (a, b)
     `a = isub(a, b)' is equivalent to `a -= b'.

     New in version 2.5.

 -- Function: operator.itruediv (a, b)
 -- Function: operator.__itruediv__ (a, b)
     `a = itruediv(a, b)' is equivalent to `a /= b' when
     `__future__.division' is in effect.

     New in version 2.5.

 -- Function: operator.ixor (a, b)
 -- Function: operator.__ixor__ (a, b)
     `a = ixor(a, b)' is equivalent to `a ^= b'.

     New in version 2.5.

  The operator module also defines a few predicates to test
the type of objects; however, these are not all reliable.  It is
preferable to test abstract base classes instead (see
collections and numbers for details).

 -- Function: operator.isCallable (obj)
     Deprecated since version 2.0: Use `isinstance(x,
     collections.Callable)' instead.

     Returns true if the object _obj_ can be called like a function,
     otherwise it returns false.  True is returned for functions, bound
     and unbound methods, class objects, and instance objects which
     support the __call__() method.

 -- Function: operator.isMappingType (obj)
     Deprecated since version 2.7: Use `isinstance(x,
     collections.Mapping)' instead.

     Returns true if the object _obj_ supports the mapping interface.
     This is true for dictionaries and all instance objects defining
     __getitem__().

 -- Function: operator.isNumberType (obj)
     Deprecated since version 2.7: Use `isinstance(x, numbers.Number)'
     instead.

     Returns true if the object _obj_ represents a number.  This is
     true for all numeric types implemented in C.

 -- Function: operator.isSequenceType (obj)
     Deprecated since version 2.7: Use `isinstance(x,
     collections.Sequence)' instead.

     Returns true if the object _obj_ supports the sequence protocol.
     This returns true for all objects which define sequence methods in
     C, and for all instance objects defining __getitem__().

  The operator module also defines tools for generalized
attribute and item lookups.  These are useful for making fast field
extractors as arguments for map(), sorted(),
itertools.groupby(), or other functions that expect a
function argument.

 -- Function: operator.attrgetter (attr[, args...])
     Return a callable object that fetches _attr_ from its operand. If
     more than one attribute is requested, returns a tuple of
     attributes. After, `f = attrgetter('name')', the call `f(b)'
     returns `b.name'.  After, `f = attrgetter('name', 'date')', the
     call `f(b)' returns `(b.name, b.date)'.  Equivalent to:

         def attrgetter(*items):
             if len(items) == 1:
                 attr = items[0]
                 def g(obj):
                     return resolve_attr(obj, attr)
             else:
                 def g(obj):
                     return tuple(resolve_att(obj, attr) for attr in items)
             return g

         def resolve_attr(obj, attr):
             for name in attr.split("."):
                 obj = getattr(obj, name)
             return obj

     The attribute names can also contain dots; after `f =
     attrgetter('date.month')', the call `f(b)' returns `b.date.month'.

     New in version 2.4.

     Changed in version 2.5: Added support for multiple attributes.

     Changed in version 2.6: Added support for dotted attributes.

 -- Function: operator.itemgetter (item[, args...])
     Return a callable object that fetches _item_ from its operand
     using the operand's __getitem__() method.  If multiple
     items are specified, returns a tuple of lookup values.  Equivalent
     to:

         def itemgetter(*items):
             if len(items) == 1:
                 item = items[0]
                 def g(obj):
                     return obj[item]
             else:
                 def g(obj):
                     return tuple(obj[item] for item in items)
             return g

     The items can be any type accepted by the operand's
     __getitem__().  method.  Dictionaries accept any hashable
     value.  Lists, tuples, and strings accept an index or a slice:

         >>> itemgetter(1)('ABCDEFG')
         'B'
         >>> itemgetter(1,3,5)('ABCDEFG')
         ('B', 'D', 'F')
         >>> itemgetter(slice(2,None))('ABCDEFG')
         'CDEFG'

     New in version 2.4.

     Changed in version 2.5: Added support for multiple item extraction.

     Example of using itemgetter() to retrieve specific
     fields from a tuple record:

         >>> inventory = [('apple', 3), ('banana', 2), ('pear', 5), ('orange', 1)]
         >>> getcount = itemgetter(1)
         >>> map(getcount, inventory)
         [3, 2, 5, 1]
         >>> sorted(inventory, key=getcount)
         [('orange', 1), ('banana', 2), ('apple', 3), ('pear', 5)]

 -- Function: operator.methodcaller (name[, args...])
     Return a callable object that calls the method _name_ on its
     operand.  If additional arguments and/or keyword arguments are
     given, they will be given to the method as well.  After `f =
     methodcaller('name')', the call `f(b)' returns `b.name()'.  After
     `f = methodcaller('name', 'foo', bar=1)', the call `f(b)' returns
     `b.name('foo', bar=1)'.  Equivalent to:

         def methodcaller(name, *args, **kwargs):
             def caller(obj):
                 return getattr(obj, name)(*args, **kwargs)
             return caller

     New in version 2.6.

5.9.9.1 Mapping Operators to Functions
......................................

This table shows how abstract operations correspond to operator symbols
in the Python syntax and the functions in the operator.
module.

Operation                   Syntax                        Function
------------------------------------------------------------------------------------------------------
Addition                    `a + b'                       `add(a, b)'
Concatenation               `seq1 + seq2'                 `concat(seq1, seq2)'
Containment Test            `obj in seq'                  `contains(seq, obj)'
Division                    `a / b'                       `div(a, b)' (without `__future__.division')
Division                    `a / b'                       `truediv(a, b)' (with
                                                          `__future__.division')
Division                    `a // b'                      `floordiv(a, b)'
Bitwise And                 `a & b'                       `and_(a, b)'
Bitwise Exclusive Or        `a ^ b'                       `xor(a, b)'
Bitwise Inversion           `~ a'                         `invert(a)'
Bitwise Or                  `a | b'                       `or_(a, b)'
Exponentiation              `a ** b'                      `pow(a, b)'
Identity                    `a is b'                      `is_(a, b)'
Identity                    `a is not b'                  `is_not(a, b)'
Indexed Assignment          `obj[k] = v'                  `setitem(obj, k, v)'
Indexed Deletion            `del obj[k]'                  `delitem(obj, k)'
Indexing                    `obj[k]'                      `getitem(obj, k)'
Left Shift                  `a << b'                      `lshift(a, b)'
Modulo                      `a % b'                       `mod(a, b)'
Multiplication              `a * b'                       `mul(a, b)'
Negation (Arithmetic)       `- a'                         `neg(a)'
Negation (Logical)          `not a'                       `not_(a)'
Positive                    `+ a'                         `pos(a)'
Right Shift                 `a >> b'                      `rshift(a, b)'
Sequence Repetition         `seq * i'                     `repeat(seq, i)'
Slice Assignment            `seq[i:j] = values'           `setitem(seq, slice(i, j), values)'
Slice Deletion              `del seq[i:j]'                `delitem(seq, slice(i, j))'
Slicing                     `seq[i:j]'                    `getitem(seq, slice(i, j))'
String Formatting           `s % obj'                     `mod(s, obj)'
Subtraction                 `a - b'                       `sub(a, b)'
Truth Test                  `obj'                         `truth(obj)'
Ordering                    `a < b'                       `lt(a, b)'
Ordering                    `a <= b'                      `le(a, b)'
Equality                    `a == b'                      `eq(a, b)'
Difference                  `a != b'                      `ne(a, b)'
Ordering                    `a >= b'                      `ge(a, b)'
Ordering                    `a > b'                       `gt(a, b)'

5.10 File and Directory Access
==============================

The modules described in this chapter deal with disk files and
directories.  For example, there are modules for reading the properties
of files, manipulating paths in a portable way, and creating temporary
files.  The full list of modules in this chapter is:

5.10.1 `os.path' -- Common pathname manipulations
-------------------------------------------------

This module implements some useful functions on pathnames. To read or
write files see open(), and for accessing the filesystem see
the os module.

     Note: On Windows, many of these functions do not properly support
     UNC pathnames.  splitunc() and ismount().
     do handle them correctly.

     Note: Since different operating systems have different path name
     conventions, there are several versions of this module in the
     standard library.  The os.path module is always the
     path module suitable for the operating system Python is running
     on, and therefore usable for local paths.  However, you can also
     import and use the individual modules if you want to manipulate a
     path that is _always_ in one of the different formats.  They all
     have the same interface:

        * `posixpath' for UNIX-style paths

        * `ntpath' for Windows paths

        * macpath for old-style MacOS paths

        * `os2emxpath' for OS/2 EMX paths

 -- Function: os.path.abspath (path)
     Return a normalized absolutized version of the pathname _path_. On
     most platforms, this is equivalent to `normpath(join(os.getcwd(),
     path))'.

     New in version 1.5.2.

 -- Function: os.path.basename (path)
     Return the base name of pathname _path_.  This is the second half
     of the pair returned by `split(path)'.  Note that the result of
     this function is different from the Unix *basename* program; where
     *basename* for `'/foo/bar/'' returns `'bar'', the
     basename() function returns an empty string (`''').

 -- Function: os.path.commonprefix (list)
     Return the longest path prefix (taken character-by-character) that
     is a prefix of all paths in  _list_.  If _list_ is empty, return
     the empty string (`''').  Note that this may return invalid paths
     because it works a character at a time.

 -- Function: os.path.dirname (path)
     Return the directory name of pathname _path_.  This is the first
     half of the pair returned by `split(path)'.

 -- Function: os.path.exists (path)
     Return `True' if _path_ refers to an existing path.  Returns
     `False' for broken symbolic links. On some platforms, this
     function may return `False' if permission is not granted to
     execute os.stat() on the requested file, even if the
     _path_ physically exists.

 -- Function: os.path.lexists (path)
     Return `True' if _path_ refers to an existing path. Returns `True'
     for broken symbolic links.   Equivalent to exists() on
     platforms lacking os.lstat().

     New in version 2.4.

 -- Function: os.path.expanduser (path)
     On Unix and Windows, return the argument with an initial component
     of `~' or `~user' replaced by that _user_'s home directory.

     On Unix, an initial `~' is replaced by the environment variable `HOME'
     if it is set; otherwise the current user's home directory is
     looked up in the password directory through the built-in module
     pwd. An initial `~user' is looked up directly in the
     password directory.

     On Windows, `HOME' and `USERPROFILE' will be used if set,
     otherwise a combination of `HOMEPATH' and `HOMEDRIVE' will be
     used.  An initial `~user' is handled by stripping the last
     directory component from the created user path derived above.

     If the expansion fails or if the path does not begin with a tilde,
     the path is returned unchanged.

 -- Function: os.path.expandvars (path)
     Return the argument with environment variables expanded.
     Substrings of the form `$name' or `${name}' are replaced by the
     value of environment variable _name_.  Malformed variable names
     and references to non-existing variables are left unchanged.

     On Windows, `%name%' expansions are supported in addition to
     `$name' and `${name}'.

 -- Function: os.path.getatime (path)
     Return the time of last access of _path_.  The return value is a
     number giving the number of seconds since the epoch (see the
     time module).  Raise os.error if the file
     does not exist or is inaccessible.

     New in version 1.5.2.

     Changed in version 2.3: If os.stat_float_times().
     returns True, the result is a floating point number.

 -- Function: os.path.getmtime (path)
     Return the time of last modification of _path_.  The return value
     is a number giving the number of seconds since the epoch (see the
     time module).  Raise os.error if the file
     does not exist or is inaccessible.

     New in version 1.5.2.

     Changed in version 2.3: If os.stat_float_times().
     returns True, the result is a floating point number.

 -- Function: os.path.getctime (path)
     Return the system's ctime which, on some systems (like Unix) is
     the time of the last change, and, on others (like Windows), is the
     creation time for _path_.  The return value is a number giving the
     number of seconds since the epoch (see the  time.
     module).  Raise os.error if the file does not exist or
     is inaccessible.

     New in version 2.3.

 -- Function: os.path.getsize (path)
     Return the size, in bytes, of _path_.  Raise os.error.
     if the file does not exist or is inaccessible.

     New in version 1.5.2.

 -- Function: os.path.isabs (path)
     Return `True' if _path_ is an absolute pathname.  On Unix, that
     means it begins with a slash, on Windows that it begins with a
     (back)slash after chopping off a potential drive letter.

 -- Function: os.path.isfile (path)
     Return `True' if _path_ is an existing regular file.  This follows
     symbolic links, so both islink() and isfile()
     can be true for the same path.

 -- Function: os.path.isdir (path)
     Return `True' if _path_ is an existing directory.  This follows
     symbolic links, so both islink() and isdir()
     can be true for the same path.

 -- Function: os.path.islink (path)
     Return `True' if _path_ refers to a directory entry that is a
     symbolic link.  Always `False' if symbolic links are not supported.

 -- Function: os.path.ismount (path)
     Return `True' if pathname _path_ is a _mount point_: a point in a
     file system where a different file system has been mounted.  The
     function checks whether _path_'s parent, `path/..', is on a
     different device than _path_, or whether `path/..' and _path_
     point to the same i-node on the same device -- this should detect
     mount points for all Unix and POSIX variants.

 -- Function: os.path.join (path1[, path2[, ...]])
     Join one or more path components intelligently.  If any component
     is an absolute path, all previous components (on Windows,
     including the previous drive letter, if there was one) are thrown
     away, and joining continues.  The return value is the
     concatenation of _path1_, and optionally _path2_, etc., with
     exactly one directory separator (`os.sep') inserted between
     components, unless _path2_ is empty.  Note that on Windows, since
     there is a current directory for each drive, `os.path.join("c:",
     "foo")' represents a path relative to the current directory on
     drive `C:' (`c:foo'), not `c:\foo'.

 -- Function: os.path.normcase (path)
     Normalize the case of a pathname.  On Unix and Mac OS X, this
     returns the path unchanged; on case-insensitive filesystems, it
     converts the path to lowercase.  On Windows, it also converts
     forward slashes to backward slashes.

 -- Function: os.path.normpath (path)
     Normalize a pathname.  This collapses redundant separators and
     up-level references so that `A//B', `A/B/', `A/./B' and
     `A/foo/../B' all become `A/B'.

     It does not normalize the case (use normcase() for
     that).  On Windows, it converts forward slashes to backward
     slashes. It should be understood that this may change the meaning
     of the path if it contains symbolic links!

 -- Function: os.path.realpath (path)
     Return the canonical path of the specified filename, eliminating
     any symbolic links encountered in the path (if they are supported
     by the operating system).

     New in version 2.2.

 -- Function: os.path.relpath (path[, start])
     Return a relative filepath to _path_ either from the current
     directory or from an optional _start_ point.

     _start_ defaults to os.curdir.

     Availability:  Windows, Unix.

     New in version 2.6.

 -- Function: os.path.samefile (path1, path2)
     Return `True' if both pathname arguments refer to the same file or
     directory (as indicated by device number and i-node number). Raise
     an exception if a os.stat() call on either pathname
     fails.

     Availability: Unix.

 -- Function: os.path.sameopenfile (fp1, fp2)
     Return `True' if the file descriptors _fp1_ and _fp2_ refer to the
     same file.

     Availability: Unix.

 -- Function: os.path.samestat (stat1, stat2)
     Return `True' if the stat tuples _stat1_ and _stat2_ refer to the
     same file.  These structures may have been returned by `fstat()',
     `lstat()', or stat().  This function implements the
     underlying comparison used by samefile() and
     sameopenfile().

     Availability: Unix.

 -- Function: os.path.split (path)
     Split the pathname _path_ into a pair, `(head, tail)' where _tail_
     is the last pathname component and _head_ is everything leading up
     to that.  The _tail_ part will never contain a slash; if _path_
     ends in a slash, _tail_ will be empty.  If there is no slash in
     _path_, _head_ will be empty.  If _path_ is empty, both _head_ and
     _tail_ are empty.  Trailing slashes are stripped from _head_
     unless it is the root (one or more slashes only).  In all cases,
     `join(head, tail)' returns a path to the same location as _path_
     (but the strings may differ).

 -- Function: os.path.splitdrive (path)
     Split the pathname _path_ into a pair `(drive, tail)' where
     _drive_ is either a drive specification or the empty string.  On
     systems which do not use drive specifications, _drive_ will always
     be the empty string.  In all cases, `drive + tail' will be the
     same as _path_.

     New in version 1.3.

 -- Function: os.path.splitext (path)
     Split the pathname _path_ into a pair `(root, ext)'  such that
     `root + ext == path', and _ext_ is empty or begins with a period
     and contains at most one period. Leading periods on the basename
     are  ignored; `splitext('.cshrc')' returns  `('.cshrc', '')'.

     Changed in version 2.6: Earlier versions could produce an empty
     root when the only period was the first character.

 -- Function: os.path.splitunc (path)
     Split the pathname _path_ into a pair `(unc, rest)' so that _unc_
     is the UNC mount point (such as `r'\\host\mount''), if present,
     and _rest_ the rest of the path (such as  `r'\path\file.ext'').
     For paths containing drive letters, _unc_ will always be the empty
     string.

     Availability:  Windows.

 -- Function: os.path.walk (path, visit, arg)
     Calls the function _visit_ with arguments `(arg, dirname, names)'
     for each directory in the directory tree rooted at _path_
     (including _path_ itself, if it is a directory).  The argument
     _dirname_ specifies the visited directory, the argument _names_
     lists the files in the directory (gotten from
     `os.listdir(dirname)'). The _visit_ function may modify _names_ to
     influence the set of directories visited below _dirname_, e.g. to
     avoid visiting certain parts of the tree.  (The object referred to
     by _names_ must be modified in place, using del or
     slice assignment.)

          Note: Symbolic links to directories are not treated as
          subdirectories, and that walk() therefore will
          not visit them. To visit linked directories you must identify
          them with `os.path.islink(file)' and `os.path.isdir(file)',
          and invoke walk() as necessary.

          Note: This function is deprecated and has been removed in 3.0
          in favor of os.walk().

 -- Data: os.path.supports_unicode_filenames
     True if arbitrary Unicode strings can be used as file names
     (within limitations imposed by the file system).

     New in version 2.3.

5.10.2 `fileinput' -- Iterate over lines from multiple input streams
--------------------------------------------------------------------

This module implements a helper class and functions to quickly write a
loop over standard input or a list of files. If you just want to read or
write one file see open().

  The typical use is:

    import fileinput
    for line in fileinput.input():
        process(line)

This iterates over the lines of all files listed in `sys.argv[1:]',
defaulting to `sys.stdin' if the list is empty.  If a filename is
`'-'', it is also replaced by `sys.stdin'.  To specify an alternative
list of filenames, pass it as the first argument to input().
A single file name is also allowed.

  All files are opened in text mode by default, but you can override
this by specifying the _mode_ parameter in the call to input()
or FileInput().  If an I/O error occurs during opening
or reading a file, see IOError is raised.

  If `sys.stdin' is used more than once, the second and further use
will return no lines, except perhaps for interactive use, or if it has
been explicitly reset (e.g. using `sys.stdin.seek(0)').

  Empty files are opened and immediately closed; the only time their
presence in the list of filenames is noticeable at all is when the last
file opened is empty.

  Lines are returned with any newlines intact, which means that the
last line in a file may not have one.

  You can control how files are opened by providing an opening hook via
the _openhook_ parameter to fileinput.input() or
FileInput(). The hook must be a function that takes two arguments,
_filename_ and _mode_, and returns an accordingly opened file-like
object. Two useful hooks are already provided by this module.

  Latest version of the fileinput Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/fileinput.py?view=markup)The
following function is the primary interface of this module:

 -- Function: fileinput.input ([files[, inplace[, backup[, mode[,
          openhook]]]]])
     Create an instance of the FileInput class.  The
     instance will be used as global state for the functions of this
     module, and is also returned to use during iteration.  The
     parameters to this function will be passed along to the
     constructor of the FileInput class.

     Changed in version 2.5: Added the _mode_ and _openhook_ parameters.

  The following functions use the global state created by
fileinput.input().; if there is no active state, see
RuntimeError is raised.

 -- Function: fileinput.filename ()
     Return the name of the file currently being read.  Before the
     first line has been read, returns `None'.

 -- Function: fileinput.fileno ()
     Return the integer "file descriptor" for the current file. When no
     file is opened (before the first line and between files), returns
     `-1'.

     New in version 2.5.

 -- Function: fileinput.lineno ()
     Return the cumulative line number of the line that has just been
     read.  Before the first line has been read, returns `0'.  After
     the last line of the last file has been read, returns the line
     number of that line.

 -- Function: fileinput.filelineno ()
     Return the line number in the current file.  Before the first line
     has been read, returns `0'.  After the last line of the last file
     has been read, returns the line number of that line within the
     file.

 -- Function: fileinput.isfirstline ()
     Returns true if the line just read is the first line of its file,
     otherwise returns false.

 -- Function: fileinput.isstdin ()
     Returns true if the last line was read from `sys.stdin', otherwise
     returns false.

 -- Function: fileinput.nextfile ()
     Close the current file so that the next iteration will read the
     first line from the next file (if any); lines not read from the
     file will not count towards the cumulative line count.  The
     filename is not changed until after the first line of the next
     file has been read.  Before the first line has been read, this
     function has no effect; it cannot be used to skip the first file.
     After the last line of the last file has been read, this function
     has no effect.

 -- Function: fileinput.close ()
     Close the sequence.

  The class which implements the sequence behavior provided by the
module is available for subclassing as well:

 -- Class: fileinput.FileInput ([files[, inplace[, backup[, mode[,
          openhook]]]]])
     Class FileInput is the implementation; its methods
     filename(), fileno(), lineno(),
     filelineno(), isfirstline(),
     isstdin(), nextfile() and close().
     correspond to the functions of the same name in the module. In
     addition it has a readline() method which returns the
     next input line, and a __getitem__() method which
     implements the sequence behavior.  The sequence must be accessed
     in strictly sequential order; random access and readline()
     cannot be mixed.

     With _mode_ you can specify which file mode will be passed to
     open(). It must be one of `'r'', `'rU'', `'U'' and
     `'rb''.

     The _openhook_, when given, must be a function that takes two
     arguments, _filename_ and _mode_, and returns an accordingly
     opened file-like object. You cannot use _inplace_ and _openhook_
     together.

     Changed in version 2.5: Added the _mode_ and _openhook_ parameters.

  *Optional in-place filtering:* if the keyword argument `inplace=1' is
passed to fileinput.input() or to the FileInput.
constructor, the file is moved to a backup file and standard output is
directed to the input file (if a file of the same name as the backup
file already exists, it will be replaced silently).  This makes it
possible to write a filter that rewrites its input file in place.  If
the _backup_ parameter is given (typically as `backup='.<some
extension>''), it specifies the extension for the backup file, and the
backup file remains around; by default, the extension is `'.bak'' and
it is deleted when the output file is closed.  In-place filtering is
disabled when standard input is read.

     Note: The current implementation does not work for MS-DOS 8+3
     filesystems.

  The two following opening hooks are provided by this module:

 -- Function: fileinput.hook_compressed (filename, mode)
     Transparently opens files compressed with gzip and bzip2
     (recognized by the extensions `'.gz'' and `'.bz2'') using the
     gzip and bz2.  modules.  If the filename
     extension is not `'.gz'' or `'.bz2'', the file is opened normally
     (ie, using open() without any decompression).

     Usage example:  `fi =
     fileinput.FileInput(openhook=fileinput.hook_compressed)'

     New in version 2.5.

 -- Function: fileinput.hook_encoded (encoding)
     Returns a hook which opens each file with codecs.open(),
     using the given _encoding_ to read the file.

     Usage example: `fi =
     fileinput.FileInput(openhook=fileinput.hook_encoded("iso-8859-1"))'

          Note: With this hook, see FileInput might return
          Unicode strings depending on the specified _encoding_.

     New in version 2.5.

5.10.3 `stat' -- Interpreting `stat()' results
----------------------------------------------

The stat module defines constants and functions for
interpreting the results of os.stat(), os.fstat()
and os.lstat(). (if they exist).  For complete details
about the `stat()', `fstat()' and `lstat()' calls, consult the
documentation for your system.

  The stat module defines the following functions to test
for specific file types:

 -- Function: stat.S_ISDIR (mode)
     Return non-zero if the mode is from a directory.

 -- Function: stat.S_ISCHR (mode)
     Return non-zero if the mode is from a character special device
     file.

 -- Function: stat.S_ISBLK (mode)
     Return non-zero if the mode is from a block special device file.

 -- Function: stat.S_ISREG (mode)
     Return non-zero if the mode is from a regular file.

 -- Function: stat.S_ISFIFO (mode)
     Return non-zero if the mode is from a FIFO (named pipe).

 -- Function: stat.S_ISLNK (mode)
     Return non-zero if the mode is from a symbolic link.

 -- Function: stat.S_ISSOCK (mode)
     Return non-zero if the mode is from a socket.

  Two additional functions are defined for more general manipulation of
the file's mode:

 -- Function: stat.S_IMODE (mode)
     Return the portion of the file's mode that can be set by
     os.chmod().--that is, the file's permission bits, plus the
     sticky bit, set-group-id, and set-user-id bits (on systems that
     support them).

 -- Function: stat.S_IFMT (mode)
     Return the portion of the file's mode that describes the file type
     (used by the `S_IS*()' functions above).

  Normally, you would use the `os.path.is*()' functions for testing the
type of a file; the functions here are useful when you are doing
multiple tests of the same file and wish to avoid the overhead of the
`stat()' system call for each test.  These are also useful when
checking for information about a file that isn't handled by
os.path, like the tests for block and character devices.

  All the variables below are simply symbolic indexes into the 10-tuple
returned by os.stat(), os.fstat() or
os.lstat().

 -- Data: stat.ST_MODE
     Inode protection mode.

 -- Data: stat.ST_INO
     Inode number.

 -- Data: stat.ST_DEV
     Device inode resides on.

 -- Data: stat.ST_NLINK
     Number of links to the inode.

 -- Data: stat.ST_UID
     User id of the owner.

 -- Data: stat.ST_GID
     Group id of the owner.

 -- Data: stat.ST_SIZE
     Size in bytes of a plain file; amount of data waiting on some
     special files.

 -- Data: stat.ST_ATIME
     Time of last access.

 -- Data: stat.ST_MTIME
     Time of last modification.

 -- Data: stat.ST_CTIME
     The "ctime" as reported by the operating system.  On some systems
     (like Unix) is the time of the last metadata change, and, on
     others (like Windows), is the creation time (see platform
     documentation for details).

  The interpretation of "file size" changes according to the file type.
For plain files this is the size of the file in bytes.  For FIFOs and
sockets under most flavors of Unix (including Linux in particular), the
"size" is the number of bytes waiting to be read at the time of the
call to os.stat(), os.fstat(), or
os.lstat().; this can sometimes be useful, especially for polling
one of these special files after a non-blocking open.  The meaning of
the size field for other character and block devices varies more,
depending on the implementation of the underlying system call.

  The variables below define the flags used in the ST_MODE.
field.

  Use of the functions above is more portable than use of the first set
of flags:

 -- Data: stat.S_IFMT
     Bit mask for the file type bit fields.

 -- Data: stat.S_IFSOCK
     Socket.

 -- Data: stat.S_IFLNK
     Symbolic link.

 -- Data: stat.S_IFREG
     Regular file.

 -- Data: stat.S_IFBLK
     Block device.

 -- Data: stat.S_IFDIR
     Directory.

 -- Data: stat.S_IFCHR
     Character device.

 -- Data: stat.S_IFIFO
     FIFO.

  The following flags can also be used in the _mode_ argument of
os.chmod().:

 -- Data: stat.S_ISUID
     Set UID bit.

 -- Data: stat.S_ISGID
     Set-group-ID bit.  This bit has several special uses.  For a
     directory it indicates that BSD semantics is to be used for that
     directory: files created there inherit their group ID from the
     directory, not from the effective group ID of the creating
     process, and directories created there will also get the
     S_ISGID bit set.  For a file that does not have the group
     execution bit (see S_IXGRP)  set, the set-group-ID bit
     indicates mandatory file/record locking (see also S_ENFMT).

 -- Data: stat.S_ISVTX
     Sticky bit.  When this bit is set on a directory it means that a
     file in that directory can be renamed or deleted only by the owner
     of the file, by the owner of the directory, or by a privileged
     process.

 -- Data: stat.S_IRWXU
     Mask for file owner permissions.

 -- Data: stat.S_IRUSR
     Owner has read permission.

 -- Data: stat.S_IWUSR
     Owner has write permission.

 -- Data: stat.S_IXUSR
     Owner has execute permission.

 -- Data: stat.S_IRWXG
     Mask for group permissions.

 -- Data: stat.S_IRGRP
     Group has read permission.

 -- Data: stat.S_IWGRP
     Group has write permission.

 -- Data: stat.S_IXGRP
     Group has execute permission.

 -- Data: stat.S_IRWXO
     Mask for permissions for others (not in group).

 -- Data: stat.S_IROTH
     Others have read permission.

 -- Data: stat.S_IWOTH
     Others have write permission.

 -- Data: stat.S_IXOTH
     Others have execute permission.

 -- Data: stat.S_ENFMT
     System V file locking enforcement.  This flag is shared with
     S_ISGID.: file/record locking is enforced on files that do
     not have the group execution bit (see S_IXGRP) set.

 -- Data: stat.S_IREAD
     Unix V7 synonym for S_IRUSR.

 -- Data: stat.S_IWRITE
     Unix V7 synonym for S_IWUSR.

 -- Data: stat.S_IEXEC
     Unix V7 synonym for S_IXUSR.

  Example:

    import os, sys
    from stat import *

    def walktree(top, callback):
        '''recursively descend the directory tree rooted at top,
           calling the callback function for each regular file'''

        for f in os.listdir(top):
            pathname = os.path.join(top, f)
            mode = os.stat(pathname)[ST_MODE]
            if S_ISDIR(mode):
                # It's a directory, recurse into it
                walktree(pathname, callback)
            elif S_ISREG(mode):
                # It's a file, call the callback function
                callback(pathname)
            else:
                # Unknown file type, print a message
                print 'Skipping %s' % pathname

    def visitfile(file):
        print 'visiting', file

    if __name__ == '__main__':
        walktree(sys.argv[1], visitfile)

5.10.4 `statvfs' -- Constants used with `os.statvfs()'
------------------------------------------------------

Deprecated since version 2.6: The statvfs module has been
deprecated for removal in Python 3.0.

  The statvfs module defines constants so interpreting the
result if os.statvfs(), which returns a tuple, can be made
without remembering "magic numbers."  Each of the constants defined in
this module is the _index_ of the entry in the tuple returned by
os.statvfs() that contains the specified information.

 -- Data: statvfs.F_BSIZE
     Preferred file system block size.

 -- Data: statvfs.F_FRSIZE
     Fundamental file system block size.

 -- Data: statvfs.F_BLOCKS
     Total number of blocks in the filesystem.

 -- Data: statvfs.F_BFREE
     Total number of free blocks.

 -- Data: statvfs.F_BAVAIL
     Free blocks available to non-super user.

 -- Data: statvfs.F_FILES
     Total number of file nodes.

 -- Data: statvfs.F_FFREE
     Total number of free file nodes.

 -- Data: statvfs.F_FAVAIL
     Free nodes available to non-super user.

 -- Data: statvfs.F_FLAG
     Flags. System dependent: see `statvfs()' man page.

 -- Data: statvfs.F_NAMEMAX
     Maximum file name length.

5.10.5 `filecmp' -- File and Directory Comparisons
--------------------------------------------------

The filecmp module defines functions to compare files and
directories, with various optional time/correctness trade-offs. For
comparing files, see also the difflib module.

  Latest version of the filecmp Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/filecmp.py?view=markup)The
filecmp module defines the following functions:

 -- Function: filecmp.cmp (f1, f2[, shallow])
     Compare the files named _f1_ and _f2_, returning `True' if they
     seem equal, `False' otherwise.

     Unless _shallow_ is given and is false, files with identical
     os.stat().  signatures are taken to be equal.

     Files that were compared using this function will not be compared
     again unless their os.stat() signature changes.

     Note that no external programs are called from this function,
     giving it portability and efficiency.

 -- Function: filecmp.cmpfiles (dir1, dir2, common[, shallow])
     Compare the files in the two directories _dir1_ and _dir2_ whose
     names are given by _common_.

     Returns three lists of file names: _match_, _mismatch_, _errors_.
     _match_ contains the list of files that match, _mismatch_ contains
     the names of those that don't, and _errors_ lists the names of
     files which could not be compared.  Files are listed in _errors_
     if they don't exist in one of the directories, the user lacks
     permission to read them or if the comparison could not be done for
     some other reason.

     The _shallow_ parameter has the same meaning and default value as
     for filecmp.cmp().

     For example, `cmpfiles('a', 'b', ['c', 'd/e'])' will compare `a/c'
     with `b/c' and `a/d/e' with `b/d/e'.  `'c'' and `'d/e'' will each
     be in one of the three returned lists.

  Example:

    >>> import filecmp
    >>> filecmp.cmp('undoc.rst', 'undoc.rst')
    True
    >>> filecmp.cmp('undoc.rst', 'index.rst')
    False

5.10.5.1 The `dircmp' class
...........................

dircmp instances are built using this constructor:

 -- Class: filecmp.dircmp (a, b[, ignore[, hide]])
     Construct a new directory comparison object, to compare the
     directories _a_ and _b_. _ignore_ is a list of names to ignore,
     and defaults to `['RCS', 'CVS', 'tags']'. _hide_ is a list of
     names to hide, and defaults to `[os.curdir, os.pardir]'.

     The dircmp class provides the following methods:

      -- Method: report ()
          Print (to `sys.stdout') a comparison between _a_ and _b_.

      -- Method: report_partial_closure ()
          Print a comparison between _a_ and _b_ and common immediate
          subdirectories.

      -- Method: report_full_closure ()
          Print a comparison between _a_ and _b_ and common
          subdirectories (recursively).

     The dircmp offers a number of interesting attributes
     that may be used to get various bits of information about the
     directory trees being compared.

     Note that via __getattr__() hooks, all attributes are
     computed lazily, so there is no speed penalty if only those
     attributes which are lightweight to compute are used.

      -- Attribute: left_list
          Files and subdirectories in _a_, filtered by _hide_ and
          _ignore_.

      -- Attribute: right_list
          Files and subdirectories in _b_, filtered by _hide_ and
          _ignore_.

      -- Attribute: common
          Files and subdirectories in both _a_ and _b_.

      -- Attribute: left_only
          Files and subdirectories only in _a_.

      -- Attribute: right_only
          Files and subdirectories only in _b_.

      -- Attribute: common_dirs
          Subdirectories in both _a_ and _b_.

      -- Attribute: common_files
          Files in both _a_ and _b_

      -- Attribute: common_funny
          Names in both _a_ and _b_, such that the type differs between
          the directories, or names for which os.stat().
          reports an error.

      -- Attribute: same_files
          Files which are identical in both _a_ and _b_.

      -- Attribute: diff_files
          Files which are in both _a_ and _b_, whose contents differ.

      -- Attribute: funny_files
          Files which are in both _a_ and _b_, but could not be
          compared.

      -- Attribute: subdirs
          A dictionary mapping names in common_dirs to
          dircmp objects.

5.10.6 `tempfile' -- Generate temporary files and directories
-------------------------------------------------------------

This module generates temporary files and directories.  It works on all
supported platforms.

  In version 2.3 of Python, this module was overhauled for enhanced
security.  It now provides three new functions, see
NamedTemporaryFile(), mkstemp(), and mkdtemp(),
which should eliminate all remaining need to use the insecure
mktemp() function.  Temporary file names created by this
module no longer contain the process ID; instead a string of six random
characters is used.

  Also, all the user-callable functions now take additional arguments
which allow direct control over the location and name of temporary
files.  It is no longer necessary to use the global _tempdir_ and
_template_ variables.  To maintain backward compatibility, the argument
order is somewhat odd; it is recommended to use keyword arguments for
clarity.

  The module defines the following user-callable functions:

 -- Function: tempfile.TemporaryFile ([mode='w+b'[, bufsize=-1[,
          suffix="[, prefix='tmp'[, dir=None]]]]])
     Return a file-like object that can be used as a temporary storage
     area.  The file is created using mkstemp(). It will be
     destroyed as soon as it is closed (including an implicit close
     when the object is garbage collected).  Under Unix, the directory
     entry for the file is removed immediately after the file is
     created.  Other platforms do not support this; your code should
     not rely on a temporary file created using this function having or
     not having a visible name in the file system.

     The _mode_ parameter defaults to `'w+b'' so that the file created
     can be read and written without being closed.  Binary mode is used
     so that it behaves consistently on all platforms without regard
     for the data that is stored.  _bufsize_ defaults to `-1', meaning
     that the operating system default is used.

     The _dir_, _prefix_ and _suffix_ parameters are passed to
     mkstemp().

     The returned object is a true file object on POSIX platforms.  On
     other platforms, it is a file-like object whose `file' attribute
     is the underlying true file object. This file-like object can be
     used in a with statement, just like a normal file.

 -- Function: tempfile.NamedTemporaryFile ([mode='w+b'[, bufsize=-1[,
          suffix="[, prefix='tmp'[, dir=None[, delete=True]]]]]])
     This function operates exactly as TemporaryFile().
     does, except that the file is guaranteed to have a visible name in
     the file system (on Unix, the directory entry is not unlinked).
     That name can be retrieved from the `name' member of the file
     object.  Whether the name can be used to open the file a second
     time, while the named temporary file is still open, varies across
     platforms (it can be so used on Unix; it cannot on Windows NT or
     later).  If _delete_ is true (the default), the file is deleted as
     soon as it is closed.

     The returned object is always a file-like object whose `file'
     attribute is the underlying true file object. This file-like
     object can be used in a with statement, just like a
     normal file.

     New in version 2.3.

     New in version 2.6: The _delete_ parameter.

 -- Function: tempfile.SpooledTemporaryFile ([max_size=0[, mode='w+b'[,
          bufsize=-1[, suffix="[, prefix='tmp'[, dir=None]]]]]])
     This function operates exactly as TemporaryFile().
     does, except that data is spooled in memory until the file size
     exceeds _max_size_, or until the file's `fileno()' method is
     called, at which point the contents are written to disk and
     operation proceeds as with TemporaryFile().

     The resulting file has one additional method, `rollover()', which
     causes the file to roll over to an on-disk file regardless of its
     size.

     The returned object is a file-like object whose `_file' attribute
     is either a StringIO object or a true file object,
     depending on whether `rollover()' has been called. This file-like
     object can be used in a with statement, just like a
     normal file.

     New in version 2.6.

 -- Function: tempfile.mkstemp ([suffix="[, prefix='tmp'[, dir=None[,
          text=False]]]])
     Creates a temporary file in the most secure manner possible.
     There are no race conditions in the file's creation, assuming that
     the platform properly implements the os.O_EXCL flag
     for os.open().  The file is readable and writable only
     by the creating user ID.  If the platform uses permission bits to
     indicate whether a file is executable, the file is executable by
     no one.  The file descriptor is not inherited by child processes.

     Unlike TemporaryFile(), the user of mkstemp()
     is responsible for deleting the temporary file when done with
     it.

     If _suffix_ is specified, the file name will end with that suffix,
     otherwise there will be no suffix.  mkstemp() does not
     put a dot between the file name and the suffix; if you need one,
     put it at the beginning of _suffix_.

     If _prefix_ is specified, the file name will begin with that
     prefix; otherwise, a default prefix is used.

     If _dir_ is specified, the file will be created in that directory;
     otherwise, a default directory is used.  The default directory is
     chosen from a platform-dependent list, but the user of the
     application can control the directory location by setting the
     _TMPDIR_, _TEMP_ or _TMP_ environment variables.  There is thus no
     guarantee that the generated filename will have any nice
     properties, such as not requiring quoting when passed to external
     commands via `os.popen()'.

     If _text_ is specified, it indicates whether to open the file in
     binary mode (the default) or text mode.  On some platforms, this
     makes no difference.

     mkstemp() returns a tuple containing an OS-level
     handle to an open file (as would be returned by os.open())
     and the absolute pathname of that file, in that order.

     New in version 2.3.

 -- Function: tempfile.mkdtemp ([suffix="[, prefix='tmp'[, dir=None]]])
     Creates a temporary directory in the most secure manner possible.
     There are no race conditions in the directory's creation.  The
     directory is readable, writable, and searchable only by the
     creating user ID.

     The user of mkdtemp() is responsible for deleting the
     temporary directory and its contents when done with it.

     The _prefix_, _suffix_, and _dir_ arguments are the same as for
     mkstemp().

     mkdtemp() returns the absolute pathname of the new
     directory.

     New in version 2.3.

 -- Function: tempfile.mktemp ([suffix="[, prefix='tmp'[, dir=None]]])
     Deprecated since version 2.3: Use mkstemp() instead.

     Return an absolute pathname of a file that did not exist at the
     time the call is made.  The _prefix_, _suffix_, and _dir_
     arguments are the same as for mkstemp().

          Warning: Use of this function may introduce a security hole
          in your program.  By the time you get around to doing
          anything with the file name it returns, someone else may have
          beaten you to the punch.  mktemp() usage can be
          replaced easily with NamedTemporaryFile(), passing
          it the `delete=False' parameter:

              >>> f = NamedTemporaryFile(delete=False)
              >>> f
              <open file '<fdopen>', mode 'w+b' at 0x384698>
              >>> f.name
              '/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/tmpG7V1Y0'
              >>> f.write("Hello World!\n")
              >>> f.close()
              >>> os.unlink(f.name)
              >>> os.path.exists(f.name)
              False

  The module uses two global variables that tell it how to construct a
temporary name.  They are initialized at the first call to any of the
functions above.  The caller may change them, but this is discouraged;
use the appropriate function arguments, instead.

 -- Data: tempfile.tempdir
     When set to a value other than `None', this variable defines the
     default value for the _dir_ argument to all the functions defined
     in this module.

     If `tempdir' is unset or `None' at any call to any of the above
     functions, Python searches a standard list of directories and sets
     _tempdir_ to the first one which the calling user can create files
     in.  The list is:

       1. The directory named by the `TMPDIR' environment variable.

       2. The directory named by the `TEMP' environment variable.

       3. The directory named by the `TMP' environment variable.

       4. A platform-specific location:

             * On RiscOS, the directory named by the `Wimp$ScrapDir'
               environment variable.

             * On Windows, the directories `C:\TEMP', `C:\TMP',
               `\TEMP', and `\TMP', in that order.

             * On all other platforms, the directories `/tmp',
               `/var/tmp', and `/usr/tmp', in that order.

       5. As a last resort, the current working directory.

 -- Function: tempfile.gettempdir ()
     Return the directory currently selected to create temporary files
     in. If tempdir is not `None', this simply returns its
     contents; otherwise, the search described above is performed, and
     the result returned.

     New in version 2.3.

 -- Data: tempfile.template
     Deprecated since version 2.0: Use gettempprefix().
     instead.

     When set to a value other than `None', this variable defines the
     prefix of the final component of the filenames returned by
     mktemp().  A string of six random letters and digits is
     appended to the prefix to make the filename unique.  The default
     prefix is `tmp'.

     Older versions of this module used to require that `template' be
     set to `None' after a call to os.fork().; this has not
     been necessary since version 1.5.2.

 -- Function: tempfile.gettempprefix ()
     Return the filename prefix used to create temporary files.  This
     does not contain the directory component.  Using this function is
     preferred over reading the _template_ variable directly.

     New in version 1.5.2.

5.10.7 `glob' -- Unix style pathname pattern expansion
------------------------------------------------------

The glob module finds all the pathnames matching a specified
pattern according to the rules used by the Unix shell.  No tilde
expansion is done, but `*', `?', and character ranges expressed with
`[]' will be correctly matched.  This is done by using the
os.listdir() and fnmatch.fnmatch() functions in
concert, and not by actually invoking a subshell.  (For tilde and shell
variable expansion, use os.path.expanduser() and
os.path.expandvars())

  Latest version of the glob module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/glob.py?view=markup)

 -- Function: glob.glob (pathname)
     Return a possibly-empty list of path names that match _pathname_,
     which must be a string containing a path specification. _pathname_
     can be either absolute (like `/usr/src/Python-1.5/Makefile') or
     relative (like `../../Tools/*/*.gif'), and can contain shell-style
     wildcards. Broken symlinks are included in the results (as in the
     shell).

 -- Function: glob.iglob (pathname)
     Return an iterator which yields the same values as
     glob().  without actually storing them all
     simultaneously.

     New in version 2.5.

  For example, consider a directory containing only the following files:
`1.gif', `2.txt', and `card.gif'.  glob() will produce the
following results.  Notice how any leading components of the path are
preserved.

    >>> import glob
    >>> glob.glob('./[0-9].*')
    ['./1.gif', './2.txt']
    >>> glob.glob('*.gif')
    ['1.gif', 'card.gif']
    >>> glob.glob('?.gif')
    ['1.gif']

Module fnmatch.
     Shell-style filename (not path) expansion

5.10.8 `fnmatch' -- Unix filename pattern matching
--------------------------------------------------

This module provides support for Unix shell-style wildcards, which are
_not_ the same as regular expressions (which are documented in the
re module).  The special characters used in shell-style
wildcards are:

Pattern          Meaning
----------------------------------------------------------
`*'              matches everything
`?'              matches any single character
`[seq]'          matches any character in _seq_
`[!seq]'         matches any character not in _seq_

  Note that the filename separator (`'/'' on Unix) is _not_ special to
this module.  See module glob for pathname expansion (see
glob uses fnmatch() to match pathname segments).
Similarly, filenames starting with a period are not special for this
module, and are matched by the `*' and `?' patterns.

  Latest version of the fnmatch Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/fnmatch.py?view=markup)

 -- Function: fnmatch.fnmatch (filename, pattern)
     Test whether the _filename_ string matches the _pattern_ string,
     returning True or False.  If the operating
     system is case-insensitive, then both parameters will be
     normalized to all lower- or upper-case before the comparison is
     performed.  fnmatchcase() can be used to perform a
     case-sensitive comparison, regardless of whether that's standard
     for the operating system.

     This example will print all file names in the current directory
     with the extension `.txt':

         import fnmatch
         import os

         for file in os.listdir('.'):
             if fnmatch.fnmatch(file, '*.txt'):
                 print file

 -- Function: fnmatch.fnmatchcase (filename, pattern)
     Test whether _filename_ matches _pattern_, returning True
     or False.; the comparison is case-sensitive.

 -- Function: fnmatch.filter (names, pattern)
     Return the subset of the list of _names_ that match _pattern_. It
     is the same as `[n for n in names if fnmatch(n, pattern)]', but
     implemented more efficiently.

     New in version 2.2.

 -- Function: fnmatch.translate (pattern)
     Return the shell-style _pattern_ converted to a regular expression.

     Be aware there is no way to quote meta-characters.

     Example:

         >>> import fnmatch, re
         >>>
         >>> regex = fnmatch.translate('*.txt')
         >>> regex
         '.*\\.txt$'
         >>> reobj = re.compile(regex)
         >>> reobj.match('foobar.txt')
         <_sre.SRE_Match object at 0x...>

Module glob.
     Unix shell-style path expansion.

5.10.9 `linecache' -- Random access to text lines
-------------------------------------------------

The linecache module allows one to get any line from any
file, while attempting to optimize internally, using a cache, the
common case where many lines are read from a single file.  This is used
by the traceback module to retrieve source lines for
inclusion in  the formatted traceback.

  Latest version of the linecache module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/linecache.py?view=markup)The
linecache module defines the following functions:

 -- Function: linecache.getline (filename, lineno[, module_globals])
     Get line _lineno_ from file named _filename_. This function will
     never raise an exception -- it will return `''' on errors (the
     terminating newline character will be included for lines that are
     found).

     If a file named _filename_ is not found, the function will look
     for it in the module search path, `sys.path', after first checking
     for a PEP 302 (http://www.python.org/dev/peps/pep-0302)
     `__loader__' in _module_globals_, in case the module was imported
     from a zipfile or other non-filesystem import source.

     New in version 2.5: The _module_globals_ parameter was added.

 -- Function: linecache.clearcache ()
     Clear the cache.  Use this function if you no longer need lines
     from files previously read using getline().

 -- Function: linecache.checkcache ([filename])
     Check the cache for validity.  Use this function if files in the
     cache  may have changed on disk, and you require the updated
     version.  If _filename_ is omitted, it will check all the entries
     in the cache.

  Example:

    >>> import linecache
    >>> linecache.getline('/etc/passwd', 4)
    'sys:x:3:3:sys:/dev:/bin/sh\n'

5.10.10 `shutil' -- High-level file operations
----------------------------------------------

The shutil module offers a number of high-level operations
on files and collections of files.  In particular, functions are
provided  which support file copying and removal. For operations on
individual files, see also the os module.

  Latest version of the shutil module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/shutil.py?view=markup)

     Warning: Even the higher-level file copying functions (see
     copy(), copy2())  can't copy all file metadata.

     On POSIX platforms, this means that file owner and group are lost
     as well as ACLs.  On Mac OS, the resource fork and other metadata
     are not used.  This means that resources will be lost and file
     type and creator codes will not be correct. On Windows, file
     owners, ACLs and alternate data streams are not copied.

5.10.10.1 Directory and files operations
........................................

 -- Function: shutil.copyfileobj (fsrc, fdst[, length])
     Copy the contents of the file-like object _fsrc_ to the file-like
     object _fdst_.  The integer _length_, if given, is the buffer
     size. In particular, a negative _length_ value means to copy the
     data without looping over the source data in chunks; by default
     the data is read in chunks to avoid uncontrolled memory
     consumption. Note that if the current file position of the _fsrc_
     object is not 0, only the contents from the current file position
     to the end of the file will be copied.

 -- Function: shutil.copyfile (src, dst)
     Copy the contents (no metadata) of the file named _src_ to a file
     named _dst_.  _dst_ must be the complete target file name; look at
     copy() for a copy that accepts a target directory path.
     If _src_ and _dst_ are the same files, see Error is raised.
     The destination location must be writable; otherwise,  an
     IOError exception will be raised. If _dst_ already exists,
     it will be replaced.   Special files such as character or block
     devices and pipes cannot be copied with this function.  _src_ and
     _dst_ are path names given as strings.

 -- Function: shutil.copymode (src, dst)
     Copy the permission bits from _src_ to _dst_.  The file contents,
     owner, and group are unaffected.  _src_ and _dst_ are path names
     given as strings.

 -- Function: shutil.copystat (src, dst)
     Copy the permission bits, last access time, last modification
     time, and flags from _src_ to _dst_.  The file contents, owner,
     and group are unaffected.  _src_ and _dst_ are path names given as
     strings.

 -- Function: shutil.copy (src, dst)
     Copy the file _src_ to the file or directory _dst_.  If _dst_ is a
     directory, a file with the same basename as _src_  is created (or
     overwritten) in the directory specified.  Permission bits are
     copied.  _src_ and _dst_ are path names given as strings.

 -- Function: shutil.copy2 (src, dst)
     Similar to copy(), but metadata is copied as well - in
     fact, this is just copy() followed by copystat().
     This is similar to the Unix command *cp -p*.

 -- Function: shutil.ignore_patterns (*patterns)
     This factory function creates a function that can be used as a
     callable for copytree().'s _ignore_ argument, ignoring
     files and directories that match one of the glob-style _patterns_
     provided.  See the example below.

     New in version 2.6.

 -- Function: shutil.copytree (src, dst[, symlinks=False[,
          ignore=None]])
     Recursively copy an entire directory tree rooted at _src_.  The
     destination directory, named by _dst_, must not already exist; it
     will be created as well as missing parent directories.
     Permissions and times of directories are copied with
     copystat(), individual files are copied using copy2().

     If _symlinks_ is true, symbolic links in the source tree are
     represented as symbolic links in the new tree; if false or
     omitted, the contents of the linked files are copied to the new
     tree.

     If _ignore_ is given, it must be a callable that will receive as
     its arguments the directory being visited by copytree(),
     and a list of its contents, as returned by
     os.listdir().  Since copytree() is called
     recursively, the _ignore_ callable will be called once for each
     directory that is copied.  The callable must return a sequence of
     directory and file names relative to the current directory (i.e. a
     subset of the items in its second argument); these names will then
     be ignored in the copy process.  ignore_patterns() can
     be used to create such a callable that ignores names based on
     glob-style patterns.

     If exception(s) occur, an Error is raised with a list
     of reasons.

     The source code for this should be considered an example rather
     than the ultimate tool.

     Changed in version 2.3: Error is raised if any
     exceptions occur during copying, rather than printing a message.

     Changed in version 2.5: Create intermediate directories needed to
     create _dst_, rather than raising an error. Copy permissions and
     times of directories using copystat().

     Changed in version 2.6: Added the _ignore_ argument to be able to
     influence what is being copied.

 -- Function: shutil.rmtree (path[, ignore_errors[, onerror]])
     Delete an entire directory tree; _path_ must point to a directory
     (but not a symbolic link to a directory).  If _ignore_errors_ is
     true, errors resulting from failed removals will be ignored; if
     false or omitted, such errors are handled by calling a handler
     specified by _onerror_ or, if that is omitted, they raise an
     exception.

     If _onerror_ is provided, it must be a callable that accepts three
     parameters: _function_, _path_, and _excinfo_. The first parameter,
     _function_, is the function which raised the exception; it will be
     os.path.islink(), os.listdir(),
     os.remove() or os.rmdir().  The second parameter,
     _path_, will be the path name passed to _function_.  The third
     parameter, _excinfo_, will be the exception information return by
     sys.exc_info().  Exceptions raised by _onerror_ will
     not be caught.

     Changed in version 2.6: Explicitly check for _path_ being a
     symbolic link and raise OSError.  in that case.

 -- Function: shutil.move (src, dst)
     Recursively move a file or directory to another location.

     If the destination is on the current filesystem, then simply use
     rename.  Otherwise, copy src (with copy2()) to the dst
     and then remove src.

     New in version 2.3.

 -- Exception: shutil.Error
     This exception collects exceptions that raised during a multi-file
     operation. For copytree(), the exception argument is a
     list of 3-tuples (_srcname_, _dstname_, _exception_).

     New in version 2.3.

5.10.10.2 copytree example
..........................

This example is the implementation of the copytree().
function, described above, with the docstring omitted.  It demonstrates
many of the other functions provided by this module.

    def copytree(src, dst, symlinks=False, ignore=None):
        names = os.listdir(src)
        if ignore is not None:
            ignored_names = ignore(src, names)
        else:
            ignored_names = set()

        os.makedirs(dst)
        errors = []
        for name in names:
            if name in ignored_names:
                continue
            srcname = os.path.join(src, name)
            dstname = os.path.join(dst, name)
            try:
                if symlinks and os.path.islink(srcname):
                    linkto = os.readlink(srcname)
                    os.symlink(linkto, dstname)
                elif os.path.isdir(srcname):
                    copytree(srcname, dstname, symlinks, ignore)
                else:
                    copy2(srcname, dstname)
                # XXX What about devices, sockets etc.?
            except (IOError, os.error), why:
                errors.append((srcname, dstname, str(why)))
            # catch the Error from the recursive copytree so that we can
            # continue with other files
            except Error, err:
                errors.extend(err.args[0])
        try:
            copystat(src, dst)
        except WindowsError:
            # can't copy file access times on Windows
            pass
        except OSError, why:
            errors.extend((src, dst, str(why)))
        if errors:
            raise Error(errors)

Another example that uses the ignore_patterns() helper:

    from shutil import copytree, ignore_patterns

    copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))

This will copy everything except `.pyc' files and files or directories
whose name starts with `tmp'.

  Another example that uses the _ignore_ argument to add a logging call:

    from shutil import copytree
    import logging

    def _logpath(path, names):
        logging.info('Working in %s' % path)
        return []   # nothing will be ignored

    copytree(source, destination, ignore=_logpath)

5.10.10.3 Archives operations
.............................

 -- Function: shutil.make_archive (base_name, format[, root_dir[,
          base_dir[, verbose[, dry_run[, owner[, group[, logger]]]]]]])
     Create an archive file (eg. zip or tar) and returns its name.

     _base_name_ is the name of the file to create, including the path,
     minus any format-specific extension. _format_ is the archive
     format: one of "zip", "tar", "bztar" or "gztar".

     _root_dir_ is a directory that will be the root directory of the
     archive; ie. we typically chdir into _root_dir_ before creating the
     archive.

     _base_dir_ is the directory where we start archiving from; ie.
     _base_dir_ will be the common prefix of all files and directories
     in the archive.

     _root_dir_ and _base_dir_ both default to the current directory.

     _owner_ and _group_ are used when creating a tar archive. By
     default, uses the current owner and group.

     New in version 2.7.

 -- Function: shutil.get_archive_formats ()
     Returns a list of supported formats for archiving.  Each element
     of the returned sequence is a tuple `(name, description)'

     By default shutil provides these formats:

        - _gztar_: gzip'ed tar-file

        - _bztar_: bzip2'ed tar-file

        - _tar_: uncompressed tar file

        - _zip_: ZIP file

     You can register new formats or provide your own archiver for any
     existing formats, by using register_archive_format().

     New in version 2.7.

 -- Function: shutil.register_archive_format (name, function[,
          extra_args[, description]])
     Registers an archiver for the format _name_. _function_ is a
     callable that will be used to invoke the archiver.

     If given, _extra_args_ is a sequence of `(name, value)' that will
     be used as extra keywords arguments when the archiver callable is
     used.

     _description_ is used by get_archive_formats() which
     returns the list of archivers. Defaults to an empty list.

     New in version 2.7.

 -- Function: shutil.unregister_archive_format (name)
     Remove the archive format _name_ from the list of supported
     formats.

     New in version 2.7.

5.10.10.4 Archiving example
...........................

In this example, we create a gzip'ed tar-file archive containing all
files found in the `.ssh' directory of the user:

    >>> from shutil import make_archive
    >>> import os
    >>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive'))
    >>> root_dir = os.path.expanduser(os.path.join('~', '.ssh'))
    >>> make_archive(archive_name, 'gztar', root_dir)
    '/Users/tarek/myarchive.tar.gz'

The resulting archive contains:

    $ tar -tzvf /Users/tarek/myarchive.tar.gz
    drwx------ tarek/staff       0 2010-02-01 16:23:40 ./
    -rw-r--r-- tarek/staff     609 2008-06-09 13:26:54 ./authorized_keys
    -rwxr-xr-x tarek/staff      65 2008-06-09 13:26:54 ./config
    -rwx------ tarek/staff     668 2008-06-09 13:26:54 ./id_dsa
    -rwxr-xr-x tarek/staff     609 2008-06-09 13:26:54 ./id_dsa.pub
    -rw------- tarek/staff    1675 2008-06-09 13:26:54 ./id_rsa
    -rw-r--r-- tarek/staff     397 2008-06-09 13:26:54 ./id_rsa.pub
    -rw-r--r-- tarek/staff   37192 2010-02-06 18:23:10 ./known_hosts

5.10.11 `dircache' -- Cached directory listings
-----------------------------------------------

Deprecated since version 2.6: The dircache module has been
removed in Python 3.0.

  The dircache module defines a function for reading
directory listing using a cache, and cache invalidation using the
_mtime_ of the directory.  Additionally, it defines a function to
annotate directories by appending a slash.

  The dircache module defines the following functions:

 -- Function: dircache.reset ()
     Resets the directory cache.

 -- Function: dircache.listdir (path)
     Return a directory listing of _path_, as gotten from
     os.listdir(). Note that unless _path_ changes, further call
     to listdir() will not re-read the directory structure.

     Note that the list returned should be regarded as read-only.
     (Perhaps a future version should change it to return a tuple?)

 -- Function: dircache.opendir (path)
     Same as listdir(). Defined for backwards compatibility.

 -- Function: dircache.annotate (head, list)
     Assume _list_ is a list of paths relative to _head_, and append,
     in place, a `'/'' to each path which points to a directory.

    >>> import dircache
    >>> a = dircache.listdir('/')
    >>> a = a[:] # Copy the return value so we can change 'a'
    >>> a
    ['bin', 'boot', 'cdrom', 'dev', 'etc', 'floppy', 'home', 'initrd', 'lib', 'lost+
    found', 'mnt', 'proc', 'root', 'sbin', 'tmp', 'usr', 'var', 'vmlinuz']
    >>> dircache.annotate('/', a)
    >>> a
    ['bin/', 'boot/', 'cdrom/', 'dev/', 'etc/', 'floppy/', 'home/', 'initrd/', 'lib/
    ', 'lost+found/', 'mnt/', 'proc/', 'root/', 'sbin/', 'tmp/', 'usr/', 'var/', 'vm
    linuz']

5.10.12 `macpath' -- Mac OS 9 path manipulation functions
---------------------------------------------------------

This module is the Mac OS 9 (and earlier) implementation of the
os.path.  module. It can be used to manipulate old-style Macintosh
pathnames on Mac OS X (or any other platform).

  The following functions are available in this module: `normcase()',
`normpath()', `isabs()', `join()', `split()', `isdir()', `isfile()',
`walk()', `exists()'. For other functions available in os.path
dummy counterparts are available.

Section File Objects.
     A description of Python's built-in file objects.

Module os.
     Operating system interfaces, including functions to work with
     files at a lower level than the built-in file object.

5.11 Data Persistence
=====================

The modules described in this chapter support storing Python data in a
persistent form on disk.  The pickle and marshal
modules can turn many Python data types into a stream of bytes and
then recreate the objects from the bytes.  The various DBM-related
modules support a family of hash-based file formats that store a
mapping of strings to other strings.  The bsddb.  module also
provides such disk-based string-to-string mappings based on hashing,
and also supports B-Tree and record-based formats.

  The list of modules described in this chapter is:

5.11.1 `pickle' -- Python object serialization
----------------------------------------------

The pickle module implements a fundamental, but powerful
algorithm for serializing and de-serializing a Python object structure.
"Pickling" is the process whereby a Python object hierarchy is
converted into a byte stream, and "unpickling" is the inverse
operation, whereby a byte stream is converted back into an object
hierarchy.  Pickling (and unpickling) is alternatively known as
"serialization", "marshalling," (1) or "flattening", however, to avoid
confusion, the terms used here are "pickling" and "unpickling".

  This documentation describes both the pickle module and
the cPickle module.

     Warning: The pickle module is not intended to be
     secure against erroneous or maliciously constructed data.  Never
     unpickle data received from an untrusted or unauthenticated source.

5.11.1.1 Relationship to other Python modules
.............................................

The pickle module has an optimized cousin called the
cPickle.  module.  As its name implies, see cPickle is
written in C, so it can be up to 1000 times faster than pickle.
However it does not support subclassing of the Pickler()
and Unpickler() classes, because in cPickle.
these are functions, not classes.  Most applications have no need for
this functionality, and can benefit from the improved performance of
cPickle.  Other than that, the interfaces of the two modules
are nearly identical; the common interface is described in this manual
and differences are pointed out where necessary.  In the following
discussions, we use the term "pickle" to collectively describe the
pickle and cPickle modules.

  The data streams the two modules produce are guaranteed to be
interchangeable.

  Python has a more primitive serialization module called
marshal, but in general pickle should always be the
preferred way to serialize Python objects.  marshal exists
primarily to support Python's `.pyc' files.

  The pickle module differs from marshal.
several significant ways:

   * The pickle module keeps track of the objects it has
     already serialized, so that later references to the same object
     won't be serialized again.  marshal doesn't do this.

     This has implications both for recursive objects and object
     sharing.  Recursive objects are objects that contain references to
     themselves.  These are not handled by marshal, and in fact,
     attempting to marshal recursive objects will crash your Python
     interpreter.  Object sharing happens when there are multiple
     references to the same object in different places in the object
     hierarchy being serialized.  pickle stores such
     objects only once, and ensures that all other references point to
     the master copy.  Shared objects remain shared, which can be very
     important for mutable objects.

   * marshal cannot be used to serialize user-defined
     classes and their instances.  pickle can save and
     restore class instances transparently, however the class
     definition must be importable and live in the same module as when
     the object was stored.

   * The marshal serialization format is not guaranteed to
     be portable across Python versions.  Because its primary job in
     life is to support `.pyc' files, the Python implementers reserve
     the right to change the serialization format in non-backwards
     compatible ways should the need arise.  The pickle.
     serialization format is guaranteed to be backwards compatible
     across Python releases.

  Note that serialization is a more primitive notion than persistence;
although pickle reads and writes file objects, it does not
handle the issue of naming persistent objects, nor the (even more
complicated) issue of concurrent access to persistent objects.  The
pickle module can transform a complex object into a byte
stream and it can transform the byte stream into an object with the
same internal structure.  Perhaps the most obvious thing to do with
these byte streams is to write them onto a file, but it is also
conceivable to send them across a network or store them in a database.
The module shelve provides a simple interface to pickle and
unpickle objects on DBM-style database files.

5.11.1.2 Data stream format
...........................

The data format used by pickle is Python-specific.  This
has the advantage that there are no restrictions imposed by external
standards such as XDR (which can't represent pointer sharing); however
it means that non-Python programs may not be able to reconstruct
pickled Python objects.

  By default, the pickle data format uses a printable ASCII
representation.  This is slightly more voluminous than a binary
representation.  The big advantage of using printable ASCII (and of
some other characteristics of pickle.'s representation) is
that for debugging or recovery purposes it is possible for a human to
read the pickled file with a standard text editor.

  There are currently 3 different protocols which can be used for
pickling.

   * Protocol version 0 is the original ASCII protocol and is backwards
     compatible with earlier versions of Python.

   * Protocol version 1 is the old binary format which is also
     compatible with earlier versions of Python.

   * Protocol version 2 was introduced in Python 2.3.  It provides much
     more efficient pickling of new-style class.es.

  Refer to PEP 307 (http://www.python.org/dev/peps/pep-0307) for more
information.

  If a _protocol_ is not specified, protocol 0 is used. If _protocol_
is specified as a negative value or HIGHEST_PROTOCOL, the
highest protocol version available will be used.

  Changed in version 2.3: Introduced the _protocol_ parameter.

  A binary format, which is slightly more efficient, can be chosen by
specifying a _protocol_ version >= 1.

5.11.1.3 Usage
..............

To serialize an object hierarchy, you first create a pickler, then you
call the pickler's dump() method.  To de-serialize a data
stream, you first create an unpickler, then you call the unpickler's
load() method.  The pickle module provides the
following constant:

 -- Data: pickle.HIGHEST_PROTOCOL
     The highest protocol version available.  This value can be passed
     as a _protocol_ value.

     New in version 2.3.

     Note: Be sure to always open pickle files created with protocols
     >= 1 in binary mode.  For the old ASCII-based pickle protocol 0
     you can use either text mode or binary mode as long as you stay
     consistent.

     A pickle file written with protocol 0 in binary mode will contain
     lone linefeeds as line terminators and therefore will look "funny"
     when viewed in Notepad or other editors which do not support this
     format.

  The pickle module provides the following functions to
make the pickling process more convenient:

 -- Function: pickle.dump (obj, file[, protocol])
     Write a pickled representation of _obj_ to the open file object
     _file_.  This is equivalent to `Pickler(file, protocol).dump(obj)'.

     If the _protocol_ parameter is omitted, protocol 0 is used. If
     _protocol_ is specified as a negative value or
     HIGHEST_PROTOCOL, the highest protocol version will be used.

     Changed in version 2.3: Introduced the _protocol_ parameter.

     _file_ must have a `write()' method that accepts a single string
     argument.  It can thus be a file object opened for writing, a
     StringIO object, or any other custom object that meets
     this interface.

 -- Function: pickle.load (file)
     Read a string from the open file object _file_ and interpret it as
     a pickle data stream, reconstructing and returning the original
     object hierarchy.  This is equivalent to `Unpickler(file).load()'.

     _file_ must have two methods, a `read()' method that takes an
     integer argument, and a readline() method that
     requires no arguments.  Both methods should return a string.  Thus
     _file_ can be a file object opened for reading, a StringIO
     object, or any other custom object that meets this interface.

     This function automatically determines whether the data stream was
     written in binary mode or not.

 -- Function: pickle.dumps (obj[, protocol])
     Return the pickled representation of the object as a string,
     instead of writing it to a file.

     If the _protocol_ parameter is omitted, protocol 0 is used. If
     _protocol_ is specified as a negative value or
     HIGHEST_PROTOCOL, the highest protocol version will be used.

     Changed in version 2.3: The _protocol_ parameter was added.

 -- Function: pickle.loads (string)
     Read a pickled object hierarchy from a string.  Characters in the
     string past the pickled object's representation are ignored.

  The pickle module also defines three exceptions:

 -- Exception: pickle.PickleError
     A common base class for the other exceptions defined below.  This
     inherits from Exception.

 -- Exception: pickle.PicklingError
     This exception is raised when an unpicklable object is passed to
     the dump() method.

 -- Exception: pickle.UnpicklingError
     This exception is raised when there is a problem unpickling an
     object. Note that other exceptions may also be raised during
     unpickling, including (but not necessarily limited to)
     AttributeError, EOFError, ImportError,
     and IndexError.

  The pickle module also exports two callables (2), see
Pickler and Unpickler.:

 -- Class: pickle.Pickler (file[, protocol])
     This takes a file-like object to which it will write a pickle data
     stream.

     If the _protocol_ parameter is omitted, protocol 0 is used. If
     _protocol_ is specified as a negative value or
     HIGHEST_PROTOCOL, the highest protocol version will be used.

     Changed in version 2.3: Introduced the _protocol_ parameter.

     _file_ must have a `write()' method that accepts a single string
     argument.  It can thus be an open file object, a StringIO
     object, or any other custom object that meets this interface.

     Pickler objects define one (or two) public methods:

      -- Method: dump (obj)
          Write a pickled representation of _obj_ to the open file
          object given in the constructor.  Either the binary or ASCII
          format will be used, depending on the value of the _protocol_
          argument passed to the constructor.

      -- Method: clear_memo ()
          Clears the pickler's "memo".  The memo is the data structure
          that remembers which objects the pickler has already seen, so
          that shared or recursive objects pickled by reference and not
          by value.  This method is useful when re-using picklers.

               Note: Prior to Python 2.3, see clear_memo() was
               only available on the picklers created by cPickle.
               In the pickle module, picklers have an
               instance variable called `memo' which is a Python
               dictionary.  So to clear the memo for a pickle
               module pickler, you could do the following:

                   mypickler.memo.clear()

               Code that does not need to support older versions of
               Python should simply use clear_memo().

  It is possible to make multiple calls to the dump().
method of the same Pickler instance.  These must then be
matched to the same number of calls to the load() method of
the corresponding Unpickler.  instance.  If the same object
is pickled by multiple dump() calls, the load().
will all yield references to the same object. (3)

  Unpickler objects are defined as:

 -- Class: pickle.Unpickler (file)
     This takes a file-like object from which it will read a pickle
     data stream.  This class automatically determines whether the data
     stream was written in binary mode or not, so it does not need a
     flag as in the Pickler.  factory.

     _file_ must have two methods, a `read()' method that takes an
     integer argument, and a readline() method that
     requires no arguments.  Both methods should return a string.  Thus
     _file_ can be a file object opened for reading, a StringIO
     object, or any other custom object that meets this interface.

     Unpickler objects have one (or two) public methods:

      -- Method: load ()
          Read a pickled object representation from the open file
          object given in the constructor, and return the reconstituted
          object hierarchy specified therein.

          This method automatically determines whether the data stream
          was written in binary mode or not.

      -- Method: noload ()
          This is just like load() except that it doesn't
          actually create any objects.  This is useful primarily for
          finding what's called "persistent ids" that may be referenced
          in a pickle data stream.  See section The pickle protocol
          below for more details.

          *Note_* the noload() method is currently only
          available on Unpickler objects created with the
          cPickle module.  pickle module
          Unpickler.s do not have the noload().  method.

5.11.1.4 What can be pickled and unpickled?
...........................................

The following types can be pickled:

   * `None', `True', and `False'

   * integers, long integers, floating point numbers, complex numbers

   * normal and Unicode strings

   * tuples, lists, sets, and dictionaries containing only picklable
     objects

   * functions defined at the top level of a module

   * built-in functions defined at the top level of a module

   * classes that are defined at the top level of a module

   * instances of such classes whose `__dict__' or
     __setstate__() is picklable  (see section The pickle protocol
     for details)

  Attempts to pickle unpicklable objects will raise the
PicklingError.  exception; when this happens, an unspecified
number of bytes may have already been written to the underlying file.
Trying to pickle a highly recursive data structure may exceed the
maximum recursion depth, a RuntimeError will be raised in
this case. You can carefully raise this limit with
sys.setrecursionlimit().

  Note that functions (built-in and user-defined) are pickled by "fully
qualified" name reference, not by value.  This means that only the
function name is pickled, along with the name of module the function is
defined in.  Neither the function's code, nor any of its function
attributes are pickled.  Thus the defining module must be importable in
the unpickling environment, and the module must contain the named
object, otherwise an exception will be raised. (4)

  Similarly, classes are pickled by named reference, so the same
restrictions in the unpickling environment apply.  Note that none of
the class's code or data is pickled, so in the following example the
class attribute `attr' is not restored in the unpickling environment:

    class Foo:
        attr = 'a class attr'

    picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be
defined in the top level of a module.

  Similarly, when class instances are pickled, their class's code and
data are not pickled along with them.  Only the instance data are
pickled.  This is done on purpose, so you can fix bugs in a class or
add methods to the class and still load objects that were created with
an earlier version of the class.  If you plan to have long-lived
objects that will see many versions of a class, it may be worthwhile to
put a version number in the objects so that suitable conversions can be
made by the class's __setstate__() method.

5.11.1.5 The pickle protocol
............................

This section describes the "pickling protocol" that defines the
interface between the pickler/unpickler and the objects that are being
serialized.  This protocol provides a standard way for you to define,
customize, and control how your objects are serialized and
de-serialized.  The description in this section doesn't cover specific
customizations that you can employ to make the unpickling environment
slightly safer from untrusted pickle data streams; see section
Subclassing Unpicklers for more details.

5.11.1.6 Pickling and unpickling normal class instances
.......................................................

 -- Method: object.__getinitargs__ ()
     When a pickled class instance is unpickled, its __init__()
     method is normally _not_ invoked.  If it is desirable that
     the __init__() method be called on unpickling, an
     old-style class can define a method __getinitargs__(),
     which should return a _tuple_ containing the arguments to be
     passed to the class constructor (see __init__() for
     example).  The __getinitargs__() method is called at
     pickle time; the tuple it returns is incorporated in the pickle
     for the instance.

 -- Method: object.__getnewargs__ ()
     New-style types can provide a __getnewargs__() method
     that is used for protocol 2.  Implementing this method is needed
     if the type establishes some internal invariants when the instance
     is created, or if the memory allocation is affected by the values
     passed to the __new__() method for the type (as it is
     for tuples and strings).  Instances of a new-style class.
     `C' are created using

         obj = C.__new__(C, *args)

     where _args_ is the result of calling __getnewargs__().
     on the original object; if there is no __getnewargs__(),
     an empty tuple is assumed.

 -- Method: object.__getstate__ ()
     Classes can further influence how their instances are pickled; if
     the class defines the method __getstate__(), it is
     called and the return state is pickled as the contents for the
     instance, instead of the contents of the instance's dictionary.
     If there is no __getstate__() method, the instance's
     __dict__ is pickled.

 -- Method: object.__setstate__ (state)
     Upon unpickling, if the class also defines the method
     __setstate__(), it is called with the unpickled state. (5) If
     there is no __setstate__() method, the pickled state
     must be a dictionary and its items are assigned to the new
     instance's dictionary.  If a class defines both
     __getstate__() and __setstate__(), the state
     object needn't be a dictionary and these methods can do what they
     want. (6)

          Note: For new-style class.es, if
          __getstate__() returns a false value, the
          __setstate__() method will not be called.

     Note: At unpickling time, some methods like __getattr__(),
     see __getattribute__(), or __setattr__().
     may be called upon the instance.  In case those methods rely on
     some internal invariant being true, the type should implement
     either __getinitargs__() or __getnewargs__()
     to establish such an invariant; otherwise, neither
     __new__() nor __init__() will be called.

5.11.1.7 Pickling and unpickling extension types
................................................

 -- Method: object.__reduce__ ()
     When the `Pickler' encounters an object of a type it knows nothing
     about -- such as an extension type -- it looks in two places for a
     hint of how to pickle it.  One alternative is for the object to
     implement a __reduce__() method.  If provided, at
     pickling time __reduce__().  will be called with no
     arguments, and it must return either a string or a tuple.

     If a string is returned, it names a global variable whose contents
     are pickled as normal.  The string returned by __reduce__()
     should be the object's local name relative to its module; the
     pickle module searches the module namespace to determine the
     object's module.

     When a tuple is returned, it must be between two and five elements
     long.  Optional elements can either be omitted, or `None' can be
     provided as their value.  The contents of this tuple are pickled
     as normal and used to reconstruct the object at unpickling time.
     The semantics of each element are:

        * A callable object that will be called to create the initial
          version of the object.  The next element of the tuple will
          provide arguments for this callable, and later elements
          provide additional state information that will subsequently
          be used to fully reconstruct the pickled data.

          In the unpickling environment this object must be either a
          class, a callable registered as a "safe constructor" (see
          below), or it must have an attribute
          `__safe_for_unpickling__' with a true value. Otherwise, an
          `UnpicklingError' will be raised in the unpickling
          environment.  Note that as usual, the callable itself is
          pickled by name.

        * A tuple of arguments for the callable object.

          Changed in version 2.5: Formerly, this argument could also be
          `None'.

        * Optionally, the object's state, which will be passed to the
          object's __setstate__() method as described in
          section Pickling and unpickling normal class instances.
          If the object has no __setstate__() method,
          then, as above, the value must be a dictionary and it will be
          added to the object's __dict__.

        * Optionally, an iterator (and not a sequence) yielding
          successive list items.  These list items will be pickled, and
          appended to the object using either `obj.append(item)' or
          `obj.extend(list_of_items)'.  This is primarily used for list
          subclasses, but may be used by other classes as long as they
          have `append()' and `extend()' methods with the appropriate
          signature.  (Whether `append()' or `extend()' is used depends
          on which pickle protocol version is used as well as the
          number of items to append, so both must be supported.)

        * Optionally, an iterator (not a sequence) yielding successive
          dictionary items, which should be tuples of the form `(key,
          value)'.  These items will be pickled and stored to the
          object using `obj[key] = value'. This is primarily used for
          dictionary subclasses, but may be used by other classes as
          long as they implement __setitem__().

 -- Method: object.__reduce_ex__ (protocol)
     It is sometimes useful to know the protocol version when
     implementing __reduce__().  This can be done by
     implementing a method named __reduce_ex__() instead of
     __reduce__(). See __reduce_ex__(), when it
     exists, is called in preference over __reduce__(). (you
     may still provide __reduce__() for backwards
     compatibility).  The __reduce_ex__() method will be
     called with a single integer argument, the protocol version.

     The object class implements both __reduce__()
     and __reduce_ex__().; however, if a subclass
     overrides __reduce__().  but not __reduce_ex__(),
     the __reduce_ex__() implementation detects this
     and calls __reduce__().

  An alternative to implementing a __reduce__() method on
the object to be pickled, is to register the callable with the
copy_reg module.  This module provides a way for programs to
register "reduction functions" and constructors for user-defined types.
 Reduction functions have the same semantics and interface as the
__reduce__() method described above, except that they are called
with a single argument, the object to be pickled.

  The registered constructor is deemed a "safe constructor" for
purposes of unpickling as described above.

5.11.1.8 Pickling and unpickling external objects
.................................................

For the benefit of object persistence, the pickle module
supports the notion of a reference to an object outside the pickled
data stream.  Such objects are referenced by a "persistent id", which
is just an arbitrary string of printable ASCII characters. The
resolution of such names is not defined by the pickle.
module; it will delegate this resolution to user defined functions on
the pickler and unpickler. (7)

  To define external persistent id resolution, you need to set the
`persistent_id' attribute of the pickler object and the
`persistent_load' attribute of the unpickler object.

  To pickle objects that have an external persistent id, the pickler
must have a custom `persistent_id()' method that takes an object as an
argument and returns either `None' or the persistent id for that
object.  When `None' is returned, the pickler simply pickles the object
as normal.  When a persistent id string is returned, the pickler will
pickle that string, along with a marker so that the unpickler will
recognize the string as a persistent id.

  To unpickle external objects, the unpickler must have a custom
`persistent_load()' function that takes a persistent id string and
returns the referenced object.

  Here's a silly example that _might_ shed more light:

    import pickle
    from cStringIO import StringIO

    src = StringIO()
    p = pickle.Pickler(src)

    def persistent_id(obj):
        if hasattr(obj, 'x'):
            return 'the value %d' % obj.x
        else:
            return None

    p.persistent_id = persistent_id

    class Integer:
        def __init__(self, x):
            self.x = x
        def __str__(self):
            return 'My name is integer %d' % self.x

    i = Integer(7)
    print i
    p.dump(i)

    datastream = src.getvalue()
    print repr(datastream)
    dst = StringIO(datastream)

    up = pickle.Unpickler(dst)

    class FancyInteger(Integer):
        def __str__(self):
            return 'I am the integer %d' % self.x

    def persistent_load(persid):
        if persid.startswith('the value '):
            value = int(persid.split()[2])
            return FancyInteger(value)
        else:
            raise pickle.UnpicklingError, 'Invalid persistent id'

    up.persistent_load = persistent_load

    j = up.load()
    print j

In the cPickle module, the unpickler's `persistent_load'
attribute can also be set to a Python list, in which case, when the
unpickler reaches a persistent id, the persistent id string will simply
be appended to this list.  This functionality exists so that a pickle
data stream can be "sniffed" for object references without actually
instantiating all the objects in a pickle.  (8)  Setting
`persistent_load' to a list is usually used in conjunction with the
`noload()' method on the Unpickler.

5.11.1.9 Subclassing Unpicklers
...............................

By default, unpickling will import any class that it finds in the
pickle data.  You can control exactly what gets unpickled and what gets
called by customizing your unpickler.  Unfortunately, exactly how you
do this is different depending on whether you're using pickle
or cPickle. (9)

  In the pickle module, you need to derive a subclass from
`Unpickler', overriding the `load_global()' method.  `load_global()'
should read two lines from the pickle data stream where the first line
will the name of the module containing the class and the second line
will be the name of the instance's class.  It then looks up the class,
possibly importing the module and digging out the attribute, then it
appends what it finds to the unpickler's stack.  Later on, this class
will be assigned to the `__class__' attribute of an empty class, as a
way of magically creating an instance without calling its class's
__init__(). Your job (should you choose to accept it), would be to
have `load_global()' push onto the unpickler's stack, a known safe
version of any class you deem safe to unpickle.  It is up to you to
produce such a class.  Or you could raise an error if you want to
disallow all unpickling of instances.  If this sounds like a hack,
you're right.  Refer to the source code to make this work.

  Things are a little cleaner with cPickle, but not by much.
To control what gets unpickled, you can set the unpickler's
`find_global' attribute to a function or `None'.  If it is `None' then
any attempts to unpickle instances will raise an `UnpicklingError'.  If
it is a function, then it should accept a module name and a class name,
and return the corresponding class object.  It is responsible for
looking up the class and performing any necessary imports, and it may
raise an error to prevent instances of the class from being unpickled.

  The moral of the story is that you should be really careful about the
source of the strings your application unpickles.

5.11.1.10 Example
.................

For the simplest code, use the `dump()' and `load()' functions.  Note
that a self-referencing list is pickled and restored correctly.

    import pickle

    data1 = {'a': [1, 2.0, 3, 4+6j],
             'b': ('string', u'Unicode string'),
             'c': None}

    selfref_list = [1, 2, 3]
    selfref_list.append(selfref_list)

    output = open('data.pkl', 'wb')

    # Pickle dictionary using protocol 0.
    pickle.dump(data1, output)

    # Pickle the list using the highest protocol available.
    pickle.dump(selfref_list, output, -1)

    output.close()

The following example reads the resulting pickled data.  When reading a
pickle-containing file, you should open the file in binary mode because
you can't be sure if the ASCII or binary format was used.

    import pprint, pickle

    pkl_file = open('data.pkl', 'rb')

    data1 = pickle.load(pkl_file)
    pprint.pprint(data1)

    data2 = pickle.load(pkl_file)
    pprint.pprint(data2)

    pkl_file.close()

Here's a larger example that shows how to modify pickling behavior for
a class.  The `TextReader' class opens a text file, and returns the
line number and line contents each time its readline().
method is called. If a `TextReader' instance is pickled, all attributes
_except_ the file object member are saved. When the instance is
unpickled, the file is reopened, and reading resumes from the last
location. The __setstate__() and __getstate__().
methods are used to implement this behavior.

    #!/usr/local/bin/python

    class TextReader:
        """Print and number lines in a text file."""
        def __init__(self, file):
            self.file = file
            self.fh = open(file)
            self.lineno = 0

        def readline(self):
            self.lineno = self.lineno + 1
            line = self.fh.readline()
            if not line:
                return None
            if line.endswith("\n"):
                line = line[:-1]
            return "%d: %s" % (self.lineno, line)

        def __getstate__(self):
            odict = self.__dict__.copy() # copy the dict since we change it
            del odict['fh']              # remove filehandle entry
            return odict

        def __setstate__(self, dict):
            fh = open(dict['file'])      # reopen file
            count = dict['lineno']       # read from file...
            while count:                 # until line count is restored
                fh.readline()
                count = count - 1
            self.__dict__.update(dict)   # update attributes
            self.fh = fh                 # save the file object

A sample usage might be something like this:

    >>> import TextReader
    >>> obj = TextReader.TextReader("TextReader.py")
    >>> obj.readline()
    '1: #!/usr/local/bin/python'
    >>> obj.readline()
    '2: '
    >>> obj.readline()
    '3: class TextReader:'
    >>> import pickle
    >>> pickle.dump(obj, open('save.p', 'wb'))

If you want to see that pickle works across Python
processes, start another Python session, before continuing.  What
follows can happen from either the same process or a new process.

    >>> import pickle
    >>> reader = pickle.load(open('save.p', 'rb'))
    >>> reader.readline()
    '4:     """Print and number lines in a text file."""'

Module copy_reg.
     Pickle interface constructor registration for extension types.

Module shelve.
     Indexed databases of objects; uses pickle.

Module copy.
     Shallow and deep object copying.

Module marshal.
     High-performance serialization of built-in types.

5.11.2 `cPickle' -- A faster `pickle'
-------------------------------------

The cPickle module supports serialization and
de-serialization of Python objects, providing an interface and
functionality nearly identical to the pickle module.  There
are several differences, the most important being performance and
subclassability.

  First, see cPickle can be up to 1000 times faster than
pickle because the former is implemented in C.  Second, in the
cPickle module the callables `Pickler()' and `Unpickler()'
are functions, not classes.  This means that you cannot use them to
derive custom pickling and unpickling subclasses.  Most applications
have no need for this functionality and should benefit from the greatly
improved performance of the cPickle module.

  The pickle data stream produced by pickle and
cPickle are identical, so it is possible to use pickle.
and cPickle.  interchangeably with existing pickles. (10)

  There are additional minor differences in API between cPickle
and pickle, however for most applications, they are
interchangeable.  More documentation is provided in the pickle
module documentation, which includes a list of the documented
differences.

Footnotes
.........

     (1) Don't confuse this with the marshal module

     (2) In the pickle module these callables are classes,
     which you could subclass to customize the behavior.  However, in
     the cPickle module these callables are factory
     functions and so cannot be subclassed.  One common reason to
     subclass is to control what objects can actually be unpickled.
     See section Subclassing Unpicklers for more details.

     (3) _Warning_: this is intended for pickling multiple objects
     without intervening modifications to the objects or their parts.
     If you modify an object and then pickle it again using the same
     `Pickler' instance, the object is not pickled again -- a reference
     to it is pickled and the `Unpickler' will return the old value,
     not the modified one. There are two problems here: (1) detecting
     changes, and (2) marshalling a minimal set of changes.  Garbage
     Collection may also become a problem here.

     (4) The exception raised will likely be an ImportError.
     or an AttributeError but it could be something else.

     (5) These methods can also be used to implement copying class
     instances.

     (6) This protocol is also used by the shallow and deep copying
     operations defined in the copy module.

     (7) The actual mechanism for associating these user defined
     functions is slightly different for pickle and
     cPickle.  The description given here works the same for both
     implementations.  Users of the pickle module could
     also use subclassing to effect the same results, overriding the
     `persistent_id()' and `persistent_load()' methods in the derived
     classes.

     (8) We'll leave you with the image of Guido and Jim sitting around
     sniffing pickles in their living rooms.

     (9) A word of caution: the mechanisms described here use internal
     attributes and methods, which are subject to change in future
     versions of Python.  We intend to someday provide a common
     interface for controlling this behavior, which will work in either
     pickle or cPickle.

     (10) Since the pickle data format is actually a tiny
     stack-oriented programming language, and some freedom is taken in
     the encodings of certain objects, it is possible that the two
     modules produce different data streams for the same input objects.
     However it is guaranteed that they will always be able to read each
     other's data streams.

5.11.3 `copy_reg' -- Register `pickle' support functions
--------------------------------------------------------

     Note: The copy_reg module has been renamed to `copyreg'
     in Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

The copy_reg module provides support for the pickle
and cPickle modules.  The copy module is
likely to use this in the future as well.  It provides configuration
information about object constructors which are not classes.  Such
constructors may be factory functions or class instances.

 -- Function: copy_reg.constructor (object)
     Declares _object_ to be a valid constructor.  If _object_ is not
     callable (and hence not valid as a constructor), raises
     TypeError.

 -- Function: copy_reg.pickle (type, function[, constructor])
     Declares that _function_ should be used as a "reduction" function
     for objects of type _type_; _type_ must not be a "classic" class
     object.  (Classic classes are handled differently; see the
     documentation for the pickle module for details.)
     _function_ should return either a string or a tuple containing two
     or three elements.

     The optional _constructor_ parameter, if provided, is a callable
     object which can be used to reconstruct the object when called
     with the tuple of arguments returned by _function_ at pickling
     time.  TypeError will be raised if _object_ is a class
     or _constructor_ is not callable.

     See the pickle module for more details on the
     interface expected of _function_ and _constructor_.

5.11.4 `shelve' -- Python object persistence
--------------------------------------------

A "shelf" is a persistent, dictionary-like object.  The difference with
"dbm" databases is that the values (not the keys!) in a shelf can be
essentially arbitrary Python objects -- anything that the pickle
module can handle.  This includes most class instances, recursive
data types, and objects containing lots of shared  sub-objects.  The
keys are ordinary strings.

  Latest version of the shelve module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/shelve.py?view=markup)

 -- Function: shelve.open (filename[, flag='c'[, protocol=None[,
          writeback=False]]])
     Open a persistent dictionary.  The filename specified is the base
     filename for the underlying database.  As a side-effect, an
     extension may be added to the filename and more than one file may
     be created.  By default, the underlying database file is opened
     for reading and writing.  The optional _flag_ parameter has the
     same interpretation as the _flag_ parameter of
     anydbm.open().

     By default, version 0 pickles are used to serialize values.  The
     version of the pickle protocol can be specified with the
     _protocol_ parameter.

     Changed in version 2.3: The _protocol_ parameter was added.

     Because of Python semantics, a shelf cannot know when a mutable
     persistent-dictionary entry is modified.  By default modified
     objects are written _only_ when assigned to the shelf (see
     Example).  If the optional _writeback_ parameter is set to
     _True_, all entries accessed are also cached in memory, and
     written back on sync() and close().; this
     can make it handier to mutate mutable entries in the persistent
     dictionary, but, if many entries are accessed, it can consume vast
     amounts of memory for the cache, and it can make the close
     operation very slow since all accessed entries are written back
     (there is no way to determine which accessed entries are mutable,
     nor which ones were actually mutated).

          Note: Do not rely on the shelf being closed automatically;
          always call `close()' explicitly when you don't need it any
          more, or use a with statement with
          contextlib.closing().

     Warning: Because the shelve module is backed by
     pickle, it is insecure to load a shelf from an untrusted
     source.  Like with pickle, loading a shelf can execute arbitrary
     code.

  Shelf objects support all methods supported by dictionaries.  This
eases the transition from dictionary based scripts to those requiring
persistent storage.

  Two additional methods are supported:

 -- Method: Shelf.sync ()
     Write back all entries in the cache if the shelf was opened with
     _writeback_ set to True.  Also empty the cache and
     synchronize the persistent dictionary on disk, if feasible.  This
     is called automatically when the shelf is closed with
     close().

 -- Method: Shelf.close ()
     Synchronize and close the persistent _dict_ object.  Operations on
     a closed shelf will fail with a ValueError.

  Persistent dictionary recipe
(http://code.activestate.com/recipes/576642/)with widely supported
storage formats and having the speed of native dictionaries.

5.11.4.1 Restrictions
.....................

   * The choice of which database package will be used (such as
     dbm, gdbm or bsddb) depends on which
     interface is available.  Therefore it is not safe to open the
     database directly using dbm.  The database is also
     (unfortunately) subject to the limitations of dbm, if it
     is used -- this means that (the pickled representation of) the
     objects stored in the database should be fairly small, and in rare
     cases key collisions may cause the database to refuse updates.

   * The shelve module does not support _concurrent_
     read/write access to shelved objects.  (Multiple simultaneous read
     accesses are safe.)  When a program has a shelf open for writing,
     no other program should have it open for reading or writing.  Unix
     file locking can be used to solve this, but this differs across
     Unix versions and requires knowledge about the database
     implementation used.

 -- Class: shelve.Shelf (dict[, protocol=None[, writeback=False]])
     A subclass of UserDict.DictMixin which stores pickled
     values in the _dict_ object.

     By default, version 0 pickles are used to serialize values.  The
     version of the pickle protocol can be specified with the
     _protocol_ parameter. See the pickle documentation for
     a discussion of the pickle protocols.

     Changed in version 2.3: The _protocol_ parameter was added.

     If the _writeback_ parameter is `True', the object will hold a
     cache of all entries accessed and write them back to the _dict_ at
     sync and close times.  This allows natural operations on mutable
     entries, but can consume much more memory and make sync and close
     take a long time.

 -- Class: shelve.BsdDbShelf (dict[, protocol=None[, writeback=False]])
     A subclass of Shelf which exposes `first()', `next()',
     `previous()', `last()' and `set_location()' which are available in
     the bsddb module but not in other database modules.
     The _dict_ object passed to the constructor must support those
     methods.  This is generally accomplished by calling one of
     bsddb.hashopen(), bsddb.btopen() or
     bsddb.rnopen().  The optional _protocol_ and _writeback_
     parameters have the same interpretation as for the Shelf
     class.

 -- Class: shelve.DbfilenameShelf (filename[, flag='c'[,
          protocol=None[, writeback=False]]])
     A subclass of Shelf which accepts a _filename_ instead
     of a dict-like object.  The underlying file will be opened using
     anydbm.open().  By default, the file will be created
     and opened for both read and write.  The optional _flag_ parameter
     has the same interpretation as for the open().
     function.  The optional _protocol_ and _writeback_ parameters have
     the same interpretation as for the Shelf class.

5.11.4.2 Example
................

To summarize the interface (`key' is a string, `data' is an arbitrary
object):

    import shelve

    d = shelve.open(filename) # open -- file may get suffix added by low-level
                              # library

    d[key] = data   # store data at key (overwrites old data if
                    # using an existing key)
    data = d[key]   # retrieve a COPY of data at key (raise KeyError if no
                    # such key)
    del d[key]      # delete data stored at key (raises KeyError
                    # if no such key)
    flag = d.has_key(key)   # true if the key exists
    klist = d.keys() # a list of all existing keys (slow!)

    # as d was opened WITHOUT writeback=True, beware:
    d['xx'] = range(4)  # this works as expected, but...
    d['xx'].append(5)   # *this doesn't!* -- d['xx'] is STILL range(4)!

    # having opened d without writeback=True, you need to code carefully:
    temp = d['xx']      # extracts the copy
    temp.append(5)      # mutates the copy
    d['xx'] = temp      # stores the copy right back, to persist it

    # or, d=shelve.open(filename,writeback=True) would let you just code
    # d['xx'].append(5) and have it work as expected, BUT it would also
    # consume more memory and make the d.close() operation slower.

    d.close()       # close it

Module anydbm.
     Generic interface to `dbm'-style databases.

Module bsddb.
     BSD `db' database interface.

Module dbhash.
     Thin layer around the bsddb which provides an
     open().  function like the other database modules.

Module dbm.
     Standard Unix database interface.

Module dumbdbm.
     Portable implementation of the `dbm' interface.

Module gdbm.
     GNU database interface, based on the `dbm' interface.

Module pickle.
     Object serialization used by shelve.

Module cPickle.
     High-performance version of pickle.

5.11.5 `marshal' -- Internal Python object serialization
--------------------------------------------------------

This module contains functions that can read and write Python values in
a binary format.  The format is specific to Python, but independent of
machine architecture issues (e.g., you can write a Python value to a
file on a PC, transport the file to a Sun, and read it back there).
Details of the format are undocumented on purpose; it may change
between Python versions (although it rarely does). (1)

  This is not a general "persistence" module.  For general persistence
and transfer of Python objects through RPC calls, see the modules
pickle and shelve.  The marshal module
exists mainly to support reading and writing the "pseudo-compiled" code
for Python modules of `.pyc' files.  Therefore, the Python maintainers
reserve the right to modify the marshal format in backward incompatible
ways should the need arise.  If you're serializing and de-serializing
Python objects, use the pickle module instead - the
performance is comparable, version independence is guaranteed, and
pickle supports a substantially wider range of objects than marshal.

     Warning: The marshal module is not intended to be
     secure against erroneous or maliciously constructed data.  Never
     unmarshal data received from an untrusted or unauthenticated
     source.

  Not all Python object types are supported; in general, only objects
whose value is independent from a particular invocation of Python can
be written and read by this module.  The following types are supported:
booleans, integers, long integers, floating point numbers, complex
numbers, strings, Unicode objects, tuples, lists, sets, frozensets,
dictionaries, and code objects, where it should be understood that
tuples, lists, sets, frozensets and dictionaries are only supported as
long as the values contained therein are themselves supported; and
recursive lists, sets and dictionaries should not be written (they will
cause infinite loops).  The singletons None, Ellipsis
and StopIteration can also be marshalled and
unmarshalled.

     Warning: On machines where C's `long int' type has more than 32
     bits (such as the DEC Alpha), it is possible to create plain
     Python integers that are longer than 32 bits. If such an integer
     is marshaled and read back in on a machine where C's `long int'
     type has only 32 bits, a Python long integer object is returned
     instead.  While of a different type, the numeric value is the
     same.  (This behavior is new in Python 2.2.  In earlier versions,
     all but the least-significant 32 bits of the value were lost, and
     a warning message was printed.)

  There are functions that read/write files as well as functions
operating on strings.

  The module defines these functions:

 -- Function: marshal.dump (value, file[, version])
     Write the value on the open file.  The value must be a supported
     type.  The file must be an open file object such as `sys.stdout'
     or returned by open() or os.popen().  It
     must be opened in binary mode (`'wb'' or `'w+b'').

     If the value has (or contains an object that has) an unsupported
     type, a ValueError exception is raised -- but garbage
     data will also be written to the file.  The object will not be
     properly read back by load().

     New in version 2.4: The _version_ argument indicates the data
     format that `dump' should use (see below).

 -- Function: marshal.load (file)
     Read one value from the open file and return it.  If no valid
     value is read (e.g. because the data has a different Python
     version's incompatible marshal format), raise EOFError,
     ValueError or TypeError.  The file must be
     an open file object opened in binary mode (`'rb'' or `'r+b'').

          Note: If an object containing an unsupported type was
          marshalled with dump(), load() will
          substitute `None' for the unmarshallable type.

 -- Function: marshal.dumps (value[, version])
     Return the string that would be written to a file by `dump(value,
     file)'.  The value must be a supported type.  Raise a
     ValueError exception if value has (or contains an object
     that has) an unsupported type.

     New in version 2.4: The _version_ argument indicates the data
     format that `dumps' should use (see below).

 -- Function: marshal.loads (string)
     Convert the string to a value.  If no valid value is found, raise
     EOFError, ValueError or TypeError.
     Extra characters in the string are ignored.

  In addition, the following constants are defined:

 -- Data: marshal.version
     Indicates the format that the module uses. Version 0 is the
     historical format, version 1 (added in Python 2.4) shares interned
     strings and version 2 (added in Python 2.5) uses a binary format
     for floating point numbers. The current version is 2.

     New in version 2.4.

Footnotes
.........

     (1) The name of this module stems from a bit of terminology used
     by the designers of Modula-3 (amongst others), who use the term
     "marshalling" for shipping of data around in a self-contained
     form. Strictly speaking, "to marshal" means to convert some data
     from internal to external form (in an RPC buffer for instance) and
     "unmarshalling" for the reverse process.

5.11.6 `anydbm' -- Generic access to DBM-style databases
--------------------------------------------------------

     Note: The anydbm module has been renamed to dbm
     in Python 3.0.  The 2to3 tool will automatically
     adapt imports when converting your sources to 3.0.

anydbm is a generic interface to variants of the DBM database
-- see dbhash. (requires bsddb), see gdbm, or
dbm.  If none of these modules is installed, the
slow-but-simple implementation in module dumbdbm will be
used.

 -- Function: anydbm.open (filename[, flag[, mode]])
     Open the database file _filename_ and return a corresponding
     object.

     If the database file already exists, the whichdb.
     module is used to determine its type and the appropriate module is
     used; if it does not exist, the first module listed above that can
     be imported is used.

     The optional _flag_ argument must be one of these values:

     Value         Meaning
     --------------------------------------------------------------
     `'r''         Open existing database for reading only
                   (default)
     `'w''         Open existing database for reading and writing
     `'c''         Open database for reading and writing,
                   creating it if it doesn't exist
     `'n''         Always create a new, empty database, open for
                   reading and writing

     If not specified, the default value is `'r''.

     The optional _mode_ argument is the Unix mode of the file, used
     only when the database has to be created.  It defaults to octal
     `0666' (and will be modified by the prevailing umask).

 -- Exception: anydbm.error
     A tuple containing the exceptions that can be raised by each of
     the supported modules, with a unique exception also named
     anydbm.error as the first item -- the latter is used when
     anydbm.error is raised.

  The object returned by open() supports most of the same
functionality as dictionaries; keys and their corresponding values can
be stored, retrieved, and deleted, and the `has_key()' and `keys()'
methods are available.  Keys and values must always be strings.

  The following example records some hostnames and a corresponding
title,  and then prints out the contents of the database:

    import anydbm

    # Open database, creating it if necessary.
    db = anydbm.open('cache', 'c')

    # Record some values
    db['www.python.org'] = 'Python Website'
    db['www.cnn.com'] = 'Cable News Network'

    # Loop through contents.  Other dictionary methods
    # such as .keys(), .values() also work.
    for k, v in db.iteritems():
        print k, '\t', v

    # Storing a non-string key or value will raise an exception (most
    # likely a TypeError).
    db['www.yahoo.com'] = 4

    # Close when done.
    db.close()

Module dbhash.
     BSD `db' database interface.

Module dbm.
     Standard Unix database interface.

Module dumbdbm.
     Portable implementation of the `dbm' interface.

Module gdbm.
     GNU database interface, based on the `dbm' interface.

Module shelve.
     General object persistence built on top of  the Python `dbm'
     interface.

Module whichdb.
     Utility module used to determine the type of an existing database.

5.11.7 `whichdb' -- Guess which DBM module created a database
-------------------------------------------------------------

     Note: The whichdb module's only function has been put
     into the dbm.  module in Python 3.0.  The 2to3
     tool will automatically adapt imports when converting your
     sources to 3.0.

The single function in this module attempts to guess which of the
several simple database modules available-see dbm, gdbm,
or dbhash.-should be used to open a given file.

 -- Function: whichdb.whichdb (filename)
     Returns one of the following values: `None' if the file can't be
     opened because it's unreadable or doesn't exist; the empty string
     (`''') if the file's format can't be guessed; or a string
     containing the required module name, such as `'dbm'' or `'gdbm''.

5.11.8 `dbm' -- Simple "database" interface
-------------------------------------------

     Note: The dbm module has been renamed to `dbm.ndbm' in
     Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

The dbm module provides an interface to the Unix "(n)dbm"
library.  Dbm objects behave like mappings (dictionaries), except that
keys and values are always strings. Printing a dbm object doesn't print
the keys and values, and the `items()' and `values()' methods are not
supported.

  This module can be used with the "classic" ndbm interface, the BSD DB
compatibility interface, or the GNU GDBM compatibility interface. On
Unix, the *configure* script will attempt to locate the appropriate
header file to simplify building this module.

  The module defines the following:

 -- Exception: dbm.error
     Raised on dbm-specific errors, such as I/O errors. KeyError
     is raised for general mapping errors like specifying an
     incorrect key.

 -- Data: dbm.library
     Name of the `ndbm' implementation library used.

 -- Function: dbm.open (filename[, flag[, mode]])
     Open a dbm database and return a dbm object.  The _filename_
     argument is the name of the database file (without the `.dir' or
     `.pag' extensions; note that the BSD DB implementation of the
     interface will append the extension `.db' and only create one
     file).

     The optional _flag_ argument must be one of these values:

     Value         Meaning
     --------------------------------------------------------------
     `'r''         Open existing database for reading only
                   (default)
     `'w''         Open existing database for reading and writing
     `'c''         Open database for reading and writing,
                   creating it if it doesn't exist
     `'n''         Always create a new, empty database, open for
                   reading and writing

     The optional _mode_ argument is the Unix mode of the file, used
     only when the database has to be created.  It defaults to octal
     `0666' (and will be modified by the prevailing umask).

Module anydbm.
     Generic interface to `dbm'-style databases.

Module gdbm.
     Similar interface to the GNU GDBM library.

Module whichdb.
     Utility module used to determine the type of an existing database.

5.11.9 `gdbm' -- GNU's reinterpretation of dbm
----------------------------------------------

     Note: The gdbm module has been renamed to `dbm.gnu' in
     Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

This module is quite similar to the dbm module, but uses
`gdbm' instead to provide some additional functionality.  Please note
that the file formats created by `gdbm' and `dbm' are incompatible.

  The gdbm module provides an interface to the GNU DBM
library.  `gdbm' objects behave like mappings (dictionaries), except
that keys and values are always strings. Printing a `gdbm' object
doesn't print the keys and values, and the `items()' and `values()'
methods are not supported.

  The module defines the following constant and functions:

 -- Exception: gdbm.error
     Raised on `gdbm'-specific errors, such as I/O errors.
     KeyError is raised for general mapping errors like
     specifying an incorrect key.

 -- Function: gdbm.open (filename[, flag[, mode]])
     Open a `gdbm' database and return a `gdbm' object.  The _filename_
     argument is the name of the database file.

     The optional _flag_ argument can be:

     Value         Meaning
     --------------------------------------------------------------
     `'r''         Open existing database for reading only
                   (default)
     `'w''         Open existing database for reading and writing
     `'c''         Open database for reading and writing,
                   creating it if it doesn't exist
     `'n''         Always create a new, empty database, open for
                   reading and writing

     The following additional characters may be appended to the flag to
     control how the database is opened:

     Value         Meaning
     ---------------------------------------------------------------
     `'f''         Open the database in fast mode.  Writes to the
                   database will not be synchronized.
     `'s''         Synchronized mode. This will cause changes to
                   the database to be immediately written to the
                   file.
     `'u''         Do not lock database.

     Not all flags are valid for all versions of `gdbm'.  The module
     constant `open_flags' is a string of supported flag characters.
     The exception error is raised if an invalid flag is
     specified.

     The optional _mode_ argument is the Unix mode of the file, used
     only when the database has to be created.  It defaults to octal
     `0666'.

  In addition to the dictionary-like methods, `gdbm' objects have the
following methods:

 -- Function: gdbm.firstkey ()
     It's possible to loop over every key in the database using this
     method  and the nextkey() method.  The traversal is
     ordered by `gdbm''s internal hash values, and won't be sorted by
     the key values.  This method returns the starting key.

 -- Function: gdbm.nextkey (key)
     Returns the key that follows _key_ in the traversal.  The
     following code prints every key in the database `db', without
     having to create a list in memory that contains them all:

         k = db.firstkey()
         while k != None:
             print k
             k = db.nextkey(k)

 -- Function: gdbm.reorganize ()
     If you have carried out a lot of deletions and would like to
     shrink the space used by the `gdbm' file, this routine will
     reorganize the database.  `gdbm' will not shorten the length of a
     database file except by using this reorganization; otherwise,
     deleted file space will be kept and reused as new (key, value)
     pairs are added.

 -- Function: gdbm.sync ()
     When the database has been opened in fast mode, this method forces
     any unwritten data to be written to the disk.

Module anydbm.
     Generic interface to `dbm'-style databases.

Module whichdb.
     Utility module used to determine the type of an existing database.

5.11.10 `dbhash' -- DBM-style interface to the BSD database library
-------------------------------------------------------------------

Deprecated since version 2.6: The dbhash module has been
deprecated for removal in Python 3.0.

  The dbhash module provides a function to open databases
using the BSD `db' library.  This module mirrors the interface of the
other Python database modules that provide access to DBM-style
databases.  The bsddb module is required  to use
dbhash.

  This module provides an exception and a function:

 -- Exception: dbhash.error
     Exception raised on database errors other than KeyError.
     It is a synonym for `bsddb.error'.

 -- Function: dbhash.open (path[, flag[, mode]])
     Open a `db' database and return the database object.  The _path_
     argument is the name of the database file.

     The _flag_ argument can be:

     Value         Meaning
     --------------------------------------------------------------
     `'r''         Open existing database for reading only
                   (default)
     `'w''         Open existing database for reading and writing
     `'c''         Open database for reading and writing,
                   creating it if it doesn't exist
     `'n''         Always create a new, empty database, open for
                   reading and writing

     For platforms on which the BSD `db' library supports locking, an
     `'l'' can be appended to indicate that locking should be used.

     The optional _mode_ parameter is used to indicate the Unix
     permission bits that should be set if a new database must be
     created; this will be masked by the current umask value for the
     process.

Module anydbm.
     Generic interface to `dbm'-style databases.

Module bsddb.
     Lower-level interface to the BSD `db' library.

Module whichdb.
     Utility module used to determine the type of an existing database.

5.11.10.1 Database Objects
..........................

The database objects returned by open() provide the methods
common to all the DBM-style databases and mapping objects.  The
following methods are available in addition to the standard methods.

 -- Method: dbhash.first ()
     It's possible to loop over every key/value pair in the database
     using this method and the `next()' method.  The traversal is
     ordered by the databases internal hash values, and won't be sorted
     by the key values.  This method returns the starting key.

 -- Method: dbhash.last ()
     Return the last key/value pair in a database traversal.  This may
     be used to begin a reverse-order traversal; see previous().

 -- Method: dbhash.next ()
     Returns the key next key/value pair in a database traversal.  The
     following code prints every key in the database `db', without
     having to create a list in memory that contains them all:

         print db.first()
         for i in xrange(1, len(db)):
             print db.next()

 -- Method: dbhash.previous ()
     Returns the previous key/value pair in a forward-traversal of the
     database. In conjunction with last(), this may be used
     to implement a reverse-order traversal.

 -- Method: dbhash.sync ()
     This method forces any unwritten data to be written to the disk.

5.11.11 `bsddb' -- Interface to Berkeley DB library
---------------------------------------------------

Deprecated since version 2.6: The bsddb module has been
deprecated for removal in Python 3.0.

  The bsddb module provides an interface to the Berkeley DB
library.  Users can create hash, btree or record based library files
using the appropriate open call. Bsddb objects behave generally like
dictionaries.  Keys and values must be strings, however, so to use
other objects as keys or to store other kinds of objects the user must
serialize them somehow, typically using marshal.dumps() or
pickle.dumps().

  The bsddb module requires a Berkeley DB library version
from 4.0 thru 4.7.

<http://www.jcea.es/programacion/pybsddb.htm>
     The website with documentation for the `bsddb.db' Python Berkeley
     DB interface that closely mirrors the object oriented interface
     provided in Berkeley DB 4.x itself.

<http://www.oracle.com/database/berkeley-db/>
     The Berkeley DB library.

  A more modern DB, DBEnv and DBSequence object interface is available
in the `bsddb.db' module which closely matches the Berkeley DB C API
documented at the above URLs.  Additional features provided by the
`bsddb.db' API include fine tuning, transactions, logging, and
multiprocess concurrent database access.

  The following is a description of the legacy bsddb.
interface compatible with the old Python bsddb module.  Starting in
Python 2.5 this interface should be safe for multithreaded access.  The
`bsddb.db' API is recommended for threading users as it provides better
control.

  The bsddb module defines the following functions that
create objects that access the appropriate type of Berkeley DB file.
The first two arguments of each function are the same.  For ease of
portability, only the first two arguments should be used in most
instances.

 -- Function: bsddb.hashopen (filename[, flag[, mode[, pgsize[,
          ffactor[, nelem[, cachesize[, lorder[, hflags]]]]]]]])
     Open the hash format file named _filename_.  Files never intended
     to be preserved on disk may be created by passing `None' as the
     _filename_.  The optional _flag_ identifies the mode used to open
     the file.  It may be `'r'' (read only), `'w'' (read-write) , `'c''
     (read-write - create if necessary; the default) or `'n''
     (read-write - truncate to zero length).  The other arguments are
     rarely used and are just passed to the low-level `dbopen()'
     function.  Consult the Berkeley DB documentation for their use and
     interpretation.

 -- Function: bsddb.btopen (filename[, flag[, mode[, btflags[,
          cachesize[, maxkeypage[, minkeypage[, pgsize[, lorder]]]]]]]])
     Open the btree format file named _filename_.  Files never intended
     to be preserved on disk may be created by passing `None' as the
     _filename_.  The optional _flag_ identifies the mode used to open
     the file.  It may be `'r'' (read only), `'w'' (read-write), `'c''
     (read-write - create if necessary; the default) or `'n''
     (read-write - truncate to zero length).  The other arguments are
     rarely used and are just passed to the low-level dbopen function.
     Consult the Berkeley DB documentation for their use and
     interpretation.

 -- Function: bsddb.rnopen (filename[, flag[, mode[, rnflags[,
          cachesize[, pgsize[, lorder[, rlen[, delim[, source[,
          pad]]]]]]]]]])
     Open a DB record format file named _filename_.  Files never
     intended  to be preserved on disk may be created by passing `None'
     as the  _filename_.  The optional _flag_ identifies the mode used
     to open the file.  It may be `'r'' (read only), `'w''
     (read-write), `'c'' (read-write - create if necessary; the
     default) or `'n'' (read-write - truncate to zero length).  The
     other arguments are rarely used and are just passed to the
     low-level dbopen function.  Consult the Berkeley DB documentation
     for their use and interpretation.

     Note: Beginning in 2.3 some Unix versions of Python may have a
     `bsddb185' module.  This is present _only_ to allow backwards
     compatibility with systems which ship with the old Berkeley DB
     1.85 database library.  The `bsddb185' module should never be used
     directly in new code. The module has been removed in Python 3.0.
     If you find you still need it look in PyPI.

Module dbhash.
     DBM-style interface to the bsddb.

5.11.11.1 Hash, BTree and Record Objects
........................................

Once instantiated, hash, btree and record objects support the same
methods as dictionaries.  In addition, they support the methods listed
below.

  Changed in version 2.3.1: Added dictionary methods.

 -- Method: bsddbobject.close ()
     Close the underlying file.  The object can no longer be accessed.
     Since there is no open open() method for these
     objects, to open the file again a new bsddb module open
     function must be called.

 -- Method: bsddbobject.keys ()
     Return the list of keys contained in the DB file.  The order of
     the list is unspecified and should not be relied on.  In
     particular, the order of the list returned is different for
     different file formats.

 -- Method: bsddbobject.has_key (key)
     Return `1' if the DB file contains the argument as a key.

 -- Method: bsddbobject.set_location (key)
     Set the cursor to the item indicated by _key_ and return a tuple
     containing the key and its value.  For binary tree databases
     (opened using btopen()), if _key_ does not actually
     exist in the database, the cursor will point to the next item in
     sorted order and return that key and value.  For other databases,
     see KeyError will be raised if _key_ is not found in the
     database.

 -- Method: bsddbobject.first ()
     Set the cursor to the first item in the DB file and return it.
     The order of keys in the file is unspecified, except in the case
     of B-Tree databases. This method raises `bsddb.error' if the
     database is empty.

 -- Method: bsddbobject.next ()
     Set the cursor to the next item in the DB file and return it.  The
     order of keys in the file is unspecified, except in the case of
     B-Tree databases.

 -- Method: bsddbobject.previous ()
     Set the cursor to the previous item in the DB file and return it.
     The order of keys in the file is unspecified, except in the case
     of B-Tree databases.  This is not supported on hashtable databases
     (those opened with hashopen()).

 -- Method: bsddbobject.last ()
     Set the cursor to the last item in the DB file and return it.  The
     order of keys in the file is unspecified.  This is not supported
     on hashtable databases (those opened with hashopen()).
     This method raises `bsddb.error' if the database is empty.

 -- Method: bsddbobject.sync ()
     Synchronize the database on disk.

  Example:

    >>> import bsddb
    >>> db = bsddb.btopen('/tmp/spam.db', 'c')
    >>> for i in range(10): db['%d'%i] = '%d'% (i*i)
    ...
    >>> db['3']
    '9'
    >>> db.keys()
    ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
    >>> db.first()
    ('0', '0')
    >>> db.next()
    ('1', '1')
    >>> db.last()
    ('9', '81')
    >>> db.set_location('2')
    ('2', '4')
    >>> db.previous()
    ('1', '1')
    >>> for k, v in db.iteritems():
    ...     print k, v
    0 0
    1 1
    2 4
    3 9
    4 16
    5 25
    6 36
    7 49
    8 64
    9 81
    >>> '8' in db
    True
    >>> db.sync()
    0

5.11.12 `dumbdbm' -- Portable DBM implementation
------------------------------------------------

     Note: The dumbdbm module has been renamed to `dbm.dumb'
     in Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

     Note: The dumbdbm module is intended as a last resort
     fallback for the anydbm module when no more robust
     module is available. The dumbdbm.  module is not written
     for speed and is not nearly as heavily used as the other database
     modules.

The dumbdbm module provides a persistent dictionary-like
interface which is written entirely in Python.  Unlike other modules
such as gdbm and bsddb, no external library is
required.  As with other persistent mappings, the keys and values must
always be strings.

  The module defines the following:

 -- Exception: dumbdbm.error
     Raised on dumbdbm-specific errors, such as I/O errors.
     KeyError is raised for general mapping errors like
     specifying an incorrect key.

 -- Function: dumbdbm.open (filename[, flag[, mode]])
     Open a dumbdbm database and return a dumbdbm object.  The
     _filename_ argument is the basename of the database file (without
     any specific extensions).  When a dumbdbm database is created,
     files with `.dat' and `.dir' extensions are created.

     The optional _flag_ argument is currently ignored; the database is
     always opened for update, and will be created if it does not exist.

     The optional _mode_ argument is the Unix mode of the file, used
     only when the database has to be created.  It defaults to octal
     `0666' (and will be modified by the prevailing umask).

     Changed in version 2.2: The _mode_ argument was ignored in earlier
     versions.

Module anydbm.
     Generic interface to `dbm'-style databases.

Module dbm.
     Similar interface to the DBM/NDBM library.

Module gdbm.
     Similar interface to the GNU GDBM library.

Module shelve.
     Persistence module which stores non-string data.

Module whichdb.
     Utility module used to determine the type of an existing database.

5.11.12.1 Dumbdbm Objects
.........................

In addition to the methods provided by the UserDict.DictMixin
class, see dumbdbm objects provide the following methods.

 -- Method: dumbdbm.sync ()
     Synchronize the on-disk directory and data files.  This method is
     called by the sync() method of `Shelve' objects.

5.11.13 `sqlite3' -- DB-API 2.0 interface for SQLite databases
--------------------------------------------------------------

New in version 2.5.

  SQLite is a C library that provides a lightweight disk-based database
that doesn't require a separate server process and allows accessing the
database using a nonstandard variant of the SQL query language. Some
applications can use SQLite for internal data storage.  It's also
possible to prototype an application using SQLite and then port the
code to a larger database such as PostgreSQL or Oracle.

  sqlite3 was written by Gerhard Häring and provides a SQL interface
compliant with the DB-API 2.0 specification described by PEP 249
(http://www.python.org/dev/peps/pep-0249).

  To use the module, you must first create a Connection.
object that represents the database.  Here the data will be stored in
the `/tmp/example' file:

    conn = sqlite3.connect('/tmp/example')

You can also supply the special name `:memory:' to create a database in
RAM.

  Once you have a Connection, you can create a Cursor.
object and call its execute() method to perform SQL
commands:

    c = conn.cursor()

    # Create table
    c.execute('''create table stocks
    (date text, trans text, symbol text,
     qty real, price real)''')

    # Insert a row of data
    c.execute("""insert into stocks
              values ('2006-01-05','BUY','RHAT',100,35.14)""")

    # Save (commit) the changes
    conn.commit()

    # We can also close the cursor if we are done with it
    c.close()

Usually your SQL operations will need to use values from Python
variables.  You shouldn't assemble your query using Python's string
operations because doing so is insecure; it makes your program
vulnerable to an SQL injection attack.

  Instead, use the DB-API's parameter substitution.  Put `?' as a
placeholder wherever you want to use a value, and then provide a tuple
of values as the second argument to the cursor's execute().
method.  (Other database modules may use a different placeholder, such
as `%s' or `:1'.) For example:

    # Never do this -- insecure!
    symbol = 'IBM'
    c.execute("... where symbol = '%s'" % symbol)

    # Do this instead
    t = (symbol,)
    c.execute('select * from stocks where symbol=?', t)

    # Larger example
    for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
              ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
              ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
             ]:
        c.execute('insert into stocks values (?,?,?,?,?)', t)

To retrieve data after executing a SELECT statement, you can either
treat the cursor as an iterator, call the cursor's
fetchone() method to retrieve a single matching row, or call
fetchall() to get a list of the matching rows.

  This example uses the iterator form:

    >>> c = conn.cursor()
    >>> c.execute('select * from stocks order by price')
    >>> for row in c:
    ...    print row
    ...
    (u'2006-01-05', u'BUY', u'RHAT', 100, 35.14)
    (u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
    (u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
    (u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)
    >>>

<http://code.google.com/p/pysqlite/>
     The pysqlite web page - sqlite3 is developed externally under the
     name "pysqlite".

<http://www.sqlite.org>
     The SQLite web page; the documentation describes the syntax and the
     available data types for the supported SQL dialect.

PEP 249 (http://www.python.org/dev/peps/pep-0249) - Database API Specification 2.0
     PEP written by Marc-André Lemburg.

5.11.13.1 Module functions and constants
........................................

 -- Data: sqlite3.PARSE_DECLTYPES
     This constant is meant to be used with the _detect_types_
     parameter of the connect() function.

     Setting it makes the sqlite3 module parse the declared
     type for each column it returns.  It will parse out the first word
     of the declared type, i. e.  for "integer primary key", it will
     parse out "integer", or for "number(10)" it will parse out
     "number". Then for that column, it will look into the converters
     dictionary and use the converter function registered for that type
     there.

 -- Data: sqlite3.PARSE_COLNAMES
     This constant is meant to be used with the _detect_types_
     parameter of the connect() function.

     Setting this makes the SQLite interface parse the column name for
     each column it returns.  It will look for a string formed [mytype]
     in there, and then decide that 'mytype' is the type of the column.
     It will try to find an entry of 'mytype' in the converters
     dictionary and then use the converter function found there to
     return the value. The column name found in
     Cursor.description.  is only the first word of the column
     name, i.  e. if you use something like `'as "x [datetime]"'' in
     your SQL, then we will parse out everything until the first blank
     for the column name: the column name would simply be "x".

 -- Function: sqlite3.connect (database[, timeout, detect_types,
          isolation_level, check_same_thread, factory,
          cached_statements])
     Opens a connection to the SQLite database file _database_. You can
     use `":memory:"' to open a database connection to a database that
     resides in RAM instead of on disk.

     When a database is accessed by multiple connections, and one of
     the processes modifies the database, the SQLite database is locked
     until that transaction is committed. The _timeout_ parameter
     specifies how long the connection should wait for the lock to go
     away until raising an exception. The default for the timeout
     parameter is 5.0 (five seconds).

     For the _isolation_level_ parameter, please see the
     Connection.isolation_level property of Connection
     objects.

     SQLite natively supports only the types TEXT, INTEGER, FLOAT, BLOB
     and NULL. If you want to use other types you must add support for
     them yourself. The _detect_types_ parameter and the using custom
     *converters* registered with the module-level
     register_converter() function allow you to easily do that.

     _detect_types_ defaults to 0 (i. e. off, no type detection), you
     can set it to any combination of PARSE_DECLTYPES and
     PARSE_COLNAMES to turn type detection on.

     By default, the sqlite3 module uses its
     Connection class for the connect call.  You can, however,
     subclass the Connection class and make
     connect() use your class instead by providing your class for
     the _factory_ parameter.

     Consult the section SQLite and Python types of this
     manual for details.

     The sqlite3 module internally uses a statement cache
     to avoid SQL parsing overhead. If you want to explicitly set the
     number of statements that are cached for the connection, you can
     set the _cached_statements_ parameter. The currently implemented
     default is to cache 100 statements.

 -- Function: sqlite3.register_converter (typename, callable)
     Registers a callable to convert a bytestring from the database
     into a custom Python type. The callable will be invoked for all
     database values that are of the type _typename_. Confer the
     parameter _detect_types_ of the connect().  function
     for how the type detection works. Note that the case of _typename_
     and the name of the type in your query must match!

 -- Function: sqlite3.register_adapter (type, callable)
     Registers a callable to convert the custom Python type _type_ into
     one of SQLite's supported types. The callable _callable_ accepts
     as single parameter the Python value, and must return a value of
     the following types: int, long, float, str (UTF-8 encoded),
     unicode or buffer.

 -- Function: sqlite3.complete_statement (sql)
     Returns True if the string _sql_ contains one or more
     complete SQL statements terminated by semicolons. It does not
     verify that the SQL is syntactically correct, only that there are
     no unclosed string literals and the statement is terminated by a
     semicolon.

     This can be used to build a shell for SQLite, as in the following
     example:

         # A minimal SQLite shell for experiments

         import sqlite3

         con = sqlite3.connect(":memory:")
         con.isolation_level = None
         cur = con.cursor()

         buffer = ""

         print "Enter your SQL commands to execute in sqlite3."
         print "Enter a blank line to exit."

         while True:
             line = raw_input()
             if line == "":
                 break
             buffer += line
             if sqlite3.complete_statement(buffer):
                 try:
                     buffer = buffer.strip()
                     cur.execute(buffer)

                     if buffer.lstrip().upper().startswith("SELECT"):
                         print cur.fetchall()
                 except sqlite3.Error, e:
                     print "An error occurred:", e.args[0]
                 buffer = ""

         con.close()

 -- Function: sqlite3.enable_callback_tracebacks (flag)
     By default you will not get any tracebacks in user-defined
     functions, aggregates, converters, authorizer callbacks etc. If
     you want to debug them, you can call this function with _flag_ as
     True. Afterwards, you will get tracebacks from callbacks on
     `sys.stderr'. Use False to disable the feature again.

5.11.13.2 Connection Objects
............................

 -- Class: sqlite3.Connection
     A SQLite database connection has the following attributes and
     methods:

 -- Attribute: Connection.isolation_level
     Get or set the current isolation level. None for
     autocommit mode or one of "DEFERRED", "IMMEDIATE" or "EXCLUSIVE".
     See section Controlling Transactions for a more
     detailed explanation.

 -- Method: Connection.cursor ([cursorClass])
     The cursor method accepts a single optional parameter
     _cursorClass_. If supplied, this must be a custom cursor class
     that extends sqlite3.Cursor.

 -- Method: Connection.commit ()
     This method commits the current transaction. If you don't call
     this method, anything you did since the last call to `commit()' is
     not visible from from other database connections. If you wonder
     why you don't see the data you've written to the database, please
     check you didn't forget to call this method.

 -- Method: Connection.rollback ()
     This method rolls back any changes to the database since the last
     call to commit().

 -- Method: Connection.close ()
     This closes the database connection. Note that this does not
     automatically call commit(). If you just close your
     database connection without calling commit() first,
     your changes will be lost!

 -- Method: Connection.execute (sql[, parameters])
     This is a nonstandard shortcut that creates an intermediate cursor
     object by calling the cursor method, then calls the cursor's
     execute method with the parameters given.

 -- Method: Connection.executemany (sql[, parameters])
     This is a nonstandard shortcut that creates an intermediate cursor
     object by calling the cursor method, then calls the cursor's
     executemany method with the parameters given.

 -- Method: Connection.executescript (sql_script)
     This is a nonstandard shortcut that creates an intermediate cursor
     object by calling the cursor method, then calls the cursor's
     executescript method with the parameters given.

 -- Method: Connection.create_function (name, num_params, func)
     Creates a user-defined function that you can later use from within
     SQL statements under the function name _name_. _num_params_ is the
     number of parameters the function accepts, and _func_ is a Python
     callable that is called as the SQL function.

     The function can return any of the types supported by SQLite:
     unicode, str, int, long, float, buffer and None.

     Example:

         import sqlite3
         import md5

         def md5sum(t):
             return md5.md5(t).hexdigest()

         con = sqlite3.connect(":memory:")
         con.create_function("md5", 1, md5sum)
         cur = con.cursor()
         cur.execute("select md5(?)", ("foo",))
         print cur.fetchone()[0]

 -- Method: Connection.create_aggregate (name, num_params,
          aggregate_class)
     Creates a user-defined aggregate function.

     The aggregate class must implement a `step' method, which accepts
     the number of parameters _num_params_, and a `finalize' method
     which will return the final result of the aggregate.

     The `finalize' method can return any of the types supported by
     SQLite: unicode, str, int, long, float, buffer and None.

     Example:

         import sqlite3

         class MySum:
             def __init__(self):
                 self.count = 0

             def step(self, value):
                 self.count += value

             def finalize(self):
                 return self.count

         con = sqlite3.connect(":memory:")
         con.create_aggregate("mysum", 1, MySum)
         cur = con.cursor()
         cur.execute("create table test(i)")
         cur.execute("insert into test(i) values (1)")
         cur.execute("insert into test(i) values (2)")
         cur.execute("select mysum(i) from test")
         print cur.fetchone()[0]

 -- Method: Connection.create_collation (name, callable)
     Creates a collation with the specified _name_ and _callable_. The
     callable will be passed two string arguments. It should return -1
     if the first is ordered lower than the second, 0 if they are
     ordered equal and 1 if the first is ordered higher than the
     second.  Note that this controls sorting (ORDER BY in SQL) so your
     comparisons don't affect other SQL operations.

     Note that the callable will get its parameters as Python
     bytestrings, which will normally be encoded in UTF-8.

     The following example shows a custom collation that sorts "the
     wrong way":

         import sqlite3

         def collate_reverse(string1, string2):
             return -cmp(string1, string2)

         con = sqlite3.connect(":memory:")
         con.create_collation("reverse", collate_reverse)

         cur = con.cursor()
         cur.execute("create table test(x)")
         cur.executemany("insert into test(x) values (?)", [("a",), ("b",)])
         cur.execute("select x from test order by x collate reverse")
         for row in cur:
             print row
         con.close()

     To remove a collation, call `create_collation' with None as
     callable:

         con.create_collation("reverse", None)

 -- Method: Connection.interrupt ()
     You can call this method from a different thread to abort any
     queries that might be executing on the connection. The query will
     then abort and the caller will get an exception.

 -- Method: Connection.set_authorizer (authorizer_callback)
     This routine registers a callback. The callback is invoked for
     each attempt to access a column of a table in the database. The
     callback should return `SQLITE_OK' if access is allowed,
     `SQLITE_DENY' if the entire SQL statement should be aborted with
     an error and `SQLITE_IGNORE' if the column should be treated as a
     NULL value. These constants are available in the sqlite3
     module.

     The first argument to the callback signifies what kind of
     operation is to be authorized. The second and third argument will
     be arguments or None.  depending on the first argument.
     The 4th argument is the name of the database ("main", "temp",
     etc.) if applicable. The 5th argument is the name of the
     inner-most trigger or view that is responsible for the access
     attempt or None if this access attempt is directly
     from input SQL code.

     Please consult the SQLite documentation about the possible values
     for the first argument and the meaning of the second and third
     argument depending on the first one. All necessary constants are
     available in the sqlite3 module.

 -- Method: Connection.set_progress_handler (handler, n)
     New in version 2.6.

     This routine registers a callback. The callback is invoked for
     every _n_ instructions of the SQLite virtual machine. This is
     useful if you want to get called from SQLite during long-running
     operations, for example to update a GUI.

     If you want to clear any previously installed progress handler,
     call the method with None for _handler_.

 -- Method: Connection.enable_load_extension (enabled)
     New in version 2.7.

     This routine allows/disallows the SQLite engine to load SQLite
     extensions from shared libraries.  SQLite extensions can define
     new functions, aggregates or whole new virtual table
     implementations.  One well-known extension is the fulltext-search
     extension distributed with SQLite.

         import sqlite3

         con = sqlite3.connect(":memory:")

         # enable extension loading
         con.enable_load_extension(True)

         # Load the fulltext search extension
         con.execute("select load_extension('./fts3.so')")

         # alternatively you can load the extension using an API call:
         # con.load_extension("./fts3.so")

         # disable extension laoding again
         con.enable_load_extension(False)

         # example from SQLite wiki
         con.execute("create virtual table recipe using fts3(name, ingredients)")
         con.executescript("""
             insert into recipe (name, ingredients) values ('broccoli stew', 'broccoli peppers cheese tomatoes');
             insert into recipe (name, ingredients) values ('pumpkin stew', 'pumpkin onions garlic celery');
             insert into recipe (name, ingredients) values ('broccoli pie', 'broccoli cheese onions flour');
             insert into recipe (name, ingredients) values ('pumpkin pie', 'pumpkin sugar flour butter');
             """)
         for row in con.execute("select rowid, name, ingredients from recipe where name match 'pie'"):
             print row

 -- Method: Connection.load_extension (path)
     New in version 2.7.

     This routine loads a SQLite extension from a shared library.  You
     have to enable extension loading with
     enable_load_extension() before you can use this routine.

 -- Attribute: Connection.row_factory
     You can change this attribute to a callable that accepts the
     cursor and the original row as a tuple and will return the real
     result row.  This way, you can implement more advanced ways of
     returning results, such  as returning an object that can also
     access columns by name.

     Example:

         import sqlite3

         def dict_factory(cursor, row):
             d = {}
             for idx, col in enumerate(cursor.description):
                 d[col[0]] = row[idx]
             return d

         con = sqlite3.connect(":memory:")
         con.row_factory = dict_factory
         cur = con.cursor()
         cur.execute("select 1 as a")
         print cur.fetchone()["a"]

     If returning a tuple doesn't suffice and you want name-based
     access to columns, you should consider setting row_factory
     to the highly-optimized sqlite3.Row type.
     Row provides both index-based and case-insensitive
     name-based access to columns with almost no memory overhead. It
     will probably be better than your own custom dictionary-based
     approach or even a db_row based solution.

 -- Attribute: Connection.text_factory
     Using this attribute you can control what objects are returned for
     the `TEXT' data type. By default, this attribute is set to
     unicode and the sqlite3 module will return
     Unicode objects for `TEXT'. If you want to return bytestrings
     instead, you can set it to str.

     For efficiency reasons, there's also a way to return Unicode
     objects only for non-ASCII data, and bytestrings otherwise. To
     activate it, set this attribute to `sqlite3.OptimizedUnicode'.

     You can also set it to any other callable that accepts a single
     bytestring parameter and returns the resulting object.

     See the following example code for illustration:

         import sqlite3

         con = sqlite3.connect(":memory:")
         cur = con.cursor()

         # Create the table
         con.execute("create table person(lastname, firstname)")

         AUSTRIA = u"\xd6sterreich"

         # by default, rows are returned as Unicode
         cur.execute("select ?", (AUSTRIA,))
         row = cur.fetchone()
         assert row[0] == AUSTRIA

         # but we can make sqlite3 always return bytestrings ...
         con.text_factory = str
         cur.execute("select ?", (AUSTRIA,))
         row = cur.fetchone()
         assert type(row[0]) == str
         # the bytestrings will be encoded in UTF-8, unless you stored garbage in the
         # database ...
         assert row[0] == AUSTRIA.encode("utf-8")

         # we can also implement a custom text_factory ...
         # here we implement one that will ignore Unicode characters that cannot be
         # decoded from UTF-8
         con.text_factory = lambda x: unicode(x, "utf-8", "ignore")
         cur.execute("select ?", ("this is latin1 and would normally create errors" +
                                  u"\xe4\xf6\xfc".encode("latin1"),))
         row = cur.fetchone()
         assert type(row[0]) == unicode

         # sqlite3 offers a built-in optimized text_factory that will return bytestring
         # objects, if the data is in ASCII only, and otherwise return unicode objects
         con.text_factory = sqlite3.OptimizedUnicode
         cur.execute("select ?", (AUSTRIA,))
         row = cur.fetchone()
         assert type(row[0]) == unicode

         cur.execute("select ?", ("Germany",))
         row = cur.fetchone()
         assert type(row[0]) == str

 -- Attribute: Connection.total_changes
     Returns the total number of database rows that have been modified,
     inserted, or deleted since the database connection was opened.

 -- Attribute: Connection.iterdump
     Returns an iterator to dump the database in an SQL text format.
     Useful when saving an in-memory database for later restoration.
     This function provides the same capabilities as the `.dump'
     command in the *sqlite3* shell.

     New in version 2.6.

     Example:

         # Convert file existing_db.db to SQL dump file dump.sql
         import sqlite3, os

         con = sqlite3.connect('existing_db.db')
         with open('dump.sql', 'w') as f:
             for line in con.iterdump():
                 f.write('%s\n' % line)

5.11.13.3 Cursor Objects
........................

 -- Class: sqlite3.Cursor
     A Cursor instance has the following attributes and
     methods.

 -- Method: Cursor.execute (sql[, parameters])
     Executes an SQL statement. The SQL statement may be parametrized
     (i. e.  placeholders instead of SQL literals). The sqlite3
     module supports two kinds of placeholders: question marks
     (qmark style) and named placeholders (named style).

     This example shows how to use parameters with qmark style:

         import sqlite3

         con = sqlite3.connect("mydb")

         cur = con.cursor()

         who = "Yeltsin"
         age = 72

         cur.execute("select name_last, age from people where name_last=? and age=?", (who, age))
         print cur.fetchone()

     This example shows how to use the named style:

         import sqlite3

         con = sqlite3.connect("mydb")

         cur = con.cursor()

         who = "Yeltsin"
         age = 72

         cur.execute("select name_last, age from people where name_last=:who and age=:age",
             {"who": who, "age": age})
         print cur.fetchone()

     execute() will only execute a single SQL statement. If
     you try to execute more than one statement with it, it will raise
     a Warning. Use executescript() if you want to execute
     multiple SQL statements with one call.

 -- Method: Cursor.executemany (sql, seq_of_parameters)
     Executes an SQL command against all parameter sequences or
     mappings found in the sequence _sql_.  The sqlite3.
     module also allows using an iterator yielding
     parameters instead of a sequence.

         import sqlite3

         class IterChars:
             def __init__(self):
                 self.count = ord('a')

             def __iter__(self):
                 return self

             def next(self):
                 if self.count > ord('z'):
                     raise StopIteration
                 self.count += 1
                 return (chr(self.count - 1),) # this is a 1-tuple

         con = sqlite3.connect(":memory:")
         cur = con.cursor()
         cur.execute("create table characters(c)")

         theIter = IterChars()
         cur.executemany("insert into characters(c) values (?)", theIter)

         cur.execute("select c from characters")
         print cur.fetchall()

     Here's a shorter example using a generator.:

         import sqlite3

         def char_generator():
             import string
             for c in string.letters[:26]:
                 yield (c,)

         con = sqlite3.connect(":memory:")
         cur = con.cursor()
         cur.execute("create table characters(c)")

         cur.executemany("insert into characters(c) values (?)", char_generator())

         cur.execute("select c from characters")
         print cur.fetchall()

 -- Method: Cursor.executescript (sql_script)
     This is a nonstandard convenience method for executing multiple
     SQL statements at once. It issues a `COMMIT' statement first, then
     executes the SQL script it gets as a parameter.

     _sql_script_ can be a bytestring or a Unicode string.

     Example:

         import sqlite3

         con = sqlite3.connect(":memory:")
         cur = con.cursor()
         cur.executescript("""
             create table person(
                 firstname,
                 lastname,
                 age
             );

             create table book(
                 title,
                 author,
                 published
             );

             insert into book(title, author, published)
             values (
                 'Dirk Gently''s Holistic Detective Agency',
                 'Douglas Adams',
                 1987
             );
             """)

 -- Method: Cursor.fetchone ()
     Fetches the next row of a query result set, returning a single
     sequence, or None when no more data is available.

 -- Method: Cursor.fetchmany ([size=cursor.arraysize])
     Fetches the next set of rows of a query result, returning a list.
     An empty list is returned when no more rows are available.

     The number of rows to fetch per call is specified by the _size_
     parameter.  If it is not given, the cursor's arraysize determines
     the number of rows to be fetched. The method should try to fetch
     as many rows as indicated by the size parameter. If this is not
     possible due to the specified number of rows not being available,
     fewer rows may be returned.

     Note there are performance considerations involved with the _size_
     parameter.  For optimal performance, it is usually best to use the
     arraysize attribute.  If the _size_ parameter is used, then it is
     best for it to retain the same value from one fetchmany()
     call to the next.

 -- Method: Cursor.fetchall ()
     Fetches all (remaining) rows of a query result, returning a list.
     Note that the cursor's arraysize attribute can affect the
     performance of this operation.  An empty list is returned when no
     rows are available.

 -- Attribute: Cursor.rowcount
     Although the Cursor class of the sqlite3.
     module implements this attribute, the database engine's own
     support for the determination of "rows affected"/"rows selected"
     is quirky.

     For `DELETE' statements, SQLite reports rowcount as 0
     if you make a `DELETE FROM table' without any condition.

     For executemany() statements, the number of
     modifications are summed up into rowcount.

     As required by the Python DB API Spec, the rowcount.
     attribute "is -1 in case no `executeXX()' has been performed on
     the cursor or the rowcount of the last operation is not
     determinable by the interface".

     This includes `SELECT' statements because we cannot determine the
     number of rows a query produced until all rows were fetched.

 -- Attribute: Cursor.lastrowid
     This read-only attribute provides the rowid of the last modified
     row. It is only set if you issued a `INSERT' statement using the
     execute().  method. For operations other than `INSERT'
     or when executemany() is called, see lastrowid.
     is set to None.

 -- Attribute: Cursor.description
     This read-only attribute provides the column names of the last
     query. To remain compatible with the Python DB API, it returns a
     7-tuple for each column where the last six items of each tuple are
     None.

     It is set for `SELECT' statements without any matching rows as
     well.

5.11.13.4 Row Objects
.....................

 -- Class: sqlite3.Row
     A Row instance serves as a highly optimized
     row_factory for Connection objects.  It tries to
     mimic a tuple in most of its features.

     It supports mapping access by column name and index, iteration,
     representation, equality testing and len().

     If two Row objects have exactly the same columns and
     their members are equal, they compare equal.

     Changed in version 2.6: Added iteration and equality (hashability).

      -- Method: keys ()
          This method returns a tuple of column names. Immediately
          after a query, it is the first member of each tuple in
          Cursor.description.

          New in version 2.6.

Let's assume we initialize a table as in the example given above:

    conn = sqlite3.connect(":memory:")
    c = conn.cursor()
    c.execute('''create table stocks
    (date text, trans text, symbol text,
     qty real, price real)''')
    c.execute("""insert into stocks
              values ('2006-01-05','BUY','RHAT',100,35.14)""")
    conn.commit()
    c.close()

Now we plug Row in:

    >>> conn.row_factory = sqlite3.Row
    >>> c = conn.cursor()
    >>> c.execute('select * from stocks')
    <sqlite3.Cursor object at 0x7f4e7dd8fa80>
    >>> r = c.fetchone()
    >>> type(r)
    <type 'sqlite3.Row'>
    >>> r
    (u'2006-01-05', u'BUY', u'RHAT', 100.0, 35.14)
    >>> len(r)
    5
    >>> r[2]
    u'RHAT'
    >>> r.keys()
    ['date', 'trans', 'symbol', 'qty', 'price']
    >>> r['qty']
    100.0
    >>> for member in r: print member
    ...
    2006-01-05
    BUY
    RHAT
    100.0
    35.14

5.11.13.5 SQLite and Python types
.................................

5.11.13.6 Introduction
......................

SQLite natively supports the following types: `NULL', `INTEGER',
`REAL', `TEXT', `BLOB'.

  The following Python types can thus be sent to SQLite without any
problem:

Python type                       SQLite type
----------------------------------------------------
see None.                  `NULL'
int.                   `INTEGER'
long.                  `INTEGER'
float.                 `REAL'
str. (UTF8-encoded)    `TEXT'
unicode.               `TEXT'
buffer.                `BLOB'

  This is how SQLite types are converted to Python types by default:

SQLite type       Python type
---------------------------------------------------------------------
`NULL'            None.
`INTEGER'         int or long, depending on
                  size
`REAL'            float.
`TEXT'            depends on text_factory,
                  unicode by default
`BLOB'            buffer.

  The type system of the sqlite3 module is extensible in
two ways: you can store additional Python types in a SQLite database
via object adaptation, and you can let the sqlite3 module
convert SQLite types to different Python types via converters.

5.11.13.7 Using adapters to store additional Python types in SQLite databases
.............................................................................

As described before, SQLite supports only a limited set of types
natively. To use other Python types with SQLite, you must *adapt* them
to one of the sqlite3 module's supported types for SQLite: one of
NoneType, int, long, float, str, unicode, buffer.

  The sqlite3 module uses Python object adaptation, as
described in PEP 246 (http://www.python.org/dev/peps/pep-0246) for
this.  The protocol to use is `PrepareProtocol'.

  There are two ways to enable the sqlite3 module to adapt
a custom Python type to one of the supported ones.

5.11.13.8 Letting your object adapt itself
..........................................

This is a good approach if you write the class yourself. Let's suppose
you have a class like this:

    class Point(object):
        def __init__(self, x, y):
            self.x, self.y = x, y

Now you want to store the point in a single SQLite column.  First
you'll have to choose one of the supported types first to be used for
representing the point.  Let's just use str and separate the
coordinates using a semicolon. Then you need to give your class a
method `__conform__(self, protocol)' which must return the converted
value. The parameter _protocol_ will be `PrepareProtocol'.

    import sqlite3

    class Point(object):
        def __init__(self, x, y):
            self.x, self.y = x, y

        def __conform__(self, protocol):
            if protocol is sqlite3.PrepareProtocol:
                return "%f;%f" % (self.x, self.y)

    con = sqlite3.connect(":memory:")
    cur = con.cursor()

    p = Point(4.0, -3.2)
    cur.execute("select ?", (p,))
    print cur.fetchone()[0]

5.11.13.9 Registering an adapter callable
.........................................

The other possibility is to create a function that converts the type to
the string representation and register the function with
register_adapter().

     Note: The type/class to adapt must be a new-style class,
     i. e. it must have object as one of its bases.

    import sqlite3

    class Point(object):
        def __init__(self, x, y):
            self.x, self.y = x, y

    def adapt_point(point):
        return "%f;%f" % (point.x, point.y)

    sqlite3.register_adapter(Point, adapt_point)

    con = sqlite3.connect(":memory:")
    cur = con.cursor()

    p = Point(4.0, -3.2)
    cur.execute("select ?", (p,))
    print cur.fetchone()[0]

The sqlite3 module has two default adapters for Python's
built-in datetime.date and datetime.datetime.
types.  Now let's suppose we want to store datetime.datetime
objects not in ISO representation, but as a Unix timestamp.

    import sqlite3
    import datetime, time

    def adapt_datetime(ts):
        return time.mktime(ts.timetuple())

    sqlite3.register_adapter(datetime.datetime, adapt_datetime)

    con = sqlite3.connect(":memory:")
    cur = con.cursor()

    now = datetime.datetime.now()
    cur.execute("select ?", (now,))
    print cur.fetchone()[0]

5.11.13.10 Converting SQLite values to custom Python types
..........................................................

Writing an adapter lets you send custom Python types to SQLite. But to
make it really useful we need to make the Python to SQLite to Python
roundtrip work.

  Enter converters.

  Let's go back to the `Point' class. We stored the x and y coordinates
separated via semicolons as strings in SQLite.

  First, we'll define a converter function that accepts the string as a
parameter and constructs a `Point' object from it.

     Note: Converter functions *always* get called with a string, no
     matter under which data type you sent the value to SQLite.

    def convert_point(s):
        x, y = map(float, s.split(";"))
        return Point(x, y)

Now you need to make the sqlite3 module know that what you
select from the database is actually a point. There are two ways of
doing this:

   * Implicitly via the declared type

   * Explicitly via the column name

  Both ways are described in section Module functions and constants,
in the entries for the constants PARSE_DECLTYPES
and PARSE_COLNAMES.

  The following example illustrates both approaches.

    import sqlite3

    class Point(object):
        def __init__(self, x, y):
            self.x, self.y = x, y

        def __repr__(self):
            return "(%f;%f)" % (self.x, self.y)

    def adapt_point(point):
        return "%f;%f" % (point.x, point.y)

    def convert_point(s):
        x, y = map(float, s.split(";"))
        return Point(x, y)

    # Register the adapter
    sqlite3.register_adapter(Point, adapt_point)

    # Register the converter
    sqlite3.register_converter("point", convert_point)

    p = Point(4.0, -3.2)

    #########################
    # 1) Using declared types
    con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES)
    cur = con.cursor()
    cur.execute("create table test(p point)")

    cur.execute("insert into test(p) values (?)", (p,))
    cur.execute("select p from test")
    print "with declared types:", cur.fetchone()[0]
    cur.close()
    con.close()

    #######################
    # 1) Using column names
    con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_COLNAMES)
    cur = con.cursor()
    cur.execute("create table test(p)")

    cur.execute("insert into test(p) values (?)", (p,))
    cur.execute('select p as "p [point]" from test')
    print "with column names:", cur.fetchone()[0]
    cur.close()
    con.close()

5.11.13.11 Default adapters and converters
..........................................

There are default adapters for the date and datetime types in the
datetime module. They will be sent as ISO dates/ISO timestamps to
SQLite.

  The default converters are registered under the name "date" for
datetime.date and under the name "timestamp" for
datetime.datetime.

  This way, you can use date/timestamps from Python without any
additional fiddling in most cases. The format of the adapters is also
compatible with the experimental SQLite date/time functions.

  The following example demonstrates this.

    import sqlite3
    import datetime

    con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    cur = con.cursor()
    cur.execute("create table test(d date, ts timestamp)")

    today = datetime.date.today()
    now = datetime.datetime.now()

    cur.execute("insert into test(d, ts) values (?, ?)", (today, now))
    cur.execute("select d, ts from test")
    row = cur.fetchone()
    print today, "=>", row[0], type(row[0])
    print now, "=>", row[1], type(row[1])

    cur.execute('select current_date as "d [date]", current_timestamp as "ts [timestamp]"')
    row = cur.fetchone()
    print "current_date", row[0], type(row[0])
    print "current_timestamp", row[1], type(row[1])

5.11.13.12 Controlling Transactions
...................................

By default, the sqlite3 module opens transactions
implicitly before a Data Modification Language (DML)  statement (i.e.
`INSERT'/`UPDATE'/`DELETE'/`REPLACE'), and commits transactions
implicitly before a non-DML, non-query statement (i. e.  anything other
than `SELECT' or the aforementioned).

  So if you are within a transaction and issue a command like `CREATE
TABLE ...', `VACUUM', `PRAGMA', the sqlite3 module will
commit implicitly before executing that command. There are two reasons
for doing that. The first is that some of these commands don't work
within transactions. The other reason is that sqlite3 needs to keep
track of the transaction state (if a transaction is active or not).

  You can control which kind of `BEGIN' statements sqlite3 implicitly
executes (or none at all) via the _isolation_level_ parameter to the
connect().  call, or via the `isolation_level' property of
connections.

  If you want *autocommit mode*, then set `isolation_level' to None.

  Otherwise leave it at its default, which will result in a plain
"BEGIN" statement, or set it to one of SQLite's supported isolation
levels: "DEFERRED", "IMMEDIATE" or "EXCLUSIVE".

5.11.13.13 Using `sqlite3' efficiently
......................................

5.11.13.14 Using shortcut methods
.................................

Using the nonstandard `execute()', `executemany()' and
`executescript()' methods of the Connection object, your
code can be written more concisely because you don't have to create the
(often superfluous) Cursor objects explicitly. Instead, the
Cursor.  objects are created implicitly and these shortcut
methods return the cursor objects. This way, you can execute a `SELECT'
statement and iterate over it directly using only a single call on the
Connection object.

    import sqlite3

    persons = [
        ("Hugo", "Boss"),
        ("Calvin", "Klein")
        ]

    con = sqlite3.connect(":memory:")

    # Create the table
    con.execute("create table person(firstname, lastname)")

    # Fill the table
    con.executemany("insert into person(firstname, lastname) values (?, ?)", persons)

    # Print the table contents
    for row in con.execute("select firstname, lastname from person"):
        print row

    # Using a dummy WHERE clause to not let SQLite take the shortcut table deletes.
    print "I just deleted", con.execute("delete from person where 1=1").rowcount, "rows"

5.11.13.15 Accessing columns by name instead of by index
........................................................

One useful feature of the sqlite3 module is the built-in
sqlite3.Row class designed to be used as a row factory.

  Rows wrapped with this class can be accessed both by index (like
tuples) and case-insensitively by name:

    import sqlite3

    con = sqlite3.connect("mydb")
    con.row_factory = sqlite3.Row

    cur = con.cursor()
    cur.execute("select name_last, age from people")
    for row in cur:
        assert row[0] == row["name_last"]
        assert row["name_last"] == row["nAmE_lAsT"]
        assert row[1] == row["age"]
        assert row[1] == row["AgE"]

5.11.13.16 Using the connection as a context manager
....................................................

New in version 2.6.

  Connection objects can be used as context managers that automatically
commit or rollback transactions.  In the event of an exception, the
transaction is rolled back; otherwise, the transaction is committed:

    import sqlite3

    con = sqlite3.connect(":memory:")
    con.execute("create table person (id integer primary key, firstname varchar unique)")

    # Successful, con.commit() is called automatically afterwards
    with con:
        con.execute("insert into person(firstname) values (?)", ("Joe",))

    # con.rollback() is called after the with block finishes with an exception, the
    # exception is still raised and must be catched
    try:
        with con:
            con.execute("insert into person(firstname) values (?)", ("Joe",))
    except sqlite3.IntegrityError:
        print "couldn't add Joe twice"

5.11.13.17 Common issues
........................

5.11.13.18 Multithreading
.........................

Older SQLite versions had issues with sharing connections between
threads.  That's why the Python module disallows sharing connections
and cursors between threads. If you still try to do so, you will get an
exception at runtime.

  The only exception is calling the interrupt() method,
which only makes sense to call from a different thread.

5.12 Data Compression and Archiving
===================================

The modules described in this chapter support data compression with the
zlib, gzip, and bzip2 algorithms, and  the creation of ZIP- and
tar-format archives.

5.12.1 `zlib' -- Compression compatible with *gzip*
---------------------------------------------------

For applications that require data compression, the functions in this
module allow compression and decompression, using the zlib library. The
zlib library has its own home page at <http://www.zlib.net>.   There
are known incompatibilities between the Python module and versions of
the zlib library earlier than 1.1.3; 1.1.3 has a security
vulnerability, so we recommend using 1.1.4 or later.

  zlib's functions have many options and often need to be used in a
particular order.  This documentation doesn't attempt to cover all of
the permutations; consult the zlib manual at
<http://www.zlib.net/manual.html> for authoritative information.

  For reading and writing `.gz' files see the gzip module.
For other archive formats, see the bz2, zipfile,
and tarfile modules.

  The available exception and functions in this module are:

 -- Exception: zlib.error
     Exception raised on compression and decompression errors.

 -- Function: zlib.adler32 (data[, value])
     Computes a Adler-32 checksum of _data_.  (An Adler-32 checksum is
     almost as reliable as a CRC32 but can be computed much more
     quickly.)  If _value_ is present, it is used as the starting value
     of the checksum; otherwise, a fixed default value is used.  This
     allows computing a running checksum over the concatenation of
     several inputs.  The algorithm is not cryptographically strong,
     and should not be used for authentication or digital signatures.
     Since the algorithm is designed for use as a checksum algorithm,
     it is not suitable for use as a general hash algorithm.

     This function always returns an integer object.

     Note: To generate the same numeric value across all Python
     versions and platforms use adler32(data) & 0xffffffff.  If you are
     only using the checksum in packed binary format this is not
     necessary as the return value is the correct 32bit binary
     representation regardless of sign.

  Changed in version 2.6: The return value is in the range [-2**31,
2**31-1] regardless of platform.  In older versions the value is signed
on some platforms and unsigned on others.

  Changed in version 3.0: The return value is unsigned and in the range
[0, 2**32-1] regardless of platform.

 -- Function: zlib.compress (string[, level])
     Compresses the data in _string_, returning a string contained
     compressed data.  _level_ is an integer from `1' to `9'
     controlling the level of compression; `1' is fastest and produces
     the least compression, `9' is slowest and produces the most.  The
     default value is `6'.  Raises the error.  exception if
     any error occurs.

 -- Function: zlib.compressobj ([level])
     Returns a compression object, to be used for compressing data
     streams that won't fit into memory at once.  _level_ is an integer
     from `1' to `9' controlling the level of compression; `1' is
     fastest and produces the least compression, `9' is slowest and
     produces the most.  The default value is `6'.

 -- Function: zlib.crc32 (data[, value])
     Computes a CRC (Cyclic Redundancy Check)  checksum of _data_. If
     _value_ is present, it is used as the starting value of the
     checksum; otherwise, a fixed default value is used.  This allows
     computing a running checksum over the concatenation of several
     inputs.  The algorithm is not cryptographically strong, and should
     not be used for authentication or digital signatures.  Since the
     algorithm is designed for use as a checksum algorithm, it is not
     suitable for use as a general hash algorithm.

     This function always returns an integer object.

     Note: To generate the same numeric value across all Python
     versions and platforms use crc32(data) & 0xffffffff.  If you are
     only using the checksum in packed binary format this is not
     necessary as the return value is the correct 32bit binary
     representation regardless of sign.

  Changed in version 2.6: The return value is in the range [-2**31,
2**31-1] regardless of platform.  In older versions the value would be
signed on some platforms and unsigned on others.

  Changed in version 3.0: The return value is unsigned and in the range
[0, 2**32-1] regardless of platform.

 -- Function: zlib.decompress (string[, wbits[, bufsize]])
     Decompresses the data in _string_, returning a string containing
     the uncompressed data.  The _wbits_ parameter controls the size of
     the window buffer, and is discussed further below.  If _bufsize_
     is given, it is used as the initial size of the output buffer.
     Raises the error exception if any error occurs.

     The absolute value of _wbits_ is the base two logarithm of the
     size of the history buffer (the "window size") used when
     compressing data.  Its absolute value should be between 8 and 15
     for the most recent versions of the zlib library, larger values
     resulting in better compression at the expense of greater memory
     usage.  When decompressing a stream, _wbits_ must not be smaller
     than the size originally used to compress the stream; using a
     too-small value will result in an exception. The default value is
     therefore the highest value, 15.  When _wbits_ is negative, the
     standard *gzip* header is suppressed.

     _bufsize_ is the initial size of the buffer used to hold
     decompressed data.  If more space is required, the buffer size
     will be increased as needed, so you don't have to get this value
     exactly right; tuning it will only save a few calls to `malloc()'.
     The default size is 16384.

 -- Function: zlib.decompressobj ([wbits])
     Returns a decompression object, to be used for decompressing data
     streams that won't fit into memory at once.  The _wbits_ parameter
     controls the size of the window buffer.

  Compression objects support the following methods:

 -- Method: Compress.compress (string)
     Compress _string_, returning a string containing compressed data
     for at least part of the data in _string_.  This data should be
     concatenated to the output produced by any preceding calls to the
     compress() method.  Some input may be kept in internal
     buffers for later processing.

 -- Method: Compress.flush ([mode])
     All pending input is processed, and a string containing the
     remaining compressed output is returned.  _mode_ can be selected
     from the constants `Z_SYNC_FLUSH',  `Z_FULL_FLUSH',  or
     `Z_FINISH', defaulting to `Z_FINISH'.  `Z_SYNC_FLUSH' and
     `Z_FULL_FLUSH' allow compressing further strings of data, while
     `Z_FINISH' finishes the compressed stream and  prevents
     compressing any more data.  After calling flush() with
     _mode_ set to `Z_FINISH', the compress() method cannot
     be called again; the only realistic action is to delete the object.

 -- Method: Compress.copy ()
     Returns a copy of the compression object.  This can be used to
     efficiently compress a set of data that share a common initial
     prefix.

     New in version 2.5.

  Decompression objects support the following methods, and two
attributes:

 -- Attribute: Decompress.unused_data
     A string which contains any bytes past the end of the compressed
     data. That is, this remains `""' until the last byte that contains
     compression data is available.  If the whole string turned out to
     contain compressed data, this is `""', the empty string.

     The only way to determine where a string of compressed data ends
     is by actually decompressing it.  This means that when compressed
     data is contained part of a larger file, you can only find the end
     of it by reading data and feeding it followed by some non-empty
     string into a decompression object's decompress().
     method until the unused_data attribute is no longer
     the empty string.

 -- Attribute: Decompress.unconsumed_tail
     A string that contains any data that was not consumed by the last
     decompress() call because it exceeded the limit for
     the uncompressed data buffer.  This data has not yet been seen by
     the zlib machinery, so you must feed it (possibly with further
     data concatenated to it) back to a subsequent decompress()
     method call in order to get correct output.

 -- Method: Decompress.decompress (string[, max_length])
     Decompress _string_, returning a string containing the
     uncompressed data corresponding to at least part of the data in
     _string_.  This data should be concatenated to the output produced
     by any preceding calls to the decompress() method.
     Some of the input data may be preserved in internal buffers for
     later processing.

     If the optional parameter _max_length_ is supplied then the return
     value will be no longer than _max_length_. This may mean that not
     all of the compressed input can be processed; and unconsumed data
     will be stored in the attribute unconsumed_tail. This
     string must be passed to a subsequent call to decompress()
     if decompression is to continue.  If _max_length_ is not
     supplied then the whole input is decompressed, and
     unconsumed_tail is an empty string.

 -- Method: Decompress.flush ([length])
     All pending input is processed, and a string containing the
     remaining uncompressed output is returned.  After calling
     flush(), the decompress() method cannot be called
     again; the only realistic action is to delete the object.

     The optional parameter _length_ sets the initial size of the
     output buffer.

 -- Method: Decompress.copy ()
     Returns a copy of the decompression object.  This can be used to
     save the state of the decompressor midway through the data stream
     in order to speed up random seeks into the stream at a future
     point.

     New in version 2.5.

Module gzip.
     Reading and writing *gzip*-format files.

<http://www.zlib.net>
     The zlib library home page.

<http://www.zlib.net/manual.html>
     The zlib manual explains  the semantics and usage of the library's
     many functions.

5.12.2 `gzip' -- Support for *gzip* files
-----------------------------------------

This module provides a simple interface to compress and decompress
files just like the GNU programs *gzip* and *gunzip* would.

  The data compression is provided by the zlib module.

  The gzip module provides the GzipFile class
which is modeled after Python's File Object. The GzipFile.
class reads and writes *gzip*-format files, automatically compressing
or decompressing the data so that it looks like an ordinary file object.

  Note that additional file formats which can be decompressed by the
*gzip* and *gunzip* programs, such  as those produced by *compress* and
*pack*, are not supported by this module.

  For other archive formats, see the bz2, zipfile,
and tarfile modules.

  The module defines the following items:

 -- Class: gzip.GzipFile ([filename[, mode[, compresslevel[, fileobj[,
          mtime]]]]])
     Constructor for the GzipFile class, which simulates
     most of the methods of a file object, with the exception of the
     `readinto()' and `truncate()' methods.  At least one of _fileobj_
     and _filename_ must be given a non-trivial value.

     The new class instance is based on _fileobj_, which can be a
     regular file, a StringIO object, or any other object
     which simulates a file.  It defaults to `None', in which case
     _filename_ is opened to provide a file object.

     When _fileobj_ is not `None', the _filename_ argument is only used
     to be included in the *gzip* file header, which may includes the
     original filename of the uncompressed file.  It defaults to the
     filename of _fileobj_, if discernible; otherwise, it defaults to
     the empty string, and in this case the original filename is not
     included in the header.

     The _mode_ argument can be any of `'r'', `'rb'', `'a'', `'ab'',
     `'w'', or `'wb'', depending on whether the file will be read or
     written.  The default is the mode of _fileobj_ if discernible;
     otherwise, the default is `'rb''. If not given, the 'b' flag will
     be added to the mode to ensure the file is opened in binary mode
     for cross-platform portability.

     The _compresslevel_ argument is an integer from `1' to `9'
     controlling the level of compression; `1' is fastest and produces
     the least compression, and `9' is slowest and produces the most
     compression.  The default is `9'.

     The _mtime_ argument is an optional numeric timestamp to be
     written to the stream when compressing.  All *gzip* compressed
     streams are required to contain a timestamp.  If omitted or
     `None', the current time is used.  This module ignores the
     timestamp when decompressing; however, some programs, such as
     *gunzip*, make use of it.  The format of the timestamp is the same
     as that of the return value of `time.time()' and of the `st_mtime'
     member of the object returned by `os.stat()'.

     Calling a GzipFile object's `close()' method does not
     close _fileobj_, since you might wish to append more material
     after the compressed data.  This also allows you to pass a
     StringIO object opened for writing as _fileobj_, and
     retrieve the resulting memory buffer using the StringIO
     object's `getvalue()' method.

     GzipFile supports iteration and the with.
     statement.

     Changed in version 2.7: Support for the with statement
     was added.

     Changed in version 2.7: Support for zero-padded files was added.

 -- Function: gzip.open (filename[, mode[, compresslevel]])
     This is a shorthand for `GzipFile(filename,' `mode,'
     `compresslevel)'.  The _filename_ argument is required; _mode_
     defaults to `'rb'' and _compresslevel_ defaults to `9'.

5.12.2.1 Examples of usage
..........................

Example of how to read a compressed file:

    import gzip
    f = gzip.open('/home/joe/file.txt.gz', 'rb')
    file_content = f.read()
    f.close()

Example of how to create a compressed GZIP file:

    import gzip
    content = "Lots of content here"
    f = gzip.open('/home/joe/file.txt.gz', 'wb')
    f.write(content)
    f.close()

Example of how to GZIP compress an existing file:

    import gzip
    f_in = open('/home/joe/file.txt', 'rb')
    f_out = gzip.open('/home/joe/file.txt.gz', 'wb')
    f_out.writelines(f_in)
    f_out.close()
    f_in.close()

Module zlib.
     The basic data compression module needed to support the *gzip* file
     format.

5.12.3 `bz2' -- Compression compatible with *bzip2*
---------------------------------------------------

New in version 2.3.

  This module provides a comprehensive interface for the bz2
compression library.  It implements a complete file interface, one-shot
(de)compression functions, and types for sequential (de)compression.

  For other archive formats, see the gzip, zipfile,
and tarfile modules.

  Here is a summary of the features offered by the bz2 module:

   * BZ2File class implements a complete file interface,
     including readline(), readlines(),
     writelines(), seek(), etc;

   * BZ2File class implements emulated seek().
     support;

   * BZ2File class implements universal newline support;

   * BZ2File class offers an optimized line iteration using
     the readahead algorithm borrowed from file objects;

   * Sequential (de)compression supported by BZ2Compressor.
     and BZ2Decompressor classes;

   * One-shot (de)compression supported by compress() and
     decompress().  functions;

   * Thread safety uses individual locking mechanism.

5.12.3.1 (De)compression of files
.................................

Handling of compressed files is offered by the BZ2File.
class.

 -- Class: bz2.BZ2File (filename[, mode[, buffering[, compresslevel]]])
     Open a bz2 file. Mode can be either `'r'' or `'w'', for reading
     (default) or writing. When opened for writing, the file will be
     created if it doesn't exist, and truncated otherwise. If
     _buffering_ is given, `0' means unbuffered, and larger numbers
     specify the buffer size; the default is `0'. If _compresslevel_ is
     given, it must be a number between `1' and `9'; the default is
     `9'. Add a `'U'' to mode to open the file for input with universal
     newline support. Any line ending in the input file will be seen as
     a `'\n'' in Python.  Also, a file so opened gains the attribute
     `newlines'; the value for this attribute is one of `None' (no
     newline read yet), `'\r'', `'\n'', `'\r\n'' or a tuple containing
     all the newline types seen. Universal newlines are available only
     when reading. Instances support iteration in the same way as
     normal file.  instances.

     BZ2File supports the with statement.

     Changed in version 2.7: Support for the with statement
     was added.

      -- Method: close ()
          Close the file. Sets data attribute `closed' to true. A
          closed file cannot be used for further I/O operations.
          close() may be called more than once without error.

      -- Method: read ([size])
          Read at most _size_ uncompressed bytes, returned as a string.
          If the _size_ argument is negative or omitted, read until EOF
          is reached.

      -- Method: readline ([size])
          Return the next line from the file, as a string, retaining
          newline. A non-negative _size_ argument limits the maximum
          number of bytes to return (an incomplete line may be returned
          then). Return an empty string at EOF.

      -- Method: readlines ([size])
          Return a list of lines read. The optional _size_ argument, if
          given, is an approximate bound on the total number of bytes
          in the lines returned.

      -- Method: xreadlines ()
          For backward compatibility. BZ2File objects now
          include the performance optimizations previously implemented
          in the `xreadlines' module.

          Deprecated since version 2.3: This exists only for
          compatibility with the method by this name on file
          objects, which is deprecated.  Use `for line in file'
          instead.

      -- Method: seek (offset[, whence])
          Move to new file position. Argument _offset_ is a byte count.
          Optional argument _whence_ defaults to `os.SEEK_SET' or `0'
          (offset from start of file; offset should be `>= 0'); other
          values are `os.SEEK_CUR' or `1' (move relative to current
          position; offset can be positive or negative), and
          `os.SEEK_END' or `2' (move relative to end of file; offset is
          usually negative, although many platforms allow seeking beyond
          the end of a file).

          Note that seeking of bz2 files is emulated, and depending on
          the parameters the operation may be extremely slow.

      -- Method: tell ()
          Return the current file position, an integer (may be a long
          integer).

      -- Method: write (data)
          Write string _data_ to file. Note that due to buffering,
          see close() may be needed before the file on disk
          reflects the data written.

      -- Method: writelines (sequence_of_strings)
          Write the sequence of strings to the file. Note that newlines
          are not added. The sequence can be any iterable object
          producing strings. This is equivalent to calling write() for
          each string.

5.12.3.2 Sequential (de)compression
...................................

Sequential compression and decompression is done using the classes
BZ2Compressor and BZ2Decompressor.

 -- Class: bz2.BZ2Compressor ([compresslevel])
     Create a new compressor object. This object may be used to
     compress data sequentially. If you want to compress data in one
     shot, use the compress() function instead. The
     _compresslevel_ parameter, if given, must be a number between `1'
     and `9'; the default is `9'.

      -- Method: compress (data)
          Provide more data to the compressor object. It will return
          chunks of compressed data whenever possible. When you've
          finished providing data to compress, call the flush()
          method to finish the compression process, and return
          what is left in internal buffers.

      -- Method: flush ()
          Finish the compression process and return what is left in
          internal buffers. You must not use the compressor object
          after calling this method.

 -- Class: bz2.BZ2Decompressor
     Create a new decompressor object. This object may be used to
     decompress data sequentially. If you want to decompress data in
     one shot, use the decompress() function instead.

      -- Method: decompress (data)
          Provide more data to the decompressor object. It will return
          chunks of decompressed data whenever possible. If you try to
          decompress data after the end of stream is found, see
          EOFError will be raised. If any data was found after
          the end of stream, it'll be ignored and saved in
          `unused_data' attribute.

5.12.3.3 One-shot (de)compression
.................................

One-shot compression and decompression is provided through the
compress().  and decompress() functions.

 -- Function: bz2.compress (data[, compresslevel])
     Compress _data_ in one shot. If you want to compress data
     sequentially, use an instance of BZ2Compressor.
     instead. The _compresslevel_ parameter, if given, must be a number
     between `1' and `9'; the default is `9'.

 -- Function: bz2.decompress (data)
     Decompress _data_ in one shot. If you want to decompress data
     sequentially, use an instance of BZ2Decompressor.
     instead.

5.12.4 `zipfile' -- Work with ZIP archives
------------------------------------------

New in version 1.6.

  The ZIP file format is a common archive and compression standard.
This module provides tools to create, read, write, append, and list a
ZIP file.  Any advanced use of this module will require an
understanding of the format, as defined in PKZIP Application Note
(http://www.pkware.com/documents/casestudies/APPNOTE.TXT).

  This module does not currently handle multi-disk ZIP files.  It can
handle ZIP files that use the ZIP64 extensions (that is ZIP files that
are more than 4 GByte in size).  It supports decryption of encrypted
files in ZIP archives, but it currently cannot create an encrypted
file.  Decryption is extremely slow as it is implemented in native
Python rather than C.

  For other archive formats, see the bz2, gzip, and
tarfile modules.

  The module defines the following items:

 -- Exception: zipfile.BadZipfile
     The error raised for bad ZIP files (old name: `zipfile.error').

 -- Exception: zipfile.LargeZipFile
     The error raised when a ZIP file would require ZIP64 functionality
     but that has not been enabled.

 -- Class: zipfile.ZipFile
     The class for reading and writing ZIP files.  See section
     ZipFile Objects for constructor details.

 -- Class: zipfile.PyZipFile
     Class for creating ZIP archives containing Python libraries.

 -- Class: zipfile.ZipInfo ([filename[, date_time]])
     Class used to represent information about a member of an archive.
     Instances of this class are returned by the `getinfo()' and
     `infolist()' methods of ZipFile objects.  Most users
     of the zipfile module will not need to create these,
     but only use those created by this module. _filename_ should be
     the full name of the archive member, and _date_time_ should be a
     tuple containing six fields which describe the time of the last
     modification to the file; the fields are described in section
     ZipInfo Objects.

 -- Function: zipfile.is_zipfile (filename)
     Returns `True' if _filename_ is a valid ZIP file based on its
     magic number, otherwise returns `False'.  _filename_ may be a file
     or file-like object too.

     Changed in version 2.7: Support for file and file-like objects.

 -- Data: zipfile.ZIP_STORED
     The numeric constant for an uncompressed archive member.

 -- Data: zipfile.ZIP_DEFLATED
     The numeric constant for the usual ZIP compression method.  This
     requires the zlib module.  No other compression methods are
     currently supported.

PKZIP Application Note (http://www.pkware.com/documents/casestudies/APPNOTE.TXT)
     Documentation on the ZIP file format by Phil Katz, the creator of
     the format and algorithms used.

Info-ZIP Home Page (http://www.info-zip.org/)
     Information about the Info-ZIP project's ZIP archive programs and
     development libraries.

5.12.4.1 ZipFile Objects
........................

 -- Class: zipfile.ZipFile (file[, mode[, compression[, allowZip64]]])
     Open a ZIP file, where _file_ can be either a path to a file (a
     string) or a file-like object.  The _mode_ parameter should be
     `'r'' to read an existing file, `'w'' to truncate and write a new
     file, or `'a'' to append to an existing file.  If _mode_ is `'a''
     and _file_ refers to an existing ZIP file, then additional files
     are added to it.  If _file_ does not refer to a ZIP file, then a
     new ZIP archive is appended to the file.  This is meant for adding
     a ZIP archive to another file (such as `python.exe').

     Changed in version 2.6: If _mode_ is `a' and the file does not
     exist at all, it is created.

     _compression_ is the ZIP compression method to use when writing
     the archive, and should be ZIP_STORED or
     ZIP_DEFLATED.; unrecognized values will cause
     RuntimeError to be raised.  If ZIP_DEFLATED.  is
     specified but the zlib module is not available, see
     RuntimeError.  is also raised. The default is
     ZIP_STORED.  If _allowZip64_ is `True' zipfile will create
     ZIP files that use the ZIP64 extensions when the zipfile is larger
     than 2 GB. If it is  false (the default) zipfile.  will
     raise an exception when the ZIP file would require ZIP64
     extensions.  ZIP64 extensions are disabled by default because the
     default *zip* and *unzip* commands on Unix (the InfoZIP utilities)
     don't support these extensions.

     If the file is created with mode `'a'' or `'w'' and then
     close().d without adding any files to the archive, the
     appropriate ZIP structures for an empty archive will be written to
     the file.

     ZipFile is also a context manager and therefore supports the
     with statement.  In the example, _myzip_ is closed after the
     with statement's suite is finished--even if an
     exception occurs:

         with ZipFile('spam.zip', 'w') as myzip:
             myzip.write('eggs.txt')

     New in version 2.7: Added the ability to use ZipFile.
     as a context manager.

 -- Method: ZipFile.close ()
     Close the archive file.  You must call close() before
     exiting your program or essential records will not be written.

 -- Method: ZipFile.getinfo (name)
     Return a ZipInfo object with information about the
     archive member _name_.  Calling getinfo() for a name
     not currently contained in the archive will raise a
     KeyError.

 -- Method: ZipFile.infolist ()
     Return a list containing a ZipInfo object for each
     member of the archive.  The objects are in the same order as their
     entries in the actual ZIP file on disk if an existing archive was
     opened.

 -- Method: ZipFile.namelist ()
     Return a list of archive members by name.

 -- Method: ZipFile.open (name[, mode[, pwd]])
     Extract a member from the archive as a file-like object
     (ZipExtFile). _name_ is the name of the file in the archive, or a
     ZipInfo object. The _mode_ parameter, if included,
     must be one of the following: `'r'' (the  default), `'U'', or
     `'rU''. Choosing `'U'' or  `'rU'' will enable universal newline
     support in the read-only object. _pwd_ is the password used for
     encrypted files.  Calling  open() on a closed ZipFile
     will raise a  RuntimeError.

          Note: The file-like object is read-only and provides the
          following methods: read(), readline(),
          `readlines()', see __iter__(), next().

          Note: If the ZipFile was created by passing in a file-like
          object as the  first argument to the constructor, then the
          object returned by open() shares the ZipFile's
          file pointer.  Under these  circumstances, the object
          returned by open() should not  be used after any
          additional operations are performed on the  ZipFile object.
          If the ZipFile was created by passing in a string (the
          filename) as the first argument to the constructor, then
          open() will create a new file object that will be
          held by the ZipExtFile, allowing it to operate independently
          of the  ZipFile.

          Note: The open(), read() and
          extract() methods can take a filename or a
          ZipInfo object.  You will appreciate this when trying
          to read a ZIP file that contains members with duplicate names.

     New in version 2.6.

 -- Method: ZipFile.extract (member[, path[, pwd]])
     Extract a member from the archive to the current working
     directory; _member_ must be its full name or a ZipInfo.
     object).  Its file information is extracted as accurately as
     possible.  _path_ specifies a different directory to extract to.
     _member_ can be a filename or a ZipInfo object.  _pwd_
     is the password used for encrypted files.

     New in version 2.6.

 -- Method: ZipFile.extractall ([path[, members[, pwd]]])
     Extract all members from the archive to the current working
     directory.  _path_ specifies a different directory to extract to.
     _members_ is optional and must be a subset of the list returned by
     namelist().  _pwd_ is the password used for encrypted
     files.

          Warning: Never extract archives from untrusted sources
          without prior inspection.  It is possible that files are
          created outside of _path_, e.g. members that have absolute
          filenames starting with `"/"' or filenames with two dots
          `".."'.

     New in version 2.6.

 -- Method: ZipFile.printdir ()
     Print a table of contents for the archive to `sys.stdout'.

 -- Method: ZipFile.setpassword (pwd)
     Set _pwd_ as default password to extract encrypted files.

     New in version 2.6.

 -- Method: ZipFile.read (name[, pwd])
     Return the bytes of the file _name_ in the archive.  _name_ is the
     name of the file in the archive, or a ZipInfo object.
     The archive must be open for read or append. _pwd_ is the password
     used for encrypted  files and, if specified, it will override the
     default password set with setpassword().  Calling
     read() on a closed ZipFile  will raise a RuntimeError.

     Changed in version 2.6: _pwd_ was added, and _name_ can now be a
     ZipInfo object.

 -- Method: ZipFile.testzip ()
     Read all the files in the archive and check their CRC's and file
     headers.  Return the name of the first bad file, or else return
     `None'. Calling testzip() on a closed ZipFile will
     raise a RuntimeError.

 -- Method: ZipFile.write (filename[, arcname[, compress_type]])
     Write the file named _filename_ to the archive, giving it the
     archive name _arcname_ (by default, this will be the same as
     _filename_, but without a drive letter and with leading path
     separators removed).  If given, _compress_type_ overrides the
     value given for the _compression_ parameter to the constructor for
     the new entry.  The archive must be open with mode `'w'' or `'a''
     - calling write() on a ZipFile created with mode `'r''
     will raise a RuntimeError.  Calling  write()
     on a closed ZipFile will raise a RuntimeError.

          Note: There is no official file name encoding for ZIP files.
          If you have unicode file names, you must convert them to byte
          strings in your desired encoding before passing them to
          write(). WinZip interprets all file names as encoded in
          CP437, also known as DOS Latin.

          Note: Archive names should be relative to the archive root,
          that is, they should not start with a path separator.

          Note: If `arcname' (or `filename', if `arcname' is  not
          given) contains a null byte, the name of the file in the
          archive will be truncated at the null byte.

 -- Method: ZipFile.writestr (zinfo_or_arcname, bytes[, compress_type])
     Write the string _bytes_ to the archive; _zinfo_or_arcname_ is
     either the file name it will be given in the archive, or a
     ZipInfo instance.  If it's an instance, at least the
     filename, date, and time must be given.  If it's a name, the date
     and time is set to the current date and time. The archive must be
     opened with mode `'w'' or `'a'' - calling  writestr().
     on a ZipFile created with mode `'r''  will raise a
     RuntimeError.  Calling writestr() on a closed
     ZipFile will raise a RuntimeError.

     If given, _compress_type_ overrides the value given for the
     _compression_ parameter to the constructor for the new entry, or
     in the _zinfo_or_arcname_ (if that is a ZipInfo.
     instance).

          Note: When passing a ZipInfo instance as the
          _zinfo_or_acrname_ parameter, the compression method used
          will be that specified in the _compress_type_ member of the
          given ZipInfo instance.  By default, the
          ZipInfo constructor sets this member to
          ZIP_STORED.

     Changed in version 2.7: The _compression_type_ argument.

  The following data attributes are also available:

 -- Attribute: ZipFile.debug
     The level of debug output to use.  This may be set from `0' (the
     default, no output) to `3' (the most output).  Debugging
     information is written to `sys.stdout'.

 -- Attribute: ZipFile.comment
     The comment text associated with the ZIP file.  If assigning a
     comment to a ZipFile instance created with mode 'a' or
     'w', this should be a string no longer than 65535 bytes.  Comments
     longer than this will be truncated in the written archive when
     ZipFile.close() is called.

5.12.4.2 PyZipFile Objects
..........................

The PyZipFile constructor takes the same parameters as the
ZipFile constructor.  Instances have one method in addition
to those of ZipFile objects.

 -- Method: PyZipFile.writepy (pathname[, basename])
     Search for files `*.py' and add the corresponding file to the
     archive.  The corresponding file is a `*.pyo' file if available,
     else a `*.pyc' file, compiling if necessary.  If the pathname is a
     file, the filename must end with `.py', and just the (corresponding
     `*.py[co]') file is added at the top level (no path information).
     If the pathname is a file that does not end with `.py', a
     RuntimeError.  will be raised.  If it is a directory, and the
     directory is not a package directory, then all the files
     `*.py[co]' are added at the top level.  If the directory is a
     package directory, then all `*.py[co]' are added under the package
     name as a file path, and if any subdirectories are package
     directories, all of these are added recursively.  _basename_ is
     intended for internal use only.  The writepy() method
     makes archives with file names like this:

         string.pyc                                # Top level name
         test/__init__.pyc                         # Package directory
         test/test_support.pyc                          # Module test.test_support
         test/bogus/__init__.pyc                   # Subpackage directory
         test/bogus/myfile.pyc                     # Submodule test.bogus.myfile

5.12.4.3 ZipInfo Objects
........................

Instances of the ZipInfo class are returned by the
`getinfo()' and `infolist()' methods of ZipFile objects.
Each object stores information about a single member of the ZIP archive.

  Instances have the following attributes:

 -- Attribute: ZipInfo.filename
     Name of the file in the archive.

 -- Attribute: ZipInfo.date_time
     The time and date of the last modification to the archive member.
     This is a tuple of six values:

     Index       Value
     -------------------------------------------
     `0'         Year
     `1'         Month (one-based)
     `2'         Day of month (one-based)
     `3'         Hours (zero-based)
     `4'         Minutes (zero-based)
     `5'         Seconds (zero-based)

 -- Attribute: ZipInfo.compress_type
     Type of compression for the archive member.

 -- Attribute: ZipInfo.comment
     Comment for the individual archive member.

 -- Attribute: ZipInfo.extra
     Expansion field data.  The PKZIP Application Note
     (http://www.pkware.com/documents/casestudies/APPNOTE.TXT) contains
     some comments on the internal structure of the data contained in
     this string.

 -- Attribute: ZipInfo.create_system
     System which created ZIP archive.

 -- Attribute: ZipInfo.create_version
     PKZIP version which created ZIP archive.

 -- Attribute: ZipInfo.extract_version
     PKZIP version needed to extract archive.

 -- Attribute: ZipInfo.reserved
     Must be zero.

 -- Attribute: ZipInfo.flag_bits
     ZIP flag bits.

 -- Attribute: ZipInfo.volume
     Volume number of file header.

 -- Attribute: ZipInfo.internal_attr
     Internal attributes.

 -- Attribute: ZipInfo.external_attr
     External file attributes.

 -- Attribute: ZipInfo.header_offset
     Byte offset to the file header.

 -- Attribute: ZipInfo.CRC
     CRC-32 of the uncompressed file.

 -- Attribute: ZipInfo.compress_size
     Size of the compressed data.

 -- Attribute: ZipInfo.file_size
     Size of the uncompressed file.

5.12.5 `tarfile' -- Read and write tar archive files
----------------------------------------------------

New in version 2.3.

  The tarfile module makes it possible to read and write tar
archives, including those using gzip or bz2 compression.  (`.zip' files
can be read and written using the zipfile module.)

  Some facts and figures:

   * reads and writes gzip and bz2 compressed
     archives.

   * read/write support for the POSIX.1-1988 (ustar) format.

   * read/write support for the GNU tar format including _longname_ and
     _longlink_ extensions, read-only support for the _sparse_
     extension.

   * read/write support for the POSIX.1-2001 (pax) format.

     New in version 2.6.

   * handles directories, regular files, hardlinks, symbolic links,
     fifos, character devices and block devices and is able to acquire
     and restore file information like timestamp, access permissions
     and owner.

 -- Function: tarfile.open (name=None, mode='r', fileobj=None,
          bufsize=10240, **kwargs)
     Return a TarFile object for the pathname _name_. For
     detailed information on TarFile objects and the
     keyword arguments that are allowed, see TarFile Objects.

     _mode_ has to be a string of the form `'filemode[:compression]'',
     it defaults to `'r''. Here is a full list of mode combinations:

     mode                   action
     -------------------------------------------------------------------------
     `'r' or 'r:*''         Open for reading with transparent compression
                            (recommended).
     `'r:''                 Open for reading exclusively without compression.
     `'r:gz''               Open for reading with gzip compression.
     `'r:bz2''              Open for reading with bzip2 compression.
     `'a' or 'a:''          Open for appending with no compression. The file
                            is created if it does not exist.
     `'w' or 'w:''          Open for uncompressed writing.
     `'w:gz''               Open for gzip compressed writing.
     `'w:bz2''              Open for bzip2 compressed writing.

     Note that `'a:gz'' or `'a:bz2'' is not possible. If _mode_ is not
     suitable to open a certain (compressed) file for reading, see
     ReadError is raised. Use _mode_ `'r'' to avoid this.  If a
     compression method is not supported, see CompressionError.
     is raised.

     If _fileobj_ is specified, it is used as an alternative to a file
     object opened for _name_. It is supposed to be at position 0.

     For special purposes, there is a second format for _mode_:
     `'filemode|[compression]''.  tarfile.open() will
     return a TarFile.  object that processes its data as a
     stream of blocks.  No random seeking will be done on the file. If
     given, _fileobj_ may be any object that has a `read()' or
     `write()' method (depending on the _mode_). _bufsize_ specifies
     the blocksize and defaults to `20 * 512' bytes. Use this variant
     in combination with e.g. `sys.stdin', a socket file object or a
     tape device. However, such a TarFile object is limited
     in that it does not allow to be accessed randomly, see
     Examples.  The currently possible modes:

     Mode              Action
     -------------------------------------------------------------------
     `'r|*''           Open a _stream_ of tar blocks for reading with
                       transparent compression.
     `'r|''            Open a _stream_ of uncompressed tar blocks for
                       reading.
     `'r|gz''          Open a gzip compressed _stream_ for reading.
     `'r|bz2''         Open a bzip2 compressed _stream_ for reading.
     `'w|''            Open an uncompressed _stream_ for writing.
     `'w|gz''          Open an gzip compressed _stream_ for writing.
     `'w|bz2''         Open an bzip2 compressed _stream_ for writing.

 -- Class: tarfile.TarFile
     Class for reading and writing tar archives. Do not use this class
     directly, better use tarfile.open() instead. See
     TarFile Objects.

 -- Function: tarfile.is_tarfile (name)
     Return True if _name_ is a tar archive file, that the
     tarfile.  module can read.

 -- Class: tarfile.TarFileCompat (filename, mode='r',
          compression=TAR_PLAIN)
     Class for limited access to tar archives with a zipfile.
     -like interface.  Please consult the documentation of the
     zipfile module for more details.  _compression_ must
     be one of the following constants:

      -- Data: TAR_PLAIN
          Constant for an uncompressed tar archive.

      -- Data: TAR_GZIPPED
          Constant for a gzip compressed tar archive.

     Deprecated since version 2.6: The TarFileCompat class
     has been deprecated for removal in Python 3.0.

 -- Exception: tarfile.TarError
     Base class for all tarfile exceptions.

 -- Exception: tarfile.ReadError
     Is raised when a tar archive is opened, that either cannot be
     handled by the tarfile module or is somehow invalid.

 -- Exception: tarfile.CompressionError
     Is raised when a compression method is not supported or when the
     data cannot be decoded properly.

 -- Exception: tarfile.StreamError
     Is raised for the limitations that are typical for stream-like
     TarFile.  objects.

 -- Exception: tarfile.ExtractError
     Is raised for _non-fatal_ errors when using
     TarFile.extract(), but only if `TarFile.errorlevel'`== 2'.

 -- Exception: tarfile.HeaderError
     Is raised by TarInfo.frombuf() if the buffer it gets
     is invalid.

     New in version 2.6.

  Each of the following constants defines a tar archive format that the
tarfile module is able to create. See section
Supported tar formats for details.

 -- Data: tarfile.USTAR_FORMAT
     POSIX.1-1988 (ustar) format.

 -- Data: tarfile.GNU_FORMAT
     GNU tar format.

 -- Data: tarfile.PAX_FORMAT
     POSIX.1-2001 (pax) format.

 -- Data: tarfile.DEFAULT_FORMAT
     The default format for creating archives. This is currently
     GNU_FORMAT.

  The following variables are available on module level:

 -- Data: tarfile.ENCODING
     The default character encoding i.e. the value from either
     sys.getfilesystemencoding() or
     sys.getdefaultencoding().

Module zipfile.
     Documentation of the zipfile standard module.

GNU tar manual, Basic Tar Format (http://www.gnu.org/software/tar/manual/html_node/Standard.html)
     Documentation for tar archive files, including GNU tar extensions.

5.12.5.1 TarFile Objects
........................

The TarFile object provides an interface to a tar archive.
A tar archive is a sequence of blocks. An archive member (a stored
file) is made up of a header block followed by data blocks. It is
possible to store a file in a tar archive several times. Each archive
member is represented by a TarInfo.  object, see
TarInfo Objects for details.

  A TarFile object can be used as a context manager in a
with.  statement. It will automatically be closed when the
block is completed. Please note that in the event of an exception an
archive opened for writing will not be finalized; only the internally
used file object will be closed. See the Examples section
for a use case.

  New in version 2.7: Added support for the context manager protocol.

 -- Class: tarfile.TarFile (name=None, mode='r', fileobj=None,
          format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False,
          ignore_zeros=False, encoding=ENCODING, errors=None,
          pax_headers=None, debug=0, errorlevel=0)
     All following arguments are optional and can be accessed as
     instance attributes as well.

     _name_ is the pathname of the archive. It can be omitted if
     _fileobj_ is given.  In this case, the file object's `name'
     attribute is used if it exists.

     _mode_ is either `'r'' to read from an existing archive, `'a'' to
     append data to an existing file or `'w'' to create a new file
     overwriting an existing one.

     If _fileobj_ is given, it is used for reading or writing data. If
     it can be determined, _mode_ is overridden by _fileobj_'s mode.
     _fileobj_ will be used from position 0.

          Note: _fileobj_ is not closed, when TarFile is
          closed.

     _format_ controls the archive format. It must be one of the
     constants USTAR_FORMAT, GNU_FORMAT or
     PAX_FORMAT that are defined at module level.

     New in version 2.6.

     The _tarinfo_ argument can be used to replace the default
     TarInfo class with a different one.

     New in version 2.6.

     If _dereference_ is False, add symbolic and hard links
     to the archive. If it is True, add the content of the
     target files to the archive. This has no effect on systems that do
     not support symbolic links.

     If _ignore_zeros_ is False, treat an empty block as the
     end of the archive.  If it is True, skip empty (and
     invalid) blocks and try to get as many members as possible. This
     is only useful for reading concatenated or damaged archives.

     _debug_ can be set from `0' (no debug messages) up to `3' (all
     debug messages). The messages are written to `sys.stderr'.

     If _errorlevel_ is `0', all errors are ignored when using
     TarFile.extract().  Nevertheless, they appear as error
     messages in the debug output, when debugging is enabled.  If `1',
     all _fatal_ errors are raised as OSError or
     IOError exceptions. If `2', all _non-fatal_ errors are
     raised as TarError exceptions as well.

     The _encoding_ and _errors_ arguments control the way strings are
     converted to unicode objects and vice versa. The default settings
     will work for most users.  See section Unicode issues.
     for in-depth information.

     New in version 2.6.

     The _pax_headers_ argument is an optional dictionary of unicode
     strings which will be added as a pax global header if _format_ is
     PAX_FORMAT.

     New in version 2.6.

 -- Method: TarFile.open (...)
     Alternative constructor. The tarfile.open() function
     is actually a shortcut to this classmethod.

 -- Method: TarFile.getmember (name)
     Return a TarInfo object for member _name_. If _name_
     can not be found in the archive, see KeyError is raised.

          Note: If a member occurs more than once in the archive, its
          last occurrence is assumed to be the most up-to-date version.

 -- Method: TarFile.getmembers ()
     Return the members of the archive as a list of TarInfo.
     objects. The list has the same order as the members in the archive.

 -- Method: TarFile.getnames ()
     Return the members as a list of their names. It has the same order
     as the list returned by getmembers().

 -- Method: TarFile.list (verbose=True)
     Print a table of contents to `sys.stdout'. If _verbose_ is
     False, only the names of the members are printed. If it is
     True, output similar to that of *ls -l* is produced.

 -- Method: TarFile.next ()
     Return the next member of the archive as a TarInfo.
     object, when TarFile is opened for reading. Return
     None if there is no more available.

 -- Method: TarFile.extractall (path=".", members=None)
     Extract all members from the archive to the current working
     directory or directory _path_. If optional _members_ is given, it
     must be a subset of the list returned by getmembers().
     Directory information like owner, modification time and
     permissions are set after all members have been extracted.  This
     is done to work around two problems: A directory's modification
     time is reset each time a file is created in it. And, if a
     directory's permissions do not allow writing, extracting files to
     it will fail.

          Warning: Never extract archives from untrusted sources
          without prior inspection.  It is possible that files are
          created outside of _path_, e.g. members that have absolute
          filenames starting with `"/"' or filenames with two dots
          `".."'.

     New in version 2.5.

 -- Method: TarFile.extract (member, path="")
     Extract a member from the archive to the current working
     directory, using its full name. Its file information is extracted
     as accurately as possible. _member_ may be a filename or a
     TarInfo object. You can specify a different directory using
     _path_.

          Note: The extract() method does not take care of
          several extraction issues.  In most cases you should consider
          using the extractall() method.

          Warning: See the warning for extractall().

 -- Method: TarFile.extractfile (member)
     Extract a member from the archive as a file object. _member_ may
     be a filename or a TarInfo object. If _member_ is a
     regular file, a file-like object is returned. If _member_ is a
     link, a file-like object is constructed from the link's target. If
     _member_ is none of the above, see None is returned.

          Note: The file-like object is read-only.  It provides the
          methods `read()', see readline(), `readlines()',
          `seek()', `tell()', and close(), and also supports
          iteration over its lines.

 -- Method: TarFile.add (name, arcname=None, recursive=True,
          exclude=None, filter=None)
     Add the file _name_ to the archive. _name_ may be any type of file
     (directory, fifo, symbolic link, etc.). If given, _arcname_
     specifies an alternative name for the file in the archive.
     Directories are added recursively by default. This can be avoided
     by setting _recursive_ to False. If _exclude_ is given
     it must be a function that takes one filename argument and returns
     a boolean value. Depending on this value the respective file is
     either excluded (see True) or added (see False).
     If _filter_ is specified it must be a function that takes a
     TarInfo object argument and returns the changed
     TarInfo object. If it instead returns None the
     TarInfo.  object will be excluded from the archive. See
     Examples for an example.

     Changed in version 2.6: Added the _exclude_ parameter.

     Changed in version 2.7: Added the _filter_ parameter.

     Deprecated since version 2.7: The _exclude_ parameter is
     deprecated, please use the _filter_ parameter instead.

 -- Method: TarFile.addfile (tarinfo, fileobj=None)
     Add the TarInfo object _tarinfo_ to the archive. If
     _fileobj_ is given, `tarinfo.size' bytes are read from it and
     added to the archive.  You can create TarInfo objects
     using gettarinfo().

          Note: On Windows platforms, _fileobj_ should always be opened
          with mode `'rb'' to avoid irritation about the file size.

 -- Method: TarFile.gettarinfo (name=None, arcname=None, fileobj=None)
     Create a TarInfo object for either the file _name_ or
     the file object _fileobj_ (using os.fstat() on its
     file descriptor).  You can modify some of the TarInfo.'
     s attributes before you add it using addfile().
     If given, _arcname_ specifies an alternative name for the file in
     the archive.

 -- Method: TarFile.close ()
     Close the TarFile. In write mode, two finishing zero
     blocks are appended to the archive.

 -- Attribute: TarFile.posix
     Setting this to True is equivalent to setting the
     format.  attribute to USTAR_FORMAT,
     False is equivalent to GNU_FORMAT.

     Changed in version 2.4: _posix_ defaults to False.

     Deprecated since version 2.6: Use the format attribute
     instead.

 -- Attribute: TarFile.pax_headers
     A dictionary containing key-value pairs of pax global headers.

     New in version 2.6.

5.12.5.2 TarInfo Objects
........................

A TarInfo object represents one member in a TarFile.
Aside from storing all required attributes of a file (like file
type, size, time, permissions, owner etc.), it provides some useful
methods to determine its type.  It does _not_ contain the file's data
itself.

  TarInfo objects are returned by TarFile.'s
methods `getmember()', `getmembers()' and `gettarinfo()'.

 -- Class: tarfile.TarInfo (name="")
     Create a TarInfo object.

 -- Method: TarInfo.frombuf (buf)
     Create and return a TarInfo object from string buffer
     _buf_.

     New in version 2.6: Raises HeaderError if the buffer
     is invalid..

 -- Method: TarInfo.fromtarfile (tarfile)
     Read the next member from the TarFile object _tarfile_
     and return it as a TarInfo object.

     New in version 2.6.

 -- Method: TarInfo.tobuf (format=DEFAULT_FORMAT, encoding=ENCODING,
          errors='strict')
     Create a string buffer from a TarInfo object. For
     information on the arguments see the constructor of the
     TarFile class.

     Changed in version 2.6: The arguments were added.

  A `TarInfo' object has the following public data attributes:

 -- Attribute: TarInfo.name
     Name of the archive member.

 -- Attribute: TarInfo.size
     Size in bytes.

 -- Attribute: TarInfo.mtime
     Time of last modification.

 -- Attribute: TarInfo.mode
     Permission bits.

 -- Attribute: TarInfo.type
     File type.  _type_ is usually one of these constants: `REGTYPE',
     `AREGTYPE', `LNKTYPE', `SYMTYPE', `DIRTYPE', `FIFOTYPE',
     `CONTTYPE', `CHRTYPE', `BLKTYPE', `GNUTYPE_SPARSE'.  To determine
     the type of a TarInfo object more conveniently, use
     the `is_*()' methods below.

 -- Attribute: TarInfo.linkname
     Name of the target file name, which is only present in
     TarInfo objects of type `LNKTYPE' and `SYMTYPE'.

 -- Attribute: TarInfo.uid
     User ID of the user who originally stored this member.

 -- Attribute: TarInfo.gid
     Group ID of the user who originally stored this member.

 -- Attribute: TarInfo.uname
     User name.

 -- Attribute: TarInfo.gname
     Group name.

 -- Attribute: TarInfo.pax_headers
     A dictionary containing key-value pairs of an associated pax
     extended header.

     New in version 2.6.

  A TarInfo object also provides some convenient query
methods:

 -- Method: TarInfo.isfile ()
     Return True if the `Tarinfo' object is a regular file.

 -- Method: TarInfo.isreg ()
     Same as isfile().

 -- Method: TarInfo.isdir ()
     Return True if it is a directory.

 -- Method: TarInfo.issym ()
     Return True if it is a symbolic link.

 -- Method: TarInfo.islnk ()
     Return True if it is a hard link.

 -- Method: TarInfo.ischr ()
     Return True if it is a character device.

 -- Method: TarInfo.isblk ()
     Return True if it is a block device.

 -- Method: TarInfo.isfifo ()
     Return True if it is a FIFO.

 -- Method: TarInfo.isdev ()
     Return True if it is one of character device, block
     device or FIFO.

5.12.5.3 Examples
.................

How to extract an entire tar archive to the current working directory:

    import tarfile
    tar = tarfile.open("sample.tar.gz")
    tar.extractall()
    tar.close()

How to extract a subset of a tar archive with
TarFile.extractall() using a generator function instead of a list:

    import os
    import tarfile

    def py_files(members):
        for tarinfo in members:
            if os.path.splitext(tarinfo.name)[1] == ".py":
                yield tarinfo

    tar = tarfile.open("sample.tar.gz")
    tar.extractall(members=py_files(tar))
    tar.close()

How to create an uncompressed tar archive from a list of filenames:

    import tarfile
    tar = tarfile.open("sample.tar", "w")
    for name in ["foo", "bar", "quux"]:
        tar.add(name)
    tar.close()

The same example using the with statement:

    import tarfile
    with tarfile.open("sample.tar", "w") as tar:
        for name in ["foo", "bar", "quux"]:
            tar.add(name)

How to read a gzip compressed tar archive and display some member
information:

    import tarfile
    tar = tarfile.open("sample.tar.gz", "r:gz")
    for tarinfo in tar:
        print tarinfo.name, "is", tarinfo.size, "bytes in size and is",
        if tarinfo.isreg():
            print "a regular file."
        elif tarinfo.isdir():
            print "a directory."
        else:
            print "something else."
    tar.close()

How to create an archive and reset the user information using the
_filter_ parameter in TarFile.add().:

    import tarfile
    def reset(tarinfo):
        tarinfo.uid = tarinfo.gid = 0
        tarinfo.uname = tarinfo.gname = "root"
        return tarinfo
    tar = tarfile.open("sample.tar.gz", "w:gz")
    tar.add("foo", filter=reset)
    tar.close()

5.12.5.4 Supported tar formats
..............................

There are three tar formats that can be created with the tarfile
module:

   * The POSIX.1-1988 ustar format (see USTAR_FORMAT). It
     supports filenames up to a length of at best 256 characters and
     linknames up to 100 characters. The maximum file size is 8
     gigabytes. This is an old and limited but widely supported format.

   * The GNU tar format (see GNU_FORMAT). It supports long
     filenames and linknames, files bigger than 8 gigabytes and sparse
     files. It is the de facto standard on GNU/Linux systems.
     tarfile fully supports the GNU tar extensions for long
     names, sparse file support is read-only.

   * The POSIX.1-2001 pax format (see PAX_FORMAT). It is the
     most flexible format with virtually no limits. It supports long
     filenames and linknames, large files and stores pathnames in a
     portable way. However, not all tar implementations today are able
     to handle pax archives properly.

     The _pax_ format is an extension to the existing _ustar_ format.
     It uses extra headers for information that cannot be stored
     otherwise. There are two flavours of pax headers: Extended headers
     only affect the subsequent file header, global headers are valid
     for the complete archive and affect all following files. All the
     data in a pax header is encoded in _UTF-8_ for portability reasons.

  There are some more variants of the tar format which can be read, but
not created:

   * The ancient V7 format. This is the first tar format from Unix
     Seventh Edition, storing only regular files and directories. Names
     must not be longer than 100 characters, there is no user/group
     name information. Some archives have miscalculated header
     checksums in case of fields with non-ASCII characters.

   * The SunOS tar extended format. This format is a variant of the
     POSIX.1-2001 pax format, but is not compatible.

5.12.5.5 Unicode issues
.......................

The tar format was originally conceived to make backups on tape drives
with the main focus on preserving file system information. Nowadays tar
archives are commonly used for file distribution and exchanging
archives over networks. One problem of the original format (that all
other formats are merely variants of) is that there is no concept of
supporting different character encodings. For example, an ordinary tar
archive created on a _UTF-8_ system cannot be read correctly on a
_Latin-1_ system if it contains non-ASCII characters. Names (i.e.
filenames, linknames, user/group names) containing these characters
will appear damaged.  Unfortunately, there is no way to autodetect the
encoding of an archive.

  The pax format was designed to solve this problem. It stores
non-ASCII names using the universal character encoding _UTF-8_. When a
pax archive is read, these _UTF-8_ names are converted to the encoding
of the local file system.

  The details of unicode conversion are controlled by the _encoding_
and _errors_ keyword arguments of the TarFile class.

  The default value for _encoding_ is the local character encoding. It
is deduced from sys.getfilesystemencoding() and
sys.getdefaultencoding(). In read mode, _encoding_ is used
exclusively to convert unicode names from a pax archive to strings in
the local character encoding. In write mode, the use of _encoding_
depends on the chosen archive format. In case of PAX_FORMAT,
input names that contain non-ASCII characters need to be decoded before
being stored as _UTF-8_ strings. The other formats do not make use of
_encoding_ unless unicode objects are used as input names. These are
converted to 8-bit character strings before they are added to the
archive.

  The _errors_ argument defines how characters are treated that cannot
be converted to or from _encoding_. Possible values are listed in
section Codec Base Classes. In read mode, there is an
additional scheme `'utf-8'' which means that bad characters are
replaced by their _UTF-8_ representation. This is the default scheme.
In write mode the default value for _errors_ is `'strict'' to ensure
that name information is not altered unnoticed.

5.13 File Formats
=================

The modules described in this chapter parse various miscellaneous file
formats that aren't markup languages or are related to e-mail.

5.13.1 `csv' -- CSV File Reading and Writing
--------------------------------------------

New in version 2.3.

  The so-called CSV (Comma Separated Values) format is the most common
import and export format for spreadsheets and databases.  There is no
"CSV standard", so the format is operationally defined by the many
applications which read and write it.  The lack of a standard means
that subtle differences often exist in the data produced and consumed
by different applications.  These differences can make it annoying to
process CSV files from multiple sources.  Still, while the delimiters
and quoting characters vary, the overall format is similar enough that
it is possible to write a single module which can efficiently manipulate
such data, hiding the details of reading and writing the data from the
programmer.

  The csv module implements classes to read and write
tabular data in CSV format.  It allows programmers to say, "write this
data in the format preferred by Excel," or "read data from this file
which was generated by Excel," without knowing the precise details of
the CSV format used by Excel.  Programmers can also describe the CSV
formats understood by other applications or define their own
special-purpose CSV formats.

  The csv module's reader and writer.
objects read and write sequences.  Programmers can also read and write
data in dictionary form using the DictReader and
DictWriter classes.

     Note: This version of the csv module doesn't support
     Unicode input.  Also, there are currently some issues regarding
     ASCII NUL characters.  Accordingly, all input should be UTF-8 or
     printable ASCII to be safe; see the examples in section
     Examples. These restrictions will be removed in the future.

PEP 305 (http://www.python.org/dev/peps/pep-0305) - CSV File API
     The Python Enhancement Proposal which proposed this addition to
     Python.

5.13.1.1 Module Contents
........................

The csv module defines the following functions:

 -- Function: csv.reader (csvfile[, dialect='excel'][, fmtparam])
     Return a reader object which will iterate over lines in the given
     _csvfile_.  _csvfile_ can be any object which supports the
     iterator protocol and returns a string each time its
     `next()' method is called -- file objects and list objects are
     both suitable.   If _csvfile_ is a file object, it must be opened
     with the 'b' flag on platforms where that makes a difference.  An
     optional _dialect_ parameter can be given which is used to define
     a set of parameters specific to a particular CSV dialect.  It may
     be an instance of a subclass of the Dialect class or
     one of the strings returned by the list_dialects().
     function.  The other optional _fmtparam_ keyword arguments can be
     given to override individual formatting parameters in the current
     dialect.  For full details about the dialect and formatting
     parameters, see section Dialects and Formatting Parameters.

     Each row read from the csv file is returned as a list of strings.
     No automatic data type conversion is performed.

     A short usage example:

         >>> import csv
         >>> spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|')
         >>> for row in spamReader:
         ...     print ', '.join(row)
         Spam, Spam, Spam, Spam, Spam, Baked Beans
         Spam, Lovely Spam, Wonderful Spam

     Changed in version 2.5: The parser is now stricter with respect to
     multi-line quoted fields. Previously, if a line ended within a
     quoted field without a terminating newline character, a newline
     would be inserted into the returned field. This behavior caused
     problems when reading files which contained carriage return
     characters within fields.  The behavior was changed to return the
     field without inserting newlines. As a consequence, if newlines
     embedded within fields are important, the input should be split
     into lines in a manner which preserves the newline characters.

 -- Function: csv.writer (csvfile[, dialect='excel'][, fmtparam])
     Return a writer object responsible for converting the user's data
     into delimited strings on the given file-like object.  _csvfile_
     can be any object with a `write()' method.  If _csvfile_ is a file
     object, it must be opened with the 'b' flag on platforms where
     that makes a difference.  An optional _dialect_ parameter can be
     given which is used to define a set of parameters specific to a
     particular CSV dialect.  It may be an instance of a subclass of the
     Dialect class or one of the strings returned by the
     list_dialects() function.  The other optional
     _fmtparam_ keyword arguments can be given to override individual
     formatting parameters in the current dialect.  For full details
     about the dialect and formatting parameters, see section
     Dialects and Formatting Parameters. To make it as easy as
     possible to interface with modules which implement the DB API, the
     value None is written as the empty string.  While this
     isn't a reversible transformation, it makes it easier to dump SQL
     NULL data values to CSV files without preprocessing the data
     returned from a `cursor.fetch*' call.  All other non-string data
     are stringified with str() before being written.

     A short usage example:

         >>> import csv
         >>> spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',
         ...                         quotechar='|', quoting=csv.QUOTE_MINIMAL)
         >>> spamWriter.writerow(['Spam'] * 5 + ['Baked Beans'])
         >>> spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])

 -- Function: csv.register_dialect (name[, dialect][, fmtparam])
     Associate _dialect_ with _name_.  _name_ must be a string or
     Unicode object. The dialect can be specified either by passing a
     sub-class of Dialect, or by _fmtparam_ keyword
     arguments, or both, with keyword arguments overriding parameters
     of the dialect. For full details about the dialect and formatting
     parameters, see section Dialects and Formatting Parameters.

 -- Function: csv.unregister_dialect (name)
     Delete the dialect associated with _name_ from the dialect
     registry.  An Error is raised if _name_ is not a
     registered dialect name.

 -- Function: csv.get_dialect (name)
     Return the dialect associated with _name_.  An Error.
     is raised if _name_ is not a registered dialect name.

     Changed in version 2.5: This function now returns an immutable
     Dialect.  Previously an instance of the requested
     dialect was returned.  Users could modify the underlying class,
     changing the behavior of active readers and writers.

 -- Function: csv.list_dialects ()
     Return the names of all registered dialects.

 -- Function: csv.field_size_limit ([new_limit])
     Returns the current maximum field size allowed by the parser. If
     _new_limit_ is given, this becomes the new limit.

     New in version 2.5.

  The csv module defines the following classes:

 -- Class: csv.DictReader (csvfile[, fieldnames=None[, restkey=None[,
          restval=None[, dialect='excel'[, *args, **kwds]]]]])
     Create an object which operates like a regular reader but maps the
     information read into a dict whose keys are given by the optional
     _fieldnames_ parameter.  If the _fieldnames_ parameter is omitted,
     the values in the first row of the _csvfile_ will be used as the
     fieldnames.  If the row read has more fields than the fieldnames
     sequence, the remaining data is added as a sequence keyed by the
     value of _restkey_.  If the row read has fewer fields than the
     fieldnames sequence, the remaining keys take the value of the
     optional _restval_ parameter.  Any other optional or keyword
     arguments are passed to the underlying reader instance.

 -- Class: csv.DictWriter (csvfile, fieldnames[, restval="[,
          extrasaction='raise'[, dialect='excel'[, *args, **kwds]]]])
     Create an object which operates like a regular writer but maps
     dictionaries onto output rows.  The _fieldnames_ parameter
     identifies the order in which values in the dictionary passed to
     the `writerow()' method are written to the _csvfile_.  The
     optional _restval_ parameter specifies the value to be written if
     the dictionary is missing a key in _fieldnames_.  If the
     dictionary passed to the `writerow()' method contains a key not
     found in _fieldnames_, the optional _extrasaction_ parameter
     indicates what action to take.  If it is set to `'raise'' a
     ValueError is raised.  If it is set to `'ignore'', extra
     values in the dictionary are ignored.  Any other optional or
     keyword arguments are passed to the underlying writer.
     instance.

     Note that unlike the DictReader class, the
     _fieldnames_ parameter of the DictWriter is not
     optional.  Since Python's dict objects are not
     ordered, there is not enough information available to deduce the
     order in which the row should be written to the _csvfile_.

 -- Class: csv.Dialect
     The Dialect class is a container class relied on
     primarily for its attributes, which are used to define the
     parameters for a specific reader or writer.
     instance.

 -- Class: csv.excel
     The excel class defines the usual properties of an
     Excel-generated CSV file.  It is registered with the dialect name
     `'excel''.

 -- Class: csv.excel_tab
     The excel_tab class defines the usual properties of an
     Excel-generated TAB-delimited file.  It is registered with the
     dialect name `'excel-tab''.

 -- Class: csv.Sniffer
     The Sniffer class is used to deduce the format of a
     CSV file.

     The Sniffer class provides two methods:

      -- Method: sniff (sample[, delimiters=None])
          Analyze the given _sample_ and return a Dialect.
          subclass reflecting the parameters found.  If the optional
          _delimiters_ parameter is given, it is interpreted as a
          string containing possible valid delimiter characters.

      -- Method: has_header (sample)
          Analyze the sample text (presumed to be in CSV format) and
          return True if the first row appears to be a
          series of column headers.

  An example for Sniffer use:

    csvfile = open("example.csv", "rb")
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)
    # ... process CSV file contents here ...

The csv module defines the following constants:

 -- Data: csv.QUOTE_ALL
     Instructs writer objects to quote all fields.

 -- Data: csv.QUOTE_MINIMAL
     Instructs writer objects to only quote those fields
     which contain special characters such as _delimiter_, _quotechar_
     or any of the characters in _lineterminator_.

 -- Data: csv.QUOTE_NONNUMERIC
     Instructs writer objects to quote all non-numeric
     fields.

     Instructs the reader to convert all non-quoted fields to type
     _float_.

 -- Data: csv.QUOTE_NONE
     Instructs writer objects to never quote fields.  When
     the current _delimiter_ occurs in output data it is preceded by
     the current _escapechar_ character.  If _escapechar_ is not set,
     the writer will raise Error if any characters that
     require escaping are encountered.

     Instructs reader to perform no special processing of
     quote characters.

  The csv module defines the following exception:

 -- Exception: csv.Error
     Raised by any of the functions when an error is detected.

5.13.1.2 Dialects and Formatting Parameters
...........................................

To make it easier to specify the format of input and output records,
specific formatting parameters are grouped together into dialects.  A
dialect is a subclass of the Dialect class having a set of
specific methods and a single `validate()' method.  When creating
reader or writer objects, the programmer can specify
a string or a subclass of the Dialect class as the dialect
parameter.  In addition to, or instead of, the _dialect_ parameter, the
programmer can also specify individual formatting parameters, which
have the same names as the attributes defined below for the
Dialect class.

  Dialects support the following attributes:

 -- Attribute: Dialect.delimiter
     A one-character string used to separate fields.  It defaults to
     `',''.

 -- Attribute: Dialect.doublequote
     Controls how instances of _quotechar_ appearing inside a field
     should be themselves be quoted.  When True, the
     character is doubled. When False, the _escapechar_ is
     used as a prefix to the _quotechar_.  It defaults to True.

     On output, if _doublequote_ is False and no
     _escapechar_ is set, see Error is raised if a _quotechar_
     is found in a field.

 -- Attribute: Dialect.escapechar
     A one-character string used by the writer to escape the
     _delimiter_ if _quoting_ is set to QUOTE_NONE and the
     _quotechar_ if _doublequote_ is False. On reading, the
     _escapechar_ removes any special meaning from the following
     character. It defaults to None, which disables escaping.

 -- Attribute: Dialect.lineterminator
     The string used to terminate lines produced by the writer.
     It defaults to `'\r\n''.

          Note: The reader is hard-coded to recognise
          either `'\r'' or `'\n'' as end-of-line, and ignores
          _lineterminator_. This behavior may change in the future.

 -- Attribute: Dialect.quotechar
     A one-character string used to quote fields containing special
     characters, such as the _delimiter_ or _quotechar_, or which
     contain new-line characters.  It defaults to `'"''.

 -- Attribute: Dialect.quoting
     Controls when quotes should be generated by the writer and
     recognised by the reader.  It can take on any of the `QUOTE_*'
     constants (see section Module Contents) and defaults
     to QUOTE_MINIMAL.

 -- Attribute: Dialect.skipinitialspace
     When True, whitespace immediately following the
     _delimiter_ is ignored.  The default is False.

5.13.1.3 Reader Objects
.......................

Reader objects (see DictReader instances and objects returned
by the reader() function) have the following public methods:

 -- Method: csvreader.next ()
     Return the next row of the reader's iterable object as a list,
     parsed according to the current dialect.

  Reader objects have the following public attributes:

 -- Attribute: csvreader.dialect
     A read-only description of the dialect in use by the parser.

 -- Attribute: csvreader.line_num
     The number of lines read from the source iterator. This is not the
     same as the number of records returned, as records can span
     multiple lines.

     New in version 2.5.

  DictReader objects have the following public attribute:

 -- Attribute: csvreader.fieldnames
     If not passed as a parameter when creating the object, this
     attribute is initialized upon first access or when the first
     record is read from the file.

     Changed in version 2.6.

5.13.1.4 Writer Objects
.......................

`Writer' objects (see DictWriter instances and objects returned
by the writer() function) have the following public
methods.  A _row_ must be a sequence of strings or numbers for `Writer'
objects and a dictionary mapping fieldnames to strings or numbers (by
passing them through str().  first) for DictWriter
objects.  Note that complex numbers are written out surrounded by
parens. This may cause some problems for other programs which read CSV
files (assuming they support complex numbers at all).

 -- Method: csvwriter.writerow (row)
     Write the _row_ parameter to the writer's file object, formatted
     according to the current dialect.

 -- Method: csvwriter.writerows (rows)
     Write all the _rows_ parameters (a list of _row_ objects as
     described above) to the writer's file object, formatted according
     to the current dialect.

  Writer objects have the following public attribute:

 -- Attribute: csvwriter.dialect
     A read-only description of the dialect in use by the writer.

  DictWriter objects have the following public method:

 -- Method: DictWriter.writeheader ()
     Write a row with the field names (as specified in the constructor).

     New in version 2.7.

5.13.1.5 Examples
.................

The simplest example of reading a CSV file:

    import csv
    reader = csv.reader(open("some.csv", "rb"))
    for row in reader:
        print row

Reading a file with an alternate format:

    import csv
    reader = csv.reader(open("passwd", "rb"), delimiter=':', quoting=csv.QUOTE_NONE)
    for row in reader:
        print row

The corresponding simplest possible writing example is:

    import csv
    writer = csv.writer(open("some.csv", "wb"))
    writer.writerows(someiterable)

Registering a new dialect:

    import csv

    csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE)

    reader = csv.reader(open("passwd", "rb"), 'unixpwd')

A slightly more advanced use of the reader -- catching and reporting
errors:

    import csv, sys
    filename = "some.csv"
    reader = csv.reader(open(filename, "rb"))
    try:
        for row in reader:
            print row
    except csv.Error, e:
        sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e))

And while the module doesn't directly support parsing strings, it can
easily be done:

    import csv
    for row in csv.reader(['one,two,three']):
        print row

The csv module doesn't directly support reading and writing
Unicode, but it is 8-bit-clean save for some problems with ASCII NUL
characters.  So you can write functions or classes that handle the
encoding and decoding for you as long as you avoid encodings like
UTF-16 that use NULs.  UTF-8 is recommended.

  `unicode_csv_reader()' below is a generator that wraps
csv.reader.  to handle Unicode CSV data (a list of Unicode
strings).  `utf_8_encoder()' is a generator that encodes
the Unicode strings as UTF-8, one string (or row) at a time.  The
encoded strings are parsed by the CSV reader, and
`unicode_csv_reader()' decodes the UTF-8-encoded cells back into
Unicode:

    import csv

    def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs):
        # csv.py doesn't do Unicode; encode temporarily as UTF-8:
        csv_reader = csv.reader(utf_8_encoder(unicode_csv_data),
                                dialect=dialect, **kwargs)
        for row in csv_reader:
            # decode UTF-8 back to Unicode, cell by cell:
            yield [unicode(cell, 'utf-8') for cell in row]

    def utf_8_encoder(unicode_csv_data):
        for line in unicode_csv_data:
            yield line.encode('utf-8')

For all other encodings the following `UnicodeReader' and
`UnicodeWriter' classes can be used. They take an additional _encoding_
parameter in their constructor and make sure that the data passes the
real reader or writer encoded as UTF-8:

    import csv, codecs, cStringIO

    class UTF8Recoder:
        """
        Iterator that reads an encoded stream and reencodes the input to UTF-8
        """
        def __init__(self, f, encoding):
            self.reader = codecs.getreader(encoding)(f)

        def __iter__(self):
            return self

        def next(self):
            return self.reader.next().encode("utf-8")

    class UnicodeReader:
        """
        A CSV reader which will iterate over lines in the CSV file "f",
        which is encoded in the given encoding.
        """

        def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
            f = UTF8Recoder(f, encoding)
            self.reader = csv.reader(f, dialect=dialect, **kwds)

        def next(self):
            row = self.reader.next()
            return [unicode(s, "utf-8") for s in row]

        def __iter__(self):
            return self

    class UnicodeWriter:
        """
        A CSV writer which will write rows to CSV file "f",
        which is encoded in the given encoding.
        """

        def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
            # Redirect output to a queue
            self.queue = cStringIO.StringIO()
            self.writer = csv.writer(self.queue, dialect=dialect, **kwds)
            self.stream = f
            self.encoder = codecs.getincrementalencoder(encoding)()

        def writerow(self, row):
            self.writer.writerow([s.encode("utf-8") for s in row])
            # Fetch UTF-8 output from the queue ...
            data = self.queue.getvalue()
            data = data.decode("utf-8")
            # ... and reencode it into the target encoding
            data = self.encoder.encode(data)
            # write to the target stream
            self.stream.write(data)
            # empty queue
            self.queue.truncate(0)

        def writerows(self, rows):
            for row in rows:
                self.writerow(row)

5.13.2 `ConfigParser' -- Configuration file parser
--------------------------------------------------

     Note: The ConfigParser module has been renamed to
     `configparser' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

This module defines the class ConfigParser.   The
ConfigParser.  class implements a basic configuration file parser
language which provides a structure similar to what you would find on
Microsoft Windows INI files.  You can use this to write Python programs
which can be customized by end users easily.

     Note: This library does _not_ interpret or write the value-type
     prefixes used in the Windows Registry extended version of INI
     syntax.

  The configuration file consists of sections, led by a `[section]'
header and followed by `name: value' entries, with continuations in the
style of RFC 822 (http://tools.ietf.org/html/rfc822.html) (see section
3.1.1, "LONG HEADER FIELDS"); `name=value' is also accepted.  Note that
leading whitespace is removed from values. The optional values can
contain format strings which refer to other values in the same section,
or values in a special `DEFAULT' section.  Additional defaults can be
provided on initialization and retrieval.  Lines beginning with `'#'' or
`';'' are ignored and may be used to provide comments.

  Configuration files may include comments, prefixed by specific
characters (`#' and `;').  Comments may appear on their own in an
otherwise empty line, or may be entered in lines holding values or
section names.  In the latter case, they need to be preceded by a
whitespace character to be recognized as a comment.  (For backwards
compatibility, only `;' starts an inline comment, while `#' does not.)

  On top of the core functionality, see SafeConfigParser.
supports interpolation.  This means values can contain format strings
which refer to other values in the same section, or values in a special
`DEFAULT' section.  Additional defaults can be provided on
initialization.

  For example:

    [My Section]
    foodir: %(dir)s/whatever
    dir=frob
    long: this value continues
       in the next line

would resolve the `%(dir)s' to the value of `dir' (`frob' in this case).
All reference expansions are done on demand.

  Default values can be specified by passing them into the
ConfigParser.  constructor as a dictionary.  Additional defaults
may be passed into the `get()' method which will override all others.

  Sections are normally stored in a built-in dictionary. An alternative
dictionary type can be passed to the ConfigParser.
constructor. For example, if a dictionary type is passed that sorts its
keys, the sections will be sorted on write-back, as will be the keys
within each section.

 -- Class: ConfigParser.RawConfigParser ([defaults[, dict_type[,
          allow_no_value]]])
     The basic configuration object.  When _defaults_ is given, it is
     initialized into the dictionary of intrinsic defaults.  When
     _dict_type_ is given, it will be used to create the dictionary
     objects for the list of sections, for the options within a
     section, and for the default values.  When _allow_no_value_ is
     true (default: `False'), options without values are accepted; the
     value presented for these is `None'.

     This class does not support the magical interpolation behavior.

     New in version 2.3.

     Changed in version 2.6: _dict_type_ was added.

     Changed in version 2.7: The default _dict_type_ is
     collections.OrderedDict.  _allow_no_value_ was added.

 -- Class: ConfigParser.ConfigParser ([defaults[, dict_type[,
          allow_no_value]]])
     Derived class of RawConfigParser that implements the
     magical interpolation feature and adds optional arguments to the
     get() and items() methods.  The values in
     _defaults_ must be appropriate for the `%()s' string
     interpolation.  Note that ___name___ is an intrinsic default; its
     value is the section name, and will override any value provided in
     _defaults_.

     All option names used in interpolation will be passed through the
     `optionxform()' method just like any other option name reference.
     For example, using the default implementation of `optionxform()'
     (which converts option names to lower case), the values `foo
     %(bar)s' and `foo %(BAR)s' are equivalent.

     New in version 2.3.

     Changed in version 2.6: _dict_type_ was added.

     Changed in version 2.7: The default _dict_type_ is
     collections.OrderedDict.  _allow_no_value_ was added.

 -- Class: ConfigParser.SafeConfigParser ([defaults[, dict_type[,
          allow_no_value]]])
     Derived class of ConfigParser that implements a
     more-sane variant of the magical interpolation feature.  This
     implementation is more predictable as well. New applications
     should prefer this version if they don't need to be compatible
     with older versions of Python.

     New in version 2.3.

     Changed in version 2.6: _dict_type_ was added.

     Changed in version 2.7: The default _dict_type_ is
     collections.OrderedDict.  _allow_no_value_ was added.

 -- Exception: ConfigParser.Error
     Base class for all other configparser exceptions.

 -- Exception: ConfigParser.NoSectionError
     Exception raised when a specified section is not found.

 -- Exception: ConfigParser.DuplicateSectionError
     Exception raised if `add_section()' is called with the name of a
     section that is already present.

 -- Exception: ConfigParser.NoOptionError
     Exception raised when a specified option is not found in the
     specified  section.

 -- Exception: ConfigParser.InterpolationError
     Base class for exceptions raised when problems occur performing
     string interpolation.

 -- Exception: ConfigParser.InterpolationDepthError
     Exception raised when string interpolation cannot be completed
     because the number of iterations exceeds
     MAX_INTERPOLATION_DEPTH. Subclass of
     InterpolationError.

 -- Exception: ConfigParser.InterpolationMissingOptionError
     Exception raised when an option referenced from a value does not
     exist. Subclass of InterpolationError.

     New in version 2.3.

 -- Exception: ConfigParser.InterpolationSyntaxError
     Exception raised when the source text into which substitutions are
     made does not conform to the required syntax. Subclass of
     InterpolationError.

     New in version 2.3.

 -- Exception: ConfigParser.MissingSectionHeaderError
     Exception raised when attempting to parse a file which has no
     section headers.

 -- Exception: ConfigParser.ParsingError
     Exception raised when errors occur attempting to parse a file.

 -- Data: ConfigParser.MAX_INTERPOLATION_DEPTH
     The maximum depth for recursive interpolation for `get()' when the
     _raw_ parameter is false.  This is relevant only for the
     ConfigParser class.

Module shlex.
     Support for a creating Unix shell-like mini-languages which can be
     used as an alternate format for application configuration files.

5.13.2.1 RawConfigParser Objects
................................

RawConfigParser instances have the following methods:

 -- Method: RawConfigParser.defaults ()
     Return a dictionary containing the instance-wide defaults.

 -- Method: RawConfigParser.sections ()
     Return a list of the sections available; `DEFAULT' is not included
     in the list.

 -- Method: RawConfigParser.add_section (section)
     Add a section named _section_ to the instance.  If a section by
     the given name already exists, see DuplicateSectionError.
     is raised. If the name `DEFAULT' (or any of it's case-insensitive
     variants) is passed, see ValueError is raised.

 -- Method: RawConfigParser.has_section (section)
     Indicates whether the named section is present in the
     configuration. The `DEFAULT' section is not acknowledged.

 -- Method: RawConfigParser.options (section)
     Returns a list of options available in the specified _section_.

 -- Method: RawConfigParser.has_option (section, option)
     If the given section exists, and contains the given option, return
     True.; otherwise return False.

     New in version 1.6.

 -- Method: RawConfigParser.read (filenames)
     Attempt to read and parse a list of filenames, returning a list of
     filenames which were successfully parsed.  If _filenames_ is a
     string or Unicode string, it is treated as a single filename. If a
     file named in _filenames_ cannot be opened, that file will be
     ignored.  This is designed so that you can specify a list of
     potential configuration file locations (for example, the current
     directory, the user's home directory, and some system-wide
     directory), and all existing configuration files in the list will
     be read.  If none of the named files exist, the
     ConfigParser instance will contain an empty dataset.  An
     application which requires initial values to be loaded from a file
     should load the required file or files using readfp().
     before calling read().  for any optional files:

         import ConfigParser, os

         config = ConfigParser.ConfigParser()
         config.readfp(open('defaults.cfg'))
         config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')])

     Changed in version 2.4: Returns list of successfully parsed
     filenames.

 -- Method: RawConfigParser.readfp (fp[, filename])
     Read and parse configuration data from the file or file-like
     object in _fp_ (only the readline() method is used).
     If _filename_ is omitted and _fp_ has a `name' attribute, that is
     used for _filename_; the default is `<???>'.

 -- Method: RawConfigParser.get (section, option)
     Get an _option_ value for the named _section_.

 -- Method: RawConfigParser.getint (section, option)
     A convenience method which coerces the _option_ in the specified
     _section_ to an integer.

 -- Method: RawConfigParser.getfloat (section, option)
     A convenience method which coerces the _option_ in the specified
     _section_ to a floating point number.

 -- Method: RawConfigParser.getboolean (section, option)
     A convenience method which coerces the _option_ in the specified
     _section_ to a Boolean value.  Note that the accepted values for
     the option are `"1"', `"yes"', `"true"', and `"on"', which cause
     this method to return `True', and `"0"', `"no"', `"false"', and
     `"off"', which cause it to return `False'.  These string values
     are checked in a case-insensitive manner.  Any other value will
     cause it to raise ValueError.

 -- Method: RawConfigParser.items (section)
     Return a list of `(name, value)' pairs for each option in the
     given _section_.

 -- Method: RawConfigParser.set (section, option, value)
     If the given section exists, set the given option to the specified
     value; otherwise raise NoSectionError.  While it is
     possible to use RawConfigParser. (or
     ConfigParser with _raw_ parameters set to true) for
     _internal_ storage of non-string values, full functionality
     (including interpolation and output to files) can only be achieved
     using string values.

     New in version 1.6.

 -- Method: RawConfigParser.write (fileobject)
     Write a representation of the configuration to the specified file
     object.  This representation can be parsed by a future
     read() call.

     New in version 1.6.

 -- Method: RawConfigParser.remove_option (section, option)
     Remove the specified _option_ from the specified _section_. If the
     section does not exist, raise NoSectionError.  If the
     option existed to be removed, return True.; otherwise
     return False.

     New in version 1.6.

 -- Method: RawConfigParser.remove_section (section)
     Remove the specified _section_ from the configuration. If the
     section in fact existed, return `True'. Otherwise return `False'.

 -- Method: RawConfigParser.optionxform (option)
     Transforms the option name _option_ as found in an input file or
     as passed in by client code to the form that should be used in the
     internal structures.  The default implementation returns a
     lower-case version of _option_; subclasses may override this or
     client code can set an attribute of this name on instances to
     affect this behavior.

     You don't necessarily need to subclass a ConfigParser to use this
     method, you can also re-set it on an instance, to a function that
     takes a string argument.  Setting it to `str', for example, would
     make option names case sensitive:

         cfgparser = ConfigParser()
         ...
         cfgparser.optionxform = str

     Note that when reading configuration files, whitespace around the
     option names are stripped before optionxform() is
     called.

5.13.2.2 ConfigParser Objects
.............................

The ConfigParser class extends some methods of the
RawConfigParser interface, adding some optional arguments.

 -- Method: ConfigParser.get (section, option[, raw[, vars]])
     Get an _option_ value for the named _section_.  If _vars_ is
     provided, it must be a dictionary.  The _option_ is looked up in
     _vars_ (if provided), _section_, and in _defaults_ in that order.

     All the `'%'' interpolations are expanded in the return values,
     unless the _raw_ argument is true.  Values for interpolation keys
     are looked up in the same manner as the option.

 -- Method: ConfigParser.items (section[, raw[, vars]])
     Return a list of `(name, value)' pairs for each option in the
     given _section_.  Optional arguments have the same meaning as for
     the get() method.

     New in version 2.3.

5.13.2.3 SafeConfigParser Objects
.................................

The SafeConfigParser class implements the same extended
interface as ConfigParser, with the following addition:

 -- Method: SafeConfigParser.set (section, option, value)
     If the given section exists, set the given option to the specified
     value; otherwise raise NoSectionError.  _value_ must be
     a string (see str.  or unicode); if not, see
     TypeError is raised.

     New in version 2.4.

5.13.2.4 Examples
.................

An example of writing to a configuration file:

    import ConfigParser

    config = ConfigParser.RawConfigParser()

    # When adding sections or items, add them in the reverse order of
    # how you want them to be displayed in the actual file.
    # In addition, please note that using RawConfigParser's and the raw
    # mode of ConfigParser's respective set functions, you can assign
    # non-string values to keys internally, but will receive an error
    # when attempting to write to a file or when you get it in non-raw
    # mode. SafeConfigParser does not allow such assignments to take place.
    config.add_section('Section1')
    config.set('Section1', 'int', '15')
    config.set('Section1', 'bool', 'true')
    config.set('Section1', 'float', '3.1415')
    config.set('Section1', 'baz', 'fun')
    config.set('Section1', 'bar', 'Python')
    config.set('Section1', 'foo', '%(bar)s is %(baz)s!')

    # Writing our configuration file to 'example.cfg'
    with open('example.cfg', 'wb') as configfile:
        config.write(configfile)

An example of reading the configuration file again:

    import ConfigParser

    config = ConfigParser.RawConfigParser()
    config.read('example.cfg')

    # getfloat() raises an exception if the value is not a float
    # getint() and getboolean() also do this for their respective types
    float = config.getfloat('Section1', 'float')
    int = config.getint('Section1', 'int')
    print float + int

    # Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'.
    # This is because we are using a RawConfigParser().
    if config.getboolean('Section1', 'bool'):
        print config.get('Section1', 'foo')

To get interpolation, you will need to use a ConfigParser or
SafeConfigParser.:

    import ConfigParser

    config = ConfigParser.ConfigParser()
    config.read('example.cfg')

    # Set the third, optional argument of get to 1 if you wish to use raw mode.
    print config.get('Section1', 'foo', 0) # -> "Python is fun!"
    print config.get('Section1', 'foo', 1) # -> "%(bar)s is %(baz)s!"

    # The optional fourth argument is a dict with members that will take
    # precedence in interpolation.
    print config.get('Section1', 'foo', 0, {'bar': 'Documentation',
                                            'baz': 'evil'})

Defaults are available in all three types of ConfigParsers. They are
used in interpolation if an option used is not defined elsewhere.

    import ConfigParser

    # New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each
    config = ConfigParser.SafeConfigParser({'bar': 'Life', 'baz': 'hard'})
    config.read('example.cfg')

    print config.get('Section1', 'foo') # -> "Python is fun!"
    config.remove_option('Section1', 'bar')
    config.remove_option('Section1', 'baz')
    print config.get('Section1', 'foo') # -> "Life is hard!"

The function `opt_move' below can be used to move options between
sections:

    def opt_move(config, section1, section2, option):
        try:
            config.set(section2, option, config.get(section1, option, 1))
        except ConfigParser.NoSectionError:
            # Create non-existent section
            config.add_section(section2)
            opt_move(config, section1, section2, option)
        else:
            config.remove_option(section1, option)

Some configuration files are known to include settings without values,
but which otherwise conform to the syntax supported by
ConfigParser.  The _allow_no_value_ parameter to the constructor
can be used to indicate that such values should be accepted:

    >>> import ConfigParser
    >>> import io

    >>> sample_config = """
    ... [mysqld]
    ... user = mysql
    ... pid-file = /var/run/mysqld/mysqld.pid
    ... skip-external-locking
    ... old_passwords = 1
    ... skip-bdb
    ... skip-innodb
    ... """
    >>> config = ConfigParser.RawConfigParser(allow_no_value=True)
    >>> config.readfp(io.BytesIO(sample_config))

    >>> # Settings with values are treated as before:
    >>> config.get("mysqld", "user")
    'mysql'

    >>> # Settings without values provide None:
    >>> config.get("mysqld", "skip-bdb")

    >>> # Settings which aren't specified still raise an error:
    >>> config.get("mysqld", "does-not-exist")
    Traceback (most recent call last):
      ...
    ConfigParser.NoOptionError: No option 'does-not-exist' in section: 'mysqld'

5.13.3 `robotparser' --  Parser for robots.txt
----------------------------------------------

     Note: The robotparser module has been renamed
     `urllib.robotparser' in Python 3.0.  The 2to3 tool
     will automatically adapt imports when converting your sources to
     3.0.

This module provides a single class, see RobotFileParser, which
answers questions about whether or not a particular user agent can
fetch a URL on the Web site that published the `robots.txt' file.  For
more details on the structure of `robots.txt' files, see
<http://www.robotstxt.org/orig.html>.

 -- Class: robotparser.RobotFileParser
     This class provides a set of methods to read, parse and answer
     questions about a single `robots.txt' file.

      -- Method: set_url (url)
          Sets the URL referring to a `robots.txt' file.

      -- Method: read ()
          Reads the `robots.txt' URL and feeds it to the parser.

      -- Method: parse (lines)
          Parses the lines argument.

      -- Method: can_fetch (useragent, url)
          Returns `True' if the _useragent_ is allowed to fetch the
          _url_ according to the rules contained in the parsed
          `robots.txt' file.

      -- Method: mtime ()
          Returns the time the `robots.txt' file was last fetched.
          This is useful for long-running web spiders that need to
          check for new `robots.txt' files periodically.

      -- Method: modified ()
          Sets the time the `robots.txt' file was last fetched to the
          current time.

  The following example demonstrates basic use of the RobotFileParser
class.

    >>> import robotparser
    >>> rp = robotparser.RobotFileParser()
    >>> rp.set_url("http://www.musi-cal.com/robots.txt")
    >>> rp.read()
    >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
    False
    >>> rp.can_fetch("*", "http://www.musi-cal.com/")
    True

5.13.4 `netrc' -- netrc file processing
---------------------------------------

New in version 1.5.2.

  The netrc class parses and encapsulates the netrc file
format used by the Unix *ftp* program and other FTP clients.

 -- Class: netrc.netrc ([file])
     A netrc instance or subclass instance encapsulates
     data from  a netrc file.  The initialization argument, if present,
     specifies the file to parse.  If no argument is given, the file
     `.netrc' in the user's home directory will be read.  Parse errors
     will raise NetrcParseError with diagnostic information
     including the file name, line number, and terminating token.

 -- Exception: netrc.NetrcParseError
     Exception raised by the netrc class when syntactical
     errors are encountered in source text.  Instances of this
     exception provide three interesting attributes:  `msg' is a
     textual explanation of the error, `filename' is the name of the
     source file, and `lineno' gives the line number on which the error
     was found.

5.13.4.1 netrc Objects
......................

A netrc instance has the following methods:

 -- Method: netrc.authenticators (host)
     Return a 3-tuple `(login, account, password)' of authenticators
     for _host_.  If the netrc file did not contain an entry for the
     given host, return the tuple associated with the 'default' entry.
     If neither matching host nor default entry is available, return
     `None'.

 -- Method: netrc.__repr__ ()
     Dump the class data as a string in the format of a netrc file.
     (This discards comments and may reorder the entries.)

  Instances of netrc have public instance variables:

 -- Attribute: netrc.hosts
     Dictionary mapping host names to `(login, account, password)'
     tuples.  The 'default' entry, if any, is represented as a
     pseudo-host by that name.

 -- Attribute: netrc.macros
     Dictionary mapping macro names to string lists.

     Note: Passwords are limited to a subset of the ASCII character
     set. Versions of this module prior to 2.3 were extremely limited.
     Starting with 2.3, all ASCII punctuation is allowed in passwords.
     However, note that whitespace and non-printable characters are not
     allowed in passwords.  This is a limitation of the way the .netrc
     file is parsed and may be removed in the future.

5.13.5 `xdrlib' -- Encode and decode XDR data
---------------------------------------------

The xdrlib module supports the External Data Representation
Standard as described in RFC 1014
(http://tools.ietf.org/html/rfc1014.html), written by Sun Microsystems,
Inc. June 1987.  It supports most of the data types described in the
RFC.

  The xdrlib module defines two classes, one for packing
variables into XDR representation, and another for unpacking from XDR
representation.  There are also two exception classes.

 -- Class: xdrlib.Packer
     Packer is the class for packing data into XDR
     representation. The Packer class is instantiated with
     no arguments.

 -- Class: xdrlib.Unpacker (data)
     `Unpacker' is the complementary class which unpacks XDR data
     values from a string buffer.  The input buffer is given as _data_.

RFC 1014 (http://tools.ietf.org/html/rfc1014.html) - XDR: External Data Representation Standard
     This RFC defined the encoding of data which was XDR at the time
     this module was originally written.  It has apparently been
     obsoleted by RFC 1832 (http://tools.ietf.org/html/rfc1832.html).

RFC 1832 (http://tools.ietf.org/html/rfc1832.html) - XDR: External Data Representation Standard
     Newer RFC that provides a revised definition of XDR.

5.13.5.1 Packer Objects
.......................

Packer instances have the following methods:

 -- Method: Packer.get_buffer ()
     Returns the current pack buffer as a string.

 -- Method: Packer.reset ()
     Resets the pack buffer to the empty string.

  In general, you can pack any of the most common XDR data types by
calling the appropriate `pack_type()' method.  Each method takes a
single argument, the value to pack.  The following simple data type
packing methods are supported: `pack_uint()', `pack_int()',
`pack_enum()', `pack_bool()', `pack_uhyper()', and `pack_hyper()'.

 -- Method: Packer.pack_float (value)
     Packs the single-precision floating point number _value_.

 -- Method: Packer.pack_double (value)
     Packs the double-precision floating point number _value_.

  The following methods support packing strings, bytes, and opaque data:

 -- Method: Packer.pack_fstring (n, s)
     Packs a fixed length string, _s_.  _n_ is the length of the string
     but it is _not_ packed into the data buffer.  The string is padded
     with null bytes if necessary to guaranteed 4 byte alignment.

 -- Method: Packer.pack_fopaque (n, data)
     Packs a fixed length opaque data stream, similarly to
     pack_fstring().

 -- Method: Packer.pack_string (s)
     Packs a variable length string, _s_.  The length of the string is
     first packed as an unsigned integer, then the string data is
     packed with pack_fstring().

 -- Method: Packer.pack_opaque (data)
     Packs a variable length opaque data string, similarly to
     pack_string().

 -- Method: Packer.pack_bytes (bytes)
     Packs a variable length byte stream, similarly to
     pack_string().

  The following methods support packing arrays and lists:

 -- Method: Packer.pack_list (list, pack_item)
     Packs a _list_ of homogeneous items.  This method is useful for
     lists with an indeterminate size; i.e. the size is not available
     until the entire list has been walked.  For each item in the list,
     an unsigned integer `1' is packed first, followed by the data
     value from the list.  _pack_item_ is the function that is called
     to pack the individual item.  At the end of the list, an unsigned
     integer `0' is packed.

     For example, to pack a list of integers, the code might appear
     like this:

         import xdrlib
         p = xdrlib.Packer()
         p.pack_list([1, 2, 3], p.pack_int)

 -- Method: Packer.pack_farray (n, array, pack_item)
     Packs a fixed length list (_array_) of homogeneous items.  _n_ is
     the length of the list; it is _not_ packed into the buffer, but a
     ValueError exception is raised if `len(array)' is not
     equal to _n_.  As above, _pack_item_ is the function used to pack
     each element.

 -- Method: Packer.pack_array (list, pack_item)
     Packs a variable length _list_ of homogeneous items.  First, the
     length of the list is packed as an unsigned integer, then each
     element is packed as in pack_farray() above.

5.13.5.2 Unpacker Objects
.........................

The Unpacker class offers the following methods:

 -- Method: Unpacker.reset (data)
     Resets the string buffer with the given _data_.

 -- Method: Unpacker.get_position ()
     Returns the current unpack position in the data buffer.

 -- Method: Unpacker.set_position (position)
     Sets the data buffer unpack position to _position_.  You should be
     careful about using get_position() and
     set_position().

 -- Method: Unpacker.get_buffer ()
     Returns the current unpack data buffer as a string.

 -- Method: Unpacker.done ()
     Indicates unpack completion.  Raises an Error.
     exception if all of the data has not been unpacked.

  In addition, every data type that can be packed with a Packer,
can be unpacked with an Unpacker.  Unpacking methods
are of the form `unpack_type()', and take no arguments.  They return
the unpacked object.

 -- Method: Unpacker.unpack_float ()
     Unpacks a single-precision floating point number.

 -- Method: Unpacker.unpack_double ()
     Unpacks a double-precision floating point number, similarly to
     unpack_float().

  In addition, the following methods unpack strings, bytes, and opaque
data:

 -- Method: Unpacker.unpack_fstring (n)
     Unpacks and returns a fixed length string.  _n_ is the number of
     characters expected.  Padding with null bytes to guaranteed 4 byte
     alignment is assumed.

 -- Method: Unpacker.unpack_fopaque (n)
     Unpacks and returns a fixed length opaque data stream, similarly to
     unpack_fstring().

 -- Method: Unpacker.unpack_string ()
     Unpacks and returns a variable length string.  The length of the
     string is first unpacked as an unsigned integer, then the string
     data is unpacked with unpack_fstring().

 -- Method: Unpacker.unpack_opaque ()
     Unpacks and returns a variable length opaque data string,
     similarly to unpack_string().

 -- Method: Unpacker.unpack_bytes ()
     Unpacks and returns a variable length byte stream, similarly to
     unpack_string().

  The following methods support unpacking arrays and lists:

 -- Method: Unpacker.unpack_list (unpack_item)
     Unpacks and returns a list of homogeneous items.  The list is
     unpacked one element at a time by first unpacking an unsigned
     integer flag.  If the flag is `1', then the item is unpacked and
     appended to the list.  A flag of `0' indicates the end of the
     list.  _unpack_item_ is the function that is called to unpack the
     items.

 -- Method: Unpacker.unpack_farray (n, unpack_item)
     Unpacks and returns (as a list) a fixed length array of
     homogeneous items.  _n_ is number of list elements to expect in
     the buffer. As above, _unpack_item_ is the function used to unpack
     each element.

 -- Method: Unpacker.unpack_array (unpack_item)
     Unpacks and returns a variable length _list_ of homogeneous items.
     First, the length of the list is unpacked as an unsigned integer,
     then each element is unpacked as in unpack_farray().
     above.

5.13.5.3 Exceptions
...................

Exceptions in this module are coded as class instances:

 -- Exception: xdrlib.Error
     The base exception class.  Error has a single public
     data member `msg' containing the description of the error.

 -- Exception: xdrlib.ConversionError
     Class derived from Error.  Contains no additional
     instance variables.

  Here is an example of how you would catch one of these exceptions:

    import xdrlib
    p = xdrlib.Packer()
    try:
        p.pack_double(8.01)
    except xdrlib.ConversionError, instance:
        print 'packing the double failed:', instance.msg

5.13.6 `plistlib' -- Generate and parse Mac OS X `.plist' files
---------------------------------------------------------------

Changed in version 2.6: This module was previously only available in
the Mac-specific library, it is now available for all platforms.

  This module provides an interface for reading and writing the
"property list" XML files used mainly by Mac OS X.

  The property list (`.plist') file format is a simple XML pickle
supporting basic object types, like dictionaries, lists, numbers and
strings.  Usually the top level object is a dictionary.

  Values can be strings, integers, floats, booleans, tuples, lists,
dictionaries (but only with string keys), see Data or
datetime.datetime.  objects.  String values (including dictionary
keys) may be unicode strings - they will be written out as UTF-8.

  The `<data>' plist type is supported through the Data.
class.  This is a thin wrapper around a Python string.  Use Data
if your strings contain control characters.

PList manual page (http://developer.apple.com/documentation/Darwin/Reference/ManPages/man5/plist.5.html)
     Apple's documentation of the file format.

  This module defines the following functions:

 -- Function: plistlib.readPlist (pathOrFile)
     Read a plist file. _pathOrFile_ may either be a file name or a
     (readable) file object.  Return the unpacked root object (which
     usually is a dictionary).

     The XML data is parsed using the Expat parser from
     xml.parsers.expat.  - see its documentation for possible
     exceptions on ill-formed XML.  Unknown elements will simply be
     ignored by the plist parser.

 -- Function: plistlib.writePlist (rootObject, pathOrFile)
     Write _rootObject_ to a plist file. _pathOrFile_ may either be a
     file name or a (writable) file object.

     A TypeError will be raised if the object is of an
     unsupported type or a container that contains objects of
     unsupported types.

 -- Function: plistlib.readPlistFromString (data)
     Read a plist from a string.  Return the root object.

 -- Function: plistlib.writePlistToString (rootObject)
     Return _rootObject_ as a plist-formatted string.

 -- Function: plistlib.readPlistFromResource (path[, restype='plst'[,
          resid=0]])
     Read a plist from the resource with type _restype_ from the
     resource fork of _path_.  Availability: Mac OS X.

          Note: In Python 3.x, this function has been removed.

 -- Function: plistlib.writePlistToResource (rootObject, path[,
          restype='plst'[, resid=0]])
     Write _rootObject_ as a resource with type _restype_ to the
     resource fork of _path_.  Availability: Mac OS X.

          Note: In Python 3.x, this function has been removed.

  The following class is available:

 -- Class: plistlib.Data (data)
     Return a "data" wrapper object around the string _data_.  This is
     used in functions converting from/to plists to represent the
     `<data>' type available in plists.

     It has one attribute, `data', that can be used to retrieve the
     Python string stored in it.

5.13.6.1 Examples
.................

Generating a plist:

    pl = dict(
        aString="Doodah",
        aList=["A", "B", 12, 32.1, [1, 2, 3]],
        aFloat = 0.1,
        anInt = 728,
        aDict=dict(
            anotherString="<hello & hi there!>",
            aUnicodeValue=u'M\xe4ssig, Ma\xdf',
            aTrueValue=True,
            aFalseValue=False,
        ),
        someData = Data("<binary gunk>"),
        someMoreData = Data("<lots of binary gunk>" * 10),
        aDate = datetime.datetime.fromtimestamp(time.mktime(time.gmtime())),
    )
    # unicode keys are possible, but a little awkward to use:
    pl[u'\xc5benraa'] = "That was a unicode key."
    writePlist(pl, fileName)

Parsing a plist:

    pl = readPlist(pathOrFile)
    print pl["aKey"]

5.14 Cryptographic Services
===========================

The modules described in this chapter implement various algorithms of a
cryptographic nature.  They are available at the discretion of the
installation.  Here's an overview:

5.14.1 `hashlib' -- Secure hashes and message digests
-----------------------------------------------------

New in version 2.5.

  This module implements a common interface to many different secure
hash and message digest algorithms.  Included are the FIPS secure hash
algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS
180-2) as well as RSA's MD5 algorithm (defined in Internet RFC 1321
(http://tools.ietf.org/html/rfc1321.html)). The terms secure hash and
message digest are interchangeable.  Older algorithms were called
message digests.  The modern term is secure hash.

     Note: If you want the adler32 or crc32 hash functions they are
     available in the zlib module.

     Warning: Some algorithms have known hash collision weaknesses, see
     the FAQ at the end.

  There is one constructor method named for each type of _hash_.  All
return a hash object with the same simple interface. For example: use
`sha1()' to create a SHA1 hash object. You can now feed this object
with arbitrary strings using the `update()' method.  At any point you
can ask it for the _digest_ of the concatenation of the strings fed to
it so far using the `digest()' or `hexdigest()' methods.

  Constructors for hash algorithms that are always present in this
module are md5(), `sha1()', `sha224()', `sha256()',
`sha384()', and `sha512()'.  Additional algorithms may also be
available depending upon the OpenSSL library that Python uses on your
platform.

  For example, to obtain the digest of the string `'Nobody inspects the
spammish repetition'':

    >>> import hashlib
    >>> m = hashlib.md5()
    >>> m.update("Nobody inspects")
    >>> m.update(" the spammish repetition")
    >>> m.digest()
    '\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9'
    >>> m.digest_size
    16
    >>> m.block_size
    64

More condensed:

    >>> hashlib.sha224("Nobody inspects the spammish repetition").hexdigest()
    'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2'

A generic new() constructor that takes the string name of
the desired algorithm as its first parameter also exists to allow
access to the above listed hashes as well as any other algorithms that
your OpenSSL library may offer.  The named constructors are much faster
than new() and should be preferred.

  Using new() with an algorithm provided by OpenSSL:

    >>> h = hashlib.new('ripemd160')
    >>> h.update("Nobody inspects the spammish repetition")
    >>> h.hexdigest()
    'cc4a5ce1b3df48aec5d22d1f16b894a0b894eccc'

This module provides the following constant attribute:

 -- Data: hashlib.algorithms
     A tuple providing the names of the hash algorithms guaranteed to be
     supported by this module.

     New in version 2.7.

  The following values are provided as constant attributes of the hash
objects returned by the constructors:

 -- Data: hash.digest_size
     The size of the resulting hash in bytes.

 -- Data: hash.block_size
     The internal block size of the hash algorithm in bytes.

  A hash object has the following methods:

 -- Method: hash.update (arg)
     Update the hash object with the string _arg_.  Repeated calls are
     equivalent to a single call with the concatenation of all the
     arguments: `m.update(a); m.update(b)' is equivalent to
     `m.update(a+b)'.

     Changed in version 2.7.

 -- Method: hash.digest ()
     Return the digest of the strings passed to the update()
     method so far.  This is a string of digest_size.
     bytes which may contain non-ASCII characters, including null bytes.

 -- Method: hash.hexdigest ()
     Like digest() except the digest is returned as a
     string of double length, containing only hexadecimal digits.  This
     may  be used to exchange the value safely in email or other
     non-binary environments.

 -- Method: hash.copy ()
     Return a copy ("clone") of the hash object.  This can be used to
     efficiently compute the digests of strings that share a common
     initial substring.

Module hmac.
     A module to generate message authentication codes using hashes.

Module base64.
     Another way to encode binary hashes for non-binary environments.

<http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf>
     The FIPS 180-2 publication on Secure Hash Algorithms.

<http://en.wikipedia.org/wiki/Cryptographic_hash_function#Cryptographic_hash_algorithms>
     Wikipedia article with information on which algorithms have known
     issues and what that means regarding their use.

5.14.2 `hmac' -- Keyed-Hashing for Message Authentication
---------------------------------------------------------

New in version 2.2.

  This module implements the HMAC algorithm as described by RFC 2104
(http://tools.ietf.org/html/rfc2104.html).

 -- Function: hmac.new (key[, msg[, digestmod]])
     Return a new hmac object.  If _msg_ is present, the method call
     `update(msg)' is made. _digestmod_ is the digest constructor or
     module for the HMAC object to use. It defaults to  the
     `hashlib.md5()' constructor.

  An HMAC object has the following methods:

 -- Method: hmac.update (msg)
     Update the hmac object with the string _msg_.  Repeated calls are
     equivalent to a single call with the concatenation of all the
     arguments: `m.update(a); m.update(b)' is equivalent to `m.update(a
     + b)'.

 -- Method: hmac.digest ()
     Return the digest of the strings passed to the update()
     method so far.  This string will be the same length as the
     _digest_size_ of the digest given to the constructor.  It may
     contain non-ASCII characters, including NUL bytes.

 -- Method: hmac.hexdigest ()
     Like digest() except the digest is returned as a
     string twice the length containing only hexadecimal digits.  This
     may be used to exchange the value safely in email or other
     non-binary environments.

 -- Method: hmac.copy ()
     Return a copy ("clone") of the hmac object.  This can be used to
     efficiently compute the digests of strings that share a common
     initial substring.

Module hashlib.
     The Python module providing secure hash functions.

5.14.3 `md5' -- MD5 message digest algorithm
--------------------------------------------

Deprecated since version 2.5: Use the hashlib module instead.

  This module implements the interface to RSA's MD5 message digest
algorithm (see also Internet RFC 1321
(http://tools.ietf.org/html/rfc1321.html)).  Its use is quite
straightforward: use new().  to create an md5 object. You
can now feed this object with arbitrary strings using the `update()'
method, and at any point you can ask it for the _digest_ (a strong kind
of 128-bit checksum, a.k.a. "fingerprint") of the concatenation of the
strings fed to it so far using the `digest()' method.

  For example, to obtain the digest of the string `'Nobody inspects the
spammish repetition'':

    >>> import md5
    >>> m = md5.new()
    >>> m.update("Nobody inspects")
    >>> m.update(" the spammish repetition")
    >>> m.digest()
    '\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9'

More condensed:

    >>> md5.new("Nobody inspects the spammish repetition").digest()
    '\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9'

The following values are provided as constants in the module and as
attributes of the md5 objects returned by new().:

 -- Data: md5.digest_size
     The size of the resulting digest in bytes.  This is always `16'.

  The md5 module provides the following functions:

 -- Function: md5.new ([arg])
     Return a new md5 object.  If _arg_ is present, the method call
     `update(arg)' is made.

 -- Function: md5.md5 ([arg])
     For backward compatibility reasons, this is an alternative name
     for the new() function.

  An md5 object has the following methods:

 -- Method: md5.update (arg)
     Update the md5 object with the string _arg_.  Repeated calls are
     equivalent to a single call with the concatenation of all the
     arguments: `m.update(a); m.update(b)' is equivalent to
     `m.update(a+b)'.

 -- Method: md5.digest ()
     Return the digest of the strings passed to the update()
     method so far.  This is a 16-byte string which may contain
     non-ASCII characters, including null bytes.

 -- Method: md5.hexdigest ()
     Like digest() except the digest is returned as a
     string of length 32, containing only hexadecimal digits.  This may
     be used to exchange the value safely in email or other non-binary
     environments.

 -- Method: md5.copy ()
     Return a copy ("clone") of the md5 object.  This can be used to
     efficiently compute the digests of strings that share a common
     initial substring.

Module sha.
     Similar module implementing the Secure Hash Algorithm (SHA).  The
     SHA algorithm is considered a more secure hash.

5.14.4 `sha' -- SHA-1 message digest algorithm
----------------------------------------------

Deprecated since version 2.5: Use the hashlib module instead.

  This module implements the interface to NIST's secure hash
algorithm, known as SHA-1.  SHA-1 is an improved version of the
original SHA hash algorithm.  It is used in the same way as the
md5 module: use new() to create an sha object, then
feed this object with arbitrary strings using the `update()' method,
and at any point you can ask it for the _digest_ of the concatenation
of the strings fed to it so far.  SHA-1 digests are 160 bits instead of
MD5's 128 bits.

 -- Function: sha.new ([string])
     Return a new sha object.  If _string_ is present, the method call
     `update(string)' is made.

  The following values are provided as constants in the module and as
attributes of the sha objects returned by new().:

 -- Data: sha.blocksize
     Size of the blocks fed into the hash function; this is always `1'.
     This size is used to allow an arbitrary string to be hashed.

 -- Data: sha.digest_size
     The size of the resulting digest in bytes.  This is always `20'.

  An sha object has the same methods as md5 objects:

 -- Method: sha.update (arg)
     Update the sha object with the string _arg_.  Repeated calls are
     equivalent to a single call with the concatenation of all the
     arguments: `m.update(a); m.update(b)' is equivalent to
     `m.update(a+b)'.

 -- Method: sha.digest ()
     Return the digest of the strings passed to the update()
     method so far.  This is a 20-byte string which may contain
     non-ASCII characters, including null bytes.

 -- Method: sha.hexdigest ()
     Like digest() except the digest is returned as a
     string of length 40, containing only hexadecimal digits.  This may
     be used to exchange the value safely in email or other non-binary
     environments.

 -- Method: sha.copy ()
     Return a copy ("clone") of the sha object.  This can be used to
     efficiently compute the digests of strings that share a common
     initial substring.

Secure Hash Standard (http://csrc.nist.gov/publications/fips/fips180-2/fips180-2withchangenotice.pdf)
     The Secure Hash Algorithm is defined by NIST document FIPS PUB
     180-2: Secure Hash Standard
     (http://csrc.nist.gov/publications/fips/fips180-2/fips180-2withchangenotice.pdf),
     published in August 2002.

Cryptographic Toolkit (Secure Hashing) (http://csrc.nist.gov/CryptoToolkit/tkhash.html)
     Links from NIST to various information on secure hashing.

  Hardcore cypherpunks will probably find the cryptographic modules
written by A.M. Kuchling of further interest; the package contains
modules for various encryption algorithms, most notably AES.  These
modules are not distributed with Python but available separately.  See
the URL <http://www.amk.ca/python/code/crypto.html>  for more
information.

5.15 Generic Operating System Services
======================================

The modules described in this chapter provide interfaces to operating
system features that are available on (almost) all operating systems,
such as files and a clock.  The interfaces are generally modeled after
the Unix or C interfaces, but they are available on most other systems
as well.  Here's an overview:

5.15.1 `os' -- Miscellaneous operating system interfaces
--------------------------------------------------------

This module provides a portable way of using operating system dependent
functionality.  If you just want to read or write a file see
open(), if you want to manipulate paths, see the os.path
module, and if you want to read all the lines in all the files on
the command line see the fileinput.  module.  For creating
temporary files and directories see the tempfile.  module,
and for high-level file and directory handling see the shutil.
module.

  Notes on the availability of these functions:

   * The design of all built-in operating system dependent modules of
     Python is such that as long as the same functionality is
     available, it uses the same interface; for example, the function
     `os.stat(path)' returns stat information about _path_ in the same
     format (which happens to have originated with the POSIX interface).

   * Extensions peculiar to a particular operating system are also
     available through the os module, but using them is of
     course a threat to portability.

   * An "Availability: Unix" note means that this function is commonly
     found on Unix systems.  It does not make any claims about its
     existence on a specific operating system.

   * If not separately noted, all functions that claim "Availability:
     Unix" are supported on Mac OS X, which builds on a Unix core.

     Note: All functions in this module raise OSError in
     the case of invalid or inaccessible file names and paths, or other
     arguments that have the correct type, but are not accepted by the
     operating system.

 -- Exception: os.error
     An alias for the built-in OSError exception.

 -- Data: os.name
     The name of the operating system dependent module imported.  The
     following names have currently been registered: `'posix'', `'nt'',
     `'os2'', `'ce'', `'java'', `'riscos''.

5.15.1.1 Process Parameters
...........................

These functions and data items provide information and operate on the
current process and user.

 -- Data: os.environ
     A mapping object representing the string environment. For example,
     `environ['HOME']' is the pathname of your home directory (on some
     platforms), and is equivalent to `getenv("HOME")' in C.

     This mapping is captured the first time the os module
     is imported, typically during Python startup as part of processing
     `site.py'.  Changes to the environment made after this time are
     not reflected in `os.environ', except for changes made by
     modifying `os.environ' directly.

     If the platform supports the putenv() function, this
     mapping may be used to modify the environment as well as query the
     environment.  putenv() will be called automatically
     when the mapping is modified.

          Note: Calling putenv() directly does not change
          `os.environ', so it's better to modify `os.environ'.

          Note: On some platforms, including FreeBSD and Mac OS X,
          setting `environ' may cause memory leaks.  Refer to the
          system documentation for `putenv()'.

     If putenv() is not provided, a modified copy of this
     mapping  may be passed to the appropriate process-creation
     functions to cause  child processes to use a modified environment.

     If the platform supports the unsetenv() function, you
     can delete items in this mapping to unset environment variables.
     unsetenv() will be called automatically when an item
     is deleted from `os.environ', and when one of the `pop()' or
     `clear()' methods is called.

     Changed in version 2.6: Also unset environment variables when
     calling `os.environ.clear()' and `os.environ.pop()'.

 -- Function: os.chdir (path)
 -- Function: os.fchdir (fd)
 -- Function: os.getcwd ()
     These functions are described in Files and Directories.

 -- Function: os.ctermid ()
     Return the filename corresponding to the controlling terminal of
     the process.

     Availability: Unix.

 -- Function: os.getegid ()
     Return the effective group id of the current process.  This
     corresponds to the "set id" bit on the file being executed in the
     current process.

     Availability: Unix.

 -- Function: os.geteuid ()
     Return the current process's effective user id.

     Availability: Unix.

 -- Function: os.getgid ()
     Return the real group id of the current process.

     Availability: Unix.

 -- Function: os.getgroups ()
     Return list of supplemental group ids associated with the current
     process.

     Availability: Unix.

 -- Function: os.initgroups (username, gid)
     Call the system initgroups() to initialize the group access list
     with all of the groups of which the specified username is a
     member, plus the specified group id.

     Availability: Unix.

     New in version 2.7.

 -- Function: os.getlogin ()
     Return the name of the user logged in on the controlling terminal
     of the process.  For most purposes, it is more useful to use the
     environment variable `LOGNAME' to find out who the user is, or
     `pwd.getpwuid(os.getuid())[0]' to get the login name of the
     currently effective user id.

     Availability: Unix.

 -- Function: os.getpgid (pid)
     Return the process group id of the process with process id _pid_.
     If _pid_ is 0, the process group id of the current process is
     returned.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.getpgrp ()
     Return the id of the current process group.

     Availability: Unix.

 -- Function: os.getpid ()
     Return the current process id.

     Availability: Unix, Windows.

 -- Function: os.getppid ()
     Return the parent's process id.

     Availability: Unix.

 -- Function: os.getresuid ()
     Return a tuple (ruid, euid, suid) denoting the current process's
     real, effective, and saved user ids.

     Availability: Unix.

     New in version 2.7.

 -- Function: os.getresgid ()
     Return a tuple (rgid, egid, sgid) denoting the current process's
     real, effective, and saved group ids.

     Availability: Unix.

     New in version 2.7.

 -- Function: os.getuid ()
     Return the current process's user id.

     Availability: Unix.

 -- Function: os.getenv (varname[, value])
     Return the value of the environment variable _varname_ if it
     exists, or _value_ if it doesn't.  _value_ defaults to `None'.

     Availability: most flavors of Unix, Windows.

 -- Function: os.putenv (varname, value)
     Set the environment variable named _varname_ to the string
     _value_.  Such changes to the environment affect subprocesses
     started with os.system(), popen() or
     fork() and execv().

     Availability: most flavors of Unix, Windows.

          Note: On some platforms, including FreeBSD and Mac OS X,
          setting `environ' may cause memory leaks. Refer to the system
          documentation for putenv.

     When putenv() is supported, assignments to items in
     `os.environ' are automatically translated into corresponding calls
     to putenv().; however, calls to putenv().
     don't update `os.environ', so it is actually preferable to assign
     to items of `os.environ'.

 -- Function: os.setegid (egid)
     Set the current process's effective group id.

     Availability: Unix.

 -- Function: os.seteuid (euid)
     Set the current process's effective user id.

     Availability: Unix.

 -- Function: os.setgid (gid)
     Set the current process' group id.

     Availability: Unix.

 -- Function: os.setgroups (groups)
     Set the list of supplemental group ids associated with the current
     process to _groups_. _groups_ must be a sequence, and each element
     must be an integer identifying a group. This operation is
     typically available only to the superuser.

     Availability: Unix.

     New in version 2.2.

 -- Function: os.setpgrp ()
     Call the system call `setpgrp()' or `setpgrp(0, 0)()' depending on
     which version is implemented (if any).  See the Unix manual for
     the semantics.

     Availability: Unix.

 -- Function: os.setpgid (pid, pgrp)
     Call the system call `setpgid()' to set the process group id of the
     process with id _pid_ to the process group with id _pgrp_.  See
     the Unix manual for the semantics.

     Availability: Unix.

 -- Function: os.setregid (rgid, egid)
     Set the current process's real and effective group ids.

     Availability: Unix.

 -- Function: os.setresgid (rgid, egid, sgid)
     Set the current process's real, effective, and saved group ids.

     Availability: Unix.

     New in version 2.7.

 -- Function: os.setresuid (ruid, euid, suid)
     Set the current process's real, effective, and saved user ids.

     Availability: Unix.

     New in version 2.7.

 -- Function: os.setreuid (ruid, euid)
     Set the current process's real and effective user ids.

     Availability: Unix.

 -- Function: os.getsid (pid)
     Call the system call `getsid()'.  See the Unix manual for the
     semantics.

     Availability: Unix.

     New in version 2.4.

 -- Function: os.setsid ()
     Call the system call `setsid()'.  See the Unix manual for the
     semantics.

     Availability: Unix.

 -- Function: os.setuid (uid)
     Set the current process's user id.

     Availability: Unix.

 -- Function: os.strerror (code)
     Return the error message corresponding to the error code in _code_.
     On platforms where `strerror()' returns `NULL' when given an
     unknown error number, see ValueError is raised.

     Availability: Unix, Windows.

 -- Function: os.umask (mask)
     Set the current numeric umask and return the previous umask.

     Availability: Unix, Windows.

 -- Function: os.uname ()
     Return a 5-tuple containing information identifying the current
     operating system.  The tuple contains 5 strings: `(sysname,
     nodename, release, version, machine)'.  Some systems truncate the
     nodename to 8 characters or to the leading component; a better way
     to get the hostname is socket.gethostname().  or even
     `socket.gethostbyaddr(socket.gethostname())'.

     Availability: recent flavors of Unix.

 -- Function: os.unsetenv (varname)
     Unset (delete) the environment variable named _varname_. Such
     changes to the environment affect subprocesses started with
     os.system(), popen() or fork() and
     execv().

     When unsetenv() is supported, deletion of items in
     `os.environ' is automatically translated into a corresponding call
     to unsetenv().; however, calls to unsetenv()
     don't update `os.environ', so it is actually preferable to
     delete items of `os.environ'.

     Availability: most flavors of Unix, Windows.

5.15.1.2 File Object Creation
.............................

These functions create new file objects. (See also open())

 -- Function: os.fdopen (fd[, mode[, bufsize]])
     Return an open file object connected to the file descriptor _fd_.
     The _mode_ and _bufsize_ arguments have the same meaning as the
     corresponding arguments to the built-in open().
     function.

     Availability: Unix, Windows.

     Changed in version 2.3: When specified, the _mode_ argument must
     now start with one of the letters `'r'', `'w'', or `'a'',
     otherwise a ValueError is raised.

     Changed in version 2.5: On Unix, when the _mode_ argument starts
     with `'a'', the _O_APPEND_ flag is set on the file descriptor
     (which the `fdopen()' implementation already does on most
     platforms).

 -- Function: os.popen (command[, mode[, bufsize]])
     Open a pipe to or from _command_.  The return value is an open
     file object connected to the pipe, which can be read or written
     depending on whether _mode_ is `'r'' (default) or `'w''. The
     _bufsize_ argument has the same meaning as the corresponding
     argument to the built-in open() function.  The exit
     status of the command (encoded in the format specified for
     wait()) is available as the return value of the
     close() method of the file object, except that when the exit
     status is zero (termination without errors), `None' is returned.

     Availability: Unix, Windows.

     Deprecated since version 2.6: This function is obsolete.  Use the
     subprocess module.  Check especially the
     Replacing Older Functions with the subprocess Module.
     section.

     Changed in version 2.0: This function worked unreliably under
     Windows in earlier versions of Python.  This was due to the use of
     the `_popen()' function from the libraries provided with Windows.
     Newer versions of Python do not use the broken implementation from
     the Windows libraries.

 -- Function: os.tmpfile ()
     Return a new file object opened in update mode (`w+b').  The file
     has no directory entries associated with it and will be
     automatically deleted once there are no file descriptors for the
     file.

     Availability: Unix, Windows.

  There are a number of different `popen*()' functions that provide
slightly different ways to create subprocesses.

  Deprecated since version 2.6: All of the `popen*()' functions are
obsolete. Use the subprocess.  module.

  For each of the `popen*()' variants, if _bufsize_ is specified, it
specifies the buffer size for the I/O pipes. _mode_, if provided,
should be the string `'b'' or `'t''; on Windows this is needed to
determine whether the file objects should be opened in binary or text
mode.  The default value for _mode_ is `'t''.

  Also, for each of these variants, on Unix, _cmd_ may be a sequence,
in which case arguments will be passed directly to the program without
shell intervention (as with os.spawnv()). If _cmd_ is a
string it will be passed to the shell (as with os.system()).

  These methods do not make it possible to retrieve the exit status
from the child processes.  The only way to control the input and output
streams and also retrieve the return codes is to use the
subprocess module; these are only available on Unix.

  For a discussion of possible deadlock conditions related to the use
of these functions, see Flow Control Issues.

 -- Function: os.popen2 (cmd[, mode[, bufsize]])
     Execute _cmd_ as a sub-process and return the file objects
     `(child_stdin, child_stdout)'.

     Deprecated since version 2.6: This function is obsolete.  Use the
     subprocess module.  Check especially the
     Replacing Older Functions with the subprocess Module.
     section.

     Availability: Unix, Windows.

     New in version 2.0.

 -- Function: os.popen3 (cmd[, mode[, bufsize]])
     Execute _cmd_ as a sub-process and return the file objects
     `(child_stdin, child_stdout, child_stderr)'.

     Deprecated since version 2.6: This function is obsolete.  Use the
     subprocess module.  Check especially the
     Replacing Older Functions with the subprocess Module.
     section.

     Availability: Unix, Windows.

     New in version 2.0.

 -- Function: os.popen4 (cmd[, mode[, bufsize]])
     Execute _cmd_ as a sub-process and return the file objects
     `(child_stdin, child_stdout_and_stderr)'.

     Deprecated since version 2.6: This function is obsolete.  Use the
     subprocess module.  Check especially the
     Replacing Older Functions with the subprocess Module.
     section.

     Availability: Unix, Windows.

     New in version 2.0.

  (Note that `child_stdin, child_stdout, and child_stderr' are named
from the point of view of the child process, so _child_stdin_ is the
child's standard input.)

  This functionality is also available in the popen2 module
using functions of the same names, but the return values of those
functions have a different order.

5.15.1.3 File Descriptor Operations
...................................

These functions operate on I/O streams referenced using file
descriptors.

  File descriptors are small integers corresponding to a file that has
been opened by the current process.  For example, standard input is
usually file descriptor 0, standard output is 1, and standard error is
2.  Further files opened by a process will then be assigned 3, 4, 5,
and so forth.  The name "file descriptor" is slightly deceptive; on
Unix platforms, sockets and pipes are also referenced by file
descriptors.

  The fileno() method can be used to obtain the file
descriptor associated with a file object when required.  Note that
using the file descriptor directly will bypass the file object methods,
ignoring aspects such as internal buffering of data.

 -- Function: os.close (fd)
     Close file descriptor _fd_.

     Availability: Unix, Windows.

          Note: This function is intended for low-level I/O and must be
          applied to a file descriptor as returned by os.open()
          or pipe().  To close a "file object"
          returned by the built-in function open() or by
          popen() or fdopen(), use its
          close() method.

 -- Function: os.closerange (fd_low, fd_high)
     Close all file descriptors from _fd_low_ (inclusive) to _fd_high_
     (exclusive), ignoring errors. Equivalent to:

         for fd in xrange(fd_low, fd_high):
             try:
                 os.close(fd)
             except OSError:
                 pass

     Availability: Unix, Windows.

     New in version 2.6.

 -- Function: os.dup (fd)
     Return a duplicate of file descriptor _fd_.

     Availability: Unix, Windows.

 -- Function: os.dup2 (fd, fd2)
     Duplicate file descriptor _fd_ to _fd2_, closing the latter first
     if necessary.

     Availability: Unix, Windows.

 -- Function: os.fchmod (fd, mode)
     Change the mode of the file given by _fd_ to the numeric _mode_.
     See the docs for chmod() for possible values of _mode_.

     Availability: Unix.

     New in version 2.6.

 -- Function: os.fchown (fd, uid, gid)
     Change the owner and group id of the file given by _fd_ to the
     numeric _uid_ and _gid_.  To leave one of the ids unchanged, set
     it to -1.

     Availability: Unix.

     New in version 2.6.

 -- Function: os.fdatasync (fd)
     Force write of file with filedescriptor _fd_ to disk. Does not
     force update of metadata.

     Availability: Unix.

          Note: This function is not available on MacOS.

 -- Function: os.fpathconf (fd, name)
     Return system configuration information relevant to an open file.
     _name_ specifies the configuration value to retrieve; it may be a
     string which is the name of a defined system value; these names
     are specified in a number of standards (POSIX.1, Unix 95, Unix 98,
     and others).  Some platforms define additional names as well.  The
     names known to the host operating system are given in the
     `pathconf_names' dictionary.  For configuration variables not
     included in that mapping, passing an integer for _name_ is also
     accepted.

     If _name_ is a string and is not known, see ValueError is
     raised.  If a specific value for _name_ is not supported by the
     host system, even if it is included in `pathconf_names', an
     OSError is raised with errno.EINVAL for the
     error number.

     Availability: Unix.

 -- Function: os.fstat (fd)
     Return status for file descriptor _fd_, like stat().

     Availability: Unix, Windows.

 -- Function: os.fstatvfs (fd)
     Return information about the filesystem containing the file
     associated with file descriptor _fd_, like statvfs().

     Availability: Unix.

 -- Function: os.fsync (fd)
     Force write of file with filedescriptor _fd_ to disk.  On Unix,
     this calls the native `fsync()' function; on Windows, the MS
     `_commit()' function.

     If you're starting with a Python file object _f_, first do
     `f.flush()', and then do `os.fsync(f.fileno())', to ensure that
     all internal buffers associated with _f_ are written to disk.

     Availability: Unix, and Windows starting in 2.2.3.

 -- Function: os.ftruncate (fd, length)
     Truncate the file corresponding to file descriptor _fd_, so that
     it is at most _length_ bytes in size.

     Availability: Unix.

 -- Function: os.isatty (fd)
     Return `True' if the file descriptor _fd_ is open and connected to
     a tty(-like) device, else `False'.

     Availability: Unix.

 -- Function: os.lseek (fd, pos, how)
     Set the current position of file descriptor _fd_ to position
     _pos_, modified by _how_: SEEK_SET or `0' to set the
     position relative to the beginning of the file; see SEEK_CUR
     or `1' to set it relative to the current position; see
     os.SEEK_END or `2' to set it relative to the end of the file.

     Availability: Unix, Windows.

 -- Data: os.SEEK_SET
 -- Data: os.SEEK_CUR
 -- Data: os.SEEK_END
     Parameters to the lseek() function. Their values are
     0, 1, and 2, respectively.

     Availability: Windows, Unix.

     New in version 2.5.

 -- Function: os.open (file, flags[, mode])
     Open the file _file_ and set various flags according to _flags_
     and possibly its mode according to _mode_. The default _mode_ is
     `0777' (octal), and the current umask value is first masked out.
     Return the file descriptor for the newly opened file.

     For a description of the flag and mode values, see the C run-time
     documentation; flag constants (like O_RDONLY and
     O_WRONLY) are defined in this module too (see
     open() flag constants).  In particular, on Windows adding
     O_BINARY is needed to open files in binary mode.

     Availability: Unix, Windows.

          Note: This function is intended for low-level I/O.  For
          normal usage, use the built-in function open(),
          which returns a "file object" with read() and
          write() methods (and many more).  To wrap a file
          descriptor in a "file object", use fdopen().

 -- Function: os.openpty ()
     Open a new pseudo-terminal pair. Return a pair of file descriptors
     `(master, slave)' for the pty and the tty, respectively. For a
     (slightly) more portable approach, use the pty module.

     Availability: some flavors of Unix.

 -- Function: os.pipe ()
     Create a pipe.  Return a pair of file descriptors `(r, w)' usable
     for reading and writing, respectively.

     Availability: Unix, Windows.

 -- Function: os.read (fd, n)
     Read at most _n_ bytes from file descriptor _fd_. Return a string
     containing the bytes read.  If the end of the file referred to by
     _fd_ has been reached, an empty string is returned.

     Availability: Unix, Windows.

          Note: This function is intended for low-level I/O and must be
          applied to a file descriptor as returned by os.open()
          or pipe().  To read a "file object" returned
          by the built-in function open() or by
          popen() or fdopen(), or sys.stdin,
          use its read() or readline().
          methods.

 -- Function: os.tcgetpgrp (fd)
     Return the process group associated with the terminal given by
     _fd_ (an open file descriptor as returned by os.open()).

     Availability: Unix.

 -- Function: os.tcsetpgrp (fd, pg)
     Set the process group associated with the terminal given by _fd_
     (an open file descriptor as returned by os.open()) to
     _pg_.

     Availability: Unix.

 -- Function: os.ttyname (fd)
     Return a string which specifies the terminal device associated with
     file descriptor _fd_.  If _fd_ is not associated with a terminal
     device, an exception is raised.

     Availability: Unix.

 -- Function: os.write (fd, str)
     Write the string _str_ to file descriptor _fd_. Return the number
     of bytes actually written.

     Availability: Unix, Windows.

          Note: This function is intended for low-level I/O and must be
          applied to a file descriptor as returned by os.open()
          or pipe().  To write a "file object"
          returned by the built-in function open() or by
          popen() or fdopen(), or
          sys.stdout or sys.stderr, use its
          write() method.

5.15.1.4 `open()' flag constants
................................

The following constants are options for the _flags_ parameter to the
open() function.  They can be combined using the bitwise OR
operator `|'.  Some of them are not available on all platforms.  For
descriptions of their availability and use, consult the `open(2)'
manual page on Unix or the MSDN
(http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx)on Windows.

 -- Data: os.O_RDONLY
 -- Data: os.O_WRONLY
 -- Data: os.O_RDWR
 -- Data: os.O_APPEND
 -- Data: os.O_CREAT
 -- Data: os.O_EXCL
 -- Data: os.O_TRUNC
     These constants are available on Unix and Windows.

 -- Data: os.O_DSYNC
 -- Data: os.O_RSYNC
 -- Data: os.O_SYNC
 -- Data: os.O_NDELAY
 -- Data: os.O_NONBLOCK
 -- Data: os.O_NOCTTY
 -- Data: os.O_SHLOCK
 -- Data: os.O_EXLOCK
     These constants are only available on Unix.

 -- Data: os.O_BINARY
 -- Data: os.O_NOINHERIT
 -- Data: os.O_SHORT_LIVED
 -- Data: os.O_TEMPORARY
 -- Data: os.O_RANDOM
 -- Data: os.O_SEQUENTIAL
 -- Data: os.O_TEXT
     These constants are only available on Windows.

 -- Data: os.O_ASYNC
 -- Data: os.O_DIRECT
 -- Data: os.O_DIRECTORY
 -- Data: os.O_NOFOLLOW
 -- Data: os.O_NOATIME
     These constants are GNU extensions and not present if they are not
     defined by the C library.

5.15.1.5 Files and Directories
..............................

 -- Function: os.access (path, mode)
     Use the real uid/gid to test for access to _path_.  Note that most
     operations will use the effective uid/gid, therefore this routine
     can be used in a suid/sgid environment to test if the invoking
     user has the specified access to _path_.  _mode_ should be
     F_OK to test the existence of _path_, or it can be the
     inclusive OR of one or more of R_OK, W_OK,
     and X_OK to test permissions.  Return True
     if access is allowed, see False if not. See the Unix
     man page `access(2)' for more information.

     Availability: Unix, Windows.

          Note: Using access() to check if a user is
          authorized to e.g. open a file before actually doing so using
          open() creates a security hole, because the user
          might exploit the short time interval between checking and
          opening the file to manipulate it.

          Note: I/O operations may fail even when access().
          indicates that they would succeed, particularly for
          operations on network filesystems which may have permissions
          semantics beyond the usual POSIX permission-bit model.

 -- Data: os.F_OK
     Value to pass as the _mode_ parameter of access() to
     test the existence of _path_.

 -- Data: os.R_OK
     Value to include in the _mode_ parameter of access().
     to test the readability of _path_.

 -- Data: os.W_OK
     Value to include in the _mode_ parameter of access().
     to test the writability of _path_.

 -- Data: os.X_OK
     Value to include in the _mode_ parameter of access().
     to determine if _path_ can be executed.

 -- Function: os.chdir (path)
     Change the current working directory to _path_.

     Availability: Unix, Windows.

 -- Function: os.fchdir (fd)
     Change the current working directory to the directory represented
     by the file descriptor _fd_.  The descriptor must refer to an
     opened directory, not an open file.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.getcwd ()
     Return a string representing the current working directory.

     Availability: Unix, Windows.

 -- Function: os.getcwdu ()
     Return a Unicode object representing the current working directory.

     Availability: Unix, Windows.

     New in version 2.3.

 -- Function: os.chflags (path, flags)
     Set the flags of _path_ to the numeric _flags_. _flags_ may take a
     combination (bitwise OR) of the following values (as defined in
     the stat module):

        * `UF_NODUMP'

        * `UF_IMMUTABLE'

        * `UF_APPEND'

        * `UF_OPAQUE'

        * `UF_NOUNLINK'

        * `SF_ARCHIVED'

        * `SF_IMMUTABLE'

        * `SF_APPEND'

        * `SF_NOUNLINK'

        * `SF_SNAPSHOT'

     Availability: Unix.

     New in version 2.6.

 -- Function: os.chroot (path)
     Change the root directory of the current process to _path_.
     Availability: Unix.

     New in version 2.2.

 -- Function: os.chmod (path, mode)
     Change the mode of _path_ to the numeric _mode_. _mode_ may take
     one of the following values (as defined in the stat.
     module) or bitwise ORed combinations of them:

        * stat.S_ISUID.

        * stat.S_ISGID.

        * stat.S_ENFMT.

        * stat.S_ISVTX.

        * stat.S_IREAD.

        * stat.S_IWRITE.

        * stat.S_IEXEC.

        * stat.S_IRWXU.

        * stat.S_IRUSR.

        * stat.S_IWUSR.

        * stat.S_IXUSR.

        * stat.S_IRWXG.

        * stat.S_IRGRP.

        * stat.S_IWGRP.

        * stat.S_IXGRP.

        * stat.S_IRWXO.

        * stat.S_IROTH.

        * stat.S_IWOTH.

        * stat.S_IXOTH.

     Availability: Unix, Windows.

          Note: Although Windows supports chmod(), you can
          only  set the file's read-only flag with it (via the
          `stat.S_IWRITE'  and `stat.S_IREAD' constants or a
          corresponding integer value).  All other bits are ignored.

 -- Function: os.chown (path, uid, gid)
     Change the owner and group id of _path_ to the numeric _uid_ and
     _gid_. To leave one of the ids unchanged, set it to -1.

     Availability: Unix.

 -- Function: os.lchflags (path, flags)
     Set the flags of _path_ to the numeric _flags_, like
     chflags(), but do not follow symbolic links.

     Availability: Unix.

     New in version 2.6.

 -- Function: os.lchmod (path, mode)
     Change the mode of _path_ to the numeric _mode_. If path is a
     symlink, this affects the symlink rather than the target. See the
     docs for chmod().  for possible values of _mode_.

     Availability: Unix.

     New in version 2.6.

 -- Function: os.lchown (path, uid, gid)
     Change the owner and group id of _path_ to the numeric _uid_ and
     _gid_. This function will not follow symbolic links.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.link (source, link_name)
     Create a hard link pointing to _source_ named _link_name_.

     Availability: Unix.

 -- Function: os.listdir (path)
     Return a list containing the names of the entries in the directory
     given by _path_.  The list is in arbitrary order.  It does not
     include the special entries `'.'' and `'..'' even if they are
     present in the directory.

     Availability: Unix, Windows.

     Changed in version 2.3: On Windows NT/2k/XP and Unix, if _path_ is
     a Unicode object, the result will be a list of Unicode objects.
     Undecodable filenames will still be returned as string objects.

 -- Function: os.lstat (path)
     Like stat(), but do not follow symbolic links.  This is
     an alias for stat() on platforms that do not support
     symbolic links, such as Windows.

 -- Function: os.mkfifo (path[, mode])
     Create a FIFO (a named pipe) named _path_ with numeric mode
     _mode_.  The default _mode_ is `0666' (octal).  The current umask
     value is first masked out from the mode.

     Availability: Unix.

     FIFOs are pipes that can be accessed like regular files.  FIFOs
     exist until they are deleted (for example with os.unlink()).
     Generally, FIFOs are used as rendezvous between "client"
     and "server" type processes: the server opens the FIFO for
     reading, and the client opens it for writing.  Note that
     mkfifo().  doesn't open the FIFO -- it just creates the
     rendezvous point.

 -- Function: os.mknod (filename[, mode=0600, device])
     Create a filesystem node (file, device special file or named pipe)
     named _filename_. _mode_ specifies both the permissions to use and
     the type of node to be created, being combined (bitwise OR) with
     one of `stat.S_IFREG', `stat.S_IFCHR', `stat.S_IFBLK', and
     `stat.S_IFIFO' (those constants are available in stat).
     For `stat.S_IFCHR' and `stat.S_IFBLK', _device_ defines the newly
     created device special file (probably using os.makedev()),
     otherwise it is ignored.

     New in version 2.3.

 -- Function: os.major (device)
     Extract the device major number from a raw device number (usually
     the `st_dev' or `st_rdev' field from `stat').

     New in version 2.3.

 -- Function: os.minor (device)
     Extract the device minor number from a raw device number (usually
     the `st_dev' or `st_rdev' field from `stat').

     New in version 2.3.

 -- Function: os.makedev (major, minor)
     Compose a raw device number from the major and minor device
     numbers.

     New in version 2.3.

 -- Function: os.mkdir (path[, mode])
     Create a directory named _path_ with numeric mode _mode_. The
     default _mode_ is `0777' (octal).  On some systems, _mode_ is
     ignored.  Where it is used, the current umask value is first
     masked out.  If the directory already exists, see OSError.
     is raised.

     It is also possible to create temporary directories; see the
     tempfile module's tempfile.mkdtemp() function.

     Availability: Unix, Windows.

 -- Function: os.makedirs (path[, mode])
     Recursive directory creation function.  Like mkdir(),
     but makes all intermediate-level directories needed to contain the
     leaf directory.  Throws an error exception if the leaf
     directory already exists or cannot be created.  The default _mode_
     is `0777' (octal).  On some systems, _mode_ is ignored. Where it
     is used, the current umask value is first masked out.

          Note: makedirs() will become confused if the
          path elements to create include os.pardir.

     New in version 1.5.2.

     Changed in version 2.3: This function now handles UNC paths
     correctly.

 -- Function: os.pathconf (path, name)
     Return system configuration information relevant to a named file.
     _name_ specifies the configuration value to retrieve; it may be a
     string which is the name of a defined system value; these names
     are specified in a number of standards (POSIX.1, Unix 95, Unix 98,
     and others).  Some platforms define additional names as well.  The
     names known to the host operating system are given in the
     `pathconf_names' dictionary.  For configuration variables not
     included in that mapping, passing an integer for _name_ is also
     accepted.

     If _name_ is a string and is not known, see ValueError is
     raised.  If a specific value for _name_ is not supported by the
     host system, even if it is included in `pathconf_names', an
     OSError is raised with errno.EINVAL for the
     error number.

     Availability: Unix.

 -- Data: os.pathconf_names
     Dictionary mapping names accepted by pathconf() and
     fpathconf() to the integer values defined for those
     names by the host operating system.  This can be used to determine
     the set of names known to the system. Availability: Unix.

 -- Function: os.readlink (path)
     Return a string representing the path to which the symbolic link
     points.  The result may be either an absolute or relative
     pathname; if it is relative, it may be converted to an absolute
     pathname using `os.path.join(os.path.dirname(path), result)'.

     Changed in version 2.6: If the _path_ is a Unicode object the
     result will also be a Unicode object.

     Availability: Unix.

 -- Function: os.remove (path)
     Remove (delete) the file _path_.  If _path_ is a directory, see
     OSError is raised; see rmdir() below to remove a
     directory.  This is identical to the unlink().
     function documented below.  On Windows, attempting to remove a
     file that is in use causes an exception to be raised; on Unix, the
     directory entry is removed but the storage allocated to the file
     is not made available until the original file is no longer in use.

     Availability: Unix, Windows.

 -- Function: os.removedirs (path)
     Remove directories recursively.  Works like rmdir().
     except that, if the leaf directory is successfully removed, see
     removedirs().  tries to successively remove every parent
     directory mentioned in  _path_ until an error is raised (which is
     ignored, because it generally means that a parent directory is not
     empty). For example, `os.removedirs('foo/bar/baz')' will first
     remove the directory `'foo/bar/baz'', and then remove `'foo/bar''
     and `'foo'' if they are empty. Raises OSError if the
     leaf directory could not be successfully removed.

     New in version 1.5.2.

 -- Function: os.rename (src, dst)
     Rename the file or directory _src_ to _dst_.  If _dst_ is a
     directory, see OSError will be raised.  On Unix, if _dst_
     exists and is a file, it will be replaced silently if the user has
     permission.  The operation may fail on some Unix flavors if _src_
     and _dst_ are on different filesystems.  If successful, the
     renaming will be an atomic operation (this is a POSIX
     requirement).  On Windows, if _dst_ already exists, see OSError
     will be raised even if it is a file; there may be no way to
     implement an atomic rename when _dst_ names an existing file.

     Availability: Unix, Windows.

 -- Function: os.renames (old, new)
     Recursive directory or file renaming function. Works like
     rename(), except creation of any intermediate directories
     needed to make the new pathname good is attempted first. After the
     rename, directories corresponding to rightmost path segments of
     the old name will be pruned away using removedirs().

     New in version 1.5.2.

          Note: This function can fail with the new directory structure
          made if you lack permissions needed to remove the leaf
          directory or file.

 -- Function: os.rmdir (path)
     Remove (delete) the directory _path_.  Only works when the
     directory is empty, otherwise, see OSError is raised.  In
     order to remove whole directory trees, see shutil.rmtree().
     can be used.

     Availability: Unix, Windows.

 -- Function: os.stat (path)
     Perform a `stat()' system call on the given path.  The return
     value is an object whose attributes correspond to the members of
     the `stat' structure, namely: `st_mode' (protection bits),
     `st_ino' (inode number), `st_dev' (device), `st_nlink' (number of
     hard links), `st_uid' (user id of owner), `st_gid' (group id of
     owner), `st_size' (size of file, in bytes), `st_atime' (time of
     most recent access), `st_mtime' (time of most recent content
     modification), `st_ctime' (platform dependent; time of most recent
     metadata change on Unix, or the time of creation on Windows):

         >>> import os
         >>> statinfo = os.stat('somefile.txt')
         >>> statinfo
         (33188, 422511L, 769L, 1, 1032, 100, 926L, 1105022698,1105022732, 1105022732)
         >>> statinfo.st_size
         926L
         >>>

     Changed in version 2.3: If stat_float_times() returns
     `True', the time values are floats, measuring seconds. Fractions
     of a second may be reported if the system supports that. On Mac
     OS, the times are always floats. See stat_float_times()
     for further discussion.

     On some Unix systems (such as Linux), the following attributes may
     also be available: `st_blocks' (number of blocks allocated for
     file), `st_blksize' (filesystem blocksize), `st_rdev' (type of
     device if an inode device). `st_flags' (user defined flags for
     file).

     On other Unix systems (such as FreeBSD), the following attributes
     may be available (but may be only filled out if root tries to use
     them): `st_gen' (file generation number), `st_birthtime' (time of
     file creation).

     On Mac OS systems, the following attributes may also be available:
     `st_rsize', `st_creator', `st_type'.

     On RISCOS systems, the following attributes are also available:
     `st_ftype' (file type), `st_attrs' (attributes), `st_obtype'
     (object type).

     For backward compatibility, the return value of stat().
     is also accessible as a tuple of at least 10 integers giving the
     most important (and portable) members of the `stat' structure, in
     the order `st_mode', `st_ino', `st_dev', `st_nlink', `st_uid',
     `st_gid', `st_size', `st_atime', `st_mtime', `st_ctime'. More
     items may be added at the end by some implementations.  The
     standard module stat defines functions and constants
     that are useful for extracting information from a `stat'
     structure. (On Windows, some items are filled with dummy values.)

          Note: The exact meaning and resolution of the `st_atime',
          `st_mtime', and `st_ctime' members depends on the operating
          system and the file system.  For example, on Windows systems
          using the FAT or FAT32 file systems, `st_mtime' has 2-second
          resolution, and `st_atime' has only 1-day resolution.  See
          your operating system documentation for details.

     Availability: Unix, Windows.

     Changed in version 2.2: Added access to values as attributes of
     the returned object.

     Changed in version 2.5: Added `st_gen' and `st_birthtime'.

 -- Function: os.stat_float_times ([newvalue])
     Determine whether `stat_result' represents time stamps as float
     objects.  If _newvalue_ is `True', future calls to stat()
     return floats, if it is `False', future calls return ints. If
     _newvalue_ is omitted, return the current setting.

     For compatibility with older Python versions, accessing
     `stat_result' as a tuple always returns integers.

     Changed in version 2.5: Python now returns float values by
     default. Applications which do not work correctly with floating
     point time stamps can use this function to restore the old
     behaviour.

     The resolution of the timestamps (that is the smallest possible
     fraction) depends on the system. Some systems only support second
     resolution; on these systems, the fraction will always be zero.

     It is recommended that this setting is only changed at program
     startup time in the ___main___ module; libraries should never
     change this setting. If an application uses a library that works
     incorrectly if floating point time stamps are processed, this
     application should turn the feature off until the library has been
     corrected.

 -- Function: os.statvfs (path)
     Perform a `statvfs()' system call on the given path.  The return
     value is an object whose attributes describe the filesystem on the
     given path, and correspond to the members of the `statvfs'
     structure, namely: `f_bsize', `f_frsize', `f_blocks', `f_bfree',
     `f_bavail', `f_files', `f_ffree', `f_favail', `f_flag',
     `f_namemax'.

     For backward compatibility, the return value is also accessible as
     a tuple whose values correspond to the attributes, in the order
     given above. The standard module statvfs defines
     constants that are useful for extracting information from a
     `statvfs' structure when accessing it as a sequence; this remains
     useful when writing code that needs to work with versions of Python
     that don't support accessing the fields as attributes.

     Availability: Unix.

     Changed in version 2.2: Added access to values as attributes of
     the returned object.

 -- Function: os.symlink (source, link_name)
     Create a symbolic link pointing to _source_ named _link_name_.

     Availability: Unix.

 -- Function: os.tempnam ([dir[, prefix]])
     Return a unique path name that is reasonable for creating a
     temporary file.  This will be an absolute path that names a
     potential directory entry in the directory _dir_ or a common
     location for temporary files if _dir_ is omitted or `None'.  If
     given and not `None', _prefix_ is used to provide a short prefix
     to the filename.  Applications are responsible for properly
     creating and managing files created using paths returned by
     tempnam().; no automatic cleanup is provided. On Unix, the
     environment variable `TMPDIR' overrides _dir_, while on Windows `TMP'
     is used.  The specific behavior of this function depends on the C
     library implementation; some aspects are underspecified in system
     documentation.

          Warning: Use of tempnam() is vulnerable to
          symlink attacks; consider using tmpfile().
          (section File Object Creation) instead.

     Availability: Unix, Windows.

 -- Function: os.tmpnam ()
     Return a unique path name that is reasonable for creating a
     temporary file.  This will be an absolute path that names a
     potential directory entry in a common location for temporary
     files.  Applications are responsible for properly creating and
     managing files created using paths returned by tmpnam().;
     no automatic cleanup is provided.

          Warning: Use of tmpnam() is vulnerable to
          symlink attacks; consider using tmpfile().
          (section File Object Creation) instead.

     Availability: Unix, Windows.  This function probably shouldn't be
     used on Windows, though: Microsoft's implementation of
     tmpnam() always creates a name in the root directory of the
     current drive, and that's generally a poor location for a temp
     file (depending on privileges, you may not even be able to open a
     file using this name).

 -- Data: os.TMP_MAX
     The maximum number of unique names that tmpnam() will
     generate before reusing names.

 -- Function: os.unlink (path)
     Remove (delete) the file _path_.  This is the same function as
     remove().; the unlink() name is its
     traditional Unix name.

     Availability: Unix, Windows.

 -- Function: os.utime (path, times)
     Set the access and modified times of the file specified by _path_.
     If _times_ is `None', then the file's access and modified times
     are set to the current time. (The effect is similar to running the
     Unix program *touch* on the path.)  Otherwise, _times_ must be a
     2-tuple of numbers, of the form `(atime, mtime)' which is used to
     set the access and modified times, respectively. Whether a
     directory can be given for _path_ depends on whether the operating
     system implements directories as files (for example, Windows does
     not).  Note that the exact times you set here may not be returned
     by a subsequent stat() call, depending on the
     resolution with which your operating system records access and
     modification times; see stat().

     Changed in version 2.0: Added support for `None' for _times_.

     Availability: Unix, Windows.

 -- Function: os.walk (top[, topdown=True[, onerror=None[,
          followlinks=False]]])
     Generate the file names in a directory tree by walking the tree
     either top-down or bottom-up. For each directory in the tree
     rooted at directory _top_ (including _top_ itself), it yields a
     3-tuple `(dirpath, dirnames, filenames)'.

     _dirpath_ is a string, the path to the directory.  _dirnames_ is a
     list of the names of the subdirectories in _dirpath_ (excluding
     `'.'' and `'..'').  _filenames_ is a list of the names of the
     non-directory files in _dirpath_.  Note that the names in the
     lists contain no path components.  To get a full path (which
     begins with _top_) to a file or directory in _dirpath_, do
     `os.path.join(dirpath, name)'.

     If optional argument _topdown_ is `True' or not specified, the
     triple for a directory is generated before the triples for any of
     its subdirectories (directories are generated top-down).  If
     _topdown_ is `False', the triple for a directory is generated
     after the triples for all of its subdirectories (directories are
     generated bottom-up).

     When _topdown_ is `True', the caller can modify the _dirnames_
     list in-place (perhaps using del or slice assignment),
     and walk() will only recurse into the subdirectories
     whose names remain in _dirnames_; this can be used to prune the
     search, impose a specific order of visiting, or even to inform
     walk() about directories the caller creates or renames
     before it resumes walk() again.  Modifying _dirnames_
     when _topdown_ is `False' is ineffective, because in bottom-up
     mode the directories in _dirnames_ are generated before _dirpath_
     itself is generated.

     By default errors from the listdir() call are ignored.
     If optional argument _onerror_ is specified, it should be a
     function; it will be called with one argument, an OSError
     instance.  It can report the error to continue with the walk,
     or raise the exception to abort the walk.  Note that the filename
     is available as the `filename' attribute of the exception object.

     By default, see walk() will not walk down into symbolic
     links that resolve to directories. Set _followlinks_ to `True' to
     visit directories pointed to by symlinks, on systems that support
     them.

     New in version 2.6: The _followlinks_ parameter.

          Note: Be aware that setting _followlinks_ to `True' can lead
          to infinite recursion if a link points to a parent directory
          of itself. walk() does not keep track of the
          directories it visited already.

          Note: If you pass a relative pathname, don't change the
          current working directory between resumptions of
          walk().  walk() never changes the current
          directory, and assumes that its caller doesn't either.

     This example displays the number of bytes taken by non-directory
     files in each directory under the starting directory, except that
     it doesn't look under any CVS subdirectory:

         import os
         from os.path import join, getsize
         for root, dirs, files in os.walk('python/Lib/email'):
             print root, "consumes",
             print sum(getsize(join(root, name)) for name in files),
             print "bytes in", len(files), "non-directory files"
             if 'CVS' in dirs:
                 dirs.remove('CVS')  # don't visit CVS directories

     In the next example, walking the tree bottom-up is essential:
     rmdir().  doesn't allow deleting a directory before the
     directory is empty:

         # Delete everything reachable from the directory named in "top",
         # assuming there are no symbolic links.
         # CAUTION:  This is dangerous!  For example, if top == '/', it
         # could delete all your disk files.
         import os
         for root, dirs, files in os.walk(top, topdown=False):
             for name in files:
                 os.remove(os.path.join(root, name))
             for name in dirs:
                 os.rmdir(os.path.join(root, name))

     New in version 2.3.

5.15.1.6 Process Management
...........................

These functions may be used to create and manage processes.

  The various `exec*()' functions take a list of arguments for the new
program loaded into the process.  In each case, the first of these
arguments is passed to the new program as its own name rather than as
an argument a user may have typed on a command line.  For the C
programmer, this is the `argv[0]' passed to a program's `main()'.  For
example, `os.execv('/bin/echo', ['foo', 'bar'])' will only print `bar'
on standard output; `foo' will seem to be ignored.

 -- Function: os.abort ()
     Generate a `SIGABRT' signal to the current process.  On Unix, the
     default behavior is to produce a core dump; on Windows, the
     process immediately returns an exit code of `3'.  Be aware that
     programs which use signal.signal().  to register a
     handler for `SIGABRT' will behave differently.

     Availability: Unix, Windows.

 -- Function: os.execl (path, arg0, arg1, ...)
 -- Function: os.execle (path, arg0, arg1, ..., env)
 -- Function: os.execlp (file, arg0, arg1, ...)
 -- Function: os.execlpe (file, arg0, arg1, ..., env)
 -- Function: os.execv (path, args)
 -- Function: os.execve (path, args, env)
 -- Function: os.execvp (file, args)
 -- Function: os.execvpe (file, args, env)
     These functions all execute a new program, replacing the current
     process; they do not return.  On Unix, the new executable is
     loaded into the current process, and will have the same process id
     as the caller.  Errors will be reported as OSError.
     exceptions.

     The current process is replaced immediately. Open file objects and
     descriptors are not flushed, so if there may be data buffered on
     these open files, you should flush them using `sys.stdout.flush()'
     or os.fsync() before calling an `exec*()' function.

     The "l" and "v" variants of the `exec*()' functions differ in how
     command-line arguments are passed.  The "l" variants are perhaps
     the easiest to work with if the number of parameters is fixed when
     the code is written; the individual parameters simply become
     additional parameters to the `execl*()' functions.  The "v"
     variants are good when the number of parameters is variable, with
     the arguments being passed in a list or tuple as the _args_
     parameter.  In either case, the arguments to the child process
     should start with the name of the command being run, but this is
     not enforced.

     The variants which include a "p" near the end (see execlp(),
     see execlpe(), execvp(), and
     execvpe()) will use the `PATH' environment variable to
     locate the program _file_.  When the environment is being replaced
     (using one of the `exec*e()' variants, discussed in the next
     paragraph), the new environment is used as the source of the `PATH'
     variable. The other variants, see execl(), execle(),
     see execv(), and execve(), will not use
     the `PATH' variable to locate the executable; _path_ must contain
     an appropriate absolute or relative path.

     For execle(), execlpe(), execve(),
     and execvpe(). (note that these all end in "e"),
     the _env_ parameter must be a mapping which is used to define the
     environment variables for the new process (these are used instead
     of the current process' environment); the functions execl(),
     see execlp(), execv(), and
     execvp() all cause the new process to inherit the
     environment of the current process.

     Availability: Unix, Windows.

 -- Function: os._exit (n)
     Exit the process with status _n_, without calling cleanup
     handlers, flushing stdio buffers, etc.

     Availability: Unix, Windows.

          Note: The standard way to exit is `sys.exit(n)'.
          _exit() should normally only be used in the child
          process after a fork().

  The following exit codes are defined and can be used with
_exit(), although they are not required.  These are typically used
for system programs written in Python, such as a mail server's external
command delivery program.

     Note: Some of these may not be available on all Unix platforms,
     since there is some variation.  These constants are defined where
     they are defined by the underlying platform.

 -- Data: os.EX_OK
     Exit code that means no error occurred.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_USAGE
     Exit code that means the command was used incorrectly, such as
     when the wrong number of arguments are given.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_DATAERR
     Exit code that means the input data was incorrect.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_NOINPUT
     Exit code that means an input file did not exist or was not
     readable.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_NOUSER
     Exit code that means a specified user did not exist.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_NOHOST
     Exit code that means a specified host did not exist.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_UNAVAILABLE
     Exit code that means that a required service is unavailable.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_SOFTWARE
     Exit code that means an internal software error was detected.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_OSERR
     Exit code that means an operating system error was detected, such
     as the inability to fork or create a pipe.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_OSFILE
     Exit code that means some system file did not exist, could not be
     opened, or had some other kind of error.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_CANTCREAT
     Exit code that means a user specified output file could not be
     created.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_IOERR
     Exit code that means that an error occurred while doing I/O on
     some file.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_TEMPFAIL
     Exit code that means a temporary failure occurred.  This indicates
     something that may not really be an error, such as a network
     connection that couldn't be made during a retryable operation.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_PROTOCOL
     Exit code that means that a protocol exchange was illegal,
     invalid, or not understood.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_NOPERM
     Exit code that means that there were insufficient permissions to
     perform the operation (but not intended for file system problems).

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_CONFIG
     Exit code that means that some kind of configuration error
     occurred.

     Availability: Unix.

     New in version 2.3.

 -- Data: os.EX_NOTFOUND
     Exit code that means something like "an entry was not found".

     Availability: Unix.

     New in version 2.3.

 -- Function: os.fork ()
     Fork a child process.  Return `0' in the child and the child's
     process id in the parent.  If an error occurs OSError.
     is raised.

     Note that some platforms including FreeBSD <= 6.3, Cygwin and OS/2
     EMX have known issues when using fork() from a thread.

     Availability: Unix.

 -- Function: os.forkpty ()
     Fork a child process, using a new pseudo-terminal as the child's
     controlling terminal. Return a pair of `(pid, fd)', where _pid_ is
     `0' in the child, the new child's process id in the parent, and
     _fd_ is the file descriptor of the master end of the
     pseudo-terminal.  For a more portable approach, use the pty
     module.  If an error occurs OSError is raised.

     Availability: some flavors of Unix.

 -- Function: os.kill (pid, sig)
     Send signal _sig_ to the process _pid_.  Constants for the
     specific signals available on the host platform are defined in the
     signal module.

     Windows: The signal.CTRL_C_EVENT and
     signal.CTRL_BREAK_EVENT signals are special signals which
     can only be sent to console processes which share a common console
     window, e.g., some subprocesses. Any other value for _sig_ will
     cause the process to be unconditionally killed by the
     TerminateProcess API, and the exit code will be set to _sig_. The
     Windows version of kill() additionally takes process
     handles to be killed.

     New in version 2.7: Windows support

 -- Function: os.killpg (pgid, sig)
     Send the signal _sig_ to the process group _pgid_.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.nice (increment)
     Add _increment_ to the process's "niceness".  Return the new
     niceness.

     Availability: Unix.

 -- Function: os.plock (op)
     Lock program segments into memory.  The value of _op_ (defined in
     `<sys/lock.h>') determines which segments are locked.

     Availability: Unix.

 -- Function: os.popen (...)
 -- Function: os.popen2 (...)
 -- Function: os.popen3 (...)
 -- Function: os.popen4 (...)
     Run child processes, returning opened pipes for communications.
     These functions are described in section File Object Creation.

 -- Function: os.spawnl (mode, path, ...)
 -- Function: os.spawnle (mode, path, ..., env)
 -- Function: os.spawnlp (mode, file, ...)
 -- Function: os.spawnlpe (mode, file, ..., env)
 -- Function: os.spawnv (mode, path, args)
 -- Function: os.spawnve (mode, path, args, env)
 -- Function: os.spawnvp (mode, file, args)
 -- Function: os.spawnvpe (mode, file, args, env)
     Execute the program _path_ in a new process.

     (Note that the subprocess module provides more
     powerful facilities for spawning new processes and retrieving
     their results; using that module is preferable to using these
     functions.  Check especially the Replacing Older Functions with the subprocess Module
     section.)

     If _mode_ is P_NOWAIT, this function returns the
     process id of the new process; if _mode_ is P_WAIT,
     returns the process's exit code if it exits normally, or
     `-signal', where _signal_ is the signal that killed the process.
     On Windows, the process id will actually be the process handle, so
     can be used with the waitpid() function.

     The "l" and "v" variants of the `spawn*()' functions differ in how
     command-line arguments are passed.  The "l" variants are perhaps
     the easiest to work with if the number of parameters is fixed when
     the code is written; the individual parameters simply become
     additional parameters to the `spawnl*()' functions.  The "v"
     variants are good when the number of parameters is variable, with
     the arguments being passed in a list or tuple as the _args_
     parameter.  In either case, the arguments to the child process must
     start with the name of the command being run.

     The variants which include a second "p" near the end (see
     spawnlp(), spawnlpe(), spawnvp(),
     and spawnvpe()) will use the `PATH' environment
     variable to locate the program _file_.  When the environment is
     being replaced (using one of the `spawn*e()' variants, discussed
     in the next paragraph), the new environment is used as the source
     of the `PATH' variable.  The other variants, see spawnl(),
     spawnle(), spawnv(), and spawnve(),
     will not use the `PATH' variable to locate the executable;
     _path_ must contain an appropriate absolute or relative path.

     For spawnle(), spawnlpe(),
     spawnve(), and spawnvpe().  (note that these all
     end in "e"), the _env_ parameter must be a mapping which is used
     to define the environment variables for the new process (they are
     used instead of the current process' environment); the functions
     spawnl(), spawnlp(), spawnv(),
     and spawnvp() all cause the new process to inherit
     the environment of the current process.  Note that keys and values
     in the _env_ dictionary must be strings; invalid keys or values
     will cause the function to fail, with a return value of `127'.

     As an example, the following calls to spawnlp() and
     spawnvpe() are equivalent:

         import os
         os.spawnlp(os.P_WAIT, 'cp', 'cp', 'index.html', '/dev/null')

         L = ['cp', 'index.html', '/dev/null']
         os.spawnvpe(os.P_WAIT, 'cp', L, os.environ)

     Availability: Unix, Windows.  See spawnlp(),
     spawnlpe(), spawnvp().  and spawnvpe()
     are not available on Windows.

     New in version 1.6.

 -- Data: os.P_NOWAIT
 -- Data: os.P_NOWAITO
     Possible values for the _mode_ parameter to the `spawn*()' family
     of functions.  If either of these values is given, the `spawn*()'
     functions will return as soon as the new process has been created,
     with the process id as the return value.

     Availability: Unix, Windows.

     New in version 1.6.

 -- Data: os.P_WAIT
     Possible value for the _mode_ parameter to the `spawn*()' family of
     functions.  If this is given as _mode_, the `spawn*()' functions
     will not return until the new process has run to completion and
     will return the exit code of the process the run is successful, or
     `-signal' if a signal kills the process.

     Availability: Unix, Windows.

     New in version 1.6.

 -- Data: os.P_DETACH
 -- Data: os.P_OVERLAY
     Possible values for the _mode_ parameter to the `spawn*()' family
     of functions.  These are less portable than those listed above.
     See P_DETACH.  is similar to P_NOWAIT, but the
     new process is detached from the console of the calling process.
     If P_OVERLAY is used, the current process will be
     replaced; the `spawn*()' function will not return.

     Availability: Windows.

     New in version 1.6.

 -- Function: os.startfile (path[, operation])
     Start a file with its associated application.

     When _operation_ is not specified or `'open'', this acts like
     double-clicking the file in Windows Explorer, or giving the file
     name as an argument to the *start* command from the interactive
     command shell: the file is opened with whatever application (if
     any) its extension is associated.

     When another _operation_ is given, it must be a "command verb"
     that specifies what should be done with the file. Common verbs
     documented by Microsoft are `'print'' and  `'edit'' (to be used on
     files) as well as `'explore'' and `'find'' (to be used on
     directories).

     startfile() returns as soon as the associated
     application is launched.  There is no option to wait for the
     application to close, and no way to retrieve the application's
     exit status.  The _path_ parameter is relative to the current
     directory.  If you want to use an absolute path, make sure the
     first character is not a slash (`'/''); the underlying Win32
     `ShellExecute()' function doesn't work if it is.  Use the
     os.path.normpath() function to ensure that the path is
     properly encoded for Win32.

     Availability: Windows.

     New in version 2.0.

     New in version 2.5: The _operation_ parameter.

 -- Function: os.system (command)
     Execute the command (a string) in a subshell.  This is implemented
     by calling the Standard C function `system()', and has the same
     limitations.  Changes to sys.stdin, etc. are not
     reflected in the environment of the executed command.

     On Unix, the return value is the exit status of the process
     encoded in the format specified for wait().  Note that
     POSIX does not specify the meaning of the return value of the C
     `system()' function, so the return value of the Python function is
     system-dependent.

     On Windows, the return value is that returned by the system shell
     after running _command_, given by the Windows environment variable `COMSPEC':
     on *command.com* systems (Windows 95, 98 and ME) this is always
     `0'; on *cmd.exe* systems (Windows NT, 2000 and XP) this is the
     exit status of the command run; on systems using a non-native
     shell, consult your shell documentation.

     The subprocess module provides more powerful
     facilities for spawning new processes and retrieving their
     results; using that module is preferable to using this function.
     See the Replacing Older Functions with the subprocess Module
     section in the subprocess documentation
     for some helpful recipes.

     Availability: Unix, Windows.

 -- Function: os.times ()
     Return a 5-tuple of floating point numbers indicating accumulated
     (processor or other) times, in seconds.  The items are: user time,
     system time, children's user time, children's system time, and
     elapsed real time since a fixed point in the past, in that order.
     See the Unix manual page `times(2)' or the corresponding Windows
     Platform API documentation.  On Windows, only the first two items
     are filled, the others are zero.

     Availability: Unix, Windows

 -- Function: os.wait ()
     Wait for completion of a child process, and return a tuple
     containing its pid and exit status indication: a 16-bit number,
     whose low byte is the signal number that killed the process, and
     whose high byte is the exit status (if the signal number is zero);
     the high bit of the low byte is set if a core file was produced.

     Availability: Unix.

 -- Function: os.waitpid (pid, options)
     The details of this function differ on Unix and Windows.

     On Unix: Wait for completion of a child process given by process
     id _pid_, and return a tuple containing its process id and exit
     status indication (encoded as for wait()).  The
     semantics of the call are affected by the value of the integer
     _options_, which should be `0' for normal operation.

     If _pid_ is greater than `0', see waitpid() requests
     status information for that specific process.  If _pid_ is `0',
     the request is for the status of any child in the process group of
     the current process.  If _pid_ is `-1', the request pertains to
     any child of the current process.  If _pid_ is less than `-1',
     status is requested for any process in the process group `-pid'
     (the absolute value of _pid_).

     An OSError is raised with the value of errno when the
     syscall returns -1.

     On Windows: Wait for completion of a process given by process
     handle _pid_, and return a tuple containing _pid_, and its exit
     status shifted left by 8 bits (shifting makes cross-platform use
     of the function easier). A _pid_ less than or equal to `0' has no
     special meaning on Windows, and raises an exception. The value of
     integer _options_ has no effect. _pid_ can refer to any process
     whose id is known, not necessarily a child process. The `spawn()'
     functions called with P_NOWAIT return suitable
     process handles.

 -- Function: os.wait3 ([options])
     Similar to waitpid(), except no process id argument is
     given and a 3-element tuple containing the child's process id,
     exit status indication, and resource usage information is
     returned.  Refer to resource.`getrusage()' for details
     on resource usage information.  The option argument is the same as
     that provided to waitpid() and wait4().

     Availability: Unix.

     New in version 2.5.

 -- Function: os.wait4 (pid, options)
     Similar to waitpid(), except a 3-element tuple,
     containing the child's process id, exit status indication, and
     resource usage information is returned.  Refer to resource.
     `getrusage()' for details on resource usage information.  The
     arguments to wait4() are the same as those provided to
     waitpid().

     Availability: Unix.

     New in version 2.5.

 -- Data: os.WNOHANG
     The option for waitpid() to return immediately if no
     child process status is available immediately. The function
     returns `(0, 0)' in this case.

     Availability: Unix.

 -- Data: os.WCONTINUED
     This option causes child processes to be reported if they have
     been continued from a job control stop since their status was last
     reported.

     Availability: Some Unix systems.

     New in version 2.3.

 -- Data: os.WUNTRACED
     This option causes child processes to be reported if they have
     been stopped but their current state has not been reported since
     they were stopped.

     Availability: Unix.

     New in version 2.3.

  The following functions take a process status code as returned by
system(), wait(), or waitpid() as a
parameter.  They may be used to determine the disposition of a process.

 -- Function: os.WCOREDUMP (status)
     Return `True' if a core dump was generated for the process,
     otherwise return `False'.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.WIFCONTINUED (status)
     Return `True' if the process has been continued from a job control
     stop, otherwise return `False'.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.WIFSTOPPED (status)
     Return `True' if the process has been stopped, otherwise return
     `False'.

     Availability: Unix.

 -- Function: os.WIFSIGNALED (status)
     Return `True' if the process exited due to a signal, otherwise
     return `False'.

     Availability: Unix.

 -- Function: os.WIFEXITED (status)
     Return `True' if the process exited using the `exit(2)' system
     call, otherwise return `False'.

     Availability: Unix.

 -- Function: os.WEXITSTATUS (status)
     If `WIFEXITED(status)' is true, return the integer parameter to the
     `exit(2)' system call.  Otherwise, the return value is meaningless.

     Availability: Unix.

 -- Function: os.WSTOPSIG (status)
     Return the signal which caused the process to stop.

     Availability: Unix.

 -- Function: os.WTERMSIG (status)
     Return the signal which caused the process to exit.

     Availability: Unix.

5.15.1.7 Miscellaneous System Information
.........................................

 -- Function: os.confstr (name)
     Return string-valued system configuration values. _name_ specifies
     the configuration value to retrieve; it may be a string which is
     the name of a defined system value; these names are specified in a
     number of standards (POSIX, Unix 95, Unix 98, and others).  Some
     platforms define additional names as well.  The names known to the
     host operating system are given as the keys of the `confstr_names'
     dictionary.  For configuration variables not included in that
     mapping, passing an integer for _name_ is also accepted.

     If the configuration value specified by _name_ isn't defined,
     `None' is returned.

     If _name_ is a string and is not known, see ValueError is
     raised.  If a specific value for _name_ is not supported by the
     host system, even if it is included in `confstr_names', an
     OSError is raised with errno.EINVAL for the
     error number.

     Availability: Unix

 -- Data: os.confstr_names
     Dictionary mapping names accepted by confstr() to the
     integer values defined for those names by the host operating
     system. This can be used to determine the set of names known to
     the system.

     Availability: Unix.

 -- Function: os.getloadavg ()
     Return the number of processes in the system run queue averaged
     over the last 1, 5, and 15 minutes or raises OSError.
     if the load average was unobtainable.

     Availability: Unix.

     New in version 2.3.

 -- Function: os.sysconf (name)
     Return integer-valued system configuration values. If the
     configuration value specified by _name_ isn't defined, `-1' is
     returned.  The comments regarding the _name_ parameter for
     confstr() apply here as well; the dictionary that provides
     information on the known names is given by `sysconf_names'.

     Availability: Unix.

 -- Data: os.sysconf_names
     Dictionary mapping names accepted by sysconf() to the
     integer values defined for those names by the host operating
     system. This can be used to determine the set of names known to
     the system.

     Availability: Unix.

  The following data values are used to support path manipulation
operations.  These are defined for all platforms.

  Higher-level operations on pathnames are defined in the
os.path module.

 -- Data: os.curdir
     The constant string used by the operating system to refer to the
     current directory. This is `'.'' for Windows and POSIX. Also
     available via os.path.

 -- Data: os.pardir
     The constant string used by the operating system to refer to the
     parent directory. This is `'..'' for Windows and POSIX. Also
     available via os.path.

 -- Data: os.sep
     The character used by the operating system to separate pathname
     components.  This is `'/'' for POSIX and `'\\'' for Windows.  Note
     that knowing this is not sufficient to be able to parse or
     concatenate pathnames -- use os.path.split() and
     os.path.join(). -- but it is occasionally useful. Also
     available via os.path.

 -- Data: os.altsep
     An alternative character used by the operating system to separate
     pathname components, or `None' if only one separator character
     exists.  This is set to `'/'' on Windows systems where `sep' is a
     backslash. Also available via os.path.

 -- Data: os.extsep
     The character which separates the base filename from the
     extension; for example, the `'.'' in `os.py'. Also available via
     os.path.

     New in version 2.2.

 -- Data: os.pathsep
     The character conventionally used by the operating system to
     separate search path components (as in `PATH'), such as `':'' for
     POSIX or `';'' for Windows. Also available via os.path.

 -- Data: os.defpath
     The default search path used by `exec*p*()' and `spawn*p*()' if the
     environment doesn't have a `'PATH'' key. Also available via
     os.path.

 -- Data: os.linesep
     The string used to separate (or, rather, terminate) lines on the
     current platform.  This may be a single character, such as `'\n''
     for POSIX, or multiple characters, for example, `'\r\n'' for
     Windows. Do not use _os.linesep_ as a line terminator when writing
     files opened in text mode (the default); use a single `'\n''
     instead, on all platforms.

 -- Data: os.devnull
     The file path of the null device. For example: `'/dev/null'' for
     POSIX, `'nul'' for Windows.  Also available via os.path.

     New in version 2.4.

5.15.1.8 Miscellaneous Functions
................................

 -- Function: os.urandom (n)
     Return a string of _n_ random bytes suitable for cryptographic use.

     This function returns random bytes from an OS-specific randomness
     source.  The returned data should be unpredictable enough for
     cryptographic applications, though its exact quality depends on
     the OS implementation.  On a UNIX-like system this will query
     /dev/urandom, and on Windows it will use CryptGenRandom.  If a
     randomness source is not found, see NotImplementedError.
     will be raised.

     New in version 2.4.

5.15.2 `io' -- Core tools for working with streams
--------------------------------------------------

New in version 2.6.

  The io module provides the Python interfaces to stream
handling.  Under Python 2.x, this is proposed as an alternative to the
built-in file object, but in Python 3.x it is the default
interface to access files and streams.

     Note: Since this module has been designed primarily for Python
     3.x, you have to be aware that all uses of "bytes" in this
     document refer to the str type (of which `bytes' is an
     alias), and all uses of "text" refer to the unicode.
     type.  Furthermore, those two types are not interchangeable in the
     io. APIs.

  At the top of the I/O hierarchy is the abstract base class
IOBase.  It defines the basic interface to a stream.  Note,
however, that there is no separation between reading and writing to
streams; implementations are allowed to raise an IOError if
they do not support a given operation.

  Extending IOBase is RawIOBase which deals
simply with the reading and writing of raw bytes to a stream.
FileIO subclasses RawIOBase to provide an interface
to files in the machine's file system.

  BufferedIOBase deals with buffering on a raw byte stream
(see RawIOBase).  Its subclasses, see BufferedWriter,
BufferedReader, and BufferedRWPair buffer
streams that are readable, writable, and both readable and writable.
BufferedRandom provides a buffered interface to random
access streams.  BytesIO is a simple stream of in-memory
bytes.

  Another IOBase subclass, see TextIOBase, deals
with streams whose bytes represent text, and handles encoding and
decoding from and to unicode strings.  See TextIOWrapper,
which extends it, is a buffered text interface to a buffered raw
stream (see BufferedIOBase). Finally, see StringIO is
an in-memory stream for unicode text.

  Argument names are not part of the specification, and only the
arguments of open() are intended to be used as keyword
arguments.

5.15.2.1 Module Interface
.........................

 -- Data: io.DEFAULT_BUFFER_SIZE
     An int containing the default buffer size used by the module's
     buffered I/O classes.  open() uses the file's blksize
     (as obtained by os.stat()) if possible.

 -- Function: io.open (file, mode='r', buffering=-1, encoding=None,
          errors=None, newline=None, closefd=True)
     Open _file_ and return a corresponding stream.  If the file cannot
     be opened, an IOError is raised.

     _file_ is either a string giving the pathname (absolute or
     relative to the current working directory) of the file to be
     opened or an integer file descriptor of the file to be wrapped.
     (If a file descriptor is given, it is closed when the returned I/O
     object is closed, unless _closefd_ is set to `False'.)

     _mode_ is an optional string that specifies the mode in which the
     file is opened.  It defaults to `'r'' which means open for reading
     in text mode.  Other common values are `'w'' for writing
     (truncating the file if it already exists), and `'a'' for
     appending (which on _some_ Unix systems, means that _all_ writes
     append to the end of the file regardless of the current seek
     position).  In text mode, if _encoding_ is not specified the
     encoding used is platform dependent. (For reading and writing raw
     bytes use binary mode and leave _encoding_ unspecified.)  The
     available modes are:

     Character     Meaning
     `'r''         open for reading (default)
     `'w''         open for writing, truncating the file first
     `'a''         open for writing, appending to the end of the file if it exists
     `'b''         binary mode
     `'t''         text mode (default)
     `'+''         open a disk file for updating (reading and writing)
     `'U''         universal newline mode (for backwards compatibility; should not be
                   used in new code)

     The default mode is `'rt'' (open for reading text).  For binary
     random access, the mode `'w+b'' opens and truncates the file to 0
     bytes, while `'r+b'' opens the file without truncation.

     Python distinguishes between files opened in binary and text
     modes, even when the underlying operating system doesn't.  Files
     opened in binary mode (including `'b'' in the _mode_ argument)
     return contents as `bytes' objects without any decoding.  In text
     mode (the default, or when `'t'' is included in the _mode_
     argument), the contents of the file are returned as unicode
     strings, the bytes having been first decoded using a
     platform-dependent encoding or using the specified _encoding_ if
     given.

     _buffering_ is an optional integer used to set the buffering
     policy.  Pass 0 to switch buffering off (only allowed in binary
     mode), 1 to select line buffering (only usable in text mode), and
     an integer > 1 to indicate the size of a fixed-size chunk buffer.
     When no _buffering_ argument is given, the default buffering
     policy works as follows:

        * Binary files are buffered in fixed-size chunks; the size of
          the buffer is chosen using a heuristic trying to determine
          the underlying device's "block size" and falling back on
          DEFAULT_BUFFER_SIZE.  On many systems, the buffer
          will typically be 4096 or 8192 bytes long.

        * "Interactive" text files (files for which `isatty()' returns
          True) use line buffering.  Other text files use the policy
          described above for binary files.

     _encoding_ is the name of the encoding used to decode or encode
     the file.  This should only be used in text mode.  The default
     encoding is platform dependent (whatever
     locale.getpreferredencoding() returns), but any encoding
     supported by Python can be used.  See the codecs module
     for the list of supported encodings.

     _errors_ is an optional string that specifies how encoding and
     decoding errors are to be handled-this cannot be used in binary
     mode.  Pass `'strict'' to raise a ValueError exception
     if there is an encoding error (the default of `None' has the same
     effect), or pass `'ignore'' to ignore errors.  (Note that ignoring
     encoding errors can lead to data loss.)  `'replace'' causes a
     replacement marker (such as `'?'') to be inserted where there is
     malformed data.  When writing, `'xmlcharrefreplace'' (replace with
     the appropriate XML character reference) or `'backslashreplace''
     (replace with backslashed escape sequences) can be used.  Any
     other error handling name that has been registered with
     codecs.register_error() is also valid.

     _newline_ controls how universal newlines works (it only applies
     to text mode).  It can be `None', `''', `'\n'', `'\r'', and
     `'\r\n''.  It works as follows:

        * On input, if _newline_ is `None', universal newlines mode is
          enabled.  Lines in the input can end in `'\n'', `'\r'', or
          `'\r\n'', and these are translated into `'\n'' before being
          returned to the caller.  If it is `''', universal newline
          mode is enabled, but line endings are returned to the caller
          untranslated.  If it has any of the other legal values, input
          lines are only terminated by the given string, and the line
          ending is returned to the caller untranslated.

        * On output, if _newline_ is `None', any `'\n'' characters
          written are translated to the system default line separator,
          see os.linesep.  If _newline_ is `''', no translation
          takes place.  If _newline_ is any of the other legal values,
          any `'\n'' characters written are translated to the given
          string.

     If _closefd_ is `False' and a file descriptor rather than a
     filename was given, the underlying file descriptor will be kept
     open when the file is closed.  If a filename is given _closefd_
     has no effect and must be `True' (the default).

     The type of file object returned by the open().
     function depends on the mode.  When open() is used to
     open a file in a text mode (`'w'', `'r'', `'wt'', `'rt'', etc.),
     it returns a subclass of TextIOBase. (specifically
     TextIOWrapper).  When used to open a file in a binary
     mode with buffering, the returned class is a subclass of
     BufferedIOBase.  The exact class varies: in read binary mode,
     it returns a BufferedReader.; in write binary and
     append binary modes, it returns a BufferedWriter, and
     in read/write mode, it returns a BufferedRandom.  When
     buffering is disabled, the raw stream, a subclass of
     RawIOBase, FileIO, is returned.

     It is also possible to use an unicode or `bytes' string
     as a file for both reading and writing.  For unicode.
     strings StringIO can be used like a file opened in
     text mode, and for `bytes' a BytesIO can be used like
     a file opened in a binary mode.

 -- Exception: io.BlockingIOError
     Error raised when blocking would occur on a non-blocking stream.
     It inherits IOError.

     In addition to those of IOError, BlockingIOError
     has one attribute:

      -- Attribute: characters_written
          An integer containing the number of characters written to the
          stream before it blocked.

 -- Exception: io.UnsupportedOperation
     An exception inheriting IOError and ValueError
     that is raised when an unsupported operation is called on a
     stream.

5.15.2.2 I/O Base Classes
.........................

 -- Class: io.IOBase
     The abstract base class for all I/O classes, acting on streams of
     bytes.  There is no public constructor.

     This class provides empty abstract implementations for many methods
     that derived classes can override selectively; the default
     implementations represent a file that cannot be read, written or
     seeked.

     Even though IOBase does not declare `read()',
     `readinto()', or `write()' because their signatures will vary,
     implementations and clients should consider those methods part of
     the interface.  Also, implementations may raise a IOError
     when operations they do not support are called.

     The basic type used for binary data read from or written to a file
     is `bytes' (also known as str).  See bytearray.
     s are accepted too, and in some cases (such as `readinto')
     required.  Text I/O classes work with unicode data.

     Note that calling any method (even inquiries) on a closed stream is
     undefined.  Implementations may raise IOError in this
     case.

     IOBase (and its subclasses) support the iterator protocol, meaning
     that an IOBase object can be iterated over yielding
     the lines in a stream.  Lines are defined slightly differently
     depending on whether the stream is a binary stream (yielding
     `bytes'), or a text stream (yielding unicode strings).
     See readline() below.

     IOBase is also a context manager and therefore supports the
     with statement.  In this example, _file_ is closed after the
     with statement's suite is finished--even if an
     exception occurs:

         with io.open('spam.txt', 'w') as file:
             file.write(u'Spam and eggs!')

     IOBase provides these data attributes and methods:

      -- Method: close ()
          Flush and close this stream. This method has no effect if the
          file is already closed. Once the file is closed, any
          operation on the file (e.g. reading or writing) will raise a
          ValueError.

          As a convenience, it is allowed to call this method more than
          once; only the first call, however, will have an effect.

      -- Attribute: closed
          True if the stream is closed.

      -- Method: fileno ()
          Return the underlying file descriptor (an integer) of the
          stream if it exists.  An IOError is raised if the
          IO object does not use a file descriptor.

      -- Method: flush ()
          Flush the write buffers of the stream if applicable.  This
          does nothing for read-only and non-blocking streams.

      -- Method: isatty ()
          Return `True' if the stream is interactive (i.e., connected to
          a terminal/tty device).

      -- Method: readable ()
          Return `True' if the stream can be read from.  If False,
          `read()' will raise IOError.

      -- Method: readline (limit=-1)
          Read and return one line from the stream.  If _limit_ is
          specified, at most _limit_ bytes will be read.

          The line terminator is always `b'\n'' for binary files; for
          text files, the _newlines_ argument to open().
          can be used to select the line terminator(s) recognized.

      -- Method: readlines (hint=-1)
          Read and return a list of lines from the stream.  _hint_ can
          be specified to control the number of lines read: no more
          lines will be read if the total size (in bytes/characters) of
          all lines so far exceeds _hint_.

      -- Method: seek (offset, whence=SEEK_SET)
          Change the stream position to the given byte _offset_.
          _offset_ is interpreted relative to the position indicated by
          _whence_.  Values for _whence_ are:

             * `SEEK_SET' or `0' - start of the stream (the default);
               _offset_ should be zero or positive

             * `SEEK_CUR' or `1' - current stream position; _offset_ may
               be negative

             * `SEEK_END' or `2' - end of the stream; _offset_ is
               usually negative

          Return the new absolute position.

          New in version 2.7: The `SEEK_*' constants

      -- Method: seekable ()
          Return `True' if the stream supports random access.  If
          `False', see seek(), tell() and
          truncate() will raise IOError.

      -- Method: tell ()
          Return the current stream position.

      -- Method: truncate (size=None)
          Resize the stream to the given _size_ in bytes (or the
          current position if _size_ is not specified).  The current
          stream position isn't changed.  This resizing can extend or
          reduce the current file size.  In case of extension, the
          contents of the new file area depend on the platform (on most
          systems, additional bytes are zero-filled, on Windows they're
          undetermined).  The new file size is returned.

      -- Method: writable ()
          Return `True' if the stream supports writing.  If `False',
          `write()' and truncate() will raise
          IOError.

      -- Method: writelines (lines)
          Write a list of lines to the stream.  Line separators are not
          added, so it is usual for each of the lines provided to have
          a line separator at the end.

 -- Class: io.RawIOBase
     Base class for raw binary I/O.  It inherits IOBase.
     There is no public constructor.

     Raw binary I/O typically provides low-level access to an
     underlying OS device or API, and does not try to encapsulate it in
     high-level primitives (this is left to Buffered I/O and Text I/O,
     described later in this page).

     In addition to the attributes and methods from IOBase,
     RawIOBase provides the following methods:

      -- Method: read (n=-1)
          Read up to _n_ bytes from the object and return them.  As a
          convenience, if _n_ is unspecified or -1, see readall()
          is called.  Otherwise, only one system call is ever
          made.  Fewer than _n_ bytes may be returned if the operating
          system call returns fewer than _n_ bytes.

          If 0 bytes are returned, and _n_ was not 0, this indicates
          end of file.  If the object is in non-blocking mode and no
          bytes are available, `None' is returned.

      -- Method: readall ()
          Read and return all the bytes from the stream until EOF,
          using multiple calls to the stream if necessary.

      -- Method: readinto (b)
          Read up to len(b) bytes into bytearray _b_ and return the
          number ofbytes read.  If the object is in non-blocking mode
          and no bytes are available, `None' is returned.

      -- Method: write (b)
          Write the given bytes or bytearray object, _b_, to the
          underlying raw stream and return the number of bytes written.
          This can be less than `len(b)', depending on specifics of the
          underlying raw stream, and especially if it is in
          non-blocking mode.  `None' is returned if the raw stream is
          set not to block and no single byte could be readily written
          to it.

 -- Class: io.BufferedIOBase
     Base class for binary streams that support some kind of buffering.
     It inherits IOBase. There is no public constructor.

     The main difference with RawIOBase is that methods
     read(), readinto() and write()
     will try (respectively) to read as much input as requested
     or to consume all given output, at the expense of making perhaps
     more than one system call.

     In addition, those methods can raise BlockingIOError.
     if the underlying raw stream is in non-blocking mode and cannot
     take or give enough data; unlike their RawIOBase.
     counterparts, they will never return `None'.

     Besides, the read() method does not have a default
     implementation that defers to readinto().

     A typical BufferedIOBase implementation should not
     inherit from a RawIOBase implementation, but wrap
     one, like BufferedWriter and BufferedReader
     do.

     BufferedIOBase provides or overrides these members in
     addition to those from IOBase.:

      -- Attribute: raw
          The underlying raw stream (a RawIOBase instance)
          that BufferedIOBase deals with.  This is not part
          of the BufferedIOBase. API and may not exist on
          some implementations.

      -- Method: detach ()
          Separate the underlying raw stream from the buffer and return
          it.

          After the raw stream has been detached, the buffer is in an
          unusable state.

          Some buffers, like BytesIO, do not have the
          concept of a single raw stream to return from this method.
          They raise UnsupportedOperation.

          New in version 2.7.

      -- Method: read (n=-1)
          Read and return up to _n_ bytes.  If the argument is omitted,
          `None', or negative, data is read and returned until EOF is
          reached.  An empty bytes object is returned if the stream is
          already at EOF.

          If the argument is positive, and the underlying raw stream is
          not interactive, multiple raw reads may be issued to satisfy
          the byte count (unless EOF is reached first).  But for
          interactive raw streams, at most one raw read will be issued,
          and a short result does not imply that EOF is imminent.

          A BlockingIOError is raised if the underlying
          raw stream is in non blocking-mode, and has no data available
          at the moment.

      -- Method: read1 (n=-1)
          Read and return up to _n_ bytes, with at most one call to the
          underlying raw stream's read() method.  This can
          be useful if you are implementing your own buffering on top
          of a BufferedIOBase.  object.

      -- Method: readinto (b)
          Read up to len(b) bytes into bytearray _b_ and return the
          number of bytes read.

          Like read(), multiple reads may be issued to the
          underlying raw stream, unless the latter is 'interactive'.

          A BlockingIOError is raised if the underlying
          raw stream is in non blocking-mode, and has no data available
          at the moment.

      -- Method: write (b)
          Write the given bytes or bytearray object, _b_ and return the
          number of bytes written (never less than `len(b)', since if
          the write fails an IOError will be raised).
          Depending on the actual implementation, these bytes may be
          readily written to the underlying stream, or held in a buffer
          for performance and latency reasons.

          When in non-blocking mode, a BlockingIOError is
          raised if the data needed to be written to the raw stream but
          it couldn't accept all the data without blocking.

5.15.2.3 Raw File I/O
.....................

 -- Class: io.FileIO (name, mode='r', closefd=True)
     FileIO represents an OS-level file containing bytes
     data.  It implements the RawIOBase interface (and
     therefore the IOBase interface, too).

     The _name_ can be one of two things:

        * a string representing the path to the file which will be
          opened;

        * an integer representing the number of an existing OS-level
          file descriptor to which the resulting FileIO.
          object will give access.

     The _mode_ can be `'r'', `'w'' or `'a'' for reading (default),
     writing, or appending.  The file will be created if it doesn't
     exist when opened for writing or appending; it will be truncated
     when opened for writing.  Add a `'+'' to the mode to allow
     simultaneous reading and writing.

     The `read()' (when called with a positive argument), `readinto()'
     and `write()' methods on this class will only make one system call.

     In addition to the attributes and methods from IOBase.
     and RawIOBase, FileIO provides the
     following data attributes and methods:

      -- Attribute: mode
          The mode as given in the constructor.

      -- Attribute: name
          The file name.  This is the file descriptor of the file when
          no name is given in the constructor.

5.15.2.4 Buffered Streams
.........................

In many situations, buffered I/O streams will provide higher performance
(bandwidth and latency) than raw I/O streams.  Their API is also more
usable.

 -- Class: io.BytesIO ([initial_bytes])
     A stream implementation using an in-memory bytes buffer.  It
     inherits BufferedIOBase.

     The argument _initial_bytes_ is an optional initial `bytes'.

     BytesIO provides or overrides these methods in
     addition to those from BufferedIOBase and
     IOBase.:

      -- Method: getvalue ()
          Return `bytes' containing the entire contents of the buffer.

      -- Method: read1 ()
          In BytesIO, this is the same as `read()'.

 -- Class: io.BufferedReader (raw, buffer_size=DEFAULT_BUFFER_SIZE)
     A buffer providing higher-level access to a readable, sequential
     RawIOBase object.  It inherits BufferedIOBase.
     When reading data from this object, a larger amount of data
     may be requested from the underlying raw stream, and kept in an
     internal buffer.  The buffered data can then be returned directly
     on subsequent reads.

     The constructor creates a BufferedReader for the given
     readable _raw_ stream and _buffer_size_.  If _buffer_size_ is
     omitted, see DEFAULT_BUFFER_SIZE is used.

     BufferedReader provides or overrides these methods in
     addition to those from BufferedIOBase and
     IOBase.:

      -- Method: peek ([n])
          Return bytes from the stream without advancing the position.
          At most one single read on the raw stream is done to satisfy
          the call. The number of bytes returned may be less or more
          than requested.

      -- Method: read ([n])
          Read and return _n_ bytes, or if _n_ is not given or
          negative, until EOF or if the read call would block in
          non-blocking mode.

      -- Method: read1 (n)
          Read and return up to _n_ bytes with only one call on the raw
          stream.  If at least one byte is buffered, only buffered
          bytes are returned.  Otherwise, one raw stream read call is
          made.

 -- Class: io.BufferedWriter (raw, buffer_size=DEFAULT_BUFFER_SIZE)
     A buffer providing higher-level access to a writeable, sequential
     RawIOBase object.  It inherits BufferedIOBase.
     When writing to this object, data is normally held into an
     internal buffer.  The buffer will be written out to the underlying
     RawIOBase.  object under various conditions, including:

        * when the buffer gets too small for all pending data;

        * when flush() is called;

        * when a `seek()' is requested (for BufferedRandom.
          objects);

        * when the BufferedWriter object is closed or
          destroyed.

     The constructor creates a BufferedWriter for the
     given writeable _raw_ stream.  If the _buffer_size_ is not given,
     it defaults to DEFAULT_BUFFER_SIZE.

     A third argument, _max_buffer_size_, is supported, but unused and
     deprecated.

     BufferedWriter provides or overrides these methods in
     addition to those from BufferedIOBase and
     IOBase.:

      -- Method: flush ()
          Force bytes held in the buffer into the raw stream.  A
          BlockingIOError should be raised if the raw stream
          blocks.

      -- Method: write (b)
          Write the bytes or bytearray object, _b_ and return the
          number of bytes written.  When in non-blocking mode, a
          BlockingIOError is raised if the buffer needs to be
          written out but the raw stream blocks.

 -- Class: io.BufferedRWPair (reader, writer,
          buffer_size=DEFAULT_BUFFER_SIZE)
     A buffered I/O object giving a combined, higher-level access to two
     sequential RawIOBase objects: one readable, the other
     writeable.  It is useful for pairs of unidirectional communication
     channels (pipes, for instance).  It inherits BufferedIOBase.

     _reader_ and _writer_ are RawIOBase objects that are
     readable and writeable respectively.  If the _buffer_size_ is
     omitted it defaults to DEFAULT_BUFFER_SIZE.

     A fourth argument, _max_buffer_size_, is supported, but unused and
     deprecated.

     BufferedRWPair implements all of
     BufferedIOBase.'s methods except for detach(),
     which raises UnsupportedOperation.

 -- Class: io.BufferedRandom (raw, buffer_size=DEFAULT_BUFFER_SIZE)
     A buffered interface to random access streams.  It inherits
     BufferedReader and BufferedWriter, and further
     supports `seek()' and `tell()' functionality.

     The constructor creates a reader and writer for a seekable raw
     stream, given in the first argument.  If the _buffer_size_ is
     omitted it defaults to DEFAULT_BUFFER_SIZE.

     A third argument, _max_buffer_size_, is supported, but unused and
     deprecated.

     BufferedRandom is capable of anything
     BufferedReader or BufferedWriter can do.

5.15.2.5 Text I/O
.................

 -- Class: io.TextIOBase
     Base class for text streams.  This class provides an unicode
     character and line based interface to stream I/O.  There is no
     `readinto()' method because Python's unicode strings
     are immutable.  It inherits IOBase.  There is no
     public constructor.

     TextIOBase provides or overrides these data attributes
     and methods in addition to those from IOBase.:

      -- Attribute: encoding
          The name of the encoding used to decode the stream's bytes
          into strings, and to encode strings into bytes.

      -- Attribute: errors
          The error setting of the decoder or encoder.

      -- Attribute: newlines
          A string, a tuple of strings, or `None', indicating the
          newlines translated so far.  Depending on the implementation
          and the initial constructor flags, this may not be available.

      -- Attribute: buffer
          The underlying binary buffer (a BufferedIOBase.
          instance) that TextIOBase deals with.  This is
          not part of the TextIOBase. API and may not exist
          on some implementations.

      -- Method: detach ()
          Separate the underlying binary buffer from the
          TextIOBase and return it.

          After the underlying buffer has been detached, the
          TextIOBase is in an unusable state.

          Some TextIOBase implementations, like
          StringIO, may not have the concept of an underlying
          buffer and calling this method will raise
          UnsupportedOperation.

          New in version 2.7.

      -- Method: read (n)
          Read and return at most _n_ characters from the stream as a
          single unicode.  If _n_ is negative or `None',
          reads until EOF.

      -- Method: readline ()
          Read until newline or EOF and return a single `unicode'.  If
          the stream is already at EOF, an empty string is returned.

      -- Method: write (s)
          Write the unicode string _s_ to the stream and
          return the number of characters written.

 -- Class: io.TextIOWrapper (buffer, encoding=None, errors=None,
          newline=None, line_buffering=False)
     A buffered text stream over a BufferedIOBase binary
     stream.  It inherits TextIOBase.

     _encoding_ gives the name of the encoding that the stream will be
     decoded or encoded with.  It defaults to
     locale.getpreferredencoding().

     _errors_ is an optional string that specifies how encoding and
     decoding errors are to be handled.  Pass `'strict'' to raise a
     ValueError.  exception if there is an encoding error
     (the default of `None' has the same effect), or pass `'ignore'' to
     ignore errors.  (Note that ignoring encoding errors can lead to
     data loss.)  `'replace'' causes a replacement marker (such as
     `'?'') to be inserted where there is malformed data.  When
     writing, `'xmlcharrefreplace'' (replace with the appropriate XML
     character reference) or `'backslashreplace'' (replace with
     backslashed escape sequences) can be used.  Any other error
     handling name that has been registered with
     codecs.register_error() is also valid.

     _newline_ can be `None', `''', `'\n'', `'\r'', or `'\r\n''.  It
     controls the handling of line endings.  If it is `None', universal
     newlines is enabled.  With this enabled, on input, the lines
     endings `'\n'', `'\r'', or `'\r\n'' are translated to `'\n''
     before being returned to the caller.  Conversely, on output,
     `'\n'' is translated to the system default line separator, see
     os.linesep.  If _newline_ is any other of its legal values,
     that newline becomes the newline when the file is read and it is
     returned untranslated.  On output, `'\n'' is converted to the
     _newline_.

     If _line_buffering_ is `True', `flush()' is implied when a call to
     write contains a newline character.

     TextIOWrapper provides one attribute in addition to
     those of TextIOBase and its parents:

      -- Attribute: line_buffering
          Whether line buffering is enabled.

 -- Class: io.StringIO (initial_value=u", newline=None)
     An in-memory stream for unicode text.  It inherits
     TextIOWrapper.

     The initial value of the buffer (an empty unicode string by
     default) can be set by providing _initial_value_.  The _newline_
     argument works like that of TextIOWrapper.  The
     default is to do no newline translation.

     StringIO provides this method in addition to those from
     TextIOWrapper and its parents:

      -- Method: getvalue ()
          Return a `unicode' containing the entire contents of the
          buffer at any time before the StringIO object's
          `close()' method is called.

     Example usage:

         import io

         output = io.StringIO()
         output.write(u'First line.\n')
         output.write(u'Second line.\n')

         # Retrieve file contents -- this will be
         # u'First line.\nSecond line.\n'
         contents = output.getvalue()

         # Close object and discard memory buffer --
         # .getvalue() will now raise an exception.
         output.close()

 -- Class: io.IncrementalNewlineDecoder
     A helper codec that decodes newlines for universal newlines mode.
     It inherits codecs.IncrementalDecoder.

5.15.3 `time' -- Time access and conversions
--------------------------------------------

This module provides various time-related functions. For related
functionality, see also the datetime and calendar.
modules.

  Although this module is always available, not all functions are
available on all platforms.  Most of the functions defined in this
module call platform C library functions with the same name.  It may
sometimes be helpful to consult the platform documentation, because the
semantics of these functions varies among platforms.

  An explanation of some terminology and conventions is in order.

   * The _epoch_ is the point where the time starts.  On January 1st of
     that year, at 0 hours, the "time since the epoch" is zero.  For
     Unix, the epoch is 1970.  To find out what the epoch is, look at
     `gmtime(0)'.

   * The functions in this module do not handle dates and times before
     the epoch or far in the future.  The cut-off point in the future
     is determined by the C library; for Unix, it is typically in 2038.

   * *Year 2000 (Y2K) issues*:  Python depends on the platform's C
     library, which generally doesn't have year 2000 issues, since all
     dates and times are represented internally as seconds since the
     epoch.  Functions accepting a struct_time. (see below)
     generally require a 4-digit year.  For backward compatibility,
     2-digit years are supported if the module variable `accept2dyear'
     is a non-zero integer; this variable is initialized to `1' unless
     the environment variable PYTHONY2K is set to a
     non-empty string, in which case it is initialized to `0'.  Thus,
     you can set PYTHONY2K to a non-empty string in the
     environment to require 4-digit years for all year input.  When
     2-digit years are accepted, they are converted according to the
     POSIX or X/Open standard: values 69-99 are mapped to 1969-1999,
     and values 0-68 are mapped to 2000-2068. Values 100-1899 are
     always illegal.  Note that this is new as of Python 1.5.2(a2);
     earlier versions, up to Python 1.5.1 and 1.5.2a1, would add 1900
     to year values below 1900.

   * UTC is Coordinated Universal Time (formerly known as Greenwich
     Mean Time, or GMT).  The acronym UTC is not a mistake but a
     compromise between English and French.

   * DST is Daylight Saving Time, an adjustment of the timezone by
     (usually) one hour during part of the year.  DST rules are magic
     (determined by local law) and can change from year to year.  The C
     library has a table containing the local rules (often it is read
     from a system file for flexibility) and is the only source of True
     Wisdom in this respect.

   * The precision of the various real-time functions may be less than
     suggested by the units in which their value or argument is
     expressed. E.g. on most Unix systems, the clock "ticks" only 50 or
     100 times a second.

   * On the other hand, the precision of time() and
     sleep() is better than their Unix equivalents: times are
     expressed as floating point numbers, see time() returns
     the most accurate time available (using Unix `gettimeofday()'
     where available), and sleep() will accept a time with
     a nonzero fraction (Unix `select()' is used to implement this,
     where available).

   * The time value as returned by gmtime(),
     localtime(), and strptime(), and accepted by
     asctime(), mktime() and strftime(),
     may be considered as a sequence of 9 integers.  The return values
     of gmtime(), localtime(), and
     strptime() also offer attribute names for individual fields.

     See struct_time for a description of these objects.

     Changed in version 2.2: The time value sequence was changed from a
     tuple to a struct_time, with the addition of attribute
     names for the fields.

   * Use the following functions to convert between time
     representations:

     From                          To                            Use
     ------------------------------------------------------------------------------------------
     seconds since the epoch       struct_time in    gmtime().
                                   UTC
     seconds since the epoch       struct_time in    localtime().
                                   local time
     struct_time in    seconds since the epoch       calendar.timegm().
     UTC
     struct_time in    seconds since the epoch       mktime().
     local time

  The module defines the following functions and data items:

 -- Data: time.accept2dyear
     Boolean value indicating whether two-digit year values will be
     accepted.  This is true by default, but will be set to false if
     the environment variable PYTHONY2K has been set to a
     non-empty string.  It may also be modified at run time.

 -- Data: time.altzone
     The offset of the local DST timezone, in seconds west of UTC, if
     one is defined.  This is negative if the local DST timezone is
     east of UTC (as in Western Europe, including the UK).  Only use
     this if `daylight' is nonzero.

 -- Function: time.asctime ([t])
     Convert a tuple or struct_time representing a time as
     returned by gmtime() or localtime() to a
     24-character string of the following form: `'Sun Jun 20 23:21:05
     1993''.  If _t_ is not provided, the current time as returned by
     localtime() is used. Locale information is not used by
     asctime().

          Note: Unlike the C function of the same name, there is no
          trailing newline.

     Changed in version 2.1: Allowed _t_ to be omitted.

 -- Function: time.clock ()
     On Unix, return the current processor time as a floating point
     number expressed in seconds.  The precision, and in fact the very
     definition of the meaning of "processor time", depends on that of
     the C function of the same name, but in any case, this is the
     function to use for benchmarking Python or timing algorithms.

     On Windows, this function returns wall-clock seconds elapsed since
     the first call to this function, as a floating point number, based
     on the Win32 function `QueryPerformanceCounter()'. The resolution
     is typically better than one microsecond.

 -- Function: time.ctime ([secs])
     Convert a time expressed in seconds since the epoch to a string
     representing local time. If _secs_ is not provided or None,
     the current time as returned by time() is used.
     `ctime(secs)' is equivalent to `asctime(localtime(secs))'. Locale
     information is not used by ctime().

     Changed in version 2.1: Allowed _secs_ to be omitted.

     Changed in version 2.4: If _secs_ is None, the current
     time is used.

 -- Data: time.daylight
     Nonzero if a DST timezone is defined.

 -- Function: time.gmtime ([secs])
     Convert a time expressed in seconds since the epoch to a
     struct_time in UTC in which the dst flag is always zero.  If
     _secs_ is not provided or None, the current time as
     returned by time() is used.  Fractions of a second are
     ignored.  See above for a description of the struct_time
     object. See calendar.timegm() for the inverse of
     this function.

     Changed in version 2.1: Allowed _secs_ to be omitted.

     Changed in version 2.4: If _secs_ is None, the current
     time is used.

 -- Function: time.localtime ([secs])
     Like gmtime() but converts to local time.  If _secs_
     is not provided or None, the current time as returned
     by time() is used.  The dst flag is set to `1' when
     DST applies to the given time.

     Changed in version 2.1: Allowed _secs_ to be omitted.

     Changed in version 2.4: If _secs_ is None, the current
     time is used.

 -- Function: time.mktime (t)
     This is the inverse function of localtime().  Its
     argument is the struct_time or full 9-tuple (since the
     dst flag is needed; use `-1' as the dst flag if it is unknown)
     which expresses the time in _local_ time, not UTC.  It returns a
     floating point number, for compatibility with time().
     If the input value cannot be represented as a valid time, either
     OverflowError or ValueError will be raised
     (which depends on whether the invalid value is caught by Python or
     the underlying C libraries).  The earliest date for which it can
     generate a time is platform-dependent.

 -- Function: time.sleep (secs)
     Suspend execution for the given number of seconds.  The argument
     may be a floating point number to indicate a more precise sleep
     time. The actual suspension time may be less than that requested
     because any caught signal will terminate the sleep().
     following execution of that signal's catching routine.  Also, the
     suspension time may be longer than requested by an arbitrary
     amount because of the scheduling of other activity in the system.

 -- Function: time.strftime (format[, t])
     Convert a tuple or struct_time representing a time as
     returned by gmtime() or localtime() to a
     string as specified by the _format_ argument.  If _t_ is not
     provided, the current time as returned by localtime().
     is used.  _format_ must be a string.  ValueError is
     raised if any field in _t_ is outside of the allowed range.

     Changed in version 2.1: Allowed _t_ to be omitted.

     Changed in version 2.4: ValueError raised if a field
     in _t_ is out of range.

     Changed in version 2.5: 0 is now a legal argument for any position
     in the time tuple; if it is normally illegal the value is forced
     to a correct one..

     The following directives can be embedded in the _format_ string.
     They are shown without the optional field width and precision
     specification, and are replaced by the indicated characters in the
     strftime() result:

     Directive       Meaning                              Notes
     -----------------------------------------------------------------
     `%a'            Locale's abbreviated weekday name.
     `%A'            Locale's full weekday name.
     `%b'            Locale's abbreviated month name.
     `%B'            Locale's full month name.
     `%c'            Locale's appropriate date and time
                     representation.
     `%d'            Day of the month as a decimal
                     number [01,31].
     `%H'            Hour (24-hour clock) as a decimal
                     number [00,23].
     `%I'            Hour (12-hour clock) as a decimal
                     number [01,12].
     `%j'            Day of the year as a decimal number
                     [001,366].
     `%m'            Month as a decimal number [01,12].
     `%M'            Minute as a decimal number [00,59].
     `%p'            Locale's equivalent of either AM or  (1)
                     PM.
     `%S'            Second as a decimal number [00,61].  (2)
     `%U'            Week number of the year (Sunday as   (3)
                     the first day of the week) as a
                     decimal number [00,53].  All days
                     in a new year preceding the first
                     Sunday are considered to be in week
                     0.
     `%w'            Weekday as a decimal number
                     [0(Sunday),6].
     `%W'            Week number of the year (Monday as   (3)
                     the first day of the week) as a
                     decimal number [00,53].  All days
                     in a new year preceding the first
                     Monday are considered to be in week
                     0.
     `%x'            Locale's appropriate date
                     representation.
     `%X'            Locale's appropriate time
                     representation.
     `%y'            Year without century as a decimal
                     number [00,99].
     `%Y'            Year with century as a decimal
                     number.
     `%Z'            Time zone name (no characters if no
                     time zone exists).
     `%%'            A literal `'%'' character.

     Notes:

       1. When used with the strptime() function, the `%p'
          directive only affects the output hour field if the `%I'
          directive is used to parse the hour.

       2. The range really is `0' to `61'; this accounts for leap
          seconds and the (very rare) double leap seconds.

       3. When used with the strptime() function, `%U' and
          `%W' are only used in calculations when the day of the week
          and the year are specified.

     Here is an example, a format for dates compatible with that
     specified  in the RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) Internet email standard.
     (1)

         >>> from time import gmtime, strftime
         >>> strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime())
         'Thu, 28 Jun 2001 14:17:15 +0000'

     Additional directives may be supported on certain platforms, but
     only the ones listed here have a meaning standardized by ANSI C.

     On some platforms, an optional field width and precision
     specification can immediately follow the initial `'%'' of a
     directive in the following order; this is also not portable. The
     field width is normally 2 except for `%j' where it is 3.

 -- Function: time.strptime (string[, format])
     Parse a string representing a time according to a format.  The
     return  value is a struct_time as returned by
     gmtime() or localtime().

     The _format_ parameter uses the same directives as those used by
     strftime().; it defaults to `"%a %b %d %H:%M:%S %Y"'
     which matches the formatting returned by ctime(). If
     _string_ cannot be parsed according to _format_, or if it has
     excess data after parsing, see ValueError is raised.  The
     default values used to fill in any missing data when more accurate
     values cannot be inferred are `(1900, 1, 1, 0, 0, 0, 0, 1, -1)'.

     For example:

         >>> import time
         >>> time.strptime("30 Nov 00", "%d %b %y")   # doctest: +NORMALIZE_WHITESPACE
         time.struct_time(tm_year=2000, tm_mon=11, tm_mday=30, tm_hour=0, tm_min=0,
                          tm_sec=0, tm_wday=3, tm_yday=335, tm_isdst=-1)

     Support for the `%Z' directive is based on the values contained in
     `tzname' and whether `daylight' is true.  Because of this, it is
     platform-specific except for recognizing UTC and GMT which are
     always known (and are considered to be non-daylight savings
     timezones).

     Only the directives specified in the documentation are supported.
     Because `strftime()' is implemented per platform it can sometimes
     offer more directives than those listed.  But `strptime()' is
     independent of any platform and thus does not necessarily support
     all directives available that are not documented as supported.

 -- Class: time.struct_time
     The type of the time value sequence returned by gmtime(),
     see localtime(), and strptime().  It is an
     object with a named tuple interface: values can be
     accessed by index and by attribute name.  The following values are
     present:

     Index       Attribute               Values
     --------------------------------------------------------------------------
     0           `tm_year'               (for example, 1993)
     1           `tm_mon'                range [1, 12]
     2           `tm_mday'               range [1, 31]
     3           `tm_hour'               range [0, 23]
     4           `tm_min'                range [0, 59]
     5           `tm_sec'                range [0, 61]; see *(1)* in
                                         strftime() description
     6           `tm_wday'               range [0, 6], Monday is 0
     7           `tm_yday'               range [1, 366]
     8           `tm_isdst'              0, 1 or -1; see below

     New in version 2.2.

     Note that unlike the C structure, the month value is a range of
     [1, 12], not [0, 11].  A year value will be handled as described
     under Year 2000 (Y2K) issues above.  A `-1' argument
     as the daylight savings flag, passed to mktime() will
     usually result in the correct daylight savings state to be filled
     in.

     When a tuple with an incorrect length is passed to a function
     expecting a struct_time, or having elements of the
     wrong type, a TypeError is raised.

 -- Function: time.time ()
     Return the time as a floating point number expressed in seconds
     since the epoch, in UTC.  Note that even though the time is always
     returned as a floating point number, not all systems provide time
     with a better precision than 1 second.  While this function
     normally returns non-decreasing values, it can return a lower
     value than a previous call if the system clock has been set back
     between the two calls.

 -- Data: time.timezone
     The offset of the local (non-DST) timezone, in seconds west of UTC
     (negative in most of Western Europe, positive in the US, zero in
     the UK).

 -- Data: time.tzname
     A tuple of two strings: the first is the name of the local non-DST
     timezone, the second is the name of the local DST timezone.  If no
     DST timezone is defined, the second string should not be used.

 -- Function: time.tzset ()
     Resets the time conversion rules used by the library routines. The
     environment variable `TZ' specifies how this is done.

     New in version 2.3.

     Availability: Unix.

          Note: Although in many cases, changing the `TZ' environment
          variable may affect the output of functions like
          localtime() without calling tzset(), this
          behavior should not be relied on.

          The `TZ' environment variable should contain no whitespace.

     The standard format of the `TZ' environment variable is (whitespace
     added for clarity):

         std offset [dst [offset [,start[/time], end[/time]]]]

     Where the components are:

    `std' and `dst'
          Three or more alphanumerics giving the timezone
          abbreviations. These will be propagated into time.tzname

    `offset'
          The offset has the form: `± hh[:mm[:ss]]'. This indicates the
          value added the local time to arrive at UTC.  If preceded by
          a '-', the timezone is east of the Prime Meridian; otherwise,
          it is west. If no offset follows dst, summer time is assumed
          to be one hour ahead of standard time.

    `start[/time], end[/time]'
          Indicates when to change to and back from DST. The format of
          the start and end dates are one of the following:

         `J_n_'
               The Julian day _n_ (1 <= _n_ <= 365). Leap days are not
               counted, so in all years February 28 is day 59 and March
               1 is day 60.

         `_n_'
               The zero-based Julian day (0 <= _n_ <= 365). Leap days
               are counted, and it is possible to refer to February 29.

         `M_m_._n_._d_'
               The _d_'th day (0 <= _d_ <= 6) or week _n_ of month _m_
               of the year (1 <= _n_ <= 5, 1 <= _m_ <= 12, where week 5
               means "the last _d_ day in month _m_" which may occur in
               either the fourth or the fifth week). Week 1 is the
               first week in which the _d_'th day occurs. Day zero is
               Sunday.

          `time' has the same format as `offset' except that no leading
          sign ('-' or '+') is allowed. The default, if time is not
          given, is 02:00:00.

         >>> os.environ['TZ'] = 'EST+05EDT,M4.1.0,M10.5.0'
         >>> time.tzset()
         >>> time.strftime('%X %x %Z')
         '02:07:36 05/08/03 EDT'
         >>> os.environ['TZ'] = 'AEST-10AEDT-11,M10.5.0,M3.5.0'
         >>> time.tzset()
         >>> time.strftime('%X %x %Z')
         '16:08:12 05/08/03 AEST'

     On many Unix systems (including *BSD, Linux, Solaris, and Darwin),
     it is more convenient to use the system's zoneinfo (`tzfile(5)')
     database to specify the timezone rules. To do this, set the `TZ'
     environment variable to the path of the required timezone
     datafile, relative to the root of the systems 'zoneinfo' timezone
     database, usually located at `/usr/share/zoneinfo'. For example,
     `'US/Eastern'', `'Australia/Melbourne'', `'Egypt'' or
     `'Europe/Amsterdam''.

         >>> os.environ['TZ'] = 'US/Eastern'
         >>> time.tzset()
         >>> time.tzname
         ('EST', 'EDT')
         >>> os.environ['TZ'] = 'Egypt'
         >>> time.tzset()
         >>> time.tzname
         ('EET', 'EEST')

Module datetime.
     More object-oriented interface to dates and times.

Module locale.
     Internationalization services.  The locale settings can affect the
     return values for some of  the functions in the time.
     module.

Module calendar.
     General calendar-related functions.   `timegm()' is the inverse of
     gmtime() from this module.

Footnotes
.........

     (1) The use of `%Z' is now deprecated, but the `%z' escape that
     expands to the preferred  hour/minute offset is not supported by
     all ANSI C libraries. Also, a strict reading of the original 1982 RFC
     822 (http://tools.ietf.org/html/rfc822.html) standard calls for a
     two-digit year (%y rather than %Y), but practice moved to 4-digit
     years long before the year 2000.  The 4-digit year has been
     mandated by RFC 2822 (http://tools.ietf.org/html/rfc2822.html),
     which obsoletes RFC 822 (http://tools.ietf.org/html/rfc822.html).

5.15.4 `argparse' -- Parser for command line options, arguments and sub-commands
--------------------------------------------------------------------------------

New in version 2.7.

  The argparse module makes it easy to write user friendly
command line interfaces. The program defines what arguments it
requires, and argparse.  will figure out how to parse those
out of sys.argv.  The argparse.  module also
automatically generates help and usage messages and issues errors when
users give the program invalid arguments.

5.15.4.1 Example
................

The following code is a Python program that takes a list of integers and
produces either the sum or the max:

    import argparse

    parser = argparse.ArgumentParser(description='Process some integers.')
    parser.add_argument('integers', metavar='N', type=int, nargs='+',
                       help='an integer for the accumulator')
    parser.add_argument('--sum', dest='accumulate', action='store_const',
                       const=sum, default=max,
                       help='sum the integers (default: find the max)')

    args = parser.parse_args()
    print args.accumulate(args.integers)

Assuming the Python code above is saved into a file called `prog.py',
it can be run at the command line and provides useful help messages:

    $ prog.py -h
    usage: prog.py [-h] [--sum] N [N ...]

    Process some integers.

    positional arguments:
     N           an integer for the accumulator

    optional arguments:
     -h, --help  show this help message and exit
     --sum       sum the integers (default: find the max)

When run with the appropriate arguments, it prints either the sum or
the max of the command-line integers:

    $ prog.py 1 2 3 4
    4

    $ prog.py 1 2 3 4 --sum
    10

If invalid arguments are passed in, it will issue an error:

    $ prog.py a b c
    usage: prog.py [-h] [--sum] N [N ...]
    prog.py: error: argument N: invalid int value: 'a'

The following sections walk you through this example.

5.15.4.2 Creating a parser
..........................

The first step in using the argparse is creating an
ArgumentParser object:

    >>> parser = argparse.ArgumentParser(description='Process some integers.')

The ArgumentParser object will hold all the information
necessary to parse the command line into python data types.

5.15.4.3 Adding arguments
.........................

Filling an ArgumentParser with information about program
arguments is done by making calls to the add_argument().
method.  Generally, these calls tell the ArgumentParser.
how to take the strings on the command line and turn them into objects.
This information is stored and used when parse_args() is
called. For example:

    >>> parser.add_argument('integers', metavar='N', type=int, nargs='+',
    ...                     help='an integer for the accumulator')
    >>> parser.add_argument('--sum', dest='accumulate', action='store_const',
    ...                     const=sum, default=max,
    ...                     help='sum the integers (default: find the max)')

Later, calling `parse_args()' will return an object with two
attributes, `integers' and `accumulate'.  The `integers' attribute will
be a list of one or more ints, and the `accumulate' attribute will be
either the sum() function, if `--sum' was specified at the
command line, or the max() function if it was not.

5.15.4.4 Parsing arguments
..........................

ArgumentParser parses args through the parse_args()
method.  This will inspect the command-line, convert each arg to
the appropriate type and then invoke the appropriate action.  In most
cases, this means a simple namespace object will be built up from
attributes parsed out of the command-line:

    >>> parser.parse_args(['--sum', '7', '-1', '42'])
    Namespace(accumulate=<built-in function sum>, integers=[7, -1, 42])

In a script, see parse_args() will typically be called with no
arguments, and the ArgumentParser will automatically
determine the command-line args from sys.argv.

5.15.4.5 ArgumentParser objects
...............................

 -- Class: argparse.ArgumentParser ([description][, epilog][, prog][,
          usage][, add_help][, argument_default][, parents][,
          prefix_chars][, conflict_handler][, formatter_class])
     Create a new ArgumentParser object.  Each parameter
     has its own more detailed description below, but in short they are:

        * description. - Text to display before the
          argument help.

        * epilog. - Text to display after the argument help.

        * add_help. - Add a -h/-help option to the parser.
          (default: `True')

        * argument_default. - Set the global default value
          for arguments.  (default: `None')

        * parents. - A list of ArgumentParser.
          objects whose arguments should also be included.

        * prefix_chars. - The set of characters that prefix
          optional arguments.  (default: '-')

        * fromfile_prefix_chars. - The set of characters
          that prefix files from which additional arguments should be
          read. (default: `None')

        * formatter_class. - A class for customizing the
          help output.

        * conflict_handler. - Usually unnecessary, defines
          strategy for resolving conflicting optionals.

        * prog. - The name of the program (default:
          `sys.argv[0]')

        * usage. - The string describing the program usage
          (default: generated)

The following sections describe how each of these are used.

5.15.4.6 description
....................

Most calls to the ArgumentParser constructor will use the
`description=' keyword argument.  This argument gives a brief
description of what the program does and how it works.  In help
messages, the description is displayed between the command-line usage
string and the help messages for the various arguments:

    >>> parser = argparse.ArgumentParser(description='A foo that bars')
    >>> parser.print_help()
    usage: argparse.py [-h]

    A foo that bars

    optional arguments:
     -h, --help  show this help message and exit

By default, the description will be line-wrapped so that it fits within
the given space.  To change this behavior, see the
formatter_class argument.

5.15.4.7 epilog
...............

Some programs like to display additional description of the program
after the description of the arguments.  Such text can be specified
using the `epilog=' argument to ArgumentParser.:

    >>> parser = argparse.ArgumentParser(
    ...     description='A foo that bars',
    ...     epilog="And that's how you'd foo a bar")
    >>> parser.print_help()
    usage: argparse.py [-h]

    A foo that bars

    optional arguments:
     -h, --help  show this help message and exit

    And that's how you'd foo a bar

As with the description argument, the `epilog=' text is by
default line-wrapped, but this behavior can be adjusted with the
formatter_class.  argument to ArgumentParser.

5.15.4.8 add_help
.................

By default, ArgumentParser objects add an option which simply displays
the parser's help message. For example, consider a file named
`myprogram.py' containing the following code:

    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--foo', help='foo help')
    args = parser.parse_args()

If `-h' or `--help' is supplied is at the command-line, the
ArgumentParser help will be printed:

    $ python myprogram.py --help
    usage: myprogram.py [-h] [--foo FOO]

    optional arguments:
     -h, --help  show this help message and exit
     --foo FOO   foo help

Occasionally, it may be useful to disable the addition of this help
option.  This can be achieved by passing `False' as the `add_help='
argument to ArgumentParser.:

    >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False)
    >>> parser.add_argument('--foo', help='foo help')
    >>> parser.print_help()
    usage: PROG [--foo FOO]

    optional arguments:
     --foo FOO  foo help

The help option is typically `-h/--help'. The exception to this is if
the `prefix_chars=' is specified and does not include `'-'', in which
case `-h' and `--help' are not valid options.  In this case, the first
character in `prefix_chars' is used to prefix the help options:

    >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='+/')
    >>> parser.print_help()
    usage: PROG [+h]

    optional arguments:
      +h, ++help  show this help message and exit

5.15.4.9 prefix_chars
.....................

Most command-line options will use `'-'' as the prefix, e.g. `-f/--foo'.
Parsers that need to support different or additional prefix characters,
e.g. for options like `+f' or `/foo', may specify them using the
`prefix_chars=' argument to the ArgumentParser constructor:

    >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='-+')
    >>> parser.add_argument('+f')
    >>> parser.add_argument('++bar')
    >>> parser.parse_args('+f X ++bar Y'.split())
    Namespace(bar='Y', f='X')

The `prefix_chars=' argument defaults to `'-''. Supplying a set of
characters that does not include `'-'' will cause `-f/--foo' options to
be disallowed.

5.15.4.10 fromfile_prefix_chars
...............................

Sometimes, for example when dealing with a particularly long argument
lists, it may make sense to keep the list of arguments in a file rather
than typing it out at the command line.  If the
`fromfile_prefix_chars=' argument is given to the ArgumentParser
constructor, then arguments that start with any of the specified
characters will be treated as files, and will be replaced by the
arguments they contain.  For example:

    >>> with open('args.txt', 'w') as fp:
    ...    fp.write('-f\nbar')
    >>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
    >>> parser.add_argument('-f')
    >>> parser.parse_args(['-f', 'foo', '@args.txt'])
    Namespace(f='bar')

Arguments read from a file must by default be one per line (but see also
`convert_arg_line_to_args()') and are treated as if they were in the
same place as the original file referencing argument on the command
line.  So in the example above, the expression `['-f', 'foo',
'@args.txt']' is considered equivalent to the expression `['-f', 'foo',
'-f', 'bar']'.

  The `fromfile_prefix_chars=' argument defaults to `None', meaning that
arguments will never be treated as file references.

5.15.4.11 argument_default
..........................

Generally, argument defaults are specified either by passing a default
to `add_argument()' or by calling the `set_defaults()' methods with a
specific set of name-value pairs.  Sometimes however, it may be useful
to specify a single parser-wide default for arguments.  This can be
accomplished by passing the `argument_default=' keyword argument to
ArgumentParser.  For example, to globally suppress
attribute creation on `parse_args()' calls, we supply
`argument_default=SUPPRESS':

    >>> parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS)
    >>> parser.add_argument('--foo')
    >>> parser.add_argument('bar', nargs='?')
    >>> parser.parse_args(['--foo', '1', 'BAR'])
    Namespace(bar='BAR', foo='1')
    >>> parser.parse_args([])
    Namespace()

5.15.4.12 parents
.................

Sometimes, several parsers share a common set of arguments. Rather than
repeating the definitions of these arguments, a single parser with all
the shared arguments and passed to `parents=' argument to
ArgumentParser.  can be used.  The `parents=' argument takes a
list of ArgumentParser.  objects, collects all the
positional and optional actions from them, and adds these actions to
the ArgumentParser object being constructed:

    >>> parent_parser = argparse.ArgumentParser(add_help=False)
    >>> parent_parser.add_argument('--parent', type=int)

    >>> foo_parser = argparse.ArgumentParser(parents=[parent_parser])
    >>> foo_parser.add_argument('foo')
    >>> foo_parser.parse_args(['--parent', '2', 'XXX'])
    Namespace(foo='XXX', parent=2)

    >>> bar_parser = argparse.ArgumentParser(parents=[parent_parser])
    >>> bar_parser.add_argument('--bar')
    >>> bar_parser.parse_args(['--bar', 'YYY'])
    Namespace(bar='YYY', parent=None)

Note that most parent parsers will specify `add_help=False'.
Otherwise, the ArgumentParser will see two `-h/--help'
options (one in the parent and one in the child) and raise an error.

5.15.4.13 formatter_class
.........................

ArgumentParser objects allow the help formatting to be
customized by specifying an alternate formatting class.  Currently,
there are three such classes: `argparse.RawDescriptionHelpFormatter',
`argparse.RawTextHelpFormatter' and
`argparse.ArgumentDefaultsHelpFormatter'.  The first two allow more
control over how textual descriptions are displayed, while the last
automatically adds information about argument default values.

  By default, see ArgumentParser objects line-wrap the
description and epilog texts in command-line help
messages:

    >>> parser = argparse.ArgumentParser(
    ...     prog='PROG',
    ...     description='''this description
    ...         was indented weird
    ...             but that is okay''',
    ...     epilog='''
    ...             likewise for this epilog whose whitespace will
    ...         be cleaned up and whose words will be wrapped
    ...         across a couple lines''')
    >>> parser.print_help()
    usage: PROG [-h]

    this description was indented weird but that is okay

    optional arguments:
     -h, --help  show this help message and exit

    likewise for this epilog whose whitespace will be cleaned up and whose words
    will be wrapped across a couple lines

Passing `argparse.RawDescriptionHelpFormatter' as `formatter_class='
indicates that description and epilog are
already correctly formatted and should not be line-wrapped:

    >>> parser = argparse.ArgumentParser(
    ...     prog='PROG',
    ...     formatter_class=argparse.RawDescriptionHelpFormatter,
    ...     description=textwrap.dedent('''\
    ...         Please do not mess up this text!
    ...         --------------------------------
    ...             I have indented it
    ...             exactly the way
    ...             I want it
    ...         '''))
    >>> parser.print_help()
    usage: PROG [-h]

    Please do not mess up this text!
    --------------------------------
       I have indented it
       exactly the way
       I want it

    optional arguments:
     -h, --help  show this help message and exit

`RawTextHelpFormatter' maintains whitespace for all sorts of help text
including argument descriptions.

  The other formatter class available, `ArgumentDefaultsHelpFormatter',
will add information about the default value of each of the arguments:

    >>> parser = argparse.ArgumentParser(
    ...     prog='PROG',
    ...     formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    >>> parser.add_argument('--foo', type=int, default=42, help='FOO!')
    >>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!')
    >>> parser.print_help()
    usage: PROG [-h] [--foo FOO] [bar [bar ...]]

    positional arguments:
     bar         BAR! (default: [1, 2, 3])

    optional arguments:
     -h, --help  show this help message and exit
     --foo FOO   FOO! (default: 42)

5.15.4.14 conflict_handler
..........................

ArgumentParser objects do not allow two actions with the
same option string.  By default, see ArgumentParser objects
raises an exception if an attempt is made to create an argument with an
option string that is already in use:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-f', '--foo', help='old foo help')
    >>> parser.add_argument('--foo', help='new foo help')
    Traceback (most recent call last):
     ..
    ArgumentError: argument --foo: conflicting option string(s): --foo

Sometimes (e.g. when using parents) it may be useful to
simply override any older arguments with the same option string.  To
get this behavior, the value `'resolve'' can be supplied to the
`conflict_handler=' argument of ArgumentParser.:

    >>> parser = argparse.ArgumentParser(prog='PROG', conflict_handler='resolve')
    >>> parser.add_argument('-f', '--foo', help='old foo help')
    >>> parser.add_argument('--foo', help='new foo help')
    >>> parser.print_help()
    usage: PROG [-h] [-f FOO] [--foo FOO]

    optional arguments:
     -h, --help  show this help message and exit
     -f FOO      old foo help
     --foo FOO   new foo help

Note that ArgumentParser objects only remove an action if
all of its option strings are overridden.  So, in the example above,
the old `-f/--foo' action is retained as the `-f' action, because only
the `--foo' option string was overridden.

5.15.4.15 prog
..............

By default, see ArgumentParser objects uses `sys.argv[0]' to
determine how to display the name of the program in help messages.
This default is almost always desirable because it will make the help
messages match how the program was invoked on the command line.  For
example, consider a file named `myprogram.py' with the following code:

    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--foo', help='foo help')
    args = parser.parse_args()

The help for this program will display `myprogram.py' as the program
name (regardless of where the program was invoked from):

    $ python myprogram.py --help
    usage: myprogram.py [-h] [--foo FOO]

    optional arguments:
     -h, --help  show this help message and exit
     --foo FOO   foo help
    $ cd ..
    $ python subdir\myprogram.py --help
    usage: myprogram.py [-h] [--foo FOO]

    optional arguments:
     -h, --help  show this help message and exit
     --foo FOO   foo help

To change this default behavior, another value can be supplied using the
`prog=' argument to ArgumentParser.:

    >>> parser = argparse.ArgumentParser(prog='myprogram')
    >>> parser.print_help()
    usage: myprogram [-h]

    optional arguments:
     -h, --help  show this help message and exit

Note that the program name, whether determined from `sys.argv[0]' or
from the `prog=' argument, is available to help messages using the
`%(prog)s' format specifier.

    >>> parser = argparse.ArgumentParser(prog='myprogram')
    >>> parser.add_argument('--foo', help='foo of the %(prog)s program')
    >>> parser.print_help()
    usage: myprogram [-h] [--foo FOO]

    optional arguments:
     -h, --help  show this help message and exit
     --foo FOO   foo of the myprogram program

5.15.4.16 usage
...............

By default, see ArgumentParser calculates the usage message
from the arguments it contains:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('--foo', nargs='?', help='foo help')
    >>> parser.add_argument('bar', nargs='+', help='bar help')
    >>> parser.print_help()
    usage: PROG [-h] [--foo [FOO]] bar [bar ...]

    positional arguments:
     bar          bar help

    optional arguments:
     -h, --help   show this help message and exit
     --foo [FOO]  foo help

The default message can be overridden with the `usage=' keyword
argument:

    >>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]')
    >>> parser.add_argument('--foo', nargs='?', help='foo help')
    >>> parser.add_argument('bar', nargs='+', help='bar help')
    >>> parser.print_help()
    usage: PROG [options]

    positional arguments:
     bar          bar help

    optional arguments:
     -h, --help   show this help message and exit
     --foo [FOO]  foo help

The `%(prog)s' format specifier is available to fill in the program
name in your usage messages.

5.15.4.17 The add_argument() method
...................................

 -- Method: ArgumentParser.add_argument (name or flags...[, action][,
          nargs][, const][, default][, type][, choices][, required][,
          help][, metavar][, dest])
     Define how a single command line argument should be parsed.  Each
     parameter has its own more detailed description below, but in
     short they are:

        * name or flags. - Either a name or a list of
          option strings, e.g. `foo' or `-f, --foo'

        * action. - The basic type of action to be taken
          when this argument is encountered at the command-line.

        * nargs. - The number of command-line arguments
          that should be consumed.

        * const. - A constant value required by some
          action and nargs selections.

        * default. - The value produced if the argument is
          absent from the command-line.

        * type. - The type to which the command-line arg
          should be converted.

        * choices. - A container of the allowable values
          for the argument.

        * required. - Whether or not the command-line
          option may be omitted (optionals only).

        * help. - A brief description of what the argument
          does.

        * metavar. - A name for the argument in usage
          messages.

        * dest. - The name of the attribute to be added to
          the object returned by parse_args().

The following sections describe how each of these are used.

5.15.4.18 name or flags
.......................

The `add_argument()' method must know whether an optional argument, like
`-f' or `--foo', or a positional argument, like a list of filenames, is
expected.  The first arguments passed to `add_argument()' must
therefore be either a series of flags, or a simple argument name.  For
example, an optional argument could be created like:

    >>> parser.add_argument('-f', '--foo')

while a positional argument could be created like:

    >>> parser.add_argument('bar')

When `parse_args()' is called, optional arguments will be identified by
the `-' prefix, and the remaining arguments will be assumed to be
positional:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-f', '--foo')
    >>> parser.add_argument('bar')
    >>> parser.parse_args(['BAR'])
    Namespace(bar='BAR', foo=None)
    >>> parser.parse_args(['BAR', '--foo', 'FOO'])
    Namespace(bar='BAR', foo='FOO')
    >>> parser.parse_args(['--foo', 'FOO'])
    usage: PROG [-h] [-f FOO] bar
    PROG: error: too few arguments

5.15.4.19 action
................

ArgumentParser objects associate command-line args with
actions.  These actions can do just about anything with the
command-line args associated with them, though most actions simply add
an attribute to the object returned by `parse_args()'.  The `action'
keyword argument specifies how the command-line args should be handled.
The supported actions are:

   *
    `'store'' - This just stores the argument's value.  This is the default
          action. For example:

              >>> parser = argparse.ArgumentParser()
              >>> parser.add_argument('--foo')
              >>> parser.parse_args('--foo 1'.split())
              Namespace(foo='1')

   *
    `'store_const'' - This stores the value specified by the const keyword
          argument.  (Note that the const keyword argument
          defaults to the rather unhelpful `None'.)  The
          `'store_const'' action is most commonly used with optional
          arguments that specify some sort of flag.  For example:

              >>> parser = argparse.ArgumentParser()
              >>> parser.add_argument('--foo', action='store_const', const=42)
              >>> parser.parse_args('--foo'.split())
              Namespace(foo=42)

   * `'store_true'' and `'store_false'' - These store the values `True'
     and `False' respectively.  These are special cases of
     `'store_const''.  For example:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', action='store_true')
         >>> parser.add_argument('--bar', action='store_false')
         >>> parser.parse_args('--foo --bar'.split())
         Namespace(bar=False, foo=True)

   * `'append'' - This stores a list, and appends each argument value
     to the list.  This is useful to allow an option to be specified
     multiple times.  Example usage:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', action='append')
         >>> parser.parse_args('--foo 1 --foo 2'.split())
         Namespace(foo=['1', '2'])

   * `'append_const'' - This stores a list, and appends the value
     specified by the const keyword argument to the list.
     (Note that the const keyword argument defaults to
     `None'.)  The `'append_const'' action is typically useful when
     multiple arguments need to store constants to the same list. For
     example:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--str', dest='types', action='append_const', const=str)
         >>> parser.add_argument('--int', dest='types', action='append_const', const=int)
         >>> parser.parse_args('--str --int'.split())
         Namespace(types=[<type 'str'>, <type 'int'>])

   * `'version'' - This expects a `version=' keyword argument in the
     `add_argument()' call, and prints version information and exits
     when invoked.

         >>> import argparse
         >>> parser = argparse.ArgumentParser(prog='PROG')
         >>> parser.add_argument('--version', action='version', version='%(prog)s 2.0')
         >>> parser.parse_args(['--version'])
         PROG 2.0

  You can also specify an arbitrary action by passing an object that
implements the Action API.  The easiest way to do this is to extend
`argparse.Action', supplying an appropriate `__call__' method.  The
`__call__' method should accept four parameters:

   * `parser' - The ArgumentParser object which contains this action.

   * `namespace' - The namespace object that will be returned by
     `parse_args()'.  Most actions add an attribute to this object.

   * `values' - The associated command-line args, with any
     type-conversions applied.  (Type-conversions are specified with
     the type keyword argument to `add_argument()'.

   * `option_string' - The option string that was used to invoke this
     action.  The `option_string' argument is optional, and will be
     absent if the action is associated with a positional argument.

  An example of a custom action:

    >>> class FooAction(argparse.Action):
    ...     def __call__(self, parser, namespace, values, option_string=None):
    ...         print '%r %r %r' % (namespace, values, option_string)
    ...         setattr(namespace, self.dest, values)
    ...
    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', action=FooAction)
    >>> parser.add_argument('bar', action=FooAction)
    >>> args = parser.parse_args('1 --foo 2'.split())
    Namespace(bar=None, foo=None) '1' None
    Namespace(bar='1', foo=None) '2' '--foo'
    >>> args
    Namespace(bar='1', foo='2')

5.15.4.20 nargs
...............

ArgumentParser objects usually associate a single command-line argument
with a single action to be taken.  The `nargs' keyword argument
associates a different number of command-line arguments with a single
action..  The supported values are:

   * N (an integer).  N args from the command-line will be gathered
     together into a list.  For example:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', nargs=2)
         >>> parser.add_argument('bar', nargs=1)
         >>> parser.parse_args('c --foo a b'.split())
         Namespace(bar=['c'], foo=['a', 'b'])

     Note that `nargs=1' produces a list of one item.  This is
     different from the default, in which the item is produced by
     itself.

   * `'?''. One arg will be consumed from the command-line if possible,
     and produced as a single item.  If no command-line arg is present,
     the value from default will be produced.  Note that
     for optional arguments, there is an additional case - the option
     string is present but not followed by a command-line arg.  In this
     case the value from const will be produced.  Some
     examples to illustrate this:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', nargs='?', const='c', default='d')
         >>> parser.add_argument('bar', nargs='?', default='d')
         >>> parser.parse_args('XX --foo YY'.split())
         Namespace(bar='XX', foo='YY')
         >>> parser.parse_args('XX --foo'.split())
         Namespace(bar='XX', foo='c')
         >>> parser.parse_args(''.split())
         Namespace(bar='d', foo='d')

     One of the more common uses of `nargs='?'' is to allow optional
     input and output files:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'),
         ...                     default=sys.stdin)
         >>> parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'),
         ...                     default=sys.stdout)
         >>> parser.parse_args(['input.txt', 'output.txt'])
         Namespace(infile=<open file 'input.txt', mode 'r' at 0x...>, outfile=<open file 'output.txt', mode 'w' at 0x...>)
         >>> parser.parse_args([])
         Namespace(infile=<open file '<stdin>', mode 'r' at 0x...>, outfile=<open file '<stdout>', mode 'w' at 0x...>)

   * `'*''.  All command-line args present are gathered into a list.
     Note that it generally doesn't make much sense to have more than
     one positional argument with `nargs='*'', but multiple optional
     arguments with `nargs='*'' is possible.  For example:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', nargs='*')
         >>> parser.add_argument('--bar', nargs='*')
         >>> parser.add_argument('baz', nargs='*')
         >>> parser.parse_args('a b --foo x y --bar 1 2'.split())
         Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y'])

   * `'+''. Just like `'*'', all command-line args present are gathered
     into a list.  Additionally, an error message will be generated if
     there wasn't at least one command-line arg present.  For example:

         >>> parser = argparse.ArgumentParser(prog='PROG')
         >>> parser.add_argument('foo', nargs='+')
         >>> parser.parse_args('a b'.split())
         Namespace(foo=['a', 'b'])
         >>> parser.parse_args(''.split())
         usage: PROG [-h] foo [foo ...]
         PROG: error: too few arguments

  If the `nargs' keyword argument is not provided, the number of args
consumed is determined by the action.  Generally this means
a single command-line arg will be consumed and a single item (not a
list) will be produced.

5.15.4.21 const
...............

The `const' argument of `add_argument()' is used to hold constant values
that are not read from the command line but are required for the various
ArgumentParser actions.  The two most common uses of it are:

   * When `add_argument()' is called with `action='store_const'' or
     `action='append_const''.  These actions add the `const' value to
     one of the attributes of the object returned by `parse_args()'.
     See the action.  description for examples.

   * When `add_argument()' is called with option strings (like `-f' or
     `--foo') and `nargs='?''.  This creates an optional argument that
     can be followed by zero or one command-line args.  When parsing
     the command-line, if the option string is encountered with no
     command-line arg following it, the value of `const' will be
     assumed instead. See the nargs description for
     examples.

  The `const' keyword argument defaults to `None'.

5.15.4.22 default
.................

All optional arguments and some positional arguments may be omitted at
the command-line.  The `default' keyword argument of `add_argument()',
whose value defaults to `None', specifies what value should be used if
the command-line arg is not present.  For optional arguments, the
`default' value is used when the option string was not present at the
command line:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', default=42)
    >>> parser.parse_args('--foo 2'.split())
    Namespace(foo='2')
    >>> parser.parse_args(''.split())
    Namespace(foo=42)

For positional arguments with nargs. `='?'' or `'*'', the
`default' value is used when no command-line arg was present:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('foo', nargs='?', default=42)
    >>> parser.parse_args('a'.split())
    Namespace(foo='a')
    >>> parser.parse_args(''.split())
    Namespace(foo=42)

Providing `default=argparse.SUPPRESS' causes no attribute to be added
if the command-line argument was not present.:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', default=argparse.SUPPRESS)
    >>> parser.parse_args([])
    Namespace()
    >>> parser.parse_args(['--foo', '1'])
    Namespace(foo='1')

5.15.4.23 type
..............

By default, ArgumentParser objects read command-line args in as simple
strings.  However, quite often the command-line string should instead
be interpreted as another type, like a float, int
or file.  The `type' keyword argument of
`add_argument()' allows any necessary type-checking and
type-conversions to be performed.  Many common built-in types can be
used directly as the value of the `type' argument:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('foo', type=int)
    >>> parser.add_argument('bar', type=file)
    >>> parser.parse_args('2 temp.txt'.split())
    Namespace(bar=<open file 'temp.txt', mode 'r' at 0x...>, foo=2)

To ease the use of various types of files, the argparse module provides
the factory FileType which takes the `mode=' and `bufsize=' arguments
of the `file' object.  For example, `FileType('w')' can be used to
create a writable file:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('bar', type=argparse.FileType('w'))
    >>> parser.parse_args(['out.txt'])
    Namespace(bar=<open file 'out.txt', mode 'w' at 0x...>)

`type=' can take any callable that takes a single string argument and
returns the type-converted value:

    >>> def perfect_square(string):
    ...     value = int(string)
    ...     sqrt = math.sqrt(value)
    ...     if sqrt != int(sqrt):
    ...         msg = "%r is not a perfect square" % string
    ...         raise argparse.ArgumentTypeError(msg)
    ...     return value
    ...
    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('foo', type=perfect_square)
    >>> parser.parse_args('9'.split())
    Namespace(foo=9)
    >>> parser.parse_args('7'.split())
    usage: PROG [-h] foo
    PROG: error: argument foo: '7' is not a perfect square

The choices keyword argument may be more convenient for
type checkers that simply check against a range of values:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('foo', type=int, choices=xrange(5, 10))
    >>> parser.parse_args('7'.split())
    Namespace(foo=7)
    >>> parser.parse_args('11'.split())
    usage: PROG [-h] {5,6,7,8,9}
    PROG: error: argument foo: invalid choice: 11 (choose from 5, 6, 7, 8, 9)

See the choices section for more details.

5.15.4.24 choices
.................

Some command-line args should be selected from a restricted set of
values.  These can be handled by passing a container object as the
`choices' keyword argument to `add_argument()'.  When the command-line
is parsed, arg values will be checked, and an error message will be
displayed if the arg was not one of the acceptable values:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('foo', choices='abc')
    >>> parser.parse_args('c'.split())
    Namespace(foo='c')
    >>> parser.parse_args('X'.split())
    usage: PROG [-h] {a,b,c}
    PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c')

Note that inclusion in the `choices' container is checked after any
type.  conversions have been performed, so the type of the
objects in the `choices' container should match the type.
specified:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('foo', type=complex, choices=[1, 1j])
    >>> parser.parse_args('1j'.split())
    Namespace(foo=1j)
    >>> parser.parse_args('-- -4'.split())
    usage: PROG [-h] {1,1j}
    PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j)

Any object that supports the `in' operator can be passed as the
`choices' value, so dict objects, see set objects,
custom containers, etc. are all supported.

5.15.4.25 required
..................

In general, the argparse module assumes that flags like `-f' and `--bar'
indicate _optional_ arguments, which can always be omitted at the
command-line.  To make an option _required_, `True' can be specified
for the `required=' keyword argument to `add_argument()':

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', required=True)
    >>> parser.parse_args(['--foo', 'BAR'])
    Namespace(foo='BAR')
    >>> parser.parse_args([])
    usage: argparse.py [-h] [--foo FOO]
    argparse.py: error: option --foo is required

As the example shows, if an option is marked as `required',
`parse_args()' will report an error if that option is not present at
the command line.

     Note: Required options are generally considered bad form because
     users expect _options_ to be _optional_, and thus they should be
     avoided when possible.

5.15.4.26 help
..............

The `help' value is a string containing a brief description of the
argument.  When a user requests help (usually by using `-h' or `--help'
at the command-line), these `help' descriptions will be displayed with
each argument:

    >>> parser = argparse.ArgumentParser(prog='frobble')
    >>> parser.add_argument('--foo', action='store_true',
    ...         help='foo the bars before frobbling')
    >>> parser.add_argument('bar', nargs='+',
    ...         help='one of the bars to be frobbled')
    >>> parser.parse_args('-h'.split())
    usage: frobble [-h] [--foo] bar [bar ...]

    positional arguments:
     bar     one of the bars to be frobbled

    optional arguments:
     -h, --help  show this help message and exit
     --foo   foo the bars before frobbling

The `help' strings can include various format specifiers to avoid
repetition of things like the program name or the argument
default.  The available specifiers include the program name,
`%(prog)s' and most keyword arguments to `add_argument()', e.g.
`%(default)s', `%(type)s', etc.:

    >>> parser = argparse.ArgumentParser(prog='frobble')
    >>> parser.add_argument('bar', nargs='?', type=int, default=42,
    ...         help='the bar to %(prog)s (default: %(default)s)')
    >>> parser.print_help()
    usage: frobble [-h] [bar]

    positional arguments:
     bar     the bar to frobble (default: 42)

    optional arguments:
     -h, --help  show this help message and exit

5.15.4.27 metavar
.................

When ArgumentParser generates help messages, it need some
way to refer to each expected argument.  By default, ArgumentParser
objects use the dest.  value as the "name" of each object.
By default, for positional argument actions, the dest.
value is used directly, and for optional argument actions, the
dest value is uppercased.  So, a single positional argument with
`dest='bar'' will that argument will be referred to as `bar'. A single
optional argument `--foo' that should be followed by a single
command-line arg will be referred to as `FOO'.  An example:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo')
    >>> parser.add_argument('bar')
    >>> parser.parse_args('X --foo Y'.split())
    Namespace(bar='X', foo='Y')
    >>> parser.print_help()
    usage:  [-h] [--foo FOO] bar

    positional arguments:
     bar

    optional arguments:
     -h, --help  show this help message and exit
     --foo FOO

An alternative name can be specified with `metavar':

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', metavar='YYY')
    >>> parser.add_argument('bar', metavar='XXX')
    >>> parser.parse_args('X --foo Y'.split())
    Namespace(bar='X', foo='Y')
    >>> parser.print_help()
    usage:  [-h] [--foo YYY] XXX

    positional arguments:
     XXX

    optional arguments:
     -h, --help  show this help message and exit
     --foo YYY

Note that `metavar' only changes the _displayed_ name - the name of the
attribute on the `parse_args()' object is still determined by the
dest.  value.

  Different values of `nargs' may cause the metavar to be used multiple
times.  Providing a tuple to `metavar' specifies a different display
for each of the arguments:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-x', nargs=2)
    >>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz'))
    >>> parser.print_help()
    usage: PROG [-h] [-x X X] [--foo bar baz]

    optional arguments:
     -h, --help     show this help message and exit
     -x X X
     --foo bar baz

5.15.4.28 dest
..............

Most ArgumentParser actions add some value as an attribute
of the object returned by `parse_args()'.  The name of this attribute
is determined by the `dest' keyword argument of `add_argument()'.  For
positional argument actions, `dest' is normally supplied as the first
argument to `add_argument()':

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('bar')
    >>> parser.parse_args('XXX'.split())
    Namespace(bar='XXX')

For optional argument actions, the value of `dest' is normally inferred
from the option strings.  ArgumentParser generates the
value of `dest' by taking the first long option string and stripping
away the initial `'--'' string.  If no long option strings were
supplied, `dest' will be derived from the first short option string by
stripping the initial `'-'' character.  Any internal `'-'' characters
will be converted to `'_'' characters to make sure the string is a
valid attribute name.  The examples below illustrate this behavior:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('-f', '--foo-bar', '--foo')
    >>> parser.add_argument('-x', '-y')
    >>> parser.parse_args('-f 1 -x 2'.split())
    Namespace(foo_bar='1', x='2')
    >>> parser.parse_args('--foo 1 -y 2'.split())
    Namespace(foo_bar='1', x='2')

`dest' allows a custom attribute name to be provided:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', dest='bar')
    >>> parser.parse_args('--foo XXX'.split())
    Namespace(bar='XXX')

5.15.4.29 The parse_args() method
.................................

 -- Method: ArgumentParser.parse_args (args=None, namespace=None)
     Convert argument strings to objects and assign them as attributes
     of the namespace.  Return the populated namespace.

     Previous calls to add_argument() determine exactly
     what objects are created and how they are assigned. See the
     documentation for add_argument() for details.

     By default, the arg strings are taken from sys.argv,
     and a new empty `Namespace' object is created for the attributes.

5.15.4.30 Option value syntax
.............................

The `parse_args()' method supports several ways of specifying the value
of an option (if it takes one).  In the simplest case, the option and
its value are passed as two separate arguments:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-x')
    >>> parser.add_argument('--foo')
    >>> parser.parse_args('-x X'.split())
    Namespace(foo=None, x='X')
    >>> parser.parse_args('--foo FOO'.split())
    Namespace(foo='FOO', x=None)

For long options (options with names longer than a single character),
the option and value can also be passed as a single command line
argument, using `=' to separate them:

    >>> parser.parse_args('--foo=FOO'.split())
    Namespace(foo='FOO', x=None)

For short options (options only one character long), the option and its
value can be concatenated:

    >>> parser.parse_args('-xX'.split())
    Namespace(foo=None, x='X')

Several short options can be joined together, using only a single `-'
prefix, as long as only the last option (or none of them) requires a
value:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-x', action='store_true')
    >>> parser.add_argument('-y', action='store_true')
    >>> parser.add_argument('-z')
    >>> parser.parse_args('-xyzZ'.split())
    Namespace(x=True, y=True, z='Z')

5.15.4.31 Invalid arguments
...........................

While parsing the command-line, `parse_args' checks for a variety of
errors, including ambiguous options, invalid types, invalid options,
wrong number of positional arguments, etc.  When it encounters such an
error, it exits and prints the error along with a usage message:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('--foo', type=int)
    >>> parser.add_argument('bar', nargs='?')

    >>> # invalid type
    >>> parser.parse_args(['--foo', 'spam'])
    usage: PROG [-h] [--foo FOO] [bar]
    PROG: error: argument --foo: invalid int value: 'spam'

    >>> # invalid option
    >>> parser.parse_args(['--bar'])
    usage: PROG [-h] [--foo FOO] [bar]
    PROG: error: no such option: --bar

    >>> # wrong number of arguments
    >>> parser.parse_args(['spam', 'badger'])
    usage: PROG [-h] [--foo FOO] [bar]
    PROG: error: extra arguments found: badger

5.15.4.32 Arguments containing `"-"'
....................................

The `parse_args' method attempts to give errors whenever the user has
clearly made a mistake, but some situations are inherently ambiguous.
For example, the command-line arg `'-1'' could either be an attempt to
specify an option or an attempt to provide a positional argument.  The
`parse_args' method is cautious here: positional arguments may only
begin with `'-'' if they look like negative numbers and there are no
options in the parser that look like negative numbers:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-x')
    >>> parser.add_argument('foo', nargs='?')

    >>> # no negative number options, so -1 is a positional argument
    >>> parser.parse_args(['-x', '-1'])
    Namespace(foo=None, x='-1')

    >>> # no negative number options, so -1 and -5 are positional arguments
    >>> parser.parse_args(['-x', '-1', '-5'])
    Namespace(foo='-5', x='-1')

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-1', dest='one')
    >>> parser.add_argument('foo', nargs='?')

    >>> # negative number options present, so -1 is an option
    >>> parser.parse_args(['-1', 'X'])
    Namespace(foo=None, one='X')

    >>> # negative number options present, so -2 is an option
    >>> parser.parse_args(['-2'])
    usage: PROG [-h] [-1 ONE] [foo]
    PROG: error: no such option: -2

    >>> # negative number options present, so both -1s are options
    >>> parser.parse_args(['-1', '-1'])
    usage: PROG [-h] [-1 ONE] [foo]
    PROG: error: argument -1: expected one argument

If you have positional arguments that must begin with `'-'' and don't
look like negative numbers, you can insert the pseudo-argument `'--''
which tells `parse_args' that everything after that is a positional
argument:

    >>> parser.parse_args(['--', '-f'])
    Namespace(foo='-f', one=None)

5.15.4.33 Argument abbreviations
................................

The `parse_args()' method allows long options to be abbreviated if the
abbreviation is unambiguous:

    >>> parser = argparse.ArgumentParser(prog='PROG')
    >>> parser.add_argument('-bacon')
    >>> parser.add_argument('-badger')
    >>> parser.parse_args('-bac MMM'.split())
    Namespace(bacon='MMM', badger=None)
    >>> parser.parse_args('-bad WOOD'.split())
    Namespace(bacon=None, badger='WOOD')
    >>> parser.parse_args('-ba BA'.split())
    usage: PROG [-h] [-bacon BACON] [-badger BADGER]
    PROG: error: ambiguous option: -ba could match -badger, -bacon

An error is produced for arguments that could produce more than one
options.

5.15.4.34 Beyond `sys.argv'
...........................

Sometimes it may be useful to have an ArgumentParser parse args other
than those of sys.argv.  This can be accomplished by passing
a list of strings to `parse_args'.  This is useful for testing at the
interactive prompt:

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument(
    ...     'integers', metavar='int', type=int, choices=xrange(10),
    ...  nargs='+', help='an integer in the range 0..9')
    >>> parser.add_argument(
    ...     '--sum', dest='accumulate', action='store_const', const=sum,
    ...   default=max, help='sum the integers (default: find the max)')
    >>> parser.parse_args(['1', '2', '3', '4'])
    Namespace(accumulate=<built-in function max>, integers=[1, 2, 3, 4])
    >>> parser.parse_args('1 2 3 4 --sum'.split())
    Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4])

5.15.4.35 Custom namespaces
...........................

It may also be useful to have an ArgumentParser assign
attributes to an already existing object, rather than the newly-created
`Namespace' object that is normally used.  This can be achieved by
specifying the `namespace=' keyword argument:

    >>> class C(object):
    ...     pass
    ...
    >>> c = C()
    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo')
    >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c)
    >>> c.foo
    'BAR'

5.15.4.36 Other utilities
.........................

5.15.4.37 Sub-commands
......................

 -- Method: ArgumentParser.add_subparsers ()
     Many programs split up their functionality into a number of
     sub-commands, for example, the `svn' program can invoke
     sub-commands like `svn checkout', `svn update', and `svn commit'.
     Splitting up functionality this way can be a particularly good
     idea when a program performs several different functions which
     require different kinds of command-line arguments.
     ArgumentParser supports the creation of such sub-commands
     with the add_subparsers() method.  The
     add_subparsers() method is normally called with no
     arguments and returns an special action object.  This object has a
     single method, `add_parser', which takes a command name and any
     ArgumentParser constructor arguments, and returns an
     ArgumentParser object that can be modified as usual.

     Some example usage:

         >>> # create the top-level parser
         >>> parser = argparse.ArgumentParser(prog='PROG')
         >>> parser.add_argument('--foo', action='store_true', help='foo help')
         >>> subparsers = parser.add_subparsers(help='sub-command help')
         >>>
         >>> # create the parser for the "a" command
         >>> parser_a = subparsers.add_parser('a', help='a help')
         >>> parser_a.add_argument('bar', type=int, help='bar help')
         >>>
         >>> # create the parser for the "b" command
         >>> parser_b = subparsers.add_parser('b', help='b help')
         >>> parser_b.add_argument('--baz', choices='XYZ', help='baz help')
         >>>
         >>> # parse some arg lists
         >>> parser.parse_args(['a', '12'])
         Namespace(bar=12, foo=False)
         >>> parser.parse_args(['--foo', 'b', '--baz', 'Z'])
         Namespace(baz='Z', foo=True)

     Note that the object returned by parse_args() will
     only contain attributes for the main parser and the subparser that
     was selected by the command line (and not any other subparsers).
     So in the example above, when the `"a"' command is specified, only
     the `foo' and `bar' attributes are present, and when the `"b"'
     command is specified, only the `foo' and `baz' attributes are
     present.

     Similarly, when a help message is requested from a subparser, only
     the help for that particular parser will be printed.  The help
     message will not include parent parser or sibling parser messages.
     (A help message for each subparser command, however, can be given
     by supplying the `help=' argument to `add_parser' as above.)

         >>> parser.parse_args(['--help'])
         usage: PROG [-h] [--foo] {a,b} ...

         positional arguments:
           {a,b}   sub-command help
         a     a help
         b     b help

         optional arguments:
           -h, --help  show this help message and exit
           --foo   foo help

         >>> parser.parse_args(['a', '--help'])
         usage: PROG a [-h] bar

         positional arguments:
           bar     bar help

         optional arguments:
           -h, --help  show this help message and exit

         >>> parser.parse_args(['b', '--help'])
         usage: PROG b [-h] [--baz {X,Y,Z}]

         optional arguments:
           -h, --help     show this help message and exit
           --baz {X,Y,Z}  baz help

     The add_subparsers() method also supports `title' and
     `description' keyword arguments.  When either is present, the
     subparser's commands will appear in their own group in the help
     output.  For example:

         >>> parser = argparse.ArgumentParser()
         >>> subparsers = parser.add_subparsers(title='subcommands',
         ...                                    description='valid subcommands',
         ...                                    help='additional help')
         >>> subparsers.add_parser('foo')
         >>> subparsers.add_parser('bar')
         >>> parser.parse_args(['-h'])
         usage:  [-h] {foo,bar} ...

         optional arguments:
           -h, --help  show this help message and exit

         subcommands:
           valid subcommands

           {foo,bar}   additional help

     One particularly effective way of handling sub-commands is to
     combine the use of the add_subparsers() method with
     calls to set_defaults() so that each subparser knows
     which Python function it should execute.  For example:

         >>> # sub-command functions
         >>> def foo(args):
         ...     print args.x * args.y
         ...
         >>> def bar(args):
         ...     print '((%s))' % args.z
         ...
         >>> # create the top-level parser
         >>> parser = argparse.ArgumentParser()
         >>> subparsers = parser.add_subparsers()
         >>>
         >>> # create the parser for the "foo" command
         >>> parser_foo = subparsers.add_parser('foo')
         >>> parser_foo.add_argument('-x', type=int, default=1)
         >>> parser_foo.add_argument('y', type=float)
         >>> parser_foo.set_defaults(func=foo)
         >>>
         >>> # create the parser for the "bar" command
         >>> parser_bar = subparsers.add_parser('bar')
         >>> parser_bar.add_argument('z')
         >>> parser_bar.set_defaults(func=bar)
         >>>
         >>> # parse the args and call whatever function was selected
         >>> args = parser.parse_args('foo 1 -x 2'.split())
         >>> args.func(args)
         2.0
         >>>
         >>> # parse the args and call whatever function was selected
         >>> args = parser.parse_args('bar XYZYX'.split())
         >>> args.func(args)
         ((XYZYX))

     This way, you can let parse_args() does the job of
     calling the appropriate function after argument parsing is
     complete.  Associating functions with actions like this is
     typically the easiest way to handle the different actions for each
     of your subparsers.  However, if it is necessary to check the name
     of the subparser that was invoked, the `dest' keyword argument to
     the add_subparsers() call will work:

         >>> parser = argparse.ArgumentParser()
         >>> subparsers = parser.add_subparsers(dest='subparser_name')
         >>> subparser1 = subparsers.add_parser('1')
         >>> subparser1.add_argument('-x')
         >>> subparser2 = subparsers.add_parser('2')
         >>> subparser2.add_argument('y')
         >>> parser.parse_args(['2', 'frobble'])
         Namespace(subparser_name='2', y='frobble')

5.15.4.38 FileType objects
..........................

 -- Class: argparse.FileType (mode='r', bufsize=None)
     The FileType factory creates objects that can be
     passed to the type argument of
     ArgumentParser.add_argument().  Arguments that have
     FileType objects as their type will open command-line args
     as files with the requested modes and buffer sizes:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--output', type=argparse.FileType('wb', 0))
         >>> parser.parse_args(['--output', 'out'])
         Namespace(output=<open file 'out', mode 'wb' at 0x...>)

     FileType objects understand the pseudo-argument `'-'' and
     automatically convert this into `sys.stdin' for readable
     FileType objects and `sys.stdout' for writable
     FileType objects:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('infile', type=argparse.FileType('r'))
         >>> parser.parse_args(['-'])
         Namespace(infile=<open file '<stdin>', mode 'r' at 0x...>)

5.15.4.39 Argument groups
.........................

 -- Method: ArgumentParser.add_argument_group (title=None,
          description=None)
     By default, see ArgumentParser groups command-line
     arguments into "positional arguments" and "optional arguments"
     when displaying help messages. When there is a better conceptual
     grouping of arguments than this default one, appropriate groups
     can be created using the add_argument_group() method:

         >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False)
         >>> group = parser.add_argument_group('group')
         >>> group.add_argument('--foo', help='foo help')
         >>> group.add_argument('bar', help='bar help')
         >>> parser.print_help()
         usage: PROG [--foo FOO] bar

         group:
           bar    bar help
           --foo FOO  foo help

     The add_argument_group() method returns an argument
     group object which has an add_argument() method just
     like a regular ArgumentParser.  When an argument is
     added to the group, the parser treats it just like a normal
     argument, but displays the argument in a separate group for help
     messages.  The add_argument_group() method accepts
     _title_ and _description_ arguments which can be used to customize
     this display:

         >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False)
         >>> group1 = parser.add_argument_group('group1', 'group1 description')
         >>> group1.add_argument('foo', help='foo help')
         >>> group2 = parser.add_argument_group('group2', 'group2 description')
         >>> group2.add_argument('--bar', help='bar help')
         >>> parser.print_help()
         usage: PROG [--bar BAR] foo

         group1:
           group1 description

           foo    foo help

         group2:
           group2 description

           --bar BAR  bar help

     Note that any arguments not your user defined groups will end up
     back in the usual "positional arguments" and "optional arguments"
     sections.

5.15.4.40 Mutual exclusion
..........................

 -- Method: argparse.add_mutually_exclusive_group (required=False)
     Create a mutually exclusive group. argparse will make sure that
     only one of the arguments in the mutually exclusive group was
     present on the command line:

         >>> parser = argparse.ArgumentParser(prog='PROG')
         >>> group = parser.add_mutually_exclusive_group()
         >>> group.add_argument('--foo', action='store_true')
         >>> group.add_argument('--bar', action='store_false')
         >>> parser.parse_args(['--foo'])
         Namespace(bar=True, foo=True)
         >>> parser.parse_args(['--bar'])
         Namespace(bar=False, foo=False)
         >>> parser.parse_args(['--foo', '--bar'])
         usage: PROG [-h] [--foo | --bar]
         PROG: error: argument --bar: not allowed with argument --foo

     The add_mutually_exclusive_group() method also
     accepts a _required_ argument, to indicate that at least one of
     the mutually exclusive arguments is required:

         >>> parser = argparse.ArgumentParser(prog='PROG')
         >>> group = parser.add_mutually_exclusive_group(required=True)
         >>> group.add_argument('--foo', action='store_true')
         >>> group.add_argument('--bar', action='store_false')
         >>> parser.parse_args([])
         usage: PROG [-h] (--foo | --bar)
         PROG: error: one of the arguments --foo --bar is required

     Note that currently mutually exclusive argument groups do not
     support the _title_ and _description_ arguments of
     `add_argument_group()'.

5.15.4.41 Parser defaults
.........................

 -- Method: ArgumentParser.set_defaults (**kwargs)
     Most of the time, the attributes of the object returned by
     parse_args().  will be fully determined by inspecting the
     command-line args and the argument actions.
     ArgumentParser.set_defaults() allows some additional
     attributes that are determined without any inspection of the
     command-line to be added:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('foo', type=int)
         >>> parser.set_defaults(bar=42, baz='badger')
         >>> parser.parse_args(['736'])
         Namespace(bar=42, baz='badger', foo=736)

     Note that parser-level defaults always override argument-level
     defaults:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', default='bar')
         >>> parser.set_defaults(foo='spam')
         >>> parser.parse_args([])
         Namespace(foo='spam')

     Parser-level defaults can be particularly useful when working with
     multiple parsers.  See the add_subparsers() method
     for an example of this type.

 -- Method: ArgumentParser.get_default (dest)
     Get the default value for a namespace attribute, as set by either
     add_argument() or by set_defaults().:

         >>> parser = argparse.ArgumentParser()
         >>> parser.add_argument('--foo', default='badger')
         >>> parser.get_default('foo')
         'badger'

5.15.4.42 Printing help
.......................

In most typical applications, `parse_args()' will take care of
formatting and printing any usage or error messages.  However, several
formatting methods are available:

 -- Method: ArgumentParser.print_usage (file=None)
     Print a brief description of how the ArgumentParser.
     should be invoked on the command line.  If _file_ is `None', see
     sys.stderr is assumed.

 -- Method: ArgumentParser.print_help (file=None)
     Print a help message, including the program usage and information
     about the arguments registered with the ArgumentParser.
     If _file_ is `None', see sys.stderr is assumed.

  There are also variants of these methods that simply return a string
instead of printing it:

 -- Method: ArgumentParser.format_usage ()
     Return a string containing a brief description of how the
     ArgumentParser should be invoked on the command line.

 -- Method: ArgumentParser.format_help ()
     Return a string containing a help message, including the program
     usage and information about the arguments registered with the
     ArgumentParser.

5.15.4.43 Partial parsing
.........................

 -- Method: ArgumentParser.parse_known_args (args=None, namespace=None)

  Sometimes a script may only parse a few of the command line
arguments, passing the remaining arguments on to another script or
program. In these cases, the `parse_known_args()' method can be useful.
It works much like parse_args() except that it does not
produce an error when extra arguments are present.  Instead, it returns
a two item tuple containing the populated namespace and the list of
remaining argument strings.

    >>> parser = argparse.ArgumentParser()
    >>> parser.add_argument('--foo', action='store_true')
    >>> parser.add_argument('bar')
    >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam'])
    (Namespace(bar='BAR', foo=True), ['--badger', 'spam'])

5.15.4.44 Customizing file parsing
..................................

 -- Method: ArgumentParser.convert_arg_line_to_args (arg_line)
     Arguments that are read from a file (see the
     _fromfile_prefix_chars_ keyword argument to the
     ArgumentParser constructor) are read one argument per line.
     convert_arg_line_to_args() can be overriden for
     fancier reading.

     This method takes a single argument _arg_line_ which is a string
     read from the argument file.  It returns a list of arguments
     parsed from this string.  The method is called once per line read
     from the argument file, in order.

     A useful override of this method is one that treats each
     space-separated word as an argument:

         def convert_arg_line_to_args(self, arg_line):
             for arg in arg_line.split():
                 if not arg.strip():
                     continue
                 yield arg

5.15.4.45 Exiting methods
.........................

 -- Method: ArgumentParser.exit (status=0, message=None)
     This method terminates the program, exiting with the specified
     _status_ and, if given, it prints a _message_ before that.

 -- Method: ArgumentParser.error (message)
     This method prints a usage message including the _message_ to the
     standard output and terminates the program with a status code of 2.

5.15.4.46 Upgrading optparse code
.................................

Originally, the argparse module had attempted to maintain compatibility
with optparse.  However, optparse was difficult to extend
transparently, particularly with the changes required to support the
new `nargs=' specifiers and better usage messages.  When most
everything in optparse had either been copy-pasted over or
monkey-patched, it no longer seemed practical to try to maintain the
backwards compatibility.

  A partial upgrade path from optparse to argparse:

   * Replace all `add_option()' calls with
     ArgumentParser.add_argument() calls.

   * Replace `options, args = parser.parse_args()' with `args =
     parser.parse_args()' and add additional
     ArgumentParser.add_argument() calls for the positional
     arguments.

   * Replace callback actions and the `callback_*' keyword arguments
     with `type' or `action' arguments.

   * Replace string names for `type' keyword arguments with the
     corresponding type objects (e.g. int, float, complex, etc).

   * Replace `optparse.Values' with `Namespace' and
     `optparse.OptionError' and `optparse.OptionValueError' with
     `ArgumentError'.

   * Replace strings with implicit arguments such as `%default' or
     `%prog' with the standard python syntax to use dictionaries to
     format strings, that is, `%(default)s' and `%(prog)s'.

   * Replace the OptionParser constructor `version' argument with a
     call to `parser.add_argument('--version', action='version',
     version='<the version>')'

5.15.5 `optparse' -- Parser for command line options
----------------------------------------------------

Deprecated since version 2.7: The optparse module is
deprecated and will not be developed further; development will continue
with the argparse module.

  New in version 2.3.

  optparse is a more convenient, flexible, and powerful
library for parsing command-line options than the old getopt.
module.  optparse uses a more declarative style of
command-line parsing: you create an instance of OptionParser,
populate it with options, and parse the command line.
optparse allows users to specify options in the conventional
GNU/POSIX syntax, and additionally generates usage and help messages
for you.

  Here's an example of using optparse in a simple script:

    from optparse import OptionParser
    [...]
    parser = OptionParser()
    parser.add_option("-f", "--file", dest="filename",
                      help="write report to FILE", metavar="FILE")
    parser.add_option("-q", "--quiet",
                      action="store_false", dest="verbose", default=True,
                      help="don't print status messages to stdout")

    (options, args) = parser.parse_args()

With these few lines of code, users of your script can now do the
"usual thing" on the command-line, for example:

    <yourscript> --file=outfile -q

As it parses the command line, see optparse sets attributes of
the `options' object returned by `parse_args()' based on user-supplied
command-line values.  When `parse_args()' returns from parsing this
command line, `options.filename' will be `"outfile"' and
`options.verbose' will be `False'.  optparse supports both
long and short options, allows short options to be merged together, and
allows options to be associated with their arguments in a variety of
ways.  Thus, the following command lines are all equivalent to the
above example:

    <yourscript> -f outfile --quiet
    <yourscript> --quiet --file outfile
    <yourscript> -q -foutfile
    <yourscript> -qfoutfile

Additionally, users can run one of

    <yourscript> -h
    <yourscript> --help

and optparse will print out a brief summary of your
script's options:

    usage: <yourscript> [options]

    options:
      -h, --help            show this help message and exit
      -f FILE, --file=FILE  write report to FILE
      -q, --quiet           don't print status messages to stdout

where the value of _yourscript_ is determined at runtime (normally from
`sys.argv[0]').

5.15.5.1 Background
...................

optparse was explicitly designed to encourage the creation
of programs with straightforward, conventional command-line interfaces.
To that end, it supports only the most common command-line syntax and
semantics conventionally used under Unix.  If you are unfamiliar with
these conventions, read this section to acquaint yourself with them.

5.15.5.2 Terminology
....................

argument
     a string entered on the command-line, and passed by the shell to
     `execl()' or `execv()'.  In Python, arguments are elements of
     `sys.argv[1:]' (`sys.argv[0]' is the name of the program being
     executed).  Unix shells also use the term "word".

     It is occasionally desirable to substitute an argument list other
     than `sys.argv[1:]', so you should read "argument" as "an element
     of `sys.argv[1:]', or of some other list provided as a substitute
     for `sys.argv[1:]'".

option
     an argument used to supply extra information to guide or customize
     the execution of a program.  There are many different syntaxes for
     options; the traditional Unix syntax is a hyphen ("-") followed by
     a single letter, e.g. `"-x"' or `"-F"'.  Also, traditional Unix
     syntax allows multiple options to be merged into a single
     argument, e.g.  `"-x -F"' is equivalent to `"-xF"'.  The GNU
     project introduced `"--"' followed by a series of hyphen-separated
     words, e.g.  `"--file"' or `"--dry-run"'.  These are the only two
     option syntaxes provided by optparse.

     Some other option syntaxes that the world has seen include:

        * a hyphen followed by a few letters, e.g. `"-pf"' (this is
          _not_ the same as multiple options merged into a single
          argument)

        * a hyphen followed by a whole word, e.g. `"-file"' (this is
          technically equivalent to the previous syntax, but they
          aren't usually seen in the same program)

        * a plus sign followed by a single letter, or a few letters, or
          a word, e.g.  `"+f"', `"+rgb"'

        * a slash followed by a letter, or a few letters, or a word,
          e.g. `"/f"', `"/file"'

     These option syntaxes are not supported by optparse,
     and they never will be.  This is deliberate: the first three are
     non-standard on any environment, and the last only makes sense if
     you're exclusively targeting VMS, MS-DOS, and/or Windows.

option argument
     an argument that follows an option, is closely associated with
     that option, and is consumed from the argument list when that
     option is. With optparse, option arguments may either
     be in a separate argument from their option:

         -f foo
         --file foo

     or included in the same argument:

         -ffoo
         --file=foo

     Typically, a given option either takes an argument or it doesn't.
     Lots of people want an "optional option arguments" feature,
     meaning that some options will take an argument if they see it,
     and won't if they don't.  This is somewhat controversial, because
     it makes parsing ambiguous: if `"-a"' takes an optional argument
     and `"-b"' is another option entirely, how do we interpret
     `"-ab"'?  Because of this ambiguity, see optparse does not
     support this feature.

positional argument
     something leftover in the argument list after options have been
     parsed, i.e.  after options and their arguments have been parsed
     and removed from the argument list.

required option
     an option that must be supplied on the command-line; note that the
     phrase "required option" is self-contradictory in English.
     optparse doesn't prevent you from implementing required
     options, but doesn't give you much help at it either.

  For example, consider this hypothetical command-line:

    prog -v --report /tmp/report.txt foo bar

`"-v"' and `"--report"' are both options.  Assuming that `--report'
takes one argument, `"/tmp/report.txt"' is an option argument.  `"foo"'
and `"bar"' are positional arguments.

5.15.5.3 What are options for?
..............................

Options are used to provide extra information to tune or customize the
execution of a program.  In case it wasn't clear, options are usually
_optional_.  A program should be able to run just fine with no options
whatsoever.  (Pick a random program from the Unix or GNU toolsets.  Can
it run without any options at all and still make sense?  The main
exceptions are `find', `tar', and `dd'--all of which are mutant
oddballs that have been rightly criticized for their non-standard
syntax and confusing interfaces.)

  Lots of people want their programs to have "required options".  Think
about it.  If it's required, then it's _not optional_!  If there is a
piece of information that your program absolutely requires in order to
run successfully, that's what positional arguments are for.

  As an example of good command-line interface design, consider the
humble `cp' utility, for copying files.  It doesn't make much sense to
try to copy files without supplying a destination and at least one
source. Hence, `cp' fails if you run it with no arguments.  However, it
has a flexible, useful syntax that does not require any options at all:

    cp SOURCE DEST
    cp SOURCE ... DEST-DIR

You can get pretty far with just that.  Most `cp' implementations
provide a bunch of options to tweak exactly how the files are copied:
you can preserve mode and modification time, avoid following symlinks,
ask before clobbering existing files, etc.  But none of this distracts
from the core mission of `cp', which is to copy either one file to
another, or several files to another directory.

5.15.5.4 What are positional arguments for?
...........................................

Positional arguments are for those pieces of information that your
program absolutely, positively requires to run.

  A good user interface should have as few absolute requirements as
possible.  If your program requires 17 distinct pieces of information
in order to run successfully, it doesn't much matter _how_ you get that
information from the user--most people will give up and walk away
before they successfully run the program.  This applies whether the
user interface is a command-line, a configuration file, or a GUI: if
you make that many demands on your users, most of them will simply give
up.

  In short, try to minimize the amount of information that users are
absolutely required to supply--use sensible defaults whenever possible.
Of course, you also want to make your programs reasonably flexible.
That's what options are for.  Again, it doesn't matter if they are
entries in a config file, widgets in the "Preferences" dialog of a GUI,
or command-line options--the more options you implement, the more
flexible your program is, and the more complicated its implementation
becomes.  Too much flexibility has drawbacks as well, of course; too
many options can overwhelm users and make your code much harder to
maintain.

5.15.5.5 Tutorial
.................

While optparse is quite flexible and powerful, it's also
straightforward to use in most cases.  This section covers the code
patterns that are common to any optparse.-based program.

  First, you need to import the OptionParser class; then, early in the
main program, create an OptionParser instance:

    from optparse import OptionParser
    [...]
    parser = OptionParser()

Then you can start defining options.  The basic syntax is:

    parser.add_option(opt_str, ...,
                      attr=value, ...)

Each option has one or more option strings, such as `"-f"' or
`"--file"', and several option attributes that tell optparse
what to expect and what to do when it encounters that option on
the command line.

  Typically, each option will have one short option string and one long
option string, e.g.:

    parser.add_option("-f", "--file", ...)

You're free to define as many short option strings and as many long
option strings as you like (including zero), as long as there is at
least one option string overall.

  The option strings passed to `add_option()' are effectively labels
for the option defined by that call.  For brevity, we will frequently
refer to _encountering an option_ on the command line; in reality,
see optparse.  encounters _option strings_ and looks up options
from them.

  Once all of your options are defined, instruct optparse.
to parse your program's command line:

    (options, args) = parser.parse_args()

(If you like, you can pass a custom argument list to `parse_args()', but
that's rarely necessary: by default it uses `sys.argv[1:]'.)

  `parse_args()' returns two values:

   * `options', an object containing values for all of your
     options--e.g. if `"--file"' takes a single string argument, then
     `options.file' will be the filename supplied by the user, or
     `None' if the user did not supply that option

   * `args', the list of positional arguments leftover after parsing
     options

  This tutorial section only covers the four most important option
attributes: action, type, dest.
(destination), and help. Of these, see action is
the most fundamental.

5.15.5.6 Understanding option actions
.....................................

Actions tell optparse what to do when it encounters an
option on the command line.  There is a fixed set of actions hard-coded
into optparse.; adding new actions is an advanced topic
covered in section Extending optparse.  Most actions tell
optparse to store a value in some variable--for example,
take a string from the command line and store it in an attribute of
`options'.

  If you don't specify an option action, see optparse defaults
to `store'.

5.15.5.7 The store action
.........................

The most common option action is `store', which tells optparse
to take the next argument (or the remainder of the current
argument), ensure that it is of the correct type, and store it to your
chosen destination.

  For example:

    parser.add_option("-f", "--file",
                      action="store", type="string", dest="filename")

Now let's make up a fake command line and ask optparse to
parse it:

    args = ["-f", "foo.txt"]
    (options, args) = parser.parse_args(args)

When optparse sees the option string `"-f"', it consumes
the next argument, `"foo.txt"', and stores it in `options.filename'.
So, after this call to `parse_args()', `options.filename' is
`"foo.txt"'.

  Some other option types supported by optparse are `int'
and `float'.  Here's an option that expects an integer argument:

    parser.add_option("-n", type="int", dest="num")

Note that this option has no long option string, which is perfectly
acceptable.  Also, there's no explicit action, since the default is
`store'.

  Let's parse another fake command-line.  This time, we'll jam the
option argument right up against the option: since `"-n42"' (one
argument) is equivalent to `"-n 42"' (two arguments), the code

    (options, args) = parser.parse_args(["-n42"])
    print options.num

will print `"42"'.

  If you don't specify a type, see optparse assumes `string'.
Combined with the fact that the default action is `store', that means
our first example can be a lot shorter:

    parser.add_option("-f", "--file", dest="filename")

If you don't supply a destination, see optparse figures out a
sensible default from the option strings: if the first long option
string is `"--foo-bar"', then the default destination is `foo_bar'.  If
there are no long option strings, see optparse looks at the
first short option string: the default destination for `"-f"' is `f'.

  optparse also includes built-in `long' and `complex'
types.  Adding types is covered in section Extending optparse.

5.15.5.8 Handling boolean (flag) options
........................................

Flag options--set a variable to true or false when a particular option
is seen --are quite common.  optparse supports them with
two separate actions, `store_true' and `store_false'.  For example, you
might have a `verbose' flag that is turned on with `"-v"' and off with
`"-q"':

    parser.add_option("-v", action="store_true", dest="verbose")
    parser.add_option("-q", action="store_false", dest="verbose")

Here we have two different options with the same destination, which is
perfectly OK.  (It just means you have to be a bit careful when setting
default values-- see below.)

  When optparse encounters `"-v"' on the command line, it
sets `options.verbose' to `True'; when it encounters `"-q"',
`options.verbose' is set to `False'.

5.15.5.9 Other actions
......................

Some other actions supported by optparse are:

`"store_const"'
     store a constant value

`"append"'
     append this option's argument to a list

`"count"'
     increment a counter by one

`"callback"'
     call a specified function

  These are covered in section Reference Guide, Reference
Guide and section Option Callbacks.

5.15.5.10 Default values
........................

All of the above examples involve setting some variable (the
"destination") when certain command-line options are seen.  What
happens if those options are never seen?  Since we didn't supply any
defaults, they are all set to `None'.  This is usually fine, but
sometimes you want more control.  optparse lets you supply
a default value for each destination, which is assigned before the
command line is parsed.

  First, consider the verbose/quiet example.  If we want
optparse to set `verbose' to `True' unless `"-q"' is seen, then
we can do this:

    parser.add_option("-v", action="store_true", dest="verbose", default=True)
    parser.add_option("-q", action="store_false", dest="verbose")

Since default values apply to the _destination_ rather than to any
particular option, and these two options happen to have the same
destination, this is exactly equivalent:

    parser.add_option("-v", action="store_true", dest="verbose")
    parser.add_option("-q", action="store_false", dest="verbose", default=True)

Consider this:

    parser.add_option("-v", action="store_true", dest="verbose", default=False)
    parser.add_option("-q", action="store_false", dest="verbose", default=True)

Again, the default value for `verbose' will be `True': the last default
value supplied for any particular destination is the one that counts.

  A clearer way to specify default values is the `set_defaults()'
method of OptionParser, which you can call at any time before calling
`parse_args()':

    parser.set_defaults(verbose=True)
    parser.add_option(...)
    (options, args) = parser.parse_args()

As before, the last value specified for a given option destination is
the one that counts.  For clarity, try to use one method or the other
of setting default values, not both.

5.15.5.11 Generating help
.........................

See optparse.'s ability to generate help and usage text
automatically is useful for creating user-friendly command-line
interfaces.  All you have to do is supply a help value for
each option, and optionally a short usage message for your whole
program.  Here's an OptionParser populated with user-friendly
(documented) options:

    usage = "usage: %prog [options] arg1 arg2"
    parser = OptionParser(usage=usage)
    parser.add_option("-v", "--verbose",
                      action="store_true", dest="verbose", default=True,
                      help="make lots of noise [default]")
    parser.add_option("-q", "--quiet",
                      action="store_false", dest="verbose",
                      help="be vewwy quiet (I'm hunting wabbits)")
    parser.add_option("-f", "--filename",
                      metavar="FILE", help="write output to FILE")
    parser.add_option("-m", "--mode",
                      default="intermediate",
                      help="interaction mode: novice, intermediate, "
                           "or expert [default: %default]")

If optparse encounters either `"-h"' or `"--help"' on the
command-line, or if you just call `parser.print_help()', it prints the
following to standard output:

    usage: <yourscript> [options] arg1 arg2

    options:
      -h, --help            show this help message and exit
      -v, --verbose         make lots of noise [default]
      -q, --quiet           be vewwy quiet (I'm hunting wabbits)
      -f FILE, --filename=FILE
                            write output to FILE
      -m MODE, --mode=MODE  interaction mode: novice, intermediate, or
                            expert [default: intermediate]

(If the help output is triggered by a help option, see optparse.
exits after printing the help text.)

  There's a lot going on here to help optparse generate the
best possible help message:

   * the script defines its own usage message:

         usage = "usage: %prog [options] arg1 arg2"

     optparse expands `"%prog"' in the usage string to the
     name of the current program, i.e. `os.path.basename(sys.argv[0])'.
     The expanded string is then printed before the detailed option
     help.

     If you don't supply a usage string, see optparse uses a
     bland but sensible default: `"usage: %prog [options]"', which is
     fine if your script doesn't take any positional arguments.

   * every option defines a help string, and doesn't worry about
     line-wrapping-- see optparse takes care of wrapping lines
     and making the help output look good.

   * options that take a value indicate this fact in their
     automatically-generated help message, e.g. for the "mode" option:

         -m MODE, --mode=MODE

     Here, "MODE" is called the meta-variable: it stands for the
     argument that the user is expected to supply to -m.
     /`--mode'.  By default, see optparse converts the
     destination variable name to uppercase and uses that for the
     meta-variable.  Sometimes, that's not what you want--for example,
     the `--filename' option explicitly sets `metavar="FILE"',
     resulting in this automatically-generated option description:

         -f FILE, --filename=FILE

     This is important for more than just saving space, though: the
     manually written help text uses the meta-variable "FILE" to clue
     the user in that there's a connection between the semi-formal
     syntax "-f FILE" and the informal semantic description "write
     output to FILE". This is a simple but effective way to make your
     help text a lot clearer and more useful for end users.

  New in version 2.4: Options that have a default value can include
`%default' in the help string--see optparse will replace it
with str() of the option's default value.  If an option has
no default value (or the default value is `None'), `%default' expands
to `none'.

  When dealing with many options, it is convenient to group these
options for better help output.  An OptionParser can
contain several option groups, each of which can contain several
options.

  Continuing with the parser defined above, adding an `OptionGroup' to a
parser is easy:

    group = OptionGroup(parser, "Dangerous Options",
                        "Caution: use these options at your own risk.  "
                        "It is believed that some of them bite.")
    group.add_option("-g", action="store_true", help="Group option.")
    parser.add_option_group(group)

This would result in the following help output:

    usage:  [options] arg1 arg2

    options:
      -h, --help           show this help message and exit
      -v, --verbose        make lots of noise [default]
      -q, --quiet          be vewwy quiet (I'm hunting wabbits)
      -fFILE, --file=FILE  write output to FILE
      -mMODE, --mode=MODE  interaction mode: one of 'novice', 'intermediate'
                           [default], 'expert'

      Dangerous Options:
      Caution: use of these options is at your own risk.  It is believed that
      some of them bite.
      -g                 Group option.

5.15.5.12 Printing a version string
...................................

Similar to the brief usage string, see optparse can also print
a version string for your program.  You have to supply the string as
the `version' argument to OptionParser:

    parser = OptionParser(usage="%prog [-f] [-q]", version="%prog 1.0")

`"%prog"' is expanded just like it is in `usage'.  Apart from that,
`version' can contain anything you like.  When you supply it, see
optparse.  automatically adds a `"--version"' option to your
parser. If it encounters this option on the command line, it expands
your `version' string (by replacing `"%prog"'), prints it to stdout,
and exits.

  For example, if your script is called `/usr/bin/foo':

    $ /usr/bin/foo --version
    foo 1.0

The following two methods can be used to print and get the `version'
string:

 -- Method: OptionParser.print_version (file=None)
     Print the version message for the current program (`self.version')
     to _file_ (default stdout).  As with print_usage(),
     any occurrence of `"%prog"' in `self.version' is replaced with the
     name of the current program.  Does nothing if `self.version' is
     empty or undefined.

 -- Method: OptionParser.get_version ()
     Same as print_version() but returns the version
     string instead of printing it.

5.15.5.13 How `optparse' handles errors
.......................................

There are two broad classes of errors that optparse has to
worry about: programmer errors and user errors.  Programmer errors are
usually erroneous calls to OptionParser.add_option(), e.g.
invalid option strings, unknown option attributes, missing option
attributes, etc.  These are dealt with in the usual way: raise an
exception (either `optparse.OptionError' or TypeError) and
let the program crash.

  Handling user errors is much more important, since they are
guaranteed to happen no matter how stable your code is.
optparse can automatically detect some user errors, such as bad
option arguments (passing `"-n 4x"' where `-n' takes an integer
argument), missing arguments (`"-n"' at the end of the command line,
where `-n' takes an argument of any type).  Also, you can call
`OptionParser.error()' to signal an application-defined error condition:

    (options, args) = parser.parse_args()
    [...]
    if options.a and options.b:
        parser.error("options -a and -b are mutually exclusive")

In either case, see optparse handles the error the same way: it
prints the program's usage message and an error message to standard
error and exits with error status 2.

  Consider the first example above, where the user passes `"4x"' to an
option that takes an integer:

    $ /usr/bin/foo -n 4x
    usage: foo [options]

    foo: error: option -n: invalid integer value: '4x'

Or, where the user fails to pass a value at all:

    $ /usr/bin/foo -n
    usage: foo [options]

    foo: error: -n option requires an argument

See optparse.-generated error messages take care always to
mention the option involved in the error; be sure to do the same when
calling `OptionParser.error()' from your application code.

  If optparse.'s default error-handling behaviour does not
suit your needs, you'll need to subclass OptionParser and override its
`exit()' and/or `error()' methods.

5.15.5.14 Putting it all together
.................................

Here's what optparse.-based scripts usually look like:

    from optparse import OptionParser
    [...]
    def main():
        usage = "usage: %prog [options] arg"
        parser = OptionParser(usage)
        parser.add_option("-f", "--file", dest="filename",
                          help="read data from FILENAME")
        parser.add_option("-v", "--verbose",
                          action="store_true", dest="verbose")
        parser.add_option("-q", "--quiet",
                          action="store_false", dest="verbose")
        [...]
        (options, args) = parser.parse_args()
        if len(args) != 1:
            parser.error("incorrect number of arguments")
        if options.verbose:
            print "reading %s..." % options.filename
        [...]

    if __name__ == "__main__":
        main()

5.15.5.15 Reference Guide
.........................

5.15.5.16 Creating the parser
.............................

The first step in using optparse is to create an
OptionParser instance.

 -- Class: optparse.OptionParser (...)
     The OptionParser constructor has no required arguments, but a
     number of optional keyword arguments.  You should always pass them
     as keyword arguments, i.e. do not rely on the order in which the
     arguments are declared.

    `usage' (default: `"%prog [options]"')
          The usage summary to print when your program is run
          incorrectly or with a help option.  When optparse.
          prints the usage string, it expands `%prog' to
          `os.path.basename(sys.argv[0])' (or to `prog' if you passed
          that keyword argument).  To suppress a usage message, pass the
          special value `optparse.SUPPRESS_USAGE'.

    `option_list' (default: `[]')
          A list of Option objects to populate the parser with.  The
          options in `option_list' are added after any options in
          `standard_option_list' (a class attribute that may be set by
          OptionParser subclasses), but before any version or help
          options. Deprecated; use add_option() after
          creating the parser instead.

    `option_class' (default: optparse.Option)
          Class to use when adding options to the parser in
          add_option().

    `version' (default: `None')
          A version string to print when the user supplies a version
          option. If you supply a true value for `version', see
          optparse automatically adds a version option with the
          single option string `"--version"'.  The substring `"%prog"'
          is expanded the same as for `usage'.

    `conflict_handler' (default: `"error"')
          Specifies what to do when options with conflicting option
          strings are added to the parser; see section Conflicts between options.

    `description' (default: `None')
          A paragraph of text giving a brief overview of your program.
          optparse reformats this paragraph to fit the
          current terminal width and prints it when the user requests
          help (after `usage', but before the list of options).

    `formatter' (default: a new `IndentedHelpFormatter')
          An instance of optparse.HelpFormatter that will be used for
          printing help text.  optparse provides two
          concrete classes for this purpose: IndentedHelpFormatter and
          TitledHelpFormatter.

    `add_help_option' (default: `True')
          If true, see optparse will add a help option (with
          option strings `"-h"' and `"--help"') to the parser.

    `prog'
          The string to use when expanding `"%prog"' in `usage' and
          `version' instead of `os.path.basename(sys.argv[0])'.

    `epilog' (default: `None')
          A paragraph of help text to print after the option help.

5.15.5.17 Populating the parser
...............................

There are several ways to populate the parser with options.  The
preferred way is by using OptionParser.add_option(), as
shown in section Tutorial.  `add_option()' can be called in
one of two ways:

   * pass it an Option instance (as returned by `make_option()')

   * pass it any combination of positional and keyword arguments that
     are acceptable to `make_option()' (i.e., to the Option
     constructor), and it will create the Option instance for you

  The other alternative is to pass a list of pre-constructed Option
instances to the OptionParser constructor, as in:

    option_list = [
        make_option("-f", "--filename",
                    action="store", type="string", dest="filename"),
        make_option("-q", "--quiet",
                    action="store_false", dest="verbose"),
        ]
    parser = OptionParser(option_list=option_list)

(`make_option()' is a factory function for creating Option instances;
currently it is an alias for the Option constructor.  A future version
of optparse may split Option into several classes, and
`make_option()' will pick the right class to instantiate.  Do not
instantiate Option directly.)

5.15.5.18 Defining options
..........................

Each Option instance represents a set of synonymous command-line option
strings, e.g. `-f' and `--file'.  You can specify any number of short or
long option strings, but you must specify at least one overall option
string.

  The canonical way to create an `Option' instance is with the
`add_option()' method of OptionParser.

 -- Method: OptionParser.add_option (opt_str[, ...], attr=value, ...)
     To define an option with only a short option string:

         parser.add_option("-f", attr=value, ...)

     And to define an option with only a long option string:

         parser.add_option("--foo", attr=value, ...)

     The keyword arguments define attributes of the new Option object.
     The most important option attribute is action, and it
     largely determines which other attributes are relevant or
     required.  If you pass irrelevant option attributes, or fail to
     pass required ones, see optparse.  raises an `OptionError'
     exception explaining your mistake.

     An option's _action_ determines what optparse does
     when it encounters this option on the command-line.  The standard
     option actions hard-coded into optparse are:

    `"store"'
          store this option's argument (default)

    `"store_const"'
          store a constant value

    `"store_true"'
          store a true value

    `"store_false"'
          store a false value

    `"append"'
          append this option's argument to a list

    `"append_const"'
          append a constant value to a list

    `"count"'
          increment a counter by one

    `"callback"'
          call a specified function

    `"help"'
          print a usage message including all options and the
          documentation for them

     (If you don't supply an action, the default is `"store"'.  For
     this action, you may also supply type and dest
     option attributes; see Standard option actions)

  As you can see, most actions involve storing or updating a value
somewhere.  optparse always creates a special object for
this, conventionally called `options' (it happens to be an instance of
`optparse.Values').  Option arguments (and various other values) are
stored as attributes of this object, according to the dest.
(destination) option attribute.

  For example, when you call

    parser.parse_args()

one of the first things optparse does is create the
`options' object:

    options = Values()

If one of the options in this parser is defined with

    parser.add_option("-f", "--file", action="store", type="string", dest="filename")

and the command-line being parsed includes any of the following:

    -ffoo
    -f foo
    --file=foo
    --file foo

then optparse, on seeing this option, will do the equivalent
of

    options.filename = "foo"

The type and dest option attributes are almost
as important as action, but action is the only
one that makes sense for _all_ options.

5.15.5.19 Option attributes
...........................

The following option attributes may be passed as keyword arguments to
OptionParser.add_option().  If you pass an option attribute
that is not relevant to a particular option, or fail to pass a required
option attribute, see optparse raises `OptionError'.

 -- Attribute: Option.action
     (default: `"store"')

     Determines optparse.'s behaviour when this option is
     seen on the command line; the available options are documented
     here.

 -- Attribute: Option.type
     (default: `"string"')

     The argument type expected by this option (e.g., `"string"' or
     `"int"'); the available option types are documented here.

 -- Attribute: Option.dest
     (default: derived from option strings)

     If the option's action implies writing or modifying a value
     somewhere, this tells optparse where to write it:
     dest names an attribute of the `options' object that
     optparse builds as it parses the command line.

 -- Attribute: Option.default
     The value to use for this option's destination if the option is
     not seen on the command line.  See also
     OptionParser.set_defaults().

 -- Attribute: Option.nargs
     (default: 1)

     How many arguments of type type should be consumed
     when this option is seen.  If > 1, see optparse will store
     a tuple of values to dest.

 -- Attribute: Option.const
     For actions that store a constant value, the constant value to
     store.

 -- Attribute: Option.choices
     For options of type `"choice"', the list of strings the user may
     choose from.

 -- Attribute: Option.callback
     For options with action `"callback"', the callable to call when
     this option is seen.  See section Option Callbacks.
     for detail on the arguments passed to the callable.

 -- Attribute: Option.callback_args
 -- Attribute: Option.callback_kwargs
     Additional positional and keyword arguments to pass to `callback'
     after the four standard callback arguments.

 -- Attribute: Option.help
     Help text to print for this option when listing all available
     options after the user supplies a help option (such
     as `"--help"').  If no help text is supplied, the option will be
     listed without help text.  To hide this option, use the special
     value `optparse.SUPPRESS_HELP'.

 -- Attribute: Option.metavar
     (default: derived from option strings)

     Stand-in for the option argument(s) to use when printing help
     text.  See section Tutorial for an example.

5.15.5.20 Standard option actions
.................................

The various option actions all have slightly different requirements and
effects.  Most actions have several relevant option attributes which
you may specify to guide optparse.'s behaviour; a few have
required attributes, which you must specify for any option using that
action.

   * `"store"' [relevant: type, dest,
     nargs, choices]

     The option must be followed by an argument, which is converted to
     a value according to type and stored in dest.
     If nargs. > 1, multiple arguments will be
     consumed from the command line; all will be converted according to
     type and stored to dest as a tuple.  See
     the Standard option types section.

     If choices is supplied (a list or tuple of strings),
     the type defaults to `"choice"'.

     If type is not supplied, it defaults to `"string"'.

     If dest is not supplied, see optparse derives
     a destination from the first long option string (e.g.,
     `"--foo-bar"' implies `foo_bar'). If there are no long option
     strings, see optparse derives a destination from the first
     short option string (e.g., `"-f"' implies `f').

     Example:

         parser.add_option("-f")
         parser.add_option("-p", type="float", nargs=3, dest="point")

     As it parses the command line

         -f foo.txt -p 1 -3.5 4 -fbar.txt

     optparse will set

         options.f = "foo.txt"
         options.point = (1.0, -3.5, 4.0)
         options.f = "bar.txt"

   * `"store_const"' [required: const.; relevant:
     dest]

     The value const is stored in dest.

     Example:

         parser.add_option("-q", "--quiet",
                           action="store_const", const=0, dest="verbose")
         parser.add_option("-v", "--verbose",
                           action="store_const", const=1, dest="verbose")
         parser.add_option("--noisy",
                           action="store_const", const=2, dest="verbose")

     If `"--noisy"' is seen, see optparse will set

         options.verbose = 2

   * `"store_true"' [relevant: dest]

     A special case of `"store_const"' that stores a true value to
     dest.

   * `"store_false"' [relevant: dest]

     Like `"store_true"', but stores a false value.

     Example:

         parser.add_option("--clobber", action="store_true", dest="clobber")
         parser.add_option("--no-clobber", action="store_false", dest="clobber")

   * `"append"' [relevant: type, dest,
     nargs, choices]

     The option must be followed by an argument, which is appended to
     the list in dest.  If no default value for dest
     is supplied, an empty list is automatically created when
     optparse first encounters this option on the
     command-line.  If nargs. > 1, multiple arguments are
     consumed, and a tuple of length nargs.  is appended to
     dest.

     The defaults for type and dest are the
     same as for the `"store"' action.

     Example:

         parser.add_option("-t", "--tracks", action="append", type="int")

     If `"-t3"' is seen on the command-line, see optparse does
     the equivalent of:

         options.tracks = []
         options.tracks.append(int("3"))

     If, a little later on, `"--tracks=4"' is seen, it does:

         options.tracks.append(int("4"))

   * `"append_const"' [required: const.; relevant:
     dest]

     Like `"store_const"', but the value const is appended
     to dest.; as with `"append"', see dest.
     defaults to `None', and an empty list is automatically created the
     first time the option is encountered.

   * `"count"' [relevant: dest]

     Increment the integer stored at dest.  If no default
     value is supplied, see dest is set to zero before being
     incremented the first time.

     Example:

         parser.add_option("-v", action="count", dest="verbosity")

     The first time `"-v"' is seen on the command line, see optparse
     does the equivalent of:

         options.verbosity = 0
         options.verbosity += 1

     Every subsequent occurrence of `"-v"' results in

         options.verbosity += 1

   * `"callback"' [required: callback.; relevant:
     type, nargs, callback_args,
     callback_kwargs]

     Call the function specified by callback, which is
     called as

         func(option, opt_str, value, parser, *args, **kwargs)

     See section Option Callbacks for more detail.

   * `"help"'

     Prints a complete help message for all the options in the current
     option parser.  The help message is constructed from the `usage'
     string passed to OptionParser's constructor and the help
     string passed to every option.

     If no help string is supplied for an option, it will
     still be listed in the help message.  To omit an option entirely,
     use the special value `optparse.SUPPRESS_HELP'.

     optparse automatically adds a help option
     to all OptionParsers, so you do not normally need to create one.

     Example:

         from optparse import OptionParser, SUPPRESS_HELP

         # usually, a help option is added automatically, but that can
         # be suppressed using the add_help_option argument
         parser = OptionParser(add_help_option=False)

         parser.add_option("-h", "--help", action="help")
         parser.add_option("-v", action="store_true", dest="verbose",
                           help="Be moderately verbose")
         parser.add_option("--file", dest="filename",
                           help="Input file to read data from")
         parser.add_option("--secret", help=SUPPRESS_HELP)

     If optparse sees either `"-h"' or `"--help"' on the
     command line, it will print something like the following help
     message to stdout (assuming `sys.argv[0]' is `"foo.py"'):

         usage: foo.py [options]

         options:
           -h, --help        Show this help message and exit
           -v                Be moderately verbose
           --file=FILENAME   Input file to read data from

     After printing the help message, see optparse terminates
     your process with `sys.exit(0)'.

   * `"version"'

     Prints the version number supplied to the OptionParser to stdout
     and exits.  The version number is actually formatted and printed
     by the `print_version()' method of OptionParser.  Generally only
     relevant if the `version' argument is supplied to the OptionParser
     constructor.  As with help options, you will rarely
     create `version' options, since optparse automatically
     adds them when needed.

5.15.5.21 Standard option types
...............................

optparse has six built-in option types: `"string"', `"int"',
`"long"', `"choice"', `"float"' and `"complex"'.  If you need to add new
option types, see section Extending optparse.

  Arguments to string options are not checked or converted in any way:
the text on the command line is stored in the destination (or passed to
the callback) as-is.

  Integer arguments (type `"int"' or `"long"') are parsed as follows:

   * if the number starts with `0x', it is parsed as a hexadecimal
     number

   * if the number starts with `0', it is parsed as an octal number

   * if the number starts with `0b', it is parsed as a binary number

   * otherwise, the number is parsed as a decimal number

  The conversion is done by calling either int() or
long() with the appropriate base (2, 8, 10, or 16).  If this
fails, so will optparse, although with a more useful error
message.

  `"float"' and `"complex"' option arguments are converted directly with
float() and complex(), with similar
error-handling.

  `"choice"' options are a subtype of `"string"' options.  The
choices option attribute (a sequence of strings) defines the set
of allowed option arguments.  `optparse.check_choice()' compares
user-supplied option arguments against this master list and raises
`OptionValueError' if an invalid string is given.

5.15.5.22 Parsing arguments
...........................

The whole point of creating and populating an OptionParser is to call
its `parse_args()' method:

    (options, args) = parser.parse_args(args=None, values=None)

where the input parameters are

`args'
     the list of arguments to process (default: `sys.argv[1:]')

`values'
     a `optparse.Values' object to store option arguments in (default: a
     new instance of `Values') - if you give an existing object, the
     option defaults will not be initialized on it

  and the return values are

`options'
     the same object that was passed in as `values', or the
     optparse.Values instance created by optparse.

`args'
     the leftover positional arguments after all options have been
     processed

  The most common usage is to supply neither keyword argument.  If you
supply `values', it will be modified with repeated setattr()
calls (roughly one for every option argument stored to an option
destination) and returned by `parse_args()'.

  If `parse_args()' encounters any errors in the argument list, it
calls the OptionParser's `error()' method with an appropriate end-user
error message.  This ultimately terminates your process with an exit
status of 2 (the traditional Unix exit status for command-line errors).

5.15.5.23 Querying and manipulating your option parser
......................................................

The default behavior of the option parser can be customized slightly,
and you can also poke around your option parser and see what's there.
OptionParser provides several methods to help you out:

 -- Method: OptionParser.disable_interspersed_args ()
     Set parsing to stop on the first non-option.  For example, if
     `"-a"' and `"-b"' are both simple options that take no arguments,
     see optparse.  normally accepts this syntax:

         prog -a arg1 -b arg2

     and treats it as equivalent to

         prog -a -b arg1 arg2

     To disable this feature, call disable_interspersed_args().
     This restores traditional Unix syntax, where option parsing
     stops with the first non-option argument.

     Use this if you have a command processor which runs another
     command which has options of its own and you want to make sure
     these options don't get confused.  For example, each command might
     have a different set of options.

 -- Method: OptionParser.enable_interspersed_args ()
     Set parsing to not stop on the first non-option, allowing
     interspersing switches with command arguments.  This is the
     default behavior.

 -- Method: OptionParser.get_option (opt_str)
     Returns the Option instance with the option string _opt_str_, or
     `None' if no options have that option string.

 -- Method: OptionParser.has_option (opt_str)
     Return true if the OptionParser has an option with option string
     _opt_str_ (e.g., `"-q"' or `"--verbose"').

 -- Method: OptionParser.remove_option (opt_str)
     If the OptionParser has an option corresponding to
     _opt_str_, that option is removed.  If that option provided any
     other option strings, all of those option strings become invalid.
     If _opt_str_ does not occur in any option belonging to this
     OptionParser, raises ValueError.

5.15.5.24 Conflicts between options
...................................

If you're not careful, it's easy to define options with conflicting
option strings:

    parser.add_option("-n", "--dry-run", ...)
    [...]
    parser.add_option("-n", "--noisy", ...)

(This is particularly true if you've defined your own OptionParser
subclass with some standard options.)

  Every time you add an option, see optparse checks for
conflicts with existing options.  If it finds any, it invokes the
current conflict-handling mechanism.  You can set the conflict-handling
mechanism either in the constructor:

    parser = OptionParser(..., conflict_handler=handler)

or with a separate call:

    parser.set_conflict_handler(handler)

The available conflict handlers are:

    `"error"' (default)
          assume option conflicts are a programming error and raise
          `OptionConflictError'

    `"resolve"'
          resolve option conflicts intelligently (see below)

  As an example, let's define an OptionParser that
resolves conflicts intelligently and add conflicting options to it:

    parser = OptionParser(conflict_handler="resolve")
    parser.add_option("-n", "--dry-run", ..., help="do no harm")
    parser.add_option("-n", "--noisy", ..., help="be noisy")

At this point, see optparse detects that a previously-added
option is already using the `"-n"' option string.  Since
`conflict_handler' is `"resolve"', it resolves the situation by
removing `"-n"' from the earlier option's list of option strings.  Now
`"--dry-run"' is the only way for the user to activate that option.  If
the user asks for help, the help message will reflect that:

    options:
      --dry-run     do no harm
      [...]
      -n, --noisy   be noisy

It's possible to whittle away the option strings for a previously-added
option until there are none left, and the user has no way of invoking
that option from the command-line.  In that case, see optparse.
removes that option completely, so it doesn't show up in help text or
anywhere else. Carrying on with our existing OptionParser:

    parser.add_option("--dry-run", ..., help="new dry-run option")

At this point, the original `-n/--dry-run' option is no longer
accessible, so optparse removes it, leaving this help text:

    options:
      [...]
      -n, --noisy   be noisy
      --dry-run     new dry-run option

5.15.5.25 Cleanup
.................

OptionParser instances have several cyclic references.  This should not
be a problem for Python's garbage collector, but you may wish to break
the cyclic references explicitly by calling `destroy()' on your
OptionParser once you are done with it.  This is particularly useful in
long-running applications where large object graphs are reachable from
your OptionParser.

5.15.5.26 Other methods
.......................

OptionParser supports several other public methods:

 -- Method: OptionParser.set_usage (usage)
     Set the usage string according to the rules described above for
     the `usage' constructor keyword argument.  Passing `None' sets the
     default usage string; use `optparse.SUPPRESS_USAGE' to suppress a
     usage message.

 -- Method: OptionParser.print_usage (file=None)
     Print the usage message for the current program (`self.usage') to
     _file_ (default stdout).  Any occurrence of the string `"%prog"'
     in `self.usage' is replaced with the name of the current program.
     Does nothing if `self.usage' is empty or not defined.

 -- Method: OptionParser.get_usage ()
     Same as print_usage() but returns the usage string
     instead of printing it.

 -- Method: OptionParser.set_defaults (dest=value, ...)
     Set default values for several option destinations at once.  Using
     set_defaults() is the preferred way to set default
     values for options, since multiple options can share the same
     destination.  For example, if several "mode" options all set the
     same destination, any one of them can set the default, and the
     last one wins:

         parser.add_option("--advanced", action="store_const",
                           dest="mode", const="advanced",
                           default="novice")    # overridden below
         parser.add_option("--novice", action="store_const",
                           dest="mode", const="novice",
                           default="advanced")  # overrides above setting

     To avoid this confusion, use set_defaults().:

         parser.set_defaults(mode="advanced")
         parser.add_option("--advanced", action="store_const",
                           dest="mode", const="advanced")
         parser.add_option("--novice", action="store_const",
                           dest="mode", const="novice")

5.15.5.27 Option Callbacks
..........................

When optparse.'s built-in actions and types aren't quite
enough for your needs, you have two choices: extend optparse
or define a callback option.  Extending optparse is
more general, but overkill for a lot of simple cases.  Quite often a
simple callback is all you need.

  There are two steps to defining a callback option:

   * define the option itself using the `"callback"' action

   * write the callback; this is a function (or method) that takes at
     least four arguments, as described below

5.15.5.28 Defining a callback option
....................................

As always, the easiest way to define a callback option is by using the
OptionParser.add_option() method.  Apart from
action, the only option attribute you must specify is `callback',
the function to call:

    parser.add_option("-c", action="callback", callback=my_callback)

`callback' is a function (or other callable object), so you must have
already defined `my_callback()' when you create this callback option.
In this simple case, see optparse doesn't even know if
-c takes any arguments, which usually means that the option takes
no arguments--the mere presence of -c on the command-line
is all it needs to know.  In some circumstances, though, you might want
your callback to consume an arbitrary number of command-line arguments.
This is where writing callbacks gets tricky; it's covered later in this
section.

  optparse always passes four particular arguments to your
callback, and it will only pass additional arguments if you specify
them via callback_args and callback_kwargs.
Thus, the minimal callback function signature is:

    def my_callback(option, opt, value, parser):

The four arguments to a callback are described below.

  There are several other option attributes that you can supply when
you define a callback option:

type.
     has its usual meaning: as with the `"store"' or `"append"'
     actions, it instructs optparse to consume one argument
     and convert it to type.  Rather than storing the
     converted value(s) anywhere, though, see optparse passes
     it to your callback function.

See nargs.
     also has its usual meaning: if it is supplied and > 1, see
     optparse will consume nargs arguments, each of
     which must be convertible to type.  It then passes a
     tuple of converted values to your callback.

See callback_args.
     a tuple of extra positional arguments to pass to the callback

See callback_kwargs.
     a dictionary of extra keyword arguments to pass to the callback

5.15.5.29 How callbacks are called
..................................

All callbacks are called as follows:

    func(option, opt_str, value, parser, *args, **kwargs)

where

`option'
     is the Option instance that's calling the callback

`opt_str'
     is the option string seen on the command-line that's triggering
     the callback.  (If an abbreviated long option was used, `opt_str'
     will be the full, canonical option string--e.g. if the user puts
     `"--foo"' on the command-line as an abbreviation for `"--foobar"',
     then `opt_str' will be `"--foobar"'.)

`value'
     is the argument to this option seen on the command-line.
     optparse will only expect an argument if type.
     is set; the type of `value' will be the type implied by the
     option's type.  If type for this option is `None' (no
     argument expected), then `value' will be `None'.  If nargs.
     > 1, `value' will be a tuple of values of the appropriate
     type.

`parser'
     is the OptionParser instance driving the whole thing, mainly
     useful because you can access some other interesting data through
     its instance attributes:

    `parser.largs'
          the current list of leftover arguments, ie. arguments that
          have been consumed but are neither options nor option
          arguments. Feel free to modify `parser.largs', e.g. by adding
          more arguments to it.  (This list will become `args', the
          second return value of `parse_args()'.)

    `parser.rargs'
          the current list of remaining arguments, ie. with `opt_str'
          and `value' (if applicable) removed, and only the arguments
          following them still there.  Feel free to modify
          `parser.rargs', e.g. by consuming more arguments.

    `parser.values'
          the object where option values are by default stored (an
          instance of optparse.OptionValues).  This lets callbacks use
          the same mechanism as the rest of optparse for
          storing option values; you don't need to mess around with
          globals or closures.  You can also access or modify the
          value(s) of any options already encountered on the
          command-line.

`args'
     is a tuple of arbitrary positional arguments supplied via the
     callback_args option attribute.

`kwargs'
     is a dictionary of arbitrary keyword arguments supplied via
     callback_kwargs.

5.15.5.30 Raising errors in a callback
......................................

The callback function should raise `OptionValueError' if there are any
problems with the option or its argument(s).  See optparse.
catches this and terminates the program, printing the error message you
supply to stderr.  Your message should be clear, concise, accurate, and
mention the option at fault.  Otherwise, the user will have a hard time
figuring out what he did wrong.

5.15.5.31 Callback example 1: trivial callback
..............................................

Here's an example of a callback option that takes no arguments, and
simply records that the option was seen:

    def record_foo_seen(option, opt_str, value, parser):
        parser.values.saw_foo = True

    parser.add_option("--foo", action="callback", callback=record_foo_seen)

Of course, you could do that with the `"store_true"' action.

5.15.5.32 Callback example 2: check option order
................................................

Here's a slightly more interesting example: record the fact that `"-a"'
is seen, but blow up if it comes after `"-b"' in the command-line.

    def check_order(option, opt_str, value, parser):
        if parser.values.b:
            raise OptionValueError("can't use -a after -b")
        parser.values.a = 1
    [...]
    parser.add_option("-a", action="callback", callback=check_order)
    parser.add_option("-b", action="store_true", dest="b")

5.15.5.33 Callback example 3: check option order (generalized)
..............................................................

If you want to re-use this callback for several similar options (set a
flag, but blow up if `"-b"' has already been seen), it needs a bit of
work: the error message and the flag that it sets must be generalized.

    def check_order(option, opt_str, value, parser):
        if parser.values.b:
            raise OptionValueError("can't use %s after -b" % opt_str)
        setattr(parser.values, option.dest, 1)
    [...]
    parser.add_option("-a", action="callback", callback=check_order, dest='a')
    parser.add_option("-b", action="store_true", dest="b")
    parser.add_option("-c", action="callback", callback=check_order, dest='c')

5.15.5.34 Callback example 4: check arbitrary condition
.......................................................

Of course, you could put any condition in there--you're not limited to
checking the values of already-defined options.  For example, if you
have options that should not be called when the moon is full, all you
have to do is this:

    def check_moon(option, opt_str, value, parser):
        if is_moon_full():
            raise OptionValueError("%s option invalid when moon is full"
                                   % opt_str)
        setattr(parser.values, option.dest, 1)
    [...]
    parser.add_option("--foo",
                      action="callback", callback=check_moon, dest="foo")

(The definition of `is_moon_full()' is left as an exercise for the
reader.)

5.15.5.35 Callback example 5: fixed arguments
.............................................

Things get slightly more interesting when you define callback options
that take a fixed number of arguments.  Specifying that a callback
option takes arguments is similar to defining a `"store"' or `"append"'
option: if you define type, then the option takes one
argument that must be convertible to that type; if you further define
nargs, then the option takes nargs arguments.

  Here's an example that just emulates the standard `"store"' action:

    def store_value(option, opt_str, value, parser):
        setattr(parser.values, option.dest, value)
    [...]
    parser.add_option("--foo",
                      action="callback", callback=store_value,
                      type="int", nargs=3, dest="foo")

Note that optparse takes care of consuming 3 arguments and
converting them to integers for you; all you have to do is store them.
(Or whatever; obviously you don't need a callback for this example.)

5.15.5.36 Callback example 6: variable arguments
................................................

Things get hairy when you want an option to take a variable number of
arguments.  For this case, you must write a callback, as
optparse doesn't provide any built-in capabilities for it.  And
you have to deal with certain intricacies of conventional Unix
command-line parsing that optparse normally handles for
you.  In particular, callbacks should implement the conventional rules
for bare `"--"' and `"-"' arguments:

   * either `"--"' or `"-"' can be option arguments

   * bare `"--"' (if not the argument to some option): halt command-line
     processing and discard the `"--"'

   * bare `"-"' (if not the argument to some option): halt command-line
     processing but keep the `"-"' (append it to `parser.largs')

  If you want an option that takes a variable number of arguments,
there are several subtle, tricky issues to worry about.  The exact
implementation you choose will be based on which trade-offs you're
willing to make for your application (which is why optparse.
doesn't support this sort of thing directly).

  Nevertheless, here's a stab at a callback for an option with variable
arguments:

     def vararg_callback(option, opt_str, value, parser):
         assert value is None
         value = []

         def floatable(str):
             try:
                 float(str)
                 return True
             except ValueError:
                 return False

         for arg in parser.rargs:
             # stop on --foo like options
             if arg[:2] == "--" and len(arg) > 2:
                 break
             # stop on -a, but not on -3 or -3.0
             if arg[:1] == "-" and len(arg) > 1 and not floatable(arg):
                 break
             value.append(arg)

         del parser.rargs[:len(value)]
         setattr(parser.values, option.dest, value)

    [...]
    parser.add_option("-c", "--callback", dest="vararg_attr",
                      action="callback", callback=vararg_callback)

5.15.5.37 Extending `optparse'
..............................

Since the two major controlling factors in how optparse.
interprets command-line options are the action and type of each option,
the most likely direction of extension is to add new actions and new
types.

5.15.5.38 Adding new types
..........................

To add new types, you need to define your own subclass of
optparse.'s `Option' class.  This class has a couple of attributes
that define optparse.'s types: TYPES and
TYPE_CHECKER.

 -- Attribute: Option.TYPES
     A tuple of type names; in your subclass, simply define a new tuple
     TYPES that builds on the standard one.

 -- Attribute: Option.TYPE_CHECKER
     A dictionary mapping type names to type-checking functions.  A
     type-checking function has the following signature:

         def check_mytype(option, opt, value)

     where `option' is an `Option' instance, `opt' is an option string
     (e.g., `"-f"'), and `value' is the string from the command line
     that must be checked and converted to your desired type.
     `check_mytype()' should return an object of the hypothetical type
     `mytype'.  The value returned by a type-checking function will
     wind up in the OptionValues instance returned by
     `OptionParser.parse_args()', or be passed to a callback as the
     `value' parameter.

     Your type-checking function should raise `OptionValueError' if it
     encounters any problems.  `OptionValueError' takes a single string
     argument, which is passed as-is to OptionParser.'s
     `error()' method, which in turn prepends the program name and the
     string `"error:"' and prints everything to stderr before
     terminating the process.

  Here's a silly example that demonstrates adding a `"complex"' option
type to parse Python-style complex numbers on the command line.  (This
is even sillier than it used to be, because optparse. 1.3
added built-in support for complex numbers, but never mind.)

  First, the necessary imports:

    from copy import copy
    from optparse import Option, OptionValueError

You need to define your type-checker first, since it's referred to
later (in the TYPE_CHECKER class attribute of your Option
subclass):

    def check_complex(option, opt, value):
        try:
            return complex(value)
        except ValueError:
            raise OptionValueError(
                "option %s: invalid complex value: %r" % (opt, value))

Finally, the Option subclass:

    class MyOption (Option):
        TYPES = Option.TYPES + ("complex",)
        TYPE_CHECKER = copy(Option.TYPE_CHECKER)
        TYPE_CHECKER["complex"] = check_complex

(If we didn't make a copy() of Option.TYPE_CHECKER,
we would end up modifying the TYPE_CHECKER attribute
of optparse.'s Option class.  This being Python, nothing
stops you from doing that except good manners and common sense.)

  That's it!  Now you can write a script that uses the new option type
just like any other optparse.-based script, except you have
to instruct your OptionParser to use MyOption instead of Option:

    parser = OptionParser(option_class=MyOption)
    parser.add_option("-c", type="complex")

Alternately, you can build your own option list and pass it to
OptionParser; if you don't use `add_option()' in the above way, you
don't need to tell OptionParser which option class to use:

    option_list = [MyOption("-c", action="store", type="complex", dest="c")]
    parser = OptionParser(option_list=option_list)

5.15.5.39 Adding new actions
............................

Adding new actions is a bit trickier, because you have to understand
that optparse has a couple of classifications for actions:

"store" actions
     actions that result in optparse storing a value to an
     attribute of the current OptionValues instance; these options
     require a dest.  attribute to be supplied to the
     Option constructor.

"typed" actions
     actions that take a value from the command line and expect it to
     be of a certain type; or rather, a string that can be converted to
     a certain type.  These options require a type.
     attribute to the Option constructor.

  These are overlapping sets: some default "store" actions are
`"store"', `"store_const"', `"append"', and `"count"', while the
default "typed" actions are `"store"', `"append"', and `"callback"'.

  When you add an action, you need to categorize it by listing it in at
least one of the following class attributes of Option (all are lists of
strings):

 -- Attribute: Option.ACTIONS
     All actions must be listed in ACTIONS.

 -- Attribute: Option.STORE_ACTIONS
     "store" actions are additionally listed here.

 -- Attribute: Option.TYPED_ACTIONS
     "typed" actions are additionally listed here.

 -- Attribute: Option.ALWAYS_TYPED_ACTIONS
     Actions that always take a type (i.e. whose options always take a
     value) are additionally listed here.  The only effect of this is
     that optparse.  assigns the default type, `"string"',
     to options with no explicit type whose action is listed in
     ALWAYS_TYPED_ACTIONS.

  In order to actually implement your new action, you must override
Option's `take_action()' method and add a case that recognizes your
action.

  For example, let's add an `"extend"' action.  This is similar to the
standard `"append"' action, but instead of taking a single value from
the command-line and appending it to an existing list, `"extend"' will
take multiple values in a single comma-delimited string, and extend an
existing list with them.  That is, if `"--names"' is an `"extend"'
option of type `"string"', the command line

    --names=foo,bar --names blah --names ding,dong

would result in a list

    ["foo", "bar", "blah", "ding", "dong"]

Again we define a subclass of Option:

    class MyOption(Option):

        ACTIONS = Option.ACTIONS + ("extend",)
        STORE_ACTIONS = Option.STORE_ACTIONS + ("extend",)
        TYPED_ACTIONS = Option.TYPED_ACTIONS + ("extend",)
        ALWAYS_TYPED_ACTIONS = Option.ALWAYS_TYPED_ACTIONS + ("extend",)

        def take_action(self, action, dest, opt, value, values, parser):
            if action == "extend":
                lvalue = value.split(",")
                values.ensure_value(dest, []).extend(lvalue)
            else:
                Option.take_action(
                    self, action, dest, opt, value, values, parser)

Features of note:

   * `"extend"' both expects a value on the command-line and stores
     that value somewhere, so it goes in both STORE_ACTIONS
     and TYPED_ACTIONS.

   * to ensure that optparse assigns the default type of
     `"string"' to `"extend"' actions, we put the `"extend"' action in
     ALWAYS_TYPED_ACTIONS as well.

   * `MyOption.take_action()' implements just this one new action, and
     passes control back to `Option.take_action()' for the standard
     optparse.  actions.

   * `values' is an instance of the optparse_parser.Values class, which
     provides the very useful `ensure_value()' method. `ensure_value()'
     is essentially getattr() with a safety valve; it is
     called as

         values.ensure_value(attr, value)

     If the `attr' attribute of `values' doesn't exist or is None, then
     ensure_value() first sets it to `value', and then returns 'value.
     This is very handy for actions like `"extend"', `"append"', and
     `"count"', all of which accumulate data in a variable and expect
     that variable to be of a certain type (a list for the first two,
     an integer for the latter).  Using `ensure_value()' means that
     scripts using your action don't have to worry about setting a
     default value for the option destinations in question; they can
     just leave the default as None and `ensure_value()' will take care
     of getting it right when it's needed.

5.15.6 `getopt' -- C-style parser for command line options
----------------------------------------------------------

     Note: The getopt module is a parser for command line
     options whose API is designed to be familiar to users of the C
     `getopt()' function. Users who are unfamiliar with the C
     `getopt()' function or who would like to write less code and get
     better help and error messages should consider using the
     argparse module instead.

This module helps scripts to parse the command line arguments in
`sys.argv'.  It supports the same conventions as the Unix `getopt()'
function (including the special meanings of arguments of the form '`-''
and '`--'').  Long options similar to those supported by GNU software
may be used as well via an optional third argument.

  A more convenient, flexible, and powerful alternative is the
optparse module.

  This module provides two functions and an exception:

 -- Function: getopt.getopt (args, options[, long_options])
     Parses command line options and parameter list.  _args_ is the
     argument list to be parsed, without the leading reference to the
     running program. Typically, this means `sys.argv[1:]'. _options_
     is the string of option letters that the script wants to
     recognize, with options that require an argument followed by a
     colon (`':''; i.e., the same format that Unix `getopt()' uses).

          Note: Unlike GNU `getopt()', after a non-option argument, all
          further arguments are considered also non-options. This is
          similar to the way non-GNU Unix systems work.

     _long_options_, if specified, must be a list of strings with the
     names of the long options which should be supported.  The leading
     `'-'`-'' characters should not be included in the option name.
     Long options which require an argument should be followed by an
     equal sign (`'='').  Optional arguments are not supported.  To
     accept only long options, _options_ should be an empty string.
     Long options on the command line can be recognized so long as they
     provide a prefix of the option name that matches exactly one of
     the accepted options.  For example, if _long_options_ is `['foo',
     'frob']', the option `--fo' will match as `--foo', but `--f' will
     not match uniquely, so GetoptError will be raised.

     The return value consists of two elements: the first is a list of
     `(option, value)' pairs; the second is the list of program
     arguments left after the option list was stripped (this is a
     trailing slice of _args_).  Each option-and-value pair returned
     has the option as its first element, prefixed with a hyphen for
     short options (e.g., `'-x'') or two hyphens for long options
     (e.g., `'-'`-long-option''), and the option argument as its second
     element, or an empty string if the option has no argument.  The
     options occur in the list in the same order in which they were
     found, thus allowing multiple occurrences.  Long and short options
     may be mixed.

 -- Function: getopt.gnu_getopt (args, options[, long_options])
     This function works like getopt(), except that GNU style
     scanning mode is used by default. This means that option and
     non-option arguments may be intermixed. The getopt().
     function stops processing options as soon as a non-option argument
     is encountered.

     If the first character of the option string is '+', or if the
     environment variable `POSIXLY_CORRECT' is set, then option
     processing stops as soon as a non-option argument is encountered.

     New in version 2.3.

 -- Exception: getopt.GetoptError
     This is raised when an unrecognized option is found in the
     argument list or when an option requiring an argument is given
     none. The argument to the exception is a string indicating the
     cause of the error.  For long options, an argument given to an
     option which does not require one will also cause this exception
     to be raised.  The attributes `msg' and `opt' give the error
     message and related option; if there is no specific option to
     which the exception relates, `opt' is an empty string.

     Changed in version 1.6: Introduced GetoptError as a
     synonym for error.

 -- Exception: getopt.error
     Alias for GetoptError.; for backward compatibility.

  An example using only Unix style options:

    >>> import getopt
    >>> args = '-a -b -cfoo -d bar a1 a2'.split()
    >>> args
    ['-a', '-b', '-cfoo', '-d', 'bar', 'a1', 'a2']
    >>> optlist, args = getopt.getopt(args, 'abc:d:')
    >>> optlist
    [('-a', ''), ('-b', ''), ('-c', 'foo'), ('-d', 'bar')]
    >>> args
    ['a1', 'a2']

Using long option names is equally easy:

    >>> s = '--condition=foo --testing --output-file abc.def -x a1 a2'
    >>> args = s.split()
    >>> args
    ['--condition=foo', '--testing', '--output-file', 'abc.def', '-x', 'a1', 'a2']
    >>> optlist, args = getopt.getopt(args, 'x', [
    ...     'condition=', 'output-file=', 'testing'])
    >>> optlist
    [('--condition', 'foo'), ('--testing', ''), ('--output-file', 'abc.def'), ('-x', '')]
    >>> args
    ['a1', 'a2']

In a script, typical usage is something like this:

    import getopt, sys

    def main():
        try:
            opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
        except getopt.GetoptError, err:
            # print help information and exit:
            print str(err) # will print something like "option -a not recognized"
            usage()
            sys.exit(2)
        output = None
        verbose = False
        for o, a in opts:
            if o == "-v":
                verbose = True
            elif o in ("-h", "--help"):
                usage()
                sys.exit()
            elif o in ("-o", "--output"):
                output = a
            else:
                assert False, "unhandled option"
        # ...

    if __name__ == "__main__":
        main()

Note that an equivalent command line interface could be produced with
less code and more informative help and error messages by using the
argparse module:

    import argparse

    if __name__ == '__main__':
        parser = argparse.ArgumentParser()
        parser.add_argument('-o', '--output')
        parser.add_argument('-v', dest='verbose', action='store_true')
        args = parser.parse_args()
        # ... do something with args.output ...
        # ... do something with args.verbose ..

Module argparse.
     Alternative command line option and argument parsing library.

5.15.7 `logging' -- Logging facility for Python
-----------------------------------------------

New in version 2.3.

  This module defines functions and classes which implement a flexible
error logging system for applications.

  Logging is performed by calling methods on instances of the `Logger'
class (hereafter called _loggers_). Each instance has a name, and they
are conceptually arranged in a namespace hierarchy using dots (periods)
as separators. For example, a logger named "scan" is the parent of
loggers "scan.text", "scan.html" and "scan.pdf". Logger names can be
anything you want, and indicate the area of an application in which a
logged message originates.

  Logged messages also have levels of importance associated with them.
The default levels provided are `DEBUG', `INFO', `WARNING', `ERROR' and
`CRITICAL'. As a convenience, you indicate the importance of a logged
message by calling an appropriate method of `Logger'. The methods are
debug(), info(), warning(),
error() and critical(), which mirror the default
levels. You are not constrained to use these levels: you can specify
your own and use a more general `Logger' method, see log(),
which takes an explicit level argument.

5.15.7.1 Logging tutorial
.........................

The key benefit of having the logging API provided by a standard
library module is that all Python modules can participate in logging,
so your application log can include messages from third-party modules.

  It is, of course, possible to log messages with different verbosity
levels or to different destinations.  Support for writing log messages
to files, HTTP GET/POST locations, email via SMTP, generic sockets, or
OS-specific logging mechanisms are all supported by the standard
module.  You can also create your own log destination class if you have
special requirements not met by any of the built-in classes.

5.15.7.2 Simple examples
........................

Most applications are probably going to want to log to a file, so let's
start with that case. Using the basicConfig() function, we
can set up the default handler so that debug messages are written to a
file (in the example, we assume that you have the appropriate
permissions to create a file called _example.log_ in the current
directory):

    import logging
    LOG_FILENAME = 'example.log'
    logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)

    logging.debug('This message should go to the log file')

And now if we open the file and look at what we have, we should find
the log message:

    DEBUG:root:This message should go to the log file

If you run the script repeatedly, the additional log messages are
appended to the file.  To create a new file each time, you can pass a
_filemode_ argument to basicConfig() with a value of
`'w''.  Rather than managing the file size yourself, though, it is
simpler to use a `RotatingFileHandler':

    import glob
    import logging
    import logging.handlers

    LOG_FILENAME = 'logging_rotatingfile_example.out'

    # Set up a specific logger with our desired output level
    my_logger = logging.getLogger('MyLogger')
    my_logger.setLevel(logging.DEBUG)

    # Add the log message handler to the logger
    handler = logging.handlers.RotatingFileHandler(
                  LOG_FILENAME, maxBytes=20, backupCount=5)

    my_logger.addHandler(handler)

    # Log some messages
    for i in range(20):
        my_logger.debug('i = %d' % i)

    # See what files are created
    logfiles = glob.glob('%s*' % LOG_FILENAME)

    for filename in logfiles:
        print filename

The result should be 6 separate files, each with part of the log
history for the application:

    logging_rotatingfile_example.out
    logging_rotatingfile_example.out.1
    logging_rotatingfile_example.out.2
    logging_rotatingfile_example.out.3
    logging_rotatingfile_example.out.4
    logging_rotatingfile_example.out.5

The most current file is always `logging_rotatingfile_example.out', and
each time it reaches the size limit it is renamed with the suffix `.1'.
Each of the existing backup files is renamed to increment the suffix
(`.1' becomes `.2', etc.)  and the `.6' file is erased.

  Obviously this example sets the log length much much too small as an
extreme example.  You would want to set _maxBytes_ to an appropriate
value.

  Another useful feature of the logging API is the ability to produce
different messages at different log levels.  This allows you to
instrument your code with debug messages, for example, but turning the
log level down so that those debug messages are not written for your
production system.  The default levels are `NOTSET', `DEBUG', `INFO',
`WARNING', `ERROR' and `CRITICAL'.

  The logger, handler, and log message call each specify a level.  The
log message is only emitted if the handler and logger are configured to
emit messages of that level or lower.  For example, if a message is
`CRITICAL', and the logger is set to `ERROR', the message is emitted.
If a message is a `WARNING', and the logger is set to produce only
`ERROR's, the message is not emitted:

    import logging
    import sys

    LEVELS = {'debug': logging.DEBUG,
              'info': logging.INFO,
              'warning': logging.WARNING,
              'error': logging.ERROR,
              'critical': logging.CRITICAL}

    if len(sys.argv) > 1:
        level_name = sys.argv[1]
        level = LEVELS.get(level_name, logging.NOTSET)
        logging.basicConfig(level=level)

    logging.debug('This is a debug message')
    logging.info('This is an info message')
    logging.warning('This is a warning message')
    logging.error('This is an error message')
    logging.critical('This is a critical error message')

Run the script with an argument like 'debug' or 'warning' to see which
messages show up at different levels:

    $ python logging_level_example.py debug
    DEBUG:root:This is a debug message
    INFO:root:This is an info message
    WARNING:root:This is a warning message
    ERROR:root:This is an error message
    CRITICAL:root:This is a critical error message

    $ python logging_level_example.py info
    INFO:root:This is an info message
    WARNING:root:This is a warning message
    ERROR:root:This is an error message
    CRITICAL:root:This is a critical error message

You will notice that these log messages all have `root' embedded in
them.  The logging module supports a hierarchy of loggers with
different names.  An easy way to tell where a specific log message
comes from is to use a separate logger object for each of your modules.
Each new logger "inherits" the configuration of its parent, and log
messages sent to a logger include the name of that logger.  Optionally,
each logger can be configured differently, so that messages from
different modules are handled in different ways.  Let's look at a simple
example of how to log from different modules so it is easy to trace the
source of the message:

    import logging

    logging.basicConfig(level=logging.WARNING)

    logger1 = logging.getLogger('package1.module1')
    logger2 = logging.getLogger('package2.module2')

    logger1.warning('This message comes from one module')
    logger2.warning('And this message comes from another module')

And the output:

    $ python logging_modules_example.py
    WARNING:package1.module1:This message comes from one module
    WARNING:package2.module2:And this message comes from another module

There are many more options for configuring logging, including
different log message formatting options, having messages delivered to
multiple destinations, and changing the configuration of a long-running
application on the fly using a socket interface.  All of these options
are covered in depth in the library module documentation.

5.15.7.3 Loggers
................

The logging library takes a modular approach and offers the several
categories of components: loggers, handlers, filters, and formatters.
Loggers expose the interface that application code directly uses.
Handlers send the log records to the appropriate destination. Filters
provide a finer grained facility for determining which log records to
send on to a handler.  Formatters specify the layout of the resultant
log record.

  `Logger' objects have a threefold job.  First, they expose several
methods to application code so that applications can log messages at
runtime.  Second, logger objects determine which log messages to act
upon based upon severity (the default filtering facility) or filter
objects.  Third, logger objects pass along relevant log messages to all
interested log handlers.

  The most widely used methods on logger objects fall into two
categories: configuration and message sending.

   * Logger.setLevel() specifies the lowest-severity log
     message a logger will handle, where debug is the lowest built-in
     severity level and critical is the highest built-in severity.  For
     example, if the severity level is info, the logger will handle
     only info, warning, error, and critical messages and will ignore
     debug messages.

   * Logger.addFilter() and Logger.removeFilter()
     add and remove filter objects from the logger object.  This
     tutorial does not address filters.

  With the logger object configured, the following methods create log
messages:

   * Logger.debug(), Logger.info(),
     Logger.warning(), Logger.error(), and
     Logger.critical() all create log records with a message and
     a level that corresponds to their respective method names. The
     message is actually a format string, which may contain the
     standard string substitution syntax of `%s', `%d', `%f', and so
     on.  The rest of their arguments is a list of objects that
     correspond with the substitution fields in the message.  With
     regard to `**kwargs', the logging methods care only about a
     keyword of `exc_info' and use it to determine whether to log
     exception information.

   * Logger.exception() creates a log message similar to
     Logger.error().  The difference is that
     Logger.exception() dumps a stack trace along with it.  Call
     this method only from an exception handler.

   * Logger.log() takes a log level as an explicit
     argument.  This is a little more verbose for logging messages than
     using the log level convenience methods listed above, but this is
     how to log at custom log levels.

  getLogger() returns a reference to a logger instance
with the specified name if it is provided, or `root' if not.  The names
are period-separated hierarchical structures.  Multiple calls to
getLogger() with the same name will return a reference to the
same logger object.  Loggers that are further down in the hierarchical
list are children of loggers higher up in the list.  For example, given
a logger with a name of `foo', loggers with names of `foo.bar',
`foo.bar.baz', and `foo.bam' are all descendants of `foo'.  Child
loggers propagate messages up to the handlers associated with their
ancestor loggers.  Because of this, it is unnecessary to define and
configure handlers for all the loggers an application uses. It is
sufficient to configure handlers for a top-level logger and create
child loggers as needed.

5.15.7.4 Handlers
.................

`Handler' objects are responsible for dispatching the appropriate log
messages (based on the log messages' severity) to the handler's
specified destination.  Logger objects can add zero or more handler
objects to themselves with an `addHandler()' method.  As an example
scenario, an application may want to send all log messages to a log
file, all log messages of error or higher to stdout, and all messages
of critical to an email address.  This scenario requires three
individual handlers where each handler is responsible for sending
messages of a specific severity to a specific location.

  The standard library includes quite a few handler types; this
tutorial uses only StreamHandler and FileHandler
in its examples.

  There are very few methods in a handler for application developers to
concern themselves with.  The only handler methods that seem relevant
for application developers who are using the built-in handler objects
(that is, not creating custom handlers) are the following configuration
methods:

   * The Handler.setLevel() method, just as in logger
     objects, specifies the lowest severity that will be dispatched to
     the appropriate destination.  Why are there two `setLevel()'
     methods?  The level set in the logger determines which severity of
     messages it will pass to its handlers.  The level set in each
     handler determines which messages that handler will send on.

   * `setFormatter()' selects a Formatter object for this handler to
     use.

   * `addFilter()' and `removeFilter()' respectively configure and
     deconfigure filter objects on handlers.

  Application code should not directly instantiate and use instances of
`Handler'.  Instead, the `Handler' class is a base class that defines
the interface that all handlers should have and establishes some
default behavior that child classes can use (or override).

5.15.7.5 Formatters
...................

Formatter objects configure the final order, structure, and contents of
the log message.  Unlike the base `logging.Handler' class, application
code may instantiate formatter classes, although you could likely
subclass the formatter if your application needs special behavior.  The
constructor takes two optional arguments: a message format string and a
date format string.  If there is no message format string, the default
is to use the raw message.  If there is no date format string, the
default date format is:

    %Y-%m-%d %H:%M:%S

with the milliseconds tacked on at the end.

  The message format string uses `%(<dictionary key>)s' styled string
substitution; the possible keys are documented in Formatter Objects.

  The following message format string will log the time in a
human-readable format, the severity of the message, and the contents of
the message, in that order:

    "%(asctime)s - %(levelname)s - %(message)s"

Formatters use a user-configurable function to convert the creation
time of a record to a tuple. By default, see time.localtime().
is used; to change this for a particular formatter instance, set the
`converter' attribute of the instance to a function with the same
signature as time.localtime() or time.gmtime().
To change it for all formatters, for example if you want all logging
times to be shown in GMT, set the `converter' attribute in the
Formatter class (to `time.gmtime' for GMT display).

5.15.7.6 Configuring Logging
............................

Programmers can configure logging in three ways:

  1. Creating loggers, handlers, and formatters explicitly using Python
     code that calls the configuration methods listed above.

  2. Creating a logging config file and reading it using the
     fileConfig().  function.

  3. Creating a dictionary of configuration information and passing it
     to the dictConfig() function.

  The following example configures a very simple logger, a console
handler, and a simple formatter using Python code:

    import logging

    # create logger
    logger = logging.getLogger("simple_example")
    logger.setLevel(logging.DEBUG)

    # create console handler and set level to debug
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)

    # create formatter
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")

    # add formatter to ch
    ch.setFormatter(formatter)

    # add ch to logger
    logger.addHandler(ch)

    # "application" code
    logger.debug("debug message")
    logger.info("info message")
    logger.warn("warn message")
    logger.error("error message")
    logger.critical("critical message")

Running this module from the command line produces the following output:

    $ python simple_logging_module.py
    2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message
    2005-03-19 15:10:26,620 - simple_example - INFO - info message
    2005-03-19 15:10:26,695 - simple_example - WARNING - warn message
    2005-03-19 15:10:26,697 - simple_example - ERROR - error message
    2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message

The following Python module creates a logger, handler, and formatter
nearly identical to those in the example listed above, with the only
difference being the names of the objects:

    import logging
    import logging.config

    logging.config.fileConfig("logging.conf")

    # create logger
    logger = logging.getLogger("simpleExample")

    # "application" code
    logger.debug("debug message")
    logger.info("info message")
    logger.warn("warn message")
    logger.error("error message")
    logger.critical("critical message")

Here is the logging.conf file:

    [loggers]
    keys=root,simpleExample

    [handlers]
    keys=consoleHandler

    [formatters]
    keys=simpleFormatter

    [logger_root]
    level=DEBUG
    handlers=consoleHandler

    [logger_simpleExample]
    level=DEBUG
    handlers=consoleHandler
    qualname=simpleExample
    propagate=0

    [handler_consoleHandler]
    class=StreamHandler
    level=DEBUG
    formatter=simpleFormatter
    args=(sys.stdout,)

    [formatter_simpleFormatter]
    format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
    datefmt=

The output is nearly identical to that of the non-config-file-based
example:

    $ python simple_logging_config.py
    2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message
    2005-03-19 15:38:55,979 - simpleExample - INFO - info message
    2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message
    2005-03-19 15:38:56,055 - simpleExample - ERROR - error message
    2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message

You can see that the config file approach has a few advantages over the
Python code approach, mainly separation of configuration and code and
the ability of noncoders to easily modify the logging properties.

  Note that the class names referenced in config files need to be
either relative to the logging module, or absolute values which can be
resolved using normal import mechanisms. Thus, you could use either
handlers.WatchedFileHandler.  (relative to the logging
module) or `mypackage.mymodule.MyHandler' (for a class defined in
package `mypackage' and module `mymodule', where `mypackage' is
available on the Python import path).

  Changed in version 2.7.

  In Python 2.7, a new means of configuring logging has been
introduced, using dictionaries to hold configuration information. This
provides a superset of the functionality of the config-file-based
approach outlined above, and is the recommended configuration method
for new applications and deployments. Because a Python dictionary is
used to hold configuration information, and since you can populate that
dictionary using different means, you have more options for
configuration. For example, you can use a configuration file in JSON
format, or, if you have access to YAML processing functionality, a file
in YAML format, to populate the configuration dictionary. Or, of
course, you can construct the dictionary in Python code, receive it in
pickled form over a socket, or use whatever approach makes sense for
your application.

  Here's an example of the same configuration as above, in YAML format
for the new dictionary-based approach:

    version: 1
    formatters:
      simple:
        format: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
    handlers:
      console:
        class: logging.StreamHandler
        level: DEBUG
        formatter: simple
        stream: ext://sys.stdout
    loggers:
      simpleExample:
        level: DEBUG
        handlers: [console]
        propagate: no
    root:
        level: DEBUG
        handlers: [console]

For more information about logging using a dictionary, see
Configuration functions.

5.15.7.7 Configuring Logging for a Library
..........................................

When developing a library which uses logging, some consideration needs
to be given to its configuration. If the using application does not use
logging, and library code makes logging calls, then a one-off message
"No handlers could be found for logger X.Y.Z" is printed to the
console. This message is intended to catch mistakes in logging
configuration, but will confuse an application developer who is not
aware of logging by the library.

  In addition to documenting how a library uses logging, a good way to
configure library logging so that it does not cause a spurious message
is to add a handler which does nothing. This avoids the message being
printed, since a handler will be found: it just doesn't produce any
output. If the library user configures logging for application use,
presumably that configuration will add some handlers, and if levels are
suitably configured then logging calls made in library code will send
output to those handlers, as normal.

  A do-nothing handler can be simply defined as follows:

    import logging

    class NullHandler(logging.Handler):
        def emit(self, record):
            pass

An instance of this handler should be added to the top-level logger of
the logging namespace used by the library. If all logging by a library
_foo_ is done using loggers with names matching "foo.x.y", then the
code:

    import logging

    h = NullHandler()
    logging.getLogger("foo").addHandler(h)

should have the desired effect. If an organisation produces a number of
libraries, then the logger name specified can be "orgname.foo" rather
than just "foo".

  *PLEASE NOTE:* It is strongly advised that you _do not add any
handlers other than_ See NullHandler. _to your library's
loggers_. This is because the configuration of handlers is the
prerogative of the application developer who uses your library. The
application developer knows their target audience and what handlers are
most appropriate for their application: if you add handlers "under the
hood", you might well interfere with their ability to carry out unit
tests and deliver logs which suit their requirements.

  New in version 2.7.

  The NullHandler class was not present in previous
versions, but is now included, so that it need not be defined in
library code.

5.15.7.8 Logging Levels
.......................

The numeric values of logging levels are given in the following table.
These are primarily of interest if you want to define your own levels,
and need them to have specific values relative to the predefined
levels. If you define a level with the same numeric value, it
overwrites the predefined value; the predefined name is lost.

Level              Numeric value
---------------------------------------
`CRITICAL'         50
`ERROR'            40
`WARNING'          30
`INFO'             20
`DEBUG'            10
`NOTSET'           0

  Levels can also be associated with loggers, being set either by the
developer or through loading a saved logging configuration. When a
logging method is called on a logger, the logger compares its own level
with the level associated with the method call. If the logger's level
is higher than the method call's, no logging message is actually
generated. This is the basic mechanism controlling the verbosity of
logging output.

  Logging messages are encoded as instances of the LogRecord
class. When a logger decides to actually log an event, a
LogRecord instance is created from the logging message.

  Logging messages are subjected to a dispatch mechanism through the
use of _handlers_, which are instances of subclasses of the `Handler'
class. Handlers are responsible for ensuring that a logged message (in
the form of a LogRecord) ends up in a particular location
(or set of locations) which is useful for the target audience for that
message (such as end users, support desk staff, system administrators,
developers). Handlers are passed LogRecord instances
intended for particular destinations. Each logger can have zero, one or
more handlers associated with it (via the `addHandler()' method of
`Logger'). In addition to any handlers directly associated with a
logger, _all handlers associated with all ancestors of the logger_ are
called to dispatch the message (unless the _propagate_ flag for a
logger is set to a false value, at which point the passing to ancestor
handlers stops).

  Just as for loggers, handlers can have levels associated with them. A
handler's level acts as a filter in the same way as a logger's level
does. If a handler decides to actually dispatch an event, the `emit()'
method is used to send the message to its destination. Most
user-defined subclasses of `Handler' will need to override this
`emit()'.

5.15.7.9 Custom Levels
......................

Defining your own levels is possible, but should not be necessary, as
the existing levels have been chosen on the basis of practical
experience.  However, if you are convinced that you need custom levels,
great care should be exercised when doing this, and it is possibly _a
very bad idea to define custom levels if you are developing a library_.
That's because if multiple library authors all define their own custom
levels, there is a chance that the logging output from such multiple
libraries used together will be difficult for the using developer to
control and/or interpret, because a given numeric value might mean
different things for different libraries.

5.15.7.10 Useful Handlers
.........................

In addition to the base `Handler' class, many useful subclasses are
provided:

  1. StreamHandler instances send error messages to
     streams (file-like objects).

  2. FileHandler instances send error messages to disk
     files.

  3. `BaseRotatingHandler' is the base class for handlers that rotate
     log files at a certain point. It is not meant to be  instantiated
     directly. Instead, use RotatingFileHandler or
     TimedRotatingFileHandler.

  4. RotatingFileHandler instances send error messages to
     disk files, with support for maximum log file sizes and log file
     rotation.

  5. TimedRotatingFileHandler instances send error
     messages to disk files, rotating the log file at certain timed
     intervals.

  6. SocketHandler instances send error messages to TCP/IP
     sockets.

  7. DatagramHandler instances send error messages to UDP
     sockets.

  8. SMTPHandler instances send error messages to a
     designated email address.

  9. SysLogHandler instances send error messages to a Unix
     syslog daemon, possibly on a remote machine.

 10. NTEventLogHandler instances send error messages to a
     Windows NT/2000/XP event log.

 11. MemoryHandler instances send error messages to a
     buffer in memory, which is flushed whenever specific criteria are
     met.

 12. HTTPHandler instances send error messages to an HTTP
     server using either `GET' or `POST' semantics.

 13. WatchedFileHandler instances watch the file they are
     logging to. If the file changes, it is closed and reopened using
     the file name. This handler is only useful on Unix-like systems;
     Windows does not support the underlying mechanism used.

 14. NullHandler instances do nothing with error messages.
     They are used by library developers who want to use logging, but
     want to avoid the "No handlers could be found for logger XXX"
     message which can be displayed if the library user has not
     configured logging. See Configuring Logging for a Library
     for more information.

  New in version 2.7.

  The NullHandler class was not present in previous
versions.

  The NullHandler, StreamHandler and
FileHandler.  classes are defined in the core logging package.
The other handlers are defined in a sub- module, `logging.handlers'.
(There is also another sub-module, `logging.config', for configuration
functionality.)

  Logged messages are formatted for presentation through instances of
the Formatter class. They are initialized with a format
string suitable for use with the % operator and a dictionary.

  For formatting multiple messages in a batch, instances of
`BufferingFormatter' can be used. In addition to the format string
(which is applied to each message in the batch), there is provision for
header and trailer format strings.

  When filtering based on logger level and/or handler level is not
enough, instances of Filter can be added to both `Logger'
and `Handler' instances (through their `addFilter()' method). Before
deciding to process a message further, both loggers and handlers
consult all their filters for permission. If any filter returns a false
value, the message is not processed further.

  The basic Filter functionality allows filtering by
specific logger name. If this feature is used, messages sent to the
named logger and its children are allowed through the filter, and all
others dropped.

5.15.7.11 Module-Level Functions
................................

In addition to the classes described above, there are a number of
module- level functions.

 -- Function: logging.getLogger ([name])
     Return a logger with the specified name or, if no name is
     specified, return a logger which is the root logger of the
     hierarchy. If specified, the name is typically a dot-separated
     hierarchical name like _"a"_, _"a.b"_ or _"a.b.c.d"_.  Choice of
     these names is entirely up to the developer who is using logging.

     All calls to this function with a given name return the same
     logger instance.  This means that logger instances never need to
     be passed between different parts of an application.

 -- Function: logging.getLoggerClass ()
     Return either the standard `Logger' class, or the last class
     passed to setLoggerClass(). This function may be
     called from within a new class definition, to ensure that
     installing a customised `Logger' class will not undo
     customisations already applied by other code. For example:

         class MyLogger(logging.getLoggerClass()):
             # ... override behaviour here

 -- Function: logging.debug (msg[, *args[, **kwargs]])
     Logs a message with level `DEBUG' on the root logger. The _msg_ is
     the message format string, and the _args_ are the arguments which
     are merged into _msg_ using the string formatting operator. (Note
     that this means that you can use keywords in the format string,
     together with a single dictionary argument.)

     There are two keyword arguments in _kwargs_ which are inspected:
     _exc_info_ which, if it does not evaluate as false, causes
     exception information to be added to the logging message. If an
     exception tuple (in the format returned by sys.exc_info())
     is provided, it is used; otherwise, see sys.exc_info().
     is called to get the exception information.

     The other optional keyword argument is _extra_ which can be used
     to pass a dictionary which is used to populate the __dict__ of the
     LogRecord created for the logging event with user-defined
     attributes. These custom attributes can then be used as you like.
     For example, they could be incorporated into logged messages. For
     example:

         FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s"
         logging.basicConfig(format=FORMAT)
         d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}
         logging.warning("Protocol problem: %s", "connection reset", extra=d)

     would print something like:

         2006-02-08 22:20:02,165 192.168.0.1 fbloggs  Protocol problem: connection reset

     The keys in the dictionary passed in _extra_ should not clash with
     the keys used by the logging system. (See the Formatter
     documentation for more information on which keys are used by
     the logging system.)

     If you choose to use these attributes in logged messages, you need
     to exercise some care. In the above example, for instance, the
     Formatter has been set up with a format string which
     expects 'clientip' and 'user' in the attribute dictionary of the
     LogRecord. If these are missing, the message will not be logged
     because a string formatting exception will occur. So in this case,
     you always need to pass the _extra_ dictionary with these keys.

     While this might be annoying, this feature is intended for use in
     specialized circumstances, such as multi-threaded servers where
     the same code executes in many contexts, and interesting
     conditions which arise are dependent on this context (such as
     remote client IP address and authenticated user name, in the above
     example). In such circumstances, it is likely that specialized
     Formatter.s would be used with particular `Handler's.

     Changed in version 2.5: _extra_ was added.

 -- Function: logging.info (msg[, *args[, **kwargs]])
     Logs a message with level `INFO' on the root logger. The arguments
     are interpreted as for debug().

 -- Function: logging.warning (msg[, *args[, **kwargs]])
     Logs a message with level `WARNING' on the root logger. The
     arguments are interpreted as for debug().

 -- Function: logging.error (msg[, *args[, **kwargs]])
     Logs a message with level `ERROR' on the root logger. The
     arguments are interpreted as for debug().

 -- Function: logging.critical (msg[, *args[, **kwargs]])
     Logs a message with level `CRITICAL' on the root logger. The
     arguments are interpreted as for debug().

 -- Function: logging.exception (msg[, *args])
     Logs a message with level `ERROR' on the root logger. The
     arguments are interpreted as for debug(). Exception
     info is added to the logging message. This function should only be
     called from an exception handler.

 -- Function: logging.log (level, msg[, *args[, **kwargs]])
     Logs a message with level _level_ on the root logger. The other
     arguments are interpreted as for debug().

     PLEASE NOTE: The above module-level functions which delegate to
     the root logger should _not_ be used in threads, in versions of
     Python earlier than 2.7.1 and 3.2, unless at least one handler has
     been added to the root logger _before_ the threads are started.
     These convenience functions call basicConfig() to
     ensure that at least one handler is available; in earlier versions
     of Python, this can (under rare circumstances) lead to handlers
     being added multiple times to the root logger, which can in turn
     lead to multiple messages for the same event.

 -- Function: logging.disable (lvl)
     Provides an overriding level _lvl_ for all loggers which takes
     precedence over the logger's own level. When the need arises to
     temporarily throttle logging output down across the whole
     application, this function can be useful. Its effect is to disable
     all logging calls of severity _lvl_ and below, so that if you call
     it with a value of INFO, then all INFO and DEBUG events would be
     discarded, whereas those of severity WARNING and above would be
     processed according to the logger's effective level.

 -- Function: logging.addLevelName (lvl, levelName)
     Associates level _lvl_ with text _levelName_ in an internal
     dictionary, which is used to map numeric levels to a textual
     representation, for example when a Formatter formats
     a message. This function can also be used to define your own
     levels. The only constraints are that all levels used must be
     registered using this function, levels should be positive integers
     and they should increase in increasing order of severity.

     NOTE: If you are thinking of defining your own levels, please see
     the section on Custom Levels.

 -- Function: logging.getLevelName (lvl)
     Returns the textual representation of logging level _lvl_. If the
     level is one of the predefined levels `CRITICAL', `ERROR',
     `WARNING', `INFO' or `DEBUG' then you get the corresponding
     string. If you have associated levels with names using
     addLevelName() then the name you have associated with _lvl_
     is returned. If a numeric value corresponding to one of the
     defined levels is passed in, the corresponding string
     representation is returned. Otherwise, the string "Level %s" % lvl
     is returned.

 -- Function: logging.makeLogRecord (attrdict)
     Creates and returns a new LogRecord instance whose
     attributes are defined by _attrdict_. This function is useful for
     taking a pickled LogRecord attribute dictionary, sent
     over a socket, and reconstituting it as a LogRecord.
     instance at the receiving end.

 -- Function: logging.basicConfig ([**kwargs])
     Does basic configuration for the logging system by creating a
     StreamHandler with a default Formatter.
     and adding it to the root logger. The functions debug(),
     see info(), warning(), error()
     and critical() will call basicConfig()
     automatically if no handlers are defined for the root logger.

     This function does nothing if the root logger already has handlers
     configured for it.

     Changed in version 2.4: Formerly, see basicConfig() did
     not take any keyword arguments.

     PLEASE NOTE: This function should be called from the main thread
     before other threads are started. In versions of Python prior to
     2.7.1 and 3.2, if this function is called from multiple threads,
     it is possible (in rare circumstances) that a handler will be added
     to the root logger more than once, leading to unexpected results
     such as messages being duplicated in the log.

     The following keyword arguments are supported.

     Format             Description
     ---------------------------------------------------------------------
     `filename'         Specifies that a FileHandler be created, using
                        the specified filename, rather than a
                        StreamHandler.
     `filemode'         Specifies the mode to open the file, if filename
                        is specified (if filemode is unspecified, it
                        defaults to 'a').
     `format'           Use the specified format string for the handler.
     `datefmt'          Use the specified date/time format.
     `level'            Set the root logger level to the specified level.
     `stream'           Use the specified stream to initialize the
                        StreamHandler. Note that this argument is
                        incompatible with 'filename' - if both are
                        present, 'stream' is ignored.

 -- Function: logging.shutdown ()
     Informs the logging system to perform an orderly shutdown by
     flushing and closing all handlers. This should be called at
     application exit and no further use of the logging system should
     be made after this call.

 -- Function: logging.setLoggerClass (klass)
     Tells the logging system to use the class _klass_ when
     instantiating a logger.  The class should define __init__()
     such that only a name argument is required, and the
     __init__() should call `Logger.__init__()'. This function is
     typically called before any loggers are instantiated by
     applications which need to use custom logger behavior.

PEP 282 (http://www.python.org/dev/peps/pep-0282) - A Logging System
     The proposal which described this feature for inclusion in the
     Python standard library.

Original Python logging package (http://www.red-dove.com/python_logging.html)
     This is the original source for the logging package.
     The version of the package available from this site is suitable
     for use with Python 1.5.2, 2.1.x and 2.2.x, which do not include
     the logging package in the standard library.

5.15.7.12 Logger Objects
........................

Loggers have the following attributes and methods. Note that Loggers
are never instantiated directly, but always through the module-level
function `logging.getLogger(name)'.

 -- Attribute: Logger.propagate
     If this evaluates to false, logging messages are not passed by
     this logger or by its child loggers to the handlers of higher
     level (ancestor) loggers. The constructor sets this attribute to 1.

 -- Method: Logger.setLevel (lvl)
     Sets the threshold for this logger to _lvl_. Logging messages
     which are less severe than _lvl_ will be ignored. When a logger is
     created, the level is set to `NOTSET' (which causes all messages
     to be processed when the logger is the root logger, or delegation
     to the parent when the logger is a non-root logger). Note that the
     root logger is created with level `WARNING'.

     The term "delegation to the parent" means that if a logger has a
     level of NOTSET, its chain of ancestor loggers is traversed until
     either an ancestor with a level other than NOTSET is found, or the
     root is reached.

     If an ancestor is found with a level other than NOTSET, then that
     ancestor's level is treated as the effective level of the logger
     where the ancestor search began, and is used to determine how a
     logging event is handled.

     If the root is reached, and it has a level of NOTSET, then all
     messages will be processed. Otherwise, the root's level will be
     used as the effective level.

 -- Method: Logger.isEnabledFor (lvl)
     Indicates if a message of severity _lvl_ would be processed by
     this logger.  This method checks first the module-level level set
     by `logging.disable(lvl)' and then the logger's effective level as
     determined by getEffectiveLevel().

 -- Method: Logger.getEffectiveLevel ()
     Indicates the effective level for this logger. If a value other
     than `NOTSET' has been set using setLevel(), it is
     returned. Otherwise, the hierarchy is traversed towards the root
     until a value other than `NOTSET' is found, and that value is
     returned.

 -- Method: Logger.getChild (suffix)
     Returns a logger which is a descendant to this logger, as
     determined by the suffix.  Thus,
     `logging.getLogger('abc').getChild('def.ghi')' would return the
     same logger as would be returned by
     `logging.getLogger('abc.def.ghi')'. This is a convenience method,
     useful when the parent logger is named using e.g. `__name__'
     rather than a literal string.

     New in version 2.7.

 -- Method: Logger.debug (msg[, *args[, **kwargs]])
     Logs a message with level `DEBUG' on this logger. The _msg_ is the
     message format string, and the _args_ are the arguments which are
     merged into _msg_ using the string formatting operator. (Note that
     this means that you can use keywords in the format string,
     together with a single dictionary argument.)

     There are two keyword arguments in _kwargs_ which are inspected:
     _exc_info_ which, if it does not evaluate as false, causes
     exception information to be added to the logging message. If an
     exception tuple (in the format returned by sys.exc_info())
     is provided, it is used; otherwise, see sys.exc_info().
     is called to get the exception information.

     The other optional keyword argument is _extra_ which can be used
     to pass a dictionary which is used to populate the __dict__ of the
     LogRecord created for the logging event with user-defined
     attributes. These custom attributes can then be used as you like.
     For example, they could be incorporated into logged messages. For
     example:

         FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s"
         logging.basicConfig(format=FORMAT)
         d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' }
         logger = logging.getLogger("tcpserver")
         logger.warning("Protocol problem: %s", "connection reset", extra=d)

     would print something like

         2006-02-08 22:20:02,165 192.168.0.1 fbloggs  Protocol problem: connection reset

     The keys in the dictionary passed in _extra_ should not clash with
     the keys used by the logging system. (See the Formatter
     documentation for more information on which keys are used by
     the logging system.)

     If you choose to use these attributes in logged messages, you need
     to exercise some care. In the above example, for instance, the
     Formatter has been set up with a format string which
     expects 'clientip' and 'user' in the attribute dictionary of the
     LogRecord. If these are missing, the message will not be logged
     because a string formatting exception will occur. So in this case,
     you always need to pass the _extra_ dictionary with these keys.

     While this might be annoying, this feature is intended for use in
     specialized circumstances, such as multi-threaded servers where
     the same code executes in many contexts, and interesting
     conditions which arise are dependent on this context (such as
     remote client IP address and authenticated user name, in the above
     example). In such circumstances, it is likely that specialized
     Formatter.s would be used with particular `Handler's.

     Changed in version 2.5: _extra_ was added.

 -- Method: Logger.info (msg[, *args[, **kwargs]])
     Logs a message with level `INFO' on this logger. The arguments are
     interpreted as for debug().

 -- Method: Logger.warning (msg[, *args[, **kwargs]])
     Logs a message with level `WARNING' on this logger. The arguments
     are interpreted as for debug().

 -- Method: Logger.error (msg[, *args[, **kwargs]])
     Logs a message with level `ERROR' on this logger. The arguments are
     interpreted as for debug().

 -- Method: Logger.critical (msg[, *args[, **kwargs]])
     Logs a message with level `CRITICAL' on this logger. The arguments
     are interpreted as for debug().

 -- Method: Logger.log (lvl, msg[, *args[, **kwargs]])
     Logs a message with integer level _lvl_ on this logger. The other
     arguments are interpreted as for debug().

 -- Method: Logger.exception (msg[, *args])
     Logs a message with level `ERROR' on this logger. The arguments are
     interpreted as for debug(). Exception info is added to
     the logging message. This method should only be called from an
     exception handler.

 -- Method: Logger.addFilter (filt)
     Adds the specified filter _filt_ to this logger.

 -- Method: Logger.removeFilter (filt)
     Removes the specified filter _filt_ from this logger.

 -- Method: Logger.filter (record)
     Applies this logger's filters to the record and returns a true
     value if the record is to be processed.

 -- Method: Logger.addHandler (hdlr)
     Adds the specified handler _hdlr_ to this logger.

 -- Method: Logger.removeHandler (hdlr)
     Removes the specified handler _hdlr_ from this logger.

 -- Method: Logger.findCaller ()
     Finds the caller's source filename and line number. Returns the
     filename, line number and function name as a 3-element tuple.

     Changed in version 2.4: The function name was added. In earlier
     versions, the filename and line number were returned as a
     2-element tuple..

 -- Method: Logger.handle (record)
     Handles a record by passing it to all handlers associated with
     this logger and its ancestors (until a false value of _propagate_
     is found). This method is used for unpickled records received from
     a socket, as well as those created locally.  Logger-level
     filtering is applied using filter().

 -- Method: Logger.makeRecord (name, lvl, fn, lno, msg, args,
          exc_info[, func, extra])
     This is a factory method which can be overridden in subclasses to
     create specialized LogRecord instances.

     Changed in version 2.5: _func_ and _extra_ were added.

5.15.7.13 Basic example
.......................

Changed in version 2.4: formerly basicConfig() did not
take any keyword arguments.

  The logging package provides a lot of flexibility, and
its configuration can appear daunting.  This section demonstrates that
simple use of the logging package is possible.

  The simplest example shows logging to the console:

    import logging

    logging.debug('A debug message')
    logging.info('Some information')
    logging.warning('A shot across the bows')

If you run the above script, you'll see this:

    WARNING:root:A shot across the bows

Because no particular logger was specified, the system used the root
logger. The debug and info messages didn't appear because by default,
the root logger is configured to only handle messages with a severity
of WARNING or above. The message format is also a configuration
default, as is the output destination of the messages - `sys.stderr'.
The severity level, the message format and destination can be easily
changed, as shown in the example below:

    import logging

    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s %(levelname)s %(message)s',
                        filename='myapp.log',
                        filemode='w')
    logging.debug('A debug message')
    logging.info('Some information')
    logging.warning('A shot across the bows')

The basicConfig() method is used to change the
configuration defaults, which results in output (written to
`myapp.log') which should look something like the following:

    2004-07-02 13:00:08,743 DEBUG A debug message
    2004-07-02 13:00:08,743 INFO Some information
    2004-07-02 13:00:08,743 WARNING A shot across the bows

This time, all messages with a severity of DEBUG or above were handled,
and the format of the messages was also changed, and output went to the
specified file rather than the console.

  Formatting uses standard Python string formatting - see section
String Formatting Operations. The format string takes the
following common specifiers. For a complete list of specifiers, consult
the Formatter.  documentation.

Format                  Description
----------------------------------------------------------------------------
`%(name)s'              Name of the logger (logging channel).
`%(levelname)s'         Text logging level for the message (`'DEBUG'',
                        `'INFO'', `'WARNING'', `'ERROR'', `'CRITICAL'').
`%(asctime)s'           Human-readable time when the LogRecord
                        was created.  By default this is of the form
                        "2003-07-08 16:49:45,896" (the numbers after the
                        comma are millisecond portion of the time).
`%(message)s'           The logged message.

  To change the date/time format, you can pass an additional keyword
parameter, _datefmt_, as in the following:

    import logging

    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s %(levelname)-8s %(message)s',
                        datefmt='%a, %d %b %Y %H:%M:%S',
                        filename='/temp/myapp.log',
                        filemode='w')
    logging.debug('A debug message')
    logging.info('Some information')
    logging.warning('A shot across the bows')

which would result in output like

    Fri, 02 Jul 2004 13:06:18 DEBUG    A debug message
    Fri, 02 Jul 2004 13:06:18 INFO     Some information
    Fri, 02 Jul 2004 13:06:18 WARNING  A shot across the bows

The date format string follows the requirements of `strftime()' - see
the documentation for the time module.

  If, instead of sending logging output to the console or a file, you'd
rather use a file-like object which you have created separately, you
can pass it to basicConfig() using the _stream_ keyword
argument. Note that if both _stream_ and _filename_ keyword arguments
are passed, the _stream_ argument is ignored.

  Of course, you can put variable information in your output. To do
this, simply have the message be a format string and pass in additional
arguments containing the variable information, as in the following
example:

    import logging

    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s %(levelname)-8s %(message)s',
                        datefmt='%a, %d %b %Y %H:%M:%S',
                        filename='/temp/myapp.log',
                        filemode='w')
    logging.error('Pack my box with %d dozen %s', 5, 'liquor jugs')

which would result in

    Wed, 21 Jul 2004 15:35:16 ERROR    Pack my box with 5 dozen liquor jugs

5.15.7.14 Logging to multiple destinations
..........................................

Let's say you want to log to console and file with different message
formats and in differing circumstances. Say you want to log messages
with levels of DEBUG and higher to file, and those messages at level
INFO and higher to the console.  Let's also assume that the file should
contain timestamps, but the console messages should not. Here's how you
can achieve this:

    import logging

    # set up logging to file - see previous section for more details
    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
                        datefmt='%m-%d %H:%M',
                        filename='/temp/myapp.log',
                        filemode='w')
    # define a Handler which writes INFO messages or higher to the sys.stderr
    console = logging.StreamHandler()
    console.setLevel(logging.INFO)
    # set a format which is simpler for console use
    formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
    # tell the handler to use this format
    console.setFormatter(formatter)
    # add the handler to the root logger
    logging.getLogger('').addHandler(console)

    # Now, we can log to the root logger, or any other logger. First the root...
    logging.info('Jackdaws love my big sphinx of quartz.')

    # Now, define a couple of other loggers which might represent areas in your
    # application:

    logger1 = logging.getLogger('myapp.area1')
    logger2 = logging.getLogger('myapp.area2')

    logger1.debug('Quick zephyrs blow, vexing daft Jim.')
    logger1.info('How quickly daft jumping zebras vex.')
    logger2.warning('Jail zesty vixen who grabbed pay from quack.')
    logger2.error('The five boxing wizards jump quickly.')

When you run this, on the console you will see

    root        : INFO     Jackdaws love my big sphinx of quartz.
    myapp.area1 : INFO     How quickly daft jumping zebras vex.
    myapp.area2 : WARNING  Jail zesty vixen who grabbed pay from quack.
    myapp.area2 : ERROR    The five boxing wizards jump quickly.

and in the file you will see something like

    10-22 22:19 root         INFO     Jackdaws love my big sphinx of quartz.
    10-22 22:19 myapp.area1  DEBUG    Quick zephyrs blow, vexing daft Jim.
    10-22 22:19 myapp.area1  INFO     How quickly daft jumping zebras vex.
    10-22 22:19 myapp.area2  WARNING  Jail zesty vixen who grabbed pay from quack.
    10-22 22:19 myapp.area2  ERROR    The five boxing wizards jump quickly.

As you can see, the DEBUG message only shows up in the file. The other
messages are sent to both destinations.

  This example uses console and file handlers, but you can use any
number and combination of handlers you choose.

5.15.7.15 Exceptions raised during logging
..........................................

The logging package is designed to swallow exceptions which occur while
logging in production. This is so that errors which occur while
handling logging events - such as logging misconfiguration, network or
other similar errors - do not cause the application using logging to
terminate prematurely.

  `SystemExit' and `KeyboardInterrupt' exceptions are never swallowed.
Other exceptions which occur during the `emit()' method of a `Handler'
subclass are passed to its `handleError()' method.

  The default implementation of `handleError()' in `Handler' checks to
see if a module-level variable, `raiseExceptions', is set. If set, a
traceback is printed to sys.stderr. If not set, the
exception is swallowed.

  *Note_* The default value of `raiseExceptions' is `True'. This is
because during development, you typically want to be notified of any
exceptions that occur. It's advised that you set `raiseExceptions' to
`False' for production usage.

5.15.7.16 Adding contextual information to your logging output
..............................................................

Sometimes you want logging output to contain contextual information in
addition to the parameters passed to the logging call. For example, in a
networked application, it may be desirable to log client-specific
information in the log (e.g. remote client's username, or IP address).
Although you could use the _extra_ parameter to achieve this, it's not
always convenient to pass the information in this way. While it might
be tempting to create `Logger' instances on a per-connection basis,
this is not a good idea because these instances are not garbage
collected. While this is not a problem in practice, when the number of
`Logger' instances is dependent on the level of granularity you want to
use in logging an application, it could be hard to manage if the number
of `Logger' instances becomes effectively unbounded.

5.15.7.17 Using LoggerAdapters to impart contextual information
...............................................................

An easy way in which you can pass contextual information to be output
along with logging event information is to use the LoggerAdapter
class.  This class is designed to look like a `Logger', so that
you can call debug(), info(), warning(),
see error(), exception(), critical()
and log(). These methods have the same signatures as
their counterparts in `Logger', so you can use the two types of
instances interchangeably.

  When you create an instance of LoggerAdapter, you pass it a
`Logger' instance and a dict-like object which contains your contextual
information. When you call one of the logging methods on an instance of
LoggerAdapter, it delegates the call to the underlying
instance of `Logger' passed to its constructor, and arranges to pass
the contextual information in the delegated call. Here's a snippet from
the code of LoggerAdapter.:

    def debug(self, msg, *args, **kwargs):
        """
        Delegate a debug call to the underlying logger, after adding
        contextual information from this adapter instance.
        """
        msg, kwargs = self.process(msg, kwargs)
        self.logger.debug(msg, *args, **kwargs)

The `process()' method of LoggerAdapter is where the
contextual information is added to the logging output. It's passed the
message and keyword arguments of the logging call, and it passes back
(potentially) modified versions of these to use in the call to the
underlying logger. The default implementation of this method leaves the
message alone, but inserts an "extra" key in the keyword argument whose
value is the dict-like object passed to the constructor. Of course, if
you had passed an "extra" keyword argument in the call to the adapter,
it will be silently overwritten.

  The advantage of using "extra" is that the values in the dict-like
object are merged into the LogRecord instance's __dict__,
allowing you to use customized strings with your Formatter.
instances which know about the keys of the dict-like object. If you
need a different method, e.g. if you want to prepend or append the
contextual information to the message string, you just need to subclass
LoggerAdapter and override `process()' to do what you need.
Here's an example script which uses this class, which also illustrates
what dict-like behaviour is needed from an arbitrary "dict-like" object
for use in the constructor:

    import logging

    class ConnInfo:
        """
        An example class which shows how an arbitrary class can be used as
        the 'extra' context information repository passed to a LoggerAdapter.
        """

        def __getitem__(self, name):
            """
            To allow this instance to look like a dict.
            """
            from random import choice
            if name == "ip":
                result = choice(["127.0.0.1", "192.168.0.1"])
            elif name == "user":
                result = choice(["jim", "fred", "sheila"])
            else:
                result = self.__dict__.get(name, "?")
            return result

        def __iter__(self):
            """
            To allow iteration over keys, which will be merged into
            the LogRecord dict before formatting and output.
            """
            keys = ["ip", "user"]
            keys.extend(self.__dict__.keys())
            return keys.__iter__()

    if __name__ == "__main__":
        from random import choice
        levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL)
        a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"),
                                   { "ip" : "123.231.231.123", "user" : "sheila" })
        logging.basicConfig(level=logging.DEBUG,
                            format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s")
        a1.debug("A debug message")
        a1.info("An info message with %s", "some parameters")
        a2 = logging.LoggerAdapter(logging.getLogger("d.e.f"), ConnInfo())
        for x in range(10):
            lvl = choice(levels)
            lvlname = logging.getLevelName(lvl)
            a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters")

When this script is run, the output should look something like this:

    2008-01-18 14:49:54,023 a.b.c DEBUG    IP: 123.231.231.123 User: sheila   A debug message
    2008-01-18 14:49:54,023 a.b.c INFO     IP: 123.231.231.123 User: sheila   An info message with some parameters
    2008-01-18 14:49:54,023 d.e.f CRITICAL IP: 192.168.0.1     User: jim      A message at CRITICAL level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f INFO     IP: 192.168.0.1     User: jim      A message at INFO level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f WARNING  IP: 192.168.0.1     User: sheila   A message at WARNING level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f ERROR    IP: 127.0.0.1       User: fred     A message at ERROR level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f ERROR    IP: 127.0.0.1       User: sheila   A message at ERROR level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f WARNING  IP: 192.168.0.1     User: sheila   A message at WARNING level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f WARNING  IP: 192.168.0.1     User: jim      A message at WARNING level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f INFO     IP: 192.168.0.1     User: fred     A message at INFO level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f WARNING  IP: 192.168.0.1     User: sheila   A message at WARNING level with 2 parameters
    2008-01-18 14:49:54,033 d.e.f WARNING  IP: 127.0.0.1       User: jim      A message at WARNING level with 2 parameters

New in version 2.6.

  The LoggerAdapter class was not present in previous
versions.

5.15.7.18 Using Filters to impart contextual information
........................................................

You can also add contextual information to log output using a
user-defined Filter. `Filter' instances are allowed to
modify the `LogRecords' passed to them, including adding additional
attributes which can then be output using a suitable format string, or
if needed a custom Formatter.

  For example in a web application, the request being processed (or at
least, the interesting parts of it) can be stored in a threadlocal
(see threading.local) variable, and then accessed from a
`Filter' to add, say, information from the request - say, the remote IP
address and remote user's username - to the `LogRecord', using the
attribute names 'ip' and 'user' as in the `LoggerAdapter' example
above. In that case, the same format string can be used to get similar
output to that shown above. Here's an example script:

    import logging
    from random import choice

    class ContextFilter(logging.Filter):
        """
        This is a filter which injects contextual information into the log.

        Rather than use actual contextual information, we just use random
        data in this demo.
        """

        USERS = ['jim', 'fred', 'sheila']
        IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1']

        def filter(self, record):

            record.ip = choice(ContextFilter.IPS)
            record.user = choice(ContextFilter.USERS)
            return True

    if __name__ == "__main__":
       levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL)
       a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"),
                                  { "ip" : "123.231.231.123", "user" : "sheila" })
       logging.basicConfig(level=logging.DEBUG,
                           format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s")
       a1 = logging.getLogger("a.b.c")
       a2 = logging.getLogger("d.e.f")

       f = ContextFilter()
       a1.addFilter(f)
       a2.addFilter(f)
       a1.debug("A debug message")
       a1.info("An info message with %s", "some parameters")
       for x in range(10):
           lvl = choice(levels)
           lvlname = logging.getLevelName(lvl)
           a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters")

which, when run, produces something like:

    2010-09-06 22:38:15,292 a.b.c DEBUG    IP: 123.231.231.123 User: fred     A debug message
    2010-09-06 22:38:15,300 a.b.c INFO     IP: 192.168.0.1     User: sheila   An info message with some parameters
    2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1       User: sheila   A message at CRITICAL level with 2 parameters
    2010-09-06 22:38:15,300 d.e.f ERROR    IP: 127.0.0.1       User: jim      A message at ERROR level with 2 parameters
    2010-09-06 22:38:15,300 d.e.f DEBUG    IP: 127.0.0.1       User: sheila   A message at DEBUG level with 2 parameters
    2010-09-06 22:38:15,300 d.e.f ERROR    IP: 123.231.231.123 User: fred     A message at ERROR level with 2 parameters
    2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1     User: jim      A message at CRITICAL level with 2 parameters
    2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1       User: sheila   A message at CRITICAL level with 2 parameters
    2010-09-06 22:38:15,300 d.e.f DEBUG    IP: 192.168.0.1     User: jim      A message at DEBUG level with 2 parameters
    2010-09-06 22:38:15,301 d.e.f ERROR    IP: 127.0.0.1       User: sheila   A message at ERROR level with 2 parameters
    2010-09-06 22:38:15,301 d.e.f DEBUG    IP: 123.231.231.123 User: fred     A message at DEBUG level with 2 parameters
    2010-09-06 22:38:15,301 d.e.f INFO     IP: 123.231.231.123 User: fred     A message at INFO level with 2 parameters

5.15.7.19 Logging to a single file from multiple processes
..........................................................

Although logging is thread-safe, and logging to a single file from
multiple threads in a single process _is_ supported, logging to a
single file from _multiple processes_ is _not_ supported, because there
is no standard way to serialize access to a single file across multiple
processes in Python. If you need to log to a single file from multiple
processes, the best way of doing this is to have all the processes log
to a `SocketHandler', and have a separate process which implements a
socket server which reads from the socket and logs to file. (If you
prefer, you can dedicate one thread in one of the existing processes to
perform this function.) The following section documents this approach
in more detail and includes a working socket receiver which can be used
as a starting point for you to adapt in your own applications.

  If you are using a recent version of Python which includes the
multiprocessing module, you can write your own handler which uses
the `Lock' class from this module to serialize access to the file from
your processes. The existing FileHandler and subclasses do
not make use of multiprocessing at present, though they may
do so in the future.  Note that at present, the multiprocessing
module does not provide working lock functionality on all
platforms (see <http://bugs.python.org/issue3770>).

5.15.7.20 Sending and receiving logging events across a network
...............................................................

Let's say you want to send logging events across a network, and handle
them at the receiving end. A simple way of doing this is attaching a
`SocketHandler' instance to the root logger at the sending end:

    import logging, logging.handlers

    rootLogger = logging.getLogger('')
    rootLogger.setLevel(logging.DEBUG)
    socketHandler = logging.handlers.SocketHandler('localhost',
                        logging.handlers.DEFAULT_TCP_LOGGING_PORT)
    # don't bother with a formatter, since a socket handler sends the event as
    # an unformatted pickle
    rootLogger.addHandler(socketHandler)

    # Now, we can log to the root logger, or any other logger. First the root...
    logging.info('Jackdaws love my big sphinx of quartz.')

    # Now, define a couple of other loggers which might represent areas in your
    # application:

    logger1 = logging.getLogger('myapp.area1')
    logger2 = logging.getLogger('myapp.area2')

    logger1.debug('Quick zephyrs blow, vexing daft Jim.')
    logger1.info('How quickly daft jumping zebras vex.')
    logger2.warning('Jail zesty vixen who grabbed pay from quack.')
    logger2.error('The five boxing wizards jump quickly.')

At the receiving end, you can set up a receiver using the
SocketServer.  module. Here is a basic working example:

    import cPickle
    import logging
    import logging.handlers
    import SocketServer
    import struct

    class LogRecordStreamHandler(SocketServer.StreamRequestHandler):
        """Handler for a streaming logging request.

        This basically logs the record using whatever logging policy is
        configured locally.
        """

        def handle(self):
            """
            Handle multiple requests - each expected to be a 4-byte length,
            followed by the LogRecord in pickle format. Logs the record
            according to whatever policy is configured locally.
            """
            while 1:
                chunk = self.connection.recv(4)
                if len(chunk) < 4:
                    break
                slen = struct.unpack(">L", chunk)[0]
                chunk = self.connection.recv(slen)
                while len(chunk) < slen:
                    chunk = chunk + self.connection.recv(slen - len(chunk))
                obj = self.unPickle(chunk)
                record = logging.makeLogRecord(obj)
                self.handleLogRecord(record)

        def unPickle(self, data):
            return cPickle.loads(data)

        def handleLogRecord(self, record):
            # if a name is specified, we use the named logger rather than the one
            # implied by the record.
            if self.server.logname is not None:
                name = self.server.logname
            else:
                name = record.name
            logger = logging.getLogger(name)
            # N.B. EVERY record gets logged. This is because Logger.handle
            # is normally called AFTER logger-level filtering. If you want
            # to do filtering, do it at the client end to save wasting
            # cycles and network bandwidth!
            logger.handle(record)

    class LogRecordSocketReceiver(SocketServer.ThreadingTCPServer):
        """simple TCP socket-based logging receiver suitable for testing.
        """

        allow_reuse_address = 1

        def __init__(self, host='localhost',
                     port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
                     handler=LogRecordStreamHandler):
            SocketServer.ThreadingTCPServer.__init__(self, (host, port), handler)
            self.abort = 0
            self.timeout = 1
            self.logname = None

        def serve_until_stopped(self):
            import select
            abort = 0
            while not abort:
                rd, wr, ex = select.select([self.socket.fileno()],
                                           [], [],
                                           self.timeout)
                if rd:
                    self.handle_request()
                abort = self.abort

    def main():
        logging.basicConfig(
            format="%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s")
        tcpserver = LogRecordSocketReceiver()
        print "About to start TCP server..."
        tcpserver.serve_until_stopped()

    if __name__ == "__main__":
        main()

First run the server, and then the client. On the client side, nothing
is printed on the console; on the server side, you should see something
like:

    About to start TCP server...
       59 root            INFO     Jackdaws love my big sphinx of quartz.
       59 myapp.area1     DEBUG    Quick zephyrs blow, vexing daft Jim.
       69 myapp.area1     INFO     How quickly daft jumping zebras vex.
       69 myapp.area2     WARNING  Jail zesty vixen who grabbed pay from quack.
       69 myapp.area2     ERROR    The five boxing wizards jump quickly.

Note that there are some security issues with pickle in some scenarios.
If these affect you, you can use an alternative serialization scheme by
overriding the `makePickle()' method and implementing your alternative
there, as well as adapting the above script to use your alternative
serialization.

5.15.7.21 Using arbitrary objects as messages
.............................................

In the preceding sections and examples, it has been assumed that the
message passed when logging the event is a string. However, this is not
the only possibility. You can pass an arbitrary object as a message,
and its __str__() method will be called when the logging
system needs to convert it to a string representation. In fact, if you
want to, you can avoid computing a string representation altogether -
for example, the `SocketHandler' emits an event by pickling it and
sending it over the wire.

5.15.7.22 Optimization
......................

Formatting of message arguments is deferred until it cannot be avoided.
However, computing the arguments passed to the logging method can also
be expensive, and you may want to avoid doing it if the logger will
just throw away your event. To decide what to do, you can call the
`isEnabledFor()' method which takes a level argument and returns true
if the event would be created by the Logger for that level of call. You
can write code like this:

    if logger.isEnabledFor(logging.DEBUG):
        logger.debug("Message with %s, %s", expensive_func1(),
                                            expensive_func2())

so that if the logger's threshold is set above `DEBUG', the calls to
`expensive_func1()' and `expensive_func2()' are never made.

  There are other optimizations which can be made for specific
applications which need more precise control over what logging
information is collected. Here's a list of things you can do to avoid
processing during logging which you don't need:

What you don't want to collect                      How to avoid collecting it
-------------------------------------------------------------------------------------------------
Information about where calls were made from.       Set `logging._srcfile' to `None'.
Threading information.                              Set `logging.logThreads' to `0'.
Process information.                                Set `logging.logProcesses' to `0'.

  Also note that the core logging module only includes the basic
handlers. If you don't import `logging.handlers' and `logging.config',
they won't take up any memory.

5.15.7.23 Handler Objects
.........................

Handlers have the following attributes and methods. Note that `Handler'
is never instantiated directly; this class acts as a base for more
useful subclasses. However, the __init__() method in
subclasses needs to call Handler.__init__().

 -- Method: Handler.__init__ (level=NOTSET)
     Initializes the `Handler' instance by setting its level, setting
     the list of filters to the empty list and creating a lock (using
     createLock()) for serializing access to an I/O
     mechanism.

 -- Method: Handler.createLock ()
     Initializes a thread lock which can be used to serialize access to
     underlying I/O functionality which may not be threadsafe.

 -- Method: Handler.acquire ()
     Acquires the thread lock created with createLock().

 -- Method: Handler.release ()
     Releases the thread lock acquired with acquire().

 -- Method: Handler.setLevel (lvl)
     Sets the threshold for this handler to _lvl_. Logging messages
     which are less severe than _lvl_ will be ignored. When a handler
     is created, the level is set to `NOTSET' (which causes all
     messages to be processed).

 -- Method: Handler.setFormatter (form)
     Sets the Formatter for this handler to _form_.

 -- Method: Handler.addFilter (filt)
     Adds the specified filter _filt_ to this handler.

 -- Method: Handler.removeFilter (filt)
     Removes the specified filter _filt_ from this handler.

 -- Method: Handler.filter (record)
     Applies this handler's filters to the record and returns a true
     value if the record is to be processed.

 -- Method: Handler.flush ()
     Ensure all logging output has been flushed. This version does
     nothing and is intended to be implemented by subclasses.

 -- Method: Handler.close ()
     Tidy up any resources used by the handler. This version does no
     output but removes the handler from an internal list of handlers
     which is closed when shutdown() is called. Subclasses
     should ensure that this gets called from overridden close()
     methods.

 -- Method: Handler.handle (record)
     Conditionally emits the specified logging record, depending on
     filters which may have been added to the handler. Wraps the actual
     emission of the record with acquisition/release of the I/O thread
     lock.

 -- Method: Handler.handleError (record)
     This method should be called from handlers when an exception is
     encountered during an emit() call. By default it does
     nothing, which means that exceptions get silently ignored. This is
     what is mostly wanted for a logging system - most users will not
     care about errors in the logging system, they are more interested
     in application errors. You could, however, replace this with a
     custom handler if you wish. The specified record is the one which
     was being processed when the exception occurred.

 -- Method: Handler.format (record)
     Do formatting for a record - if a formatter is set, use it.
     Otherwise, use the default formatter for the module.

 -- Method: Handler.emit (record)
     Do whatever it takes to actually log the specified logging record.
     This version is intended to be implemented by subclasses and so
     raises a NotImplementedError.

5.15.7.24 StreamHandler
.......................

The StreamHandler class, located in the core
logging package, sends logging output to streams such as
_sys.stdout_, _sys.stderr_ or any file-like object (or, more precisely,
any object which supports `write()' and `flush()' methods).

 -- Class: logging.StreamHandler ([stream])
     Returns a new instance of the StreamHandler class. If
     _stream_ is specified, the instance will use it for logging
     output; otherwise, _sys.stderr_ will be used.

      -- Method: emit (record)
          If a formatter is specified, it is used to format the record.
          The record is then written to the stream with a trailing
          newline. If exception information is present, it is formatted
          using traceback.print_exception() and appended
          to the stream.

      -- Method: flush ()
          Flushes the stream by calling its flush().
          method. Note that the `close()' method is inherited from
          `Handler' and so does no output, so an explicit
          flush() call may be needed at times.

5.15.7.25 FileHandler
.....................

The FileHandler class, located in the core logging
package, sends logging output to a disk file.  It inherits the
output functionality from StreamHandler.

 -- Class: logging.FileHandler (filename[, mode[, encoding[, delay]]])
     Returns a new instance of the FileHandler class. The
     specified file is opened and used as the stream for logging. If
     _mode_ is not specified, `'a'' is used.  If _encoding_ is not
     _None_, it is used to open the file with that encoding.  If
     _delay_ is true, then file opening is deferred until the first
     call to emit(). By default, the file grows
     indefinitely.

     Changed in version 2.6: _delay_ was added.

      -- Method: close ()
          Closes the file.

      -- Method: emit (record)
          Outputs the record to the file.

5.15.7.26 NullHandler
.....................

New in version 2.7.

  The NullHandler class, located in the core
logging package, does not do any formatting or output. It is
essentially a "no-op" handler for use by library developers.

 -- Class: logging.NullHandler
     Returns a new instance of the NullHandler class.

      -- Method: emit (record)
          This method does nothing.

      -- Method: handle (record)
          This method does nothing.

      -- Method: createLock ()
          This method returns `None' for the lock, since there is no
          underlying I/O to which access needs to be serialized.

  See Configuring Logging for a Library for more
information on how to use NullHandler.

5.15.7.27 WatchedFileHandler
............................

New in version 2.6.

  The WatchedFileHandler class, located in the
`logging.handlers' module, is a `FileHandler' which watches the file it
is logging to. If the file changes, it is closed and reopened using the
file name.

  A file change can happen because of usage of programs such as
_newsyslog_ and _logrotate_ which perform log file rotation. This
handler, intended for use under Unix/Linux, watches the file to see if
it has changed since the last emit.  (A file is deemed to have changed
if its device or inode have changed.) If the file has changed, the old
file stream is closed, and the file opened to get a new stream.

  This handler is not appropriate for use under Windows, because under
Windows open log files cannot be moved or renamed - logging opens the
files with exclusive locks - and so there is no need for such a
handler. Furthermore, _ST_INO_ is not supported under Windows; see
stat() always returns zero for this value.

 -- Class: logging.handlers.WatchedFileHandler (filename[, mode[,
          encoding[, delay]]])
     Returns a new instance of the WatchedFileHandler.
     class. The specified file is opened and used as the stream for
     logging. If _mode_ is not specified, `'a'' is used.  If _encoding_
     is not _None_, it is used to open the file with that encoding.  If
     _delay_ is true, then file opening is deferred until the first
     call to emit().  By default, the file grows
     indefinitely.

     Changed in version 2.6: _delay_ was added.

      -- Method: emit (record)
          Outputs the record to the file, but first checks to see if
          the file has changed.  If it has, the existing stream is
          flushed and closed and the file opened again, before
          outputting the record to the file.

5.15.7.28 RotatingFileHandler
.............................

The RotatingFileHandler class, located in the
`logging.handlers' module, supports rotation of disk log files.

 -- Class: logging.handlers.RotatingFileHandler (filename[, mode[,
          maxBytes[, backupCount[, encoding[, delay]]]]])
     Returns a new instance of the RotatingFileHandler.
     class. The specified file is opened and used as the stream for
     logging. If _mode_ is not specified, `'a'' is used.  If _encoding_
     is not _None_, it is used to open the file with that encoding.  If
     _delay_ is true, then file opening is deferred until the first
     call to emit().  By default, the file grows
     indefinitely.

     You can use the _maxBytes_ and _backupCount_ values to allow the
     file to _rollover_ at a predetermined size. When the size is about
     to be exceeded, the file is closed and a new file is silently
     opened for output. Rollover occurs whenever the current log file
     is nearly _maxBytes_ in length; if _maxBytes_ is zero, rollover
     never occurs.  If _backupCount_ is non-zero, the system will save
     old log files by appending the extensions ".1", ".2" etc., to the
     filename. For example, with a _backupCount_ of 5 and a base file
     name of `app.log', you would get `app.log', `app.log.1',
     `app.log.2', up to `app.log.5'. The file being written to is
     always `app.log'.  When this file is filled, it is closed and
     renamed to `app.log.1', and if files `app.log.1', `app.log.2',
     etc.  exist, then they are renamed to `app.log.2', `app.log.3'
     etc.  respectively.

     Changed in version 2.6: _delay_ was added.

      -- Method: doRollover ()
          Does a rollover, as described above.

      -- Method: emit (record)
          Outputs the record to the file, catering for rollover as
          described previously.

5.15.7.29 TimedRotatingFileHandler
..................................

The TimedRotatingFileHandler class, located in the
`logging.handlers' module, supports rotation of disk log files at
certain timed intervals.

 -- Class: logging.handlers.TimedRotatingFileHandler (filename[, when[,
          interval[, backupCount[, encoding[, delay[, utc]]]]]])
     Returns a new instance of the TimedRotatingFileHandler
     class. The specified file is opened and used as the stream
     for logging. On rotating it also sets the filename suffix.
     Rotating happens based on the product of _when_ and _interval_.

     You can use the _when_ to specify the type of _interval_. The list
     of possible values is below.  Note that they are not case
     sensitive.

     Value                Type of interval
     -------------------------------------------------
     `'S''                Seconds
     `'M''                Minutes
     `'H''                Hours
     `'D''                Days
     `'W''                Week day (0=Monday)
     `'midnight''         Roll over at midnight

     The system will save old log files by appending extensions to the
     filename.  The extensions are date-and-time based, using the
     strftime format `%Y-%m-%d_%H-%M-%S' or a leading portion thereof,
     depending on the rollover interval.

     When computing the next rollover time for the first time (when the
     handler is created), the last modification time of an existing log
     file, or else the current time, is used to compute when the next
     rotation will occur.

     If the _utc_ argument is true, times in UTC will be used; otherwise
     local time is used.

     If _backupCount_ is nonzero, at most _backupCount_ files will be
     kept, and if more would be created when rollover occurs, the oldest
     one is deleted. The deletion logic uses the interval to determine
     which files to delete, so changing the interval may leave old
     files lying around.

     If _delay_ is true, then file opening is deferred until the first
     call to emit().

     Changed in version 2.6: _delay_ was added.

      -- Method: doRollover ()
          Does a rollover, as described above.

      -- Method: emit (record)
          Outputs the record to the file, catering for rollover as
          described above.

5.15.7.30 SocketHandler
.......................

The SocketHandler class, located in the `logging.handlers'
module, sends logging output to a network socket. The base class uses a
TCP socket.

 -- Class: logging.handlers.SocketHandler (host, port)
     Returns a new instance of the SocketHandler class
     intended to communicate with a remote machine whose address is
     given by _host_ and _port_.

      -- Method: close ()
          Closes the socket.

      -- Method: emit ()
          Pickles the record's attribute dictionary and writes it to
          the socket in binary format. If there is an error with the
          socket, silently drops the packet. If the connection was
          previously lost, re-establishes the connection. To unpickle
          the record at the receiving end into a `LogRecord', use the
          `makeLogRecord()' function.

      -- Method: handleError ()
          Handles an error which has occurred during emit().
          The most likely cause is a lost connection. Closes the
          socket so that we can retry on the next event.

      -- Method: makeSocket ()
          This is a factory method which allows subclasses to define
          the precise type of socket they want. The default
          implementation creates a TCP socket (see
          socket.SOCK_STREAM).

      -- Method: makePickle (record)
          Pickles the record's attribute dictionary in binary format
          with a length prefix, and returns it ready for transmission
          across the socket.

          Note that pickles aren't completely secure. If you are
          concerned about security, you may want to override this
          method to implement a more secure mechanism. For example, you
          can sign pickles using HMAC and then verify them on the
          receiving end, or alternatively you can disable unpickling of
          global objects on the receiving end.

      -- Method: send (packet)
          Send a pickled string _packet_ to the socket. This function
          allows for partial sends which can happen when the network is
          busy.

5.15.7.31 DatagramHandler
.........................

The DatagramHandler class, located in the
`logging.handlers' module, inherits from SocketHandler to
support sending logging messages over UDP sockets.

 -- Class: logging.handlers.DatagramHandler (host, port)
     Returns a new instance of the DatagramHandler class
     intended to communicate with a remote machine whose address is
     given by _host_ and _port_.

      -- Method: emit ()
          Pickles the record's attribute dictionary and writes it to
          the socket in binary format. If there is an error with the
          socket, silently drops the packet. To unpickle the record at
          the receiving end into a `LogRecord', use the
          `makeLogRecord()' function.

      -- Method: makeSocket ()
          The factory method of SocketHandler is here
          overridden to create a UDP socket (see socket.SOCK_DGRAM).

      -- Method: send (s)
          Send a pickled string to a socket.

5.15.7.32 SysLogHandler
.......................

The SysLogHandler class, located in the `logging.handlers'
module, supports sending logging messages to a remote or local Unix
syslog.

 -- Class: logging.handlers.SysLogHandler ([address[, facility[,
          socktype]]])
     Returns a new instance of the SysLogHandler class
     intended to communicate with a remote Unix machine whose address
     is given by _address_ in the form of a `(host, port)' tuple.  If
     _address_ is not specified, `('localhost', 514)' is used.  The
     address is used to open a socket.  An alternative to providing a
     `(host, port)' tuple is providing an address as a string, for
     example "/dev/log". In this case, a Unix domain socket is used to
     send the message to the syslog. If _facility_ is not specified,
     `LOG_USER' is used. The type of socket opened depends on the
     _socktype_ argument, which defaults to socket.SOCK_DGRAM
     and thus opens a UDP socket. To open a TCP socket (for use
     with the newer syslog daemons such as rsyslog), specify a value of
     socket.SOCK_STREAM.

     Changed in version 2.7: _socktype_ was added.

      -- Method: close ()
          Closes the socket to the remote host.

      -- Method: emit (record)
          The record is formatted, and then sent to the syslog server.
          If exception information is present, it is _not_ sent to the
          server.

      -- Method: encodePriority (facility, priority)
          Encodes the facility and priority into an integer. You can
          pass in strings or integers - if strings are passed, internal
          mapping dictionaries are used to convert them to integers.

          The symbolic `LOG_' values are defined in
          SysLogHandler and mirror the values defined in the
          `sys/syslog.h' header file.

          *Priorities*

          Name (string)                  Symbolic value
          ---------------------------------------------------
          `alert'                        LOG_ALERT
          `crit' or `critical'           LOG_CRIT
          `debug'                        LOG_DEBUG
          `emerg' or `panic'             LOG_EMERG
          `err' or `error'               LOG_ERR
          `info'                         LOG_INFO
          `notice'                       LOG_NOTICE
          `warn' or `warning'            LOG_WARNING

          *Facilities*

          Name (string)       Symbolic value
          ----------------------------------------
          `auth'              LOG_AUTH
          `authpriv'          LOG_AUTHPRIV
          `cron'              LOG_CRON
          `daemon'            LOG_DAEMON
          `ftp'               LOG_FTP
          `kern'              LOG_KERN
          `lpr'               LOG_LPR
          `mail'              LOG_MAIL
          `news'              LOG_NEWS
          `syslog'            LOG_SYSLOG
          `user'              LOG_USER
          `uucp'              LOG_UUCP
          `local0'            LOG_LOCAL0
          `local1'            LOG_LOCAL1
          `local2'            LOG_LOCAL2
          `local3'            LOG_LOCAL3
          `local4'            LOG_LOCAL4
          `local5'            LOG_LOCAL5
          `local6'            LOG_LOCAL6
          `local7'            LOG_LOCAL7

      -- Method: mapPriority (levelname)
          Maps a logging level name to a syslog priority name.  You may
          need to override this if you are using custom levels, or if
          the default algorithm is not suitable for your needs. The
          default algorithm maps `DEBUG', `INFO', `WARNING', `ERROR' and
          `CRITICAL' to the equivalent syslog names, and all other level
          names to "warning".

5.15.7.33 NTEventLogHandler
...........................

The NTEventLogHandler class, located in the
`logging.handlers' module, supports sending logging messages to a local
Windows NT, Windows 2000 or Windows XP event log. Before you can use
it, you need Mark Hammond's Win32 extensions for Python installed.

 -- Class: logging.handlers.NTEventLogHandler (appname[, dllname[,
          logtype]])
     Returns a new instance of the NTEventLogHandler.
     class. The _appname_ is used to define the application name as it
     appears in the event log. An appropriate registry entry is created
     using this name. The _dllname_ should give the fully qualified
     pathname of a .dll or .exe which contains message definitions to
     hold in the log (if not specified, `'win32service.pyd'' is used -
     this is installed with the Win32 extensions and contains some basic
     placeholder message definitions. Note that use of these
     placeholders will make your event logs big, as the entire message
     source is held in the log. If you want slimmer logs, you have to
     pass in the name of your own .dll or .exe which contains the
     message definitions you want to use in the event log). The
     _logtype_ is one of `'Application'', `'System'' or `'Security'',
     and defaults to `'Application''.

      -- Method: close ()
          At this point, you can remove the application name from the
          registry as a source of event log entries. However, if you do
          this, you will not be able to see the events as you intended
          in the Event Log Viewer - it needs to be able to access the
          registry to get the .dll name. The current version does not
          do this.

      -- Method: emit (record)
          Determines the message ID, event category and event type, and
          then logs the message in the NT event log.

      -- Method: getEventCategory (record)
          Returns the event category for the record. Override this if
          you want to specify your own categories. This version returns
          0.

      -- Method: getEventType (record)
          Returns the event type for the record. Override this if you
          want to specify your own types. This version does a mapping
          using the handler's typemap attribute, which is set up in
          __init__() to a dictionary which contains
          mappings for `DEBUG', `INFO', `WARNING', `ERROR' and
          `CRITICAL'. If you are using your own levels, you will either
          need to override this method or place a suitable dictionary
          in the handler's _typemap_ attribute.

      -- Method: getMessageID (record)
          Returns the message ID for the record. If you are using your
          own messages, you could do this by having the _msg_ passed to
          the logger being an ID rather than a format string. Then, in
          here, you could use a dictionary lookup to get the message
          ID. This version returns 1, which is the base message ID in
          `win32service.pyd'.

5.15.7.34 SMTPHandler
.....................

The SMTPHandler class, located in the `logging.handlers'
module, supports sending logging messages to an email address via SMTP.

 -- Class: logging.handlers.SMTPHandler (mailhost, fromaddr, toaddrs,
          subject[, credentials])
     Returns a new instance of the SMTPHandler class. The
     instance is initialized with the from and to addresses and subject
     line of the email. The _toaddrs_ should be a list of strings. To
     specify a non-standard SMTP port, use the (host, port) tuple
     format for the _mailhost_ argument. If you use a string, the
     standard SMTP port is used. If your SMTP server requires
     authentication, you can specify a (username, password) tuple for
     the _credentials_ argument.

     Changed in version 2.6: _credentials_ was added.

      -- Method: emit (record)
          Formats the record and sends it to the specified addressees.

      -- Method: getSubject (record)
          If you want to specify a subject line which is
          record-dependent, override this method.

5.15.7.35 MemoryHandler
.......................

The MemoryHandler class, located in the `logging.handlers'
module, supports buffering of logging records in memory, periodically
flushing them to a _target_ handler. Flushing occurs whenever the
buffer is full, or when an event of a certain severity or greater is
seen.

  MemoryHandler is a subclass of the more general
BufferingHandler, which is an abstract class. This buffers logging
records in memory. Whenever each record is added to the buffer, a check
is made by calling `shouldFlush()' to see if the buffer should be
flushed.  If it should, then `flush()' is expected to do the needful.

 -- Class: logging.handlers.BufferingHandler (capacity)
     Initializes the handler with a buffer of the specified capacity.

      -- Method: emit (record)
          Appends the record to the buffer. If shouldFlush()
          returns true, calls flush() to process the
          buffer.

      -- Method: flush ()
          You can override this to implement custom flushing behavior.
          This version just zaps the buffer to empty.

      -- Method: shouldFlush (record)
          Returns true if the buffer is up to capacity. This method can
          be overridden to implement custom flushing strategies.

 -- Class: logging.handlers.MemoryHandler (capacity[, flushLevel[,
          target]])
     Returns a new instance of the MemoryHandler class.
     The instance is initialized with a buffer size of _capacity_. If
     _flushLevel_ is not specified, `ERROR' is used. If no _target_ is
     specified, the target will need to be set using setTarget()
     before this handler does anything useful.

      -- Method: close ()
          Calls flush(), sets the target to None
          and clears the buffer.

      -- Method: flush ()
          For a MemoryHandler, flushing means just sending
          the buffered records to the target, if there is one. Override
          if you want different behavior.

      -- Method: setTarget (target)
          Sets the target handler for this handler.

      -- Method: shouldFlush (record)
          Checks for buffer full or a record at the _flushLevel_ or
          higher.

5.15.7.36 HTTPHandler
.....................

The HTTPHandler class, located in the `logging.handlers'
module, supports sending logging messages to a Web server, using either
`GET' or `POST' semantics.

 -- Class: logging.handlers.HTTPHandler (host, url[, method])
     Returns a new instance of the HTTPHandler class. The
     instance is initialized with a host address, url and HTTP method.
     The _host_ can be of the form `host:port', should you need to use
     a specific port number. If no _method_ is specified, `GET' is used.

      -- Method: emit (record)
          Sends the record to the Web server as a percent-encoded
          dictionary.

5.15.7.37 Formatter Objects
...........................

See Formatter.s have the following attributes and methods. They
are responsible for converting a LogRecord to (usually) a
string which can be interpreted by either a human or an external
system. The base Formatter allows a formatting string to
be specified. If none is supplied, the default value of `'%(message)s''
is used.

  A Formatter can be initialized with a format string which makes use
of knowledge of the LogRecord attributes - such as the
default value mentioned above making use of the fact that the user's
message and arguments are pre-formatted into a LogRecord.'s
_message_ attribute.  This format string contains standard Python
%-style mapping keys. See section String Formatting Operations.
for more information on string formatting.

  Currently, the useful mapping keys in a LogRecord are:

Format                        Description
----------------------------------------------------------------------------------
`%(name)s'                    Name of the logger (logging channel).
`%(levelno)s'                 Numeric logging level for the message (`DEBUG',
                              `INFO', `WARNING', `ERROR', `CRITICAL').
`%(levelname)s'               Text logging level for the message (`'DEBUG'',
                              `'INFO'', `'WARNING'', `'ERROR'', `'CRITICAL'').
`%(pathname)s'                Full pathname of the source file where the logging
                              call was issued (if available).
`%(filename)s'                Filename portion of pathname.
`%(module)s'                  Module (name portion of filename).
`%(funcName)s'                Name of function containing the logging call.
`%(lineno)d'                  Source line number where the logging call was
                              issued (if available).
`%(created)f'                 Time when the LogRecord was created
                              (as returned by time.time()).
`%(relativeCreated)d'         Time in milliseconds when the LogRecord was
                              created, relative to the time the logging module
                              was loaded.
`%(asctime)s'                 Human-readable time when the LogRecord
                              was created.  By default this is of the form
                              "2003-07-08 16:49:45,896" (the numbers after the
                              comma are millisecond portion of the time).
`%(msecs)d'                   Millisecond portion of the time when the
                              LogRecord was created.
`%(thread)d'                  Thread ID (if available).
`%(threadName)s'              Thread name (if available).
`%(process)d'                 Process ID (if available).
`%(processName)s'             Process name (if available).
`%(message)s'                 The logged message, computed as `msg % args'.

  Changed in version 2.5: _funcName_ was added.

  Changed in version 2.6: _processName_ was added.

 -- Class: logging.Formatter ([fmt[, datefmt]])
     Returns a new instance of the Formatter class. The
     instance is initialized with a format string for the message as a
     whole, as well as a format string for the date/time portion of a
     message. If no _fmt_ is specified, `'%(message)s'' is used. If no
     _datefmt_ is specified, the ISO8601 date format is used.

      -- Method: format (record)
          The record's attribute dictionary is used as the operand to a
          string formatting operation. Returns the resulting string.
          Before formatting the dictionary, a couple of preparatory
          steps are carried out. The _message_ attribute of the record
          is computed using _msg_ % _args_. If the formatting string
          contains `'(asctime)'', see formatTime() is called
          to format the event time. If there is exception information,
          it is formatted using formatException() and
          appended to the message. Note that the formatted exception
          information is cached in attribute _exc_text_. This is useful
          because the exception information can be pickled and sent
          across the wire, but you should be careful if you have more
          than one Formatter subclass which customizes the
          formatting of exception information. In this case, you will
          have to clear the cached value after a formatter has done its
          formatting, so that the next formatter to handle the event
          doesn't use the cached value but recalculates it afresh.

      -- Method: formatTime (record[, datefmt])
          This method should be called from format() by a
          formatter which wants to make use of a formatted time. This
          method can be overridden in formatters to provide for any
          specific requirement, but the basic behavior is as follows:
          if _datefmt_ (a string) is specified, it is used with
          time.strftime() to format the creation time of the
          record. Otherwise, the ISO8601 format is used.  The resulting
          string is returned.

      -- Method: formatException (exc_info)
          Formats the specified exception information (a standard
          exception tuple as returned by sys.exc_info()) as
          a string. This default implementation just uses
          traceback.print_exception(). The resulting string is
          returned.

5.15.7.38 Filter Objects
........................

See Filter.s can be used by `Handler's and `Logger's for more
sophisticated filtering than is provided by levels. The base filter
class only allows events which are below a certain point in the logger
hierarchy. For example, a filter initialized with "A.B" will allow
events logged by loggers "A.B", "A.B.C", "A.B.C.D", "A.B.D" etc. but
not "A.BB", "B.A.B" etc. If initialized with the empty string, all
events are passed.

 -- Class: logging.Filter ([name])
     Returns an instance of the Filter class. If _name_ is
     specified, it names a logger which, together with its children,
     will have its events allowed through the filter. If _name_ is the
     empty string, allows every event.

      -- Method: filter (record)
          Is the specified record to be logged? Returns zero for no,
          nonzero for yes. If deemed appropriate, the record may be
          modified in-place by this method.

  Note that filters attached to handlers are consulted whenever an
event is emitted by the handler, whereas filters attached to loggers
are consulted whenever an event is logged to the handler (using
debug(), info(), etc.) This means that events which
have been generated by descendant loggers will not be filtered by a
logger's filter setting, unless the filter has also been applied to
those descendant loggers.

  You don't actually need to subclass `Filter': you can pass any
instance which has a `filter' method with the same semantics.

5.15.7.39 Other uses for filters
................................

Although filters are used primarily to filter records based on more
sophisticated criteria than levels, they get to see every record which
is processed by the handler or logger they're attached to: this can be
useful if you want to do things like counting how many records were
processed by a particular logger or handler, or adding, changing or
removing attributes in the LogRecord being processed. Obviously
changing the LogRecord needs to be done with some care, but it does
allow the injection of contextual information into logs (see
Using Filters to impart contextual information).

5.15.7.40 LogRecord Objects
...........................

LogRecord instances are created automatically by the
`Logger' every time something is logged, and can be created manually via
makeLogRecord(). (for example, from a pickled event
received over the wire).

 -- Class: logging.LogRecord (name, lvl, pathname, lineno, msg, args,
          exc_info[, func=None])
     Contains all the information pertinent to the event being logged.

     The primary information is passed in msg and
     args, which are combined using `msg % args' to create the
     message field of the record.

      -- Attribute: args
          Tuple of arguments to be used in formatting msg.

      -- Attribute: exc_info
          Exception tuple (à la `sys.exc_info') or `None' if no
          exception information is available.

      -- Attribute: func
          Name of the function of origin (i.e. in which the logging
          call was made).

      -- Attribute: lineno
          Line number in the source file of origin.

      -- Attribute: lvl
          Numeric logging level.

      -- Attribute: message
          Bound to the result of getMessage() when
          Formatter.format(record) is invoked.

      -- Attribute: msg
          User-supplied format string or arbitrary object
          (see Using arbitrary objects as messages) used
          in getMessage().

      -- Attribute: name
          Name of the logger that emitted the record.

      -- Attribute: pathname
          Absolute pathname of the source file of origin.

      -- Method: getMessage ()
          Returns the message for this LogRecord instance
          after merging any user-supplied arguments with the message.
          If the user-supplied message argument to the logging call is
          not a string, see str() is called on it to convert it
          to a string. This allows use of user-defined classes as
          messages, whose `__str__' method can return the actual format
          string to be used.

     Changed in version 2.5: _func_ was added.

5.15.7.41 LoggerAdapter Objects
...............................

New in version 2.6.

  LoggerAdapter instances are used to conveniently pass
contextual information into logging calls. For a usage example , see
the section on adding contextual information to your logging output.

 -- Class: logging.LoggerAdapter (logger, extra)
     Returns an instance of LoggerAdapter initialized with
     an underlying `Logger' instance and a dict-like object.

      -- Method: process (msg, kwargs)
          Modifies the message and/or keyword arguments passed to a
          logging call in order to insert contextual information. This
          implementation takes the object passed as _extra_ to the
          constructor and adds it to _kwargs_ using key 'extra'. The
          return value is a (_msg_, _kwargs_) tuple which has the
          (possibly modified) versions of the arguments passed in.

  In addition to the above, see LoggerAdapter supports all the
logging methods of `Logger', i.e. see debug(), info(),
see warning(), error(), exception(),
see critical() and log(). These methods have
the same signatures as their counterparts in `Logger', so you can use
the two types of instances interchangeably.

  Changed in version 2.7.

  The `isEnabledFor()' method was added to LoggerAdapter.
This method delegates to the underlying logger.

5.15.7.42 Thread Safety
.......................

The logging module is intended to be thread-safe without any special
work needing to be done by its clients. It achieves this though using
threading locks; there is one lock to serialize access to the module's
shared data, and each handler also creates a lock to serialize access
to its underlying I/O.

  If you are implementing asynchronous signal handlers using the
signal.  module, you may not be able to use logging from within
such handlers. This is because lock implementations in the
threading module are not always re-entrant, and so cannot be
invoked from such signal handlers.

5.15.7.43 Integration with the warnings module
..............................................

The captureWarnings() function can be used to integrate
logging.  with the warnings module.

 -- Function: logging.captureWarnings (capture)
     This function is used to turn the capture of warnings by logging
     on and off.

     If _capture_ is `True', warnings issued by the warnings
     module will be redirected to the logging system.
     Specifically, a warning will be formatted using
     warnings.formatwarning() and the resulting string logged to
     a logger named "py.warnings" with a severity of `WARNING'.

     If _capture_ is `False', the redirection of warnings to the
     logging system will stop, and warnings will be redirected to their
     original destinations (i.e. those in effect before
     `captureWarnings(True)' was called).

5.15.7.44 Configuration
.......................

5.15.7.45 Configuration functions
.................................

The following functions configure the logging module. They are located
in the `logging.config' module.  Their use is optional -- you can
configure the logging module using these functions or by making calls
to the main API (defined in logging itself) and defining
handlers which are declared either in logging or
`logging.handlers'.

 -- Function: logging.dictConfig (config)
     Takes the logging configuration from a dictionary.  The contents of
     this dictionary are described in Configuration dictionary schema.
     below.

     If an error is encountered during configuration, this function will
     raise a ValueError, TypeError,
     AttributeError.  or ImportError with a suitably
     descriptive message.  The following is a (possibly incomplete)
     list of conditions which will raise an error:

        * A `level' which is not a string or which is a string not
          corresponding to an actual logging level.

        * A `propagate' value which is not a boolean.

        * An id which does not have a corresponding destination.

        * A non-existent handler id found during an incremental call.

        * An invalid logger name.

        * Inability to resolve to an internal or external object.

     Parsing is performed by the `DictConfigurator' class, whose
     constructor is passed the dictionary used for configuration, and
     has a `configure()' method.  The `logging.config' module has a
     callable attribute `dictConfigClass' which is initially set to
     `DictConfigurator'.  You can replace the value of
     `dictConfigClass' with a suitable implementation of your own.

     dictConfig() calls `dictConfigClass' passing the
     specified dictionary, and then calls the `configure()' method on
     the returned object to put the configuration into effect:

         def dictConfig(config):
             dictConfigClass(config).configure()

     For example, a subclass of `DictConfigurator' could call
     `DictConfigurator.__init__()' in its own __init__(),
     then set up custom prefixes which would be usable in the subsequent
     `configure()' call. `dictConfigClass' would be bound to this new
     subclass, and then dictConfig() could be called
     exactly as in the default, uncustomized state.

 -- Function: logging.fileConfig (fname[, defaults])
     Reads the logging configuration from a ConfigParser.
     -format file named _fname_. This function can be called several
     times from an application, allowing an end user to select from
     various pre-canned configurations (if the developer provides a
     mechanism to present the choices and load the chosen
     configuration). Defaults to be passed to the ConfigParser can be
     specified in the _defaults_ argument.

 -- Function: logging.listen ([port])
     Starts up a socket server on the specified port, and listens for
     new configurations. If no port is specified, the module's default
     `DEFAULT_LOGGING_CONFIG_PORT' is used. Logging configurations will
     be sent as a file suitable for processing by fileConfig().
     Returns a `Thread' instance on which you can call `start()'
     to start the server, and which you can `join()' when appropriate.
     To stop the server, call stopListening().

     To send a configuration to the socket, read in the configuration
     file and send it to the socket as a string of bytes preceded by a
     four-byte length string packed in binary using `struct.pack('>L',
     n)'.

 -- Function: logging.stopListening ()
     Stops the listening server which was created with a call to
     listen().  This is typically called before calling `join()'
     on the return value from listen().

5.15.7.46 Configuration dictionary schema
.........................................

Describing a logging configuration requires listing the various objects
to create and the connections between them; for example, you may create
a handler named "console" and then say that the logger named "startup"
will send its messages to the "console" handler.  These objects aren't
limited to those provided by the logging.  module because
you might write your own formatter or handler class.  The parameters to
these classes may also need to include external objects such as
`sys.stderr'.  The syntax for describing these objects and connections
is defined in Object connections.  below.

5.15.7.47 Dictionary Schema Details
...................................

The dictionary passed to dictConfig() must contain the
following keys:

   * `version' - to be set to an integer value representing the schema
     version.  The only valid value at present is 1, but having this key
     allows the schema to evolve while still preserving backwards
     compatibility.

  All other keys are optional, but if present they will be interpreted
as described below.  In all cases below where a 'configuring dict' is
mentioned, it will be checked for the special `'()'' key to see if a
custom instantiation is required.  If so, the mechanism described in
User-defined objects below is used to create an instance;
otherwise, the context is used to determine what to instantiate.

   * `formatters' - the corresponding value will be a dict in which each
     key is a formatter id and each value is a dict describing how to
     configure the corresponding Formatter instance.

     The configuring dict is searched for keys `format' and `datefmt'
     (with defaults of `None') and these are used to construct a
     logging.Formatter instance.

   * `filters' - the corresponding value will be a dict in which each
     key is a filter id and each value is a dict describing how to
     configure the corresponding Filter instance.

     The configuring dict is searched for the key `name' (defaulting to
     the empty string) and this is used to construct a
     logging.Filter.  instance.

   * `handlers' - the corresponding value will be a dict in which each
     key is a handler id and each value is a dict describing how to
     configure the corresponding Handler instance.

     The configuring dict is searched for the following keys:

        * `class' (mandatory).  This is the fully qualified name of the
          handler class.

        * `level' (optional).  The level of the handler.

        * `formatter' (optional).  The id of the formatter for this
          handler.

        * `filters' (optional).  A list of ids of the filters for this
          handler.

     All _other_ keys are passed through as keyword arguments to the
     handler's constructor.  For example, given the snippet:

         handlers:
           console:
             class : logging.StreamHandler
             formatter: brief
             level   : INFO
             filters: [allow_foo]
             stream  : ext://sys.stdout
           file:
             class : logging.handlers.RotatingFileHandler
             formatter: precise
             filename: logconfig.log
             maxBytes: 1024
             backupCount: 3

     the handler with id `console' is instantiated as a
     logging.StreamHandler, using `sys.stdout' as the underlying
     stream.  The handler with id `file' is instantiated as a
     logging.handlers.RotatingFileHandler with the keyword
     arguments `filename='logconfig.log', maxBytes=1024, backupCount=3'.

   * `loggers' - the corresponding value will be a dict in which each
     key is a logger name and each value is a dict describing how to
     configure the corresponding Logger instance.

     The configuring dict is searched for the following keys:

        * `level' (optional).  The level of the logger.

        * `propagate' (optional).  The propagation setting of the
          logger.

        * `filters' (optional).  A list of ids of the filters for this
          logger.

        * `handlers' (optional).  A list of ids of the handlers for this
          logger.

     The specified loggers will be configured according to the level,
     propagation, filters and handlers specified.

   * `root' - this will be the configuration for the root logger.
     Processing of the configuration will be as for any logger, except
     that the `propagate' setting will not be applicable.

   * `incremental' - whether the configuration is to be interpreted as
     incremental to the existing configuration.  This value defaults to
     `False', which means that the specified configuration replaces the
     existing configuration with the same semantics as used by the
     existing fileConfig(). API.

     If the specified value is `True', the configuration is processed
     as described in the section on Incremental Configuration.

   * `disable_existing_loggers' - whether any existing loggers are to be
     disabled. This setting mirrors the parameter of the same name in
     fileConfig(). If absent, this parameter defaults to
     `True'.  This value is ignored if `incremental' is `True'.

5.15.7.48 Incremental Configuration
...................................

It is difficult to provide complete flexibility for incremental
configuration.  For example, because objects such as filters and
formatters are anonymous, once a configuration is set up, it is not
possible to refer to such anonymous objects when augmenting a
configuration.

  Furthermore, there is not a compelling case for arbitrarily altering
the object graph of loggers, handlers, filters, formatters at run-time,
once a configuration is set up; the verbosity of loggers and handlers
can be controlled just by setting levels (and, in the case of loggers,
propagation flags).  Changing the object graph arbitrarily in a safe
way is problematic in a multi-threaded environment; while not
impossible, the benefits are not worth the complexity it adds to the
implementation.

  Thus, when the `incremental' key of a configuration dict is present
and is `True', the system will completely ignore any `formatters' and
`filters' entries, and process only the `level' settings in the
`handlers' entries, and the `level' and `propagate' settings in the
`loggers' and `root' entries.

  Using a value in the configuration dict lets configurations to be sent
over the wire as pickled dicts to a socket listener. Thus, the logging
verbosity of a long-running application can be altered over time with
no need to stop and restart the application.

5.15.7.49 Object connections
............................

The schema describes a set of logging objects - loggers, handlers,
formatters, filters - which are connected to each other in an object
graph.  Thus, the schema needs to represent connections between the
objects.  For example, say that, once configured, a particular logger
has attached to it a particular handler.  For the purposes of this
discussion, we can say that the logger represents the source, and the
handler the destination, of a connection between the two.  Of course in
the configured objects this is represented by the logger holding a
reference to the handler.  In the configuration dict, this is done by
giving each destination object an id which identifies it unambiguously,
and then using the id in the source object's configuration to indicate
that a connection exists between the source and the destination object
with that id.

  So, for example, consider the following YAML snippet:

    formatters:
      brief:
        # configuration for formatter with id 'brief' goes here
      precise:
        # configuration for formatter with id 'precise' goes here
    handlers:
      h1: #This is an id
       # configuration of handler with id 'h1' goes here
       formatter: brief
      h2: #This is another id
       # configuration of handler with id 'h2' goes here
       formatter: precise
    loggers:
      foo.bar.baz:
        # other configuration for logger 'foo.bar.baz'
        handlers: [h1, h2]

(Note: YAML used here because it's a little more readable than the
equivalent Python source form for the dictionary.)

  The ids for loggers are the logger names which would be used
programmatically to obtain a reference to those loggers, e.g.
`foo.bar.baz'.  The ids for Formatters and Filters can be any string
value (such as `brief', `precise' above) and they are transient, in
that they are only meaningful for processing the configuration
dictionary and used to determine connections between objects, and are
not persisted anywhere when the configuration call is complete.

  The above snippet indicates that logger named `foo.bar.baz' should
have two handlers attached to it, which are described by the handler
ids `h1' and `h2'. The formatter for `h1' is that described by id
`brief', and the formatter for `h2' is that described by id `precise'.

5.15.7.50 User-defined objects
..............................

The schema supports user-defined objects for handlers, filters and
formatters.  (Loggers do not need to have different types for different
instances, so there is no support in this configuration schema for
user-defined logger classes.)

  Objects to be configured are described by dictionaries which detail
their configuration.  In some places, the logging system will be able
to infer from the context how an object is to be instantiated, but when
a user-defined object is to be instantiated, the system will not know
how to do this.  In order to provide complete flexibility for
user-defined object instantiation, the user needs to provide a
'factory' - a callable which is called with a configuration dictionary
and which returns the instantiated object.  This is signalled by an
absolute import path to the factory being made available under the
special key `'()''.  Here's a concrete example:

    formatters:
      brief:
        format: '%(message)s'
      default:
        format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
        datefmt: '%Y-%m-%d %H:%M:%S'
      custom:
          (): my.package.customFormatterFactory
          bar: baz
          spam: 99.9
          answer: 42

The above YAML snippet defines three formatters.  The first, with id
`brief', is a standard logging.Formatter instance with the
specified format string.  The second, with id `default', has a longer
format and also defines the time format explicitly, and will result in
a logging.Formatter initialized with those two format
strings.  Shown in Python source form, the `brief' and `default'
formatters have configuration sub-dictionaries:

    {
      'format' : '%(message)s'
    }

and:

    {
      'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s',
      'datefmt' : '%Y-%m-%d %H:%M:%S'
    }

respectively, and as these dictionaries do not contain the special key
`'()'', the instantiation is inferred from the context: as a result,
standard logging.Formatter instances are created.  The
configuration sub-dictionary for the third formatter, with id `custom',
is:

    {
      '()' : 'my.package.customFormatterFactory',
      'bar' : 'baz',
      'spam' : 99.9,
      'answer' : 42
    }

and this contains the special key `'()'', which means that user-defined
instantiation is wanted.  In this case, the specified factory callable
will be used. If it is an actual callable it will be used directly -
otherwise, if you specify a string (as in the example) the actual
callable will be located using normal import mechanisms.  The callable
will be called with the *remaining* items in the configuration
sub-dictionary as keyword arguments.  In the above example, the
formatter with id `custom' will be assumed to be returned by the call:

    my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42)

The key `'()'' has been used as the special key because it is not a
valid keyword parameter name, and so will not clash with the names of
the keyword arguments used in the call.  The `'()'' also serves as a
mnemonic that the corresponding value is a callable.

5.15.7.51 Access to external objects
....................................

There are times where a configuration needs to refer to objects
external to the configuration, for example `sys.stderr'.  If the
configuration dict is constructed using Python code, this is
straightforward, but a problem arises when the configuration is
provided via a text file (e.g. JSON, YAML).  In a text file, there is
no standard way to distinguish `sys.stderr' from the literal string
`'sys.stderr''.  To facilitate this distinction, the configuration
system looks for certain special prefixes in string values and treat
them specially.  For example, if the literal string
`'ext://sys.stderr'' is provided as a value in the configuration, then
the `ext://' will be stripped off and the remainder of the value
processed using normal import mechanisms.

  The handling of such prefixes is done in a way analogous to protocol
handling: there is a generic mechanism to look for prefixes which match
the regular expression `^(?P<prefix>[a-z]+)://(?P<suffix>.*)$' whereby,
if the `prefix' is recognised, the `suffix' is processed in a
prefix-dependent manner and the result of the processing replaces the
string value.  If the prefix is not recognised, then the string value
will be left as-is.

5.15.7.52 Access to internal objects
....................................

As well as external objects, there is sometimes also a need to refer to
objects in the configuration.  This will be done implicitly by the
configuration system for things that it knows about.  For example, the
string value `'DEBUG'' for a `level' in a logger or handler will
automatically be converted to the value `logging.DEBUG', and the
`handlers', `filters' and `formatter' entries will take an object id
and resolve to the appropriate destination object.

  However, a more generic mechanism is needed for user-defined objects
which are not known to the logging module.  For example,
consider logging.handlers.MemoryHandler, which takes a
`target' argument which is another handler to delegate to. Since the
system already knows about this class, then in the configuration, the
given `target' just needs to be the object id of the relevant target
handler, and the system will resolve to the handler from the id.  If,
however, a user defines a `my.package.MyHandler' which has an
`alternate' handler, the configuration system would not know that the
`alternate' referred to a handler.  To cater for this, a generic
resolution system allows the user to specify:

    handlers:
      file:
        # configuration of file handler goes here

      custom:
        (): my.package.MyHandler
        alternate: cfg://handlers.file

The literal string `'cfg://handlers.file'' will be resolved in an
analogous way to strings with the `ext://' prefix, but looking in the
configuration itself rather than the import namespace.  The mechanism
allows access by dot or by index, in a similar way to that provided by
`str.format'.  Thus, given the following snippet:

    handlers:
      email:
        class: logging.handlers.SMTPHandler
        mailhost: localhost
        fromaddr: my_app@domain.tld
        toaddrs:
          - support_team@domain.tld
          - dev_team@domain.tld
        subject: Houston, we have a problem.

in the configuration, the string `'cfg://handlers'' would resolve to
the dict with key `handlers', the string `'cfg://handlers.email' would
resolve to the dict with key `email' in the `handlers' dict, and so on.
The string `'cfg://handlers.email.toaddrs[1]' would resolve to
`'dev_team.domain.tld'' and the string
`'cfg://handlers.email.toaddrs[0]'' would resolve to the value
`'support_team@domain.tld''. The `subject' value could be accessed
using either `'cfg://handlers.email.subject'' or, equivalently,
`'cfg://handlers.email[subject]''.  The latter form only needs to be
used if the key contains spaces or non-alphanumeric characters.  If an
index value consists only of decimal digits, access will be attempted
using the corresponding integer value, falling back to the string value
if needed.

  Given a string `cfg://handlers.myhandler.mykey.123', this will
resolve to `config_dict['handlers']['myhandler']['mykey']['123']'.  If
the string is specified as `cfg://handlers.myhandler.mykey[123]', the
system will attempt to retrieve the value from
`config_dict['handlers']['myhandler']['mykey'][123]', and fall back to
`config_dict['handlers']['myhandler']['mykey']['123']' if that fails.

5.15.7.53 Configuration file format
...................................

The configuration file format understood by fileConfig() is
based on ConfigParser functionality. The file must contain
sections called `[loggers]', `[handlers]' and `[formatters]' which
identify by name the entities of each type which are defined in the
file. For each such entity, there is a separate section which
identifies how that entity is configured.  Thus, for a logger named
`log01' in the `[loggers]' section, the relevant configuration details
are held in a section `[logger_log01]'. Similarly, a handler called
`hand01' in the `[handlers]' section will have its configuration held
in a section called `[handler_hand01]', while a formatter called
`form01' in the `[formatters]' section will have its configuration
specified in a section called `[formatter_form01]'. The root logger
configuration must be specified in a section called `[logger_root]'.

  Examples of these sections in the file are given below.

    [loggers]
    keys=root,log02,log03,log04,log05,log06,log07

    [handlers]
    keys=hand01,hand02,hand03,hand04,hand05,hand06,hand07,hand08,hand09

    [formatters]
    keys=form01,form02,form03,form04,form05,form06,form07,form08,form09

The root logger must specify a level and a list of handlers. An example
of a root logger section is given below.

    [logger_root]
    level=NOTSET
    handlers=hand01

The `level' entry can be one of `DEBUG, INFO, WARNING, ERROR, CRITICAL'
or `NOTSET'. For the root logger only, `NOTSET' means that all messages
will be logged. Level values are eval().uated in the context
of the `logging' package's namespace.

  The `handlers' entry is a comma-separated list of handler names,
which must appear in the `[handlers]' section. These names must appear
in the `[handlers]' section and have corresponding sections in the
configuration file.

  For loggers other than the root logger, some additional information
is required.  This is illustrated by the following example.

    [logger_parser]
    level=DEBUG
    handlers=hand01
    propagate=1
    qualname=compiler.parser

The `level' and `handlers' entries are interpreted as for the root
logger, except that if a non-root logger's level is specified as
`NOTSET', the system consults loggers higher up the hierarchy to
determine the effective level of the logger. The `propagate' entry is
set to 1 to indicate that messages must propagate to handlers higher up
the logger hierarchy from this logger, or 0 to indicate that messages
are *not* propagated to handlers up the hierarchy. The `qualname' entry
is the hierarchical channel name of the logger, that is to say the name
used by the application to get the logger.

  Sections which specify handler configuration are exemplified by the
following.

    [handler_hand01]
    class=StreamHandler
    level=NOTSET
    formatter=form01
    args=(sys.stdout,)

The `class' entry indicates the handler's class (as determined by
eval().  in the `logging' package's namespace). The `level' is
interpreted as for loggers, and `NOTSET' is taken to mean "log
everything".

  Changed in version 2.6: Added support for resolving the handler's
class as a dotted module and class name.

  The `formatter' entry indicates the key name of the formatter for this
handler. If blank, a default formatter (`logging._defaultFormatter') is
used.  If a name is specified, it must appear in the `[formatters]'
section and have a corresponding section in the configuration file.

  The `args' entry, when eval().uated in the context of the
`logging' package's namespace, is the list of arguments to the
constructor for the handler class. Refer to the constructors for the
relevant handlers, or to the examples below, to see how typical entries
are constructed.

    [handler_hand02]
    class=FileHandler
    level=DEBUG
    formatter=form02
    args=('python.log', 'w')

    [handler_hand03]
    class=handlers.SocketHandler
    level=INFO
    formatter=form03
    args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

    [handler_hand04]
    class=handlers.DatagramHandler
    level=WARN
    formatter=form04
    args=('localhost', handlers.DEFAULT_UDP_LOGGING_PORT)

    [handler_hand05]
    class=handlers.SysLogHandler
    level=ERROR
    formatter=form05
    args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER)

    [handler_hand06]
    class=handlers.NTEventLogHandler
    level=CRITICAL
    formatter=form06
    args=('Python Application', '', 'Application')

    [handler_hand07]
    class=handlers.SMTPHandler
    level=WARN
    formatter=form07
    args=('localhost', 'from@abc', ['user1@abc', 'user2@xyz'], 'Logger Subject')

    [handler_hand08]
    class=handlers.MemoryHandler
    level=NOTSET
    formatter=form08
    target=
    args=(10, ERROR)

    [handler_hand09]
    class=handlers.HTTPHandler
    level=NOTSET
    formatter=form09
    args=('localhost:9022', '/log', 'GET')

Sections which specify formatter configuration are typified by the
following.

    [formatter_form01]
    format=F1 %(asctime)s %(levelname)s %(message)s
    datefmt=
    class=logging.Formatter

The `format' entry is the overall format string, and the `datefmt'
entry is the `strftime()'-compatible date/time format string.  If
empty, the package substitutes ISO8601 format date/times, which is
almost equivalent to specifying the date format string `"%Y-%m-%d
%H:%M:%S"'.  The ISO8601 format also specifies milliseconds, which are
appended to the result of using the above format string, with a comma
separator.  An example time in ISO8601 format is `2003-01-23
00:29:50,411'.

  The `class' entry is optional.  It indicates the name of the
formatter's class (as a dotted module and class name.)  This option is
useful for instantiating a Formatter subclass.  Subclasses
of Formatter can present exception tracebacks in an
expanded or condensed format.

5.15.7.54 Configuration server example
......................................

Here is an example of a module using the logging configuration server:

    import logging
    import logging.config
    import time
    import os

    # read initial config file
    logging.config.fileConfig("logging.conf")

    # create and start listener on port 9999
    t = logging.config.listen(9999)
    t.start()

    logger = logging.getLogger("simpleExample")

    try:
        # loop through logging calls to see the difference
        # new configurations make, until Ctrl+C is pressed
        while True:
            logger.debug("debug message")
            logger.info("info message")
            logger.warn("warn message")
            logger.error("error message")
            logger.critical("critical message")
            time.sleep(5)
    except KeyboardInterrupt:
        # cleanup
        logging.config.stopListening()
        t.join()

And here is a script that takes a filename and sends that file to the
server, properly preceded with the binary-encoded length, as the new
logging configuration:

    #!/usr/bin/env python
    import socket, sys, struct

    data_to_send = open(sys.argv[1], "r").read()

    HOST = 'localhost'
    PORT = 9999
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print "connecting..."
    s.connect((HOST, PORT))
    print "sending config..."
    s.send(struct.pack(">L", len(data_to_send)))
    s.send(data_to_send)
    s.close()
    print "complete"

5.15.7.55 More examples
.......................

5.15.7.56 Multiple handlers and formatters
..........................................

Loggers are plain Python objects.  The `addHandler()' method has no
minimum or maximum quota for the number of handlers you may add.
Sometimes it will be beneficial for an application to log all messages
of all severities to a text file while simultaneously logging errors or
above to the console.  To set this up, simply configure the appropriate
handlers.  The logging calls in the application code will remain
unchanged.  Here is a slight modification to the previous simple
module-based configuration example:

    import logging

    logger = logging.getLogger("simple_example")
    logger.setLevel(logging.DEBUG)
    # create file handler which logs even debug messages
    fh = logging.FileHandler("spam.log")
    fh.setLevel(logging.DEBUG)
    # create console handler with a higher log level
    ch = logging.StreamHandler()
    ch.setLevel(logging.ERROR)
    # create formatter and add it to the handlers
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    ch.setFormatter(formatter)
    fh.setFormatter(formatter)
    # add the handlers to logger
    logger.addHandler(ch)
    logger.addHandler(fh)

    # "application" code
    logger.debug("debug message")
    logger.info("info message")
    logger.warn("warn message")
    logger.error("error message")
    logger.critical("critical message")

Notice that the "application" code does not care about multiple
handlers.  All that changed was the addition and configuration of a new
handler named _fh_.

  The ability to create new handlers with higher- or lower-severity
filters can be very helpful when writing and testing an application.
Instead of using many `print' statements for debugging, use
`logger.debug': Unlike the print statements, which you will have to
delete or comment out later, the logger.debug statements can remain
intact in the source code and remain dormant until you need them again.
At that time, the only change that needs to happen is to modify the
severity level of the logger and/or handler to debug.

5.15.7.57 Using logging in multiple modules
...........................................

It was mentioned above that multiple calls to
`logging.getLogger('someLogger')' return a reference to the same logger
object.  This is true not only within the same module, but also across
modules as long as it is in the same Python interpreter process.  It is
true for references to the same object; additionally, application code
can define and configure a parent logger in one module and create (but
not configure) a child logger in a separate module, and all logger
calls to the child will pass up to the parent.  Here is a main module:

    import logging
    import auxiliary_module

    # create logger with "spam_application"
    logger = logging.getLogger("spam_application")
    logger.setLevel(logging.DEBUG)
    # create file handler which logs even debug messages
    fh = logging.FileHandler("spam.log")
    fh.setLevel(logging.DEBUG)
    # create console handler with a higher log level
    ch = logging.StreamHandler()
    ch.setLevel(logging.ERROR)
    # create formatter and add it to the handlers
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)
    # add the handlers to the logger
    logger.addHandler(fh)
    logger.addHandler(ch)

    logger.info("creating an instance of auxiliary_module.Auxiliary")
    a = auxiliary_module.Auxiliary()
    logger.info("created an instance of auxiliary_module.Auxiliary")
    logger.info("calling auxiliary_module.Auxiliary.do_something")
    a.do_something()
    logger.info("finished auxiliary_module.Auxiliary.do_something")
    logger.info("calling auxiliary_module.some_function()")
    auxiliary_module.some_function()
    logger.info("done with auxiliary_module.some_function()")

Here is the auxiliary module:

    import logging

    # create logger
    module_logger = logging.getLogger("spam_application.auxiliary")

    class Auxiliary:
        def __init__(self):
            self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary")
            self.logger.info("creating an instance of Auxiliary")
        def do_something(self):
            self.logger.info("doing something")
            a = 1 + 1
            self.logger.info("done doing something")

    def some_function():
        module_logger.info("received a call to \"some_function\"")

The output looks like this:

    2005-03-23 23:47:11,663 - spam_application - INFO -
       creating an instance of auxiliary_module.Auxiliary
    2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO -
       creating an instance of Auxiliary
    2005-03-23 23:47:11,665 - spam_application - INFO -
       created an instance of auxiliary_module.Auxiliary
    2005-03-23 23:47:11,668 - spam_application - INFO -
       calling auxiliary_module.Auxiliary.do_something
    2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO -
       doing something
    2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO -
       done doing something
    2005-03-23 23:47:11,670 - spam_application - INFO -
       finished auxiliary_module.Auxiliary.do_something
    2005-03-23 23:47:11,671 - spam_application - INFO -
       calling auxiliary_module.some_function()
    2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO -
       received a call to "some_function"
    2005-03-23 23:47:11,673 - spam_application - INFO -
       done with auxiliary_module.some_function()

5.15.8 `getpass' -- Portable password input
-------------------------------------------

The getpass module provides two functions:

 -- Function: getpass.getpass ([prompt[, stream]])
     Prompt the user for a password without echoing.  The user is
     prompted using the string _prompt_, which defaults to `'Password:
     ''. On Unix, the prompt is written to the file-like object
     _stream_.  _stream_ defaults to the controlling terminal
     (/dev/tty) or if that is unavailable to `sys.stderr' (this
     argument is ignored on Windows).

     If echo free input is unavailable getpass() falls back to printing
     a warning message to _stream_ and reading from `sys.stdin' and
     issuing a GetPassWarning.

     Availability: Macintosh, Unix, Windows.

     Changed in version 2.5: The _stream_ parameter was added.

     Changed in version 2.6: On Unix it defaults to using /dev/tty
     before falling back to `sys.stdin' and `sys.stderr'.

          Note: If you call getpass from within IDLE, the input may be
          done in the terminal you launched IDLE from rather than the
          idle window itself.

 -- Exception: getpass.GetPassWarning
     A UserWarning subclass issued when password input may
     be echoed.

 -- Function: getpass.getuser ()
     Return the "login name" of the user. Availability: Unix, Windows.

     This function checks the environment variables `LOGNAME', `USER', `LNAME'
     and `USERNAME', in order, and returns the value of the first one
     which is set to a non-empty string.  If none are set, the login
     name from the password database is returned on systems which
     support the pwd module, otherwise, an exception is
     raised.

5.15.9 `curses' -- Terminal handling for character-cell displays
----------------------------------------------------------------

Changed in version 1.6: Added support for the `ncurses' library and
converted to a package.

  The curses module provides an interface to the curses
library, the de-facto standard for portable advanced terminal handling.

  While curses is most widely used in the Unix environment, versions
are available for DOS, OS/2, and possibly other systems as well.  This
extension module is designed to match the API of ncurses, an
open-source curses library hosted on Linux and the BSD variants of Unix.

     Note: Since version 5.4, the ncurses library decides how to
     interpret non-ASCII data using the `nl_langinfo' function.  That
     means that you have to call locale.setlocale() in the
     application and encode Unicode strings using one of the system's
     available encodings.  This example uses the system's default
     encoding:

         import locale
         locale.setlocale(locale.LC_ALL, '')
         code = locale.getpreferredencoding()

     Then use _code_ as the encoding for str.encode() calls.

Module curses.ascii.
     Utilities for working with ASCII characters, regardless of your
     locale settings.

Module curses.panel.
     A panel stack extension that adds depth to  curses windows.

Module curses.textpad.
     Editable text widget for curses supporting  *Emacs*-like bindings.

Module curses.wrapper.
     Convenience function to ensure proper terminal setup and resetting
     on application entry and exit.

See Curses Programming with Python.
     Tutorial material on using curses with Python, by Andrew Kuchling
     and Eric Raymond.

  The `Demo/curses/' directory in the Python source distribution
contains some example programs using the curses bindings provided by
this module.

5.15.9.1 Functions
..................

The module curses defines the following exception:

 -- Exception: curses.error
     Exception raised when a curses library function returns an error.

     Note: Whenever _x_ or _y_ arguments to a function or a method are
     optional, they default to the current cursor location. Whenever
     _attr_ is optional, it defaults to `A_NORMAL'.

  The module curses defines the following functions:

 -- Function: curses.baudrate ()
     Returns the output speed of the terminal in bits per second.  On
     software terminal emulators it will have a fixed high value.
     Included for historical reasons; in former times, it was used to
     write output loops for time delays and occasionally to change
     interfaces depending on the line speed.

 -- Function: curses.beep ()
     Emit a short attention sound.

 -- Function: curses.can_change_color ()
     Returns true or false, depending on whether the programmer can
     change the colors displayed by the terminal.

 -- Function: curses.cbreak ()
     Enter cbreak mode.  In cbreak mode (sometimes called "rare" mode)
     normal tty line buffering is turned off and characters are
     available to be read one by one.  However, unlike raw mode,
     special characters (interrupt, quit, suspend, and flow control)
     retain their effects on the tty driver and calling program.
     Calling first raw() then cbreak() leaves
     the terminal in cbreak mode.

 -- Function: curses.color_content (color_number)
     Returns the intensity of the red, green, and blue (RGB) components
     in the color _color_number_, which must be between `0' and
     `COLORS'.  A 3-tuple is returned, containing the R,G,B values for
     the given color, which will be between `0' (no component) and
     `1000' (maximum amount of component).

 -- Function: curses.color_pair (color_number)
     Returns the attribute value for displaying text in the specified
     color.  This attribute value can be combined with `A_STANDOUT',
     `A_REVERSE', and the other `A_*' attributes.  pair_number()
     is the counterpart to this function.

 -- Function: curses.curs_set (visibility)
     Sets the cursor state.  _visibility_ can be set to 0, 1, or 2, for
     invisible, normal, or very visible.  If the terminal supports the
     visibility requested, the previous cursor state is returned;
     otherwise, an exception is raised.  On many terminals, the
     "visible" mode is an underline cursor and the "very visible" mode
     is a block cursor.

 -- Function: curses.def_prog_mode ()
     Saves the current terminal mode as the "program" mode, the mode
     when the running program is using curses.  (Its counterpart is the
     "shell" mode, for when the program is not in curses.)  Subsequent
     calls to reset_prog_mode() will restore this mode.

 -- Function: curses.def_shell_mode ()
     Saves the current terminal mode as the "shell" mode, the mode when
     the running program is not using curses.  (Its counterpart is the
     "program" mode, when the program is using curses capabilities.)
     Subsequent calls to reset_shell_mode() will restore
     this mode.

 -- Function: curses.delay_output (ms)
     Inserts an _ms_ millisecond pause in output.

 -- Function: curses.doupdate ()
     Update the physical screen.  The curses library keeps two data
     structures, one representing the current physical screen contents
     and a virtual screen representing the desired next state.  The
     doupdate() ground updates the physical screen to
     match the virtual screen.

     The virtual screen may be updated by a `noutrefresh()' call after
     write operations such as `addstr()' have been performed on a
     window.  The normal `refresh()' call is simply `noutrefresh()'
     followed by doupdate().; if you have to update
     multiple windows, you can speed performance and perhaps reduce
     screen flicker by issuing `noutrefresh()' calls on all windows,
     followed by a single doupdate().

 -- Function: curses.echo ()
     Enter echo mode.  In echo mode, each character input is echoed to
     the screen as it is entered.

 -- Function: curses.endwin ()
     De-initialize the library, and return terminal to normal status.

 -- Function: curses.erasechar ()
     Returns the user's current erase character.  Under Unix operating
     systems this is a property of the controlling tty of the curses
     program, and is not set by the curses library itself.

 -- Function: curses.filter ()
     The filter() routine, if used, must be called before
     initscr() is called.  The effect is that, during
     those calls, LINES is set to 1; the capabilities clear, cup, cud,
     cud1, cuu1, cuu, vpa are disabled; and the home string is set to
     the value of cr. The effect is that the cursor is confined to the
     current line, and so are screen updates.  This may be used for
     enabling character-at-a-time  line editing without touching the
     rest of the screen.

 -- Function: curses.flash ()
     Flash the screen.  That is, change it to reverse-video and then
     change it back in a short interval.  Some people prefer such as
     'visible bell' to the audible attention signal produced by
     beep().

 -- Function: curses.flushinp ()
     Flush all input buffers.  This throws away any  typeahead  that
     has been typed by the user and has not yet been processed by the
     program.

 -- Function: curses.getmouse ()
     After `getch()' returns `KEY_MOUSE' to signal a mouse event, this
     method should be call to retrieve the queued mouse event,
     represented as a 5-tuple `(id, x, y, z, bstate)'. _id_ is an ID
     value used to distinguish multiple devices, and _x_, _y_, _z_ are
     the event's coordinates.  (_z_ is currently unused.).  _bstate_ is
     an integer value whose bits will be set to indicate the type of
     event, and will be the bitwise OR of one or more of the following
     constants, where _n_ is the button number from 1 to 4:
     `BUTTONn_PRESSED', `BUTTONn_RELEASED', `BUTTONn_CLICKED',
     `BUTTONn_DOUBLE_CLICKED', `BUTTONn_TRIPLE_CLICKED',
     `BUTTON_SHIFT', `BUTTON_CTRL', `BUTTON_ALT'.

 -- Function: curses.getsyx ()
     Returns the current coordinates of the virtual screen cursor in y
     and x.  If leaveok is currently true, then -1,-1 is returned.

 -- Function: curses.getwin (file)
     Reads window related data stored in the file by an earlier
     `putwin()' call.  The routine then creates and initializes a new
     window using that data, returning the new window object.

 -- Function: curses.has_colors ()
     Returns true if the terminal can display colors; otherwise, it
     returns false.

 -- Function: curses.has_ic ()
     Returns true if the terminal has insert- and delete- character
     capabilities.  This function is included for historical reasons
     only, as all modern software terminal emulators have such
     capabilities.

 -- Function: curses.has_il ()
     Returns true if the terminal has insert- and delete-line
     capabilities,  or  can simulate  them  using scrolling regions.
     This function is included for historical reasons only, as all
     modern software terminal emulators have such capabilities.

 -- Function: curses.has_key (ch)
     Takes a key value _ch_, and returns true if the current terminal
     type recognizes a key with that value.

 -- Function: curses.halfdelay (tenths)
     Used for half-delay mode, which is similar to cbreak mode in that
     characters typed by the user are immediately available to the
     program. However, after blocking for _tenths_ tenths of seconds,
     an exception is raised if nothing has been typed.  The value of
     _tenths_ must be a number between 1 and 255.  Use
     nocbreak() to leave half-delay mode.

 -- Function: curses.init_color (color_number, r, g, b)
     Changes the definition of a color, taking the number of the color
     to be changed followed by three RGB values (for the amounts of
     red, green, and blue components).  The value of _color_number_
     must be between `0' and `COLORS'.  Each of _r_, _g_, _b_, must be
     a value between `0' and `1000'.  When init_color() is
     used, all occurrences of that color on the screen immediately
     change to the new definition.  This function is a no-op on most
     terminals; it is active only if can_change_color().
     returns `1'.

 -- Function: curses.init_pair (pair_number, fg, bg)
     Changes the definition of a color-pair.  It takes three arguments:
     the number of the color-pair to be changed, the foreground color
     number, and the background color number.  The value of
     _pair_number_ must be between `1' and `COLOR_PAIRS - 1' (the `0'
     color pair is wired to white on black and cannot be changed).  The
     value of _fg_ and _bg_ arguments must be between `0' and `COLORS'.
     If the color-pair was previously initialized, the screen is
     refreshed and all occurrences of that color-pair are changed to
     the new definition.

 -- Function: curses.initscr ()
     Initialize the library. Returns a `WindowObject' which represents
     the whole screen.

          Note: If there is an error opening the terminal, the
          underlying curses library may cause the interpreter to exit.

 -- Function: curses.isendwin ()
     Returns true if endwin() has been called (that is,
     the  curses library has been deinitialized).

 -- Function: curses.keyname (k)
     Return the name of the key numbered _k_.  The name of a key
     generating printable ASCII character is the key's character.  The
     name of a control-key combination is a two-character string
     consisting of a caret followed by the corresponding printable
     ASCII character.  The name of an alt-key combination (128-255) is a
     string consisting of the prefix 'M-' followed by the name of the
     corresponding ASCII character.

 -- Function: curses.killchar ()
     Returns the user's current line kill character. Under Unix
     operating systems this is a property of the controlling tty of the
     curses program, and is not set by the curses library itself.

 -- Function: curses.longname ()
     Returns a string containing the terminfo long name field
     describing the current terminal.  The maximum length of a verbose
     description is 128 characters.  It is defined only after the call
     to initscr().

 -- Function: curses.meta (yes)
     If _yes_ is 1, allow 8-bit characters to be input. If _yes_ is 0,
     allow only 7-bit chars.

 -- Function: curses.mouseinterval (interval)
     Sets the maximum time in milliseconds that can elapse between
     press and release events in order for them to be recognized as a
     click, and returns the previous interval value.  The default value
     is 200 msec, or one fifth of a second.

 -- Function: curses.mousemask (mousemask)
     Sets the mouse events to be reported, and returns a tuple
     `(availmask, oldmask)'.   _availmask_ indicates which of the
     specified mouse events can be reported; on complete failure it
     returns 0.  _oldmask_ is the previous value of the given window's
     mouse event mask.  If this function is never called, no mouse
     events are ever reported.

 -- Function: curses.napms (ms)
     Sleep for _ms_ milliseconds.

 -- Function: curses.newpad (nlines, ncols)
     Creates and returns a pointer to a new pad data structure with the
     given number of lines and columns.  A pad is returned as a window
     object.

     A pad is like a window, except that it is not restricted by the
     screen size, and is not necessarily associated with a particular
     part of the screen.  Pads can be used when a large window is
     needed, and only a part of the window will be on the screen at one
     time.  Automatic refreshes of pads (such as from scrolling or
     echoing of input) do not occur.  The `refresh()' and
     `noutrefresh()' methods of a pad require 6 arguments to specify
     the part of the pad to be displayed and the location on the screen
     to be used for the display. The arguments are pminrow, pmincol,
     sminrow, smincol, smaxrow, smaxcol; the p arguments refer to the
     upper left corner of the pad region to be displayed and the s
     arguments define a clipping box on the screen within which the pad
     region is to be displayed.

 -- Function: curses.newwin ([nlines, ncols], begin_y, begin_x)
     Return a new window, whose left-upper corner is at  `(begin_y,
     begin_x)', and whose height/width is  _nlines_/_ncols_.

     By default, the window will extend from the  specified position to
     the lower right corner of the screen.

 -- Function: curses.nl ()
     Enter newline mode.  This mode translates the return key into
     newline on input, and translates newline into return and line-feed
     on output. Newline mode is initially on.

 -- Function: curses.nocbreak ()
     Leave cbreak mode.  Return to normal "cooked" mode with line
     buffering.

 -- Function: curses.noecho ()
     Leave echo mode.  Echoing of input characters is turned off.

 -- Function: curses.nonl ()
     Leave newline mode.  Disable translation of return into newline on
     input, and disable low-level translation of newline into
     newline/return on output (but this does not change the behavior of
     `addch('\n')', which always does the equivalent of return and line
     feed on the virtual screen).  With translation off, curses can
     sometimes speed up vertical motion a little; also, it will be able
     to detect the return key on input.

 -- Function: curses.noqiflush ()
     When the noqiflush routine is used, normal flush of input and
     output queues associated with the INTR, QUIT and SUSP characters
     will not be done.  You may want to call noqiflush().
     in a signal handler if you want output to continue as though the
     interrupt had not occurred, after the handler exits.

 -- Function: curses.noraw ()
     Leave raw mode. Return to normal "cooked" mode with line buffering.

 -- Function: curses.pair_content (pair_number)
     Returns a tuple `(fg, bg)' containing the colors for the requested
     color pair.  The value of _pair_number_ must be between `1' and
     `COLOR_PAIRS - 1'.

 -- Function: curses.pair_number (attr)
     Returns the number of the color-pair set by the attribute value
     _attr_.  color_pair() is the counterpart to this
     function.

 -- Function: curses.putp (string)
     Equivalent to `tputs(str, 1, putchar)'; emits the value of a
     specified terminfo capability for the current terminal.  Note that
     the output of putp always goes to standard output.

 -- Function: curses.qiflush ([flag])
     If _flag_ is false, the effect is the same as calling
     noqiflush(). If _flag_ is true, or no argument is provided,
     the queues will be flushed when these control characters are read.

 -- Function: curses.raw ()
     Enter raw mode.  In raw mode, normal line buffering and
     processing of interrupt, quit, suspend, and flow control keys are
     turned off; characters are presented to curses input functions one
     by one.

 -- Function: curses.reset_prog_mode ()
     Restores the  terminal  to "program" mode, as previously saved  by
     def_prog_mode().

 -- Function: curses.reset_shell_mode ()
     Restores the  terminal  to "shell" mode, as previously saved  by
     def_shell_mode().

 -- Function: curses.setsyx (y, x)
     Sets the virtual screen cursor to _y_, _x_. If _y_ and _x_ are
     both -1, then leaveok is set.

 -- Function: curses.setupterm ([termstr, fd])
     Initializes the terminal.  _termstr_ is a string giving the
     terminal name; if omitted, the value of the TERM environment
     variable will be used.  _fd_ is the file descriptor to which any
     initialization sequences will be sent; if not supplied, the file
     descriptor for `sys.stdout' will be used.

 -- Function: curses.start_color ()
     Must be called if the programmer wants to use colors, and before
     any other color manipulation routine is called.  It is good
     practice to call this routine right after initscr().

     start_color() initializes eight basic colors (black,
     red,  green, yellow, blue, magenta, cyan, and white), and two
     global variables in the curses.  module, `COLORS' and
     `COLOR_PAIRS', containing the maximum number of colors and
     color-pairs the terminal can support.  It also restores the colors
     on the terminal to the values they had when the terminal was just
     turned on.

 -- Function: curses.termattrs ()
     Returns a logical OR of all video attributes supported by the
     terminal.  This information is useful when a curses program needs
     complete control over the appearance of the screen.

 -- Function: curses.termname ()
     Returns the value of the environment variable TERM, truncated to
     14 characters.

 -- Function: curses.tigetflag (capname)
     Returns the value of the Boolean capability corresponding to the
     terminfo capability name _capname_.  The value `-1' is returned if
     _capname_ is not a Boolean capability, or `0' if it is canceled or
     absent from the terminal description.

 -- Function: curses.tigetnum (capname)
     Returns the value of the numeric capability corresponding to the
     terminfo capability name _capname_.  The value `-2' is returned if
     _capname_ is not a numeric capability, or `-1' if it is canceled
     or absent from the terminal description.

 -- Function: curses.tigetstr (capname)
     Returns the value of the string capability corresponding to the
     terminfo capability name _capname_.  `None' is returned if
     _capname_ is not a string capability, or is canceled or absent
     from the terminal description.

 -- Function: curses.tparm (str[, ...])
     Instantiates the string _str_ with the supplied parameters, where
     _str_ should be a parameterized string obtained from the terminfo
     database.  E.g.  `tparm(tigetstr("cup"), 5, 3)' could result in
     `'\033[6;4H'', the exact result depending on terminal type.

 -- Function: curses.typeahead (fd)
     Specifies that the file descriptor _fd_ be used for typeahead
     checking.  If _fd_ is `-1', then no typeahead checking is done.

     The curses library does "line-breakout optimization" by looking
     for typeahead periodically while updating the screen.  If input is
     found, and it is coming from a tty, the current update is
     postponed until refresh or doupdate is called again, allowing
     faster response to commands typed in advance. This function allows
     specifying a different file descriptor for typeahead checking.

 -- Function: curses.unctrl (ch)
     Returns a string which is a printable representation of the
     character _ch_.  Control characters are displayed as a caret
     followed by the character, for example as `^C'. Printing
     characters are left as they are.

 -- Function: curses.ungetch (ch)
     Push _ch_ so the next `getch()' will return it.

          Note: Only one _ch_ can be pushed before `getch()' is called.

 -- Function: curses.ungetmouse (id, x, y, z, bstate)
     Push a `KEY_MOUSE' event onto the input queue, associating the
     given state data with it.

 -- Function: curses.use_env (flag)
     If used, this function should be called before initscr()
     or newterm are called.  When _flag_ is false, the values of
     lines and columns specified in the terminfo database will be used,
     even if environment variables `LINES' and `COLUMNS' (used by
     default) are set, or if curses is running in a window (in which
     case default behavior would be to use the window size if `LINES'
     and `COLUMNS' are not set).

 -- Function: curses.use_default_colors ()
     Allow use of default values for colors on terminals supporting
     this feature. Use this to support transparency in your
     application.  The default color is assigned to the color number
     -1. After calling this function,  `init_pair(x, curses.COLOR_RED,
     -1)' initializes, for instance, color pair _x_ to a red foreground
     color on the default background.

5.15.9.2 Window Objects
.......................

Window objects, as returned by initscr() and
newwin() above, have the following methods:

 -- Method: window.addch ([y, x], ch[, attr])
          Note: A _character_ means a C character (an ASCII code),
          rather then a Python character (a string of length 1). (This
          note is true whenever the documentation mentions a
          character.) The built-in ord() is handy for
          conveying strings to codes.

     Paint character _ch_ at `(y, x)' with attributes _attr_,
     overwriting any character previously painter at that location.  By
     default, the character position and attributes are the current
     settings for the window object.

 -- Method: window.addnstr ([y, x], str, n[, attr])
     Paint at most _n_ characters of the  string _str_ at `(y, x)' with
     attributes _attr_, overwriting anything previously on the display.

 -- Method: window.addstr ([y, x], str[, attr])
     Paint the string _str_ at `(y, x)' with attributes _attr_,
     overwriting anything previously on the display.

 -- Method: window.attroff (attr)
     Remove attribute _attr_ from the "background" set applied to all
     writes to the current window.

 -- Method: window.attron (attr)
     Add attribute _attr_ from the "background" set applied to all
     writes to the current window.

 -- Method: window.attrset (attr)
     Set the "background" set of attributes to _attr_.  This set is
     initially 0 (no attributes).

 -- Method: window.bkgd (ch[, attr])
     Sets the background property of the window to the character _ch_,
     with attributes _attr_.  The change is then applied to every
     character position in that window:

        * The attribute of every character in the window  is changed to
          the new background attribute.

        * Wherever  the  former background character appears, it is
          changed to the new background character.

 -- Method: window.bkgdset (ch[, attr])
     Sets the window's background.  A window's background consists of a
     character and any combination of attributes.  The attribute part
     of the background is combined (OR'ed) with all non-blank
     characters that are written into the window.  Both the character
     and attribute parts of the background are combined with the blank
     characters.  The background becomes a property of the character
     and moves with the character through any scrolling and
     insert/delete line/character operations.

 -- Method: window.border ([ls[, rs[, ts[, bs[, tl[, tr[, bl[,
          br]]]]]]]])
     Draw a border around the edges of the window. Each parameter
     specifies  the character to use for a specific part of the border;
     see the table below for more details.  The characters can be
     specified as integers or as one-character strings.

          Note: A `0' value for any parameter will cause the default
          character to be used for that parameter.  Keyword parameters
          can _not_ be used.  The defaults are listed in this table:

     Parameter       Description               Default value
     ----------------------------------------------------------------------
     _ls_            Left side                 `ACS_VLINE'
     _rs_            Right side                `ACS_VLINE'
     _ts_            Top                       `ACS_HLINE'
     _bs_            Bottom                    `ACS_HLINE'
     _tl_            Upper-left corner         `ACS_ULCORNER'
     _tr_            Upper-right corner        `ACS_URCORNER'
     _bl_            Bottom-left corner        `ACS_LLCORNER'
     _br_            Bottom-right corner       `ACS_LRCORNER'

 -- Method: window.box ([vertch, horch])
     Similar to border(), but both _ls_ and _rs_ are
     _vertch_ and both _ts_ and bs are _horch_.  The default corner
     characters are always used by this function.

 -- Method: window.chgat ([y, x, ] [num,] attr)
     Sets the attributes of _num_ characters at the current cursor
     position, or at position `(y, x)' if supplied. If no value of
     _num_ is given or _num_ = -1, the attribute will  be set on all
     the characters to the end of the line.  This function does not
     move the cursor. The changed line will be touched using the
     touchline() method so that the contents will be redisplayed
     by the next window refresh.

 -- Method: window.clear ()
     Like erase(), but also causes the whole window to be
     repainted upon next call to refresh().

 -- Method: window.clearok (yes)
     If _yes_ is 1, the next call to refresh() will clear
     the window completely.

 -- Method: window.clrtobot ()
     Erase from cursor to the end of the window: all lines below the
     cursor are deleted, and then the equivalent of clrtoeol()
     is performed.

 -- Method: window.clrtoeol ()
     Erase from cursor to the end of the line.

 -- Method: window.cursyncup ()
     Updates the current cursor position of all the ancestors of the
     window to reflect the current cursor position of the window.

 -- Method: window.delch ([y, x])
     Delete any character at `(y, x)'.

 -- Method: window.deleteln ()
     Delete the line under the cursor. All following lines are moved up
     by 1 line.

 -- Method: window.derwin ([nlines, ncols], begin_y, begin_x)
     An abbreviation for "derive window", see derwin() is the
     same as calling subwin(), except that _begin_y_ and
     _begin_x_ are relative to the origin of the window, rather than
     relative to the entire screen.  Returns a window object for the
     derived window.

 -- Method: window.echochar (ch[, attr])
     Add character _ch_ with attribute _attr_, and immediately  call
     refresh().  on the window.

 -- Method: window.enclose (y, x)
     Tests whether the given pair of screen-relative character-cell
     coordinates are enclosed by the given window, returning true or
     false.  It is useful for determining what subset of the screen
     windows enclose the location of a mouse event.

 -- Method: window.erase ()
     Clear the window.

 -- Method: window.getbegyx ()
     Return a tuple `(y, x)' of co-ordinates of upper-left corner.

 -- Method: window.getch ([y, x])
     Get a character. Note that the integer returned does _not_ have to
     be in ASCII range: function keys, keypad keys and so on return
     numbers higher than 256. In no-delay mode, -1 is returned if there
     is no input, else getch() waits until a key is
     pressed.

 -- Method: window.getkey ([y, x])
     Get a character, returning a string instead of an integer, as
     getch().  does. Function keys, keypad keys and so on
     return a multibyte string containing the key name.  In no-delay
     mode, an exception is raised if there is no input.

 -- Method: window.getmaxyx ()
     Return a tuple `(y, x)' of the height and width of the window.

 -- Method: window.getparyx ()
     Returns the beginning coordinates of this window relative to its
     parent window into two integer variables y and x.  Returns `-1,-1'
     if this window has no parent.

 -- Method: window.getstr ([y, x])
     Read a string from the user, with primitive line editing capacity.

 -- Method: window.getyx ()
     Return a tuple `(y, x)' of current cursor position  relative to
     the window's upper-left corner.

 -- Method: window.hline ([y, x], ch, n)
     Display a horizontal line starting at `(y, x)' with length _n_
     consisting of the character _ch_.

 -- Method: window.idcok (flag)
     If _flag_ is false, curses no longer considers using the hardware
     insert/delete character feature of the terminal; if _flag_ is
     true, use of character insertion and deletion is enabled.  When
     curses is first initialized, use of character insert/delete is
     enabled by default.

 -- Method: window.idlok (yes)
     If called with _yes_ equal to 1, see curses will try and
     use hardware line editing facilities. Otherwise, line
     insertion/deletion are disabled.

 -- Method: window.immedok (flag)
     If _flag_ is true, any change in the window image automatically
     causes the window to be refreshed; you no longer have to call
     refresh() yourself.  However, it may degrade
     performance considerably, due to repeated calls to wrefresh.  This
     option is disabled by default.

 -- Method: window.inch ([y, x])
     Return the character at the given position in the window. The
     bottom 8 bits are the character proper, and upper bits are the
     attributes.

 -- Method: window.insch ([y, x], ch[, attr])
     Paint character _ch_ at `(y, x)' with attributes _attr_, moving
     the line from position _x_ right by one character.

 -- Method: window.insdelln (nlines)
     Inserts _nlines_ lines into the specified window above the current
     line.  The _nlines_ bottom lines are lost.  For negative _nlines_,
     delete _nlines_ lines starting with the one under the cursor, and
     move the remaining lines up.  The bottom _nlines_ lines are
     cleared.  The current cursor position remains the same.

 -- Method: window.insertln ()
     Insert a blank line under the cursor. All following lines are
     moved down by 1 line.

 -- Method: window.insnstr ([y, x], str, n[, attr])
     Insert a character string (as many characters as will fit on the
     line) before the character under the cursor, up to _n_ characters.
      If _n_ is zero or negative, the entire string is inserted. All
     characters to the right of the cursor are shifted right, with the
     rightmost characters on the line being lost.  The cursor position
     does not change (after moving to _y_, _x_, if specified).

 -- Method: window.insstr ([y, x], str[, attr])
     Insert a character string (as many characters as will fit on the
     line) before the character under the cursor.  All characters to
     the right of the cursor are shifted right, with the rightmost
     characters on the line being lost.  The cursor position does not
     change (after moving to _y_, _x_, if specified).

 -- Method: window.instr ([y, x] [, n])
     Returns a string of characters, extracted from the window starting
     at the current cursor position, or at _y_, _x_ if specified.
     Attributes are stripped from the characters.  If _n_ is specified,
     see instr() returns return a string at most _n_
     characters long (exclusive of the trailing NUL).

 -- Method: window.is_linetouched (line)
     Returns true if the specified line was modified since the last
     call to refresh().; otherwise returns false.  Raises a
     curses.error.  exception if _line_ is not valid for
     the given window.

 -- Method: window.is_wintouched ()
     Returns true if the specified window was modified since the last
     call to refresh().; otherwise returns false.

 -- Method: window.keypad (yes)
     If _yes_ is 1, escape sequences generated by some keys (keypad,
     function keys) will be interpreted by curses. If _yes_
     is 0, escape sequences will be left as is in the input stream.

 -- Method: window.leaveok (yes)
     If _yes_ is 1, cursor is left where it is on update, instead of
     being at "cursor position."  This reduces cursor movement where
     possible. If possible the cursor will be made invisible.

     If _yes_ is 0, cursor will always be at "cursor position" after an
     update.

 -- Method: window.move (new_y, new_x)
     Move cursor to `(new_y, new_x)'.

 -- Method: window.mvderwin (y, x)
     Moves the window inside its parent window.  The screen-relative
     parameters of the window are not changed.  This routine is used to
     display different parts of the parent window at the same physical
     position on the screen.

 -- Method: window.mvwin (new_y, new_x)
     Move the window so its upper-left corner is at `(new_y, new_x)'.

 -- Method: window.nodelay (yes)
     If _yes_ is `1', see getch() will be non-blocking.

 -- Method: window.notimeout (yes)
     If _yes_ is `1', escape sequences will not be timed out.

     If _yes_ is `0', after a few milliseconds, an escape sequence will
     not be interpreted, and will be left in the input stream as is.

 -- Method: window.noutrefresh ()
     Mark for refresh but wait.  This function updates the data
     structure representing the desired state of the window, but does
     not force an update of the physical screen.  To accomplish that,
     call  doupdate().

 -- Method: window.overlay (destwin[, sminrow, smincol, dminrow,
          dmincol, dmaxrow, dmaxcol])
     Overlay the window on top of _destwin_. The windows need not be
     the same size, only the overlapping region is copied. This copy is
     non-destructive, which means that the current background character
     does not overwrite the old contents of _destwin_.

     To get fine-grained control over the copied region, the second
     form of overlay() can be used. _sminrow_ and
     _smincol_ are the upper-left coordinates of the source window, and
     the other variables mark a rectangle in the destination window.

 -- Method: window.overwrite (destwin[, sminrow, smincol, dminrow,
          dmincol, dmaxrow, dmaxcol])
     Overwrite the window on top of _destwin_. The windows need not be
     the same size, in which case only the overlapping region is
     copied. This copy is destructive, which means that the current
     background character overwrites the old contents of _destwin_.

     To get fine-grained control over the copied region, the second
     form of overwrite() can be used. _sminrow_ and
     _smincol_ are the upper-left coordinates of the source window, the
     other variables mark a rectangle in the destination window.

 -- Method: window.putwin (file)
     Writes all data associated with the window into the provided file
     object.  This information can be later retrieved using the
     getwin() function.

 -- Method: window.redrawln (beg, num)
     Indicates that the _num_ screen lines, starting at line _beg_, are
     corrupted and should be completely redrawn on the next
     refresh() call.

 -- Method: window.redrawwin ()
     Touches the entire window, causing it to be completely redrawn on
     the next refresh() call.

 -- Method: window.refresh ([pminrow, pmincol, sminrow, smincol,
          smaxrow, smaxcol])
     Update the display immediately (sync actual screen with previous
     drawing/deleting methods).

     The 6 optional arguments can only be specified when the window is
     a pad created with newpad().  The additional
     parameters are needed to indicate what part of the pad and screen
     are involved. _pminrow_ and _pmincol_ specify the upper left-hand
     corner of the rectangle to be displayed in the pad.  _sminrow_,
     _smincol_, _smaxrow_, and _smaxcol_ specify the edges of the
     rectangle to be displayed on the screen.  The lower right-hand
     corner of the rectangle to be displayed in the pad is calculated
     from the screen coordinates, since the rectangles must be the same
     size.  Both rectangles must be entirely contained within their
     respective structures.  Negative values of _pminrow_, _pmincol_,
     _sminrow_, or _smincol_ are treated as if they were zero.

 -- Method: window.scroll ([lines=1])
     Scroll the screen or scrolling region upward by _lines_ lines.

 -- Method: window.scrollok (flag)
     Controls what happens when the cursor of a window is moved off the
     edge of the window or scrolling region, either as a result of a
     newline action on the bottom line, or typing the last character of
     the last line.  If _flag_ is false, the cursor is left on the
     bottom line.  If _flag_ is true, the window is scrolled up one
     line.  Note that in order to get the physical scrolling effect on
     the terminal, it is also necessary to call idlok().

 -- Method: window.setscrreg (top, bottom)
     Set the scrolling region from line _top_ to line _bottom_. All
     scrolling actions will take place in this region.

 -- Method: window.standend ()
     Turn off the standout attribute.  On some terminals this has the
     side effect of turning off all attributes.

 -- Method: window.standout ()
     Turn on attribute _A_STANDOUT_.

 -- Method: window.subpad ([nlines, ncols], begin_y, begin_x)
     Return a sub-window, whose upper-left corner is at `(begin_y,
     begin_x)', and whose width/height is _ncols_/_nlines_.

 -- Method: window.subwin ([nlines, ncols], begin_y, begin_x)
     Return a sub-window, whose upper-left corner is at `(begin_y,
     begin_x)', and whose width/height is _ncols_/_nlines_.

     By default, the sub-window will extend from the specified position
     to the lower right corner of the window.

 -- Method: window.syncdown ()
     Touches each location in the window that has been touched in any
     of its ancestor windows.  This routine is called by
     refresh(), so it should almost never be necessary to call it
     manually.

 -- Method: window.syncok (flag)
     If called with _flag_ set to true, then syncup() is
     called automatically whenever there is a change in the window.

 -- Method: window.syncup ()
     Touches all locations in ancestors of the window that have been
     changed in  the window.

 -- Method: window.timeout (delay)
     Sets blocking or non-blocking read behavior for the window.  If
     _delay_ is negative, blocking read is used (which will wait
     indefinitely for input).  If _delay_ is zero, then non-blocking
     read is used, and -1 will be returned by getch() if
     no input is waiting.  If _delay_ is positive, then getch()
     will block for _delay_ milliseconds, and return -1 if there
     is still no input at the end of that time.

 -- Method: window.touchline (start, count[, changed])
     Pretend _count_ lines have been changed, starting with line
     _start_.  If _changed_ is supplied, it specifies whether the
     affected lines are marked as having been changed (_changed_=1) or
     unchanged (_changed_=0).

 -- Method: window.touchwin ()
     Pretend the whole window has been changed, for purposes of drawing
     optimizations.

 -- Method: window.untouchwin ()
     Marks all lines in  the  window  as unchanged since the last call
     to refresh().

 -- Method: window.vline ([y, x], ch, n)
     Display a vertical line starting at `(y, x)' with length _n_
     consisting of the character _ch_.

5.15.9.3 Constants
..................

The curses module defines the following data members:

 -- Data: curses.ERR
     Some curses routines  that  return  an integer, such as
     `getch()', return ERR upon failure.

 -- Data: curses.OK
     Some curses routines  that  return  an integer, such as
     napms(), return OK upon success.

 -- Data: curses.version
     A string representing the current version of the module.  Also
     available as `__version__'.

  Several constants are available to specify character cell attributes:

Attribute              Meaning
-----------------------------------------------------------
`A_ALTCHARSET'         Alternate character set mode.
`A_BLINK'              Blink mode.
`A_BOLD'               Bold mode.
`A_DIM'                Dim mode.
`A_NORMAL'             Normal attribute.
`A_STANDOUT'           Standout mode.
`A_UNDERLINE'          Underline mode.

  Keys are referred to by integer constants with names starting with
`KEY_'.  The exact keycaps available are system dependent.

Key constant            Key
-------------------------------------------------------------------------
`KEY_MIN'               Minimum key value
`KEY_BREAK'             Break key (unreliable)
`KEY_DOWN'              Down-arrow
`KEY_UP'                Up-arrow
`KEY_LEFT'              Left-arrow
`KEY_RIGHT'             Right-arrow
`KEY_HOME'              Home key (upward+left arrow)
`KEY_BACKSPACE'         Backspace (unreliable)
`KEY_F0'                Function keys.  Up to 64 function keys are
                        supported.
`KEY_Fn'                Value of function key _n_
`KEY_DL'                Delete line
`KEY_IL'                Insert line
`KEY_DC'                Delete character
`KEY_IC'                Insert char or enter insert mode
`KEY_EIC'               Exit insert char mode
`KEY_CLEAR'             Clear screen
`KEY_EOS'               Clear to end of screen
`KEY_EOL'               Clear to end of line
`KEY_SF'                Scroll 1 line forward
`KEY_SR'                Scroll 1 line backward (reverse)
`KEY_NPAGE'             Next page
`KEY_PPAGE'             Previous page
`KEY_STAB'              Set tab
`KEY_CTAB'              Clear tab
`KEY_CATAB'             Clear all tabs
`KEY_ENTER'             Enter or send (unreliable)
`KEY_SRESET'            Soft (partial) reset (unreliable)
`KEY_RESET'             Reset or hard reset (unreliable)
`KEY_PRINT'             Print
`KEY_LL'                Home down or bottom (lower left)
`KEY_A1'                Upper left of keypad
`KEY_A3'                Upper right of keypad
`KEY_B2'                Center of keypad
`KEY_C1'                Lower left of keypad
`KEY_C3'                Lower right of keypad
`KEY_BTAB'              Back tab
`KEY_BEG'               Beg (beginning)
`KEY_CANCEL'            Cancel
`KEY_CLOSE'             Close
`KEY_COMMAND'           Cmd (command)
`KEY_COPY'              Copy
`KEY_CREATE'            Create
`KEY_END'               End
`KEY_EXIT'              Exit
`KEY_FIND'              Find
`KEY_HELP'              Help
`KEY_MARK'              Mark
`KEY_MESSAGE'           Message
`KEY_MOVE'              Move
`KEY_NEXT'              Next
`KEY_OPEN'              Open
`KEY_OPTIONS'           Options
`KEY_PREVIOUS'          Prev (previous)
`KEY_REDO'              Redo
`KEY_REFERENCE'         Ref (reference)
`KEY_REFRESH'           Refresh
`KEY_REPLACE'           Replace
`KEY_RESTART'           Restart
`KEY_RESUME'            Resume
`KEY_SAVE'              Save
`KEY_SBEG'              Shifted Beg (beginning)
`KEY_SCANCEL'           Shifted Cancel
`KEY_SCOMMAND'          Shifted Command
`KEY_SCOPY'             Shifted Copy
`KEY_SCREATE'           Shifted Create
`KEY_SDC'               Shifted Delete char
`KEY_SDL'               Shifted Delete line
`KEY_SELECT'            Select
`KEY_SEND'              Shifted End
`KEY_SEOL'              Shifted Clear line
`KEY_SEXIT'             Shifted Dxit
`KEY_SFIND'             Shifted Find
`KEY_SHELP'             Shifted Help
`KEY_SHOME'             Shifted Home
`KEY_SIC'               Shifted Input
`KEY_SLEFT'             Shifted Left arrow
`KEY_SMESSAGE'          Shifted Message
`KEY_SMOVE'             Shifted Move
`KEY_SNEXT'             Shifted Next
`KEY_SOPTIONS'          Shifted Options
`KEY_SPREVIOUS'         Shifted Prev
`KEY_SPRINT'            Shifted Print
`KEY_SREDO'             Shifted Redo
`KEY_SREPLACE'          Shifted Replace
`KEY_SRIGHT'            Shifted Right arrow
`KEY_SRSUME'            Shifted Resume
`KEY_SSAVE'             Shifted Save
`KEY_SSUSPEND'          Shifted Suspend
`KEY_SUNDO'             Shifted Undo
`KEY_SUSPEND'           Suspend
`KEY_UNDO'              Undo
`KEY_MOUSE'             Mouse event has occurred
`KEY_RESIZE'            Terminal resize event
`KEY_MAX'               Maximum key value

  On VT100s and their software emulations, such as X terminal
emulators, there are normally at least four function keys (`KEY_F1',
`KEY_F2', `KEY_F3', `KEY_F4') available, and the arrow keys mapped to
`KEY_UP', `KEY_DOWN', `KEY_LEFT' and `KEY_RIGHT' in the obvious way.
If your machine has a PC keyboard, it is safe to expect arrow keys and
twelve function keys (older PC keyboards may have only ten function
keys); also, the following keypad mappings are standard:

Keycap                 Constant
---------------------------------------
`Insert'               KEY_IC
`Delete'               KEY_DC
`Home'                 KEY_HOME
`End'                  KEY_END
`Page Up'              KEY_NPAGE
`Page Down'            KEY_PPAGE

  The following table lists characters from the alternate character
set. These are inherited from the VT100 terminal, and will generally be
available on software emulations such as X terminals.  When there is no
graphic available, curses falls back on a crude printable ASCII
approximation.

     Note: These are available only after initscr() has
     been called.

ACS code               Meaning
----------------------------------------------------------------------
`ACS_BBSS'             alternate name for upper right corner
`ACS_BLOCK'            solid square block
`ACS_BOARD'            board of squares
`ACS_BSBS'             alternate name for horizontal line
`ACS_BSSB'             alternate name for upper left corner
`ACS_BSSS'             alternate name for top tee
`ACS_BTEE'             bottom tee
`ACS_BULLET'           bullet
`ACS_CKBOARD'          checker board (stipple)
`ACS_DARROW'           arrow pointing down
`ACS_DEGREE'           degree symbol
`ACS_DIAMOND'          diamond
`ACS_GEQUAL'           greater-than-or-equal-to
`ACS_HLINE'            horizontal line
`ACS_LANTERN'          lantern symbol
`ACS_LARROW'           left arrow
`ACS_LEQUAL'           less-than-or-equal-to
`ACS_LLCORNER'         lower left-hand corner
`ACS_LRCORNER'         lower right-hand corner
`ACS_LTEE'             left tee
`ACS_NEQUAL'           not-equal sign
`ACS_PI'               letter pi
`ACS_PLMINUS'          plus-or-minus sign
`ACS_PLUS'             big plus sign
`ACS_RARROW'           right arrow
`ACS_RTEE'             right tee
`ACS_S1'               scan line 1
`ACS_S3'               scan line 3
`ACS_S7'               scan line 7
`ACS_S9'               scan line 9
`ACS_SBBS'             alternate name for lower right corner
`ACS_SBSB'             alternate name for vertical line
`ACS_SBSS'             alternate name for right tee
`ACS_SSBB'             alternate name for lower left corner
`ACS_SSBS'             alternate name for bottom tee
`ACS_SSSB'             alternate name for left tee
`ACS_SSSS'             alternate name for crossover or big plus
`ACS_STERLING'         pound sterling
`ACS_TTEE'             top tee
`ACS_UARROW'           up arrow
`ACS_ULCORNER'         upper left corner
`ACS_URCORNER'         upper right corner
`ACS_VLINE'            vertical line

  The following table lists the predefined colors:

Constant                Color
---------------------------------------------------------
`COLOR_BLACK'           Black
`COLOR_BLUE'            Blue
`COLOR_CYAN'            Cyan (light greenish blue)
`COLOR_GREEN'           Green
`COLOR_MAGENTA'         Magenta (purplish red)
`COLOR_RED'             Red
`COLOR_WHITE'           White
`COLOR_YELLOW'          Yellow

5.15.10 `curses.textpad' -- Text input widget for curses programs
-----------------------------------------------------------------

New in version 1.6.

  The curses.textpad module provides a Textbox.
class that handles elementary text editing in a curses window,
supporting a set of keybindings resembling those of Emacs (thus, also
of Netscape Navigator, BBedit 6.x, FrameMaker, and many other
programs).  The module also provides a rectangle-drawing function
useful for framing text boxes or for other purposes.

  The module curses.textpad defines the following function:

 -- Function: curses.textpad.rectangle (win, uly, ulx, lry, lrx)
     Draw a rectangle.  The first argument must be a window object; the
     remaining arguments are coordinates relative to that window.  The
     second and third arguments are the y and x coordinates of the
     upper left hand corner of the rectangle to be drawn; the fourth
     and fifth arguments are the y and x coordinates of the lower right
     hand corner. The rectangle will be drawn using VT100/IBM PC forms
     characters on terminals that make this possible (including xterm
     and most other software terminal emulators).  Otherwise it will be
     drawn with ASCII  dashes, vertical bars, and plus signs.

5.15.10.1 Textbox objects
.........................

You can instantiate a Textbox object as follows:

 -- Class: curses.textpad.Textbox (win)
     Return a textbox widget object.  The _win_ argument should be a
     curses `WindowObject' in which the textbox is to be contained. The
     edit cursor of the textbox is initially located at the upper left
     hand corner of the containing window, with coordinates `(0, 0)'.
     The instance's stripspaces flag is initially on.

     Textbox objects have the following methods:

      -- Method: edit ([validator])
          This is the entry point you will normally use.  It accepts
          editing keystrokes until one of the termination keystrokes is
          entered.  If _validator_ is supplied, it must be a function.
          It will be called for each keystroke entered with the
          keystroke as a parameter; command dispatch is done on the
          result. This method returns the window contents as a string;
          whether blanks in the window are included is affected by the
          stripspaces member.

      -- Method: do_command (ch)
          Process a single command keystroke.  Here are the supported
          special keystrokes:

          Keystroke              Action
          -----------------------------------------------------------------------
          `Control-A'            Go to left edge of window.
          `Control-B'            Cursor left, wrapping to previous line if
                                 appropriate.
          `Control-D'            Delete character under cursor.
          `Control-E'            Go to right edge (stripspaces off) or end of
                                 line (stripspaces on).
          `Control-F'            Cursor right, wrapping to next line when
                                 appropriate.
          `Control-G'            Terminate, returning the window contents.
          `Control-H'            Delete character backward.
          `Control-J'            Terminate if the window is 1 line, otherwise
                                 insert newline.
          `Control-K'            If line is blank, delete it, otherwise clear
                                 to end of line.
          `Control-L'            Refresh screen.
          `Control-N'            Cursor down; move down one line.
          `Control-O'            Insert a blank line at cursor location.
          `Control-P'            Cursor up; move up one line.

          Move operations do nothing if the cursor is at an edge where
          the movement is not possible.  The following synonyms are
          supported where possible:

          Constant                     Keystroke
          ----------------------------------------------------
          `KEY_LEFT'                   `Control-B'
          `KEY_RIGHT'                  `Control-F'
          `KEY_UP'                     `Control-P'
          `KEY_DOWN'                   `Control-N'
          `KEY_BACKSPACE'              `Control-h'

          All other keystrokes are treated as a command to insert the
          given character and move right (with line wrapping).

      -- Method: gather ()
          This method returns the window contents as a string; whether
          blanks in the window are included is affected by the
          stripspaces member.

      -- Attribute: stripspaces
          This data member is a flag which controls the interpretation
          of blanks in the window.  When it is on, trailing blanks on
          each line are ignored; any cursor motion that would land the
          cursor on a trailing blank goes to the end of that line
          instead, and trailing blanks are stripped when the window
          contents are gathered.

5.15.11 `curses.wrapper' -- Terminal handler for curses programs
----------------------------------------------------------------

New in version 1.6.

  This module supplies one function, see wrapper(), which runs
another function which should be the rest of your curses-using
application.  If the application raises an exception, see wrapper()
will restore the terminal to a sane state before re-raising the
exception and generating a traceback.

 -- Function: curses.wrapper.wrapper (func, ...)
     Wrapper function that initializes curses and calls another
     function, _func_, restoring normal keyboard/screen behavior on
     error. The callable object _func_ is then passed the main window
     'stdscr' as its first argument, followed by any other arguments
     passed to wrapper().

  Before calling the hook function, see wrapper() turns on
cbreak mode, turns off echo, enables the terminal keypad, and
initializes colors if the terminal has color support.  On exit (whether
normally or by exception) it restores cooked mode, turns on echo, and
disables the terminal keypad.

5.15.12 `curses.ascii' -- Utilities for ASCII characters
--------------------------------------------------------

New in version 1.6.

  The curses.ascii module supplies name constants for ASCII
characters and functions to test membership in various ASCII character
classes.  The constants supplied are names for control characters as
follows:

Name               Meaning
----------------------------------------------------------------------
`NUL'
`SOH'              Start of heading, console interrupt
`STX'              Start of text
`ETX'              End of text
`EOT'              End of transmission
`ENQ'              Enquiry, goes with `ACK' flow control
`ACK'              Acknowledgement
`BEL'              Bell
`BS'               Backspace
`TAB'              Tab
`HT'               Alias for `TAB': "Horizontal tab"
`LF'               Line feed
`NL'               Alias for `LF': "New line"
`VT'               Vertical tab
`FF'               Form feed
`CR'               Carriage return
`SO'               Shift-out, begin alternate character set
`SI'               Shift-in, resume default character set
`DLE'              Data-link escape
`DC1'              XON, for flow control
`DC2'              Device control 2, block-mode flow control
`DC3'              XOFF, for flow control
`DC4'              Device control 4
`NAK'              Negative acknowledgement
`SYN'              Synchronous idle
`ETB'              End transmission block
`CAN'              Cancel
`EM'               End of medium
`SUB'              Substitute
`ESC'              Escape
`FS'               File separator
`GS'               Group separator
`RS'               Record separator, block-mode terminator
`US'               Unit separator
`SP'               Space
`DEL'              Delete

  Note that many of these have little practical significance in modern
usage.  The mnemonics derive from teleprinter conventions that predate
digital computers.

  The module supplies the following functions, patterned on those in
the standard C library:

 -- Function: curses.ascii.isalnum (c)
     Checks for an ASCII alphanumeric character; it is equivalent to
     `isalpha(c) or isdigit(c)'.

 -- Function: curses.ascii.isalpha (c)
     Checks for an ASCII alphabetic character; it is equivalent to
     `isupper(c) or islower(c)'.

 -- Function: curses.ascii.isascii (c)
     Checks for a character value that fits in the 7-bit ASCII set.

 -- Function: curses.ascii.isblank (c)
     Checks for an ASCII whitespace character.

 -- Function: curses.ascii.iscntrl (c)
     Checks for an ASCII control character (in the range 0x00 to 0x1f).

 -- Function: curses.ascii.isdigit (c)
     Checks for an ASCII decimal digit, `'0'' through `'9''.  This is
     equivalent to `c in string.digits'.

 -- Function: curses.ascii.isgraph (c)
     Checks for ASCII any printable character except space.

 -- Function: curses.ascii.islower (c)
     Checks for an ASCII lower-case character.

 -- Function: curses.ascii.isprint (c)
     Checks for any ASCII printable character including space.

 -- Function: curses.ascii.ispunct (c)
     Checks for any printable ASCII character which is not a space or
     an alphanumeric character.

 -- Function: curses.ascii.isspace (c)
     Checks for ASCII white-space characters; space, line feed,
     carriage return, form feed, horizontal tab, vertical tab.

 -- Function: curses.ascii.isupper (c)
     Checks for an ASCII uppercase letter.

 -- Function: curses.ascii.isxdigit (c)
     Checks for an ASCII hexadecimal digit.  This is equivalent to `c in
     string.hexdigits'.

 -- Function: curses.ascii.isctrl (c)
     Checks for an ASCII control character (ordinal values 0 to 31).

 -- Function: curses.ascii.ismeta (c)
     Checks for a non-ASCII character (ordinal values 0x80 and above).

  These functions accept either integers or strings; when the argument
is a string, it is first converted using the built-in function
ord().

  Note that all these functions check ordinal bit values derived from
the  first character of the string you pass in; they do not actually
know anything about the host machine's character encoding.  For
functions  that know about the character encoding (and handle
internationalization properly) see the string module.

  The following two functions take either a single-character string or
integer byte value; they return a value of the same type.

 -- Function: curses.ascii.ascii (c)
     Return the ASCII value corresponding to the low 7 bits of _c_.

 -- Function: curses.ascii.ctrl (c)
     Return the control character corresponding to the given character
     (the character bit value is bitwise-anded with 0x1f).

 -- Function: curses.ascii.alt (c)
     Return the 8-bit character corresponding to the given ASCII
     character (the character bit value is bitwise-ored with 0x80).

  The following function takes either a single-character string or
integer value; it returns a string.

 -- Function: curses.ascii.unctrl (c)
     Return a string representation of the ASCII character _c_.  If _c_
     is printable, this string is the character itself.  If the
     character is a control character (0x00-0x1f) the string consists
     of a caret (`'^'') followed by the corresponding uppercase letter.
     If the character is an ASCII delete (0x7f) the string is `'^?''.
     If the character has its meta bit (0x80) set, the meta bit is
     stripped, the preceding rules applied, and `'!'' prepended to the
     result.

 -- Data: curses.ascii.controlnames
     A 33-element string array that contains the ASCII mnemonics for
     the thirty-two ASCII control characters from 0 (NUL) to 0x1f (US),
     in order, plus the mnemonic `SP' for the space character.

5.15.13 `curses.panel' -- A panel stack extension for curses
------------------------------------------------------------

Panels are windows with the added feature of depth, so they can be
stacked on top of each other, and only the visible portions of each
window will be displayed.  Panels can be added, moved up or down in the
stack, and removed.

5.15.13.1 Functions
...................

The module curses.panel defines the following functions:

 -- Function: curses.panel.bottom_panel ()
     Returns the bottom panel in the panel stack.

 -- Function: curses.panel.new_panel (win)
     Returns a panel object, associating it with the given window
     _win_. Be aware that you need to keep the returned panel object
     referenced explicitly.  If you don't, the panel object is garbage
     collected and removed from the panel stack.

 -- Function: curses.panel.top_panel ()
     Returns the top panel in the panel stack.

 -- Function: curses.panel.update_panels ()
     Updates the virtual screen after changes in the panel stack. This
     does not call curses.doupdate(), so you'll have to do
     this yourself.

5.15.13.2 Panel Objects
.......................

Panel objects, as returned by new_panel() above, are
windows with a stacking order. There's always a window associated with
a panel which determines the content, while the panel methods are
responsible for the window's depth in the panel stack.

  Panel objects have the following methods:

 -- Method: Panel.above ()
     Returns the panel above the current panel.

 -- Method: Panel.below ()
     Returns the panel below the current panel.

 -- Method: Panel.bottom ()
     Push the panel to the bottom of the stack.

 -- Method: Panel.hidden ()
     Returns true if the panel is hidden (not visible), false otherwise.

 -- Method: Panel.hide ()
     Hide the panel. This does not delete the object, it just makes the
     window on screen invisible.

 -- Method: Panel.move (y, x)
     Move the panel to the screen coordinates `(y, x)'.

 -- Method: Panel.replace (win)
     Change the window associated with the panel to the window _win_.

 -- Method: Panel.set_userptr (obj)
     Set the panel's user pointer to _obj_. This is used to associate
     an arbitrary piece of data with the panel, and can be any Python
     object.

 -- Method: Panel.show ()
     Display the panel (which might have been hidden).

 -- Method: Panel.top ()
     Push panel to the top of the stack.

 -- Method: Panel.userptr ()
     Returns the user pointer for the panel.  This might be any Python
     object.

 -- Method: Panel.window ()
     Returns the window object associated with the panel.

5.15.14 `platform' --  Access to underlying platform's identifying data
-----------------------------------------------------------------------

New in version 2.3.

     Note: Specific platforms listed alphabetically, with Linux
     included in the Unix section.

5.15.14.1 Cross Platform
........................

 -- Function: platform.architecture (executable=sys.executable, bits=",
          linkage=")
     Queries the given executable (defaults to the Python interpreter
     binary) for various architecture information.

     Returns a tuple `(bits, linkage)' which contain information about
     the bit architecture and the linkage format used for the
     executable. Both values are returned as strings.

     Values that cannot be determined are returned as given by the
     parameter presets.  If bits is given as `''', the
     `sizeof(pointer)()' (or `sizeof(long)()' on Python version <
     1.5.2) is used as indicator for the supported pointer size.

     The function relies on the system's `file' command to do the
     actual work.  This is available on most if not all Unix  platforms
     and some non-Unix platforms and then only if the executable points
     to the Python interpreter.  Reasonable defaults are used when the
     above needs are not met.

 -- Function: platform.machine ()
     Returns the machine type, e.g. `'i386''. An empty string is
     returned if the value cannot be determined.

 -- Function: platform.node ()
     Returns the computer's network name (may not be fully qualified!).
     An empty string is returned if the value cannot be determined.

 -- Function: platform.platform (aliased=0, terse=0)
     Returns a single string identifying the underlying platform with
     as much useful information as possible.

     The output is intended to be _human readable_ rather than machine
     parseable. It may look different on different platforms and this
     is intended.

     If _aliased_ is true, the function will use aliases for various
     platforms that report system names which differ from their common
     names, for example SunOS will be reported as Solaris.  The
     system_alias() function is used to implement this.

     Setting _terse_ to true causes the function to return only the
     absolute minimum information needed to identify the platform.

 -- Function: platform.processor ()
     Returns the (real) processor name, e.g. `'amdk6''.

     An empty string is returned if the value cannot be determined.
     Note that many platforms do not provide this information or simply
     return the same value as for machine().  NetBSD does
     this.

 -- Function: platform.python_build ()
     Returns a tuple `(buildno, builddate)' stating the Python build
     number and date as strings.

 -- Function: platform.python_compiler ()
     Returns a string identifying the compiler used for compiling
     Python.

 -- Function: platform.python_branch ()
     Returns a string identifying the Python implementation SCM branch.

     New in version 2.6.

 -- Function: platform.python_implementation ()
     Returns a string identifying the Python implementation. Possible
     return values are: 'CPython', 'IronPython', 'Jython'.

     New in version 2.6.

 -- Function: platform.python_revision ()
     Returns a string identifying the Python implementation SCM
     revision.

     New in version 2.6.

 -- Function: platform.python_version ()
     Returns the Python version as string `'major.minor.patchlevel''

     Note that unlike the Python `sys.version', the returned value will
     always include the patchlevel (it defaults to 0).

 -- Function: platform.python_version_tuple ()
     Returns the Python version as tuple `(major, minor, patchlevel)'
     of strings.

     Note that unlike the Python `sys.version', the returned value will
     always include the patchlevel (it defaults to `'0'').

 -- Function: platform.release ()
     Returns the system's release, e.g. `'2.2.0'' or `'NT'' An empty
     string is returned if the value cannot be determined.

 -- Function: platform.system ()
     Returns the system/OS name, e.g. `'Linux'', `'Windows'', or
     `'Java''. An empty string is returned if the value cannot be
     determined.

 -- Function: platform.system_alias (system, release, version)
     Returns `(system, release, version)' aliased to common marketing
     names used for some systems.  It also does some reordering of the
     information in some cases where it would otherwise cause confusion.

 -- Function: platform.version ()
     Returns the system's release version, e.g. `'#3 on degas''. An
     empty string is returned if the value cannot be determined.

 -- Function: platform.uname ()
     Fairly portable uname interface. Returns a tuple of strings
     `(system, node, release, version, machine, processor)' identifying
     the underlying platform.

     Note that unlike the os.uname() function this also
     returns possible processor information as additional tuple entry.

     Entries which cannot be determined are set to `'''.

5.15.14.2 Java Platform
.......................

 -- Function: platform.java_ver (release=", vendor=", vminfo=(", ", "),
          osinfo=(", ", "))
     Version interface for Jython.

     Returns a tuple `(release, vendor, vminfo, osinfo)' with _vminfo_
     being a tuple `(vm_name, vm_release, vm_vendor)' and _osinfo_
     being a tuple `(os_name, os_version, os_arch)'. Values which
     cannot be determined are set to the defaults given as parameters
     (which all default to `''').

5.15.14.3 Windows Platform
..........................

 -- Function: platform.win32_ver (release=", version=", csd=", ptype=")
     Get additional version information from the Windows Registry and
     return a tuple `(version, csd, ptype)' referring to version
     number, CSD level and OS type (multi/single processor).

     As a hint: _ptype_ is `'Uniprocessor Free'' on single processor NT
     machines and `'Multiprocessor Free'' on multi processor machines.
     The _'Free'_ refers to the OS version being free of debugging
     code. It could also state _'Checked'_ which means the OS version
     uses debugging code, i.e. code that checks arguments, ranges, etc.

          Note: Note: this function works best with Mark Hammond's
          `win32all' package installed, but also on Python 2.3 and
          later (support for this was added in Python 2.6). It obviously
          only runs on Win32 compatible platforms.

5.15.14.4 Win95/98 specific
...........................

 -- Function: platform.popen (cmd, mode='r', bufsize=None)
     Portable popen() interface.  Find a working popen
     implementation preferring `win32pipe.popen()'.  On Windows NT,
     `win32pipe.popen()' should work; on Windows 9x it hangs due to
     bugs in the MS C library.

5.15.14.5 Mac OS Platform
.........................

 -- Function: platform.mac_ver (release=", versioninfo=(", ", "),
          machine=")
     Get Mac OS version information and return it as tuple `(release,
     versioninfo, machine)' with _versioninfo_ being a tuple `(version,
     dev_stage, non_release_version)'.

     Entries which cannot be determined are set to `'''.  All tuple
     entries are strings.

     Documentation for the underlying `gestalt()' API is available
     online at <http://www.rgaros.nl/gestalt/>.

5.15.14.6 Unix Platforms
........................

 -- Function: platform.dist (distname=", version=", id=",
          supported_dists=('SuSE', 'debian', 'redhat', 'mandrake', ...))
     This is an old version of the functionality now provided by
     linux_distribution(). For new code, please use the
     linux_distribution().

     The only difference between the two is that `dist()' always
     returns the short name of the distribution taken from the
     `supported_dists' parameter.

     Deprecated since version 2.6.

 -- Function: platform.linux_distribution (distname=", version=", id=",
          supported_dists=('SuSE', 'debian', 'redhat', 'mandrake',
          ...), full_distribution_name=1)
     Tries to determine the name of the Linux OS distribution name.

     `supported_dists' may be given to define the set of Linux
     distributions to look for. It defaults to a list of currently
     supported Linux distributions identified by their release file
     name.

     If `full_distribution_name' is true (default), the full
     distribution read from the OS is returned. Otherwise the short
     name taken from `supported_dists' is used.

     Returns a tuple `(distname,version,id)' which defaults to the args
     given as parameters.  `id' is the item in parentheses after the
     version number.  It is usually the version codename.

     New in version 2.6.

 -- Function: platform.libc_ver (executable=sys.executable, lib=",
          version=", chunksize=2048)
     Tries to determine the libc version against which the file
     executable (defaults to the Python interpreter) is linked.
     Returns a tuple of strings `(lib, version)' which default to the
     given parameters in case the lookup fails.

     Note that this function has intimate knowledge of how different
     libc versions add symbols to the executable is probably only
     usable for executables compiled using *gcc*.

     The file is read and scanned in chunks of _chunksize_ bytes.

5.15.15 `errno' -- Standard errno system symbols
------------------------------------------------

This module makes available standard `errno' system symbols. The value
of each symbol is the corresponding integer value. The names and
descriptions are borrowed from `linux/include/errno.h', which should be
pretty all-inclusive.

 -- Data: errno.errorcode
     Dictionary providing a mapping from the errno value to the string
     name in the underlying system.  For instance,
     `errno.errorcode[errno.EPERM]' maps to `'EPERM''.

  To translate a numeric error code to an error message, use
os.strerror().

  Of the following list, symbols that are not used on the current
platform are not defined by the module.  The specific list of defined
symbols is available as `errno.errorcode.keys()'.  Symbols available
can include:

 -- Data: errno.EPERM
     Operation not permitted

 -- Data: errno.ENOENT
     No such file or directory

 -- Data: errno.ESRCH
     No such process

 -- Data: errno.EINTR
     Interrupted system call

 -- Data: errno.EIO
     I/O error

 -- Data: errno.ENXIO
     No such device or address

 -- Data: errno.E2BIG
     Arg list too long

 -- Data: errno.ENOEXEC
     Exec format error

 -- Data: errno.EBADF
     Bad file number

 -- Data: errno.ECHILD
     No child processes

 -- Data: errno.EAGAIN
     Try again

 -- Data: errno.ENOMEM
     Out of memory

 -- Data: errno.EACCES
     Permission denied

 -- Data: errno.EFAULT
     Bad address

 -- Data: errno.ENOTBLK
     Block device required

 -- Data: errno.EBUSY
     Device or resource busy

 -- Data: errno.EEXIST
     File exists

 -- Data: errno.EXDEV
     Cross-device link

 -- Data: errno.ENODEV
     No such device

 -- Data: errno.ENOTDIR
     Not a directory

 -- Data: errno.EISDIR
     Is a directory

 -- Data: errno.EINVAL
     Invalid argument

 -- Data: errno.ENFILE
     File table overflow

 -- Data: errno.EMFILE
     Too many open files

 -- Data: errno.ENOTTY
     Not a typewriter

 -- Data: errno.ETXTBSY
     Text file busy

 -- Data: errno.EFBIG
     File too large

 -- Data: errno.ENOSPC
     No space left on device

 -- Data: errno.ESPIPE
     Illegal seek

 -- Data: errno.EROFS
     Read-only file system

 -- Data: errno.EMLINK
     Too many links

 -- Data: errno.EPIPE
     Broken pipe

 -- Data: errno.EDOM
     Math argument out of domain of func

 -- Data: errno.ERANGE
     Math result not representable

 -- Data: errno.EDEADLK
     Resource deadlock would occur

 -- Data: errno.ENAMETOOLONG
     File name too long

 -- Data: errno.ENOLCK
     No record locks available

 -- Data: errno.ENOSYS
     Function not implemented

 -- Data: errno.ENOTEMPTY
     Directory not empty

 -- Data: errno.ELOOP
     Too many symbolic links encountered

 -- Data: errno.EWOULDBLOCK
     Operation would block

 -- Data: errno.ENOMSG
     No message of desired type

 -- Data: errno.EIDRM
     Identifier removed

 -- Data: errno.ECHRNG
     Channel number out of range

 -- Data: errno.EL2NSYNC
     Level 2 not synchronized

 -- Data: errno.EL3HLT
     Level 3 halted

 -- Data: errno.EL3RST
     Level 3 reset

 -- Data: errno.ELNRNG
     Link number out of range

 -- Data: errno.EUNATCH
     Protocol driver not attached

 -- Data: errno.ENOCSI
     No CSI structure available

 -- Data: errno.EL2HLT
     Level 2 halted

 -- Data: errno.EBADE
     Invalid exchange

 -- Data: errno.EBADR
     Invalid request descriptor

 -- Data: errno.EXFULL
     Exchange full

 -- Data: errno.ENOANO
     No anode

 -- Data: errno.EBADRQC
     Invalid request code

 -- Data: errno.EBADSLT
     Invalid slot

 -- Data: errno.EDEADLOCK
     File locking deadlock error

 -- Data: errno.EBFONT
     Bad font file format

 -- Data: errno.ENOSTR
     Device not a stream

 -- Data: errno.ENODATA
     No data available

 -- Data: errno.ETIME
     Timer expired

 -- Data: errno.ENOSR
     Out of streams resources

 -- Data: errno.ENONET
     Machine is not on the network

 -- Data: errno.ENOPKG
     Package not installed

 -- Data: errno.EREMOTE
     Object is remote

 -- Data: errno.ENOLINK
     Link has been severed

 -- Data: errno.EADV
     Advertise error

 -- Data: errno.ESRMNT
     Srmount error

 -- Data: errno.ECOMM
     Communication error on send

 -- Data: errno.EPROTO
     Protocol error

 -- Data: errno.EMULTIHOP
     Multihop attempted

 -- Data: errno.EDOTDOT
     RFS specific error

 -- Data: errno.EBADMSG
     Not a data message

 -- Data: errno.EOVERFLOW
     Value too large for defined data type

 -- Data: errno.ENOTUNIQ
     Name not unique on network

 -- Data: errno.EBADFD
     File descriptor in bad state

 -- Data: errno.EREMCHG
     Remote address changed

 -- Data: errno.ELIBACC
     Can not access a needed shared library

 -- Data: errno.ELIBBAD
     Accessing a corrupted shared library

 -- Data: errno.ELIBSCN
     .lib section in a.out corrupted

 -- Data: errno.ELIBMAX
     Attempting to link in too many shared libraries

 -- Data: errno.ELIBEXEC
     Cannot exec a shared library directly

 -- Data: errno.EILSEQ
     Illegal byte sequence

 -- Data: errno.ERESTART
     Interrupted system call should be restarted

 -- Data: errno.ESTRPIPE
     Streams pipe error

 -- Data: errno.EUSERS
     Too many users

 -- Data: errno.ENOTSOCK
     Socket operation on non-socket

 -- Data: errno.EDESTADDRREQ
     Destination address required

 -- Data: errno.EMSGSIZE
     Message too long

 -- Data: errno.EPROTOTYPE
     Protocol wrong type for socket

 -- Data: errno.ENOPROTOOPT
     Protocol not available

 -- Data: errno.EPROTONOSUPPORT
     Protocol not supported

 -- Data: errno.ESOCKTNOSUPPORT
     Socket type not supported

 -- Data: errno.EOPNOTSUPP
     Operation not supported on transport endpoint

 -- Data: errno.EPFNOSUPPORT
     Protocol family not supported

 -- Data: errno.EAFNOSUPPORT
     Address family not supported by protocol

 -- Data: errno.EADDRINUSE
     Address already in use

 -- Data: errno.EADDRNOTAVAIL
     Cannot assign requested address

 -- Data: errno.ENETDOWN
     Network is down

 -- Data: errno.ENETUNREACH
     Network is unreachable

 -- Data: errno.ENETRESET
     Network dropped connection because of reset

 -- Data: errno.ECONNABORTED
     Software caused connection abort

 -- Data: errno.ECONNRESET
     Connection reset by peer

 -- Data: errno.ENOBUFS
     No buffer space available

 -- Data: errno.EISCONN
     Transport endpoint is already connected

 -- Data: errno.ENOTCONN
     Transport endpoint is not connected

 -- Data: errno.ESHUTDOWN
     Cannot send after transport endpoint shutdown

 -- Data: errno.ETOOMANYREFS
     Too many references: cannot splice

 -- Data: errno.ETIMEDOUT
     Connection timed out

 -- Data: errno.ECONNREFUSED
     Connection refused

 -- Data: errno.EHOSTDOWN
     Host is down

 -- Data: errno.EHOSTUNREACH
     No route to host

 -- Data: errno.EALREADY
     Operation already in progress

 -- Data: errno.EINPROGRESS
     Operation now in progress

 -- Data: errno.ESTALE
     Stale NFS file handle

 -- Data: errno.EUCLEAN
     Structure needs cleaning

 -- Data: errno.ENOTNAM
     Not a XENIX named type file

 -- Data: errno.ENAVAIL
     No XENIX semaphores available

 -- Data: errno.EISNAM
     Is a named type file

 -- Data: errno.EREMOTEIO
     Remote I/O error

 -- Data: errno.EDQUOT
     Quota exceeded

5.15.16 `ctypes' -- A foreign function library for Python
---------------------------------------------------------

New in version 2.5.

  ctypes is a foreign function library for Python.  It
provides C compatible data types, and allows calling functions in DLLs
or shared libraries.  It can be used to wrap these libraries in pure
Python.

5.15.16.1 ctypes tutorial
.........................

Note: The code samples in this tutorial use doctest to make
sure that they actually work.  Since some code samples behave
differently under Linux, Windows, or Mac OS X, they contain doctest
directives in comments.

  Note: Some code samples reference the ctypes c_int type.
This type is an alias for the c_long type on 32-bit
systems.  So, you should not be confused if c_long is
printed if you would expect c_int. -- they are actually the
same type.

5.15.16.2 Loading dynamic link libraries
........................................

ctypes exports the _cdll_, and on Windows _windll_ and
_oledll_ objects, for loading dynamic link libraries.

  You load libraries by accessing them as attributes of these objects.
_cdll_ loads libraries which export functions using the standard
`cdecl' calling convention, while _windll_ libraries call functions
using the `stdcall' calling convention. _oledll_ also uses the
`stdcall' calling convention, and assumes the functions return a
Windows `HRESULT' error code. The error code is used to automatically
raise a `WindowsError' exception when the function call fails.

  Here are some examples for Windows. Note that `msvcrt' is the MS
standard C library containing most standard C functions, and uses the
cdecl calling convention:

    >>> from ctypes import *
    >>> print windll.kernel32 # doctest: +WINDOWS
    <WinDLL 'kernel32', handle ... at ...>
    >>> print cdll.msvcrt # doctest: +WINDOWS
    <CDLL 'msvcrt', handle ... at ...>
    >>> libc = cdll.msvcrt # doctest: +WINDOWS
    >>>

Windows appends the usual `.dll' file suffix automatically.

  On Linux, it is required to specify the filename _including_ the
extension to load a library, so attribute access can not be used to
load libraries. Either the `LoadLibrary()' method of the dll loaders
should be used, or you should load the library by creating an instance
of CDLL by calling the constructor:

    >>> cdll.LoadLibrary("libc.so.6") # doctest: +LINUX
    <CDLL 'libc.so.6', handle ... at ...>
    >>> libc = CDLL("libc.so.6")     # doctest: +LINUX
    >>> libc                         # doctest: +LINUX
    <CDLL 'libc.so.6', handle ... at ...>
    >>>

5.15.16.3 Accessing functions from loaded dlls
..............................................

Functions are accessed as attributes of dll objects:

    >>> from ctypes import *
    >>> libc.printf
    <_FuncPtr object at 0x...>
    >>> print windll.kernel32.GetModuleHandleA # doctest: +WINDOWS
    <_FuncPtr object at 0x...>
    >>> print windll.kernel32.MyOwnFunction # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "ctypes.py", line 239, in __getattr__
        func = _StdcallFuncPtr(name, self)
    AttributeError: function 'MyOwnFunction' not found
    >>>

Note that win32 system dlls like `kernel32' and `user32' often export
ANSI as well as UNICODE versions of a function. The UNICODE version is
exported with an `W' appended to the name, while the ANSI version is
exported with an `A' appended to the name. The win32 `GetModuleHandle'
function, which returns a _module handle_ for a given module name, has
the following C prototype, and a macro is used to expose one of them as
`GetModuleHandle' depending on whether UNICODE is defined or not:

    /* ANSI version */
    HMODULE GetModuleHandleA(LPCSTR lpModuleName);
    /* UNICODE version */
    HMODULE GetModuleHandleW(LPCWSTR lpModuleName);

_windll_ does not try to select one of them by magic, you must access
the version you need by specifying `GetModuleHandleA' or
`GetModuleHandleW' explicitly, and then call it with strings or unicode
strings respectively.

  Sometimes, dlls export functions with names which aren't valid Python
identifiers, like `"??2@YAPAXI@Z"'. In this case you have to use
getattr() to retrieve the function:

    >>> getattr(cdll.msvcrt, "??2@YAPAXI@Z") # doctest: +WINDOWS
    <_FuncPtr object at 0x...>
    >>>

On Windows, some dlls export functions not by name but by ordinal. These
functions can be accessed by indexing the dll object with the ordinal
number:

    >>> cdll.kernel32[1] # doctest: +WINDOWS
    <_FuncPtr object at 0x...>
    >>> cdll.kernel32[0] # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "ctypes.py", line 310, in __getitem__
        func = _StdcallFuncPtr(name, self)
    AttributeError: function ordinal 0 not found
    >>>

5.15.16.4 Calling functions
...........................

You can call these functions like any other Python callable. This
example uses the `time()' function, which returns system time in
seconds since the Unix epoch, and the `GetModuleHandleA()' function,
which returns a win32 module handle.

  This example calls both functions with a NULL pointer (`None' should
be used as the NULL pointer):

    >>> print libc.time(None) # doctest: +SKIP
    1150640792
    >>> print hex(windll.kernel32.GetModuleHandleA(None)) # doctest: +WINDOWS
    0x1d000000
    >>>

ctypes tries to protect you from calling functions with the
wrong number of arguments or the wrong calling convention.
Unfortunately this only works on Windows.  It does this by examining
the stack after the function returns, so although an error is raised
the function _has_ been called:

    >>> windll.kernel32.GetModuleHandleA() # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: Procedure probably called with not enough arguments (4 bytes missing)
    >>> windll.kernel32.GetModuleHandleA(0, 0) # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: Procedure probably called with too many arguments (4 bytes in excess)
    >>>

The same exception is raised when you call an `stdcall' function with
the `cdecl' calling convention, or vice versa:

    >>> cdll.kernel32.GetModuleHandleA(None) # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: Procedure probably called with not enough arguments (4 bytes missing)
    >>>

    >>> windll.msvcrt.printf("spam") # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: Procedure probably called with too many arguments (4 bytes in excess)
    >>>

To find out the correct calling convention you have to look into the C
header file or the documentation for the function you want to call.

  On Windows, see ctypes uses win32 structured exception
handling to prevent crashes from general protection faults when
functions are called with invalid argument values:

    >>> windll.kernel32.GetModuleHandleA(32) # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    WindowsError: exception: access violation reading 0x00000020
    >>>

There are, however, enough ways to crash Python with ctypes,
so you should be careful anyway.

  `None', integers, longs, byte strings and unicode strings are the
only native Python objects that can directly be used as parameters in
these function calls.  `None' is passed as a C `NULL' pointer, byte
strings and unicode strings are passed as pointer to the memory block
that contains their data (`char *' or `wchar_t *').  Python integers
and Python longs are passed as the platforms default C `int' type,
their value is masked to fit into the C type.

  Before we move on calling functions with other parameter types, we
have to learn more about ctypes data types.

5.15.16.5 Fundamental data types
................................

ctypes defines a number of primitive C compatible data types
:

ctypes type                C type                                       Python type
---------------------------------------------------------------------------------------------------------
see c_char.        `char'                                       1-character string
c_wchar.       `wchar_t'                                    1-character unicode string
c_byte.        `char'                                       int/long
c_ubyte.       `unsigned char'                              int/long
c_short.       `short'                                      int/long
c_ushort.      `unsigned short'                             int/long
c_int.         `int'                                        int/long
c_uint.        `unsigned int'                               int/long
c_long.        `long'                                       int/long
c_ulong.       `unsigned long'                              int/long
c_longlong.    `__int64' or `long long'                     int/long
c_ulonglong.   `unsigned __int64' or `unsigned long long'   int/long
c_float.       `float'                                      float
c_double.      `double'                                     float
c_longdouble.  `long double'                                float
c_char_p.      `char *' (NUL terminated)                    string or `None'
c_wchar_p.     `wchar_t *' (NUL terminated)                 unicode or `None'
c_void_p.      `void *'                                     int/long or `None'

  All these types can be created by calling them with an optional
initializer of the correct type and value:

    >>> c_int()
    c_long(0)
    >>> c_char_p("Hello, World")
    c_char_p('Hello, World')
    >>> c_ushort(-3)
    c_ushort(65533)
    >>>

Since these types are mutable, their value can also be changed
afterwards:

    >>> i = c_int(42)
    >>> print i
    c_long(42)
    >>> print i.value
    42
    >>> i.value = -99
    >>> print i.value
    -99
    >>>

Assigning a new value to instances of the pointer types c_char_p,
see c_wchar_p, and c_void_p changes the
_memory location_ they point to, _not the contents_ of the memory block
(of course not, because Python strings are immutable):

    >>> s = "Hello, World"
    >>> c_s = c_char_p(s)
    >>> print c_s
    c_char_p('Hello, World')
    >>> c_s.value = "Hi, there"
    >>> print c_s
    c_char_p('Hi, there')
    >>> print s                 # first string is unchanged
    Hello, World
    >>>

You should be careful, however, not to pass them to functions expecting
pointers to mutable memory. If you need mutable memory blocks, ctypes
has a create_string_buffer() function which creates these
in various ways.  The current memory block contents can be accessed (or
changed) with the `raw' property; if you want to access it as NUL
terminated string, use the `value' property:

    >>> from ctypes import *
    >>> p = create_string_buffer(3)      # create a 3 byte buffer, initialized to NUL bytes
    >>> print sizeof(p), repr(p.raw)
    3 '\x00\x00\x00'
    >>> p = create_string_buffer("Hello")      # create a buffer containing a NUL terminated string
    >>> print sizeof(p), repr(p.raw)
    6 'Hello\x00'
    >>> print repr(p.value)
    'Hello'
    >>> p = create_string_buffer("Hello", 10)  # create a 10 byte buffer
    >>> print sizeof(p), repr(p.raw)
    10 'Hello\x00\x00\x00\x00\x00'
    >>> p.value = "Hi"
    >>> print sizeof(p), repr(p.raw)
    10 'Hi\x00lo\x00\x00\x00\x00\x00'
    >>>

The create_string_buffer() function replaces the
`c_buffer()' function (which is still available as an alias), as well
as the `c_string()' function from earlier ctypes releases.  To create a
mutable memory block containing unicode characters of the C type
`wchar_t' use the create_unicode_buffer() function.

5.15.16.6 Calling functions, continued
......................................

Note that printf prints to the real standard output channel, _not_ to
sys.stdout, so these examples will only work at the console
prompt, not from within _IDLE_ or _PythonWin_:

    >>> printf = libc.printf
    >>> printf("Hello, %s\n", "World!")
    Hello, World!
    14
    >>> printf("Hello, %S\n", u"World!")
    Hello, World!
    14
    >>> printf("%d bottles of beer\n", 42)
    42 bottles of beer
    19
    >>> printf("%f bottles of beer\n", 42.5)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ArgumentError: argument 2: exceptions.TypeError: Don't know how to convert parameter 2
    >>>

As has been mentioned before, all Python types except integers,
strings, and unicode strings have to be wrapped in their corresponding
ctypes type, so that they can be converted to the required C
data type:

    >>> printf("An int %d, a double %f\n", 1234, c_double(3.14))
    An int 1234, a double 3.140000
    31
    >>>

5.15.16.7 Calling functions with your own custom data types
...........................................................

You can also customize ctypes argument conversion to allow
instances of your own classes be used as function arguments.
ctypes looks for an `_as_parameter_' attribute and uses this as
the function argument.  Of course, it must be one of integer, string,
or unicode:

    >>> class Bottles(object):
    ...     def __init__(self, number):
    ...         self._as_parameter_ = number
    ...
    >>> bottles = Bottles(42)
    >>> printf("%d bottles of beer\n", bottles)
    42 bottles of beer
    19
    >>>

If you don't want to store the instance's data in the `_as_parameter_'
instance variable, you could define a property() which
makes the data available.

5.15.16.8 Specifying the required argument types (function prototypes)
......................................................................

It is possible to specify the required argument types of functions
exported from DLLs by setting the `argtypes' attribute.

  `argtypes' must be a sequence of C data types (the `printf' function
is probably not a good example here, because it takes a variable number
and different types of parameters depending on the format string, on
the other hand this is quite handy to experiment with this feature):

    >>> printf.argtypes = [c_char_p, c_char_p, c_int, c_double]
    >>> printf("String '%s', Int %d, Double %f\n", "Hi", 10, 2.2)
    String 'Hi', Int 10, Double 2.200000
    37
    >>>

Specifying a format protects against incompatible argument types (just
as a prototype for a C function), and tries to convert the arguments to
valid types:

    >>> printf("%d %d %d", 1, 2, 3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ArgumentError: argument 2: exceptions.TypeError: wrong type
    >>> printf("%s %d %f\n", "X", 2, 3)
    X 2 3.000000
    13
    >>>

If you have defined your own classes which you pass to function calls,
you have to implement a `from_param()' class method for them to be able
to use them in the `argtypes' sequence. The `from_param()' class method
receives the Python object passed to the function call, it should do a
typecheck or whatever is needed to make sure this object is acceptable,
and then return the object itself, its `_as_parameter_' attribute, or
whatever you want to pass as the C function argument in this case.
Again, the result should be an integer, string, unicode, a
ctypes instance, or an object with an `_as_parameter_' attribute.

5.15.16.9 Return types
......................

By default functions are assumed to return the C `int' type.  Other
return types can be specified by setting the `restype' attribute of the
function object.

  Here is a more advanced example, it uses the `strchr' function, which
expects a string pointer and a char, and returns a pointer to a string:

    >>> strchr = libc.strchr
    >>> strchr("abcdef", ord("d")) # doctest: +SKIP
    8059983
    >>> strchr.restype = c_char_p # c_char_p is a pointer to a string
    >>> strchr("abcdef", ord("d"))
    'def'
    >>> print strchr("abcdef", ord("x"))
    None
    >>>

If you want to avoid the `ord("x")' calls above, you can set the
`argtypes' attribute, and the second argument will be converted from a
single character Python string into a C char:

    >>> strchr.restype = c_char_p
    >>> strchr.argtypes = [c_char_p, c_char]
    >>> strchr("abcdef", "d")
    'def'
    >>> strchr("abcdef", "def")
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ArgumentError: argument 2: exceptions.TypeError: one character string expected
    >>> print strchr("abcdef", "x")
    None
    >>> strchr("abcdef", "d")
    'def'
    >>>

You can also use a callable Python object (a function or a class for
example) as the `restype' attribute, if the foreign function returns an
integer.  The callable will be called with the _integer_ the C function
returns, and the result of this call will be used as the result of your
function call. This is useful to check for error return values and
automatically raise an exception:

    >>> GetModuleHandle = windll.kernel32.GetModuleHandleA # doctest: +WINDOWS
    >>> def ValidHandle(value):
    ...     if value == 0:
    ...         raise WinError()
    ...     return value
    ...
    >>>
    >>> GetModuleHandle.restype = ValidHandle # doctest: +WINDOWS
    >>> GetModuleHandle(None) # doctest: +WINDOWS
    486539264
    >>> GetModuleHandle("something silly") # doctest: +WINDOWS
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "<stdin>", line 3, in ValidHandle
    WindowsError: [Errno 126] The specified module could not be found.
    >>>

`WinError' is a function which will call Windows `FormatMessage()' api
to get the string representation of an error code, and _returns_ an
exception.  `WinError' takes an optional error code parameter, if no
one is used, it calls GetLastError() to retrieve it.

  Please note that a much more powerful error checking mechanism is
available through the `errcheck' attribute; see the reference manual
for details.

5.15.16.10 Passing pointers (or: passing parameters by reference)
.................................................................

Sometimes a C api function expects a _pointer_ to a data type as
parameter, probably to write into the corresponding location, or if the
data is too large to be passed by value. This is also known as _passing
parameters by reference_.

  ctypes exports the byref() function which is
used to pass parameters by reference.  The same effect can be achieved
with the pointer() function, although pointer()
does a lot more work since it constructs a real pointer object,
so it is faster to use byref() if you don't need the
pointer object in Python itself:

    >>> i = c_int()
    >>> f = c_float()
    >>> s = create_string_buffer('\000' * 32)
    >>> print i.value, f.value, repr(s.value)
    0 0.0 ''
    >>> libc.sscanf("1 3.14 Hello", "%d %f %s",
    ...             byref(i), byref(f), s)
    3
    >>> print i.value, f.value, repr(s.value)
    1 3.1400001049 'Hello'
    >>>

5.15.16.11 Structures and unions
................................

Structures and unions must derive from the Structure and
Union.  base classes which are defined in the ctypes
module. Each subclass must define a `_fields_' attribute.
`_fields_' must be a list of _2-tuples_, containing a _field name_ and
a _field type_.

  The field type must be a ctypes type like c_int,
or any other derived ctypes type: structure, union,
array, pointer.

  Here is a simple example of a POINT structure, which contains two
integers named _x_ and _y_, and also shows how to initialize a
structure in the constructor:

    >>> from ctypes import *
    >>> class POINT(Structure):
    ...     _fields_ = [("x", c_int),
    ...                 ("y", c_int)]
    ...
    >>> point = POINT(10, 20)
    >>> print point.x, point.y
    10 20
    >>> point = POINT(y=5)
    >>> print point.x, point.y
    0 5
    >>> POINT(1, 2, 3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: too many initializers
    >>>

You can, however, build much more complicated structures. Structures
can itself contain other structures by using a structure as a field
type.

  Here is a RECT structure which contains two POINTs named _upperleft_
and _lowerright_:

    >>> class RECT(Structure):
    ...     _fields_ = [("upperleft", POINT),
    ...                 ("lowerright", POINT)]
    ...
    >>> rc = RECT(point)
    >>> print rc.upperleft.x, rc.upperleft.y
    0 5
    >>> print rc.lowerright.x, rc.lowerright.y
    0 0
    >>>

Nested structures can also be initialized in the constructor in several
ways:

    >>> r = RECT(POINT(1, 2), POINT(3, 4))
    >>> r = RECT((1, 2), (3, 4))

Field descriptor.s can be retrieved from the _class_, they
are useful for debugging because they can provide useful information:

    >>> print POINT.x
    <Field type=c_long, ofs=0, size=4>
    >>> print POINT.y
    <Field type=c_long, ofs=4, size=4>
    >>>

5.15.16.12 Structure/union alignment and byte order
...................................................

By default, Structure and Union fields are aligned in the same way the C
compiler does it. It is possible to override this behavior be
specifying a `_pack_' class attribute in the subclass definition. This
must be set to a positive integer and specifies the maximum alignment
for the fields. This is what `#pragma pack(n)' also does in MSVC.

  ctypes uses the native byte order for Structures and
Unions.  To build structures with non-native byte order, you can use
one of the BigEndianStructure, LittleEndianStructure,
`BigEndianUnion', and `LittleEndianUnion' base classes.  These
classes cannot contain pointer fields.

5.15.16.13 Bit fields in structures and unions
..............................................

It is possible to create structures and unions containing bit fields.
Bit fields are only possible for integer fields, the bit width is
specified as the third item in the `_fields_' tuples:

    >>> class Int(Structure):
    ...     _fields_ = [("first_16", c_int, 16),
    ...                 ("second_16", c_int, 16)]
    ...
    >>> print Int.first_16
    <Field type=c_long, ofs=0:0, bits=16>
    >>> print Int.second_16
    <Field type=c_long, ofs=0:16, bits=16>
    >>>

5.15.16.14 Arrays
.................

Arrays are sequences, containing a fixed number of instances of the
same type.

  The recommended way to create array types is by multiplying a data
type with a positive integer:

    TenPointsArrayType = POINT * 10

Here is an example of an somewhat artificial data type, a structure
containing 4 POINTs among other stuff:

    >>> from ctypes import *
    >>> class POINT(Structure):
    ...    _fields_ = ("x", c_int), ("y", c_int)
    ...
    >>> class MyStruct(Structure):
    ...    _fields_ = [("a", c_int),
    ...                ("b", c_float),
    ...                ("point_array", POINT * 4)]
    >>>
    >>> print len(MyStruct().point_array)
    4
    >>>

Instances are created in the usual way, by calling the class:

    arr = TenPointsArrayType()
    for pt in arr:
        print pt.x, pt.y

The above code print a series of `0 0' lines, because the array
contents is initialized to zeros.

  Initializers of the correct type can also be specified:

    >>> from ctypes import *
    >>> TenIntegers = c_int * 10
    >>> ii = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
    >>> print ii
    <c_long_Array_10 object at 0x...>
    >>> for i in ii: print i,
    ...
    1 2 3 4 5 6 7 8 9 10
    >>>

5.15.16.15 Pointers
...................

Pointer instances are created by calling the pointer().
function on a ctypes type:

    >>> from ctypes import *
    >>> i = c_int(42)
    >>> pi = pointer(i)
    >>>

Pointer instances have a `contents' attribute which returns the object
to which the pointer points, the `i' object above:

    >>> pi.contents
    c_long(42)
    >>>

Note that ctypes does not have OOR (original object return),
it constructs a new, equivalent object each time you retrieve an
attribute:

    >>> pi.contents is i
    False
    >>> pi.contents is pi.contents
    False
    >>>

Assigning another c_int instance to the pointer's contents
attribute would cause the pointer to point to the memory location where
this is stored:

    >>> i = c_int(99)
    >>> pi.contents = i
    >>> pi.contents
    c_long(99)
    >>>

Pointer instances can also be indexed with integers:

    >>> pi[0]
    99
    >>>

Assigning to an integer index changes the pointed to value:

    >>> print i
    c_long(99)
    >>> pi[0] = 22
    >>> print i
    c_long(22)
    >>>

It is also possible to use indexes different from 0, but you must know
what you're doing, just as in C: You can access or change arbitrary
memory locations.  Generally you only use this feature if you receive a
pointer from a C function, and you _know_ that the pointer actually
points to an array instead of a single item.

  Behind the scenes, the pointer() function does more than
simply create pointer instances, it has to create pointer _types_
first.  This is done with the POINTER() function, which
accepts any ctypes type, and returns a new type:

    >>> PI = POINTER(c_int)
    >>> PI
    <class 'ctypes.LP_c_long'>
    >>> PI(42)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: expected c_long instead of int
    >>> PI(c_int(42))
    <ctypes.LP_c_long object at 0x...>
    >>>

Calling the pointer type without an argument creates a `NULL' pointer.
`NULL' pointers have a `False' boolean value:

    >>> null_ptr = POINTER(c_int)()
    >>> print bool(null_ptr)
    False
    >>>

ctypes checks for `NULL' when dereferencing pointers (but
dereferencing invalid non-`NULL' pointers would crash Python):

    >>> null_ptr[0]
    Traceback (most recent call last):
        ....
    ValueError: NULL pointer access
    >>>

    >>> null_ptr[0] = 1234
    Traceback (most recent call last):
        ....
    ValueError: NULL pointer access
    >>>

5.15.16.16 Type conversions
...........................

Usually, ctypes does strict type checking.  This means, if you have
`POINTER(c_int)' in the `argtypes' list of a function or as the type of
a member field in a structure definition, only instances of exactly the
same type are accepted.  There are some exceptions to this rule, where
ctypes accepts other objects.  For example, you can pass compatible
array instances instead of pointer types.  So, for `POINTER(c_int)',
ctypes accepts an array of c_int:

    >>> class Bar(Structure):
    ...     _fields_ = [("count", c_int), ("values", POINTER(c_int))]
    ...
    >>> bar = Bar()
    >>> bar.values = (c_int * 3)(1, 2, 3)
    >>> bar.count = 3
    >>> for i in range(bar.count):
    ...     print bar.values[i]
    ...
    1
    2
    3
    >>>

To set a POINTER type field to `NULL', you can assign `None':

    >>> bar.values = None
    >>>

Sometimes you have instances of incompatible types.  In C, you can cast
one type into another type.  ctypes provides a cast()
function which can be used in the same way.  The `Bar' structure
defined above accepts `POINTER(c_int)' pointers or c_int.
arrays for its `values' field, but not instances of other types:

    >>> bar.values = (c_byte * 4)()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: incompatible types, c_byte_Array_4 instance instead of LP_c_long instance
    >>>

For these cases, the cast() function is handy.

  The cast() function can be used to cast a ctypes
instance into a pointer to a different ctypes data type.  cast()
takes two parameters, a ctypes object that is or can be converted
to a pointer of some kind, and a ctypes pointer type.  It returns an
instance of the second argument, which references the same memory block
as the first argument:

    >>> a = (c_byte * 4)()
    >>> cast(a, POINTER(c_int))
    <ctypes.LP_c_long object at ...>
    >>>

So, see cast() can be used to assign to the `values' field of
`Bar' the structure:

    >>> bar = Bar()
    >>> bar.values = cast((c_byte * 4)(), POINTER(c_int))
    >>> print bar.values[0]
    0
    >>>

5.15.16.17 Incomplete Types
...........................

_Incomplete Types_ are structures, unions or arrays whose members are
not yet specified. In C, they are specified by forward declarations,
which are defined later:

    struct cell; /* forward declaration */

    struct {
        char *name;
        struct cell *next;
    } cell;

The straightforward translation into ctypes code would be this, but it
does not work:

    >>> class cell(Structure):
    ...     _fields_ = [("name", c_char_p),
    ...                 ("next", POINTER(cell))]
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "<stdin>", line 2, in cell
    NameError: name 'cell' is not defined
    >>>

because the new `class cell' is not available in the class statement
itself.  In ctypes, we can define the `cell' class and set
the `_fields_' attribute later, after the class statement:

    >>> from ctypes import *
    >>> class cell(Structure):
    ...     pass
    ...
    >>> cell._fields_ = [("name", c_char_p),
    ...                  ("next", POINTER(cell))]
    >>>

Lets try it. We create two instances of `cell', and let them point to
each other, and finally follow the pointer chain a few times:

    >>> c1 = cell()
    >>> c1.name = "foo"
    >>> c2 = cell()
    >>> c2.name = "bar"
    >>> c1.next = pointer(c2)
    >>> c2.next = pointer(c1)
    >>> p = c1
    >>> for i in range(8):
    ...     print p.name,
    ...     p = p.next[0]
    ...
    foo bar foo bar foo bar foo bar
    >>>

5.15.16.18 Callback functions
.............................

ctypes allows to create C callable function pointers from
Python callables.  These are sometimes called _callback functions_.

  First, you must create a class for the callback function, the class
knows the calling convention, the return type, and the number and types
of arguments this function will receive.

  The CFUNCTYPE factory function creates types for callback functions
using the normal cdecl calling convention, and, on Windows, the
WINFUNCTYPE factory function creates types for callback functions using
the stdcall calling convention.

  Both of these factory functions are called with the result type as
first argument, and the callback functions expected argument types as
the remaining arguments.

  I will present an example here which uses the standard C library's
`qsort()' function, this is used to sort items with the help of a
callback function.  `qsort()' will be used to sort an array of integers:

    >>> IntArray5 = c_int * 5
    >>> ia = IntArray5(5, 1, 7, 33, 99)
    >>> qsort = libc.qsort
    >>> qsort.restype = None
    >>>

`qsort()' must be called with a pointer to the data to sort, the number
of items in the data array, the size of one item, and a pointer to the
comparison function, the callback. The callback will then be called
with two pointers to items, and it must return a negative integer if
the first item is smaller than the second, a zero if they are equal,
and a positive integer else.

  So our callback function receives pointers to integers, and must
return an integer. First we create the `type' for the callback function:

    >>> CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int))
    >>>

For the first implementation of the callback function, we simply print
the arguments we get, and return 0 (incremental development ;-):

    >>> def py_cmp_func(a, b):
    ...     print "py_cmp_func", a, b
    ...     return 0
    ...
    >>>

Create the C callable callback:

    >>> cmp_func = CMPFUNC(py_cmp_func)
    >>>

And we're ready to go:

    >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +WINDOWS
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...>
    >>>

We know how to access the contents of a pointer, so lets redefine our
callback:

    >>> def py_cmp_func(a, b):
    ...     print "py_cmp_func", a[0], b[0]
    ...     return 0
    ...
    >>> cmp_func = CMPFUNC(py_cmp_func)
    >>>

Here is what we get on Windows:

    >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +WINDOWS
    py_cmp_func 7 1
    py_cmp_func 33 1
    py_cmp_func 99 1
    py_cmp_func 5 1
    py_cmp_func 7 5
    py_cmp_func 33 5
    py_cmp_func 99 5
    py_cmp_func 7 99
    py_cmp_func 33 99
    py_cmp_func 7 33
    >>>

It is funny to see that on linux the sort function seems to work much
more efficiently, it is doing less comparisons:

    >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +LINUX
    py_cmp_func 5 1
    py_cmp_func 33 99
    py_cmp_func 7 33
    py_cmp_func 5 7
    py_cmp_func 1 7
    >>>

Ah, we're nearly done! The last step is to actually compare the two
items and return a useful result:

    >>> def py_cmp_func(a, b):
    ...     print "py_cmp_func", a[0], b[0]
    ...     return a[0] - b[0]
    ...
    >>>

Final run on Windows:

    >>> qsort(ia, len(ia), sizeof(c_int), CMPFUNC(py_cmp_func)) # doctest: +WINDOWS
    py_cmp_func 33 7
    py_cmp_func 99 33
    py_cmp_func 5 99
    py_cmp_func 1 99
    py_cmp_func 33 7
    py_cmp_func 1 33
    py_cmp_func 5 33
    py_cmp_func 5 7
    py_cmp_func 1 7
    py_cmp_func 5 1
    >>>

and on Linux:

    >>> qsort(ia, len(ia), sizeof(c_int), CMPFUNC(py_cmp_func)) # doctest: +LINUX
    py_cmp_func 5 1
    py_cmp_func 33 99
    py_cmp_func 7 33
    py_cmp_func 1 7
    py_cmp_func 5 7
    >>>

It is quite interesting to see that the Windows `qsort()' function needs
more comparisons than the linux version!

  As we can easily check, our array is sorted now:

    >>> for i in ia: print i,
    ...
    1 5 7 33 99
    >>>

*Important note for callback functions:*

  Make sure you keep references to CFUNCTYPE objects as long as they
are used from C code. ctypes doesn't, and if you don't, they
may be garbage collected, crashing your program when a callback is made.

5.15.16.19 Accessing values exported from dlls
..............................................

Some shared libraries not only export functions, they also export
variables. An example in the Python library itself is the
`Py_OptimizeFlag', an integer set to 0, 1, or 2, depending on the
-O or -OO flag given on startup.

  ctypes can access values like this with the `in_dll()'
class methods of the type.  _pythonapi_ is a predefined symbol giving
access to the Python C api:

    >>> opt_flag = c_int.in_dll(pythonapi, "Py_OptimizeFlag")
    >>> print opt_flag
    c_long(0)
    >>>

If the interpreter would have been started with -O, the
sample would have printed `c_long(1)', or `c_long(2)' if -OO
would have been specified.

  An extended example which also demonstrates the use of pointers
accesses the `PyImport_FrozenModules' pointer exported by Python.

  Quoting the Python docs: _This pointer is initialized to point to an
array of "struct _frozen" records, terminated by one whose members are
all NULL or zero.  When a frozen module is imported, it is searched in
this table. Third-party code could play tricks with this to provide a
dynamically created collection of frozen modules._

  So manipulating this pointer could even prove useful. To restrict the
example size, we show only how this table can be read with
ctypes.:

    >>> from ctypes import *
    >>>
    >>> class struct_frozen(Structure):
    ...     _fields_ = [("name", c_char_p),
    ...                 ("code", POINTER(c_ubyte)),
    ...                 ("size", c_int)]
    ...
    >>>

We have defined the `struct _frozen' data type, so we can get the
pointer to the table:

    >>> FrozenTable = POINTER(struct_frozen)
    >>> table = FrozenTable.in_dll(pythonapi, "PyImport_FrozenModules")
    >>>

Since `table' is a `pointer' to the array of `struct_frozen' records, we
can iterate over it, but we just have to make sure that our loop
terminates, because pointers have no size. Sooner or later it would
probably crash with an access violation or whatever, so it's better to
break out of the loop when we hit the NULL entry:

    >>> for item in table:
    ...    print item.name, item.size
    ...    if item.name is None:
    ...        break
    ...
    __hello__ 104
    __phello__ -104
    __phello__.spam 104
    None 0
    >>>

The fact that standard Python has a frozen module and a frozen package
(indicated by the negative size member) is not well known, it is only
used for testing. Try it out with `import __hello__' for example.

5.15.16.20 Surprises
....................

There are some edges in ctypes where you may be expect
something else than what actually happens.

  Consider the following example:

    >>> from ctypes import *
    >>> class POINT(Structure):
    ...     _fields_ = ("x", c_int), ("y", c_int)
    ...
    >>> class RECT(Structure):
    ...     _fields_ = ("a", POINT), ("b", POINT)
    ...
    >>> p1 = POINT(1, 2)
    >>> p2 = POINT(3, 4)
    >>> rc = RECT(p1, p2)
    >>> print rc.a.x, rc.a.y, rc.b.x, rc.b.y
    1 2 3 4
    >>> # now swap the two points
    >>> rc.a, rc.b = rc.b, rc.a
    >>> print rc.a.x, rc.a.y, rc.b.x, rc.b.y
    3 4 3 4
    >>>

Hm. We certainly expected the last statement to print `3 4 1 2'. What
happened? Here are the steps of the `rc.a, rc.b = rc.b, rc.a' line
above:

    >>> temp0, temp1 = rc.b, rc.a
    >>> rc.a = temp0
    >>> rc.b = temp1
    >>>

Note that `temp0' and `temp1' are objects still using the internal
buffer of the `rc' object above. So executing `rc.a = temp0' copies the
buffer contents of `temp0' into `rc' 's buffer.  This, in turn, changes
the contents of `temp1'. So, the last assignment `rc.b = temp1',
doesn't have the expected effect.

  Keep in mind that retrieving sub-objects from Structure, Unions, and
Arrays doesn't _copy_ the sub-object, instead it retrieves a wrapper
object accessing the root-object's underlying buffer.

  Another example that may behave different from what one would expect
is this:

    >>> s = c_char_p()
    >>> s.value = "abc def ghi"
    >>> s.value
    'abc def ghi'
    >>> s.value is s.value
    False
    >>>

Why is it printing `False'?  ctypes instances are objects containing a
memory block plus some descriptor.s accessing the contents
of the memory.  Storing a Python object in the memory block does not
store the object itself, instead the `contents' of the object is
stored.  Accessing the contents again constructs a new Python object
each time!

5.15.16.21 Variable-sized data types
....................................

ctypes provides some support for variable-sized arrays and
structures.

  The resize() function can be used to resize the memory
buffer of an existing ctypes object.  The function takes the object as
first argument, and the requested size in bytes as the second argument.
The memory block cannot be made smaller than the natural memory block
specified by the objects type, a ValueError is raised if
this is tried:

    >>> short_array = (c_short * 4)()
    >>> print sizeof(short_array)
    8
    >>> resize(short_array, 4)
    Traceback (most recent call last):
        ...
    ValueError: minimum size is 8
    >>> resize(short_array, 32)
    >>> sizeof(short_array)
    32
    >>> sizeof(type(short_array))
    8
    >>>

This is nice and fine, but how would one access the additional elements
contained in this array?  Since the type still only knows about 4
elements, we get errors accessing other elements:

    >>> short_array[:]
    [0, 0, 0, 0]
    >>> short_array[7]
    Traceback (most recent call last):
        ...
    IndexError: invalid index
    >>>

Another way to use variable-sized data types with ctypes is
to use the dynamic nature of Python, and (re-)define the data type
after the required size is already known, on a case by case basis.

5.15.16.22 ctypes reference
...........................

5.15.16.23 Finding shared libraries
...................................

When programming in a compiled language, shared libraries are accessed
when compiling/linking a program, and when the program is run.

  The purpose of the `find_library()' function is to locate a library
in a way similar to what the compiler does (on platforms with several
versions of a shared library the most recent should be loaded), while
the ctypes library loaders act like when a program is run, and call the
runtime loader directly.

  The `ctypes.util' module provides a function which can help to
determine the library to load.

 -- Data: ctypes.util.find_library (name)
     Try to find a library and return a pathname.  _name_ is the
     library name without any prefix like _lib_, suffix like `.so',
     `.dylib' or version number (this is the form used for the posix
     linker option `-l').  If no library can be found, returns `None'.

  The exact functionality is system dependent.

  On Linux, `find_library()' tries to run external programs
(`/sbin/ldconfig', `gcc', and `objdump') to find the library file.  It
returns the filename of the library file.  Here are some examples:

    >>> from ctypes.util import find_library
    >>> find_library("m")
    'libm.so.6'
    >>> find_library("c")
    'libc.so.6'
    >>> find_library("bz2")
    'libbz2.so.1.0'
    >>>

On OS X, `find_library()' tries several predefined naming schemes and
paths to locate the library, and returns a full pathname if successful:

    >>> from ctypes.util import find_library
    >>> find_library("c")
    '/usr/lib/libc.dylib'
    >>> find_library("m")
    '/usr/lib/libm.dylib'
    >>> find_library("bz2")
    '/usr/lib/libbz2.dylib'
    >>> find_library("AGL")
    '/System/Library/Frameworks/AGL.framework/AGL'
    >>>

On Windows, `find_library()' searches along the system search path, and
returns the full pathname, but since there is no predefined naming
scheme a call like `find_library("c")' will fail and return `None'.

  If wrapping a shared library with ctypes, it _may_ be
better to determine the shared library name at development type, and
hardcode that into the wrapper module instead of using `find_library()'
to locate the library at runtime.

5.15.16.24 Loading shared libraries
...................................

There are several ways to loaded shared libraries into the Python
process.  One way is to instantiate one of the following classes:

 -- Class: ctypes.CDLL (name, mode=DEFAULT_MODE, handle=None,
          use_errno=False, use_last_error=False)
     Instances of this class represent loaded shared libraries.
     Functions in these libraries use the standard C calling
     convention, and are assumed to return `int'.

 -- Class: ctypes.OleDLL (name, mode=DEFAULT_MODE, handle=None,
          use_errno=False, use_last_error=False)
     Windows only: Instances of this class represent loaded shared
     libraries, functions in these libraries use the `stdcall' calling
     convention, and are assumed to return the windows specific
     HRESULT code.  See HRESULT.  values contain
     information specifying whether the function call failed or
     succeeded, together with additional error code.  If the return
     value signals a failure, an `WindowsError' is automatically raised.

 -- Class: ctypes.WinDLL (name, mode=DEFAULT_MODE, handle=None,
          use_errno=False, use_last_error=False)
     Windows only: Instances of this class represent loaded shared
     libraries, functions in these libraries use the `stdcall' calling
     convention, and are assumed to return `int' by default.

     On Windows CE only the standard calling convention is used, for
     convenience the WinDLL and OleDLL use
     the standard calling convention on this platform.

  The Python global interpreter lock is released before
calling any function exported by these libraries, and reacquired
afterwards.

 -- Class: ctypes.PyDLL (name, mode=DEFAULT_MODE, handle=None)
     Instances of this class behave like CDLL instances,
     except that the Python GIL is _not_ released during the function
     call, and after the function execution the Python error flag is
     checked. If the error flag is set, a Python exception is raised.

     Thus, this is only useful to call Python C api functions directly.

  All these classes can be instantiated by calling them with at least
one argument, the pathname of the shared library.  If you have an
existing handle to an already loaded shared library, it can be passed
as the `handle' named parameter, otherwise the underlying platforms
`dlopen' or `LoadLibrary' function is used to load the library into the
process, and to get a handle to it.

  The _mode_ parameter can be used to specify how the library is
loaded.  For details, consult the `dlopen(3)' manpage, on Windows,
_mode_ is ignored.

  The _use_errno_ parameter, when set to True, enables a ctypes
mechanism that allows to access the system errno error
number in a safe way.  ctypes maintains a thread-local copy
of the systems errno.  variable; if you call foreign
functions created with `use_errno=True' then the errno value
before the function call is swapped with the ctypes private copy, the
same happens immediately after the function call.

  The function ctypes.get_errno() returns the value of the
ctypes private copy, and the function ctypes.set_errno().
changes the ctypes private copy to a new value and returns the former
value.

  The _use_last_error_ parameter, when set to True, enables the same
mechanism for the Windows error code which is managed by the
GetLastError() and `SetLastError()' Windows API functions; see
ctypes.get_last_error() and ctypes.set_last_error().
are used to request and change the ctypes private copy of the windows
error code.

  New in version 2.6: The _use_last_error_ and _use_errno_ optional
parameters were added.

 -- Data: ctypes.RTLD_GLOBAL
     Flag to use as _mode_ parameter.  On platforms where this flag is
     not available, it is defined as the integer zero.

 -- Data: ctypes.RTLD_LOCAL
     Flag to use as _mode_ parameter.  On platforms where this is not
     available, it is the same as _RTLD_GLOBAL_.

 -- Data: ctypes.DEFAULT_MODE
     The default mode which is used to load shared libraries.  On OSX
     10.3, this is _RTLD_GLOBAL_, otherwise it is the same as
     _RTLD_LOCAL_.

  Instances of these classes have no public methods, however
__getattr__().  and __getitem__() have special
behavior: functions exported by the shared library can be accessed as
attributes of by index.  Please note that both __getattr__()
and __getitem__() cache their result, so calling them
repeatedly returns the same object each time.

  The following public attributes are available, their name starts with
an underscore to not clash with exported function names:

 -- Attribute: PyDLL._handle
     The system handle used to access the library.

 -- Attribute: PyDLL._name
     The name of the library passed in the constructor.

  Shared libraries can also be loaded by using one of the prefabricated
objects, which are instances of the LibraryLoader class,
either by calling the `LoadLibrary()' method, or by retrieving the
library as attribute of the loader instance.

 -- Class: ctypes.LibraryLoader (dlltype)
     Class which loads shared libraries.  _dlltype_ should be one of the
     CDLL, PyDLL, WinDLL, or
     OleDLL types.

     __getattr__() has special behavior: It allows to load
     a shared library by accessing it as attribute of a library loader
     instance.  The result is cached, so repeated attribute accesses
     return the same library each time.

      -- Method: LoadLibrary (name)
          Load a shared library into the process and return it.  This
          method always returns a new instance of the library.

  These prefabricated library loaders are available:

 -- Data: ctypes.cdll
     Creates CDLL instances.

 -- Data: ctypes.windll
     Windows only: Creates WinDLL instances.

 -- Data: ctypes.oledll
     Windows only: Creates OleDLL instances.

 -- Data: ctypes.pydll
     Creates PyDLL instances.

  For accessing the C Python api directly, a ready-to-use Python shared
library object is available:

 -- Data: ctypes.pythonapi
     An instance of PyDLL that exposes Python C API
     functions as attributes.  Note that all these functions are
     assumed to return C `int', which is of course not always the
     truth, so you have to assign the correct `restype' attribute to
     use these functions.

5.15.16.25 Foreign functions
............................

As explained in the previous section, foreign functions can be accessed
as attributes of loaded shared libraries.  The function objects created
in this way by default accept any number of arguments, accept any
ctypes data instances as arguments, and return the default result type
specified by the library loader.  They are instances of a private class:

 -- Class: ctypes._FuncPtr
     Base class for C callable foreign functions.

     Instances of foreign functions are also C compatible data types;
     they represent C function pointers.

     This behavior can be customized by assigning to special attributes
     of the foreign function object.

      -- Attribute: restype
          Assign a ctypes type to specify the result type of the
          foreign function.  Use `None' for `void', a function not
          returning anything.

          It is possible to assign a callable Python object that is not
          a ctypes type, in this case the function is assumed to return
          a C `int', and the callable will be called with this integer,
          allowing to do further processing or error checking.  Using
          this is deprecated, for more flexible post processing or
          error checking use a ctypes data type as restype.
          and assign a callable to the errcheck attribute.

      -- Attribute: argtypes
          Assign a tuple of ctypes types to specify the argument types
          that the function accepts.  Functions using the `stdcall'
          calling convention can only be called with the same number of
          arguments as the length of this tuple; functions using the C
          calling convention accept additional, unspecified arguments
          as well.

          When a foreign function is called, each actual argument is
          passed to the `from_param()' class method of the items in the
          argtypes.  tuple, this method allows to adapt the
          actual argument to an object that the foreign function
          accepts.  For example, a c_char_p item in the
          argtypes tuple will convert a unicode string
          passed as argument into an byte string using ctypes
          conversion rules.

          New: It is now possible to put items in argtypes which are
          not ctypes types, but each item must have a `from_param()'
          method which returns a value usable as argument (integer,
          string, ctypes instance).  This allows to define adapters
          that can adapt custom objects as function parameters.

      -- Attribute: errcheck
          Assign a Python function or another callable to this
          attribute. The callable will be called with three or more
          arguments:

           -- Function: callable (result, func, arguments)
               _result_ is what the foreign function returns, as
               specified by the restype attribute.

               _func_ is the foreign function object itself, this
               allows to reuse the same callable object to check or
               post process the results of several functions.

               _arguments_ is a tuple containing the parameters
               originally passed to the function call, this allows to
               specialize the behavior on the arguments used.

          The object that this function returns will be returned from
          the foreign function call, but it can also check the result
          value and raise an exception if the foreign function call
          failed.

 -- Exception: ctypes.ArgumentError
     This exception is raised when a foreign function call cannot
     convert one of the passed arguments.

5.15.16.26 Function prototypes
..............................

Foreign functions can also be created by instantiating function
prototypes.  Function prototypes are similar to function prototypes in
C; they describe a function (return type, argument types, calling
convention) without defining an implementation.  The factory functions
must be called with the desired result type and the argument types of
the function.

 -- Function: ctypes.CFUNCTYPE (restype, *argtypes, use_errno=False,
          use_last_error=False)
     The returned function prototype creates functions that use the
     standard C calling convention.  The function will release the GIL
     during the call.  If _use_errno_ is set to True, the ctypes
     private copy of the system errno variable is exchanged
     with the real errno value before and after the call;
     _use_last_error_ does the same for the Windows error code.

     Changed in version 2.6: The optional _use_errno_ and
     _use_last_error_ parameters were added.

 -- Function: ctypes.WINFUNCTYPE (restype, *argtypes, use_errno=False,
          use_last_error=False)
     Windows only: The returned function prototype creates functions
     that use the `stdcall' calling convention, except on Windows CE
     where WINFUNCTYPE() is the same as CFUNCTYPE().
     The function will release the GIL during the call.
     _use_errno_ and _use_last_error_ have the same meaning as above.

 -- Function: ctypes.PYFUNCTYPE (restype, *argtypes)
     The returned function prototype creates functions that use the
     Python calling convention.  The function will _not_ release the
     GIL during the call.

  Function prototypes created by these factory functions can be
instantiated in different ways, depending on the type and number of the
parameters in the call:

      -- Function: prototype (address)
          Returns a foreign function at the specified address which
          must be an integer.

      -- Function: prototype (callable)
          Create a C callable function (a callback function) from a
          Python _callable_.

      -- Function: prototype (func_spec[, paramflags])
          Returns a foreign function exported by a shared library.
          _func_spec_ must be a 2-tuple `(name_or_ordinal, library)'.
          The first item is the name of the exported function as
          string, or the ordinal of the exported function as small
          integer.  The second item is the shared library instance.

      -- Function: prototype (vtbl_index, name[, paramflags[, iid]])
          Returns a foreign function that will call a COM method.
          _vtbl_index_ is the index into the virtual function table, a
          small non-negative integer. _name_ is name of the COM method.
          _iid_ is an optional pointer to the interface identifier
          which is used in extended error reporting.

          COM methods use a special calling convention: They require a
          pointer to the COM interface as first argument, in addition
          to those parameters that are specified in the `argtypes'
          tuple.

     The optional _paramflags_ parameter creates foreign function
     wrappers with much more functionality than the features described
     above.

     _paramflags_ must be a tuple of the same length as `argtypes'.

     Each item in this tuple contains further information about a
     parameter, it must be a tuple containing one, two, or three items.

     The first item is an integer containing a combination of direction
     flags for the parameter:

         1
               Specifies an input parameter to the function.

         2
               Output parameter.  The foreign function fills in a value.

         4
               Input parameter which defaults to the integer zero.

     The optional second item is the parameter name as string.  If this
     is specified, the foreign function can be called with named
     parameters.

     The optional third item is the default value for this parameter.

  This example demonstrates how to wrap the Windows `MessageBoxA'
function so that it supports default parameters and named arguments.
The C declaration from the windows header file is this:

    WINUSERAPI int WINAPI
    MessageBoxA(
        HWND hWnd ,
        LPCSTR lpText,
        LPCSTR lpCaption,
        UINT uType);

Here is the wrapping with ctypes.:

    >>> from ctypes import c_int, WINFUNCTYPE, windll
    >>> from ctypes.wintypes import HWND, LPCSTR, UINT
    >>> prototype = WINFUNCTYPE(c_int, HWND, LPCSTR, LPCSTR, UINT)
    >>> paramflags = (1, "hwnd", 0), (1, "text", "Hi"), (1, "caption", None), (1, "flags", 0)
    >>> MessageBox = prototype(("MessageBoxA", windll.user32), paramflags)
    >>>

The MessageBox foreign function can now be called in these ways:

    >>> MessageBox()
    >>> MessageBox(text="Spam, spam, spam")
    >>> MessageBox(flags=2, text="foo bar")
    >>>

A second example demonstrates output parameters.  The win32
`GetWindowRect' function retrieves the dimensions of a specified window
by copying them into `RECT' structure that the caller has to supply.
Here is the C declaration:

    WINUSERAPI BOOL WINAPI
    GetWindowRect(
         HWND hWnd,
         LPRECT lpRect);

Here is the wrapping with ctypes.:

    >>> from ctypes import POINTER, WINFUNCTYPE, windll, WinError
    >>> from ctypes.wintypes import BOOL, HWND, RECT
    >>> prototype = WINFUNCTYPE(BOOL, HWND, POINTER(RECT))
    >>> paramflags = (1, "hwnd"), (2, "lprect")
    >>> GetWindowRect = prototype(("GetWindowRect", windll.user32), paramflags)
    >>>

Functions with output parameters will automatically return the output
parameter value if there is a single one, or a tuple containing the
output parameter values when there are more than one, so the
GetWindowRect function now returns a RECT instance, when called.

  Output parameters can be combined with the `errcheck' protocol to do
further output processing and error checking.  The win32
`GetWindowRect' api function returns a `BOOL' to signal success or
failure, so this function could do the error checking, and raises an
exception when the api call failed:

    >>> def errcheck(result, func, args):
    ...     if not result:
    ...         raise WinError()
    ...     return args
    ...
    >>> GetWindowRect.errcheck = errcheck
    >>>

If the `errcheck' function returns the argument tuple it receives
unchanged, see ctypes continues the normal processing it does on
the output parameters.  If you want to return a tuple of window
coordinates instead of a `RECT' instance, you can retrieve the fields
in the function and return them instead, the normal processing will no
longer take place:

    >>> def errcheck(result, func, args):
    ...     if not result:
    ...         raise WinError()
    ...     rc = args[1]
    ...     return rc.left, rc.top, rc.bottom, rc.right
    ...
    >>> GetWindowRect.errcheck = errcheck
    >>>

5.15.16.27 Utility functions
............................

 -- Function: ctypes.addressof (obj)
     Returns the address of the memory buffer as integer.  _obj_ must
     be an instance of a ctypes type.

 -- Function: ctypes.alignment (obj_or_type)
     Returns the alignment requirements of a ctypes type. _obj_or_type_
     must be a ctypes type or instance.

 -- Function: ctypes.byref (obj[, offset])
     Returns a light-weight pointer to _obj_, which must be an instance
     of a ctypes type.  _offset_ defaults to zero, and must be an
     integer that will be added to the internal pointer value.

     `byref(obj, offset)' corresponds to this C code:

         (((char *)&obj) + offset)

     The returned object can only be used as a foreign function call
     parameter.  It behaves similar to `pointer(obj)', but the
     construction is a lot faster.

     New in version 2.6: The _offset_ optional argument was added.

 -- Function: ctypes.cast (obj, type)
     This function is similar to the cast operator in C.  It returns a
     new instance of _type_ which points to the same memory block as
     _obj_.  _type_ must be a pointer type, and _obj_ must be an object
     that can be interpreted as a pointer.

 -- Function: ctypes.create_string_buffer (init_or_size[, size])
     This function creates a mutable character buffer. The returned
     object is a ctypes array of c_char.

     _init_or_size_ must be an integer which specifies the size of the
     array, or a string which will be used to initialize the array
     items.

     If a string is specified as first argument, the buffer is made one
     item larger than the length of the string so that the last element
     in the array is a NUL termination character. An integer can be
     passed as second argument which allows to specify the size of the
     array if the length of the string should not be used.

     If the first parameter is a unicode string, it is converted into
     an 8-bit string according to ctypes conversion rules.

 -- Function: ctypes.create_unicode_buffer (init_or_size[, size])
     This function creates a mutable unicode character buffer. The
     returned object is a ctypes array of c_wchar.

     _init_or_size_ must be an integer which specifies the size of the
     array, or a unicode string which will be used to initialize the
     array items.

     If a unicode string is specified as first argument, the buffer is
     made one item larger than the length of the string so that the
     last element in the array is a NUL termination character. An
     integer can be passed as second argument which allows to specify
     the size of the array if the length of the string should not be
     used.

     If the first parameter is a 8-bit string, it is converted into an
     unicode string according to ctypes conversion rules.

 -- Function: ctypes.DllCanUnloadNow ()
     Windows only: This function is a hook which allows to implement
     in-process COM servers with ctypes.  It is called from the
     DllCanUnloadNow function that the _ctypes extension dll exports.

 -- Function: ctypes.DllGetClassObject ()
     Windows only: This function is a hook which allows to implement
     in-process COM servers with ctypes.  It is called from the
     DllGetClassObject function that the `_ctypes' extension dll
     exports.

 -- Function: ctypes.util.find_library (name)
     Try to find a library and return a pathname.  _name_ is the
     library name without any prefix like `lib', suffix like `.so',
     `.dylib' or version number (this is the form used for the posix
     linker option `-l').  If no library can be found, returns `None'.

     The exact functionality is system dependent.

     Changed in version 2.6: Windows only: `find_library("m")' or
     `find_library("c")' return the result of a call to `find_msvcrt()'.

 -- Function: ctypes.util.find_msvcrt ()
     Windows only: return the filename of the VC runtype library used
     by Python, and by the extension modules.  If the name of the
     library cannot be determined, `None' is returned.

     If you need to free memory, for example, allocated by an extension
     module with a call to the `free(void *)', it is important that you
     use the function in the same library that allocated the memory.

     New in version 2.6.

 -- Function: ctypes.FormatError ([code])
     Windows only: Returns a textual description of the error code
     _code_.  If no error code is specified, the last error code is
     used by calling the Windows api function GetLastError.

 -- Function: ctypes.GetLastError ()
     Windows only: Returns the last error code set by Windows in the
     calling thread.  This function calls the Windows `GetLastError()'
     function directly, it does not return the ctypes-private copy of
     the error code.

 -- Function: ctypes.get_errno ()
     Returns the current value of the ctypes-private copy of the system
     errno variable in the calling thread.

     New in version 2.6.

 -- Function: ctypes.get_last_error ()
     Windows only: returns the current value of the ctypes-private copy
     of the system `LastError' variable in the calling thread.

     New in version 2.6.

 -- Function: ctypes.memmove (dst, src, count)
     Same as the standard C memmove library function: copies _count_
     bytes from _src_ to _dst_. _dst_ and _src_ must be integers or
     ctypes instances that can be converted to pointers.

 -- Function: ctypes.memset (dst, c, count)
     Same as the standard C memset library function: fills the memory
     block at address _dst_ with _count_ bytes of value _c_. _dst_ must
     be an integer specifying an address, or a ctypes instance.

 -- Function: ctypes.POINTER (type)
     This factory function creates and returns a new ctypes pointer
     type. Pointer types are cached an reused internally, so calling
     this function repeatedly is cheap. _type_ must be a ctypes type.

 -- Function: ctypes.pointer (obj)
     This function creates a new pointer instance, pointing to _obj_.
     The returned object is of the type `POINTER(type(obj))'.

     Note: If you just want to pass a pointer to an object to a foreign
     function call, you should use `byref(obj)' which is much faster.

 -- Function: ctypes.resize (obj, size)
     This function resizes the internal memory buffer of _obj_, which
     must be an instance of a ctypes type.  It is not possible to make
     the buffer smaller than the native size of the objects type, as
     given by `sizeof(type(obj))', but it is possible to enlarge the
     buffer.

 -- Function: ctypes.set_conversion_mode (encoding, errors)
     This function sets the rules that ctypes objects use when
     converting between 8-bit strings and unicode strings.  _encoding_
     must be a string specifying an encoding, like `'utf-8'' or
     `'mbcs'', _errors_ must be a string specifying the error handling
     on encoding/decoding errors.  Examples of possible values are
     `"strict"', `"replace"', or `"ignore"'.

     set_conversion_mode() returns a 2-tuple containing
     the previous conversion rules.  On windows, the initial conversion
     rules are `('mbcs', 'ignore')', on other systems `('ascii',
     'strict')'.

 -- Function: ctypes.set_errno (value)
     Set the current value of the ctypes-private copy of the system
     errno.  variable in the calling thread to _value_ and
     return the previous value.

     New in version 2.6.

 -- Function: ctypes.set_last_error (value)
     Windows only: set the current value of the ctypes-private copy of
     the system `LastError' variable in the calling thread to _value_
     and return the previous value.

     New in version 2.6.

 -- Function: ctypes.sizeof (obj_or_type)
     Returns the size in bytes of a ctypes type or instance memory
     buffer. Does the same as the C `sizeof()' function.

 -- Function: ctypes.string_at (address[, size])
     This function returns the string starting at memory address
     address. If size is specified, it is used as size, otherwise the
     string is assumed to be zero-terminated.

 -- Function: ctypes.WinError (code=None, descr=None)
     Windows only: this function is probably the worst-named thing in
     ctypes.  It creates an instance of WindowsError.  If _code_ is not
     specified, `GetLastError' is called to determine the error code.
     If `descr' is not specified, see FormatError() is called
     to get a textual description of the error.

 -- Function: ctypes.wstring_at (address[, size])
     This function returns the wide character string starting at memory
     address _address_ as unicode string.  If _size_ is specified, it
     is used as the number of characters of the string, otherwise the
     string is assumed to be zero-terminated.

5.15.16.28 Data types
.....................

 -- Class: ctypes._CData
     This non-public class is the common base class of all ctypes data
     types.  Among other things, all ctypes type instances contain a
     memory block that hold C compatible data; the address of the
     memory block is returned by the addressof() helper
     function.  Another instance variable is exposed as _objects.;
     this contains other Python objects that need to be kept
     alive in case the memory block contains pointers.

     Common methods of ctypes data types, these are all class methods
     (to be exact, they are methods of the metaclass):

      -- Method: from_buffer (source[, offset])
          This method returns a ctypes instance that shares the buffer
          of the _source_ object.  The _source_ object must support the
          writeable buffer interface.  The optional _offset_ parameter
          specifies an offset into the source buffer in bytes; the
          default is zero.  If the source buffer is not large enough a
          ValueError is raised.

          New in version 2.6.

      -- Method: from_buffer_copy (source[, offset])
          This method creates a ctypes instance, copying the buffer
          from the _source_ object buffer which must be readable.  The
          optional _offset_ parameter specifies an offset into the
          source buffer in bytes; the default is zero.  If the source
          buffer is not large enough a ValueError is raised.

          New in version 2.6.

      -- Method: from_address (address)
          This method returns a ctypes type instance using the memory
          specified by _address_ which must be an integer.

      -- Method: from_param (obj)
          This method adapts _obj_ to a ctypes type.  It is called with
          the actual object used in a foreign function call when the
          type is present in the foreign function's `argtypes' tuple;
          it must return an object that can be used as a function call
          parameter.

          All ctypes data types have a default implementation of this
          classmethod that normally returns _obj_ if that is an
          instance of the type.  Some types accept other objects as
          well.

      -- Method: in_dll (library, name)
          This method returns a ctypes type instance exported by a
          shared library. _name_ is the name of the symbol that exports
          the data, _library_ is the loaded shared library.

     Common instance variables of ctypes data types:

      -- Attribute: _b_base_
          Sometimes ctypes data instances do not own the memory block
          they contain, instead they share part of the memory block of
          a base object.  The _b_base_ read-only member is
          the root ctypes object that owns the memory block.

      -- Attribute: _b_needsfree_
          This read-only variable is true when the ctypes data instance
          has allocated the memory block itself, false otherwise.

      -- Attribute: _objects
          This member is either `None' or a dictionary containing
          Python objects that need to be kept alive so that the memory
          block contents is kept valid.  This object is only exposed
          for debugging; never modify the contents of this dictionary.

5.15.16.29 Fundamental data types
.................................

 -- Class: ctypes._SimpleCData
     This non-public class is the base class of all fundamental ctypes
     data types. It is mentioned here because it contains the common
     attributes of the fundamental ctypes data types.
     _SimpleCData is a subclass of _CData, so it
     inherits their methods and attributes.

     Changed in version 2.6: ctypes data types that are not and do not
     contain pointers can now be pickled.

     Instances have a single attribute:

      -- Attribute: value
          This attribute contains the actual value of the instance. For
          integer and pointer types, it is an integer, for character
          types, it is a single character string, for character pointer
          types it is a Python string or unicode string.

          When the `value' attribute is retrieved from a ctypes
          instance, usually a new object is returned each time.
          ctypes does _not_ implement original object return,
          always a new object is constructed.  The same is true for all
          other ctypes object instances.

Fundamental data types, when returned as foreign function call results,
or, for example, by retrieving structure field members or array items,
are transparently converted to native Python types.  In other words, if
a foreign function has a `restype' of c_char_p, you will
always receive a Python string, _not_ a c_char_p instance.

  Subclasses of fundamental data types do _not_ inherit this behavior.
So, if a foreign functions `restype' is a subclass of c_void_p,
you will receive an instance of this subclass from the function
call. Of course, you can get the value of the pointer by accessing the
`value' attribute.

  These are the fundamental ctypes data types:

 -- Class: ctypes.c_byte
     Represents the C `signed char' datatype, and interprets the value
     as small integer.  The constructor accepts an optional integer
     initializer; no overflow checking is done.

 -- Class: ctypes.c_char
     Represents the C `char' datatype, and interprets the value as a
     single character.  The constructor accepts an optional string
     initializer, the length of the string must be exactly one
     character.

 -- Class: ctypes.c_char_p
     Represents the C `char *' datatype when it points to a
     zero-terminated string.  For a general character pointer that may
     also point to binary data, `POINTER(c_char)' must be used.  The
     constructor accepts an integer address, or a string.

 -- Class: ctypes.c_double
     Represents the C `double' datatype.  The constructor accepts an
     optional float initializer.

 -- Class: ctypes.c_longdouble
     Represents the C `long double' datatype.  The constructor accepts
     an optional float initializer.  On platforms where `sizeof(long
     double) == sizeof(double)' it is an alias to c_double.

     New in version 2.6.

 -- Class: ctypes.c_float
     Represents the C `float' datatype.  The constructor accepts an
     optional float initializer.

 -- Class: ctypes.c_int
     Represents the C `signed int' datatype.  The constructor accepts an
     optional integer initializer; no overflow checking is done.  On
     platforms where `sizeof(int) == sizeof(long)' it is an alias to
     c_long.

 -- Class: ctypes.c_int8
     Represents the C 8-bit `signed int' datatype.  Usually an alias for
     c_byte.

 -- Class: ctypes.c_int16
     Represents the C 16-bit `signed int' datatype.  Usually an alias
     for c_short.

 -- Class: ctypes.c_int32
     Represents the C 32-bit `signed int' datatype.  Usually an alias
     for c_int.

 -- Class: ctypes.c_int64
     Represents the C 64-bit `signed int' datatype.  Usually an alias
     for c_longlong.

 -- Class: ctypes.c_long
     Represents the C `signed long' datatype.  The constructor accepts
     an optional integer initializer; no overflow checking is done.

 -- Class: ctypes.c_longlong
     Represents the C `signed long long' datatype.  The constructor
     accepts an optional integer initializer; no overflow checking is
     done.

 -- Class: ctypes.c_short
     Represents the C `signed short' datatype.  The constructor accepts
     an optional integer initializer; no overflow checking is done.

 -- Class: ctypes.c_size_t
     Represents the C `size_t' datatype.

 -- Class: ctypes.c_ssize_t
     Represents the C `ssize_t' datatype.

     New in version 2.7.

 -- Class: ctypes.c_ubyte
     Represents the C `unsigned char' datatype, it interprets the value
     as small integer.  The constructor accepts an optional integer
     initializer; no overflow checking is done.

 -- Class: ctypes.c_uint
     Represents the C `unsigned int' datatype.  The constructor accepts
     an optional integer initializer; no overflow checking is done.  On
     platforms where `sizeof(int) == sizeof(long)' it is an alias for
     c_ulong.

 -- Class: ctypes.c_uint8
     Represents the C 8-bit `unsigned int' datatype.  Usually an alias
     for c_ubyte.

 -- Class: ctypes.c_uint16
     Represents the C 16-bit `unsigned int' datatype.  Usually an alias
     for c_ushort.

 -- Class: ctypes.c_uint32
     Represents the C 32-bit `unsigned int' datatype.  Usually an alias
     for c_uint.

 -- Class: ctypes.c_uint64
     Represents the C 64-bit `unsigned int' datatype.  Usually an alias
     for c_ulonglong.

 -- Class: ctypes.c_ulong
     Represents the C `unsigned long' datatype.  The constructor
     accepts an optional integer initializer; no overflow checking is
     done.

 -- Class: ctypes.c_ulonglong
     Represents the C `unsigned long long' datatype.  The constructor
     accepts an optional integer initializer; no overflow checking is
     done.

 -- Class: ctypes.c_ushort
     Represents the C `unsigned short' datatype.  The constructor
     accepts an optional integer initializer; no overflow checking is
     done.

 -- Class: ctypes.c_void_p
     Represents the C `void *' type.  The value is represented as
     integer.  The constructor accepts an optional integer initializer.

 -- Class: ctypes.c_wchar
     Represents the C `wchar_t' datatype, and interprets the value as a
     single character unicode string.  The constructor accepts an
     optional string initializer, the length of the string must be
     exactly one character.

 -- Class: ctypes.c_wchar_p
     Represents the C `wchar_t *' datatype, which must be a pointer to a
     zero-terminated wide character string.  The constructor accepts an
     integer address, or a string.

 -- Class: ctypes.c_bool
     Represent the C `bool' datatype (more accurately, `_Bool' from
     C99).  Its value can be True or False, and the constructor accepts
     any object that has a truth value.

     New in version 2.6.

 -- Class: ctypes.HRESULT
     Windows only: Represents a `HRESULT' value, which contains success
     or error information for a function or method call.

 -- Class: ctypes.py_object
     Represents the C PyObject * datatype.  Calling this
     without an argument creates a `NULL' PyObject *.
     pointer.

  The `ctypes.wintypes' module provides quite some other Windows
specific data types, for example `HWND', `WPARAM', or `DWORD'.  Some
useful structures like `MSG' or `RECT' are also defined.

5.15.16.30 Structured data types
................................

 -- Class: ctypes.Union (*args, **kw)
     Abstract base class for unions in native byte order.

 -- Class: ctypes.BigEndianStructure (*args, **kw)
     Abstract base class for structures in _big endian_ byte order.

 -- Class: ctypes.LittleEndianStructure (*args, **kw)
     Abstract base class for structures in _little endian_ byte order.

  Structures with non-native byte order cannot contain pointer type
fields, or any other data types containing pointer type fields.

 -- Class: ctypes.Structure (*args, **kw)
     Abstract base class for structures in _native_ byte order.

     Concrete structure and union types must be created by subclassing
     one of these types, and at least define a _fields_.
     class variable. ctypes will create descriptor.
     s which allow reading and writing the fields by direct
     attribute accesses.  These are the

      -- Attribute: _fields_
          A sequence defining the structure fields.  The items must be
          2-tuples or 3-tuples.  The first item is the name of the
          field, the second item specifies the type of the field; it
          can be any ctypes data type.

          For integer type fields like c_int, a third
          optional item can be given.  It must be a small positive
          integer defining the bit width of the field.

          Field names must be unique within one structure or union.
          This is not checked, only one field can be accessed when
          names are repeated.

          It is possible to define the _fields_ class
          variable _after_ the class statement that defines the
          Structure subclass, this allows to create data types that
          directly or indirectly reference themselves:

              class List(Structure):
                  pass
              List._fields_ = [("pnext", POINTER(List)),
                               ...
                              ]

          The _fields_ class variable must, however, be
          defined before the type is first used (an instance is
          created, `sizeof()' is called on it, and so on).  Later
          assignments to the _fields_ class variable will
          raise an AttributeError.

          Structure and union subclass constructors accept both
          positional and named arguments.  Positional arguments are
          used to initialize the fields in the same order as they
          appear in the _fields_ definition, named
          arguments are used to initialize the fields with the
          corresponding name.

          It is possible to defined sub-subclasses of structure types,
          they inherit the fields of the base class plus the
          _fields_ defined in the sub-subclass, if any.

      -- Attribute: _pack_
          An optional small integer that allows to override the
          alignment of structure fields in the instance.  _pack_
          must already be defined when _fields_ is
          assigned, otherwise it will have no effect.

      -- Attribute: _anonymous_
          An optional sequence that lists the names of unnamed
          (anonymous) fields.  _anonymous_ must be already
          defined when _fields_ is assigned, otherwise it
          will have no effect.

          The fields listed in this variable must be structure or union
          type fields.  ctypes will create descriptors in
          the structure type that allows to access the nested fields
          directly, without the need to create the structure or union
          field.

          Here is an example type (Windows):

              class _U(Union):
                  _fields_ = [("lptdesc", POINTER(TYPEDESC)),
                              ("lpadesc", POINTER(ARRAYDESC)),
                              ("hreftype", HREFTYPE)]

              class TYPEDESC(Structure):
                  _anonymous_ = ("u",)
                  _fields_ = [("u", _U),
                              ("vt", VARTYPE)]

          The `TYPEDESC' structure describes a COM data type, the `vt'
          field specifies which one of the union fields is valid.
          Since the `u' field is defined as anonymous field, it is now
          possible to access the members directly off the TYPEDESC
          instance. `td.lptdesc' and `td.u.lptdesc' are equivalent, but
          the former is faster since it does not need to create a
          temporary union instance:

              td = TYPEDESC()
              td.vt = VT_PTR
              td.lptdesc = POINTER(some_type)
              td.u.lptdesc = POINTER(some_type)

     It is possible to defined sub-subclasses of structures, they
     inherit the fields of the base class.  If the subclass definition
     has a separate _fields_ variable, the fields
     specified in this are appended to the fields of the base class.

     Structure and union constructors accept both positional and keyword
     arguments.  Positional arguments are used to initialize member
     fields in the same order as they are appear in _fields_.
     Keyword arguments in the constructor are interpreted as
     attribute assignments, so they will initialize _fields_
     with the same name, or create new attributes for names not
     present in _fields_.

5.15.16.31 Arrays and pointers
..............................

Not yet written - please see the sections Pointers and
section Arrays in the tutorial.

5.16 Optional Operating System Services
=======================================

The modules described in this chapter provide interfaces to operating
system features that are available on selected operating systems only.
The interfaces are generally modeled after the Unix or C interfaces but
they are available on some other systems as well (e.g. Windows or NT).
Here's an overview:

5.16.1 `select' -- Waiting for I/O completion
---------------------------------------------

This module provides access to the `select()' and `poll()' functions
available in most operating systems, `epoll()' available on Linux 2.5+
and `kqueue()' available on most BSD.  Note that on Windows, it only
works for sockets; on other operating systems, it also works for other
file types (in particular, on Unix, it works on pipes).  It cannot be
used on regular files to determine whether a file has grown since it
was last read.

  The module defines the following:

 -- Exception: select.error
     The exception raised when an error occurs.  The accompanying value
     is a pair containing the numeric error code from `errno' and the
     corresponding string, as would be printed by the C function
     `perror()'.

 -- Function: select.epoll ([sizehint=-1])
     (Only supported on Linux 2.5.44 and newer.)  Returns an edge
     polling object, which can be used as Edge or Level Triggered
     interface for I/O events; see section Edge and Level Trigger Polling (epoll) Objects
     below for the methods supported by
     epolling objects.

     New in version 2.6.

 -- Function: select.poll ()
     (Not supported by all operating systems.)  Returns a polling
     object, which supports registering and unregistering file
     descriptors, and then polling them for I/O events; see section
     Polling Objects below for the methods supported by
     polling objects.

 -- Function: select.kqueue ()
     (Only supported on BSD.)  Returns a kernel queue object; see
     section Kqueue Objects below for the methods
     supported by kqueue objects.

     New in version 2.6.

 -- Function: select.kevent (ident, filter=KQ_FILTER_READ,
          flags=KQ_EV_ADD, fflags=0, data=0, udata=0)
     (Only supported on BSD.)  Returns a kernel event object; see
     section Kevent Objects below for the methods
     supported by kevent objects.

     New in version 2.6.

 -- Function: select.select (rlist, wlist, xlist[, timeout])
     This is a straightforward interface to the Unix `select()' system
     call.  The first three arguments are sequences of 'waitable
     objects': either integers representing file descriptors or objects
     with a parameterless method named `fileno()' returning such an
     integer:

        * _rlist_: wait until ready for reading

        * _wlist_: wait until ready for writing

        * _xlist_: wait for an "exceptional condition" (see the manual
          page for what your system considers such a condition)

     Empty sequences are allowed, but acceptance of three empty
     sequences is platform-dependent. (It is known to work on Unix but
     not on Windows.)  The optional _timeout_ argument specifies a
     time-out as a floating point number in seconds.  When the
     _timeout_ argument is omitted the function blocks until at least
     one file descriptor is ready.  A time-out value of zero specifies a
     poll and never blocks.

     The return value is a triple of lists of objects that are ready:
     subsets of the first three arguments.  When the time-out is
     reached without a file descriptor becoming ready, three empty
     lists are returned.

     Among the acceptable object types in the sequences are Python file
     objects (e.g.  `sys.stdin', or objects returned by open()
     or os.popen()), socket objects returned by
     socket.socket().  You may also define a _wrapper_ class
     yourself, as long as it has an appropriate `fileno()' method (that
     really returns a file descriptor, not just a random integer).

          Note: File objects on Windows are not acceptable, but sockets
          are.  On Windows, the underlying `select()' function is
          provided by the WinSock library, and does not handle file
          descriptors that don't originate from WinSock.

 -- Attribute: select.PIPE_BUF
     Files reported as ready for writing by select(),
     poll() or similar interfaces in this module are guaranteed
     to not block on a write of up to PIPE_BUF bytes.
     This value is guaranteed by POSIX to be at least 512.
     Availability: Unix.

     New in version 2.7.

5.16.1.1 Edge and Level Trigger Polling (epoll) Objects
.......................................................

     <http://linux.die.net/man/4/epoll>

     _eventmask_

     Constant                    Meaning
     --------------------------------------------------------------------------------
     `EPOLLIN'                   Available for read
     `EPOLLOUT'                  Available for write
     `EPOLLPRI'                  Urgent data for read
     `EPOLLERR'                  Error condition happened on the assoc. fd
     `EPOLLHUP'                  Hang up happened on the assoc. fd
     `EPOLLET'                   Set Edge Trigger behavior, the default is Level
                                 Trigger behavior
     `EPOLLONESHOT'              Set one-shot behavior. After one event is pulled
                                 out, the fd is internally disabled
     `EPOLLRDNORM'               Equivalent to `EPOLLIN'
     `EPOLLRDBAND'               Priority data band can be read.
     `EPOLLWRNORM'               Equivalent to `EPOLLOUT'
     `EPOLLWRBAND'               Priority data may be written.
     `EPOLLMSG'                  Ignored.

 -- Method: epoll.close ()
     Close the control file descriptor of the epoll object.

 -- Method: epoll.fileno ()
     Return the file descriptor number of the control fd.

 -- Method: epoll.fromfd (fd)
     Create an epoll object from a given file descriptor.

 -- Method: epoll.register (fd[, eventmask])
     Register a fd descriptor with the epoll object.

          Note: Registering a file descriptor that's already registered
          raises an IOError - contrary to Polling Objects.'
          s register.

 -- Method: epoll.modify (fd, eventmask)
     Modify a register file descriptor.

 -- Method: epoll.unregister (fd)
     Remove a registered file descriptor from the epoll object.

 -- Method: epoll.poll ([timeout=-1[, maxevents=-1]])
     Wait for events. timeout in seconds (float)

5.16.1.2 Polling Objects
........................

The `poll()' system call, supported on most Unix systems, provides
better scalability for network servers that service many, many clients
at the same time. `poll()' scales better because the system call only
requires listing the file descriptors of interest, while `select()'
builds a bitmap, turns on bits for the fds of interest, and then
afterward the whole bitmap has to be linearly scanned again. `select()'
is O(highest file descriptor), while `poll()' is O(number of file
descriptors).

 -- Method: poll.register (fd[, eventmask])
     Register a file descriptor with the polling object.  Future calls
     to the poll() method will then check whether the file
     descriptor has any pending I/O events.  _fd_ can be either an
     integer, or an object with a `fileno()' method that returns an
     integer.  File objects implement `fileno()', so they can also be
     used as the argument.

     _eventmask_ is an optional bitmask describing the type of events
     you want to check for, and can be a combination of the constants
     `POLLIN', `POLLPRI', and `POLLOUT', described in the table below.
     If not specified, the default value used will check for all 3
     types of events.

     Constant                Meaning
     -----------------------------------------------------------------------
     `POLLIN'                There is data to read
     `POLLPRI'               There is urgent data to read
     `POLLOUT'               Ready for output: writing will not block
     `POLLERR'               Error condition of some sort
     `POLLHUP'               Hung up
     `POLLNVAL'              Invalid request: descriptor not open

     Registering a file descriptor that's already registered is not an
     error, and has the same effect as registering the descriptor
     exactly once.

 -- Method: poll.modify (fd, eventmask)
     Modifies an already registered fd. This has the same effect as
     `register(fd, eventmask)'.  Attempting to modify a file descriptor
     that was never registered causes an IOError exception
     with errno `ENOENT' to be raised.

     New in version 2.6.

 -- Method: poll.unregister (fd)
     Remove a file descriptor being tracked by a polling object.  Just
     like the register() method, _fd_ can be an integer or
     an object with a `fileno()' method that returns an integer.

     Attempting to remove a file descriptor that was never registered
     causes a KeyError exception to be raised.

 -- Method: poll.poll ([timeout])
     Polls the set of registered file descriptors, and returns a
     possibly-empty list containing `(fd, event)' 2-tuples for the
     descriptors that have events or errors to report. _fd_ is the file
     descriptor, and _event_ is a bitmask with bits set for the
     reported events for that descriptor -- `POLLIN' for waiting input,
     `POLLOUT' to indicate that the descriptor can be written to, and
     so forth. An empty list indicates that the call timed out and no
     file descriptors had any events to report. If _timeout_ is given,
     it specifies the length of time in milliseconds which the system
     will wait for events before returning. If _timeout_ is omitted,
     negative, or None, the call will block until there is
     an event for this poll object.

5.16.1.3 Kqueue Objects
.......................

 -- Method: kqueue.close ()
     Close the control file descriptor of the kqueue object.

 -- Method: kqueue.fileno ()
     Return the file descriptor number of the control fd.

 -- Method: kqueue.fromfd (fd)
     Create a kqueue object from a given file descriptor.

 -- Method: kqueue.control (changelist, max_events[, timeout=None]) ->
          eventlist
     Low level interface to kevent

        - changelist must be an iterable of kevent object or None

        - max_events must be 0 or a positive integer

        - timeout in seconds (floats possible)

5.16.1.4 Kevent Objects
.......................

<http://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2>

 -- Attribute: kevent.ident
     Value used to identify the event. The interpretation depends on
     the filter but it's usually the file descriptor. In the
     constructor ident can either be an int or an object with a
     fileno() function. kevent stores the integer internally.

 -- Attribute: kevent.filter
     Name of the kernel filter.

     Constant                        Meaning
     ----------------------------------------------------------------------------------
     `KQ_FILTER_READ'                Takes a descriptor and returns whenever there is
                                     data available to read
     `KQ_FILTER_WRITE'               Takes a descriptor and returns whenever there is
                                     data available to write
     `KQ_FILTER_AIO'                 AIO requests
     `KQ_FILTER_VNODE'               Returns when one or more of the requested events
                                     watched in _fflag_ occurs
     `KQ_FILTER_PROC'                Watch for events on a process id
     `KQ_FILTER_NETDEV'              Watch for events on a network device [not
                                     available on Mac OS X]
     `KQ_FILTER_SIGNAL'              Returns whenever the watched signal is delivered
                                     to the process
     `KQ_FILTER_TIMER'               Establishes an arbitrary timer

 -- Attribute: kevent.flags
     Filter action.

     Constant                        Meaning
     ----------------------------------------------------------------------------------
     `KQ_EV_ADD'                     Adds or modifies an event
     `KQ_EV_DELETE'                  Removes an event from the queue
     `KQ_EV_ENABLE'                  Permitscontrol() to returns the event
     `KQ_EV_DISABLE'                 Disablesevent
     `KQ_EV_ONESHOT'                 Removes event after first occurrence
     `KQ_EV_CLEAR'                   Reset the state after an event is retrieved
     `KQ_EV_SYSFLAGS'                internal event
     `KQ_EV_FLAG1'                   internal event
     `KQ_EV_EOF'                     Filter specific EOF condition
     `KQ_EV_ERROR'                   See return values

 -- Attribute: kevent.fflags
     Filter specific flags.

     `KQ_FILTER_READ' and  `KQ_FILTER_WRITE' filter flags:

     Constant                         Meaning
     ----------------------------------------------------------------------------------
     `KQ_NOTE_LOWAT'                  low water mark of a socket buffer

     `KQ_FILTER_VNODE' filter flags:

     Constant                         Meaning
     ----------------------------------------------------------------------------------
     `KQ_NOTE_DELETE'                 _unlink()_ was called
     `KQ_NOTE_WRITE'                  a write occurred
     `KQ_NOTE_EXTEND'                 the file was extended
     `KQ_NOTE_ATTRIB'                 an attribute was changed
     `KQ_NOTE_LINK'                   the link count has changed
     `KQ_NOTE_RENAME'                 the file was renamed
     `KQ_NOTE_REVOKE'                 access to the file was revoked

     `KQ_FILTER_PROC' filter flags:

     Constant                         Meaning
     ----------------------------------------------------------------------------------
     `KQ_NOTE_EXIT'                   the process has exited
     `KQ_NOTE_FORK'                   the process has called _fork()_
     `KQ_NOTE_EXEC'                   the process has executed a new process
     `KQ_NOTE_PCTRLMASK'              internal filter flag
     `KQ_NOTE_PDATAMASK'              internal filter flag
     `KQ_NOTE_TRACK'                  follow a process across _fork()_
     `KQ_NOTE_CHILD'                  returned on the child process for _NOTE_TRACK_
     `KQ_NOTE_TRACKERR'               unable to attach to a child

     `KQ_FILTER_NETDEV' filter flags (not available on Mac OS X):

     Constant                         Meaning
     ----------------------------------------------------------------------------------
     `KQ_NOTE_LINKUP'                 link is up
     `KQ_NOTE_LINKDOWN'               link is down
     `KQ_NOTE_LINKINV'                link state is invalid

 -- Attribute: kevent.data
     Filter specific data.

 -- Attribute: kevent.udata
     User defined value.

5.16.2 `threading' -- Higher-level threading interface
------------------------------------------------------

This module constructs higher-level threading interfaces on top of the
lower level thread module.  See also the mutex.
and Queue modules.

  The dummy_threading module is provided for situations where
threading cannot be used because thread is
missing.

     Note: Starting with Python 2.6, this module provides PEP 8
     (http://www.python.org/dev/peps/pep-0008) compliant aliases and
     properties to replace the `camelCase' names that were inspired by
     Java's threading API. This updated API is compatible with that of
     the multiprocessing module. However, no schedule has
     been set for the deprecation of the `camelCase' names and they
     remain fully supported in both Python 2.x and 3.x.

     Note: Starting with Python 2.5, several Thread methods raise
     RuntimeError.  instead of AssertionError if
     called erroneously.

  Latest version of the threading module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/threading.py?view=markup)This
module defines the following functions and objects:

 -- Function: threading.active_count ()
 -- Function: threading.activeCount ()
     Return the number of Thread objects currently alive.
     The returned count is equal to the length of the list returned by
     enumerate().

 -- Function: threading.Condition ()
     A factory function that returns a new condition variable object. A
     condition variable allows one or more threads to wait until they
     are notified by another thread.

     See Condition Objects.

 -- Function: threading.current_thread ()
 -- Function: threading.currentThread ()
     Return the current Thread object, corresponding to
     the caller's thread of control.  If the caller's thread of control
     was not created through the threading module, a dummy
     thread object with limited functionality is returned.

 -- Function: threading.enumerate ()
     Return a list of all Thread objects currently alive.
     The list includes daemonic threads, dummy thread objects created by
     current_thread(), and the main thread.  It excludes
     terminated threads and threads that have not yet been started.

 -- Function: threading.Event ()
     A factory function that returns a new event object.  An event
     manages a flag that can be set to true with the set().
     method and reset to false with the `clear()' method.  The `wait()'
     method blocks until the flag is true.

     See Event Objects.

 -- Class: threading.local
     A class that represents thread-local data.  Thread-local data are
     data whose values are thread specific.  To manage thread-local
     data, just create an instance of local. (or a
     subclass) and store attributes on it:

         mydata = threading.local()
         mydata.x = 1

     The instance's values will be different for separate threads.

     For more details and extensive examples, see the documentation
     string of the `_threading_local' module.

     New in version 2.4.

 -- Function: threading.Lock ()
     A factory function that returns a new primitive lock object.  Once
     a thread has acquired it, subsequent attempts to acquire it block,
     until it is released; any thread may release it.

     See Lock Objects.

 -- Function: threading.RLock ()
     A factory function that returns a new reentrant lock object. A
     reentrant lock must be released by the thread that acquired it.
     Once a thread has acquired a reentrant lock, the same thread may
     acquire it again without blocking; the thread must release it once
     for each time it has acquired it.

     See RLock Objects.

 -- Function: threading.Semaphore ([value])
     A factory function that returns a new semaphore object.  A
     semaphore manages a counter representing the number of `release()'
     calls minus the number of `acquire()' calls, plus an initial
     value. The `acquire()' method blocks if necessary until it can
     return without making the counter negative.  If not given, _value_
     defaults to 1.

     See Semaphore Objects.

 -- Function: threading.BoundedSemaphore ([value])
     A factory function that returns a new bounded semaphore object.  A
     bounded semaphore checks to make sure its current value doesn't
     exceed its initial value.  If it does, see ValueError is
     raised. In most situations semaphores are used to guard resources
     with limited capacity.  If the semaphore is released too many
     times it's a sign of a bug.  If not given, _value_ defaults to 1.

 -- Class: threading.Thread
     A class that represents a thread of control.  This class can be
     safely subclassed in a limited fashion.

     See Thread Objects.

 -- Class: threading.Timer
     A thread that executes a function after a specified interval has
     passed.

     See Timer Objects.

 -- Function: threading.settrace (func)
     Set a trace function for all threads started from the
     threading module.  The _func_ will be passed to
     sys.settrace() for each thread, before its `run()' method is
     called.

     New in version 2.3.

 -- Function: threading.setprofile (func)
     Set a profile function for all threads started from the
     threading module.  The _func_ will be passed to
     sys.setprofile() for each thread, before its `run()' method
     is called.

     New in version 2.3.

 -- Function: threading.stack_size ([size])
     Return the thread stack size used when creating new threads.  The
     optional _size_ argument specifies the stack size to be used for
     subsequently created threads, and must be 0 (use platform or
     configured default) or a positive integer value of at least 32,768
     (32kB). If changing the thread stack size is unsupported, a
     `ThreadError' is raised.  If the specified stack size is invalid,
     a ValueError is raised and the stack size is
     unmodified.  32kB is currently the minimum supported stack size
     value to guarantee sufficient stack space for the interpreter
     itself.  Note that some platforms may have particular restrictions
     on values for the stack size, such as requiring a minimum stack
     size > 32kB or requiring allocation in multiples of the system
     memory page size - platform documentation should be referred to
     for more information (4kB pages are common; using multiples of
     4096 for the stack size is the suggested approach in the absence
     of more specific information).  Availability: Windows, systems
     with POSIX threads.

     New in version 2.5.

  Detailed interfaces for the objects are documented below.

  The design of this module is loosely based on Java's threading model.
However, where Java makes locks and condition variables basic behavior
of every object, they are separate objects in Python.  Python's
Thread class supports a subset of the behavior of Java's Thread
class; currently, there are no priorities, no thread groups, and
threads cannot be destroyed, stopped, suspended, resumed, or
interrupted.  The static methods of Java's Thread class, when
implemented, are mapped to module-level functions.

  All of the methods described below are executed atomically.

5.16.2.1 Thread Objects
.......................

This class represents an activity that is run in a separate thread of
control.  There are two ways to specify the activity: by passing a
callable object to the constructor, or by overriding the `run()' method
in a subclass.  No other methods (except for the constructor) should be
overridden in a subclass.  In other words,  _only_  override the
__init__() and `run()' methods of this class.

  Once a thread object is created, its activity must be started by
calling the thread's `start()' method.  This invokes the `run()' method
in a separate thread of control.

  Once the thread's activity is started, the thread is considered
'alive'. It stops being alive when its `run()' method terminates -
either normally, or by raising an unhandled exception.  The
`is_alive()' method tests whether the thread is alive.

  Other threads can call a thread's `join()' method.  This blocks the
calling thread until the thread whose `join()' method is called is
terminated.

  A thread has a name.  The name can be passed to the constructor, and
read or changed through the `name' attribute.

  A thread can be flagged as a "daemon thread".  The significance of
this flag is that the entire Python program exits when only daemon
threads are left.  The initial value is inherited from the creating
thread.  The flag can be set through the `daemon' property.

  There is a "main thread" object; this corresponds to the initial
thread of control in the Python program.  It is not a daemon thread.

  There is the possibility that "dummy thread objects" are created.
These are thread objects corresponding to "alien threads", which are
threads of control started outside the threading module, such as
directly from C code.  Dummy thread objects have limited functionality;
they are always considered alive and daemonic, and cannot be
`join()'ed.  They are never deleted, since it is impossible to detect
the termination of alien threads.

 -- Class: threading.Thread (group=None, target=None, name=None,
          args=(), kwargs={})
     This constructor should always be called with keyword arguments.
     Arguments are:

     _group_ should be `None'; reserved for future extension when a
     `ThreadGroup' class is implemented.

     _target_ is the callable object to be invoked by the run()
     method.  Defaults to `None', meaning nothing is called.

     _name_ is the thread name.  By default, a unique name is
     constructed of the form "Thread-_N_" where _N_ is a small decimal
     number.

     _args_ is the argument tuple for the target invocation.  Defaults
     to `()'.

     _kwargs_ is a dictionary of keyword arguments for the target
     invocation.  Defaults to `{}'.

     If the subclass overrides the constructor, it must make sure to
     invoke the base class constructor (`Thread.__init__()') before
     doing anything else to the thread.

      -- Method: start ()
          Start the thread's activity.

          It must be called at most once per thread object.  It
          arranges for the object's run() method to be
          invoked in a separate thread of control.

          This method will raise a `RuntimeException' if called more
          than once on the same thread object.

      -- Method: run ()
          Method representing the thread's activity.

          You may override this method in a subclass.  The standard
          run().  method invokes the callable object passed
          to the object's constructor as the _target_ argument, if any,
          with sequential and keyword arguments taken from the _args_
          and _kwargs_ arguments, respectively.

      -- Method: join ([timeout])
          Wait until the thread terminates. This blocks the calling
          thread until the thread whose join() method is
          called terminates - either normally or through an unhandled
          exception - or until the optional timeout occurs.

          When the _timeout_ argument is present and not `None', it
          should be a floating point number specifying a timeout for
          the operation in seconds (or fractions thereof). As
          join() always returns `None', you must call
          isAlive() after join() to decide whether
          a timeout happened - if the thread is still alive, the
          join() call timed out.

          When the _timeout_ argument is not present or `None', the
          operation will block until the thread terminates.

          A thread can be join().ed many times.

          join() raises a RuntimeError if an
          attempt is made to join the current thread as that would
          cause a deadlock. It is also an error to join().
          a thread before it has been started and attempts to do so
          raises the same exception.

      -- Method: getName ()
      -- Method: setName ()
          Old API for name.

      -- Attribute: name
          A string used for identification purposes only. It has no
          semantics.  Multiple threads may be given the same name.  The
          initial name is set by the constructor.

      -- Attribute: ident
          The 'thread identifier' of this thread or `None' if the
          thread has not been started.  This is a nonzero integer.  See
          the thread.get_ident() function.  Thread
          identifiers may be recycled when a thread exits and another
          thread is created.  The identifier is available even after
          the thread has exited.

          New in version 2.6.

      -- Method: is_alive ()
      -- Method: isAlive ()
          Return whether the thread is alive.

          This method returns `True' just before the run().
          method starts until just after the run() method
          terminates.  The module function enumerate().
          returns a list of all alive threads.

      -- Method: isDaemon ()
      -- Method: setDaemon ()
          Old API for daemon.

      -- Attribute: daemon
          A boolean value indicating whether this thread is a daemon
          thread (True) or not (False).  This must be set before
          start() is called, otherwise RuntimeError.
          is raised.  Its initial value is inherited from the creating
          thread; the main thread is not a daemon thread and therefore
          all threads created in the main thread default to
          daemon.  = `False'.

          The entire Python program exits when no alive non-daemon
          threads are left.

5.16.2.2 Lock Objects
.....................

A primitive lock is a synchronization primitive that is not owned by a
particular thread when locked.  In Python, it is currently the lowest
level synchronization primitive available, implemented directly by the
thread.  extension module.

  A primitive lock is in one of two states, "locked" or "unlocked". It
is created in the unlocked state.  It has two basic methods,
`acquire()' and `release()'.  When the state is unlocked, `acquire()'
changes the state to locked and returns immediately.  When the state is
locked, `acquire()' blocks until a call to `release()' in another
thread changes it to unlocked, then the `acquire()' call resets it to
locked and returns.  The `release()' method should only be called in
the locked state; it changes the state to unlocked and returns
immediately. If an attempt is made to release an unlocked lock, a
RuntimeError will be raised.

  When more than one thread is blocked in `acquire()' waiting for the
state to turn to unlocked, only one thread proceeds when a `release()'
call resets the state to unlocked; which one of the waiting threads
proceeds is not defined, and may vary across implementations.

  All methods are executed atomically.

 -- Method: Lock.acquire ([blocking=1])
     Acquire a lock, blocking or non-blocking.

     When invoked without arguments, block until the lock is unlocked,
     then set it to locked, and return true.

     When invoked with the _blocking_ argument set to true, do the same
     thing as when called without arguments, and return true.

     When invoked with the _blocking_ argument set to false, do not
     block.  If a call without an argument would block, return false
     immediately; otherwise, do the same thing as when called without
     arguments, and return true.

 -- Method: Lock.release ()
     Release a lock.

     When the lock is locked, reset it to unlocked, and return.  If any
     other threads are blocked waiting for the lock to become unlocked,
     allow exactly one of them to proceed.

     Do not call this method when the lock is unlocked.

     There is no return value.

5.16.2.3 RLock Objects
......................

A reentrant lock is a synchronization primitive that may be acquired
multiple times by the same thread.  Internally, it uses the concepts of
"owning thread" and "recursion level" in addition to the
locked/unlocked state used by primitive locks.  In the locked state,
some thread owns the lock; in the unlocked state, no thread owns it.

  To lock the lock, a thread calls its `acquire()' method; this returns
once the thread owns the lock.  To unlock the lock, a thread calls its
`release()' method. `acquire()'/`release()' call pairs may be nested;
only the final `release()' (the `release()' of the outermost pair)
resets the lock to unlocked and allows another thread blocked in
`acquire()' to proceed.

 -- Method: RLock.acquire ([blocking=1])
     Acquire a lock, blocking or non-blocking.

     When invoked without arguments: if this thread already owns the
     lock, increment the recursion level by one, and return
     immediately.  Otherwise, if another thread owns the lock, block
     until the lock is unlocked.  Once the lock is unlocked (not owned
     by any thread), then grab ownership, set the recursion level to
     one, and return.  If more than one thread is blocked waiting until
     the lock is unlocked, only one at a time will be able to grab
     ownership of the lock.  There is no return value in this case.

     When invoked with the _blocking_ argument set to true, do the same
     thing as when called without arguments, and return true.

     When invoked with the _blocking_ argument set to false, do not
     block.  If a call without an argument would block, return false
     immediately; otherwise, do the same thing as when called without
     arguments, and return true.

 -- Method: RLock.release ()
     Release a lock, decrementing the recursion level.  If after the
     decrement it is zero, reset the lock to unlocked (not owned by any
     thread), and if any other threads are blocked waiting for the lock
     to become unlocked, allow exactly one of them to proceed.  If
     after the decrement the recursion level is still nonzero, the lock
     remains locked and owned by the calling thread.

     Only call this method when the calling thread owns the lock. A
     RuntimeError is raised if this method is called when
     the lock is unlocked.

     There is no return value.

5.16.2.4 Condition Objects
..........................

A condition variable is always associated with some kind of lock; this
can be passed in or one will be created by default.  (Passing one in is
useful when several condition variables must share the same lock.)

  A condition variable has `acquire()' and `release()' methods that call
the corresponding methods of the associated lock. It also has a `wait()'
method, and `notify()' and `notifyAll()' methods.  These three must only
be called when the calling thread has acquired the lock, otherwise a
RuntimeError is raised.

  The `wait()' method releases the lock, and then blocks until it is
awakened by a `notify()' or `notifyAll()' call for the same condition
variable in another thread.  Once awakened, it re-acquires the lock and
returns.  It is also possible to specify a timeout.

  The `notify()' method wakes up one of the threads waiting for the
condition variable, if any are waiting.  The `notifyAll()' method wakes
up all threads waiting for the condition variable.

  Note: the `notify()' and `notifyAll()' methods don't release the lock;
this means that the thread or threads awakened will not return from
their `wait()' call immediately, but only when the thread that called
`notify()' or `notifyAll()' finally relinquishes ownership of the lock.

  Tip: the typical programming style using condition variables uses the
lock to synchronize access to some shared state; threads that are
interested in a particular change of state call `wait()' repeatedly
until they see the desired state, while threads that modify the state
call `notify()' or `notifyAll()' when they change the state in such a
way that it could possibly be a desired state for one of the waiters.
For example, the following code is a generic producer-consumer
situation with unlimited buffer capacity:

    # Consume one item
    cv.acquire()
    while not an_item_is_available():
        cv.wait()
    get_an_available_item()
    cv.release()

    # Produce one item
    cv.acquire()
    make_an_item_available()
    cv.notify()
    cv.release()

To choose between `notify()' and `notifyAll()', consider whether one
state change can be interesting for only one or several waiting
threads.  E.g.  in a typical producer-consumer situation, adding one
item to the buffer only needs to wake up one consumer thread.

 -- Class: threading.Condition ([lock])
     If the _lock_ argument is given and not `None', it must be a
     Lock.  or RLock object, and it is used as the
     underlying lock.  Otherwise, a new RLock object is
     created and used as the underlying lock.

      -- Method: acquire (*args)
          Acquire the underlying lock. This method calls the
          corresponding method on the underlying lock; the return value
          is whatever that method returns.

      -- Method: release ()
          Release the underlying lock. This method calls the
          corresponding method on the underlying lock; there is no
          return value.

      -- Method: wait ([timeout])
          Wait until notified or until a timeout occurs. If the calling
          thread has not acquired the lock when this method is called,
          a RuntimeError is raised.

          This method releases the underlying lock, and then blocks
          until it is awakened by a notify() or
          notifyAll() call for the same condition variable in
          another thread, or until the optional timeout occurs.  Once
          awakened or timed out, it re-acquires the lock and returns.

          When the _timeout_ argument is present and not `None', it
          should be a floating point number specifying a timeout for
          the operation in seconds (or fractions thereof).

          When the underlying lock is an RLock, it is not
          released using its release() method, since this
          may not actually unlock the lock when it was acquired
          multiple times recursively.  Instead, an internal interface
          of the RLock class is used, which really unlocks
          it even when it has been recursively acquired several times.
          Another internal interface is then used to restore the
          recursion level when the lock is reacquired.

      -- Method: notify ()
          Wake up a thread waiting on this condition, if any.  If the
          calling thread has not acquired the lock when this method is
          called, a RuntimeError is raised.

          This method wakes up one of the threads waiting for the
          condition variable, if any are waiting; it is a no-op if no
          threads are waiting.

          The current implementation wakes up exactly one thread, if
          any are waiting.  However, it's not safe to rely on this
          behavior.  A future, optimized implementation may
          occasionally wake up more than one thread.

          Note: the awakened thread does not actually return from its
          wait().  call until it can reacquire the lock.
          Since notify() does not release the lock, its
          caller should.

      -- Method: notify_all ()
      -- Method: notifyAll ()
          Wake up all threads waiting on this condition.  This method
          acts like notify(), but wakes up all waiting
          threads instead of one. If the calling thread has not
          acquired the lock when this method is called, a
          RuntimeError is raised.

5.16.2.5 Semaphore Objects
..........................

This is one of the oldest synchronization primitives in the history of
computer science, invented by the early Dutch computer scientist Edsger
W. Dijkstra (he used `P()' and `V()' instead of `acquire()' and
`release()').

  A semaphore manages an internal counter which is decremented by each
`acquire()' call and incremented by each `release()' call.  The counter
can never go below zero; when `acquire()' finds that it is zero, it
blocks, waiting until some other thread calls `release()'.

 -- Class: threading.Semaphore ([value])
     The optional argument gives the initial _value_ for the internal
     counter; it defaults to `1'. If the _value_ given is less than 0,
     see ValueError is raised.

      -- Method: acquire ([blocking])
          Acquire a semaphore.

          When invoked without arguments: if the internal counter is
          larger than zero on entry, decrement it by one and return
          immediately.  If it is zero on entry, block, waiting until
          some other thread has called release() to make
          it larger than zero.  This is done with proper interlocking
          so that if multiple acquire() calls are blocked,
          see release() will wake exactly one of them up.  The
          implementation may pick one at random, so the order in which
          blocked threads are awakened should not be relied on.  There
          is no return value in this case.

          When invoked with _blocking_ set to true, do the same thing
          as when called without arguments, and return true.

          When invoked with _blocking_ set to false, do not block.  If
          a call without an argument would block, return false
          immediately; otherwise, do the same thing as when called
          without arguments, and return true.

      -- Method: release ()
          Release a semaphore, incrementing the internal counter by
          one.  When it was zero on entry and another thread is waiting
          for it to become larger than zero again, wake up that thread.

5.16.2.6 `Semaphore' Example
............................

Semaphores are often used to guard resources with limited capacity, for
example, a database server.  In any situation where the size of the
resource size is fixed, you should use a bounded semaphore.  Before
spawning any worker threads, your main thread would initialize the
semaphore:

    maxconnections = 5
    ...
    pool_sema = BoundedSemaphore(value=maxconnections)

Once spawned, worker threads call the semaphore's acquire and release
methods when they need to connect to the server:

    pool_sema.acquire()
    conn = connectdb()
    ... use connection ...
    conn.close()
    pool_sema.release()

The use of a bounded semaphore reduces the chance that a programming
error which causes the semaphore to be released more than it's acquired
will go undetected.

5.16.2.7 Event Objects
......................

This is one of the simplest mechanisms for communication between
threads: one thread signals an event and other threads wait for it.

  An event object manages an internal flag that can be set to true with
the set() method and reset to false with the `clear()'
method.  The `wait()' method blocks until the flag is true.

 -- Class: threading.Event
     The internal flag is initially false.

      -- Method: is_set ()
      -- Method: isSet ()
          Return true if and only if the internal flag is true.

          Changed in version 2.6: The `is_set()' syntax is new.

      -- Method: set ()
          Set the internal flag to true. All threads waiting for it to
          become true are awakened. Threads that call wait()
          once the flag is true will not block at all.

      -- Method: clear ()
          Reset the internal flag to false. Subsequently, threads
          calling wait() will block until set()
          is called to set the internal flag to true again.

      -- Method: wait ([timeout])
          Block until the internal flag is true.  If the internal flag
          is true on entry, return immediately.  Otherwise, block until
          another thread calls set() to set the flag to
          true, or until the optional timeout occurs.

          When the timeout argument is present and not `None', it
          should be a floating point number specifying a timeout for
          the operation in seconds (or fractions thereof).

          This method returns the internal flag on exit, so it will
          always return `True' except if a timeout is given and the
          operation times out.

          Changed in version 2.7: Previously, the method always
          returned `None'.

5.16.2.8 Timer Objects
......................

This class represents an action that should be run only after a certain
amount of time has passed -- a timer.  Timer is a subclass
of Thread.  and as such also functions as an example of
creating custom threads.

  Timers are started, as with threads, by calling their `start()'
method.  The timer can be stopped (before its action has begun) by
calling the `cancel()' method.  The interval the timer will wait before
executing its action may not be exactly the same as the interval
specified by the user.

  For example:

    def hello():
        print "hello, world"

    t = Timer(30.0, hello)
    t.start() # after 30 seconds, "hello, world" will be printed

 -- Class: threading.Timer (interval, function, args=[], kwargs={})
     Create a timer that will run _function_ with arguments _args_ and
     keyword arguments _kwargs_, after _interval_ seconds have passed.

      -- Method: cancel ()
          Stop the timer, and cancel the execution of the timer's
          action.  This will only work if the timer is still in its
          waiting stage.

5.16.2.9 Using locks, conditions, and semaphores in the `with' statement
........................................................................

All of the objects provided by this module that have `acquire()' and
`release()' methods can be used as context managers for a with.
statement.  The `acquire()' method will be called when the block
is entered, and `release()' will be called when the block is exited.

  Currently, see Lock, RLock, Condition,
Semaphore, and BoundedSemaphore objects may be
used as with statement context managers.  For example:

    import threading

    some_rlock = threading.RLock()

    with some_rlock:
        print "some_rlock is locked while this executes"

5.16.2.10 Importing in threaded code
....................................

While the import machinery is thread-safe, there are two key
restrictions on threaded imports due to inherent limitations in the way
that thread-safety is provided:

   * Firstly, other than in the main module, an import should not have
     the side effect of spawning a new thread and then waiting for that
     thread in any way. Failing to abide by this restriction can lead
     to a deadlock if the spawned thread directly or indirectly
     attempts to import a module.

   * Secondly, all import attempts must be completed before the
     interpreter starts shutting itself down. This can be most easily
     achieved by only performing imports from non-daemon threads
     created through the threading module. Daemon threads and threads
     created directly with the thread module will require some other
     form of synchronization to ensure they do not attempt imports
     after system shutdown has commenced. Failure to abide by this
     restriction will lead to intermittent exceptions and crashes
     during interpreter shutdown (as the late imports attempt to access
     machinery which is no longer in a valid state).

5.16.3 `thread' -- Multiple threads of control
----------------------------------------------

     Note: The thread module has been renamed to `_thread'
     in Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0; however, you should
     consider using the high-level threading module instead.

This module provides low-level primitives for working with multiple
threads (also called _light-weight processes_ or _tasks_) -- multiple
threads of control sharing their global data space.  For
synchronization, simple locks (also called _mutexes_ or _binary
semaphores_) are provided.  The threading module provides
an easier to use and higher-level threading API built on top of this
module.

  The module is optional.  It is supported on Windows, Linux, SGI IRIX,
Solaris 2.x, as well as on systems that have a POSIX thread (a.k.a.
"pthread") implementation.  For systems lacking the thread.
module, the dummy_thread module is available. It duplicates
this module's interface and can be used as a drop-in replacement.

  It defines the following constant and functions:

 -- Exception: thread.error
     Raised on thread-specific errors.

 -- Data: thread.LockType
     This is the type of lock objects.

 -- Function: thread.start_new_thread (function, args[, kwargs])
     Start a new thread and return its identifier.  The thread executes
     the function _function_ with the argument list _args_ (which must
     be a tuple).  The optional _kwargs_ argument specifies a
     dictionary of keyword arguments. When the function returns, the
     thread silently exits.  When the function terminates with an
     unhandled exception, a stack trace is printed and then the thread
     exits (but other threads continue to run).

 -- Function: thread.interrupt_main ()
     Raise a KeyboardInterrupt exception in the main
     thread.  A subthread can use this function to interrupt the main
     thread.

     New in version 2.3.

 -- Function: thread.exit ()
     Raise the SystemExit exception.  When not caught, this
     will cause the thread to exit silently.

 -- Function: thread.allocate_lock ()
     Return a new lock object.  Methods of locks are described below.
     The lock is initially unlocked.

 -- Function: thread.get_ident ()
     Return the 'thread identifier' of the current thread.  This is a
     nonzero integer.  Its value has no direct meaning; it is intended
     as a magic cookie to be used e.g. to index a dictionary of
     thread-specific data.  Thread identifiers may be recycled when a
     thread exits and another thread is created.

 -- Function: thread.stack_size ([size])
     Return the thread stack size used when creating new threads.  The
     optional _size_ argument specifies the stack size to be used for
     subsequently created threads, and must be 0 (use platform or
     configured default) or a positive integer value of at least 32,768
     (32kB). If changing the thread stack size is unsupported, the
     error exception is raised.  If the specified stack
     size is invalid, a ValueError is raised and the stack
     size is unmodified.  32kB is currently the minimum supported stack
     size value to guarantee sufficient stack space for the interpreter
     itself.  Note that some platforms may have particular restrictions
     on values for the stack size, such as requiring a minimum stack
     size > 32kB or requiring allocation in multiples of the system
     memory page size - platform documentation should be referred to
     for more information (4kB pages are common; using multiples of
     4096 for the stack size is the suggested approach in the absence
     of more specific information).  Availability: Windows, systems
     with POSIX threads.

     New in version 2.5.

  Lock objects have the following methods:

 -- Method: lock.acquire ([waitflag])
     Without the optional argument, this method acquires the lock
     unconditionally, if necessary waiting until it is released by
     another thread (only one thread at a time can acquire a lock --
     that's their reason for existence).  If the integer _waitflag_
     argument is present, the action depends on its value: if it is
     zero, the lock is only acquired if it can be acquired immediately
     without waiting, while if it is nonzero, the lock is acquired
     unconditionally as before.  The return value is `True' if the lock
     is acquired successfully, `False' if not.

 -- Method: lock.release ()
     Releases the lock.  The lock must have been acquired earlier, but
     not necessarily by the same thread.

 -- Method: lock.locked ()
     Return the status of the lock: `True' if it has been acquired by
     some thread, `False' if not.

  In addition to these methods, lock objects can also be used via the
with statement, e.g.:

    import thread

    a_lock = thread.allocate_lock()

    with a_lock:
        print "a_lock is locked while this executes"

*Caveats:*

   * Threads interact strangely with interrupts: the
     KeyboardInterrupt.  exception will be received by an
     arbitrary thread.  (When the signal.  module is
     available, interrupts always go to the main thread.)

   * Calling sys.exit() or raising the SystemExit
     exception is equivalent to calling thread.exit().

   * Not all built-in functions that may block waiting for I/O allow
     other threads to run.  (The most popular ones (see time.sleep(),
     see file.read(), select.select()) work
     as expected.)

   * It is not possible to interrupt the `acquire()' method on a lock
     -- the KeyboardInterrupt exception will happen after
     the lock has been acquired.

   * When the main thread exits, it is system defined whether the other
     threads survive.  On SGI IRIX using the native thread
     implementation, they survive.  On most other systems, they are
     killed without executing try...  See finally.
     clauses or executing object destructors.

   * When the main thread exits, it does not do any of its usual
     cleanup (except that try... See finally.
     clauses are honored), and the standard I/O files are not flushed.

5.16.4 `dummy_threading' -- Drop-in replacement for the `threading' module
--------------------------------------------------------------------------

This module provides a duplicate interface to the threading.
module.  It is meant to be imported when the thread module
is not provided on a platform.

  Suggested usage is:

    try:
        import threading as _threading
    except ImportError:
        import dummy_threading as _threading

Be careful to not use this module where deadlock might occur from a
thread being created that blocks waiting for another thread to be
created.  This  often occurs with blocking I/O.

5.16.5 `dummy_thread' -- Drop-in replacement for the `thread' module
--------------------------------------------------------------------

     Note: The dummy_thread module has been renamed to
     `_dummy_thread' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0;
     however, you should consider using the high-lever
     dummy_threading module instead.

This module provides a duplicate interface to the thread.
module.  It is meant to be imported when the thread module
is not provided on a platform.

  Suggested usage is:

    try:
        import thread as _thread
    except ImportError:
        import dummy_thread as _thread

Be careful to not use this module where deadlock might occur from a
thread being created that blocks waiting for another thread to be
created.  This  often occurs with blocking I/O.

5.16.6 `multiprocessing' -- Process-based "threading" interface
---------------------------------------------------------------

New in version 2.6.

5.16.6.1 Introduction
.....................

multiprocessing is a package that supports spawning
processes using an API similar to the threading module.
The multiprocessing package offers both local and remote
concurrency, effectively side-stepping the Global Interpreter Lock
by using subprocesses instead of threads.  Due to this, the
multiprocessing module allows the programmer to fully
leverage multiple processors on a given machine.  It runs on both Unix
and Windows.

     Warning: Some of this package's functionality requires a
     functioning shared semaphore implementation on the host operating
     system. Without one, the `multiprocessing.synchronize' module will
     be disabled, and attempts to import it will result in an
     ImportError. See issue 3770
     (http://bugs.python.org/issue3770) for additional information.

     Note: Functionality within this package requires that the
     `__main__' method be importable by the children. This is covered
     in Programming guidelines.  however it is worth
     pointing out here. This means that some examples, such as the
     `multiprocessing.Pool' examples will not work in the interactive
     interpreter. For example:

         >>> from multiprocessing import Pool
         >>> p = Pool(5)
         >>> def f(x):
         ...     return x*x
         ...
         >>> p.map(f, [1,2,3])
         Process PoolWorker-1:
         Process PoolWorker-2:
         Process PoolWorker-3:
         Traceback (most recent call last):
         Traceback (most recent call last):
         Traceback (most recent call last):
         AttributeError: 'module' object has no attribute 'f'
         AttributeError: 'module' object has no attribute 'f'
         AttributeError: 'module' object has no attribute 'f'

     (If you try this it will actually output three full tracebacks
     interleaved in a semi-random fashion, and then you may have to
     stop the master process somehow.)

5.16.6.2 The `Process' class
............................

In multiprocessing, processes are spawned by creating a
Process.  object and then calling its start().
method.  See Process.  follows the API of
threading.Thread.  A trivial example of a multiprocess program is

    from multiprocessing import Process

    def f(name):
        print 'hello', name

    if __name__ == '__main__':
        p = Process(target=f, args=('bob',))
        p.start()
        p.join()

To show the individual process IDs involved, here is an expanded
example:

    from multiprocessing import Process
    import os

    def info(title):
        print title
        print 'module name:', __name__
        print 'parent process:', os.getppid()
        print 'process id:', os.getpid()

    def f(name):
        info('function f')
        print 'hello', name

    if __name__ == '__main__':
        info('main line')
        p = Process(target=f, args=('bob',))
        p.start()
        p.join()

For an explanation of why (on Windows) the `if __name__ == '__main__''
part is necessary, see Programming guidelines.

5.16.6.3 Exchanging objects between processes
.............................................

multiprocessing supports two types of communication channel
between processes:

  *Queues*

     The Queue class is a near clone of Queue.Queue.
     For example:

         from multiprocessing import Process, Queue

         def f(q):
             q.put([42, None, 'hello'])

         if __name__ == '__main__':
             q = Queue()
             p = Process(target=f, args=(q,))
             p.start()
             print q.get()    # prints "[42, None, 'hello']"
             p.join()

     Queues are thread and process safe.

  *Pipes*

     The Pipe() function returns a pair of connection
     objects connected by a pipe which by default is duplex (two-way).
     For example:

         from multiprocessing import Process, Pipe

         def f(conn):
             conn.send([42, None, 'hello'])
             conn.close()

         if __name__ == '__main__':
             parent_conn, child_conn = Pipe()
             p = Process(target=f, args=(child_conn,))
             p.start()
             print parent_conn.recv()   # prints "[42, None, 'hello']"
             p.join()

     The two connection objects returned by Pipe().
     represent the two ends of the pipe.  Each connection object has
     send() and recv() methods (among
     others).  Note that data in a pipe may become corrupted if two
     processes (or threads) try to read from or write to the _same_ end
     of the pipe at the same time.  Of course there is no risk of
     corruption from processes using different ends of the pipe at the
     same time.

5.16.6.4 Synchronization between processes
..........................................

multiprocessing contains equivalents of all the
synchronization primitives from threading.  For instance one
can use a lock to ensure that only one process prints to standard
output at a time:

    from multiprocessing import Process, Lock

    def f(l, i):
        l.acquire()
        print 'hello world', i
        l.release()

    if __name__ == '__main__':
        lock = Lock()

        for num in range(10):
            Process(target=f, args=(lock, num)).start()

Without using the lock output from the different processes is liable to
get all mixed up.

5.16.6.5 Sharing state between processes
........................................

As mentioned above, when doing concurrent programming it is usually
best to avoid using shared state as far as possible.  This is
particularly true when using multiple processes.

  However, if you really do need to use some shared data then
multiprocessing provides a couple of ways of doing so.

  *Shared memory*

     Data can be stored in a shared memory map using Value.
     or Array.  For example, the following code

         from multiprocessing import Process, Value, Array

         def f(n, a):
             n.value = 3.1415927
             for i in range(len(a)):
                 a[i] = -a[i]

         if __name__ == '__main__':
             num = Value('d', 0.0)
             arr = Array('i', range(10))

             p = Process(target=f, args=(num, arr))
             p.start()
             p.join()

             print num.value
             print arr[:]

     will print

         3.1415927
         [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]

     The `'d'' and `'i'' arguments used when creating `num' and `arr'
     are typecodes of the kind used by the array module:
     `'d'' indicates a double precision float and `'i'' indicates a
     signed integer.  These shared objects will be process and
     thread-safe.

     For more flexibility in using shared memory one can use the
     multiprocessing.sharedctypes module which supports the
     creation of arbitrary ctypes objects allocated from shared memory.

  *Server process*

     A manager object returned by `Manager()' controls a server process
     which holds Python objects and allows other processes to
     manipulate them using proxies.

     A manager returned by `Manager()' will support types list,
     see dict, `Namespace', see Lock, RLock,
     see Semaphore, BoundedSemaphore,
     Condition, Event, Queue, Value
     and Array.  For example,

         from multiprocessing import Process, Manager

         def f(d, l):
             d[1] = '1'
             d['2'] = 2
             d[0.25] = None
             l.reverse()

         if __name__ == '__main__':
             manager = Manager()

             d = manager.dict()
             l = manager.list(range(10))

             p = Process(target=f, args=(d, l))
             p.start()
             p.join()

             print d
             print l

     will print

         {0.25: None, 1: '1', '2': 2}
         [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

     Server process managers are more flexible than using shared memory
     objects because they can be made to support arbitrary object
     types.  Also, a single manager can be shared by processes on
     different computers over a network.  They are, however, slower
     than using shared memory.

5.16.6.6 Using a pool of workers
................................

The `Pool' class represents a pool of worker processes.  It has methods
which allows tasks to be offloaded to the worker processes in a few
different ways.

  For example:

    from multiprocessing import Pool

    def f(x):
        return x*x

    if __name__ == '__main__':
        pool = Pool(processes=4)              # start 4 worker processes
        result = pool.apply_async(f, [10])     # evaluate "f(10)" asynchronously
        print result.get(timeout=1)           # prints "100" unless your computer is *very* slow
        print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

5.16.6.7 Reference
..................

The multiprocessing package mostly replicates the API of the
threading module.

5.16.6.8 `Process' and exceptions
.................................

 -- Class: multiprocessing.Process ([group[, target[, name[, args[,
          kwargs]]]]])
     Process objects represent activity that is run in a separate
     process. The Process class has equivalents of all the
     methods of threading.Thread.

     The constructor should always be called with keyword arguments.
     _group_ should always be `None'; it exists solely for
     compatibility with threading.Thread.  _target_ is the
     callable object to be invoked by the run() method.
     It defaults to `None', meaning nothing is called. _name_ is the
     process name.  By default, a unique name is constructed of the
     form 'Process-N[1]:N[2]:...:N[k]' where N[1],N[2],...,N[k] is a
     sequence of integers whose length is determined by the
     _generation_ of the process.  _args_ is the argument tuple for the
     target invocation.  _kwargs_ is a dictionary of keyword arguments
     for the target invocation.  By default, no arguments are passed to
     _target_.

     If a subclass overrides the constructor, it must make sure it
     invokes the base class constructor (`Process.__init__()') before
     doing anything else to the process.

      -- Method: run ()
          Method representing the process's activity.

          You may override this method in a subclass.  The standard
          run().  method invokes the callable object passed
          to the object's constructor as the target argument, if any,
          with sequential and keyword arguments taken from the _args_
          and _kwargs_ arguments, respectively.

      -- Method: start ()
          Start the process's activity.

          This must be called at most once per process object.  It
          arranges for the object's run() method to be
          invoked in a separate process.

      -- Method: join ([timeout])
          Block the calling thread until the process whose
          join() method is called terminates or until the
          optional timeout occurs.

          If _timeout_ is `None' then there is no timeout.

          A process can be joined many times.

          A process cannot join itself because this would cause a
          deadlock.  It is an error to attempt to join a process before
          it has been started.

      -- Attribute: name
          The process's name.

          The name is a string used for identification purposes only.
          It has no semantics.  Multiple processes may be given the
          same name.  The initial name is set by the constructor.

      -- Method: is_alive ()
          Return whether the process is alive.

          Roughly, a process object is alive from the moment the
          start().  method returns until the child process
          terminates.

      -- Attribute: daemon
          The process's daemon flag, a Boolean value.  This must be set
          before start() is called.

          The initial value is inherited from the creating process.

          When a process exits, it attempts to terminate all of its
          daemonic child processes.

          Note that a daemonic process is not allowed to create child
          processes.  Otherwise a daemonic process would leave its
          children orphaned if it gets terminated when its parent
          process exits. Additionally, these are *not* Unix daemons or
          services, they are normal processes that will be terminated
          (and not joined) if non-daemonic processes have exited.

     In addition to the  `Threading.Thread' API, see Process.
     objects also support the following attributes and methods:

      -- Attribute: pid
          Return the process ID.  Before the process is spawned, this
          will be `None'.

      -- Attribute: exitcode
          The child's exit code.  This will be `None' if the process
          has not yet terminated.  A negative value _-N_ indicates that
          the child was terminated by signal _N_.

      -- Attribute: authkey
          The process's authentication key (a byte string).

          When multiprocessing is initialized the main
          process is assigned a random string using `os.random()'.

          When a Process object is created, it will
          inherit the authentication key of its parent process,
          although this may be changed by setting authkey.
          to another byte string.

          See Authentication keys.

      -- Method: terminate ()
          Terminate the process.  On Unix this is done using the
          `SIGTERM' signal; on Windows `TerminateProcess()' is used.
          Note that exit handlers and finally clauses, etc., will not
          be executed.

          Note that descendant processes of the process will _not_ be
          terminated - they will simply become orphaned.

               Warning: If this method is used when the associated
               process is using a pipe or queue then the pipe or queue
               is liable to become corrupted and may become unusable by
               other process.  Similarly, if the process has acquired a
               lock or semaphore etc. then terminating it is liable to
               cause other processes to deadlock.

     Note that the start(), join(),
     is_alive() and `exit_code' methods should only be called by
     the process that created the process object.

     Example usage of some of the methods of Process.:

         >>> import multiprocessing, time, signal
         >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
         >>> print p, p.is_alive()
         <Process(Process-1, initial)> False
         >>> p.start()
         >>> print p, p.is_alive()
         <Process(Process-1, started)> True
         >>> p.terminate()
         >>> time.sleep(0.1)
         >>> print p, p.is_alive()
         <Process(Process-1, stopped[SIGTERM])> False
         >>> p.exitcode == -signal.SIGTERM
         True

 -- Exception: multiprocessing.BufferTooShort
     Exception raised by Connection.recv_bytes_into() when
     the supplied buffer object is too small for the message read.

     If `e' is an instance of BufferTooShort then
     `e.args[0]' will give the message as a byte string.

5.16.6.9 Pipes and Queues
.........................

When using multiple processes, one generally uses message passing for
communication between processes and avoids having to use any
synchronization primitives like locks.

  For passing messages one can use Pipe(). (for a
connection between two processes) or a queue (which allows multiple
producers and consumers).

  The Queue and JoinableQueue types are
multi-producer, multi-consumer FIFO queues modelled on the
Queue.Queue class in the standard library.  They differ in that
Queue lacks the task_done() and join()
methods introduced into Python 2.5's Queue.Queue class.

  If you use JoinableQueue then you *must* call
JoinableQueue.task_done() for each task removed from the queue
or else the semaphore used to count the number of unfinished tasks may
eventually overflow raising an exception.

  Note that one can also create a shared queue by using a manager
object - see Managers.

     Note: multiprocessing uses the usual
     Queue.Empty and Queue.Full exceptions to signal
     a timeout.  They are not available in the multiprocessing
     namespace so you need to import them from Queue.

     Warning: If a process is killed using Process.terminate()
     or os.kill().  while it is trying to use a
     Queue, then the data in the queue is likely to become
     corrupted.  This may cause any other processes to get an exception
     when it tries to use the queue later on.

     Warning: As mentioned above, if a child process has put items on a
     queue (and it has not used `JoinableQueue.cancel_join_thread()'),
     then that process will not terminate until all buffered items have
     been flushed to the pipe.

     This means that if you try joining that process you may get a
     deadlock unless you are sure that all items which have been put on
     the queue have been consumed.  Similarly, if the child process is
     non-daemonic then the parent process may hang on exit when it
     tries to join all its non-daemonic children.

     Note that a queue created using a manager does not have this
     issue.  See Programming guidelines.

  For an example of the usage of queues for interprocess communication
see Examples.

 -- Function: multiprocessing.Pipe ([duplex])
     Returns a pair `(conn1, conn2)' of Connection objects
     representing the ends of a pipe.

     If _duplex_ is `True' (the default) then the pipe is
     bidirectional.  If _duplex_ is `False' then the pipe is
     unidirectional: `conn1' can only be used for receiving messages
     and `conn2' can only be used for sending messages.

 -- Class: multiprocessing.Queue ([maxsize])
     Returns a process shared queue implemented using a pipe and a few
     locks/semaphores.  When a process first puts an item on the queue
     a feeder thread is started which transfers objects from a buffer
     into the pipe.

     The usual Queue.Empty and Queue.Full.
     exceptions from the standard library's Queue module
     are raised to signal timeouts.

     Queue implements all the methods of Queue.Queue
     except for task_done() and join().

      -- Method: qsize ()
          Return the approximate size of the queue.  Because of
          multithreading/multiprocessing semantics, this number is not
          reliable.

          Note that this may raise NotImplementedError on
          Unix platforms like Mac OS X where `sem_getvalue()' is not
          implemented.

      -- Method: empty ()
          Return `True' if the queue is empty, `False' otherwise.
          Because of multithreading/multiprocessing semantics, this is
          not reliable.

      -- Method: full ()
          Return `True' if the queue is full, `False' otherwise.
          Because of multithreading/multiprocessing semantics, this is
          not reliable.

      -- Method: put (item[, block[, timeout]])
          Put item into the queue.  If the optional argument _block_ is
          `True' (the default) and _timeout_ is `None' (the default),
          block if necessary until a free slot is available.  If
          _timeout_ is a positive number, it blocks at most _timeout_
          seconds and raises the Queue.Full exception if no
          free slot was available within that time.  Otherwise (_block_
          is `False'), put an item on the queue if a free slot is
          immediately available, else raise the Queue.Full.
          exception (_timeout_ is ignored in that case).

      -- Method: put_nowait (item)
          Equivalent to `put(item, False)'.

      -- Method: get ([block[, timeout]])
          Remove and return an item from the queue.  If optional args
          _block_ is `True' (the default) and _timeout_ is `None' (the
          default), block if necessary until an item is available.  If
          _timeout_ is a positive number, it blocks at most _timeout_
          seconds and raises the Queue.Empty.  exception if
          no item was available within that time.  Otherwise (block is
          `False'), return an item if one is immediately available,
          else raise the Queue.Empty exception (_timeout_
          is ignored in that case).

      -- Method: get_nowait ()
      -- Method: get_no_wait ()
          Equivalent to `get(False)'.

     multiprocessing.Queue has a few additional methods
     not found in Queue.Queue.  These methods are usually
     unnecessary for most code:

      -- Method: close ()
          Indicate that no more data will be put on this queue by the
          current process.  The background thread will quit once it has
          flushed all buffered data to the pipe.  This is called
          automatically when the queue is garbage collected.

      -- Method: join_thread ()
          Join the background thread.  This can only be used after
          close() has been called.  It blocks until the
          background thread exits, ensuring that all data in the buffer
          has been flushed to the pipe.

          By default if a process is not the creator of the queue then
          on exit it will attempt to join the queue's background
          thread.  The process can call cancel_join_thread()
          to make join_thread() do nothing.

      -- Method: cancel_join_thread ()
          Prevent join_thread() from blocking.  In
          particular, this prevents the background thread from being
          joined automatically when the process exits - see
          join_thread().

 -- Class: multiprocessing.JoinableQueue ([maxsize])
     See JoinableQueue, a Queue subclass, is a
     queue which additionally has task_done() and
     join() methods.

      -- Method: task_done ()
          Indicate that a formerly enqueued task is complete. Used by
          queue consumer threads.  For each get() used to
          fetch a task, a subsequent call to task_done().
          tells the queue that the processing on the task is complete.

          If a `join()' is currently blocking, it will resume when all
          items have been processed (meaning that a task_done()
          call was received for every item that had been
          put() into the queue).

          Raises a ValueError if called more times than
          there were items placed in the queue.

      -- Method: join ()
          Block until all items in the queue have been gotten and
          processed.

          The count of unfinished tasks goes up whenever an item is
          added to the queue.  The count goes down whenever a consumer
          thread calls task_done() to indicate that the
          item was retrieved and all work on it is complete.  When the
          count of unfinished tasks drops to zero, `join()' unblocks.

5.16.6.10 Miscellaneous
.......................

 -- Function: multiprocessing.active_children ()
     Return list of all live children of the current process.

     Calling this has the side affect of "joining" any processes which
     have already finished.

 -- Function: multiprocessing.cpu_count ()
     Return the number of CPUs in the system.  May raise
     NotImplementedError.

 -- Function: multiprocessing.current_process ()
     Return the Process object corresponding to the
     current process.

     An analogue of threading.current_thread().

 -- Function: multiprocessing.freeze_support ()
     Add support for when a program which uses multiprocessing
     has been frozen to produce a Windows executable.  (Has been
     tested with *py2exe*, *PyInstaller* and *cx_Freeze*.)

     One needs to call this function straight after the `if __name__ ==
     '__main__'' line of the main module.  For example:

         from multiprocessing import Process, freeze_support

         def f():
             print 'hello world!'

         if __name__ == '__main__':
             freeze_support()
             Process(target=f).start()

     If the `freeze_support()' line is omitted then trying to run the
     frozen executable will raise RuntimeError.

     If the module is being run normally by the Python interpreter then
     freeze_support() has no effect.

 -- Function: multiprocessing.set_executable ()
     Sets the path of the Python interpreter to use when starting a
     child process.  (By default sys.executable is used).
     Embedders will probably need to do some thing like

         setExecutable(os.path.join(sys.exec_prefix, 'pythonw.exe'))

     before they can create child processes.  (Windows only)

     Note: multiprocessing contains no analogues of
     threading.active_count(), threading.enumerate(),
     threading.settrace(), threading.setprofile(),
     see threading.Timer, or threading.local.

5.16.6.11 Connection Objects
............................

Connection objects allow the sending and receiving of picklable objects
or strings.  They can be thought of as message oriented connected
sockets.

  Connection objects usually created using Pipe(). - see
also Listeners and Clients.

 -- Class: multiprocessing.Connection
      -- Method: send (obj)
          Send an object to the other end of the connection which
          should be read using recv().

          The object must be picklable.  Very large pickles
          (approximately 32 MB+, though it depends on the OS) may raise
          a ValueError exception.

      -- Method: recv ()
          Return an object sent from the other end of the connection
          using send().  Raises EOFError if
          there is nothing left to receive and the other end was closed.

      -- Method: fileno ()
          Returns the file descriptor or handle used by the connection.

      -- Method: close ()
          Close the connection.

          This is called automatically when the connection is garbage
          collected.

      -- Method: poll ([timeout])
          Return whether there is any data available to be read.

          If _timeout_ is not specified then it will return
          immediately.  If _timeout_ is a number then this specifies
          the maximum time in seconds to block.  If _timeout_ is `None'
          then an infinite timeout is used.

      -- Method: send_bytes (buffer[, offset[, size]])
          Send byte data from an object supporting the buffer interface
          as a complete message.

          If _offset_ is given then data is read from that position in
          _buffer_.  If _size_ is given then that many bytes will be
          read from buffer.  Very large buffers (approximately 32 MB+,
          though it depends on the OS) may raise a ValueError exception

      -- Method: recv_bytes ([maxlength])
          Return a complete message of byte data sent from the other
          end of the connection as a string.  Raises EOFError
          if there is nothing left to receive and the other end
          has closed.

          If _maxlength_ is specified and the message is longer than
          _maxlength_ then IOError is raised and the
          connection will no longer be readable.

      -- Method: recv_bytes_into (buffer[, offset])
          Read into _buffer_ a complete message of byte data sent from
          the other end of the connection and return the number of
          bytes in the message.  Raises EOFError if there
          is nothing left to receive and the other end was closed.

          _buffer_ must be an object satisfying the writable buffer
          interface.  If _offset_ is given then the message will be
          written into the buffer from that position.  Offset must be a
          non-negative integer less than the length of _buffer_ (in
          bytes).

          If the buffer is too short then a BufferTooShort.
          exception is raised and the complete message is available as
          `e.args[0]' where `e' is the exception instance.

  For example:

    >>> from multiprocessing import Pipe
    >>> a, b = Pipe()
    >>> a.send([1, 'hello', None])
    >>> b.recv()
    [1, 'hello', None]
    >>> b.send_bytes('thank you')
    >>> a.recv_bytes()
    'thank you'
    >>> import array
    >>> arr1 = array.array('i', range(5))
    >>> arr2 = array.array('i', [0] * 10)
    >>> a.send_bytes(arr1)
    >>> count = b.recv_bytes_into(arr2)
    >>> assert count == len(arr1) * arr1.itemsize
    >>> arr2
    array('i', [0, 1, 2, 3, 4, 0, 0, 0, 0, 0])

     Warning: The Connection.recv() method automatically
     unpickles the data it receives, which can be a security risk
     unless you can trust the process which sent the message.

     Therefore, unless the connection object was produced using
     Pipe() you should only use the recv() and
     send().  methods after performing some sort of
     authentication.  See Authentication keys.

     Warning: If a process is killed while it is trying to read or
     write to a pipe then the data in the pipe is likely to become
     corrupted, because it may become impossible to be sure where the
     message boundaries lie.

5.16.6.12 Synchronization primitives
....................................

Generally synchronization primitives are not as necessary in a
multiprocess program as they are in a multithreaded program.  See the
documentation for threading module.

  Note that one can also create synchronization primitives by using a
manager object - see Managers.

 -- Class: multiprocessing.BoundedSemaphore ([value])
     A bounded semaphore object: a clone of
     threading.BoundedSemaphore.

     (On Mac OS X, this is indistinguishable from Semaphore
     because `sem_getvalue()' is not implemented on that
     platform).

 -- Class: multiprocessing.Condition ([lock])
     A condition variable: a clone of threading.Condition.

     If _lock_ is specified then it should be a Lock or
     RLock.  object from multiprocessing.

 -- Class: multiprocessing.Event
     A clone of threading.Event.  This method returns the
     state of the internal semaphore on exit, so it will always return
     `True' except if a timeout is given and the operation times out.

     Changed in version 2.7: Previously, the method always returned
     `None'.

 -- Class: multiprocessing.Lock
     A non-recursive lock object: a clone of threading.Lock.

 -- Class: multiprocessing.RLock
     A recursive lock object: a clone of threading.RLock.

 -- Class: multiprocessing.Semaphore ([value])
     A bounded semaphore object: a clone of threading.Semaphore.

     Note: The `acquire()' method of BoundedSemaphore,
     Lock, RLock and Semaphore.
     has a timeout parameter not supported by the equivalents in
     threading.  The signature is `acquire(block=True,
     timeout=None)' with keyword parameters being acceptable.  If
     _block_ is `True' and _timeout_ is not `None' then it specifies a
     timeout in seconds.  If _block_ is `False' then _timeout_ is
     ignored.

     On Mac OS X, `sem_timedwait' is unsupported, so calling
     `acquire()' with a timeout will emulate that function's behavior
     using a sleeping loop.

     Note: If the SIGINT signal generated by Ctrl-C arrives while the
     main thread is blocked by a call to `BoundedSemaphore.acquire()',
     `Lock.acquire()', `RLock.acquire()', `Semaphore.acquire()',
     `Condition.acquire()' or `Condition.wait()' then the call will be
     immediately interrupted and KeyboardInterrupt will be
     raised.

     This differs from the behaviour of threading where
     SIGINT will be ignored while the equivalent blocking calls are in
     progress.

5.16.6.13 Shared `ctypes' Objects
.................................

It is possible to create shared objects using shared memory which can be
inherited by child processes.

 -- Function: multiprocessing.Value (typecode_or_type, *args[, lock])
     Return a ctypes object allocated from shared memory.
     By default the return value is actually a synchronized wrapper for
     the object.

     _typecode_or_type_ determines the type of the returned object: it
     is either a ctypes type or a one character typecode of the kind
     used by the array.  module.  _*args_ is passed on to the
     constructor for the type.

     If _lock_ is `True' (the default) then a new lock object is
     created to synchronize access to the value.  If _lock_ is a
     Lock or RLock object then that will be used to
     synchronize access to the value.  If _lock_ is `False' then access
     to the returned object will not be automatically protected by a
     lock, so it will not necessarily be "process-safe".

     Note that _lock_ is a keyword-only argument.

 -- Function: multiprocessing.Array (typecode_or_type,
          size_or_initializer, *, lock=True)
     Return a ctypes array allocated from shared memory.  By default
     the return value is actually a synchronized wrapper for the array.

     _typecode_or_type_ determines the type of the elements of the
     returned array: it is either a ctypes type or a one character
     typecode of the kind used by the array module.  If
     _size_or_initializer_ is an integer, then it determines the length
     of the array, and the array will be initially zeroed.  Otherwise,
     _size_or_initializer_ is a sequence which is used to initialize
     the array and whose length determines the length of the array.

     If _lock_ is `True' (the default) then a new lock object is
     created to synchronize access to the value.  If _lock_ is a
     Lock or RLock object then that will be used to
     synchronize access to the value.  If _lock_ is `False' then access
     to the returned object will not be automatically protected by a
     lock, so it will not necessarily be "process-safe".

     Note that _lock_ is a keyword only argument.

     Note that an array of ctypes.c_char has _value_ and
     _raw_ attributes which allow one to use it to store and retrieve
     strings.

5.16.6.14 The `multiprocessing.sharedctypes' module
...................................................

The multiprocessing.sharedctypes module provides functions
for allocating ctypes objects from shared memory which can
be inherited by child processes.

     Note: Although it is possible to store a pointer in shared memory
     remember that this will refer to a location in the address space
     of a specific process.  However, the pointer is quite likely to be
     invalid in the context of a second process and trying to
     dereference the pointer from the second process may cause a crash.

 -- Function: multiprocessing.sharedctypes.RawArray (typecode_or_type,
          size_or_initializer)
     Return a ctypes array allocated from shared memory.

     _typecode_or_type_ determines the type of the elements of the
     returned array: it is either a ctypes type or a one character
     typecode of the kind used by the array module.  If
     _size_or_initializer_ is an integer then it determines the length
     of the array, and the array will be initially zeroed.  Otherwise
     _size_or_initializer_ is a sequence which is used to initialize the
     array and whose length determines the length of the array.

     Note that setting and getting an element is potentially non-atomic
     - use Array() instead to make sure that access is
     automatically synchronized using a lock.

 -- Function: multiprocessing.sharedctypes.RawValue (typecode_or_type,
          *args)
     Return a ctypes object allocated from shared memory.

     _typecode_or_type_ determines the type of the returned object: it
     is either a ctypes type or a one character typecode of the kind
     used by the array.  module.  _*args_ is passed on to the
     constructor for the type.

     Note that setting and getting the value is potentially non-atomic
     - use Value() instead to make sure that access is
     automatically synchronized using a lock.

     Note that an array of ctypes.c_char has `value' and
     `raw' attributes which allow one to use it to store and retrieve
     strings - see documentation for ctypes.

 -- Function: multiprocessing.sharedctypes.Array (typecode_or_type,
          size_or_initializer, *args[, lock])
     The same as RawArray() except that depending on the
     value of _lock_ a process-safe synchronization wrapper may be
     returned instead of a raw ctypes array.

     If _lock_ is `True' (the default) then a new lock object is
     created to synchronize access to the value.  If _lock_ is a `Lock'
     or `RLock' object then that will be used to synchronize access to
     the value.  If _lock_ is `False' then access to the returned
     object will not be automatically protected by a lock, so it will
     not necessarily be "process-safe".

     Note that _lock_ is a keyword-only argument.

 -- Function: multiprocessing.sharedctypes.Value (typecode_or_type,
          *args[, lock])
     The same as RawValue() except that depending on the
     value of _lock_ a process-safe synchronization wrapper may be
     returned instead of a raw ctypes object.

     If _lock_ is `True' (the default) then a new lock object is
     created to synchronize access to the value.  If _lock_ is a `Lock'
     or `RLock' object then that will be used to synchronize access to
     the value.  If _lock_ is `False' then access to the returned
     object will not be automatically protected by a lock, so it will
     not necessarily be "process-safe".

     Note that _lock_ is a keyword-only argument.

 -- Function: multiprocessing.sharedctypes.copy (obj)
     Return a ctypes object allocated from shared memory which is a
     copy of the ctypes object _obj_.

 -- Function: multiprocessing.sharedctypes.synchronized (obj[, lock])
     Return a process-safe wrapper object for a ctypes object which
     uses _lock_ to synchronize access.  If _lock_ is `None' (the
     default) then a multiprocessing.RLock object is
     created automatically.

     A synchronized wrapper will have two methods in addition to those
     of the object it wraps: `get_obj()' returns the wrapped object and
     `get_lock()' returns the lock object used for synchronization.

     Note that accessing the ctypes object through the wrapper can be a
     lot slower than accessing the raw ctypes object.

  The table below compares the syntax for creating shared ctypes
objects from shared memory with the normal ctypes syntax.  (In the
table `MyStruct' is some subclass of ctypes.Structure)

ctypes                   sharedctypes using type        sharedctypes using typecode
----------------------------------------------------------------------------------------
c_double(2.4)            RawValue(c_double, 2.4)        RawValue('d', 2.4)
MyStruct(4, 6)           RawValue(MyStruct, 4, 6)
(c_short * 7)()          RawArray(c_short, 7)           RawArray('h', 7)
(c_int * 3)(9, 2, 8)     RawArray(c_int, (9, 2, 8))     RawArray('i', (9, 2, 8))

  Below is an example where a number of ctypes objects are modified by
a child process:

    from multiprocessing import Process, Lock
    from multiprocessing.sharedctypes import Value, Array
    from ctypes import Structure, c_double

    class Point(Structure):
        _fields_ = [('x', c_double), ('y', c_double)]

    def modify(n, x, s, A):
        n.value **= 2
        x.value **= 2
        s.value = s.value.upper()
        for a in A:
            a.x **= 2
            a.y **= 2

    if __name__ == '__main__':
        lock = Lock()

        n = Value('i', 7)
        x = Value(c_double, 1.0/3.0, lock=False)
        s = Array('c', 'hello world', lock=lock)
        A = Array(Point, [(1.875,-6.25), (-5.75,2.0), (2.375,9.5)], lock=lock)

        p = Process(target=modify, args=(n, x, s, A))
        p.start()
        p.join()

        print n.value
        print x.value
        print s.value
        print [(a.x, a.y) for a in A]

The results printed are

    49
    0.1111111111111111
    HELLO WORLD
    [(3.515625, 39.0625), (33.0625, 4.0), (5.640625, 90.25)]

5.16.6.15 Managers
..................

Managers provide a way to create data which can be shared between
different processes. A manager object controls a server process which
manages _shared objects_.  Other processes can access the shared
objects by using proxies.

 -- Function: multiprocessing.Manager ()
     Returns a started SyncManager object which can be
     used for sharing objects between processes.  The returned manager
     object corresponds to a spawned child process and has methods
     which will create shared objects and return corresponding proxies.

  Manager processes will be shutdown as soon as they are garbage
collected or their parent process exits.  The manager classes are
defined in the multiprocessing.managers module:

 -- Class: multiprocessing.managers.BaseManager ([address[, authkey]])
     Create a BaseManager object.

     Once created one should call start() or
     `get_server().serve_forever()' to ensure that the manager object
     refers to a started manager process.

     _address_ is the address on which the manager process listens for
     new connections.  If _address_ is `None' then an arbitrary one is
     chosen.

     _authkey_ is the authentication key which will be used to check
     the validity of incoming connections to the server process.  If
     _authkey_ is `None' then `current_process().authkey'.  Otherwise
     _authkey_ is used and it must be a string.

      -- Method: start ([initializer[, initargs]])
          Start a subprocess to start the manager.  If _initializer_ is
          not `None' then the subprocess will call
          `initializer(*initargs)' when it starts.

      -- Method: get_server ()
          Returns a `Server' object which represents the actual server
          under the control of the Manager. The `Server' object
          supports the `serve_forever()' method:

              >>> from multiprocessing.managers import BaseManager
              >>> manager = BaseManager(address=('', 50000), authkey='abc')
              >>> server = manager.get_server()
              >>> server.serve_forever()

          `Server' additionally has an address attribute.

      -- Method: connect ()
          Connect a local manager object to a remote manager process:

              >>> from multiprocessing.managers import BaseManager
              >>> m = BaseManager(address=('127.0.0.1', 5000), authkey='abc')
              >>> m.connect()

      -- Method: shutdown ()
          Stop the process used by the manager.  This is only available
          if start() has been used to start the server
          process.

          This can be called multiple times.

      -- Method: register (typeid[, callable[, proxytype[, exposed[,
               method_to_typeid[, create_method]]]]])
          A classmethod which can be used for registering a type or
          callable with the manager class.

          _typeid_ is a "type identifier" which is used to identify a
          particular type of shared object.  This must be a string.

          _callable_ is a callable used for creating objects for this
          type identifier.  If a manager instance will be created using
          the `from_address()' classmethod or if the _create_method_
          argument is `False' then this can be left as `None'.

          _proxytype_ is a subclass of BaseProxy which is
          used to create proxies for shared objects with this _typeid_.
          If `None' then a proxy class is created automatically.

          _exposed_ is used to specify a sequence of method names which
          proxies for this typeid should be allowed to access using
          `BaseProxy._callMethod()'.  (If _exposed_ is `None' then
          `proxytype._exposed_' is used instead if it exists.)  In the
          case where no exposed list is specified, all "public methods"
          of the shared object will be accessible.  (Here a "public
          method" means any attribute which has a __call__()
          method and whose name does not begin with `'_''.)

          _method_to_typeid_ is a mapping used to specify the return
          type of those exposed methods which should return a proxy.
          It maps method names to typeid strings.  (If
          _method_to_typeid_ is `None' then
          `proxytype._method_to_typeid_' is used instead if it exists.)
          If a method's name is not a key of this mapping or if the
          mapping is `None' then the object returned by the method will
          be copied by value.

          _create_method_ determines whether a method should be created
          with name _typeid_ which can be used to tell the server
          process to create a new shared object and return a proxy for
          it.  By default it is `True'.

     BaseManager instances also have one read-only
     property:

      -- Attribute: address
          The address used by the manager.

 -- Class: multiprocessing.managers.SyncManager
     A subclass of BaseManager which can be used for the
     synchronization of processes.  Objects of this type are returned by
     `multiprocessing.Manager()'.

     It also supports creation of shared lists and dictionaries.

      -- Method: BoundedSemaphore ([value])
          Create a shared threading.BoundedSemaphore.
          object and return a proxy for it.

      -- Method: Condition ([lock])
          Create a shared threading.Condition object and
          return a proxy for it.

          If _lock_ is supplied then it should be a proxy for a
          threading.Lock or threading.RLock object.

      -- Method: Event ()
          Create a shared threading.Event object and return
          a proxy for it.

      -- Method: Lock ()
          Create a shared threading.Lock object and return
          a proxy for it.

      -- Method: Namespace ()
          Create a shared Namespace object and return a
          proxy for it.

      -- Method: Queue ([maxsize])
          Create a shared Queue.Queue object and return a
          proxy for it.

      -- Method: RLock ()
          Create a shared threading.RLock object and
          return a proxy for it.

      -- Method: Semaphore ([value])
          Create a shared threading.Semaphore object and
          return a proxy for it.

      -- Method: Array (typecode, sequence)
          Create an array and return a proxy for it.

      -- Method: Value (typecode, value)
          Create an object with a writable `value' attribute and return
          a proxy for it.

      -- Method: dict ()
      -- Method: dict (mapping)
      -- Method: dict (sequence)
          Create a shared `dict' object and return a proxy for it.

      -- Method: list ()
      -- Method: list (sequence)
          Create a shared `list' object and return a proxy for it.

          Note: Modifications to mutable values or items in dict and
          list proxies will not be propagated through the manager,
          because the proxy has no way of knowing when its values or
          items are modified.  To modify such an item, you can
          re-assign the modified object to the container proxy:

              # create a list proxy and append a mutable object (a dictionary)
              lproxy = manager.list()
              lproxy.append({})
              # now mutate the dictionary
              d = lproxy[0]
              d['a'] = 1
              d['b'] = 2
              # at this point, the changes to d are not yet synced, but by
              # reassigning the dictionary, the proxy is notified of the change
              lproxy[0] = d

5.16.6.16 Namespace objects
...........................

A namespace object has no public methods, but does have writable
attributes.  Its representation shows the values of its attributes.

  However, when using a proxy for a namespace object, an attribute
beginning with `'_'' will be an attribute of the proxy and not an
attribute of the referent:

    >>> manager = multiprocessing.Manager()
    >>> Global = manager.Namespace()
    >>> Global.x = 10
    >>> Global.y = 'hello'
    >>> Global._z = 12.3    # this is an attribute of the proxy
    >>> print Global
    Namespace(x=10, y='hello')

5.16.6.17 Customized managers
.............................

To create one's own manager, one creates a subclass of
BaseManager and use the register() classmethod to
register new types or callables with the manager class.  For example:

    from multiprocessing.managers import BaseManager

    class MathsClass(object):
        def add(self, x, y):
            return x + y
        def mul(self, x, y):
            return x * y

    class MyManager(BaseManager):
        pass

    MyManager.register('Maths', MathsClass)

    if __name__ == '__main__':
        manager = MyManager()
        manager.start()
        maths = manager.Maths()
        print maths.add(4, 3)         # prints 7
        print maths.mul(7, 8)         # prints 56

5.16.6.18 Using a remote manager
................................

It is possible to run a manager server on one machine and have clients
use it from other machines (assuming that the firewalls involved allow
it).

  Running the following commands creates a server for a single shared
queue which remote clients can access:

    >>> from multiprocessing.managers import BaseManager
    >>> import Queue
    >>> queue = Queue.Queue()
    >>> class QueueManager(BaseManager): pass
    >>> QueueManager.register('get_queue', callable=lambda:queue)
    >>> m = QueueManager(address=('', 50000), authkey='abracadabra')
    >>> s = m.get_server()
    >>> s.serve_forever()

One client can access the server as follows:

    >>> from multiprocessing.managers import BaseManager
    >>> class QueueManager(BaseManager): pass
    >>> QueueManager.register('get_queue')
    >>> m = QueueManager(address=('foo.bar.org', 50000), authkey='abracadabra')
    >>> m.connect()
    >>> queue = m.get_queue()
    >>> queue.put('hello')

Another client can also use it:

    >>> from multiprocessing.managers import BaseManager
    >>> class QueueManager(BaseManager): pass
    >>> QueueManager.register('get_queue')
    >>> m = QueueManager(address=('foo.bar.org', 50000), authkey='abracadabra')
    >>> m.connect()
    >>> queue = m.get_queue()
    >>> queue.get()
    'hello'

Local processes can also access that queue, using the code from above
on the client to access it remotely:

    >>> from multiprocessing import Process, Queue
    >>> from multiprocessing.managers import BaseManager
    >>> class Worker(Process):
    ...     def __init__(self, q):
    ...         self.q = q
    ...         super(Worker, self).__init__()
    ...     def run(self):
    ...         self.q.put('local hello')
    ...
    >>> queue = Queue()
    >>> w = Worker(queue)
    >>> w.start()
    >>> class QueueManager(BaseManager): pass
    ...
    >>> QueueManager.register('get_queue', callable=lambda: queue)
    >>> m = QueueManager(address=('', 50000), authkey='abracadabra')
    >>> s = m.get_server()
    >>> s.serve_forever()

5.16.6.19 Proxy Objects
.......................

A proxy is an object which _refers_ to a shared object which lives
(presumably) in a different process.  The shared object is said to be
the _referent_ of the proxy.  Multiple proxy objects may have the same
referent.

  A proxy object has methods which invoke corresponding methods of its
referent (although not every method of the referent will necessarily be
available through the proxy).  A proxy can usually be used in most of
the same ways that its referent can:

    >>> from multiprocessing import Manager
    >>> manager = Manager()
    >>> l = manager.list([i*i for i in range(10)])
    >>> print l
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >>> print repr(l)
    <ListProxy object, typeid 'list' at 0x...>
    >>> l[4]
    16
    >>> l[2:5]
    [4, 9, 16]

Notice that applying str() to a proxy will return the
representation of the referent, whereas applying repr().
will return the representation of the proxy.

  An important feature of proxy objects is that they are picklable so
they can be passed between processes.  Note, however, that if a proxy
is sent to the corresponding manager's process then unpickling it will
produce the referent itself.  This means, for example, that one shared
object can contain a second:

    >>> a = manager.list()
    >>> b = manager.list()
    >>> a.append(b)         # referent of a now contains referent of b
    >>> print a, b
    [[]] []
    >>> b.append('hello')
    >>> print a, b
    [['hello']] ['hello']

     Note: The proxy types in multiprocessing do nothing to
     support comparisons by value.  So, for instance, we have:

         >>> manager.list([1,2,3]) == [1,2,3]
         False

     One should just use a copy of the referent instead when making
     comparisons.

 -- Class: multiprocessing.managers.BaseProxy
     Proxy objects are instances of subclasses of BaseProxy.

      -- Method: _callmethod (methodname[, args[, kwds]])
          Call and return the result of a method of the proxy's
          referent.

          If `proxy' is a proxy whose referent is `obj' then the
          expression

              proxy._callmethod(methodname, args, kwds)

          will evaluate the expression

              getattr(obj, methodname)(*args, **kwds)

          in the manager's process.

          The returned value will be a copy of the result of the call
          or a proxy to a new shared object - see documentation for the
          _method_to_typeid_ argument of BaseManager.register().

          If an exception is raised by the call, then then is re-raised
          by _callmethod().  If some other exception is
          raised in the manager's process then this is converted into a
          `RemoteError' exception and is raised by _callmethod().

          Note in particular that an exception will be raised if
          _methodname_ has not been _exposed_

          An example of the usage of _callmethod().:

              >>> l = manager.list(range(10))
              >>> l._callmethod('__len__')
              10
              >>> l._callmethod('__getslice__', (2, 7))   # equiv to `l[2:7]`
              [2, 3, 4, 5, 6]
              >>> l._callmethod('__getitem__', (20,))     # equiv to `l[20]`
              Traceback (most recent call last):
              ...
              IndexError: list index out of range

      -- Method: _getvalue ()
          Return a copy of the referent.

          If the referent is unpicklable then this will raise an
          exception.

      -- Method: __repr__ ()
          Return a representation of the proxy object.

      -- Method: __str__ ()
          Return the representation of the referent.

5.16.6.20 Cleanup
.................

A proxy object uses a weakref callback so that when it gets garbage
collected it deregisters itself from the manager which owns its
referent.

  A shared object gets deleted from the manager process when there are
no longer any proxies referring to it.

5.16.6.21 Process Pools
.......................

One can create a pool of processes which will carry out tasks submitted
to it with the `Pool' class.

 -- Class: multiprocessing.Pool ([processes[, initializer[, initargs[,
          maxtasksperchild]]]])
     A process pool object which controls a pool of worker processes to
     which jobs can be submitted.  It supports asynchronous results
     with timeouts and callbacks and has a parallel map implementation.

     _processes_ is the number of worker processes to use.  If
     _processes_ is `None' then the number returned by `cpu_count()' is
     used.  If _initializer_ is not `None' then each worker process
     will call `initializer(*initargs)' when it starts.

     New in version 2.7: _maxtasksperchild_ is the number of tasks a
     worker process can complete before it will exit and be replaced
     with a fresh worker process, to enable unused resources to be
     freed. The default _maxtasksperchild_ is None, which means worker
     processes will live as long as the pool.

          Note: Worker processes within a `Pool' typically live for the
          complete duration of the Pool's work queue. A frequent
          pattern found in other systems (such as Apache, mod_wsgi,
          etc) to free resources held by workers is to allow a worker
          within a pool to complete only a set amount of work before
          being exiting, being cleaned up and a new process spawned to
          replace the old one. The _maxtasksperchild_ argument to the
          `Pool' exposes this ability to the end user.

      -- Method: apply (func[, args[, kwds]])
          Equivalent of the apply() built-in function.  It
          blocks till the result is ready.  Given this blocks, see
          apply_async() is better suited for performing work in
          parallel. Additionally, the passed in function is only
          executed in one of the workers of the pool.

      -- Method: apply_async (func[, args[, kwds[, callback]]])
          A variant of the apply() method which returns a
          result object.

          If _callback_ is specified then it should be a callable which
          accepts a single argument.  When the result becomes ready
          _callback_ is applied to it (unless the call failed).
          _callback_ should complete immediately since otherwise the
          thread which handles the results will get blocked.

      -- Method: map (func, iterable[, chunksize])
          A parallel equivalent of the map() built-in
          function (it supports only one _iterable_ argument though).
          It blocks till the result is ready.

          This method chops the iterable into a number of chunks which
          it submits to the process pool as separate tasks.  The
          (approximate) size of these chunks can be specified by
          setting _chunksize_ to a positive integer.

      -- Method: map_async (func, iterable[, chunksize[, callback]])
          A variant of the map() method which returns a
          result object.

          If _callback_ is specified then it should be a callable which
          accepts a single argument.  When the result becomes ready
          _callback_ is applied to it (unless the call failed).
          _callback_ should complete immediately since otherwise the
          thread which handles the results will get blocked.

      -- Method: imap (func, iterable[, chunksize])
          An equivalent of itertools.imap().

          The _chunksize_ argument is the same as the one used by the
          map().  method.  For very long iterables using a
          large value for _chunksize_ can make make the job complete
          *much* faster than using the default value of `1'.

          Also if _chunksize_ is `1' then the `next()' method of the
          iterator returned by the imap() method has an
          optional _timeout_ parameter: `next(timeout)' will raise
          `multiprocessing.TimeoutError' if the result cannot be
          returned within _timeout_ seconds.

      -- Method: imap_unordered (func, iterable[, chunksize])
          The same as imap() except that the ordering of
          the results from the returned iterator should be considered
          arbitrary.  (Only when there is only one worker process is
          the order guaranteed to be "correct".)

      -- Method: close ()
          Prevents any more tasks from being submitted to the pool.
          Once all the tasks have been completed the worker processes
          will exit.

      -- Method: terminate ()
          Stops the worker processes immediately without completing
          outstanding work.  When the pool object is garbage collected
          terminate() will be called immediately.

      -- Method: join ()
          Wait for the worker processes to exit.  One must call
          close() or terminate() before using
          join().

 -- Class: multiprocessing.pool.AsyncResult
     The class of the result returned by `Pool.apply_async()' and
     `Pool.map_async()'.

      -- Method: get ([timeout])
          Return the result when it arrives.  If _timeout_ is not
          `None' and the result does not arrive within _timeout_
          seconds then `multiprocessing.TimeoutError' is raised.  If
          the remote call raised an exception then that exception will
          be reraised by get().

      -- Method: wait ([timeout])
          Wait until the result is available or until _timeout_ seconds
          pass.

      -- Method: ready ()
          Return whether the call has completed.

      -- Method: successful ()
          Return whether the call completed without raising an
          exception.  Will raise AssertionError if the
          result is not ready.

  The following example demonstrates the use of a pool:

    from multiprocessing import Pool

    def f(x):
        return x*x

    if __name__ == '__main__':
        pool = Pool(processes=4)              # start 4 worker processes

        result = pool.apply_async(f, (10,))    # evaluate "f(10)" asynchronously
        print result.get(timeout=1)           # prints "100" unless your computer is *very* slow

        print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

        it = pool.imap(f, range(10))
        print it.next()                       # prints "0"
        print it.next()                       # prints "1"
        print it.next(timeout=1)              # prints "4" unless your computer is *very* slow

        import time
        result = pool.apply_async(time.sleep, (10,))
        print result.get(timeout=1)           # raises TimeoutError

5.16.6.22 Listeners and Clients
...............................

Usually message passing between processes is done using queues or by
using `Connection' objects returned by `Pipe()'.

  However, the multiprocessing.connection module allows
some extra flexibility.  It basically gives a high level message
oriented API for dealing with sockets or Windows named pipes, and also
has support for _digest authentication_ using the hmac.
module.

 -- Function: multiprocessing.connection.deliver_challenge (connection,
          authkey)
     Send a randomly generated message to the other end of the
     connection and wait for a reply.

     If the reply matches the digest of the message using _authkey_ as
     the key then a welcome message is sent to the other end of the
     connection.  Otherwise AuthenticationError is raised.

 -- Function: multiprocessing.connection.answerChallenge (connection,
          authkey)
     Receive a message, calculate the digest of the message using
     _authkey_ as the key, and then send the digest back.

     If a welcome message is not received, then
     AuthenticationError is raised.

 -- Function: multiprocessing.connection.Client (address[, family[,
          authenticate[, authkey]]])
     Attempt to set up a connection to the listener which is using
     address _address_, returning a Connection.

     The type of the connection is determined by _family_ argument, but
     this can generally be omitted since it can usually be inferred
     from the format of _address_. (See Address Formats)

     If _authenticate_ is `True' or _authkey_ is a string then digest
     authentication is used.  The key used for authentication will be
     either _authkey_ or `current_process().authkey)' if _authkey_ is
     `None'.  If authentication fails then AuthenticationError
     is raised.  See Authentication keys.

 -- Class: multiprocessing.connection.Listener ([address[, family[,
          backlog[, authenticate[, authkey]]]]])
     A wrapper for a bound socket or Windows named pipe which is
     'listening' for connections.

     _address_ is the address to be used by the bound socket or named
     pipe of the listener object.

          Note: If an address of '0.0.0.0' is used, the address will
          not be a connectable end point on Windows. If you require a
          connectable end-point, you should use '127.0.0.1'.

     _family_ is the type of socket (or named pipe) to use.  This can
     be one of the strings `'AF_INET'' (for a TCP socket), `'AF_UNIX''
     (for a Unix domain socket) or `'AF_PIPE'' (for a Windows named
     pipe).  Of these only the first is guaranteed to be available.  If
     _family_ is `None' then the family is inferred from the format of
     _address_.  If _address_ is also `None' then a default is chosen.
     This default is the family which is assumed to be the fastest
     available.  See Address Formats.  Note that if
     _family_ is `'AF_UNIX'' and address is `None' then the socket will
     be created in a private temporary directory created using
     tempfile.mkstemp().

     If the listener object uses a socket then _backlog_ (1 by default)
     is passed to the `listen()' method of the socket once it has been
     bound.

     If _authenticate_ is `True' (`False' by default) or _authkey_ is
     not `None' then digest authentication is used.

     If _authkey_ is a string then it will be used as the
     authentication key; otherwise it must be _None_.

     If _authkey_ is `None' and _authenticate_ is `True' then
     `current_process().authkey' is used as the authentication key.  If
     _authkey_ is `None' and _authenticate_ is `False' then no
     authentication is done.  If authentication fails then
     AuthenticationError is raised.  See Authentication keys.

      -- Method: accept ()
          Accept a connection on the bound socket or named pipe of the
          listener object and return a `Connection' object.  If
          authentication is attempted and fails, then
          AuthenticationError is raised.

      -- Method: close ()
          Close the bound socket or named pipe of the listener object.
          This is called automatically when the listener is garbage
          collected.  However it is advisable to call it explicitly.

     Listener objects have the following read-only properties:

      -- Attribute: address
          The address which is being used by the Listener object.

      -- Attribute: last_accepted
          The address from which the last accepted connection came.  If
          this is unavailable then it is `None'.

  The module defines two exceptions:

 -- Exception: multiprocessing.connection.AuthenticationError
     Exception raised when there is an authentication error.

  *Examples*

  The following server code creates a listener which uses `'secret
password'' as an authentication key.  It then waits for a connection
and sends some data to the client:

    from multiprocessing.connection import Listener
    from array import array

    address = ('localhost', 6000)     # family is deduced to be 'AF_INET'
    listener = Listener(address, authkey='secret password')

    conn = listener.accept()
    print 'connection accepted from', listener.last_accepted

    conn.send([2.25, None, 'junk', float])

    conn.send_bytes('hello')

    conn.send_bytes(array('i', [42, 1729]))

    conn.close()
    listener.close()

The following code connects to the server and receives some data from
the server:

    from multiprocessing.connection import Client
    from array import array

    address = ('localhost', 6000)
    conn = Client(address, authkey='secret password')

    print conn.recv()                 # => [2.25, None, 'junk', float]

    print conn.recv_bytes()            # => 'hello'

    arr = array('i', [0, 0, 0, 0, 0])
    print conn.recv_bytes_into(arr)     # => 8
    print arr                         # => array('i', [42, 1729, 0, 0, 0])

    conn.close()

5.16.6.23 Address Formats
.........................

   * An `'AF_INET'' address is a tuple of the form `(hostname, port)'
     where _hostname_ is a string and _port_ is an integer.

   * An `'AF_UNIX'' address is a string representing a filename on the
     filesystem.

   *
    An `'AF_PIPE'' address is a string of the form
          `r'\\.\pipe\_PipeName_''.  To use Client() to
          connect to a named pipe on a remote computer called
          _ServerName_ one should use an address of the form
          `r'\\_ServerName_\pipe\_PipeName_'' instead.

Note that any string beginning with two backslashes is assumed by
default to be an `'AF_PIPE'' address rather than an `'AF_UNIX'' address.

5.16.6.24 Authentication keys
.............................

When one uses `Connection.recv()', the data received is automatically
unpickled.  Unfortunately unpickling data from an untrusted source is a
security risk.  Therefore Listener and Client()
use the hmac module to provide digest authentication.

  An authentication key is a string which can be thought of as a
password: once a connection is established both ends will demand proof
that the other knows the authentication key.  (Demonstrating that both
ends are using the same key does *not* involve sending the key over the
connection.)

  If authentication is requested but do authentication key is specified
then the return value of `current_process().authkey' is used (see
Process).  This value will automatically inherited by any
Process object that the current process creates.  This means
that (by default) all processes of a multi-process program will share a
single authentication key which can be used when setting up connections
between themselves.

  Suitable authentication keys can also be generated by using
os.urandom().

5.16.6.25 Logging
.................

Some support for logging is available.  Note, however, that the
logging.  package does not use process shared locks so it is
possible (depending on the handler type) for messages from different
processes to get mixed up.

 -- Function: multiprocessing.get_logger ()
     Returns the logger used by multiprocessing.  If
     necessary, a new one will be created.

     When first created the logger has level `logging.NOTSET' and no
     default handler. Messages sent to this logger will not by default
     propagate to the root logger.

     Note that on Windows child processes will only inherit the level
     of the parent process's logger - any other customization of the
     logger will not be inherited.

 -- Function: multiprocessing.log_to_stderr ()
     This function performs a call to get_logger() but in
     addition to returning the logger created by get_logger, it adds a
     handler which sends output to sys.stderr using format
     `'[%(levelname)s/%(processName)s] %(message)s''.

  Below is an example session with logging turned on:

    >>> import multiprocessing, logging
    >>> logger = multiprocessing.log_to_stderr()
    >>> logger.setLevel(logging.INFO)
    >>> logger.warning('doomed')
    [WARNING/MainProcess] doomed
    >>> m = multiprocessing.Manager()
    [INFO/SyncManager-...] child process calling self.run()
    [INFO/SyncManager-...] created temp directory /.../pymp-...
    [INFO/SyncManager-...] manager serving at '/.../listener-...'
    >>> del m
    [INFO/MainProcess] sending shutdown message to manager
    [INFO/SyncManager-...] manager exiting with exitcode 0

In addition to having these two logging functions, the multiprocessing
also exposes two additional logging level attributes. These are
`SUBWARNING' and `SUBDEBUG'. The table below illustrates where theses
fit in the normal level hierarchy.

Level                Numeric value
------------------------------------------
`SUBWARNING'         25
`SUBDEBUG'           5

  For a full table of logging levels, see the logging.
module.

  These additional logging levels are used primarily for certain debug
messages within the multiprocessing module. Below is the same example
as above, except with `SUBDEBUG' enabled:

    >>> import multiprocessing, logging
    >>> logger = multiprocessing.log_to_stderr()
    >>> logger.setLevel(multiprocessing.SUBDEBUG)
    >>> logger.warning('doomed')
    [WARNING/MainProcess] doomed
    >>> m = multiprocessing.Manager()
    [INFO/SyncManager-...] child process calling self.run()
    [INFO/SyncManager-...] created temp directory /.../pymp-...
    [INFO/SyncManager-...] manager serving at '/.../pymp-djGBXN/listener-...'
    >>> del m
    [SUBDEBUG/MainProcess] finalizer calling ...
    [INFO/MainProcess] sending shutdown message to manager
    [DEBUG/SyncManager-...] manager received shutdown message
    [SUBDEBUG/SyncManager-...] calling <Finalize object, callback=unlink, ...
    [SUBDEBUG/SyncManager-...] finalizer calling <built-in function unlink> ...
    [SUBDEBUG/SyncManager-...] calling <Finalize object, dead>
    [SUBDEBUG/SyncManager-...] finalizer calling <function rmtree at 0x5aa730> ...
    [INFO/SyncManager-...] manager exiting with exitcode 0

5.16.6.26 The `multiprocessing.dummy' module
............................................

multiprocessing.dummy replicates the API of
multiprocessing but is no more than a wrapper around the
threading module.

5.16.6.27 Programming guidelines
................................

There are certain guidelines and idioms which should be adhered to when
using multiprocessing.

5.16.6.28 All platforms
.......................

Avoid shared state

     As far as possible one should try to avoid shifting large amounts
     of data between processes.

     It is probably best to stick to using queues or pipes for
     communication between processes rather than using the lower level
     synchronization primitives from the threading module.

  Picklability

     Ensure that the arguments to the methods of proxies are picklable.

  Thread safety of proxies

     Do not use a proxy object from more than one thread unless you
     protect it with a lock.

     (There is never a problem with different processes using the
     _same_ proxy.)

  Joining zombie processes

     On Unix when a process finishes but has not been joined it becomes
     a zombie.  There should never be very many because each time a new
     process starts (or `active_children()' is called) all completed
     processes which have not yet been joined will be joined.  Also
     calling a finished process's `Process.is_alive()' will join the
     process.  Even so it is probably good practice to explicitly join
     all the processes that you start.

  Better to inherit than pickle/unpickle

     On Windows many types from multiprocessing need to be
     picklable so that child processes can use them.  However, one
     should generally avoid sending shared objects to other processes
     using pipes or queues.  Instead you should arrange the program so
     that a process which need access to a shared resource created
     elsewhere can inherit it from an ancestor process.

  Avoid terminating processes

     Using the `Process.terminate()' method to stop a process is liable
     to cause any shared resources (such as locks, semaphores, pipes
     and queues) currently being used by the process to become broken
     or unavailable to other processes.

     Therefore it is probably best to only consider using
     `Process.terminate()' on processes which never use any shared
     resources.

  Joining processes that use queues

     Bear in mind that a process that has put items in a queue will
     wait before terminating until all the buffered items are fed by
     the "feeder" thread to the underlying pipe.  (The child process
     can call the `Queue.cancel_join_thread()' method of the queue to
     avoid this behaviour.)

     This means that whenever you use a queue you need to make sure
     that all items which have been put on the queue will eventually be
     removed before the process is joined.  Otherwise you cannot be
     sure that processes which have put items on the queue will
     terminate.  Remember also that non-daemonic processes will be
     automatically be joined.

     An example which will deadlock is the following:

         from multiprocessing import Process, Queue

         def f(q):
             q.put('X' * 1000000)

         if __name__ == '__main__':
             queue = Queue()
             p = Process(target=f, args=(queue,))
             p.start()
             p.join()                    # this deadlocks
             obj = queue.get()

     A fix here would be to swap the last two lines round (or simply
     remove the `p.join()' line).

  Explicitly pass resources to child processes

     On Unix a child process can make use of a shared resource created
     in a parent process using a global resource.  However, it is
     better to pass the object as an argument to the constructor for
     the child process.

     Apart from making the code (potentially) compatible with Windows
     this also ensures that as long as the child process is still alive
     the object will not be garbage collected in the parent process.
     This might be important if some resource is freed when the object
     is garbage collected in the parent process.

     So for instance

         from multiprocessing import Process, Lock

         def f():
             ... do something using "lock" ...

         if __name__ == '__main__':
            lock = Lock()
            for i in range(10):
                 Process(target=f).start()

     should be rewritten as

         from multiprocessing import Process, Lock

         def f(l):
             ... do something using "l" ...

         if __name__ == '__main__':
            lock = Lock()
            for i in range(10):
                 Process(target=f, args=(lock,)).start()

  Beware replacing sys.stdin with a "file like object"

     multiprocessing originally unconditionally called:

         os.close(sys.stdin.fileno())

     in the `multiprocessing.Process._bootstrap()' method -- this
     resulted in issues with processes-in-processes. This has been
     changed to:

         sys.stdin.close()
         sys.stdin = open(os.devnull)

     Which solves the fundamental issue of processes colliding with
     each other resulting in a bad file descriptor error, but
     introduces a potential danger to applications which replace
     sys.stdin() with a "file-like object" with output buffering.
     This danger is that if multiple processes call `close()' on this
     file-like object, it could result in the same data being flushed
     to the object multiple times, resulting in corruption.

     If you write a file-like object and implement your own caching,
     you can make it fork-safe by storing the pid whenever you append
     to the cache, and discarding the cache when the pid changes. For
     example:

         @property
         def cache(self):
             pid = os.getpid()
             if pid != self._pid:
                 self._pid = pid
                 self._cache = []
             return self._cache

     For more information, see issue 5155
     (http://bugs.python.org/issue5155), issue 5313
     (http://bugs.python.org/issue5313) and issue 5331
     (http://bugs.python.org/issue5331)

5.16.6.29 Windows
.................

Since Windows lacks os.fork() it has a few extra
restrictions:

  More picklability

     Ensure that all arguments to `Process.__init__()' are picklable.
     This means, in particular, that bound or unbound methods cannot be
     used directly as the `target' argument on Windows -- just define a
     function and use that instead.

     Also, if you subclass `Process' then make sure that instances will
     be picklable when the `Process.start()' method is called.

  Global variables

     Bear in mind that if code run in a child process tries to access a
     global variable, then the value it sees (if any) may not be the
     same as the value in the parent process at the time that
     `Process.start()' was called.

     However, global variables which are just module level constants
     cause no problems.

  Safe importing of main module

     Make sure that the main module can be safely imported by a new
     Python interpreter without causing unintended side effects (such a
     starting a new process).

     For example, under Windows running the following module would fail
     with a RuntimeError.:

         from multiprocessing import Process

         def foo():
             print 'hello'

         p = Process(target=foo)
         p.start()

     Instead one should protect the "entry point" of the program by
     using `if __name__ == '__main__':' as follows:

         from multiprocessing import Process, freeze_support

         def foo():
             print 'hello'

         if __name__ == '__main__':
             freeze_support()
             p = Process(target=foo)
             p.start()

     (The `freeze_support()' line can be omitted if the program will be
     run normally instead of frozen.)

     This allows the newly spawned Python interpreter to safely import
     the module and then run the module's `foo()' function.

     Similar restrictions apply if a pool or manager is created in the
     main module.

5.16.6.30 Examples
..................

Demonstration of how to create and use customized managers and proxies:

    #
    # This module shows how to use arbitrary callables with a subclass of
    # `BaseManager`.
    #
    # Copyright (c) 2006-2008, R Oudkerk
    # All rights reserved.
    #

    from multiprocessing import freeze_support
    from multiprocessing.managers import BaseManager, BaseProxy
    import operator

    ##

    class Foo(object):
        def f(self):
            print 'you called Foo.f()'
        def g(self):
            print 'you called Foo.g()'
        def _h(self):
            print 'you called Foo._h()'

    # A simple generator function
    def baz():
        for i in xrange(10):
            yield i*i

    # Proxy type for generator objects
    class GeneratorProxy(BaseProxy):
        _exposed_ = ('next', '__next__')
        def __iter__(self):
            return self
        def next(self):
            return self._callmethod('next')
        def __next__(self):
            return self._callmethod('__next__')

    # Function to return the operator module
    def get_operator_module():
        return operator

    ##

    class MyManager(BaseManager):
        pass

    # register the Foo class; make `f()` and `g()` accessible via proxy
    MyManager.register('Foo1', Foo)

    # register the Foo class; make `g()` and `_h()` accessible via proxy
    MyManager.register('Foo2', Foo, exposed=('g', '_h'))

    # register the generator function baz; use `GeneratorProxy` to make proxies
    MyManager.register('baz', baz, proxytype=GeneratorProxy)

    # register get_operator_module(); make public functions accessible via proxy
    MyManager.register('operator', get_operator_module)

    ##

    def test():
        manager = MyManager()
        manager.start()

        print '-' * 20

        f1 = manager.Foo1()
        f1.f()
        f1.g()
        assert not hasattr(f1, '_h')
        assert sorted(f1._exposed_) == sorted(['f', 'g'])

        print '-' * 20

        f2 = manager.Foo2()
        f2.g()
        f2._h()
        assert not hasattr(f2, 'f')
        assert sorted(f2._exposed_) == sorted(['g', '_h'])

        print '-' * 20

        it = manager.baz()
        for i in it:
            print '<%d>' % i,
        print

        print '-' * 20

        op = manager.operator()
        print 'op.add(23, 45) =', op.add(23, 45)
        print 'op.pow(2, 94) =', op.pow(2, 94)
        print 'op.getslice(range(10), 2, 6) =', op.getslice(range(10), 2, 6)
        print 'op.repeat(range(5), 3) =', op.repeat(range(5), 3)
        print 'op._exposed_ =', op._exposed_

    ##

    if __name__ == '__main__':
        freeze_support()
        test()

Using `Pool':

    #
    # A test of `multiprocessing.Pool` class
    #
    # Copyright (c) 2006-2008, R Oudkerk
    # All rights reserved.
    #

    import multiprocessing
    import time
    import random
    import sys

    #
    # Functions used by test code
    #

    def calculate(func, args):
        result = func(*args)
        return '%s says that %s%s = %s' % (
            multiprocessing.current_process().name,
            func.__name__, args, result
            )

    def calculatestar(args):
        return calculate(*args)

    def mul(a, b):
        time.sleep(0.5*random.random())
        return a * b

    def plus(a, b):
        time.sleep(0.5*random.random())
        return a + b

    def f(x):
        return 1.0 / (x-5.0)

    def pow3(x):
        return x**3

    def noop(x):
        pass

    #
    # Test code
    #

    def test():
        print 'cpu_count() = %d\n' % multiprocessing.cpu_count()

        #
        # Create pool
        #

        PROCESSES = 4
        print 'Creating pool with %d processes\n' % PROCESSES
        pool = multiprocessing.Pool(PROCESSES)
        print 'pool = %s' % pool
        print

        #
        # Tests
        #

        TASKS = [(mul, (i, 7)) for i in range(10)] + \
                [(plus, (i, 8)) for i in range(10)]

        results = [pool.apply_async(calculate, t) for t in TASKS]
        imap_it = pool.imap(calculatestar, TASKS)
        imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

        print 'Ordered results using pool.apply_async():'
        for r in results:
            print '\t', r.get()
        print

        print 'Ordered results using pool.imap():'
        for x in imap_it:
            print '\t', x
        print

        print 'Unordered results using pool.imap_unordered():'
        for x in imap_unordered_it:
            print '\t', x
        print

        print 'Ordered results using pool.map() --- will block till complete:'
        for x in pool.map(calculatestar, TASKS):
            print '\t', x
        print

        #
        # Simple benchmarks
        #

        N = 100000
        print 'def pow3(x): return x**3'

        t = time.time()
        A = map(pow3, xrange(N))
        print '\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
              (N, time.time() - t)

        t = time.time()
        B = pool.map(pow3, xrange(N))
        print '\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
              (N, time.time() - t)

        t = time.time()
        C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
        print '\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
              ' seconds' % (N, N//8, time.time() - t)

        assert A == B == C, (len(A), len(B), len(C))
        print

        L = [None] * 1000000
        print 'def noop(x): pass'
        print 'L = [None] * 1000000'

        t = time.time()
        A = map(noop, L)
        print '\tmap(noop, L):\n\t\t%s seconds' % \
              (time.time() - t)

        t = time.time()
        B = pool.map(noop, L)
        print '\tpool.map(noop, L):\n\t\t%s seconds' % \
              (time.time() - t)

        t = time.time()
        C = list(pool.imap(noop, L, chunksize=len(L)//8))
        print '\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
              (len(L)//8, time.time() - t)

        assert A == B == C, (len(A), len(B), len(C))
        print

        del A, B, C, L

        #
        # Test error handling
        #

        print 'Testing error handling:'

        try:
            print pool.apply(f, (5,))
        except ZeroDivisionError:
            print '\tGot ZeroDivisionError as expected from pool.apply()'
        else:
            raise AssertionError('expected ZeroDivisionError')

        try:
            print pool.map(f, range(10))
        except ZeroDivisionError:
            print '\tGot ZeroDivisionError as expected from pool.map()'
        else:
            raise AssertionError('expected ZeroDivisionError')

        try:
            print list(pool.imap(f, range(10)))
        except ZeroDivisionError:
            print '\tGot ZeroDivisionError as expected from list(pool.imap())'
        else:
            raise AssertionError('expected ZeroDivisionError')

        it = pool.imap(f, range(10))
        for i in range(10):
            try:
                x = it.next()
            except ZeroDivisionError:
                if i == 5:
                    pass
            except StopIteration:
                break
            else:
                if i == 5:
                    raise AssertionError('expected ZeroDivisionError')

        assert i == 9
        print '\tGot ZeroDivisionError as expected from IMapIterator.next()'
        print

        #
        # Testing timeouts
        #

        print 'Testing ApplyResult.get() with timeout:',
        res = pool.apply_async(calculate, TASKS[0])
        while 1:
            sys.stdout.flush()
            try:
                sys.stdout.write('\n\t%s' % res.get(0.02))
                break
            except multiprocessing.TimeoutError:
                sys.stdout.write('.')
        print
        print

        print 'Testing IMapIterator.next() with timeout:',
        it = pool.imap(calculatestar, TASKS)
        while 1:
            sys.stdout.flush()
            try:
                sys.stdout.write('\n\t%s' % it.next(0.02))
            except StopIteration:
                break
            except multiprocessing.TimeoutError:
                sys.stdout.write('.')
        print
        print

        #
        # Testing callback
        #

        print 'Testing callback:'

        A = []
        B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]

        r = pool.apply_async(mul, (7, 8), callback=A.append)
        r.wait()

        r = pool.map_async(pow3, range(10), callback=A.extend)
        r.wait()

        if A == B:
            print '\tcallbacks succeeded\n'
        else:
            print '\t*** callbacks failed\n\t\t%s != %s\n' % (A, B)

        #
        # Check there are no outstanding tasks
        #

        assert not pool._cache, 'cache = %r' % pool._cache

        #
        # Check close() methods
        #

        print 'Testing close():'

        for worker in pool._pool:
            assert worker.is_alive()

        result = pool.apply_async(time.sleep, [0.5])
        pool.close()
        pool.join()

        assert result.get() is None

        for worker in pool._pool:
            assert not worker.is_alive()

        print '\tclose() succeeded\n'

        #
        # Check terminate() method
        #

        print 'Testing terminate():'

        pool = multiprocessing.Pool(2)
        DELTA = 0.1
        ignore = pool.apply(pow3, [2])
        results = [pool.apply_async(time.sleep, [DELTA]) for i in range(100)]
        pool.terminate()
        pool.join()

        for worker in pool._pool:
            assert not worker.is_alive()

        print '\tterminate() succeeded\n'

        #
        # Check garbage collection
        #

        print 'Testing garbage collection:'

        pool = multiprocessing.Pool(2)
        DELTA = 0.1
        processes = pool._pool
        ignore = pool.apply(pow3, [2])
        results = [pool.apply_async(time.sleep, [DELTA]) for i in range(100)]

        results = pool = None

        time.sleep(DELTA * 2)

        for worker in processes:
            assert not worker.is_alive()

        print '\tgarbage collection succeeded\n'

    if __name__ == '__main__':
        multiprocessing.freeze_support()

        assert len(sys.argv) in (1, 2)

        if len(sys.argv) == 1 or sys.argv[1] == 'processes':
            print ' Using processes '.center(79, '-')
        elif sys.argv[1] == 'threads':
            print ' Using threads '.center(79, '-')
            import multiprocessing.dummy as multiprocessing
        else:
            print 'Usage:\n\t%s [processes | threads]' % sys.argv[0]
            raise SystemExit(2)

        test()

Synchronization types like locks, conditions and queues:

    #
    # A test file for the `multiprocessing` package
    #
    # Copyright (c) 2006-2008, R Oudkerk
    # All rights reserved.
    #

    import time, sys, random
    from Queue import Empty

    import multiprocessing               # may get overwritten

    #### TEST_VALUE

    def value_func(running, mutex):
        random.seed()
        time.sleep(random.random()*4)

        mutex.acquire()
        print '\n\t\t\t' + str(multiprocessing.current_process()) + ' has finished'
        running.value -= 1
        mutex.release()

    def test_value():
        TASKS = 10
        running = multiprocessing.Value('i', TASKS)
        mutex = multiprocessing.Lock()

        for i in range(TASKS):
            p = multiprocessing.Process(target=value_func, args=(running, mutex))
            p.start()

        while running.value > 0:
            time.sleep(0.08)
            mutex.acquire()
            print running.value,
            sys.stdout.flush()
            mutex.release()

        print
        print 'No more running processes'

    #### TEST_QUEUE

    def queue_func(queue):
        for i in range(30):
            time.sleep(0.5 * random.random())
            queue.put(i*i)
        queue.put('STOP')

    def test_queue():
        q = multiprocessing.Queue()

        p = multiprocessing.Process(target=queue_func, args=(q,))
        p.start()

        o = None
        while o != 'STOP':
            try:
                o = q.get(timeout=0.3)
                print o,
                sys.stdout.flush()
            except Empty:
                print 'TIMEOUT'

        print

    #### TEST_CONDITION

    def condition_func(cond):
        cond.acquire()
        print '\t' + str(cond)
        time.sleep(2)
        print '\tchild is notifying'
        print '\t' + str(cond)
        cond.notify()
        cond.release()

    def test_condition():
        cond = multiprocessing.Condition()

        p = multiprocessing.Process(target=condition_func, args=(cond,))
        print cond

        cond.acquire()
        print cond
        cond.acquire()
        print cond

        p.start()

        print 'main is waiting'
        cond.wait()
        print 'main has woken up'

        print cond
        cond.release()
        print cond
        cond.release()

        p.join()
        print cond

    #### TEST_SEMAPHORE

    def semaphore_func(sema, mutex, running):
        sema.acquire()

        mutex.acquire()
        running.value += 1
        print running.value, 'tasks are running'
        mutex.release()

        random.seed()
        time.sleep(random.random()*2)

        mutex.acquire()
        running.value -= 1
        print '%s has finished' % multiprocessing.current_process()
        mutex.release()

        sema.release()

    def test_semaphore():
        sema = multiprocessing.Semaphore(3)
        mutex = multiprocessing.RLock()
        running = multiprocessing.Value('i', 0)

        processes = [
            multiprocessing.Process(target=semaphore_func,
                                    args=(sema, mutex, running))
            for i in range(10)
            ]

        for p in processes:
            p.start()

        for p in processes:
            p.join()

    #### TEST_JOIN_TIMEOUT

    def join_timeout_func():
        print '\tchild sleeping'
        time.sleep(5.5)
        print '\n\tchild terminating'

    def test_join_timeout():
        p = multiprocessing.Process(target=join_timeout_func)
        p.start()

        print 'waiting for process to finish'

        while 1:
            p.join(timeout=1)
            if not p.is_alive():
                break
            print '.',
            sys.stdout.flush()

    #### TEST_EVENT

    def event_func(event):
        print '\t%r is waiting' % multiprocessing.current_process()
        event.wait()
        print '\t%r has woken up' % multiprocessing.current_process()

    def test_event():
        event = multiprocessing.Event()

        processes = [multiprocessing.Process(target=event_func, args=(event,))
                     for i in range(5)]

        for p in processes:
            p.start()

        print 'main is sleeping'
        time.sleep(2)

        print 'main is setting event'
        event.set()

        for p in processes:
            p.join()

    #### TEST_SHAREDVALUES

    def sharedvalues_func(values, arrays, shared_values, shared_arrays):
        for i in range(len(values)):
            v = values[i][1]
            sv = shared_values[i].value
            assert v == sv

        for i in range(len(values)):
            a = arrays[i][1]
            sa = list(shared_arrays[i][:])
            assert a == sa

        print 'Tests passed'

    def test_sharedvalues():
        values = [
            ('i', 10),
            ('h', -2),
            ('d', 1.25)
            ]
        arrays = [
            ('i', range(100)),
            ('d', [0.25 * i for i in range(100)]),
            ('H', range(1000))
            ]

        shared_values = [multiprocessing.Value(id, v) for id, v in values]
        shared_arrays = [multiprocessing.Array(id, a) for id, a in arrays]

        p = multiprocessing.Process(
            target=sharedvalues_func,
            args=(values, arrays, shared_values, shared_arrays)
            )
        p.start()
        p.join()

        assert p.exitcode == 0

    ####

    def test(namespace=multiprocessing):
        global multiprocessing

        multiprocessing = namespace

        for func in [ test_value, test_queue, test_condition,
                      test_semaphore, test_join_timeout, test_event,
                      test_sharedvalues ]:

            print '\n\t######## %s\n' % func.__name__
            func()

        ignore = multiprocessing.active_children()      # cleanup any old processes
        if hasattr(multiprocessing, '_debug_info'):
            info = multiprocessing._debug_info()
            if info:
                print info
                raise ValueError('there should be no positive refcounts left')

    if __name__ == '__main__':
        multiprocessing.freeze_support()

        assert len(sys.argv) in (1, 2)

        if len(sys.argv) == 1 or sys.argv[1] == 'processes':
            print ' Using processes '.center(79, '-')
            namespace = multiprocessing
        elif sys.argv[1] == 'manager':
            print ' Using processes and a manager '.center(79, '-')
            namespace = multiprocessing.Manager()
            namespace.Process = multiprocessing.Process
            namespace.current_process = multiprocessing.current_process
            namespace.active_children = multiprocessing.active_children
        elif sys.argv[1] == 'threads':
            print ' Using threads '.center(79, '-')
            import multiprocessing.dummy as namespace
        else:
            print 'Usage:\n\t%s [processes | manager | threads]' % sys.argv[0]
            raise SystemExit(2)

        test(namespace)

An example showing how to use queues to feed tasks to a collection of
worker process and collect the results:

    #
    # Simple example which uses a pool of workers to carry out some tasks.
    #
    # Notice that the results will probably not come out of the output
    # queue in the same in the same order as the corresponding tasks were
    # put on the input queue.  If it is important to get the results back
    # in the original order then consider using `Pool.map()` or
    # `Pool.imap()` (which will save on the amount of code needed anyway).
    #
    # Copyright (c) 2006-2008, R Oudkerk
    # All rights reserved.
    #

    import time
    import random

    from multiprocessing import Process, Queue, current_process, freeze_support

    #
    # Function run by worker processes
    #

    def worker(input, output):
        for func, args in iter(input.get, 'STOP'):
            result = calculate(func, args)
            output.put(result)

    #
    # Function used to calculate result
    #

    def calculate(func, args):
        result = func(*args)
        return '%s says that %s%s = %s' % \
            (current_process().name, func.__name__, args, result)

    #
    # Functions referenced by tasks
    #

    def mul(a, b):
        time.sleep(0.5*random.random())
        return a * b

    def plus(a, b):
        time.sleep(0.5*random.random())
        return a + b

    #
    #
    #

    def test():
        NUMBER_OF_PROCESSES = 4
        TASKS1 = [(mul, (i, 7)) for i in range(20)]
        TASKS2 = [(plus, (i, 8)) for i in range(10)]

        # Create queues
        task_queue = Queue()
        done_queue = Queue()

        # Submit tasks
        for task in TASKS1:
            task_queue.put(task)

        # Start worker processes
        for i in range(NUMBER_OF_PROCESSES):
            Process(target=worker, args=(task_queue, done_queue)).start()

        # Get and print results
        print 'Unordered results:'
        for i in range(len(TASKS1)):
            print '\t', done_queue.get()

        # Add more tasks using `put()`
        for task in TASKS2:
            task_queue.put(task)

        # Get and print some more results
        for i in range(len(TASKS2)):
            print '\t', done_queue.get()

        # Tell child processes to stop
        for i in range(NUMBER_OF_PROCESSES):
            task_queue.put('STOP')

    if __name__ == '__main__':
        freeze_support()
        test()

An example of how a pool of worker processes can each run a
`SimpleHTTPServer.HttpServer' instance while sharing a single listening
socket.

    #
    # Example where a pool of http servers share a single listening socket
    #
    # On Windows this module depends on the ability to pickle a socket
    # object so that the worker processes can inherit a copy of the server
    # object.  (We import `multiprocessing.reduction` to enable this pickling.)
    #
    # Not sure if we should synchronize access to `socket.accept()` method by
    # using a process-shared lock -- does not seem to be necessary.
    #
    # Copyright (c) 2006-2008, R Oudkerk
    # All rights reserved.
    #

    import os
    import sys

    from multiprocessing import Process, current_process, freeze_support
    from BaseHTTPServer import HTTPServer
    from SimpleHTTPServer import SimpleHTTPRequestHandler

    if sys.platform == 'win32':
        import multiprocessing.reduction    # make sockets pickable/inheritable

    def note(format, *args):
        sys.stderr.write('[%s]\t%s\n' % (current_process().name, format%args))

    class RequestHandler(SimpleHTTPRequestHandler):
        # we override log_message() to show which process is handling the request
        def log_message(self, format, *args):
            note(format, *args)

    def serve_forever(server):
        note('starting server')
        try:
            server.serve_forever()
        except KeyboardInterrupt:
            pass

    def runpool(address, number_of_processes):
        # create a single server object -- children will each inherit a copy
        server = HTTPServer(address, RequestHandler)

        # create child processes to act as workers
        for i in range(number_of_processes-1):
            Process(target=serve_forever, args=(server,)).start()

        # main process also acts as a worker
        serve_forever(server)

    def test():
        DIR = os.path.join(os.path.dirname(__file__), '..')
        ADDRESS = ('localhost', 8000)
        NUMBER_OF_PROCESSES = 4

        print 'Serving at http://%s:%d using %d worker processes' % \
              (ADDRESS[0], ADDRESS[1], NUMBER_OF_PROCESSES)
        print 'To exit press Ctrl-' + ['C', 'Break'][sys.platform=='win32']

        os.chdir(DIR)
        runpool(ADDRESS, NUMBER_OF_PROCESSES)

    if __name__ == '__main__':
        freeze_support()
        test()

Some simple benchmarks comparing multiprocessing with
threading.:

    #
    # Simple benchmarks for the multiprocessing package
    #
    # Copyright (c) 2006-2008, R Oudkerk
    # All rights reserved.
    #

    import time, sys, multiprocessing, threading, Queue, gc

    if sys.platform == 'win32':
        _timer = time.clock
    else:
        _timer = time.time

    delta = 1

    #### TEST_QUEUESPEED

    def queuespeed_func(q, c, iterations):
        a = '0' * 256
        c.acquire()
        c.notify()
        c.release()

        for i in xrange(iterations):
            q.put(a)

        q.put('STOP')

    def test_queuespeed(Process, q, c):
        elapsed = 0
        iterations = 1

        while elapsed < delta:
            iterations *= 2

            p = Process(target=queuespeed_func, args=(q, c, iterations))
            c.acquire()
            p.start()
            c.wait()
            c.release()

            result = None
            t = _timer()

            while result != 'STOP':
                result = q.get()

            elapsed = _timer() - t

            p.join()

        print iterations, 'objects passed through the queue in', elapsed, 'seconds'
        print 'average number/sec:', iterations/elapsed

    #### TEST_PIPESPEED

    def pipe_func(c, cond, iterations):
        a = '0' * 256
        cond.acquire()
        cond.notify()
        cond.release()

        for i in xrange(iterations):
            c.send(a)

        c.send('STOP')

    def test_pipespeed():
        c, d = multiprocessing.Pipe()
        cond = multiprocessing.Condition()
        elapsed = 0
        iterations = 1

        while elapsed < delta:
            iterations *= 2

            p = multiprocessing.Process(target=pipe_func,
                                        args=(d, cond, iterations))
            cond.acquire()
            p.start()
            cond.wait()
            cond.release()

            result = None
            t = _timer()

            while result != 'STOP':
                result = c.recv()

            elapsed = _timer() - t
            p.join()

        print iterations, 'objects passed through connection in',elapsed,'seconds'
        print 'average number/sec:', iterations/elapsed

    #### TEST_SEQSPEED

    def test_seqspeed(seq):
        elapsed = 0
        iterations = 1

        while elapsed < delta:
            iterations *= 2

            t = _timer()

            for i in xrange(iterations):
                a = seq[5]

            elapsed = _timer()-t

        print iterations, 'iterations in', elapsed, 'seconds'
        print 'average number/sec:', iterations/elapsed

    #### TEST_LOCK

    def test_lockspeed(l):
        elapsed = 0
        iterations = 1

        while elapsed < delta:
            iterations *= 2

            t = _timer()

            for i in xrange(iterations):
                l.acquire()
                l.release()

            elapsed = _timer()-t

        print iterations, 'iterations in', elapsed, 'seconds'
        print 'average number/sec:', iterations/elapsed

    #### TEST_CONDITION

    def conditionspeed_func(c, N):
        c.acquire()
        c.notify()

        for i in xrange(N):
            c.wait()
            c.notify()

        c.release()

    def test_conditionspeed(Process, c):
        elapsed = 0
        iterations = 1

        while elapsed < delta:
            iterations *= 2

            c.acquire()
            p = Process(target=conditionspeed_func, args=(c, iterations))
            p.start()

            c.wait()

            t = _timer()

            for i in xrange(iterations):
                c.notify()
                c.wait()

            elapsed = _timer()-t

            c.release()
            p.join()

        print iterations * 2, 'waits in', elapsed, 'seconds'
        print 'average number/sec:', iterations * 2 / elapsed

    ####

    def test():
        manager = multiprocessing.Manager()

        gc.disable()

        print '\n\t######## testing Queue.Queue\n'
        test_queuespeed(threading.Thread, Queue.Queue(),
                        threading.Condition())
        print '\n\t######## testing multiprocessing.Queue\n'
        test_queuespeed(multiprocessing.Process, multiprocessing.Queue(),
                        multiprocessing.Condition())
        print '\n\t######## testing Queue managed by server process\n'
        test_queuespeed(multiprocessing.Process, manager.Queue(),
                        manager.Condition())
        print '\n\t######## testing multiprocessing.Pipe\n'
        test_pipespeed()

        print

        print '\n\t######## testing list\n'
        test_seqspeed(range(10))
        print '\n\t######## testing list managed by server process\n'
        test_seqspeed(manager.list(range(10)))
        print '\n\t######## testing Array("i", ..., lock=False)\n'
        test_seqspeed(multiprocessing.Array('i', range(10), lock=False))
        print '\n\t######## testing Array("i", ..., lock=True)\n'
        test_seqspeed(multiprocessing.Array('i', range(10), lock=True))

        print

        print '\n\t######## testing threading.Lock\n'
        test_lockspeed(threading.Lock())
        print '\n\t######## testing threading.RLock\n'
        test_lockspeed(threading.RLock())
        print '\n\t######## testing multiprocessing.Lock\n'
        test_lockspeed(multiprocessing.Lock())
        print '\n\t######## testing multiprocessing.RLock\n'
        test_lockspeed(multiprocessing.RLock())
        print '\n\t######## testing lock managed by server process\n'
        test_lockspeed(manager.Lock())
        print '\n\t######## testing rlock managed by server process\n'
        test_lockspeed(manager.RLock())

        print

        print '\n\t######## testing threading.Condition\n'
        test_conditionspeed(threading.Thread, threading.Condition())
        print '\n\t######## testing multiprocessing.Condition\n'
        test_conditionspeed(multiprocessing.Process, multiprocessing.Condition())
        print '\n\t######## testing condition managed by a server process\n'
        test_conditionspeed(multiprocessing.Process, manager.Condition())

        gc.enable()

    if __name__ == '__main__':
        multiprocessing.freeze_support()
        test()

5.16.7 `mmap' -- Memory-mapped file support
-------------------------------------------

Memory-mapped file objects behave like both strings and like file
objects.  Unlike normal string objects, however, these are mutable.
You can use mmap objects in most places where strings are expected; for
example, you can use the re module to search through a
memory-mapped file.  Since they're mutable, you can change a single
character by doing `obj[index] = 'a'', or change a substring by
assigning to a slice: `obj[i1:i2] = '...''.  You can also read and
write data starting at the current file position, and seek()
through the file to different positions.

  A memory-mapped file is created by the mmap constructor,
which is different on Unix and on Windows.  In either case you must
provide a file descriptor for a file opened for update. If you wish to
map an existing Python file object, use its `fileno()' method to obtain
the correct value for the _fileno_ parameter.  Otherwise, you can open
the file using the os.open() function, which returns a file
descriptor directly (the file still needs to be closed when done).

  For both the Unix and Windows versions of the constructor, _access_
may be specified as an optional keyword parameter. _access_ accepts one
of three values: `ACCESS_READ', `ACCESS_WRITE', or `ACCESS_COPY' to
specify read-only, write-through or copy-on-write memory respectively.
_access_ can be used on both Unix and Windows.  If _access_ is not
specified, Windows mmap returns a write-through mapping.  The initial
memory values for all three access types are taken from the specified
file.  Assignment to an `ACCESS_READ' memory map raises a
TypeError exception.  Assignment to an `ACCESS_WRITE' memory map
affects both memory and the underlying file.  Assignment to an
`ACCESS_COPY' memory map affects memory but does not update the
underlying file.

  Changed in version 2.5: To map anonymous memory, -1 should be passed
as the fileno along with the length.

  Changed in version 2.6: mmap.mmap has formerly been a factory
function creating mmap objects. Now mmap.mmap is the class itself.

 -- Class: mmap.mmap (fileno, length[, tagname[, access[, offset]]])
     *(Windows version)* Maps _length_ bytes from the file specified by
     the file handle _fileno_, and creates a mmap object.  If _length_
     is larger than the current size of the file, the file is extended
     to contain _length_ bytes.  If _length_ is `0', the maximum length
     of the map is the current size of the file, except that if the
     file is empty Windows raises an exception (you cannot create an
     empty mapping on Windows).

     _tagname_, if specified and not `None', is a string giving a tag
     name for the mapping.  Windows allows you to have many different
     mappings against the same file.  If you specify the name of an
     existing tag, that tag is opened, otherwise a new tag of this name
     is created.  If this parameter is omitted or `None', the mapping
     is created without a name.  Avoiding the use of the tag parameter
     will assist in keeping your code portable between Unix and Windows.

     _offset_ may be specified as a non-negative integer offset. mmap
     references will be relative to the offset from the beginning of
     the file. _offset_ defaults to 0.  _offset_ must be a multiple of
     the ALLOCATIONGRANULARITY.

 -- Class: mmap.mmap (fileno, length[, flags[, prot[, access[,
          offset]]]])
     *(Unix version)* Maps _length_ bytes from the file specified by
     the file descriptor _fileno_, and returns a mmap object.  If
     _length_ is `0', the maximum length of the map will be the current
     size of the file when mmap is called.

     _flags_ specifies the nature of the mapping. `MAP_PRIVATE' creates
     a private copy-on-write mapping, so changes to the contents of the
     mmap object will be private to this process, and `MAP_SHARED'
     creates a mapping that's shared with all other processes mapping
     the same areas of the file.  The default value is `MAP_SHARED'.

     _prot_, if specified, gives the desired memory protection; the two
     most useful values are `PROT_READ' and `PROT_WRITE', to specify
     that the pages may be read or written.  _prot_ defaults to
     `PROT_READ | PROT_WRITE'.

     _access_ may be specified in lieu of _flags_ and _prot_ as an
     optional keyword parameter.  It is an error to specify both
     _flags_, _prot_ and _access_.  See the description of _access_
     above for information on how to use this parameter.

     _offset_ may be specified as a non-negative integer offset. mmap
     references will be relative to the offset from the beginning of
     the file. _offset_ defaults to 0.  _offset_ must be a multiple of
     the PAGESIZE or ALLOCATIONGRANULARITY.

     This example shows a simple way of using mmap.:

         import mmap

         # write a simple example file
         with open("hello.txt", "wb") as f:
             f.write("Hello Python!\n")

         with open("hello.txt", "r+b") as f:
             # memory-map the file, size 0 means whole file
             map = mmap.mmap(f.fileno(), 0)
             # read content via standard file methods
             print map.readline()  # prints "Hello Python!"
             # read content via slice notation
             print map[:5]  # prints "Hello"
             # update content using slice notation;
             # note that new content must have same size
             map[6:] = " world!\n"
             # ... and read again using standard file methods
             map.seek(0)
             print map.readline()  # prints "Hello  world!"
             # close the map
             map.close()

     The next example demonstrates how to create an anonymous map and
     exchange data between the parent and child processes:

         import mmap
         import os

         map = mmap.mmap(-1, 13)
         map.write("Hello world!")

         pid = os.fork()

         if pid == 0: # In a child process
             map.seek(0)
             print map.readline()

             map.close()

     Memory-mapped file objects support the following methods:

      -- Method: mmap.close ()
          Close the file.  Subsequent calls to other methods of the
          object will result in an exception being raised.

      -- Method: mmap.find (string[, start[, end]])
          Returns the lowest index in the object where the substring
          _string_ is found, such that _string_ is contained in the
          range [_start_, _end_].  Optional arguments _start_ and _end_
          are interpreted as in slice notation.  Returns `-1' on
          failure.

      -- Method: mmap.flush ([offset, size])
          Flushes changes made to the in-memory copy of a file back to
          disk. Without use of this call there is no guarantee that
          changes are written back before the object is destroyed.  If
          _offset_ and _size_ are specified, only changes to the given
          range of bytes will be flushed to disk; otherwise, the whole
          extent of the mapping is flushed.

          *(Windows version)* A nonzero value returned indicates
          success; zero indicates failure.

          *(Unix version)* A zero value is returned to indicate
          success. An exception is raised when the call failed.

      -- Method: mmap.move (dest, src, count)
          Copy the _count_ bytes starting at offset _src_ to the
          destination index _dest_.  If the mmap was created with
          `ACCESS_READ', then calls to move will raise a
          TypeError exception.

      -- Method: mmap.read (num)
          Return a string containing up to _num_ bytes starting from
          the current file position; the file position is updated to
          point after the bytes that were returned.

      -- Method: mmap.read_byte ()
          Returns a string of length 1 containing the character at the
          current file position, and advances the file position by 1.

      -- Method: mmap.readline ()
          Returns a single line, starting at the current file position
          and up to the next newline.

      -- Method: mmap.resize (newsize)
          Resizes the map and the underlying file, if any. If the mmap
          was created with `ACCESS_READ' or `ACCESS_COPY', resizing the
          map will raise a TypeError exception.

      -- Method: mmap.rfind (string[, start[, end]])
          Returns the highest index in the object where the substring
          _string_ is found, such that _string_ is contained in the
          range [_start_, _end_].  Optional arguments _start_ and _end_
          are interpreted as in slice notation.  Returns `-1' on
          failure.

      -- Method: mmap.seek (pos[, whence])
          Set the file's current position.  _whence_ argument is
          optional and defaults to `os.SEEK_SET' or `0' (absolute file
          positioning); other values are `os.SEEK_CUR' or `1' (seek
          relative to the current position) and `os.SEEK_END' or `2'
          (seek relative to the file's end).

      -- Method: mmap.size ()
          Return the length of the file, which can be larger than the
          size of the memory-mapped area.

      -- Method: mmap.tell ()
          Returns the current position of the file pointer.

      -- Method: mmap.write (string)
          Write the bytes in _string_ into memory at the current
          position of the file pointer; the file position is updated to
          point after the bytes that were written. If the mmap was
          created with `ACCESS_READ', then writing to it will raise a
          TypeError exception.

      -- Method: mmap.write_byte (byte)
          Write the single-character string _byte_ into memory at the
          current position of the file pointer; the file position is
          advanced by `1'. If the mmap was created with `ACCESS_READ',
          then writing to it will raise a TypeError.
          exception.

5.16.8 `readline' -- GNU readline interface
-------------------------------------------

The readline module defines a number of functions to
facilitate completion and reading/writing of history files from the
Python interpreter.  This module can be used directly or via the
rlcompleter module.  Settings made using  this module affect the
behaviour of both the interpreter's interactive prompt  and the prompts
offered by the raw_input() and input() built-in
functions.

     Note: On MacOS X the readline module can be
     implemented using the `libedit' library instead of GNU readline.

     The configuration file for `libedit' is different from that of GNU
     readline. If you programmatically load configuration strings you
     can check for the text "libedit" in `readline.__doc__' to
     differentiate between GNU readline and libedit.

  The readline module defines the following functions:

 -- Function: readline.parse_and_bind (string)
     Parse and execute single line of a readline init file.

 -- Function: readline.get_line_buffer ()
     Return the current contents of the line buffer.

 -- Function: readline.insert_text (string)
     Insert text into the command line.

 -- Function: readline.read_init_file ([filename])
     Parse a readline initialization file. The default filename is the
     last filename used.

 -- Function: readline.read_history_file ([filename])
     Load a readline history file. The default filename is `~/.history'.

 -- Function: readline.write_history_file ([filename])
     Save a readline history file. The default filename is `~/.history'.

 -- Function: readline.clear_history ()
     Clear the current history.  (Note: this function is not available
     if the installed version of GNU readline doesn't support it.)

     New in version 2.4.

 -- Function: readline.get_history_length ()
     Return the desired length of the history file.  Negative values
     imply unlimited history file size.

 -- Function: readline.set_history_length (length)
     Set the number of lines to save in the history file. See
     write_history_file().  uses this value to truncate the
     history file when saving.  Negative values imply unlimited history
     file size.

 -- Function: readline.get_current_history_length ()
     Return the number of lines currently in the history.  (This is
     different from get_history_length(), which returns the
     maximum number of lines that will be written to a history file.)

     New in version 2.3.

 -- Function: readline.get_history_item (index)
     Return the current contents of history item at _index_.

     New in version 2.3.

 -- Function: readline.remove_history_item (pos)
     Remove history item specified by its position from the history.

     New in version 2.4.

 -- Function: readline.replace_history_item (pos, line)
     Replace history item specified by its position with the given line.

     New in version 2.4.

 -- Function: readline.redisplay ()
     Change what's displayed on the screen to reflect the current
     contents of the line buffer.

     New in version 2.3.

 -- Function: readline.set_startup_hook ([function])
     Set or remove the startup_hook function.  If _function_ is
     specified, it will be used as the new startup_hook function; if
     omitted or `None', any hook function already installed is removed.
     The startup_hook function is called with no arguments just before
     readline prints the first prompt.

 -- Function: readline.set_pre_input_hook ([function])
     Set or remove the pre_input_hook function.  If _function_ is
     specified, it will be used as the new pre_input_hook function; if
     omitted or `None', any hook function already installed is removed.
     The pre_input_hook function is called with no arguments after the
     first prompt has been printed and just before readline starts
     reading input characters.

 -- Function: readline.set_completer ([function])
     Set or remove the completer function.  If _function_ is specified,
     it will be used as the new completer function; if omitted or
     `None', any completer function already installed is removed.  The
     completer function is called as `function(text, state)', for
     _state_ in `0', `1', `2', ..., until it returns a non-string
     value.  It should return the next possible completion starting
     with _text_.

 -- Function: readline.get_completer ()
     Get the completer function, or `None' if no completer function has
     been set.

     New in version 2.3.

 -- Function: readline.get_completion_type ()
     Get the type of completion being attempted.

     New in version 2.6.

 -- Function: readline.get_begidx ()
     Get the beginning index of the readline tab-completion scope.

 -- Function: readline.get_endidx ()
     Get the ending index of the readline tab-completion scope.

 -- Function: readline.set_completer_delims (string)
     Set the readline word delimiters for tab-completion.

 -- Function: readline.get_completer_delims ()
     Get the readline word delimiters for tab-completion.

 -- Function: readline.set_completion_display_matches_hook ([function])
     Set or remove the completion display function.  If _function_ is
     specified, it will be used as the new completion display function;
     if omitted or `None', any completion display function already
     installed is removed.  The completion display function is called as
     `function(substitution, [matches], longest_match_length)' once
     each time matches need to be displayed.

     New in version 2.6.

 -- Function: readline.add_history (line)
     Append a line to the history buffer, as if it was the last line
     typed.

Module rlcompleter.
     Completion of Python identifiers at the interactive prompt.

5.16.8.1 Example
................

The following example demonstrates how to use the readline.
module's history reading and writing functions to automatically load
and save a history file named `.pyhist' from the user's home directory.
The code below would normally be executed automatically during
interactive sessions from the user's PYTHONSTARTUP file.

    import os
    import readline
    histfile = os.path.join(os.environ["HOME"], ".pyhist")
    try:
        readline.read_history_file(histfile)
    except IOError:
        pass
    import atexit
    atexit.register(readline.write_history_file, histfile)
    del os, histfile

The following example extends the code.InteractiveConsole.
class to support history save/restore.

    import code
    import readline
    import atexit
    import os

    class HistoryConsole(code.InteractiveConsole):
        def __init__(self, locals=None, filename="<console>",
                     histfile=os.path.expanduser("~/.console-history")):
            code.InteractiveConsole.__init__(self, locals, filename)
            self.init_history(histfile)

        def init_history(self, histfile):
            readline.parse_and_bind("tab: complete")
            if hasattr(readline, "read_history_file"):
                try:
                    readline.read_history_file(histfile)
                except IOError:
                    pass
                atexit.register(self.save_history, histfile)

        def save_history(self, histfile):
            readline.write_history_file(histfile)

5.16.9 `rlcompleter' -- Completion function for GNU readline
------------------------------------------------------------

The rlcompleter module defines a completion function
suitable for the readline module by completing valid Python
identifiers and keywords.

  When this module is imported on a Unix platform with the
readline module available, an instance of the `Completer' class
is automatically created and its `complete()' method is set as the
readline completer.

  Example:

    >>> import rlcompleter
    >>> import readline
    >>> readline.parse_and_bind("tab: complete")
    >>> readline. <TAB PRESSED>
    readline.__doc__          readline.get_line_buffer(  readline.read_init_file(
    readline.__file__         readline.insert_text(      readline.set_completer(
    readline.__name__         readline.parse_and_bind(
    >>> readline.

The rlcompleter module is designed for use with Python's
interactive mode.  A user can add the following lines to his or her
initialization file (identified by the PYTHONSTARTUP.
environment variable) to get automatic `Tab' completion:

    try:
        import readline
    except ImportError:
        print "Module readline not available."
    else:
        import rlcompleter
        readline.parse_and_bind("tab: complete")

On platforms without readline, the `Completer' class defined
by this module can still be used for custom purposes.

5.16.9.1 Completer Objects
..........................

Completer objects have the following method:

 -- Method: Completer.complete (text, state)
     Return the _state_th completion for _text_.

     If called for _text_ that doesn't include a period character
     (`'.''), it will complete from names currently defined in
     __main__, __builtin__ and keywords (as defined by the
     keyword module).

     If called for a dotted name, it will try to evaluate anything
     without obvious side-effects (functions will not be evaluated, but
     it can generate calls to __getattr__()) up to the last
     part, and find matches for the rest via the dir().
     function.  Any exception raised during the evaluation of the
     expression is caught, silenced and None is returned.

5.17 Interprocess Communication and Networking
==============================================

The modules described in this chapter provide mechanisms for different
processes to communicate.

  Some modules only work for two processes that are on the same
machine, e.g.  see signal and subprocess.  Other
modules support networking protocols that two or more processes can
used to communicate across machines.

  The list of modules described in this chapter is:

5.17.1 `subprocess' -- Subprocess management
--------------------------------------------

New in version 2.4.

  The subprocess module allows you to spawn new processes,
connect to their input/output/error pipes, and obtain their return
codes.  This module intends to replace several other, older modules and
functions, such as:

    os.system
    os.spawn*
    os.popen*
    popen2.*
    commands.*

Information about how the subprocess module can be used to
replace these modules and functions can be found in the following
sections.

  PEP 324 (http://www.python.org/dev/peps/pep-0324) - PEP proposing the
subprocess module

5.17.1.1 Using the subprocess Module
....................................

This module defines one class called Popen.:

 -- Class: subprocess.Popen (args, bufsize=0, executable=None,
          stdin=None, stdout=None, stderr=None, preexec_fn=None,
          close_fds=False, shell=False, cwd=None, env=None,
          universal_newlines=False, startupinfo=None, creationflags=0)
     Arguments are:

     _args_ should be a string, or a sequence of program arguments.
     The program to execute is normally the first item in the args
     sequence or the string if a string is given, but can be explicitly
     set by using the _executable_ argument.  When _executable_ is
     given, the first item in the args sequence is still treated by
     most programs as the command name, which can then be different
     from the actual executable name.  On Unix, it becomes the display
     name for the executing program in utilities such as *ps*.

     On Unix, with _shell=False_ (default): In this case, the Popen
     class uses os.execvp() to execute the child program.
     _args_ should normally be a sequence.  If a string is specified
     for _args_, it will be used as the name or path of the program to
     execute; this will only work if the program is being given no
     arguments.

          Note: shlex.split() can be useful when
          determining the correct tokenization for _args_, especially
          in complex cases:

              >>> import shlex, subprocess
              >>> command_line = raw_input()
              /bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
              >>> args = shlex.split(command_line)
              >>> print args
              ['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
              >>> p = subprocess.Popen(args) # Success!

          Note in particular that options (such as _-input_) and
          arguments (such as _eggs.txt_) that are separated by
          whitespace in the shell go in separate list elements, while
          arguments that need quoting or backslash escaping when used
          in the shell (such as filenames containing spaces or the
          _echo_ command shown above) are single list elements.

     On Unix, with _shell=True_: If args is a string, it specifies the
     command string to execute through the shell.  This means that the
     string must be formatted exactly as it would be when typed at the
     shell prompt.  This includes, for example, quoting or backslash
     escaping filenames with spaces in them.  If _args_ is a sequence,
     the first item specifies the command string, and any additional
     items will be treated as additional arguments to the shell itself.
     That is to say, _Popen_ does the equivalent of:

         Popen(['/bin/sh', '-c', args[0], args[1], ...])

          Warning: Executing shell commands that incorporate
          unsanitized input from an untrusted source makes a program
          vulnerable to shell injection
          (http://en.wikipedia.org/wiki/Shell_injection#Shell_injection),
          a serious security flaw which can result in arbitrary command
          execution.  For this reason, the use of _shell=True_ is
          *strongly discouraged* in cases where the command string is
          constructed from external input:

              >>> from subprocess import call
              >>> filename = input("What file would you like to display?\n")
              What file would you like to display?
              non_existent; rm -rf / #
              >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...

          _shell=False_ does not suffer from this vulnerability; the
          above Note may be helpful in getting code using _shell=False_
          to work.

     On Windows: the Popen class uses CreateProcess() to
     execute the child program, which operates on strings.  If _args_
     is a sequence, it will be converted to a string using the
     `list2cmdline()' method.  Please note that not all MS Windows
     applications interpret the command line the same way:
     `list2cmdline()' is designed for applications using the same rules
     as the MS C runtime.

     _bufsize_, if given, has the same meaning as the corresponding
     argument to the built-in open() function: `0' means unbuffered,
     `1' means line buffered, any other positive value means use a
     buffer of (approximately) that size.  A negative _bufsize_ means
     to use the system default, which usually means fully buffered.
     The default value for _bufsize_ is `0' (unbuffered).

          Note: If you experience performance issues, it is recommended
          that you try to enable buffering by setting _bufsize_ to
          either -1 or a large enough positive value (such as 4096).

     The _executable_ argument specifies the program to execute. It is
     very seldom needed: Usually, the program to execute is defined by
     the _args_ argument. If `shell=True', the _executable_ argument
     specifies which shell to use. On Unix, the default shell is
     `/bin/sh'.  On Windows, the default shell is specified by the `COMSPEC'
     environment variable. The only reason you would need to specify
     `shell=True' on Windows is where the command you wish to execute
     is actually built in to the shell, eg `dir', `copy'.  You don't
     need `shell=True' to run a batch file, nor to run a console-based
     executable.

     _stdin_, _stdout_ and _stderr_ specify the executed programs'
     standard input, standard output and standard error file handles,
     respectively.  Valid values are PIPE, an existing file
     descriptor (a positive integer), an existing file object, and
     `None'.  PIPE indicates that a new pipe to the child
     should be created.  With `None', no redirection will occur; the
     child's file handles will be inherited from the parent.
     Additionally, _stderr_ can be STDOUT, which indicates
     that the stderr data from the applications should be captured into
     the same file handle as for stdout.

     If _preexec_fn_ is set to a callable object, this object will be
     called in the child process just before the child is executed.
     (Unix only)

     If _close_fds_ is true, all file descriptors except `0', `1' and
     `2' will be closed before the child process is executed. (Unix
     only).  Or, on Windows, if _close_fds_ is true then no handles
     will be inherited by the child process.  Note that on Windows, you
     cannot set _close_fds_ to true and also redirect the standard
     handles by setting _stdin_, _stdout_ or _stderr_.

     If _shell_ is True, the specified command will be
     executed through the shell.

     If _cwd_ is not `None', the child's current directory will be
     changed to _cwd_ before it is executed.  Note that this directory
     is not considered when searching the executable, so you can't
     specify the program's path relative to _cwd_.

     If _env_ is not `None', it must be a mapping that defines the
     environment variables for the new process; these are used instead
     of inheriting the current process' environment, which is the
     default behavior.

          Note: If specified, _env_ must provide any variables required
          for the program to execute.  On Windows, in order to run a
          side-by-side assembly
          (http://en.wikipedia.org/wiki/Side-by-Side_Assembly) the
          specified _env_ *must* include a valid `SystemRoot'.

     If _universal_newlines_ is True, the file objects
     stdout and stderr are opened as text files, but lines may be
     terminated by any of `'\n'', the Unix end-of-line convention,
     `'\r'', the old Macintosh convention or `'\r\n'', the Windows
     convention. All of these external representations are seen as
     `'\n'' by the Python program.

          Note: This feature is only available if Python is built with
          universal newline support (the default).  Also, the newlines
          attribute of the file objects stdout,
          stdin and stderr are not updated by the
          communicate() method.

     The _startupinfo_ and _creationflags_, if given, will be passed to
     the underlying CreateProcess() function.  They can specify things
     such as appearance of the main window and priority for the new
     process.  (Windows only)

 -- Data: subprocess.PIPE
     Special value that can be used as the _stdin_, _stdout_ or
     _stderr_ argument to Popen and indicates that a pipe
     to the standard stream should be opened.

 -- Data: subprocess.STDOUT
     Special value that can be used as the _stderr_ argument to
     Popen and indicates that standard error should go into the
     same handle as standard output.

5.17.1.2 Convenience Functions
..............................

This module also defines two shortcut functions:

 -- Function: subprocess.call (*popenargs, **kwargs)
     Run command with arguments.  Wait for command to complete, then
     return the `returncode' attribute.

     The arguments are the same as for the Popen.
     constructor.  Example:

         >>> retcode = subprocess.call(["ls", "-l"])

          Warning: Like Popen.wait(), this will deadlock
          when using `stdout=PIPE' and/or `stderr=PIPE' and the child
          process generates enough output to a pipe such that it blocks
          waiting for the OS pipe buffer to accept more data.

 -- Function: subprocess.check_call (*popenargs, **kwargs)
     Run command with arguments.  Wait for command to complete. If the
     exit code was zero then return, otherwise raise
     `CalledProcessError'. The `CalledProcessError' object will have
     the return code in the `returncode' attribute.

     The arguments are the same as for the Popen.
     constructor.  Example:

         >>> subprocess.check_call(["ls", "-l"])
         0

     New in version 2.5.

          Warning: See the warning for call().

 -- Function: subprocess.check_output (*popenargs, **kwargs)
     Run command with arguments and return its output as a byte string.

     If the exit code was non-zero it raises a `CalledProcessError'.
     The `CalledProcessError' object will have the return code in the
     `returncode' attribute and output in the `output' attribute.

     The arguments are the same as for the Popen.
     constructor.  Example:

         >>> subprocess.check_output(["ls", "-l", "/dev/null"])
         'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

     The stdout argument is not allowed as it is used internally.  To
     capture standard error in the result, use
     `stderr=subprocess.STDOUT':

         >>> subprocess.check_output(
         ...     ["/bin/sh", "-c", "ls non_existent_file; exit 0"],
         ...     stderr=subprocess.STDOUT)
         'ls: non_existent_file: No such file or directory\n'

     New in version 2.7.

5.17.1.3 Exceptions
...................

Exceptions raised in the child process, before the new program has
started to execute, will be re-raised in the parent.  Additionally, the
exception object will have one extra attribute called
`child_traceback', which is a string containing traceback information
from the child's point of view.

  The most common exception raised is OSError.  This occurs,
for example, when trying to execute a non-existent file.  Applications
should prepare for OSError exceptions.

  A ValueError will be raised if Popen is
called with invalid arguments.

  check_call() will raise `CalledProcessError', if the called process
returns a non-zero return code.

5.17.1.4 Security
.................

Unlike some other popen functions, this implementation will never call
/bin/sh implicitly.  This means that all characters, including shell
metacharacters, can safely be passed to child processes.

5.17.1.5 Popen Objects
......................

Instances of the Popen class have the following methods:

 -- Method: Popen.poll ()
     Check if child process has terminated.  Set and return
     returncode.  attribute.

 -- Method: Popen.wait ()
     Wait for child process to terminate.  Set and return
     returncode.  attribute.

          Warning: This will deadlock when using `stdout=PIPE' and/or
          `stderr=PIPE' and the child process generates enough output to
          a pipe such that it blocks waiting for the OS pipe buffer to
          accept more data.  Use communicate() to avoid
          that.

 -- Method: Popen.communicate (input=None)
     Interact with process: Send data to stdin.  Read data from stdout
     and stderr, until end-of-file is reached.  Wait for process to
     terminate. The optional _input_ argument should be a string to be
     sent to the child process, or `None', if no data should be sent to
     the child.

     communicate() returns a tuple `(stdoutdata,
     stderrdata)'.

     Note that if you want to send data to the process's stdin, you
     need to create the Popen object with `stdin=PIPE'.  Similarly, to
     get anything other than `None' in the result tuple, you need to
     give `stdout=PIPE' and/or `stderr=PIPE' too.

          Note: The data read is buffered in memory, so do not use this
          method if the data size is large or unlimited.

 -- Method: Popen.send_signal (signal)
     Sends the signal _signal_ to the child.

          Note: On Windows, SIGTERM is an alias for terminate().
          CTRL_C_EVENT and CTRL_BREAK_EVENT can be sent to
          processes started with a _creationflags_ parameter which
          includes `CREATE_NEW_PROCESS_GROUP'.

     New in version 2.6.

 -- Method: Popen.terminate ()
     Stop the child. On Posix OSs the method sends SIGTERM to the
     child. On Windows the Win32 API function `TerminateProcess()' is
     called to stop the child.

     New in version 2.6.

 -- Method: Popen.kill ()
     Kills the child. On Posix OSs the function sends SIGKILL to the
     child.  On Windows kill() is an alias for
     terminate().

     New in version 2.6.

  The following attributes are also available:

     Warning: Use `communicate()' rather than `.stdin.write',
     `.stdout.read' or `.stderr.read' to avoid deadlocks due to any of
     the other OS pipe buffers filling up and blocking the child
     process.

 -- Attribute: Popen.stdin
     If the _stdin_ argument was PIPE, this attribute is a
     file object that provides input to the child process.  Otherwise,
     it is `None'.

 -- Attribute: Popen.stdout
     If the _stdout_ argument was PIPE, this attribute is a
     file object that provides output from the child process.
     Otherwise, it is `None'.

 -- Attribute: Popen.stderr
     If the _stderr_ argument was PIPE, this attribute is a
     file object that provides error output from the child process.
     Otherwise, it is `None'.

 -- Attribute: Popen.pid
     The process ID of the child process.

     Note that if you set the _shell_ argument to `True', this is the
     process ID of the spawned shell.

 -- Attribute: Popen.returncode
     The child return code, set by poll() and
     wait(). (and indirectly by communicate()).  A
     `None' value indicates that the process hasn't terminated yet.

     A negative value `-N' indicates that the child was terminated by
     signal `N' (Unix only).

5.17.1.6 Replacing Older Functions with the subprocess Module
.............................................................

In this section, "a ==> b" means that b can be used as a replacement
for a.

     Note: All functions in this section fail (more or less) silently
     if the executed program cannot be found; this module raises an
     OSError exception.

  In the following examples, we assume that the subprocess module is
imported with "from subprocess import *".

5.17.1.7 Replacing /bin/sh shell backquote
..........................................

    output=`mycmd myarg`
    ==>
    output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

5.17.1.8 Replacing shell pipeline
.................................

    output=`dmesg | grep hda`
    ==>
    p1 = Popen(["dmesg"], stdout=PIPE)
    p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
    output = p2.communicate()[0]

5.17.1.9 Replacing `os.system()'
................................

    sts = os.system("mycmd" + " myarg")
    ==>
    p = Popen("mycmd" + " myarg", shell=True)
    sts = os.waitpid(p.pid, 0)[1]

Notes:

   * Calling the program through the shell is usually not required.

   * It's easier to look at the `returncode' attribute than the exit
     status.

  A more realistic example would look like this:

    try:
        retcode = call("mycmd" + " myarg", shell=True)
        if retcode < 0:
            print >>sys.stderr, "Child was terminated by signal", -retcode
        else:
            print >>sys.stderr, "Child returned", retcode
    except OSError, e:
        print >>sys.stderr, "Execution failed:", e

5.17.1.10 Replacing the `os.spawn' family
.........................................

P_NOWAIT example:

    pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
    ==>
    pid = Popen(["/bin/mycmd", "myarg"]).pid

P_WAIT example:

    retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
    ==>
    retcode = call(["/bin/mycmd", "myarg"])

Vector example:

    os.spawnvp(os.P_NOWAIT, path, args)
    ==>
    Popen([path] + args[1:])

Environment example:

    os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
    ==>
    Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})

5.17.1.11 Replacing `os.popen()', `os.popen2()', `os.popen3()'
..............................................................

    pipe = os.popen("cmd", 'r', bufsize)
    ==>
    pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE).stdout

    pipe = os.popen("cmd", 'w', bufsize)
    ==>
    pipe = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE).stdin

    (child_stdin, child_stdout) = os.popen2("cmd", mode, bufsize)
    ==>
    p = Popen("cmd", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdin, child_stdout) = (p.stdin, p.stdout)

    (child_stdin,
     child_stdout,
     child_stderr) = os.popen3("cmd", mode, bufsize)
    ==>
    p = Popen("cmd", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
    (child_stdin,
     child_stdout,
     child_stderr) = (p.stdin, p.stdout, p.stderr)

    (child_stdin, child_stdout_and_stderr) = os.popen4("cmd", mode,
                                                       bufsize)
    ==>
    p = Popen("cmd", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
    (child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)

On Unix, os.popen2, os.popen3 and os.popen4 also accept a sequence as
the command to execute, in which case arguments will be passed directly
to the program without shell intervention.  This usage can be replaced
as follows:

    (child_stdin, child_stdout) = os.popen2(["/bin/ls", "-l"], mode,
                                            bufsize)
    ==>
    p = Popen(["/bin/ls", "-l"], bufsize=bufsize, stdin=PIPE, stdout=PIPE)
    (child_stdin, child_stdout) = (p.stdin, p.stdout)

Return code handling translates as follows:

    pipe = os.popen("cmd", 'w')
    ...
    rc = pipe.close()
    if rc is not None and rc >> 8:
        print "There were some errors"
    ==>
    process = Popen("cmd", 'w', shell=True, stdin=PIPE)
    ...
    process.stdin.close()
    if process.wait() != 0:
        print "There were some errors"

5.17.1.12 Replacing functions from the `popen2' module
......................................................

    (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
    ==>
    p = Popen(["somestring"], shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdout, child_stdin) = (p.stdout, p.stdin)

On Unix, popen2 also accepts a sequence as the command to execute, in
which case arguments will be passed directly to the program without
shell intervention.  This usage can be replaced as follows:

    (child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize,
                                                mode)
    ==>
    p = Popen(["mycmd", "myarg"], bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdout, child_stdin) = (p.stdout, p.stdin)

popen2.Popen3 and popen2.Popen4 basically
work as subprocess.Popen, except that:

   * Popen raises an exception if the execution fails.

   * the _capturestderr_ argument is replaced with the _stderr_
     argument.

   * `stdin=PIPE' and `stdout=PIPE' must be specified.

   * popen2 closes all file descriptors by default, but you have to
     specify `close_fds=True' with Popen.

5.17.2 `socket' -- Low-level networking interface
-------------------------------------------------

This module provides access to the BSD _socket_ interface. It is
available on all modern Unix systems, Windows, Mac OS X, BeOS, OS/2,
and probably additional platforms.

     Note: Some behavior may be platform dependent, since calls are
     made to the operating system socket APIs.

  For an introduction to socket programming (in C), see the following
papers: An Introductory 4.3BSD Interprocess Communication Tutorial, by
Stuart Sechrest and An Advanced 4.3BSD Interprocess Communication
Tutorial, by Samuel J.  Leffler et al, both in the UNIX Programmer's
Manual, Supplementary Documents 1 (sections PS1:7 and PS1:8).  The
platform-specific reference material for the various socket-related
system calls are also a valuable source of information on the details
of socket semantics.  For Unix, refer to the manual pages; for Windows,
see the WinSock (or Winsock 2) specification. For IPv6-ready APIs,
readers may want to refer to RFC 3493
(http://tools.ietf.org/html/rfc3493.html) titled Basic Socket Interface
Extensions for IPv6.

  The Python interface is a straightforward transliteration of the Unix
system call and library interface for sockets to Python's
object-oriented style: the socket() function returns a
_socket object_ whose methods implement the various socket system
calls.  Parameter types are somewhat higher-level than in the C
interface: as with `read()' and `write()' operations on Python files,
buffer allocation on receive operations is automatic, and buffer length
is implicit on send operations.

  Socket addresses are represented as follows: A single string is used
for the AF_UNIX address family. A pair `(host, port)' is
used for the AF_INET address family, where _host_ is a
string representing either a hostname in Internet domain notation like
`'daring.cwi.nl'' or an IPv4 address like `'100.50.200.5'', and _port_
is an integral port number. For AF_INET6 address family, a
four-tuple `(host, port, flowinfo, scopeid)' is used, where _flowinfo_
and _scopeid_ represents `sin6_flowinfo' and `sin6_scope_id' member in
`struct sockaddr_in6' in C. For socket module methods,
_flowinfo_ and _scopeid_ can be omitted just for backward
compatibility. Note, however, omission of _scopeid_ can cause problems
in manipulating scoped IPv6 addresses. Other address families are
currently not supported. The address format required by a particular
socket object is automatically selected based on the address family
specified when the socket object was created.

  For IPv4 addresses, two special forms are accepted instead of a host
address: the empty string represents `INADDR_ANY', and the string
`'<broadcast>'' represents `INADDR_BROADCAST'. The behavior is not
available for IPv6 for backward compatibility, therefore, you may want
to avoid these if you intend to support IPv6 with your Python programs.

  If you use a hostname in the _host_ portion of IPv4/v6 socket
address, the program may show a nondeterministic behavior, as Python
uses the first address returned from the DNS resolution.  The socket
address will be resolved differently into an actual IPv4/v6 address,
depending on the results from DNS resolution and/or the host
configuration.  For deterministic behavior use a numeric address in
_host_ portion.

  New in version 2.5: AF_NETLINK sockets are represented as  pairs
`pid, groups'.

  New in version 2.6: Linux-only support for TIPC is also available
using the `AF_TIPC' address family. TIPC is an open, non-IP based
networked protocol designed for use in clustered computer environments.
Addresses are represented by a tuple, and the fields depend on the
address type. The general tuple form is `(addr_type, v1, v2, v3 [,
scope])', where:

   - _addr_type_ is one of TIPC_ADDR_NAMESEQ, TIPC_ADDR_NAME, or
     TIPC_ADDR_ID.

   - _scope_ is one of TIPC_ZONE_SCOPE, TIPC_CLUSTER_SCOPE, and
     TIPC_NODE_SCOPE.

   - If _addr_type_ is TIPC_ADDR_NAME, then _v1_ is the server type,
     _v2_ is the port identifier, and _v3_ should be 0.

     If _addr_type_ is TIPC_ADDR_NAMESEQ, then _v1_ is the server type,
     _v2_ is the lower port number, and _v3_ is the upper port number.

     If _addr_type_ is TIPC_ADDR_ID, then _v1_ is the node, _v2_ is the
     reference, and _v3_ should be set to 0.

  All errors raise exceptions.  The normal exceptions for invalid
argument types and out-of-memory conditions can be raised; errors
related to socket or address semantics raise the error
socket.error.

  Non-blocking mode is supported through setblocking().  A
generalization of this based on timeouts is supported through
settimeout().

  The module socket exports the following constants and
functions:

 -- Exception: socket.error
     This exception is raised for socket-related errors. The
     accompanying value is either a string telling what went wrong or a
     pair `(errno, string)' representing an error returned by a system
     call, similar to the value accompanying os.error. See
     the module errno, which contains names for the error
     codes defined by the underlying operating system.

     Changed in version 2.6: socket.error is now a child
     class of IOError.

 -- Exception: socket.herror
     This exception is raised for address-related errors, i.e. for
     functions that use _h_errno_ in the C API, including
     gethostbyname_ex() and gethostbyaddr().

     The accompanying value is a pair `(h_errno, string)' representing
     an error returned by a library call. _string_ represents the
     description of _h_errno_, as returned by the `hstrerror()' C
     function.

 -- Exception: socket.gaierror
     This exception is raised for address-related errors, for
     getaddrinfo() and getnameinfo(). The
     accompanying value is a pair `(error, string)' representing an
     error returned by a library call. _string_ represents the
     description of _error_, as returned by the `gai_strerror()' C
     function. The _error_ value will match one of the `EAI_*'
     constants defined in this module.

 -- Exception: socket.timeout
     This exception is raised when a timeout occurs on a socket which
     has had timeouts enabled via a prior call to `settimeout()'.  The
     accompanying value is a string whose value is currently always
     "timed out".

     New in version 2.3.

 -- Data: socket.AF_UNIX
 -- Data: socket.AF_INET
 -- Data: socket.AF_INET6
     These constants represent the address (and protocol) families,
     used for the first argument to socket().  If the
     AF_UNIX constant is not defined then this protocol is
     unsupported.

 -- Data: socket.SOCK_STREAM
 -- Data: socket.SOCK_DGRAM
 -- Data: socket.SOCK_RAW
 -- Data: socket.SOCK_RDM
 -- Data: socket.SOCK_SEQPACKET
     These constants represent the socket types, used for the second
     argument to socket(). (Only SOCK_STREAM and
     SOCK_DGRAM appear to be generally useful.)

 -- Data: SO_*
 -- Data: socket.SOMAXCONN
 -- Data: MSG_*
 -- Data: SOL_*
 -- Data: IPPROTO_*
 -- Data: IPPORT_*
 -- Data: INADDR_*
 -- Data: IP_*
 -- Data: IPV6_*
 -- Data: EAI_*
 -- Data: AI_*
 -- Data: NI_*
 -- Data: TCP_*
     Many constants of these forms, documented in the Unix
     documentation on sockets and/or the IP protocol, are also defined
     in the socket module. They are generally used in arguments to the
     `setsockopt()' and `getsockopt()' methods of socket objects.  In
     most cases, only those symbols that are defined in the Unix header
     files are defined; for a few symbols, default values are provided.

 -- Data: SIO_*
 -- Data: RCVALL_*
     Constants for Windows' WSAIoctl(). The constants are used as
     arguments to the `ioctl()' method of socket objects.

     New in version 2.6.

 -- Data: TIPC_*
     TIPC related constants, matching the ones exported by the C socket
     API. See the TIPC documentation for more information.

     New in version 2.6.

 -- Data: socket.has_ipv6
     This constant contains a boolean value which indicates if IPv6 is
     supported on this platform.

     New in version 2.3.

 -- Function: socket.create_connection (address[, timeout[,
          source_address]])
     Convenience function.  Connect to _address_ (a 2-tuple `(host,
     port)'), and return the socket object.  Passing the optional
     _timeout_ parameter will set the timeout on the socket instance
     before attempting to connect.  If no _timeout_ is supplied, the
     global default timeout setting returned by
     getdefaulttimeout() is used.

     If supplied, _source_address_ must be a 2-tuple `(host, port)' for
     the socket to bind to as its source address before connecting.  If
     host or port are " or 0 respectively the OS default behavior will
     be used.

     New in version 2.6.

     Changed in version 2.7: _source_address_ was added.

 -- Function: socket.getaddrinfo (host, port, family=0, socktype=0,
          proto=0, flags=0)
     Translate the _host_/_port_ argument into a sequence of 5-tuples
     that contain all the necessary arguments for creating a socket
     connected to that service.  _host_ is a domain name, a string
     representation of an IPv4/v6 address or `None'. _port_ is a string
     service name such as `'http'', a numeric port number or `None'.
     By passing `None' as the value of _host_ and _port_, you can pass
     `NULL' to the underlying C API.

     The _family_, _socktype_ and _proto_ arguments can be optionally
     specified in order to narrow the list of addresses returned.
     Passing zero as a value for each of these arguments selects the
     full range of results.  The _flags_ argument can be one or several
     of the `AI_*' constants, and will influence how results are
     computed and returned.  For example, `AI_NUMERICHOST' will disable
     domain name resolution and will raise an error if _host_ is a
     domain name.

     The function returns a list of 5-tuples with the following
     structure:

     `(family, socktype, proto, canonname, sockaddr)'

     In these tuples, _family_, _socktype_, _proto_ are all integers
     and are meant to be passed to the socket() function.
     _canonname_ will be a string representing the canonical name of
     the _host_ if `AI_CANONNAME' is part of the _flags_ argument; else
     _canonname_ will be empty.  _sockaddr_ is a tuple describing a
     socket address, whose format depends on the returned _family_ (a
     `(address, port)' 2-tuple for AF_INET, a `(address,
     port, flow info, scope id)' 4-tuple for AF_INET6),
     and is meant to be passed to the socket.connect().
     method.

     The following example fetches address information for a
     hypothetical TCP connection to `www.python.org' on port 80
     (results may differ on your system if IPv6 isn't enabled):

         >>> socket.getaddrinfo("www.python.org", 80, 0, 0, socket.SOL_TCP)
         [(2, 1, 6, '', ('82.94.164.162', 80)),
          (10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))]

     New in version 2.2.

 -- Function: socket.getfqdn ([name])
     Return a fully qualified domain name for _name_. If _name_ is
     omitted or empty, it is interpreted as the local host.  To find
     the fully qualified name, the hostname returned by
     gethostbyaddr() is checked, followed by aliases for the
     host, if available.  The first name which includes a period is
     selected.  In case no fully qualified domain name is available,
     the hostname as returned by gethostname() is returned.

     New in version 2.0.

 -- Function: socket.gethostbyname (hostname)
     Translate a host name to IPv4 address format.  The IPv4 address is
     returned as a string, such as  `'100.50.200.5''.  If the host name
     is an IPv4 address itself it is returned unchanged.  See
     gethostbyname_ex() for a more complete interface.
     gethostbyname() does not support IPv6 name resolution, and
     getaddrinfo() should be used instead for IPv4/v6 dual
     stack support.

 -- Function: socket.gethostbyname_ex (hostname)
     Translate a host name to IPv4 address format, extended interface.
     Return a triple `(hostname, aliaslist, ipaddrlist)' where
     _hostname_ is the primary host name responding to the given
     _ip_address_, _aliaslist_ is a (possibly empty) list of
     alternative host names for the same address, and _ipaddrlist_ is a
     list of IPv4 addresses for the same interface on the same host
     (often but not always a single address). gethostbyname_ex()
     does not support IPv6 name resolution, and
     getaddrinfo() should be used instead for IPv4/v6 dual stack
     support.

 -- Function: socket.gethostname ()
     Return a string containing the hostname of the machine where  the
     Python interpreter is currently executing.

     If you want to know the current machine's IP address, you may want
     to use `gethostbyname(gethostname())'. This operation assumes that
     there is a valid address-to-host mapping for the host, and the
     assumption does not always hold.

     Note: gethostname() doesn't always return the fully
     qualified domain name; use `getfqdn()' (see above).

 -- Function: socket.gethostbyaddr (ip_address)
     Return a triple `(hostname, aliaslist, ipaddrlist)' where
     _hostname_ is the primary host name responding to the given
     _ip_address_, _aliaslist_ is a (possibly empty) list of
     alternative host names for the same address, and _ipaddrlist_ is a
     list of IPv4/v6 addresses for the same interface on the same host
     (most likely containing only a single address). To find the fully
     qualified domain name, use the function getfqdn().
     gethostbyaddr() supports both IPv4 and IPv6.

 -- Function: socket.getnameinfo (sockaddr, flags)
     Translate a socket address _sockaddr_ into a 2-tuple `(host,
     port)'. Depending on the settings of _flags_, the result can
     contain a fully-qualified domain name or numeric address
     representation in _host_.  Similarly, _port_ can contain a string
     port name or a numeric port number.

     New in version 2.2.

 -- Function: socket.getprotobyname (protocolname)
     Translate an Internet protocol name (for example, `'icmp'') to a
     constant suitable for passing as the (optional) third argument to
     the socket().  function.  This is usually only needed
     for sockets opened in "raw" mode (see SOCK_RAW); for the
     normal socket modes, the correct protocol is chosen automatically
     if the protocol is omitted or zero.

 -- Function: socket.getservbyname (servicename[, protocolname])
     Translate an Internet service name and protocol name to a port
     number for that service.  The optional protocol name, if given,
     should be `'tcp'' or `'udp'', otherwise any protocol will match.

 -- Function: socket.getservbyport (port[, protocolname])
     Translate an Internet port number and protocol name to a service
     name for that service.  The optional protocol name, if given,
     should be `'tcp'' or `'udp'', otherwise any protocol will match.

 -- Function: socket.socket ([family[, type[, proto]]])
     Create a new socket using the given address family, socket type
     and protocol number.  The address family should be AF_INET.
     (the default), see AF_INET6 or AF_UNIX.
     The socket type should be SOCK_STREAM. (the default),
     see SOCK_DGRAM or perhaps one of the other `SOCK_'
     constants.  The protocol number is usually zero and may be omitted
     in that case.

 -- Function: socket.socketpair ([family[, type[, proto]]])
     Build a pair of connected socket objects using the given address
     family, socket type, and protocol number.  Address family, socket
     type, and protocol number are as for the socket().
     function above. The default family is AF_UNIX.  if
     defined on the platform; otherwise, the default is AF_INET.
     Availability: Unix.

     New in version 2.4.

 -- Function: socket.fromfd (fd, family, type[, proto])
     Duplicate the file descriptor _fd_ (an integer as returned by a
     file object's `fileno()' method) and build a socket object from
     the result.  Address family, socket type and protocol number are
     as for the socket() function above. The file
     descriptor should refer to a socket, but this is not checked --
     subsequent operations on the object may fail if the file
     descriptor is invalid.  This function is rarely needed, but can be
     used to get or set socket options on a socket passed to a program
     as standard input or output (such as a server started by the Unix
     inet daemon).  The socket is assumed to be in blocking mode.
     Availability: Unix.

 -- Function: socket.ntohl (x)
     Convert 32-bit positive integers from network to host byte order.
     On machines where the host byte order is the same as network byte
     order, this is a no-op; otherwise, it performs a 4-byte swap
     operation.

 -- Function: socket.ntohs (x)
     Convert 16-bit positive integers from network to host byte order.
     On machines where the host byte order is the same as network byte
     order, this is a no-op; otherwise, it performs a 2-byte swap
     operation.

 -- Function: socket.htonl (x)
     Convert 32-bit positive integers from host to network byte order.
     On machines where the host byte order is the same as network byte
     order, this is a no-op; otherwise, it performs a 4-byte swap
     operation.

 -- Function: socket.htons (x)
     Convert 16-bit positive integers from host to network byte order.
     On machines where the host byte order is the same as network byte
     order, this is a no-op; otherwise, it performs a 2-byte swap
     operation.

 -- Function: socket.inet_aton (ip_string)
     Convert an IPv4 address from dotted-quad string format (for
     example, '123.45.67.89') to 32-bit packed binary format, as a
     string four characters in length.  This is useful when conversing
     with a program that uses the standard C library and needs objects
     of type `struct in_addr', which is the C type for the 32-bit
     packed binary this function returns.

     inet_aton() also accepts strings with less than three
     dots; see the Unix manual page `inet(3)' for details.

     If the IPv4 address string passed to this function is invalid,
     see socket.error will be raised. Note that exactly what is
     valid depends on the underlying C implementation of `inet_aton()'.

     inet_aton() does not support IPv6, and
     inet_pton() should be used instead for IPv4/v6 dual stack
     support.

 -- Function: socket.inet_ntoa (packed_ip)
     Convert a 32-bit packed IPv4 address (a string four characters in
     length) to its standard dotted-quad string representation (for
     example, '123.45.67.89').  This is useful when conversing with a
     program that uses the standard C library and needs objects of type
     `struct in_addr', which is the C type for the 32-bit packed binary
     data this function takes as an argument.

     If the string passed to this function is not exactly 4 bytes in
     length, see socket.error will be raised.
     inet_ntoa() does not support IPv6, and inet_ntop()
     should be used instead for IPv4/v6 dual stack support.

 -- Function: socket.inet_pton (address_family, ip_string)
     Convert an IP address from its family-specific string format to a
     packed, binary format. inet_pton() is useful when a
     library or network protocol calls for an object of type `struct
     in_addr' (similar to inet_aton()) or `struct
     in6_addr'.

     Supported values for _address_family_ are currently AF_INET
     and AF_INET6. If the IP address string
     _ip_string_ is invalid, see socket.error will be raised.
     Note that exactly what is valid depends on both the value of
     _address_family_ and the underlying implementation of
     `inet_pton()'.

     Availability: Unix (maybe not all platforms).

     New in version 2.3.

 -- Function: socket.inet_ntop (address_family, packed_ip)
     Convert a packed IP address (a string of some number of
     characters) to its standard, family-specific string representation
     (for example, `'7.10.0.5'' or `'5aef:2b::8'') inet_ntop()
     is useful when a library or network protocol returns an
     object of type `struct in_addr' (similar to inet_ntoa())
     or `struct in6_addr'.

     Supported values for _address_family_ are currently AF_INET
     and AF_INET6. If the string _packed_ip_ is not
     the correct length for the specified address family, see
     ValueError will be raised.  A socket.error is
     raised for errors from the call to inet_ntop().

     Availability: Unix (maybe not all platforms).

     New in version 2.3.

 -- Function: socket.getdefaulttimeout ()
     Return the default timeout in floating seconds for new socket
     objects. A value of `None' indicates that new socket objects have
     no timeout. When the socket module is first imported, the default
     is `None'.

     New in version 2.3.

 -- Function: socket.setdefaulttimeout (timeout)
     Set the default timeout in floating seconds for new socket
     objects. A value of `None' indicates that new socket objects have
     no timeout. When the socket module is first imported, the default
     is `None'.

     New in version 2.3.

 -- Data: socket.SocketType
     This is a Python type object that represents the socket object
     type. It is the same as `type(socket(...))'.

Module SocketServer.
     Classes that simplify writing network servers.

5.17.2.1 Socket Objects
.......................

Socket objects have the following methods.  Except for `makefile()'
these correspond to Unix system calls applicable to sockets.

 -- Method: socket.accept ()
     Accept a connection. The socket must be bound to an address and
     listening for connections. The return value is a pair `(conn,
     address)' where _conn_ is a _new_ socket object usable to send and
     receive data on the connection, and _address_ is the address bound
     to the socket on the other end of the connection.

 -- Method: socket.bind (address)
     Bind the socket to _address_.  The socket must not already be
     bound. (The format of _address_ depends on the address family --
     see above.)

          Note: This method has historically accepted a pair of
          parameters for AF_INET.  addresses instead of
          only a tuple.  This was never intentional and is no longer
          available in Python 2.0 and later.

 -- Method: socket.close ()
     Close the socket.  All future operations on the socket object will
     fail. The remote end will receive no more data (after queued data
     is flushed). Sockets are automatically closed when they are
     garbage-collected.

 -- Method: socket.connect (address)
     Connect to a remote socket at _address_. (The format of _address_
     depends on the address family -- see above.)

          Note: This method has historically accepted a pair of
          parameters for AF_INET.  addresses instead of
          only a tuple.  This was never intentional and is no longer
          available in Python 2.0 and later.

 -- Method: socket.connect_ex (address)
     Like `connect(address)', but return an error indicator instead of
     raising an exception for errors returned by the C-level
     `connect()' call (other problems, such as "host not found," can
     still raise exceptions).  The error indicator is `0' if the
     operation succeeded, otherwise the value of the `errno' variable.
     This is useful to support, for example, asynchronous connects.

          Note: This method has historically accepted a pair of
          parameters for AF_INET.  addresses instead of
          only a tuple. This was never intentional and is no longer
          available in Python 2.0 and later.

 -- Method: socket.fileno ()
     Return the socket's file descriptor (a small integer).  This is
     useful with select.select().

     Under Windows the small integer returned by this method cannot be
     used where a file descriptor can be used (such as
     os.fdopen()).  Unix does not have this limitation.

 -- Method: socket.getpeername ()
     Return the remote address to which the socket is connected.  This
     is useful to find out the port number of a remote IPv4/v6 socket,
     for instance. (The format of the address returned depends on the
     address family -- see above.)  On some systems this function is
     not supported.

 -- Method: socket.getsockname ()
     Return the socket's own address.  This is useful to find out the
     port number of an IPv4/v6 socket, for instance. (The format of the
     address returned depends on the address family -- see above.)

 -- Method: socket.getsockopt (level, optname[, buflen])
     Return the value of the given socket option (see the Unix man page
     `getsockopt(2)').  The needed symbolic constants (`SO_*' etc.)
     are defined in this module.  If _buflen_ is absent, an integer
     option is assumed and its integer value is returned by the
     function.  If _buflen_ is present, it specifies the maximum length
     of the buffer used to receive the option in, and this buffer is
     returned as a string.  It is up to the caller to decode the
     contents of the buffer (see the optional built-in module
     struct for a way to decode C structures encoded as strings).

 -- Method: socket.ioctl (control, option)
          Platform : Windows

     The ioctl() method is a limited interface to the
     WSAIoctl system interface.  Please refer to the Win32
     documentation
     (http://msdn.microsoft.com/en-us/library/ms741621%28VS.85%29.aspx)for
     more information.

     On other platforms, the generic fcntl.fcntl() and
     fcntl.ioctl().  functions may be used; they accept a
     socket object as their first argument.

     New in version 2.6.

 -- Method: socket.listen (backlog)
     Listen for connections made to the socket.  The _backlog_ argument
     specifies the maximum number of queued connections and should be
     at least 1; the maximum value is system-dependent (usually 5).

 -- Method: socket.makefile ([mode[, bufsize]])
     Return a _file object_ associated with the socket.  (File objects
     are described in File Objects) The file object
     references a `dup()'ped version of the socket file descriptor, so
     the file object and socket object may be closed or
     garbage-collected independently.  The socket must be in blocking
     mode (it can not have a timeout). The optional _mode_ and
     _bufsize_ arguments are interpreted the same way as by the built-in
     file() function.

 -- Method: socket.recv (bufsize[, flags])
     Receive data from the socket.  The return value is a string
     representing the data received.  The maximum amount of data to be
     received at once is specified by _bufsize_.  See the Unix manual
     page `recv(2)' for the meaning of the optional argument _flags_;
     it defaults to zero.

          Note: For best match with hardware and network realities, the
          value of  _bufsize_ should be a relatively small power of 2,
          for example, 4096.

 -- Method: socket.recvfrom (bufsize[, flags])
     Receive data from the socket.  The return value is a pair
     `(string, address)' where _string_ is a string representing the
     data received and _address_ is the address of the socket sending
     the data.  See the Unix manual page `recv(2)' for the meaning of
     the optional argument _flags_; it defaults to zero. (The format of
     _address_ depends on the address family -- see above.)

 -- Method: socket.recvfrom_into (buffer[, nbytes[, flags]])
     Receive data from the socket, writing it into _buffer_ instead of
     creating a new string.  The return value is a pair `(nbytes,
     address)' where _nbytes_ is the number of bytes received and
     _address_ is the address of the socket sending the data.  See the
     Unix manual page `recv(2)' for the meaning of the optional
     argument _flags_; it defaults to zero.  (The format of _address_
     depends on the address family -- see above.)

     New in version 2.5.

 -- Method: socket.recv_into (buffer[, nbytes[, flags]])
     Receive up to _nbytes_ bytes from the socket, storing the data
     into a buffer rather than creating a new string.  If _nbytes_ is
     not specified (or 0), receive up to the size available in the
     given buffer.  Returns the number of bytes received.  See the Unix
     manual page `recv(2)' for the meaning of the optional argument
     _flags_; it defaults to zero.

     New in version 2.5.

 -- Method: socket.send (string[, flags])
     Send data to the socket.  The socket must be connected to a remote
     socket.  The optional _flags_ argument has the same meaning as for
     recv() above.  Returns the number of bytes sent.
     Applications are responsible for checking that all data has been
     sent; if only some of the data was transmitted, the application
     needs to attempt delivery of the remaining data.

 -- Method: socket.sendall (string[, flags])
     Send data to the socket.  The socket must be connected to a remote
     socket.  The optional _flags_ argument has the same meaning as for
     recv() above.  Unlike send(), this method
     continues to send data from _string_ until either all data has
     been sent or an error occurs.  `None' is returned on success.  On
     error, an exception is raised, and there is no way to determine how
     much data, if any, was successfully sent.

 -- Method: socket.sendto (string[, flags], address)
     Send data to the socket.  The socket should not be connected to a
     remote socket, since the destination socket is specified by
     _address_.  The optional _flags_ argument has the same meaning as
     for recv() above.  Return the number of bytes sent.
     (The format of _address_ depends on the address family -- see
     above.)

 -- Method: socket.setblocking (flag)
     Set blocking or non-blocking mode of the socket: if _flag_ is 0,
     the socket is set to non-blocking, else to blocking mode.
     Initially all sockets are in blocking mode.  In non-blocking mode,
     if a recv() call doesn't find any data, or if a
     send() call can't immediately dispose of the data, a
     error exception is raised; in blocking mode, the calls block
     until they can proceed. `s.setblocking(0)' is equivalent to
     `s.settimeout(0.0)'; `s.setblocking(1)' is equivalent to
     `s.settimeout(None)'.

 -- Method: socket.settimeout (value)
     Set a timeout on blocking socket operations.  The _value_ argument
     can be a nonnegative float expressing seconds, or `None'. If a
     float is given, subsequent socket operations will raise a
     timeout exception if the timeout period _value_ has elapsed
     before the operation has completed.  Setting a timeout of `None'
     disables timeouts on socket operations.  `s.settimeout(0.0)' is
     equivalent to `s.setblocking(0)'; `s.settimeout(None)' is
     equivalent to `s.setblocking(1)'.

     New in version 2.3.

 -- Method: socket.gettimeout ()
     Return the timeout in floating seconds associated with socket
     operations, or `None' if no timeout is set.  This reflects the
     last call to setblocking() or settimeout().

     New in version 2.3.

  Some notes on socket blocking and timeouts: A socket object can be in
one of three modes: blocking, non-blocking, or timeout.  Sockets are
always created in blocking mode.  In blocking mode, operations block
until complete or the system returns an error (such as connection timed
out).  In non-blocking mode, operations fail (with an error that is
unfortunately system-dependent) if they cannot be completed
immediately.  In timeout mode, operations fail if they cannot be
completed within the timeout specified for the socket or if the system
returns an error.  The setblocking().  method is simply a
shorthand for certain settimeout() calls.

  Timeout mode internally sets the socket in non-blocking mode.  The
blocking and timeout modes are shared between file descriptors and
socket objects that refer to the same network endpoint.  A consequence
of this is that file objects returned by the makefile().
method must only be used when the socket is in blocking mode; in
timeout or non-blocking mode file operations that cannot be completed
immediately will fail.

  Note that the connect() operation is subject to the
timeout setting, and in general it is recommended to call
settimeout().  before calling connect() or pass a
timeout parameter to create_connection().  The system
network stack may return a connection timeout error of its own
regardless of any Python socket timeout setting.

 -- Method: socket.setsockopt (level, optname, value)
     Set the value of the given socket option (see the Unix manual page
     `setsockopt(2)').  The needed symbolic constants are defined in the
     socket module (`SO_*' etc.).  The value can be an
     integer or a string representing a buffer.  In the latter case it
     is up to the caller to ensure that the string contains the proper
     bits (see the optional built-in module struct for a
     way to encode C structures as strings).

 -- Method: socket.shutdown (how)
     Shut down one or both halves of the connection.  If _how_ is
     `SHUT_RD', further receives are disallowed.  If _how_ is
     `SHUT_WR', further sends are disallowed.  If _how_ is `SHUT_RDWR',
     further sends and receives are disallowed.  Depending on the
     platform, shutting down one half of the connection can also close
     the opposite half (e.g. on Mac OS X, `shutdown(SHUT_WR)' does not
     allow further reads on the other end of the connection).

  Note that there are no methods `read()' or `write()'; use
recv() and send() without _flags_ argument instead.

  Socket objects also have these (read-only) attributes that correspond
to the values given to the socket constructor.

 -- Attribute: socket.family
     The socket family.

     New in version 2.5.

 -- Attribute: socket.type
     The socket type.

     New in version 2.5.

 -- Attribute: socket.proto
     The socket protocol.

     New in version 2.5.

5.17.2.2 Example
................

Here are four minimal example programs using the TCP/IP protocol: a
server that echoes all data that it receives back (servicing only one
client), and a client using it.  Note that a server must perform the
sequence socket(), bind(), listen(),
accept(). (possibly repeating the accept() to
service more than one client), while a client only needs the sequence
socket(), connect().  Also note that the server
does not send()./see recv() on the socket it is
listening on but on the new socket returned by accept().

  The first two examples support IPv4 only.

    # Echo server program
    import socket

    HOST = ''                 # Symbolic name meaning all available interfaces
    PORT = 50007              # Arbitrary non-privileged port
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, PORT))
    s.listen(1)
    conn, addr = s.accept()
    print 'Connected by', addr
    while 1:
        data = conn.recv(1024)
        if not data: break
        conn.send(data)
    conn.close()

    # Echo client program
    import socket

    HOST = 'daring.cwi.nl'    # The remote host
    PORT = 50007              # The same port as used by the server
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((HOST, PORT))
    s.send('Hello, world')
    data = s.recv(1024)
    s.close()
    print 'Received', repr(data)

The next two examples are identical to the above two, but support both
IPv4 and IPv6. The server side will listen to the first address family
available (it should listen to both instead). On most of IPv6-ready
systems, IPv6 will take precedence and the server may not accept IPv4
traffic. The client side will try to connect to the all addresses
returned as a result of the name resolution, and sends traffic to the
first one connected successfully.

    # Echo server program
    import socket
    import sys

    HOST = None               # Symbolic name meaning all available interfaces
    PORT = 50007              # Arbitrary non-privileged port
    s = None
    for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC,
                                  socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
        af, socktype, proto, canonname, sa = res
        try:
            s = socket.socket(af, socktype, proto)
        except socket.error, msg:
            s = None
            continue
        try:
            s.bind(sa)
            s.listen(1)
        except socket.error, msg:
            s.close()
            s = None
            continue
        break
    if s is None:
        print 'could not open socket'
        sys.exit(1)
    conn, addr = s.accept()
    print 'Connected by', addr
    while 1:
        data = conn.recv(1024)
        if not data: break
        conn.send(data)
    conn.close()

    # Echo client program
    import socket
    import sys

    HOST = 'daring.cwi.nl'    # The remote host
    PORT = 50007              # The same port as used by the server
    s = None
    for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC, socket.SOCK_STREAM):
        af, socktype, proto, canonname, sa = res
        try:
            s = socket.socket(af, socktype, proto)
        except socket.error, msg:
            s = None
            continue
        try:
            s.connect(sa)
        except socket.error, msg:
            s.close()
            s = None
            continue
        break
    if s is None:
        print 'could not open socket'
        sys.exit(1)
    s.send('Hello, world')
    data = s.recv(1024)
    s.close()
    print 'Received', repr(data)

The last example shows how to write a very simple network sniffer with
raw sockets on Windows. The example requires administrator privileges
to modify the interface:

    import socket

    # the public network interface
    HOST = socket.gethostbyname(socket.gethostname())

    # create a raw socket and bind it to the public interface
    s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
    s.bind((HOST, 0))

    # Include IP headers
    s.setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1)

    # receive all packages
    s.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)

    # receive a package
    print s.recvfrom(65565)

    # disabled promiscuous mode
    s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)

5.17.3 `ssl' -- SSL wrapper for socket objects
----------------------------------------------

New in version 2.6.

  This module provides access to Transport Layer Security (often known
as "Secure Sockets Layer") encryption and peer authentication
facilities for network sockets, both client-side and server-side.  This
module uses the OpenSSL library. It is available on all modern Unix
systems, Windows, Mac OS X, and probably additional platforms, as long
as OpenSSL is installed on that platform.

     Note: Some behavior may be platform dependent, since calls are
     made to the operating system socket APIs.  The installed version
     of OpenSSL may also cause variations in behavior.

  This section documents the objects and functions in the `ssl' module;
for more general information about TLS, SSL, and certificates, the
reader is referred to the documents in the "See Also" section at the
bottom.

  This module provides a class, `ssl.SSLSocket', which is derived from
the socket.socket type, and provides a socket-like wrapper
that also encrypts and decrypts the data going over the socket with
SSL.  It supports additional `read()' and `write()' methods, along with
a method, `getpeercert()', to retrieve the certificate of the other
side of the connection, and a method, `cipher()', to retrieve the
cipher being used for the secure connection.

5.17.3.1 Functions, Constants, and Exceptions
.............................................

 -- Exception: ssl.SSLError
     Raised to signal an error from the underlying SSL implementation.
     This signifies some problem in the higher-level encryption and
     authentication layer that's superimposed on the underlying network
     connection.  This error is a subtype of socket.error,
     which in turn is a subtype of IOError.

 -- Function: ssl.wrap_socket (sock, keyfile=None, certfile=None,
          server_side=False, cert_reqs=CERT_NONE, ssl_version={see
          docs}, ca_certs=None, do_handshake_on_connect=True,
          suppress_ragged_eofs=True, ciphers=None)
     Takes an instance `sock' of socket.socket, and returns
     an instance of `ssl.SSLSocket', a subtype of socket.socket,
     which wraps the underlying socket in an SSL context.  For
     client-side sockets, the context construction is lazy; if the
     underlying socket isn't connected yet, the context construction
     will be performed after `connect()' is called on the socket.  For
     server-side sockets, if the socket has no remote peer, it is
     assumed to be a listening socket, and the server-side SSL wrapping
     is automatically performed on client connections accepted via the
     `accept()' method.  wrap_socket() may raise
     SSLError.

     The `keyfile' and `certfile' parameters specify optional files
     which contain a certificate to be used to identify the local side
     of the connection.  See the discussion of Certificates
     for more information on how the certificate is stored in the
     `certfile'.

     Often the private key is stored in the same file as the
     certificate; in this case, only the `certfile' parameter need be
     passed.  If the private key is stored in a separate file, both
     parameters must be used.  If the private key is stored in the
     `certfile', it should come before the first certificate in the
     certificate chain:

         -----BEGIN RSA PRIVATE KEY-----
         ... (private key in base64 encoding) ...
         -----END RSA PRIVATE KEY-----
         -----BEGIN CERTIFICATE-----
         ... (certificate in base64 PEM encoding) ...
         -----END CERTIFICATE-----

     The parameter `server_side' is a boolean which identifies whether
     server-side or client-side behavior is desired from this socket.

     The parameter `cert_reqs' specifies whether a certificate is
     required from the other side of the connection, and whether it
     will be validated if provided.  It must be one of the three values
     CERT_NONE.  (certificates ignored), see
     CERT_OPTIONAL. (not required, but validated if provided), or
     CERT_REQUIRED. (required and validated).  If the value
     of this parameter is not CERT_NONE, then the `ca_certs'
     parameter must point to a file of CA certificates.

     The `ca_certs' file contains a set of concatenated "certification
     authority" certificates, which are used to validate certificates
     passed from the other end of the connection.  See the discussion of
     Certificates for more information about how to
     arrange the certificates in this file.

     The parameter `ssl_version' specifies which version of the SSL
     protocol to use.  Typically, the server chooses a particular
     protocol version, and the client must adapt to the server's
     choice.  Most of the versions are not interoperable with the other
     versions.  If not specified, for client-side operation, the
     default SSL version is SSLv3; for server-side operation, SSLv23.
     These version selections provide the most compatibility with other
     versions.

     Here's a table showing which versions in a client (down the side)
     can connect to which versions in a server (along the top):

          _client_ / *server*          *SSLv2*       *SSLv3*       *SSLv23*       *TLSv1*
          _SSLv2_                      yes           no            yes            no
          _SSLv3_                      yes           yes           yes            no
          _SSLv23_                     yes           no            yes            no
          _TLSv1_                      no            no            yes            yes

          Note: Which connections succeed will vary depending on the
          version of OpenSSL.  For instance, in some older versions of
          OpenSSL (such as 0.9.7l on OS X 10.4), an SSLv2 client could
          not connect to an SSLv23 server.  Another example: beginning
          with OpenSSL 1.0.0, an SSLv23 client will not actually
          attempt SSLv2 connections unless you explicitly enable SSLv2
          ciphers; for example, you might specify `"ALL"' or `"SSLv2"'
          as the _ciphers_ parameter to enable them.

     The _ciphers_ parameter sets the available ciphers for this SSL
     object.  It should be a string in the OpenSSL cipher list format
     (http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT).

     The parameter `do_handshake_on_connect' specifies whether to do
     the SSL handshake automatically after doing a `socket.connect()',
     or whether the application program will call it explicitly, by
     invoking the SSLSocket.do_handshake() method.  Calling
     SSLSocket.do_handshake() explicitly gives the program
     control over the blocking behavior of the socket I/O involved in
     the handshake.

     The parameter `suppress_ragged_eofs' specifies how the
     SSLSocket.read() method should signal unexpected EOF from
     the other end of the connection.  If specified as True.
     (the default), it returns a normal EOF in response to unexpected
     EOF errors raised from the underlying socket; if False,
     it will raise the exceptions back to the caller.

     Changed in version 2.7: New optional argument _ciphers_.

 -- Function: ssl.RAND_status ()
     Returns True if the SSL pseudo-random number generator has been
     seeded with 'enough' randomness, and False otherwise.  You can use
     ssl.RAND_egd().  and ssl.RAND_add() to
     increase the randomness of the pseudo-random number generator.

 -- Function: ssl.RAND_egd (path)
     If you are running an entropy-gathering daemon (EGD) somewhere,
     and `path' is the pathname of a socket connection open to it, this
     will read 256 bytes of randomness from the socket, and add it to
     the SSL pseudo-random number generator to increase the security of
     generated secret keys.  This is typically only necessary on
     systems without better sources of randomness.

     See <http://egd.sourceforge.net/> or
     <http://prngd.sourceforge.net/> for sources of entropy-gathering
     daemons.

 -- Function: ssl.RAND_add (bytes, entropy)
     Mixes the given `bytes' into the SSL pseudo-random number
     generator.  The parameter `entropy' (a float) is a lower bound on
     the entropy contained in string (so you can always use `0.0').  See RFC
     1750 (http://tools.ietf.org/html/rfc1750.html) for more
     information on sources of entropy.

 -- Function: ssl.cert_time_to_seconds (timestring)
     Returns a floating-point value containing a normal
     seconds-after-the-epoch time value, given the time-string
     representing the "notBefore" or "notAfter" date from a certificate.

     Here's an example:

         >>> import ssl
         >>> ssl.cert_time_to_seconds("May  9 00:00:00 2007 GMT")
         1178694000.0
         >>> import time
         >>> time.ctime(ssl.cert_time_to_seconds("May  9 00:00:00 2007 GMT"))
         'Wed May  9 00:00:00 2007'
         >>>

 -- Function: ssl.get_server_certificate (addr,
          ssl_version=PROTOCOL_SSLv3, ca_certs=None)
     Given the address `addr' of an SSL-protected server, as a
     (_hostname_, _port-number_) pair, fetches the server's
     certificate, and returns it as a PEM-encoded string.  If
     `ssl_version' is specified, uses that version of the SSL protocol
     to attempt to connect to the server.  If `ca_certs' is specified,
     it should be a file containing a list of root certificates, the
     same format as used for the same parameter in wrap_socket().
     The call will attempt to validate the server certificate
     against that set of root certificates, and will fail if the
     validation attempt fails.

 -- Function: ssl.DER_cert_to_PEM_cert (DER_cert_bytes)
     Given a certificate as a DER-encoded blob of bytes, returns a
     PEM-encoded string version of the same certificate.

 -- Function: ssl.PEM_cert_to_DER_cert (PEM_cert_string)
     Given a certificate as an ASCII PEM string, returns a DER-encoded
     sequence of bytes for that same certificate.

 -- Data: ssl.CERT_NONE
     Value to pass to the `cert_reqs' parameter to `sslobject()' when no
     certificates will be required or validated from the other side of
     the socket connection.

 -- Data: ssl.CERT_OPTIONAL
     Value to pass to the `cert_reqs' parameter to `sslobject()' when no
     certificates will be required from the other side of the socket
     connection, but if they are provided, will be validated.  Note
     that use of this setting requires a valid certificate validation
     file also be passed as a value of the `ca_certs' parameter.

 -- Data: ssl.CERT_REQUIRED
     Value to pass to the `cert_reqs' parameter to `sslobject()' when
     certificates will be required from the other side of the socket
     connection.  Note that use of this setting requires a valid
     certificate validation file also be passed as a value of the
     `ca_certs' parameter.

 -- Data: ssl.PROTOCOL_SSLv2
     Selects SSL version 2 as the channel encryption protocol.

          Warning: SSL version 2 is insecure.  Its use is highly
          discouraged.

 -- Data: ssl.PROTOCOL_SSLv23
     Selects SSL version 2 or 3 as the channel encryption protocol.
     This is a setting to use with servers for maximum compatibility
     with the other end of an SSL connection, but it may cause the
     specific ciphers chosen for the encryption to be of fairly low
     quality.

 -- Data: ssl.PROTOCOL_SSLv3
     Selects SSL version 3 as the channel encryption protocol.  For
     clients, this is the maximally compatible SSL variant.

 -- Data: ssl.PROTOCOL_TLSv1
     Selects TLS version 1 as the channel encryption protocol.  This is
     the most modern version, and probably the best choice for maximum
     protection, if both sides can speak it.

 -- Data: ssl.OPENSSL_VERSION
     The version string of the OpenSSL library loaded by the
     interpreter:

         >>> ssl.OPENSSL_VERSION
         'OpenSSL 0.9.8k 25 Mar 2009'

     New in version 2.7.

 -- Data: ssl.OPENSSL_VERSION_INFO
     A tuple of five integers representing version information about the
     OpenSSL library:

         >>> ssl.OPENSSL_VERSION_INFO
         (0, 9, 8, 11, 15)

     New in version 2.7.

 -- Data: ssl.OPENSSL_VERSION_NUMBER
     The raw version number of the OpenSSL library, as a single integer:

         >>> ssl.OPENSSL_VERSION_NUMBER
         9470143L
         >>> hex(ssl.OPENSSL_VERSION_NUMBER)
         '0x9080bfL'

     New in version 2.7.

5.17.3.2 SSLSocket Objects
..........................

 -- Method: SSLSocket.read ([nbytes=1024])
     Reads up to `nbytes' bytes from the SSL-encrypted channel and
     returns them.

 -- Method: SSLSocket.write (data)
     Writes the `data' to the other side of the connection, using the
     SSL channel to encrypt.  Returns the number of bytes written.

 -- Method: SSLSocket.getpeercert (binary_form=False)
     If there is no certificate for the peer on the other end of the
     connection, returns `None'.

     If the parameter `binary_form' is False, and a
     certificate was received from the peer, this method returns a
     dict instance.  If the certificate was not validated,
     the dict is empty.  If the certificate was validated, it returns a
     dict with the keys `subject' (the principal for which the
     certificate was issued), and `notAfter' (the time after which the
     certificate should not be trusted).  The certificate was already
     validated, so the `notBefore' and `issuer' fields are not
     returned.  If a certificate contains an instance of the _Subject
     Alternative Name_ extension (see RFC 3280
     (http://tools.ietf.org/html/rfc3280.html)), there will also be a
     `subjectAltName' key in the dictionary.

     The "subject" field is a tuple containing the sequence of relative
     distinguished names (RDNs) given in the certificate's data
     structure for the principal, and each RDN is a sequence of
     name-value pairs:

         {'notAfter': 'Feb 16 16:54:50 2013 GMT',
          'subject': ((('countryName', u'US'),),
                      (('stateOrProvinceName', u'Delaware'),),
                      (('localityName', u'Wilmington'),),
                      (('organizationName', u'Python Software Foundation'),),
                      (('organizationalUnitName', u'SSL'),),
                      (('commonName', u'somemachine.python.org'),))}

     If the `binary_form' parameter is True, and a
     certificate was provided, this method returns the DER-encoded form
     of the entire certificate as a sequence of bytes, or None
     if the peer did not provide a certificate.  This return value
     is independent of validation; if validation was required (see
     CERT_OPTIONAL or CERT_REQUIRED), it will have
     been validated, but if CERT_NONE was used to
     establish the connection, the certificate, if present, will not
     have been validated.

 -- Method: SSLSocket.cipher ()
     Returns a three-value tuple containing the name of the cipher
     being used, the version of the SSL protocol that defines its use,
     and the number of secret bits being used.  If no connection has
     been established, returns `None'.

 -- Method: SSLSocket.do_handshake ()
     Perform a TLS/SSL handshake.  If this is used with a non-blocking
     socket, it may raise SSLError with an `arg[0]' of
     `SSL_ERROR_WANT_READ' or `SSL_ERROR_WANT_WRITE', in which case it
     must be called again until it completes successfully.  For
     example, to simulate the behavior of a blocking socket, one might
     write:

         while True:
             try:
                 s.do_handshake()
                 break
             except ssl.SSLError, err:
                 if err.args[0] == ssl.SSL_ERROR_WANT_READ:
                     select.select([s], [], [])
                 elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
                     select.select([], [s], [])
                 else:
                     raise

 -- Method: SSLSocket.unwrap ()
     Performs the SSL shutdown handshake, which removes the TLS layer
     from the underlying socket, and returns the underlying socket
     object.  This can be used to go from encrypted operation over a
     connection to unencrypted.  The socket instance returned should
     always be used for further communication with the other side of
     the connection, rather than the original socket instance (which
     may not function properly after the unwrap).

5.17.3.3 Certificates
.....................

Certificates in general are part of a public-key / private-key system.
In this system, each _principal_, (which may be a machine, or a person,
or an organization) is assigned a unique two-part encryption key.  One
part of the key is public, and is called the _public key_; the other
part is kept secret, and is called the _private key_.  The two parts
are related, in that if you encrypt a message with one of the parts,
you can decrypt it with the other part, and *only* with the other part.

  A certificate contains information about two principals.  It contains
the name of a _subject_, and the subject's public key.  It also
contains a statement by a second principal, the _issuer_, that the
subject is who he claims to be, and that this is indeed the subject's
public key.  The issuer's statement is signed with the issuer's private
key, which only the issuer knows.  However, anyone can verify the
issuer's statement by finding the issuer's public key, decrypting the
statement with it, and comparing it to the other information in the
certificate.  The certificate also contains information about the time
period over which it is valid.  This is expressed as two fields, called
"notBefore" and "notAfter".

  In the Python use of certificates, a client or server can use a
certificate to prove who they are.  The other side of a network
connection can also be required to produce a certificate, and that
certificate can be validated to the satisfaction of the client or
server that requires such validation.  The connection attempt can be
set to raise an exception if the validation fails.  Validation is done
automatically, by the underlying OpenSSL framework; the application
need not concern itself with its mechanics.  But the application does
usually need to provide sets of certificates to allow this process to
take place.

  Python uses files to contain certificates.  They should be formatted
as "PEM" (see RFC 1422 (http://tools.ietf.org/html/rfc1422.html)),
which is a base-64 encoded form wrapped with a header line and a footer
line:

    -----BEGIN CERTIFICATE-----
    ... (certificate in base64 PEM encoding) ...
    -----END CERTIFICATE-----

The Python files which contain certificates can contain a sequence of
certificates, sometimes called a _certificate chain_.  This chain
should start with the specific certificate for the principal who "is"
the client or server, and then the certificate for the issuer of that
certificate, and then the certificate for the issuer of _that_
certificate, and so on up the chain till you get to a certificate which
is _self-signed_, that is, a certificate which has the same subject and
issuer, sometimes called a _root certificate_.  The certificates should
just be concatenated together in the certificate file.  For example,
suppose we had a three certificate chain, from our server certificate
to the certificate of the certification authority that signed our server
certificate, to the root certificate of the agency which issued the
certification authority's certificate:

    -----BEGIN CERTIFICATE-----
    ... (certificate for your server)...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    ... (the certificate for the CA)...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    ... (the root certificate for the CA's issuer)...
    -----END CERTIFICATE-----

If you are going to require validation of the other side of the
connection's certificate, you need to provide a "CA certs" file, filled
with the certificate chains for each issuer you are willing to trust.
Again, this file just contains these chains concatenated together.  For
validation, Python will use the first chain it finds in the file which
matches.

  Some "standard" root certificates are available from various
certification authorities: CACert.org
(http://www.cacert.org/index.php?id=3), Thawte
(http://www.thawte.com/roots/), Verisign
(http://www.verisign.com/support/roots.html), Positive SSL
(http://www.PositiveSSL.com/ssl-certificate-support/cert_installation/UTN-USERFirst-Hardware.crt)(used
by python.org), Equifax and GeoTrust
(http://www.geotrust.com/resources/root_certificates/index.asp).

  In general, if you are using SSL3 or TLS1, you don't need to put the
full chain in your "CA certs" file; you only need the root
certificates, and the remote peer is supposed to furnish the other
certificates necessary to chain from its certificate to a root
certificate.  See RFC 4158 (http://tools.ietf.org/html/rfc4158.html)
for more discussion of the way in which certification chains can be
built.

  If you are going to create a server that provides SSL-encrypted
connection services, you will need to acquire a certificate for that
service.  There are many ways of acquiring appropriate certificates,
such as buying one from a certification authority.  Another common
practice is to generate a self-signed certificate.  The simplest way to
do this is with the OpenSSL package, using something like the following:

    % openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem
    Generating a 1024 bit RSA private key
    .......++++++
    .............................++++++
    writing new private key to 'cert.pem'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:MyState
    Locality Name (eg, city) []:Some City
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Organization, Inc.
    Organizational Unit Name (eg, section) []:My Group
    Common Name (eg, YOUR name) []:myserver.mygroup.myorganization.com
    Email Address []:ops@myserver.mygroup.myorganization.com
    %

The disadvantage of a self-signed certificate is that it is its own root
certificate, and no one else will have it in their cache of known (and
trusted) root certificates.

5.17.3.4 Examples
.................

5.17.3.5 Testing for SSL support
................................

To test for the presence of SSL support in a Python installation, user
code should use the following idiom:

    try:
       import ssl
    except ImportError:
       pass
    else:
       [ do something that requires SSL support ]

5.17.3.6 Client-side operation
..............................

This example connects to an SSL server, prints the server's address and
certificate, sends some bytes, and reads part of the response:

    import socket, ssl, pprint

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # require a certificate from the server
    ssl_sock = ssl.wrap_socket(s,
                               ca_certs="/etc/ca_certs_file",
                               cert_reqs=ssl.CERT_REQUIRED)

    ssl_sock.connect(('www.verisign.com', 443))

    print repr(ssl_sock.getpeername())
    print ssl_sock.cipher()
    print pprint.pformat(ssl_sock.getpeercert())

    # Set a simple HTTP request -- use httplib in actual code.
    ssl_sock.write("""GET / HTTP/1.0\r
    Host: www.verisign.com\r\n\r\n""")

    # Read a chunk of data.  Will not necessarily
    # read all the data returned by the server.
    data = ssl_sock.read()

    # note that closing the SSLSocket will also close the underlying socket
    ssl_sock.close()

As of September 6, 2007, the certificate printed by this program looked
like this:

    {'notAfter': 'May  8 23:59:59 2009 GMT',
     'subject': ((('serialNumber', u'2497886'),),
                 (('1.3.6.1.4.1.311.60.2.1.3', u'US'),),
                 (('1.3.6.1.4.1.311.60.2.1.2', u'Delaware'),),
                 (('countryName', u'US'),),
                 (('postalCode', u'94043'),),
                 (('stateOrProvinceName', u'California'),),
                 (('localityName', u'Mountain View'),),
                 (('streetAddress', u'487 East Middlefield Road'),),
                 (('organizationName', u'VeriSign, Inc.'),),
                 (('organizationalUnitName',
                   u'Production Security Services'),),
                 (('organizationalUnitName',
                   u'Terms of use at www.verisign.com/rpa (c)06'),),
                 (('commonName', u'www.verisign.com'),))}

which is a fairly poorly-formed `subject' field.

5.17.3.7 Server-side operation
..............................

For server operation, typically you'd need to have a server
certificate, and private key, each in a file.  You'd open a socket,
bind it to a port, call `listen()' on it, then start waiting for
clients to connect:

    import socket, ssl

    bindsocket = socket.socket()
    bindsocket.bind(('myaddr.mydomain.com', 10023))
    bindsocket.listen(5)

When one did, you'd call `accept()' on the socket to get the new socket
from the other end, and use wrap_socket() to create a
server-side SSL context for it:

    while True:
       newsocket, fromaddr = bindsocket.accept()
       connstream = ssl.wrap_socket(newsocket,
                                    server_side=True,
                                    certfile="mycertfile",
                                    keyfile="mykeyfile",
                                    ssl_version=ssl.PROTOCOL_TLSv1)
       deal_with_client(connstream)

Then you'd read data from the `connstream' and do something with it
till you are finished with the client (or the client is finished with
you):

    def deal_with_client(connstream):

       data = connstream.read()
       # null data means the client is finished with us
       while data:
          if not do_something(connstream, data):
             # we'll assume do_something returns False
             # when we're finished with client
             break
          data = connstream.read()
       # finished with client
       connstream.close()

And go back to listening for new client connections.

Class socket.socket.
     Documentation of underlying socket class

Introducing SSL and Certificates using OpenSSL (http://old.pseudonym.org/ssl/wwwj-index.html)
     Frederick J. Hirsch

RFC 1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management (http://www.ietf.org/rfc/rfc1422)
     Steve Kent

RFC 1750: Randomness Recommendations for Security (http://www.ietf.org/rfc/rfc1750)
     D. Eastlake et. al.

RFC 3280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile (http://www.ietf.org/rfc/rfc3280)
     Housley et. al.

5.17.4 `signal' -- Set handlers for asynchronous events
-------------------------------------------------------

This module provides mechanisms to use signal handlers in Python. Some
general rules for working with signals and their handlers:

   * A handler for a particular signal, once set, remains installed
     until it is explicitly reset (Python emulates the BSD style
     interface regardless of the underlying implementation), with the
     exception of the handler for `SIGCHLD', which follows the
     underlying implementation.

   * There is no way to "block" signals temporarily from critical
     sections (since this is not supported by all Unix flavors).

   * Although Python signal handlers are called asynchronously as far
     as the Python user is concerned, they can only occur between the
     "atomic" instructions of the Python interpreter.  This means that
     signals arriving during long calculations implemented purely in C
     (such as regular expression matches on large bodies of text) may
     be delayed for an arbitrary amount of time.

   * When a signal arrives during an I/O operation, it is possible that
     the I/O operation raises an exception after the signal handler
     returns. This is dependent on the underlying Unix system's
     semantics regarding interrupted system calls.

   * Because the C signal handler always returns, it makes little sense
     to catch synchronous errors like `SIGFPE' or `SIGSEGV'.

   * Python installs a small number of signal handlers by default:
     `SIGPIPE' is ignored (so write errors on pipes and sockets can be
     reported as ordinary Python exceptions) and `SIGINT' is translated
     into a KeyboardInterrupt exception.  All of these can
     be overridden.

   * Some care must be taken if both signals and threads are used in
     the same program.  The fundamental thing to remember in using
     signals and threads simultaneously is: always perform
     signal() operations in the main thread of execution.  Any
     thread can perform an alarm(), getsignal(),
     pause(), setitimer() or
     getitimer().; only the main thread can set a new signal
     handler, and the main thread will be the only one to receive
     signals (this is enforced by the Python signal module,
     even if the underlying thread implementation supports sending
     signals to individual threads).  This means that signals can't be
     used as a means of inter-thread communication.  Use locks instead.

  The variables defined in the signal module are:

 -- Data: signal.SIG_DFL
     This is one of two standard signal handling options; it will
     simply perform the default function for the signal.  For example,
     on most systems the default action for `SIGQUIT' is to dump core
     and exit, while the default action for `SIGCHLD' is to simply
     ignore it.

 -- Data: signal.SIG_IGN
     This is another standard signal handler, which will simply ignore
     the given signal.

 -- Data: SIG*
     All the signal numbers are defined symbolically.  For example, the
     hangup signal is defined as `signal.SIGHUP'; the variable names
     are identical to the names used in C programs, as found in
     `<signal.h>'. The Unix man page for '`signal()'' lists the
     existing signals (on some systems this is `signal(2)', on others
     the list is in `signal(7)'). Note that not all systems define the
     same set of signal names; only those names defined by the system
     are defined by this module.

 -- Data: signal.CTRL_C_EVENT
     The signal corresponding to the CTRL+C keystroke event. This
     signal can only be used with os.kill().

     Availability: Windows.

     New in version 2.7.

 -- Data: signal.CTRL_BREAK_EVENT
     The signal corresponding to the CTRL+BREAK keystroke event. This
     signal can only be used with os.kill().

     Availability: Windows.

     New in version 2.7.

 -- Data: signal.NSIG
     One more than the number of the highest signal number.

 -- Data: signal.ITIMER_REAL
     Decrements interval timer in real time, and delivers `SIGALRM'
     upon expiration.

 -- Data: signal.ITIMER_VIRTUAL
     Decrements interval timer only when the process is executing, and
     delivers SIGVTALRM upon expiration.

 -- Data: signal.ITIMER_PROF
     Decrements interval timer both when the process executes and when
     the system is executing on behalf of the process. Coupled with
     ITIMER_VIRTUAL, this timer is usually used to profile the time
     spent by the application in user and kernel space. SIGPROF is
     delivered upon expiration.

  The signal module defines one exception:

 -- Exception: signal.ItimerError
     Raised to signal an error from the underlying setitimer()
     or getitimer() implementation. Expect this
     error if an invalid interval timer or a negative time is passed to
     setitimer().  This error is a subtype of
     IOError.

  The signal module defines the following functions:

 -- Function: signal.alarm (time)
     If _time_ is non-zero, this function requests that a `SIGALRM'
     signal be sent to the process in _time_ seconds. Any previously
     scheduled alarm is canceled (only one alarm can be scheduled at
     any time).  The returned value is then the number of seconds
     before any previously set alarm was to have been delivered. If
     _time_ is zero, no alarm is scheduled, and any scheduled alarm is
     canceled.  If the return value is zero, no alarm is currently
     scheduled.  (See the Unix man page `alarm(2)'.) Availability: Unix.

 -- Function: signal.getsignal (signalnum)
     Return the current signal handler for the signal _signalnum_. The
     returned value may be a callable Python object, or one of the
     special values signal.SIG_IGN, signal.SIG_DFL
     or None.  Here, see signal.SIG_IGN means
     that the signal was previously ignored, see signal.SIG_DFL
     means that the default way of handling the signal was
     previously in use, and `None' means that the previous signal
     handler was not installed from Python.

 -- Function: signal.pause ()
     Cause the process to sleep until a signal is received; the
     appropriate handler will then be called.  Returns nothing.  Not on
     Windows. (See the Unix man page `signal(2)'.)

 -- Function: signal.setitimer (which, seconds[, interval])
     Sets given interval timer (one of signal.ITIMER_REAL,
     signal.ITIMER_VIRTUAL or signal.ITIMER_PROF)
     specified by _which_ to fire after _seconds_ (float is
     accepted, different from alarm()) and after that
     every _interval_ seconds. The interval timer specified by _which_
     can be cleared by setting seconds to zero.

     When an interval timer fires, a signal is sent to the process.
     The signal sent is dependent on the timer being used; see
     signal.ITIMER_REAL will deliver `SIGALRM', see
     signal.ITIMER_VIRTUAL sends `SIGVTALRM', and
     signal.ITIMER_PROF will deliver `SIGPROF'.

     The old values are returned as a tuple: (delay, interval).

     Attempting to pass an invalid interval timer will cause an
     ItimerError.  Availability: Unix.

     New in version 2.6.

 -- Function: signal.getitimer (which)
     Returns current value of a given interval timer specified by
     _which_.  Availability: Unix.

     New in version 2.6.

 -- Function: signal.set_wakeup_fd (fd)
     Set the wakeup fd to _fd_.  When a signal is received, a `'\0''
     byte is written to the fd.  This can be used by a library to
     wakeup a poll or select call, allowing the signal to be fully
     processed.

     The old wakeup fd is returned.  _fd_ must be non-blocking.  It is
     up to the library to remove any bytes before calling poll or
     select again.

     When threads are enabled, this function can only be called from
     the main thread; attempting to call it from other threads will
     cause a ValueError.  exception to be raised.

 -- Function: signal.siginterrupt (signalnum, flag)
     Change system call restart behaviour: if _flag_ is False,
     system calls will be restarted when interrupted by signal
     _signalnum_, otherwise system calls will be interrupted.  Returns
     nothing.  Availability: Unix (see the man page `siginterrupt(3)'
     for further information).

     Note that installing a signal handler with signal().
     will reset the restart behaviour to interruptible by implicitly
     calling `siginterrupt()' with a true _flag_ value for the given
     signal.

     New in version 2.6.

 -- Function: signal.signal (signalnum, handler)
     Set the handler for signal _signalnum_ to the function _handler_.
     _handler_ can be a callable Python object taking two arguments
     (see below), or one of the special values signal.SIG_IGN
     or signal.SIG_DFL.  The previous signal handler
     will be returned (see the description of getsignal().
     above).  (See the Unix man page `signal(2)'.)

     When threads are enabled, this function can only be called from
     the main thread; attempting to call it from other threads will
     cause a ValueError.  exception to be raised.

     The _handler_ is called with two arguments: the signal number and
     the current stack frame (`None' or a frame object; for a
     description of frame objects, see the description in the type hierarchy
     or see the attribute descriptions in the
     inspect module).

     On Windows, see signal() can only be called with `SIGABRT',
     `SIGFPE', `SIGILL', `SIGINT', `SIGSEGV', or `SIGTERM'. A
     ValueError will be raised in any other case.

5.17.4.1 Example
................

Here is a minimal example program. It uses the alarm().
function to limit the time spent waiting to open a file; this is useful
if the file is for a serial device that may not be turned on, which
would normally cause the os.open() to hang indefinitely.
The solution is to set a 5-second alarm before opening the file; if the
operation takes too long, the alarm signal will be sent, and the
handler raises an exception.

    import signal, os

    def handler(signum, frame):
        print 'Signal handler called with signal', signum
        raise IOError("Couldn't open device!")

    # Set the signal handler and a 5-second alarm
    signal.signal(signal.SIGALRM, handler)
    signal.alarm(5)

    # This open() may hang indefinitely
    fd = os.open('/dev/ttyS0', os.O_RDWR)

    signal.alarm(0)          # Disable the alarm

5.17.5 `popen2' -- Subprocesses with accessible I/O streams
-----------------------------------------------------------

Deprecated since version 2.6: This module is obsolete.  Use the
subprocess module.  Check especially the Replacing Older Functions with the subprocess Module
section.

  This module allows you to spawn processes and connect to their
input/output/error pipes and obtain their return codes under Unix and
Windows.

  The subprocess module provides more powerful facilities
for spawning new processes and retrieving their results.  Using the
subprocess module is preferable to using the popen2
module.

  The primary interface offered by this module is a trio of factory
functions.  For each of these, if _bufsize_ is specified,  it specifies
the buffer size for the I/O pipes.  _mode_, if provided, should be the
string `'b'' or `'t''; on Windows this is needed to determine whether
the file objects should be opened in binary or text mode.  The default
value for _mode_ is `'t''.

  On Unix, _cmd_ may be a sequence, in which case arguments will be
passed directly to the program without shell intervention (as with
os.spawnv()).  If _cmd_ is a string it will be passed to
the shell (as with os.system()).

  The only way to retrieve the return codes for the child processes is
by using the `poll()' or `wait()' methods on the Popen3 and
Popen4 classes; these are only available on Unix.  This
information is not available when using the popen2(),
popen3(), and popen4().  functions, or the equivalent
functions in the os module. (Note that the tuples returned
by the os module's functions are in a different order from
the ones returned by the popen2 module.)

 -- Function: popen2.popen2 (cmd[, bufsize[, mode]])
     Executes _cmd_ as a sub-process.  Returns the file objects
     `(child_stdout, child_stdin)'.

 -- Function: popen2.popen3 (cmd[, bufsize[, mode]])
     Executes _cmd_ as a sub-process.  Returns the file objects
     `(child_stdout, child_stdin, child_stderr)'.

 -- Function: popen2.popen4 (cmd[, bufsize[, mode]])
     Executes _cmd_ as a sub-process.  Returns the file objects
     `(child_stdout_and_stderr, child_stdin)'.

     New in version 2.0.

  On Unix, a class defining the objects returned by the factory
functions is also available.  These are not used for the Windows
implementation, and are not available on that platform.

 -- Class: popen2.Popen3 (cmd[, capturestderr[, bufsize]])
     This class represents a child process.  Normally, see Popen3
     instances are created using the popen2() and
     popen3() factory functions described above.

     If not using one of the helper functions to create Popen3
     objects, the parameter _cmd_ is the shell command to execute
     in a sub-process.  The _capturestderr_ flag, if true, specifies
     that the object should capture standard error output of the child
     process. The default is false.  If the _bufsize_ parameter is
     specified, it specifies the size of the I/O buffers to/from the
     child process.

 -- Class: popen2.Popen4 (cmd[, bufsize])
     Similar to Popen3, but always captures standard error
     into the same file object as standard output.  These are typically
     created using popen4().

     New in version 2.0.

5.17.5.1 Popen3 and Popen4 Objects
..................................

Instances of the Popen3 and Popen4 classes
have the following methods:

 -- Method: Popen3.poll ()
     Returns `-1' if child process hasn't completed yet, or its status
     code (see wait()) otherwise.

 -- Method: Popen3.wait ()
     Waits for and returns the status code of the child process.  The
     status code encodes both the return code of the process and
     information about whether it exited using the `exit()' system call
     or died due to a signal.  Functions to help interpret the status
     code are defined in the os module; see section
     Process Management for the `W*()' family of functions.

  The following attributes are also available:

 -- Attribute: Popen3.fromchild
     A file object that provides output from the child process.  For
     Popen4.  instances, this will provide both the
     standard output and standard error streams.

 -- Attribute: Popen3.tochild
     A file object that provides input to the child process.

 -- Attribute: Popen3.childerr
     A file object that provides error output from the child process, if
     _capturestderr_ was true for the constructor, otherwise `None'.
     This will always be `None' for Popen4 instances.

 -- Attribute: Popen3.pid
     The process ID of the child process.

5.17.5.2 Flow Control Issues
............................

Any time you are working with any form of inter-process communication,
control flow needs to be carefully thought out.  This remains the case
with the file objects provided by this module (or the os.
module equivalents).

  When reading output from a child process that writes a lot of data to
standard error while the parent is reading from the child's standard
output, a deadlock can occur.  A similar situation can occur with other
combinations of reads and writes.  The essential factors are that more
than `_PC_PIPE_BUF' bytes are being written by one process in a
blocking fashion, while the other process is reading from the first
process, also in a blocking fashion.

  There are several ways to deal with this situation.

  The simplest application change, in many cases, will be to follow
this model in the parent process:

    import popen2

    r, w, e = popen2.popen3('python slave.py')
    e.readlines()
    r.readlines()
    r.close()
    e.close()
    w.close()

with code like this in the child:

    import os
    import sys

    # note that each of these print statements
    # writes a single long string

    print >>sys.stderr, 400 * 'this is a test\n'
    os.close(sys.stderr.fileno())
    print >>sys.stdout, 400 * 'this is another test\n'

In particular, note that `sys.stderr' must be closed after writing all
data, or `readlines()' won't return.  Also note that os.close()
must be used, as `sys.stderr.close()' won't close `stderr'
(otherwise assigning to `sys.stderr' will silently close it, so no
further errors can be printed).

  Applications which need to support a more general approach should
integrate I/O over pipes with their select() loops, or use
separate threads to read each of the individual files provided by
whichever `popen*()' function or `Popen*' class was used.

Module subprocess.
     Module for spawning and managing subprocesses.

5.17.6 `asyncore' -- Asynchronous socket handler
------------------------------------------------

This module provides the basic infrastructure for writing asynchronous
socket service clients and servers.

  There are only two ways to have a program on a single processor do
"more than one thing at a time." Multi-threaded programming is the
simplest and most popular way to do it, but there is another very
different technique, that lets you have nearly all the advantages of
multi-threading, without actually using multiple threads.  It's really
only practical if your program is largely I/O bound.  If your program
is processor bound, then pre-emptive scheduled threads are probably
what you really need.  Network servers are rarely processor bound,
however.

  If your operating system supports the `select()' system call in its
I/O library (and nearly all do), then you can use it to juggle multiple
communication channels at once; doing other work while your I/O is
taking place in the "background."  Although this strategy can seem
strange and complex, especially at first, it is in many ways easier to
understand and control than multi-threaded programming.  The
asyncore module solves many of the difficult problems for you,
making the task of building sophisticated high-performance network
servers and clients a snap.  For "conversational" applications and
protocols the companion asynchat.  module is invaluable.

  The basic idea behind both modules is to create one or more network
_channels_, instances of class asyncore.dispatcher and
asynchat.async_chat.  Creating the channels adds them to a
global map, used by the loop() function if you do not
provide it with your own _map_.

  Once the initial channel(s) is(are) created, calling the
loop() function activates channel service, which continues until
the last channel (including any that have been added to the map during
asynchronous service) is closed.

 -- Function: asyncore.loop ([timeout[, use_poll[, map[, count]]]])
     Enter a polling loop that terminates after count passes or all open
     channels have been closed.  All arguments are optional.  The
     _count_ parameter defaults to None, resulting in the loop
     terminating only when all channels have been closed.  The
     _timeout_ argument sets the timeout parameter for the appropriate
     select() or `poll()' call, measured in seconds; the
     default is 30 seconds.  The _use_poll_ parameter, if true,
     indicates that `poll()' should be used in preference to
     select().  (the default is `False').

     The _map_ parameter is a dictionary whose items are the channels
     to watch.  As channels are closed they are deleted from their map.
     If _map_ is omitted, a global map is used. Channels (instances of
     asyncore.dispatcher, asynchat.async_chat.
     and subclasses thereof) can freely be mixed in the map.

 -- Class: asyncore.dispatcher
     The dispatcher class is a thin wrapper around a
     low-level socket object. To make it more useful, it has a few
     methods for event-handling which are called from the asynchronous
     loop.   Otherwise, it can be treated as a normal non-blocking
     socket object.

     The firing of low-level events at certain times or in certain
     connection states tells the asynchronous loop that certain
     higher-level events have taken place.  For example, if we have
     asked for a socket to connect to another host, we know that the
     connection has been made when the socket becomes writable for the
     first time (at this point you know that you may write to it with
     the expectation of success).  The implied higher-level events are:

     Event                      Description
     ------------------------------------------------------------------------
     `handle_connect()'         Implied by the first read or write event
     `handle_close()'           Implied by a read event with no data
                                available
     `handle_accept()'          Implied by a read event on a listening
                                socket

     During asynchronous processing, each mapped channel's
     readable() and writable() methods are used to
     determine whether the channel's socket should be added to the list
     of channels `select()'ed or `poll()'ed for read and write events.

     Thus, the set of channel events is larger than the basic socket
     events.  The full set of methods that can be overridden in your
     subclass follows:

      -- Method: handle_read ()
          Called when the asynchronous loop detects that a `read()'
          call on the channel's socket will succeed.

      -- Method: handle_write ()
          Called when the asynchronous loop detects that a writable
          socket can be written.  Often this method will implement the
          necessary buffering for performance.  For example:

              def handle_write(self):
                  sent = self.send(self.buffer)
                  self.buffer = self.buffer[sent:]

      -- Method: handle_expt ()
          Called when there is out of band (OOB) data for a socket
          connection.  This will almost never happen, as OOB is
          tenuously supported and rarely used.

      -- Method: handle_connect ()
          Called when the active opener's socket actually makes a
          connection.  Might send a "welcome" banner, or initiate a
          protocol negotiation with the remote endpoint, for example.

      -- Method: handle_close ()
          Called when the socket is closed.

      -- Method: handle_error ()
          Called when an exception is raised and not otherwise handled.
          The default version prints a condensed traceback.

      -- Method: handle_accept ()
          Called on listening channels (passive openers) when a
          connection can be established with a new remote endpoint that
          has issued a connect().  call for the local
          endpoint.

      -- Method: readable ()
          Called each time around the asynchronous loop to determine
          whether a channel's socket should be added to the list on
          which read events can occur.  The default method simply
          returns `True', indicating that by default, all channels will
          be interested in read events.

      -- Method: writable ()
          Called each time around the asynchronous loop to determine
          whether a channel's socket should be added to the list on
          which write events can occur.  The default method simply
          returns `True', indicating that by default, all channels will
          be interested in write events.

     In addition, each channel delegates or extends many of the socket
     methods.  Most of these are nearly identical to their socket
     partners.

      -- Method: create_socket (family, type)
          This is identical to the creation of a normal socket, and
          will use the same options for creation.  Refer to the
          socket documentation for information on creating
          sockets.

      -- Method: connect (address)
          As with the normal socket object, _address_ is a tuple with
          the first element the host to connect to, and the second the
          port number.

      -- Method: send (data)
          Send _data_ to the remote end-point of the socket.

      -- Method: recv (buffer_size)
          Read at most _buffer_size_ bytes from the socket's remote
          end-point.  An empty string implies that the channel has been
          closed from the other end.

      -- Method: listen (backlog)
          Listen for connections made to the socket.  The _backlog_
          argument specifies the maximum number of queued connections
          and should be at least 1; the maximum value is
          system-dependent (usually 5).

      -- Method: bind (address)
          Bind the socket to _address_.  The socket must not already be
          bound.  (The format of _address_ depends on the address
          family -- refer to the socket documentation for
          more information.)  To mark the socket as re-usable (setting
          the `SO_REUSEADDR' option), call the dispatcher.
          object's `set_reuse_addr()' method.

      -- Method: accept ()
          Accept a connection.  The socket must be bound to an address
          and listening for connections.  The return value can be
          either `None' or a pair `(conn, address)' where _conn_ is a
          _new_ socket object usable to send and receive data on the
          connection, and _address_ is the address bound to the socket
          on the other end of the connection.  When `None' is returned
          it means the connection didn't take place, in which case the
          server should just ignore this event and keep listening for
          further incoming connections.

      -- Method: close ()
          Close the socket.  All future operations on the socket object
          will fail.  The remote end-point will receive no more data
          (after queued data is flushed).  Sockets are automatically
          closed when they are garbage-collected.

 -- Class: asyncore.dispatcher_with_send
     A dispatcher subclass which adds simple buffered
     output capability, useful for simple clients. For more
     sophisticated usage use asynchat.async_chat.

 -- Class: asyncore.file_dispatcher
     A file_dispatcher takes a file descriptor or file object along
     with an optional map argument and wraps it for use with the
     `poll()' or `loop()' functions.  If provided a file object or
     anything with a `fileno()' method, that method will be called and
     passed to the file_wrapper constructor.
     Availability: UNIX.

 -- Class: asyncore.file_wrapper
     A file_wrapper takes an integer file descriptor and calls
     os.dup() to duplicate the handle so that the original
     handle may be closed independently of the file_wrapper.  This
     class implements sufficient methods to emulate a socket for use by
     the file_dispatcher class.  Availability: UNIX.

5.17.6.1 asyncore Example basic HTTP client
...........................................

Here is a very basic HTTP client that uses the dispatcher.
class to implement its socket handling:

    import asyncore, socket

    class HTTPClient(asyncore.dispatcher):

        def __init__(self, host, path):
            asyncore.dispatcher.__init__(self)
            self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
            self.connect( (host, 80) )
            self.buffer = 'GET %s HTTP/1.0\r\n\r\n' % path

        def handle_connect(self):
            pass

        def handle_close(self):
            self.close()

        def handle_read(self):
            print self.recv(8192)

        def writable(self):
            return (len(self.buffer) > 0)

        def handle_write(self):
            sent = self.send(self.buffer)
            self.buffer = self.buffer[sent:]

    client = HTTPClient('www.python.org', '/')
    asyncore.loop()

5.17.6.2 asyncore Example basic echo server
...........................................

Here is abasic echo server that uses the dispatcher class
to accept connections and dispatches the incoming connections to a
handler:

    import asyncore
    import socket

    class EchoHandler(asyncore.dispatcher_with_send):

        def handle_read(self):
            data = self.recv(8192)
            self.send(data)

    class EchoServer(asyncore.dispatcher):

        def __init__(self, host, port):
            asyncore.dispatcher.__init__(self)
            self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
            self.set_reuse_addr()
            self.bind((host, port))
            self.listen(5)

        def handle_accept(self):
            pair = self.accept()
            if pair is None:
                pass
            else:
                sock, addr = pair
                print 'Incoming connection from %s' % repr(addr)
                handler = EchoHandler(sock)

    server = EchoServer('localhost', 8080)
    asyncore.loop()

5.17.7 `asynchat' -- Asynchronous socket command/response handler
-----------------------------------------------------------------

This module builds on the asyncore infrastructure,
simplifying asynchronous clients and servers and making it easier to
handle protocols whose elements are terminated by arbitrary strings, or
are of variable length.  asynchat defines the abstract class
async_chat that you subclass, providing implementations of
the `collect_incoming_data()' and `found_terminator()' methods. It uses
the same asynchronous loop as asyncore, and the two types of
channel, see asyncore.dispatcher.  and
asynchat.async_chat, can freely be mixed in the channel map.
Typically an asyncore.dispatcher server channel generates
new asynchat.async_chat channel objects as it receives
incoming connection requests.

 -- Class: asynchat.async_chat
     This class is an abstract subclass of asyncore.dispatcher.
     To make practical use of the code you must subclass
     async_chat, providing meaningful
     collect_incoming_data() and found_terminator().
     methods.  The asyncore.dispatcher methods can be
     used, although not all make sense in a message/response context.

     Like asyncore.dispatcher, async_chat.
     defines a set of events that are generated by an analysis of
     socket conditions after a `select()' call. Once the polling loop
     has been started the async_chat object's methods are
     called by the event-processing framework with no action on the
     part of the programmer.

     Two class attributes can be modified, to improve performance, or
     possibly even to conserve memory.

      -- Data: ac_in_buffer_size
          The asynchronous input buffer size (default `4096').

      -- Data: ac_out_buffer_size
          The asynchronous output buffer size (default `4096').

     Unlike asyncore.dispatcher, async_chat.
     allows you to define a first-in-first-out queue (fifo) of
     _producers_. A producer need have only one method, `more()', which
     should return data to be transmitted on the channel.  The producer
     indicates exhaustion (_i.e._ that it contains no more data) by
     having its `more()' method return the empty string. At this point
     the async_chat object removes the producer from the
     fifo and starts using the next producer, if any. When the producer
     fifo is empty the `handle_write()' method does nothing. You use
     the channel object's set_terminator() method to
     describe how to recognize the end of, or an important breakpoint
     in, an incoming transmission from the remote endpoint.

     To build a functioning async_chat subclass your
     input methods collect_incoming_data() and
     found_terminator() must handle the data that the channel
     receives asynchronously. The methods are described below.

 -- Method: async_chat.close_when_done ()
     Pushes a `None' on to the producer fifo. When this producer is
     popped off the fifo it causes the channel to be closed.

 -- Method: async_chat.collect_incoming_data (data)
     Called with _data_ holding an arbitrary amount of received data.
     The default method, which must be overridden, raises a
     NotImplementedError exception.

 -- Method: async_chat.discard_buffers ()
     In emergencies this method will discard any data held in the input
     and/or output buffers and the producer fifo.

 -- Method: async_chat.found_terminator ()
     Called when the incoming data stream  matches the termination
     condition set by set_terminator(). The default method,
     which must be overridden, raises a NotImplementedError.
     exception. The buffered input data should be available via an
     instance attribute.

 -- Method: async_chat.get_terminator ()
     Returns the current terminator for the channel.

 -- Method: async_chat.push (data)
     Pushes data on to the channel's fifo to ensure its transmission.
     This is all you need to do to have the channel write the data out
     to the network, although it is possible to use your own producers
     in more complex schemes to implement encryption and chunking, for
     example.

 -- Method: async_chat.push_with_producer (producer)
     Takes a producer object and adds it to the producer fifo
     associated with the channel.  When all currently-pushed producers
     have been exhausted the channel will consume this producer's data
     by calling its `more()' method and send the data to the remote
     endpoint.

 -- Method: async_chat.set_terminator (term)
     Sets the terminating condition to be recognized on the channel.
     `term' may be any of three types of value, corresponding to three
     different ways to handle incoming protocol data.

     term            Description
     ------------------------------------------------------------------
     _string_        Will call found_terminator() when
                     the string is found in the input stream
     _integer_       Will call found_terminator() when
                     the indicated number of characters have been
                     received
     `None'          The channel continues to collect data forever

     Note that any data following the terminator will be available for
     reading by the channel after found_terminator() is
     called.

5.17.7.1 asynchat - Auxiliary Classes
.....................................

 -- Class: asynchat.fifo ([list=None])
     A fifo holding data which has been pushed by the
     application but not yet popped for writing to the channel.  A
     fifo is a list used to hold data and/or producers
     until they are required.  If the _list_ argument is provided then
     it should contain producers or data items to be written to the
     channel.

      -- Method: is_empty ()
          Returns `True' if and only if the fifo is empty.

      -- Method: first ()
          Returns the least-recently push().ed item from
          the fifo.

      -- Method: push (data)
          Adds the given data (which may be a string or a producer
          object) to the producer fifo.

      -- Method: pop ()
          If the fifo is not empty, returns `True, first()', deleting
          the popped item.  Returns `False, None' for an empty fifo.

5.17.7.2 asynchat Example
.........................

The following partial example shows how HTTP requests can be read with
async_chat.  A web server might create an
`http_request_handler' object for each incoming client connection.
Notice that initially the channel terminator is set to match the blank
line at the end of the HTTP headers, and a flag indicates that the
headers are being read.

  Once the headers have been read, if the request is of type POST
(indicating that further data are present in the input stream) then the
`Content-Length:' header is used to set a numeric terminator to read the
right amount of data from the channel.

  The `handle_request()' method is called once all relevant input has
been marshalled, after setting the channel terminator to `None' to
ensure that any extraneous data sent by the web client are ignored.

    class http_request_handler(asynchat.async_chat):

        def __init__(self, sock, addr, sessions, log):
            asynchat.async_chat.__init__(self, sock=sock)
            self.addr = addr
            self.sessions = sessions
            self.ibuffer = []
            self.obuffer = ""
            self.set_terminator("\r\n\r\n")
            self.reading_headers = True
            self.handling = False
            self.cgi_data = None
            self.log = log

        def collect_incoming_data(self, data):
            """Buffer the data"""
            self.ibuffer.append(data)

        def found_terminator(self):
            if self.reading_headers:
                self.reading_headers = False
                self.parse_headers("".join(self.ibuffer))
                self.ibuffer = []
                if self.op.upper() == "POST":
                    clen = self.headers.getheader("content-length")
                    self.set_terminator(int(clen))
                else:
                    self.handling = True
                    self.set_terminator(None)
                    self.handle_request()
            elif not self.handling:
                self.set_terminator(None) # browsers sometimes over-send
                self.cgi_data = parse(self.headers, "".join(self.ibuffer))
                self.handling = True
                self.ibuffer = []
                self.handle_request()

5.18 Internet Data Handling
===========================

This chapter describes modules which support handling data formats
commonly used on the Internet.

5.18.1 `email' -- An email and MIME handling package
----------------------------------------------------

New in version 2.2.

  The email package is a library for managing email
messages, including MIME and other RFC 2822
(http://tools.ietf.org/html/rfc2822.html)-based message documents.  It
subsumes most of the functionality in several older standard modules
such as rfc822, mimetools, multifile,
and other non-standard packages such as `mimecntl'.  It is specifically
_not_ designed to do any sending of email messages to SMTP ( RFC 2821
(http://tools.ietf.org/html/rfc2821.html)), NNTP, or other servers;
those are functions of modules such as smtplib and
nntplib. The email package attempts to be as
RFC-compliant as possible, supporting in addition to RFC 2822
(http://tools.ietf.org/html/rfc2822.html), such MIME-related RFCs as RFC
2045 (http://tools.ietf.org/html/rfc2045.html), RFC 2046
(http://tools.ietf.org/html/rfc2046.html), RFC 2047
(http://tools.ietf.org/html/rfc2047.html), and RFC 2231
(http://tools.ietf.org/html/rfc2231.html).

  The primary distinguishing feature of the email package is
that it splits the parsing and generating of email messages from the
internal _object model_ representation of email.  Applications using
the email package deal primarily with objects; you can add
sub-objects to messages, remove sub-objects from messages, completely
re-arrange the contents, etc.  There is a separate parser and a
separate generator which handles the transformation from flat text to
the object model, and then back to flat text again.  There are also
handy subclasses for some common MIME object types, and a few
miscellaneous utilities that help with such common tasks as extracting
and parsing message field values, creating RFC-compliant dates, etc.

  The following sections describe the functionality of the email
package.  The ordering follows a progression that should be common
in applications: an email message is read as flat text from a file or
other source, the text is parsed to produce the object structure of the
email message, this structure is manipulated, and finally, the object
tree is rendered back into flat text.

  It is perfectly feasible to create the object structure out of whole
cloth -- i.e. completely from scratch.  From there, a similar
progression can be taken as above.

  Also included are detailed specifications of all the classes and
modules that the email package provides, the exception
classes you might encounter while using the email package,
some auxiliary utilities, and a few examples.  For users of the older
`mimelib' package, or previous versions of the email.
package, a section on differences and porting is provided.

  Contents of the email package documentation:

5.18.1.1 `email': Representing an email message
...............................................

The central class in the email package is the Message
class, imported from the email.message module.  It is
the base class for the email object model.  Message
provides the core functionality for setting and querying header
fields, and for accessing message bodies.

  Conceptually, a Message object consists of _headers_ and
_payloads_.  Headers are RFC 2822
(http://tools.ietf.org/html/rfc2822.html) style field names and values
where the field name and value are separated by a colon.  The colon is
not part of either the field name or the field value.

  Headers are stored and returned in case-preserving form but are
matched case-insensitively.  There may also be a single envelope
header, also known as the _Unix-From_ header or the `From_' header.
The payload is either a string in the case of simple message objects or
a list of Message objects for MIME container documents
(e.g. `multipart/*' and `message/rfc822').

  Message objects provide a mapping style interface for
accessing the message headers, and an explicit interface for accessing
both the headers and the payload.  It provides convenience methods for
generating a flat text representation of the message object tree, for
accessing commonly used header parameters, and for recursively walking
over the object tree.

  Here are the methods of the Message class:

 -- Class: email.message.Message
     The constructor takes no arguments.

      -- Method: as_string ([unixfrom])
          Return the entire message flattened as a string.  When
          optional _unixfrom_ is `True', the envelope header is
          included in the returned string.  _unixfrom_ defaults to
          `False'.  Flattening the message may trigger changes to the
          Message if defaults need to be filled in to
          complete the transformation to a string (for example, MIME
          boundaries may be generated or modified).

          Note that this method is provided as a convenience and may
          not always format the message the way you want.  For example,
          by default it mangles lines that begin with `From'.  For more
          flexibility, instantiate a Generator instance
          and use its `flatten()' method directly.  For example:

              from cStringIO import StringIO
              from email.generator import Generator
              fp = StringIO()
              g = Generator(fp, mangle_from_=False, maxheaderlen=60)
              g.flatten(msg)
              text = fp.getvalue()

      -- Method: __str__ ()
          Equivalent to `as_string(unixfrom=True)'.

      -- Method: is_multipart ()
          Return `True' if the message's payload is a list of sub-see
          Message objects, otherwise return `False'.  When
          is_multipart() returns False, the payload should be a
          string object.

      -- Method: set_unixfrom (unixfrom)
          Set the message's envelope header to _unixfrom_, which should
          be a string.

      -- Method: get_unixfrom ()
          Return the message's envelope header.  Defaults to `None' if
          the envelope header was never set.

      -- Method: attach (payload)
          Add the given _payload_ to the current payload, which must be
          `None' or a list of Message objects before the
          call. After the call, the payload will always be a list of
          Message objects.  If you want to set the payload
          to a scalar object (e.g. a string), use set_payload()
          instead.

      -- Method: get_payload ([i[, decode]])
          Return the current payload, which will be a list of
          Message objects when is_multipart() is
          `True', or a string when is_multipart() is
          `False'.  If the payload is a list and you mutate the list
          object, you modify the message's payload in place.

          With optional argument _i_, see get_payload() will
          return the _i_-th element of the payload, counting from zero,
          if is_multipart() is `True'.  An
          IndexError will be raised if _i_ is less than 0 or
          greater than or equal to the number of items in the payload.
          If the payload is a string (i.e.  see is_multipart().
          is `False') and _i_ is given, a TypeError is
          raised.

          Optional _decode_ is a flag indicating whether the payload
          should be decoded or not, according to the
          `Content-Transfer-Encoding' header. When `True' and the
          message is not a multipart, the payload will be decoded if
          this header's value is `quoted-printable' or `base64'.  If
          some other encoding is used, or `Content-Transfer-Encoding'
          header is missing, or if the payload has bogus base64 data,
          the payload is returned as-is (undecoded).  If the message is
          a multipart and the _decode_ flag is `True', then `None' is
          returned.  The default for _decode_ is `False'.

      -- Method: set_payload (payload[, charset])
          Set the entire message object's payload to _payload_.  It is
          the client's responsibility to ensure the payload invariants.
          Optional _charset_ sets the message's default character set;
          see set_charset() for details.

          Changed in version 2.2.2: _charset_ argument added.

      -- Method: set_charset (charset)
          Set the character set of the payload to _charset_, which can
          either be a Charset instance (see
          email.charset), a string naming a character set, or
          `None'.  If it is a string, it will be converted to a
          Charset instance.  If _charset_ is `None', the
          `charset' parameter will be removed from the `Content-Type'
          header. Anything else will generate a TypeError.

          The message will be assumed to be of type `text/*', with the
          payload either in unicode or encoded with
          _charset.input_charset_.  It will be encoded or converted to
          _charset.output_charset_ and transfer encoded properly, if
          needed, when generating the plain text representation of the
          message.  MIME headers (`MIME-Version', `Content-Type',
          `Content-Transfer-Encoding') will be added as needed.

          New in version 2.2.2.

      -- Method: get_charset ()
          Return the Charset instance associated with the
          message's payload.

          New in version 2.2.2.

     The following methods implement a mapping-like interface for
     accessing the message's RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) headers.  Note that
     there are some semantic differences between these methods and a
     normal mapping (i.e. dictionary) interface.  For example, in a
     dictionary there are no duplicate keys, but here there may be
     duplicate message headers.  Also, in dictionaries there is no
     guaranteed order to the keys returned by keys(), but
     in a Message object, headers are always returned in
     the order they appeared in the original message, or were added to
     the message later.  Any header deleted and then re-added are
     always appended to the end of the header list.

     These semantic differences are intentional and are biased toward
     maximal convenience.

     Note that in all cases, any envelope header present in the message
     is not included in the mapping interface.

      -- Method: __len__ ()
          Return the total number of headers, including duplicates.

      -- Method: __contains__ (name)
          Return true if the message object has a field named _name_.
          Matching is done case-insensitively and _name_ should not
          include the trailing colon.  Used for the `in' operator, e.g.:

              if 'message-id' in myMessage:
                  print 'Message-ID:', myMessage['message-id']

      -- Method: __getitem__ (name)
          Return the value of the named header field.  _name_ should
          not include the colon field separator.  If the header is
          missing, `None' is returned; a KeyError is never
          raised.

          Note that if the named field appears more than once in the
          message's headers, exactly which of those field values will
          be returned is undefined.  Use the get_all().
          method to get the values of all the extant named headers.

      -- Method: __setitem__ (name, val)
          Add a header to the message with field name _name_ and value
          _val_.  The field is appended to the end of the message's
          existing fields.

          Note that this does _not_ overwrite or delete any existing
          header with the same name.  If you want to ensure that the
          new header is the only one present in the message with field
          name _name_, delete the field first, e.g.:

              del msg['subject']
              msg['subject'] = 'Python roolz!'

      -- Method: __delitem__ (name)
          Delete all occurrences of the field with name _name_ from the
          message's headers.  No exception is raised if the named field
          isn't present in the headers.

      -- Method: has_key (name)
          Return true if the message contains a header field named
          _name_, otherwise return false.

      -- Method: keys ()
          Return a list of all the message's header field names.

      -- Method: values ()
          Return a list of all the message's field values.

      -- Method: items ()
          Return a list of 2-tuples containing all the message's field
          headers and values.

      -- Method: get (name[, failobj])
          Return the value of the named header field.  This is
          identical to __getitem__() except that optional
          _failobj_ is returned if the named header is missing
          (defaults to `None').

     Here are some additional useful methods:

      -- Method: get_all (name[, failobj])
          Return a list of all the values for the field named _name_.
          If there are no such named headers in the message, _failobj_
          is returned (defaults to `None').

      -- Method: add_header (_name, _value, **_params)
          Extended header setting.  This method is similar to
          __setitem__().  except that additional header
          parameters can be provided as keyword arguments.  __name_ is
          the header field to add and __value_ is the _primary_ value
          for the header.

          For each item in the keyword argument dictionary __params_,
          the key is taken as the parameter name, with underscores
          converted to dashes (since dashes are illegal in Python
          identifiers).  Normally, the parameter will be added as
          `key="value"' unless the value is `None', in which case only
          the key will be added.

          Here's an example:

              msg.add_header('Content-Disposition', 'attachment', filename='bud.gif')

          This will add a header that looks like

              Content-Disposition: attachment; filename="bud.gif"

      -- Method: replace_header (_name, _value)
          Replace a header.  Replace the first header found in the
          message that matches __name_, retaining header order and
          field name case.  If no matching header was found, a
          KeyError is raised.

          New in version 2.2.2.

      -- Method: get_content_type ()
          Return the message's content type.  The returned string is
          coerced to lower case of the form `maintype/subtype'.  If
          there was no `Content-Type' header in the message the default
          type as given by get_default_type() will be
          returned.  Since according to RFC 2045
          (http://tools.ietf.org/html/rfc2045.html), messages always
          have a default type, see get_content_type().  will
          always return a value.

          RFC 2045 (http://tools.ietf.org/html/rfc2045.html) defines a
          message's default type to be `text/plain' unless it appears
          inside a `multipart/digest' container, in which case it would
          be `message/rfc822'.  If the `Content-Type' header has an
          invalid type specification, RFC 2045
          (http://tools.ietf.org/html/rfc2045.html) mandates that the
          default type be `text/plain'.

          New in version 2.2.2.

      -- Method: get_content_maintype ()
          Return the message's main content type.  This is the
          `maintype' part of the string returned by
          get_content_type().

          New in version 2.2.2.

      -- Method: get_content_subtype ()
          Return the message's sub-content type.  This is the `subtype'
          part of the string returned by get_content_type().

          New in version 2.2.2.

      -- Method: get_default_type ()
          Return the default content type.  Most messages have a
          default content type of `text/plain', except for messages
          that are subparts of `multipart/digest' containers.  Such
          subparts have a default content type of `message/rfc822'.

          New in version 2.2.2.

      -- Method: set_default_type (ctype)
          Set the default content type.  _ctype_ should either be
          `text/plain' or `message/rfc822', although this is not
          enforced.  The default content type is not stored in the
          `Content-Type' header.

          New in version 2.2.2.

      -- Method: get_params ([failobj[, header[, unquote]]])
          Return the message's `Content-Type' parameters, as a list.
          The elements of the returned list are 2-tuples of key/value
          pairs, as split on the `'='' sign.  The left hand side of the
          `'='' is the key, while the right hand side is the value.  If
          there is no `'='' sign in the parameter the value is the
          empty string, otherwise the value is as described in
          get_param() and is unquoted if optional _unquote_ is
          `True' (the default).

          Optional _failobj_ is the object to return if there is no
          `Content-Type' header.  Optional _header_ is the header to
          search instead of `Content-Type'.

          Changed in version 2.2.2: _unquote_ argument added.

      -- Method: get_param (param[, failobj[, header[, unquote]]])
          Return the value of the `Content-Type' header's parameter
          _param_ as a string.  If the message has no `Content-Type'
          header or if there is no such parameter, then _failobj_ is
          returned (defaults to `None').

          Optional _header_ if given, specifies the message header to
          use instead of `Content-Type'.

          Parameter keys are always compared case insensitively.  The
          return value can either be a string, or a 3-tuple if the
          parameter was RFC 2231
          (http://tools.ietf.org/html/rfc2231.html) encoded.  When it's
          a 3-tuple, the elements of the value are of the form
          `(CHARSET, LANGUAGE, VALUE)'.  Note that both `CHARSET' and
          `LANGUAGE' can be `None', in which case you should consider
          `VALUE' to be encoded in the `us-ascii' charset.  You can
          usually ignore `LANGUAGE'.

          If your application doesn't care whether the parameter was
          encoded as in RFC 2231
          (http://tools.ietf.org/html/rfc2231.html), you can collapse
          the parameter value by calling
          email.utils.collapse_rfc2231_value(), passing in the
          return value from get_param().  This will return
          a suitably decoded Unicode string whn the value is a tuple,
          or the original string unquoted if it isn't.  For example:

              rawparam = msg.get_param('foo')
              param = email.utils.collapse_rfc2231_value(rawparam)

          In any case, the parameter value (either the returned string,
          or the `VALUE' item in the 3-tuple) is always unquoted,
          unless _unquote_ is set to `False'.

          Changed in version 2.2.2: _unquote_ argument added, and
          3-tuple return value possible.

      -- Method: set_param (param, value[, header[, requote[, charset[,
               language]]]])
          Set a parameter in the `Content-Type' header.  If the
          parameter already exists in the header, its value will be
          replaced with _value_.  If the `Content-Type' header as not
          yet been defined for this message, it will be set to
          `text/plain' and the new parameter value will be appended as
          per RFC 2045 (http://tools.ietf.org/html/rfc2045.html).

          Optional _header_ specifies an alternative header to
          `Content-Type', and all parameters will be quoted as necessary
          unless optional _requote_ is `False' (the default is `True').

          If optional _charset_ is specified, the parameter will be
          encoded according to RFC 2231
          (http://tools.ietf.org/html/rfc2231.html). Optional
          _language_ specifies the RFC 2231 language, defaulting to the
          empty string.  Both _charset_ and _language_ should be
          strings.

          New in version 2.2.2.

      -- Method: del_param (param[, header[, requote]])
          Remove the given parameter completely from the `Content-Type'
          header.  The header will be re-written in place without the
          parameter or its value.  All values will be quoted as
          necessary unless _requote_ is `False' (the default is
          `True').  Optional _header_ specifies an alternative to
          `Content-Type'.

          New in version 2.2.2.

      -- Method: set_type (type[, header][, requote])
          Set the main type and subtype for the `Content-Type' header.
          _type_ must be a string in the form `maintype/subtype',
          otherwise a ValueError is raised.

          This method replaces the `Content-Type' header, keeping all
          the parameters in place.  If _requote_ is `False', this
          leaves the existing header's quoting as is, otherwise the
          parameters will be quoted (the default).

          An alternative header can be specified in the _header_
          argument. When the `Content-Type' header is set a
          `MIME-Version' header is also added.

          New in version 2.2.2.

      -- Method: get_filename ([failobj])
          Return the value of the `filename' parameter of the
          `Content-Disposition' header of the message.  If the header
          does not have a `filename' parameter, this method falls back
          to looking for the `name' parameter on the `Content-Type'
          header.  If neither is found, or the header is missing, then
          _failobj_ is returned.  The returned string will always be
          unquoted as per email.utils.unquote().

      -- Method: get_boundary ([failobj])
          Return the value of the `boundary' parameter of the
          `Content-Type' header of the message, or _failobj_ if either
          the header is missing, or has no `boundary' parameter.  The
          returned string will always be unquoted as per
          email.utils.unquote().

      -- Method: set_boundary (boundary)
          Set the `boundary' parameter of the `Content-Type' header to
          _boundary_.  set_boundary() will always quote
          _boundary_ if necessary.  A `HeaderParseError' is raised if
          the message object has no `Content-Type' header.

          Note that using this method is subtly different than deleting
          the old `Content-Type' header and adding a new one with the
          new boundary via add_header(), because
          set_boundary() preserves the order of the
          `Content-Type' header in the list of headers. However, it
          does _not_ preserve any continuation lines which may have
          been present in the original `Content-Type' header.

      -- Method: get_content_charset ([failobj])
          Return the `charset' parameter of the `Content-Type' header,
          coerced to lower case.  If there is no `Content-Type' header,
          or if that header has no `charset' parameter, _failobj_ is
          returned.

          Note that this method differs from get_charset().
          which returns the Charset instance for the
          default encoding of the message body.

          New in version 2.2.2.

      -- Method: get_charsets ([failobj])
          Return a list containing the character set names in the
          message.  If the message is a `multipart', then the list will
          contain one element for each subpart in the payload,
          otherwise, it will be a list of length 1.

          Each item in the list will be a string which is the value of
          the `charset' parameter in the `Content-Type' header for the
          represented subpart.  However, if the subpart has no
          `Content-Type' header, no `charset' parameter, or is not of
          the `text' main MIME type, then that item in the returned list
          will be _failobj_.

      -- Method: walk ()
          The walk() method is an all-purpose generator
          which can be used to iterate over all the parts and subparts
          of a message object tree, in depth-first traversal order.
          You will typically use walk() as the iterator in
          a `for' loop; each iteration returns the next subpart.

          Here's an example that prints the MIME type of every part of
          a multipart message structure:

              >>> for part in msg.walk():
              ...     print part.get_content_type()
              multipart/report
              text/plain
              message/delivery-status
              text/plain
              text/plain
              message/rfc822

     Changed in version 2.5: The previously deprecated methods
     `get_type()', `get_main_type()', and `get_subtype()' were removed.

     Message objects can also optionally contain two
     instance attributes, which can be used when generating the plain
     text of a MIME message.

      -- Attribute: preamble
          The format of a MIME document allows for some text between
          the blank line following the headers, and the first multipart
          boundary string. Normally, this text is never visible in a
          MIME-aware mail reader because it falls outside the standard
          MIME armor.  However, when viewing the raw text of the
          message, or when viewing the message in a non-MIME aware
          reader, this text can become visible.

          The _preamble_ attribute contains this leading extra-armor
          text for MIME documents.  When the Parser.
          discovers some text after the headers but before the first
          boundary string, it assigns this text to the message's
          _preamble_ attribute.  When the Generator is
          writing out the plain text representation of a MIME message,
          and it finds the message has a _preamble_ attribute, it will
          write this text in the area between the headers and the first
          boundary.  See email.parser and
          email.generator for details.

          Note that if the message object has no preamble, the
          _preamble_ attribute will be `None'.

      -- Attribute: epilogue
          The _epilogue_ attribute acts the same way as the _preamble_
          attribute, except that it contains text that appears between
          the last boundary and the end of the message.

          Changed in version 2.5: You do not need to set the epilogue
          to the empty string in order for the `Generator' to print a
          newline at the end of the file.

      -- Attribute: defects
          The _defects_ attribute contains a list of all the problems
          found when parsing this message.  See email.errors.
          for a detailed description of the possible parsing defects.

          New in version 2.4.

5.18.1.2 `email': Parsing email messages
........................................

Message object structures can be created in one of two ways: they can
be created from whole cloth by instantiating Message.
objects and stringing them together via `attach()' and `set_payload()'
calls, or they can be created by parsing a flat text representation of
the email message.

  The email package provides a standard parser that
understands most email document structures, including MIME documents.
You can pass the parser a string or a file object, and the parser will
return to you the root Message instance of the object
structure.  For simple, non-MIME messages the payload of this root
object will likely be a string containing the text of the message.  For
MIME messages, the root object will return `True' from its
`is_multipart()' method, and the subparts can be accessed via the
`get_payload()' and `walk()' methods.

  There are actually two parser interfaces available for use, the
classic Parser. API and the incremental FeedParser.
API.  The classic Parser. API is fine if you have the
entire text of the message in memory as a string, or if the entire
message lives in a file on the file system.  FeedParser is
more appropriate for when you're reading the message from a stream
which might block waiting for more input (e.g. reading an email message
from a socket).  The FeedParser can consume and parse the
message incrementally, and only returns the root object when you close
the parser (1).

  Note that the parser can be extended in limited ways, and of course
you can implement your own parser completely from scratch.  There is no
magical connection between the email package's bundled
parser and the Message class, so your custom parser can
create message object trees any way it finds necessary.

5.18.1.3 FeedParser API
.......................

New in version 2.4.

  The FeedParser, imported from the `email.feedparser'
module, provides an API that is conducive to incremental parsing of
email messages, such as would be necessary when reading the text of an
email message from a source that can block (e.g. a socket).  The
FeedParser can of course be used to parse an email message fully
contained in a string or a file, but the classic Parser.
API may be more convenient for such use cases.  The semantics and
results of the two parser APIs are identical.

  The FeedParser.'s API is simple; you create an instance,
feed it a bunch of text until there's no more to feed it, then close
the parser to retrieve the root message object.  The FeedParser
is extremely accurate when parsing standards-compliant messages,
and it does a very good job of parsing non-compliant messages,
providing information about how a message was deemed broken.  It will
populate a message object's _defects_ attribute with a list of any
problems it found in a message.  See the email.errors module
for the list of defects that it can find.

  Here is the API for the FeedParser.:

 -- Class: email.parser.FeedParser ([_factory])
     Create a FeedParser instance.  Optional __factory_ is
     a no-argument callable that will be called whenever a new message
     object is needed.  It defaults to the email.message.Message
     class.

      -- Method: feed (data)
          Feed the FeedParser some more data.  _data_
          should be a string containing one or more lines.  The lines
          can be partial and the FeedParser will stitch
          such partial lines together properly.  The lines in the
          string can have any of the common three line endings,
          carriage return, newline, or carriage return and newline
          (they can even be mixed).

      -- Method: close ()
          Closing a FeedParser completes the parsing of
          all previously fed data, and returns the root message object.
          It is undefined what happens if you feed more data to a
          closed FeedParser.

5.18.1.4 Parser class API
.........................

The Parser class, imported from the email.parser
module, provides an API that can be used to parse a message when
the complete contents of the message are available in a string or file.
The email.parser.  module also provides a second class,
called `HeaderParser' which can be used if you're only interested in
the headers of the message.  `HeaderParser' can be much faster in these
situations, since it does not attempt to parse the message body,
instead setting the payload to the raw body as a string. `HeaderParser'
has the same API as the Parser.  class.

 -- Class: email.parser.Parser ([_class])
     The constructor for the Parser class takes an
     optional argument __class_.  This must be a callable factory (such
     as a function or a class), and it is used whenever a sub-message
     object needs to be created.  It defaults to Message.
     (see email.message).  The factory will be called
     without arguments.

     The optional _strict_ flag is ignored.

     Deprecated since version 2.4: Because the Parser.
     class is a backward compatible API wrapper around the
     new-in-Python 2.4 FeedParser, _all_ parsing is
     effectively non-strict.  You should simply stop passing a _strict_
     flag to the Parser constructor.

     Changed in version 2.2.2: The _strict_ flag was added.

     Changed in version 2.4: The _strict_ flag was deprecated.

     The other public Parser methods are:

      -- Method: parse (fp[, headersonly])
          Read all the data from the file-like object _fp_, parse the
          resulting text, and return the root message object.  _fp_
          must support both the readline() and the `read()'
          methods on file-like objects.

          The text contained in _fp_ must be formatted as a block of RFC
          2822 (http://tools.ietf.org/html/rfc2822.html) style headers
          and header continuation lines, optionally preceded by a
          envelope header.  The header block is terminated either by
          the end of the data or by a blank line.  Following the header
          block is the body of the message (which may contain
          MIME-encoded subparts).

          Optional _headersonly_ is as with the parse().
          method.

          Changed in version 2.2.2: The _headersonly_ flag was added.

      -- Method: parsestr (text[, headersonly])
          Similar to the parse() method, except it takes a
          string object instead of a file-like object.  Calling this
          method on a string is exactly equivalent to wrapping _text_
          in a StringIO instance first and calling
          parse().

          Optional _headersonly_ is a flag specifying whether to stop
          parsing after reading the headers or not.  The default is
          `False', meaning it parses the entire contents of the file.

          Changed in version 2.2.2: The _headersonly_ flag was added.

  Since creating a message object structure from a string or a file
object is such a common task, two functions are provided as a
convenience.  They are available in the top-level email.
package namespace.

 -- Function: email.message_from_string (s[, _class[, strict]])
     Return a message object structure from a string.  This is exactly
     equivalent to `Parser().parsestr(s)'.  Optional __class_ and
     _strict_ are interpreted as with the `Parser' class constructor.

     Changed in version 2.2.2: The _strict_ flag was added.

 -- Function: email.message_from_file (fp[, _class[, strict]])
     Return a message object structure tree from an open file object.
     This is exactly equivalent to `Parser().parse(fp)'.  Optional
     __class_ and _strict_ are interpreted as with the `Parser' class
     constructor.

     Changed in version 2.2.2: The _strict_ flag was added.

  Here's an example of how you might use this at an interactive Python
prompt:

    >>> import email
    >>> msg = email.message_from_string(myString)

5.18.1.5 Additional notes
.........................

Here are some notes on the parsing semantics:

   * Most non-`multipart' type messages are parsed as a single message
     object with a string payload.  These objects will return `False'
     for `is_multipart()'.  Their `get_payload()' method will return a
     string object.

   * All `multipart' type messages will be parsed as a container message
     object with a list of sub-message objects for their payload.  The
     outer container message will return `True' for `is_multipart()'
     and their `get_payload()' method will return the list of
     Message.  subparts.

   * Most messages with a content type of `message/*' (e.g.
     `message/delivery-status' and `message/rfc822') will also be
     parsed as container object containing a list payload of length 1.
     Their `is_multipart()' method will return `True'.  The single
     element in the list payload will be a sub-message object.

   * Some non-standards compliant messages may not be internally
     consistent about their `multipart'-edness.  Such messages may have
     a `Content-Type' header of type `multipart', but their
     `is_multipart()' method may return `False'.  If such messages were
     parsed with the `FeedParser', they will have an instance of the
     `MultipartInvariantViolationDefect' class in their _defects_
     attribute list.  See email.errors for details.

Footnotes
.........

     (1) As of email package version 3.0, introduced in Python 2.4, the
     classic `Parser' was re-implemented in terms of the `FeedParser',
     so the semantics and results are identical between the two parsers.

5.18.1.6 `email': Generating MIME documents
...........................................

One of the most common tasks is to generate the flat text of the email
message represented by a message object structure.  You will need to do
this if you want to send your message via the smtplib.
module or the nntplib module, or print the message on the
console.  Taking a message object structure and producing a flat text
document is the job of the Generator class.

  Again, as with the email.parser module, you aren't limited
to the functionality of the bundled generator; you could write one from
scratch yourself.  However the bundled generator knows how to generate
most email in a standards-compliant way, should handle MIME and
non-MIME email messages just fine, and is designed so that the
transformation from flat text, to a message structure via the
Parser class, and back to flat text, is idempotent (the input is
identical to the output).  On the other hand, using the Generator on a
Message constructed by program may result in changes to the
Message object as defaults are filled in.

  Here are the public methods of the Generator class,
imported from the email.generator module:

 -- Class: email.generator.Generator (outfp[, mangle_from_[,
          maxheaderlen]])
     The constructor for the Generator class takes a
     file-like object called _outfp_ for an argument.  _outfp_ must
     support the write() method and be usable as the
     output file in a Python extended print statement.

     Optional _mangle_from__ is a flag that, when `True', puts a `>'
     character in front of any line in the body that starts exactly as
     `From', i.e. `From' followed by a space at the beginning of the
     line.  This is the only guaranteed portable way to avoid having
     such lines be mistaken for a Unix mailbox format envelope header
     separator (see WHY THE CONTENT-LENGTH FORMAT IS BAD
     (http://www.jwz.org/doc/content-length.html)for details).
     _mangle_from__ defaults to `True', but you might want to set this
     to `False' if you are not writing Unix mailbox format files.

     Optional _maxheaderlen_ specifies the longest length for a
     non-continued header.  When a header line is longer than
     _maxheaderlen_ (in characters, with tabs expanded to 8 spaces),
     the header will be split as defined in the Header.
     class.  Set to zero to disable header wrapping.  The default is
     78, as recommended (but not required) by RFC 2822
     (http://tools.ietf.org/html/rfc2822.html).

     The other public Generator methods are:

      -- Method: flatten (msg[, unixfrom])
          Print the textual representation of the message object
          structure rooted at _msg_ to the output file specified when
          the Generator instance was created.  Subparts
          are visited depth-first and the resulting text will be
          properly MIME encoded.

          Optional _unixfrom_ is a flag that forces the printing of the
          envelope header delimiter before the first RFC 2822
          (http://tools.ietf.org/html/rfc2822.html) header of the root
          message object.  If the root object has no envelope header, a
          standard one is crafted.  By default, this is set to `False'
          to inhibit the printing of the envelope delimiter.

          Note that for subparts, no envelope header is ever printed.

          New in version 2.2.2.

      -- Method: clone (fp)
          Return an independent clone of this Generator.
          instance with the exact same options.

          New in version 2.2.2.

      -- Method: write (s)
          Write the string _s_ to the underlying file object, i.e.
          _outfp_ passed to Generator.'s constructor.  This
          provides just enough file-like API for Generator.
          instances to be used in extended print statements.

  As a convenience, see the methods `Message.as_string()' and
`str(aMessage)', a.k.a. `Message.__str__()', which simplify the
generation of a formatted string representation of a message object.
For more detail, see email.message.

  The email.generator module also provides a derived class,
called DecodedGenerator which is like the Generator
base class, except that non-`text' parts are substituted with a
format string representing the part.

 -- Class: email.generator.DecodedGenerator (outfp[, mangle_from_[,
          maxheaderlen[, fmt]]])
     This class, derived from Generator walks through all
     the subparts of a message.  If the subpart is of main type `text',
     then it prints the decoded payload of the subpart. Optional
     __mangle_from__ and _maxheaderlen_ are as with the
     Generator base class.

     If the subpart is not of main type `text', optional _fmt_ is a
     format string that is used instead of the message payload. _fmt_
     is expanded with the following keywords, `%(keyword)s' format:

        * `type' - Full MIME type of the non-`text' part

        * `maintype' - Main MIME type of the non-`text' part

        * `subtype' - Sub-MIME type of the non-`text' part

        * `filename' - Filename of the non-`text' part

        * `description' - Description associated with the non-`text'
          part

        * `encoding' - Content transfer encoding of the non-`text' part

     The default value for _fmt_ is `None', meaning

         [Non-text (%(type)s) part of message omitted, filename %(filename)s]

     New in version 2.2.2.

  Changed in version 2.5: The previously deprecated method
__call__() was removed.

5.18.1.7 `email': Creating email and MIME objects from scratch
..............................................................

Ordinarily, you get a message object structure by passing a file or
some text to a parser, which parses the text and returns the root
message object.  However you can also build a complete message
structure from scratch, or even individual Message objects
by hand.  In fact, you can also take an existing structure and add new
Message objects, move them around, etc.  This makes a very
convenient interface for slicing-and-dicing MIME messages.

  You can create a new object structure by creating Message.
instances, adding attachments and all the appropriate headers manually.
For MIME messages though, the email package provides some
convenient subclasses to make things easier.

  Here are the classes:

 -- Class: email.mime.base.MIMEBase (_maintype, _subtype, **_params)
     Module: `email.mime.base'

     This is the base class for all the MIME-specific subclasses of
     Message.  Ordinarily you won't create instances
     specifically of MIMEBase, although you could.  See
     MIMEBase.  is provided primarily as a convenient base class
     for more specific MIME-aware subclasses.

     __maintype_ is the `Content-Type' major type (e.g. `text' or
     `image'), and __subtype_ is the `Content-Type' minor type  (e.g.
     `plain' or `gif').  __params_ is a parameter key/value dictionary
     and is passed directly to `Message.add_header()'.

     The MIMEBase class always adds a `Content-Type' header
     (based on __maintype_, __subtype_, and __params_), and a
     `MIME-Version' header (always set to `1.0').

 -- Class: email.mime.nonmultipart.MIMENonMultipart
     Module: `email.mime.nonmultipart'

     A subclass of MIMEBase, this is an intermediate base
     class for MIME messages that are not `multipart'.  The primary
     purpose of this class is to prevent the use of the `attach()'
     method, which only makes sense for `multipart' messages.  If
     `attach()' is called, a MultipartConversionError.
     exception is raised.

     New in version 2.2.2.

 -- Class: email.mime.multipart.MIMEMultipart ([_subtype[, boundary[,
          _subparts[, _params]]]])
     Module: `email.mime.multipart'

     A subclass of MIMEBase, this is an intermediate base
     class for MIME messages that are `multipart'.  Optional __subtype_
     defaults to `mixed', but can be used to specify the subtype of the
     message.  A `Content-Type' header of `multipart/_subtype' will be
     added to the message object.  A `MIME-Version' header will also be
     added.

     Optional _boundary_ is the multipart boundary string.  When `None'
     (the default), the boundary is calculated when needed (for
     example, when the message is serialized).

     __subparts_ is a sequence of initial subparts for the payload.  It
     must be possible to convert this sequence to a list.  You can
     always attach new subparts to the message by using the
     `Message.attach()' method.

     Additional parameters for the `Content-Type' header are taken from
     the keyword arguments, or passed into the __params_ argument,
     which is a keyword dictionary.

     New in version 2.2.2.

 -- Class: email.mime.application.MIMEApplication (_data[, _subtype[,
          _encoder[, **_params]]])
     Module: `email.mime.application'

     A subclass of MIMENonMultipart, the
     MIMEApplication class is used to represent MIME message
     objects of major type `application'.  __data_ is a string
     containing the raw byte data.  Optional __subtype_ specifies the
     MIME subtype and defaults to `octet-stream'.

     Optional __encoder_ is a callable (i.e. function) which will
     perform the actual encoding of the data for transport.  This
     callable takes one argument, which is the MIMEApplication
     instance. It should use `get_payload()' and `set_payload()'
     to change the payload to encoded form.  It should also add any
     `Content-Transfer-Encoding' or other headers to the message object
     as necessary.  The default encoding is base64.  See the
     email.encoders module for a list of the built-in encoders.

     __params_ are passed straight through to the base class
     constructor.

     New in version 2.5.

 -- Class: email.mime.audio.MIMEAudio (_audiodata[, _subtype[,
          _encoder[, **_params]]])
     Module: `email.mime.audio'

     A subclass of MIMENonMultipart, the MIMEAudio
     class is used to create MIME message objects of major type
     `audio'. __audiodata_ is a string containing the raw audio data.
     If this data can be decoded by the standard Python module
     sndhdr, then the subtype will be automatically included in
     the `Content-Type' header.  Otherwise you can explicitly specify
     the audio subtype via the __subtype_ parameter.  If the minor type
     could not be guessed and __subtype_ was not given, then
     TypeError is raised.

     Optional __encoder_ is a callable (i.e. function) which will
     perform the actual encoding of the audio data for transport.  This
     callable takes one argument, which is the MIMEAudio.
     instance. It should use `get_payload()' and `set_payload()' to
     change the payload to encoded form.  It should also add any
     `Content-Transfer-Encoding' or other headers to the message object
     as necessary.  The default encoding is base64.  See the
     email.encoders module for a list of the built-in encoders.

     __params_ are passed straight through to the base class
     constructor.

 -- Class: email.mime.image.MIMEImage (_imagedata[, _subtype[,
          _encoder[, **_params]]])
     Module: `email.mime.image'

     A subclass of MIMENonMultipart, the MIMEImage
     class is used to create MIME message objects of major type
     `image'. __imagedata_ is a string containing the raw image data.
     If this data can be decoded by the standard Python module
     imghdr, then the subtype will be automatically included in the
     `Content-Type' header.  Otherwise you can explicitly specify the
     image subtype via the __subtype_ parameter.  If the minor type
     could not be guessed and __subtype_ was not given, then
     TypeError is raised.

     Optional __encoder_ is a callable (i.e. function) which will
     perform the actual encoding of the image data for transport.  This
     callable takes one argument, which is the MIMEImage.
     instance. It should use `get_payload()' and `set_payload()' to
     change the payload to encoded form.  It should also add any
     `Content-Transfer-Encoding' or other headers to the message object
     as necessary.  The default encoding is base64.  See the
     email.encoders module for a list of the built-in encoders.

     __params_ are passed straight through to the MIMEBase.
     constructor.

 -- Class: email.mime.message.MIMEMessage (_msg[, _subtype])
     Module: `email.mime.message'

     A subclass of MIMENonMultipart, the MIMEMessage
     class is used to create MIME objects of main type `message'.
     __msg_ is used as the payload, and must be an instance of class
     Message. (or a subclass thereof), otherwise a
     TypeError is raised.

     Optional __subtype_ sets the subtype of the message; it defaults to
     `rfc822'.

 -- Class: email.mime.text.MIMEText (_text[, _subtype[, _charset]])
     Module: `email.mime.text'

     A subclass of MIMENonMultipart, the MIMEText
     class is used to create MIME objects of major type `text'.
     __text_ is the string for the payload.  __subtype_ is the minor
     type and defaults to `plain'.  __charset_ is the character set of
     the text and is passed as a parameter to the
     MIMENonMultipart constructor; it defaults to `us-ascii'.
     If __text_ is unicode, it is encoded using the _output_charset_ of
     __charset_, otherwise it is used as-is.

     Changed in version 2.4: The previously deprecated __encoding_
     argument has been removed.  Content Transfer Encoding now happens
     happens implicitly based on the __charset_ argument.

5.18.1.8 `email': Internationalized headers
...........................................

RFC 2822 (http://tools.ietf.org/html/rfc2822.html) is the base standard
that describes the format of email messages.  It derives from the older RFC
822 (http://tools.ietf.org/html/rfc822.html) standard which came into
widespread use at a time when most email was composed of ASCII
characters only.  RFC 2822 (http://tools.ietf.org/html/rfc2822.html) is
a specification written assuming email contains only 7-bit ASCII
characters.

  Of course, as email has been deployed worldwide, it has become
internationalized, such that language specific character sets can now
be used in email messages.  The base standard still requires email
messages to be transferred using only 7-bit ASCII characters, so a slew
of RFCs have been written describing how to encode email containing
non-ASCII characters into RFC 2822
(http://tools.ietf.org/html/rfc2822.html)-compliant format. These RFCs
include RFC 2045 (http://tools.ietf.org/html/rfc2045.html), RFC 2046
(http://tools.ietf.org/html/rfc2046.html), RFC 2047
(http://tools.ietf.org/html/rfc2047.html), and RFC 2231
(http://tools.ietf.org/html/rfc2231.html). The email package
supports these standards in its email.header and
email.charset modules.

  If you want to include non-ASCII characters in your email headers,
say in the `Subject' or `To' fields, you should use the Header
class and assign the field in the Message.  object to
an instance of Header instead of using a string for the
header value.  Import the Header class from the
email.header module.  For example:

    >>> from email.message import Message
    >>> from email.header import Header
    >>> msg = Message()
    >>> h = Header('p\xf6stal', 'iso-8859-1')
    >>> msg['Subject'] = h
    >>> print msg.as_string()
    Subject: =?iso-8859-1?q?p=F6stal?=

Notice here how we wanted the `Subject' field to contain a non-ASCII
character?  We did this by creating a Header instance and
passing in the character set that the byte string was encoded in.  When
the subsequent Message instance was flattened, the `Subject'
field was properly RFC 2047 (http://tools.ietf.org/html/rfc2047.html)
encoded.  MIME-aware mail readers would show this header using the
embedded ISO-8859-1 character.

  New in version 2.2.2.

  Here is the Header class description:

 -- Class: email.header.Header ([s[, charset[, maxlinelen[,
          header_name[, continuation_ws[, errors]]]]]])
     Create a MIME-compliant header that can contain strings in
     different character sets.

     Optional _s_ is the initial header value.  If `None' (the
     default), the initial header value is not set.  You can later
     append to the header with append() method calls.  _s_
     may be a byte string or a Unicode string, but see the
     append() documentation for semantics.

     Optional _charset_ serves two purposes: it has the same meaning as
     the _charset_ argument to the append() method.  It
     also sets the default character set for all subsequent
     append() calls that omit the _charset_ argument.  If
     _charset_ is not provided in the constructor (the default), the
     `us-ascii' character set is used both as _s_'s initial charset and
     as the default for subsequent append() calls.

     The maximum line length can be specified explicit via
     _maxlinelen_.  For splitting the first line to a shorter value (to
     account for the field header which isn't included in _s_, e.g.
     `Subject') pass in the name of the field in _header_name_.  The
     default _maxlinelen_ is 76, and the default value for
     _header_name_ is `None', meaning it is not taken into account for
     the first line of a long, split header.

     Optional _continuation_ws_ must be RFC 2822
     (http://tools.ietf.org/html/rfc2822.html)-compliant folding
     whitespace, and is usually either a space or a hard tab character.
     This character will be prepended to continuation lines.
     _continuation_ws_ defaults to a single space character (" ").

     Optional _errors_ is passed straight through to the
     append() method.

      -- Method: append (s[, charset[, errors]])
          Append the string _s_ to the MIME header.

          Optional _charset_, if given, should be a Charset.
          instance (see email.charset) or the name of a
          character set, which will be converted to a Charset
          instance.  A value of `None' (the default) means that
          the _charset_ given in the constructor is used.

          _s_ may be a byte string or a Unicode string.  If it is a
          byte string (i.e.  `isinstance(s, str)' is true), then
          _charset_ is the encoding of that byte string, and a
          UnicodeError will be raised if the string cannot be
          decoded with that character set.

          If _s_ is a Unicode string, then _charset_ is a hint
          specifying the character set of the characters in the string.
          In this case, when producing an RFC 2822
          (http://tools.ietf.org/html/rfc2822.html)-compliant header
          using RFC 2047 (http://tools.ietf.org/html/rfc2047.html)
          rules, the Unicode string will be encoded using the following
          charsets in order: `us-ascii', the _charset_ hint, `utf-8'.
          The first character set to not provoke a UnicodeError
          is used.

          Optional _errors_ is passed through to any unicode()
          or `ustr.encode()' call, and defaults to "strict".

      -- Method: encode ([splitchars])
          Encode a message header into an RFC-compliant format,
          possibly wrapping long lines and encapsulating non-ASCII
          parts in base64 or quoted-printable encodings.  Optional
          _splitchars_ is a string containing characters to split long
          ASCII lines on, in rough support of RFC 2822
          (http://tools.ietf.org/html/rfc2822.html)'s _highest level
          syntactic breaks_.  This doesn't affect RFC 2047
          (http://tools.ietf.org/html/rfc2047.html) encoded lines.

     The Header class also provides a number of methods to
     support standard operators and built-in functions.

      -- Method: __str__ ()
          A synonym for Header.encode().  Useful for
          `str(aHeader)'.

      -- Method: __unicode__ ()
          A helper for the built-in unicode() function.
          Returns the header as a Unicode string.

      -- Method: __eq__ (other)
          This method allows you to compare two Header.
          instances for equality.

      -- Method: __ne__ (other)
          This method allows you to compare two Header.
          instances for inequality.

  The email.header module also provides the following
convenient functions.

 -- Function: email.header.decode_header (header)
     Decode a message header value without converting the character
     set. The header value is in _header_.

     This function returns a list of `(decoded_string, charset)' pairs
     containing each of the decoded parts of the header.  _charset_ is
     `None' for non-encoded parts of the header, otherwise a lower case
     string containing the name of the character set specified in the
     encoded string.

     Here's an example:

         >>> from email.header import decode_header
         >>> decode_header('=?iso-8859-1?q?p=F6stal?=')
         [('p\xf6stal', 'iso-8859-1')]

 -- Function: email.header.make_header (decoded_seq[, maxlinelen[,
          header_name[, continuation_ws]]])
     Create a Header instance from a sequence of pairs as
     returned by decode_header().

     decode_header() takes a header value string and
     returns a sequence of pairs of the format `(decoded_string,
     charset)' where _charset_ is the name of the character set.

     This function takes one of those sequence of pairs and returns a
     Header.  instance.  Optional _maxlinelen_,
     _header_name_, and _continuation_ws_ are as in the Header
     constructor.

5.18.1.9 `email': Representing character sets
.............................................

This module provides a class Charset for representing
character sets and character set conversions in email messages, as well
as a character set registry and several convenience methods for
manipulating this registry.  Instances of Charset are used
in several other modules within the email package.

  Import this class from the email.charset module.

  New in version 2.2.2.

 -- Class: email.charset.Charset ([input_charset])
     Map character sets to their email properties.

     This class provides information about the requirements imposed on
     email for a specific character set.  It also provides convenience
     routines for converting between character sets, given the
     availability of the applicable codecs.  Given a character set, it
     will do its best to provide information on how to use that
     character set in an email message in an RFC-compliant way.

     Certain character sets must be encoded with quoted-printable or
     base64 when used in email headers or bodies.  Certain character
     sets must be converted outright, and are not allowed in email.

     Optional _input_charset_ is as described below; it is always
     coerced to lower case.  After being alias normalized it is also
     used as a lookup into the registry of character sets to find out
     the header encoding, body encoding, and output conversion codec to
     be used for the character set.  For example, if _input_charset_ is
     `iso-8859-1', then headers and bodies will be encoded using
     quoted-printable and no output conversion codec is necessary.  If
     _input_charset_ is `euc-jp', then headers will be encoded with
     base64, bodies will not be encoded, but output text will be
     converted from the `euc-jp' character set to the `iso-2022-jp'
     character set.

     Charset instances have the following data attributes:

      -- Attribute: input_charset
          The initial character set specified.  Common aliases are
          converted to their _official_ email names (e.g. `latin_1' is
          converted to `iso-8859-1').  Defaults to 7-bit `us-ascii'.

      -- Attribute: header_encoding
          If the character set must be encoded before it can be used in
          an email header, this attribute will be set to `Charset.QP'
          (for quoted-printable), `Charset.BASE64' (for base64
          encoding), or `Charset.SHORTEST' for the shortest of QP or
          BASE64 encoding. Otherwise, it will be `None'.

      -- Attribute: body_encoding
          Same as _header_encoding_, but describes the encoding for the
          mail message's body, which indeed may be different than the
          header encoding.  `Charset.SHORTEST' is not allowed for
          _body_encoding_.

      -- Attribute: output_charset
          Some character sets must be converted before they can be used
          in email headers or bodies.  If the _input_charset_ is one of
          them, this attribute will contain the name of the character
          set output will be converted to.  Otherwise, it will be
          `None'.

      -- Attribute: input_codec
          The name of the Python codec used to convert the
          _input_charset_ to Unicode.  If no conversion codec is
          necessary, this attribute will be `None'.

      -- Attribute: output_codec
          The name of the Python codec used to convert Unicode to the
          _output_charset_.  If no conversion codec is necessary, this
          attribute will have the same value as the _input_codec_.

     Charset instances also have the following methods:

      -- Method: get_body_encoding ()
          Return the content transfer encoding used for body encoding.

          This is either the string `quoted-printable' or `base64'
          depending on the encoding used, or it is a function, in which
          case you should call the function with a single argument, the
          Message object being encoded.  The function should then set
          the `Content-Transfer-Encoding' header itself to whatever is
          appropriate.

          Returns the string `quoted-printable' if _body_encoding_ is
          `QP', returns the string `base64' if _body_encoding_ is
          `BASE64', and returns the string `7bit' otherwise.

      -- Method: convert (s)
          Convert the string _s_ from the _input_codec_ to the
          _output_codec_.

      -- Method: to_splittable (s)
          Convert a possibly multibyte string to a safely splittable
          format. _s_ is the string to split.

          Uses the _input_codec_ to try and convert the string to
          Unicode, so it can be safely split on character boundaries
          (even for multibyte characters).

          Returns the string as-is if it isn't known how to convert _s_
          to Unicode with the _input_charset_.

          Characters that could not be converted to Unicode will be
          replaced with the Unicode replacement character `'U+FFFD''.

      -- Method: from_splittable (ustr[, to_output])
          Convert a splittable string back into an encoded string.
          _ustr_ is a Unicode string to "unsplit".

          This method uses the proper codec to try and convert the
          string from Unicode back into an encoded format.  Return the
          string as-is if it is not Unicode, or if it could not be
          converted from Unicode.

          Characters that could not be converted from Unicode will be
          replaced with an appropriate character (usually `'?'').

          If _to_output_ is `True' (the default), uses _output_codec_
          to convert to an encoded format.  If _to_output_ is `False',
          it uses _input_codec_.

      -- Method: get_output_charset ()
          Return the output character set.

          This is the _output_charset_ attribute if that is not `None',
          otherwise it is _input_charset_.

      -- Method: encoded_header_len ()
          Return the length of the encoded header string, properly
          calculating for quoted-printable or base64 encoding.

      -- Method: header_encode (s[, convert])
          Header-encode the string _s_.

          If _convert_ is `True', the string will be converted from the
          input charset to the output charset automatically.  This is
          not useful for multibyte character sets, which have line
          length issues (multibyte characters must be split on a
          character, not a byte boundary); use the higher-level
          Header class to deal with these issues (see
          email.header).  _convert_ defaults to `False'.

          The type of encoding (base64 or quoted-printable) will be
          based on the _header_encoding_ attribute.

      -- Method: body_encode (s[, convert])
          Body-encode the string _s_.

          If _convert_ is `True' (the default), the string will be
          converted from the input charset to output charset
          automatically. Unlike header_encode(), there are
          no issues with byte boundaries and multibyte charsets in
          email bodies, so this is usually pretty safe.

          The type of encoding (base64 or quoted-printable) will be
          based on the _body_encoding_ attribute.

     The Charset class also provides a number of methods
     to support standard operations and built-in functions.

      -- Method: __str__ ()
          Returns _input_charset_ as a string coerced to lower case.
          __repr__() is an alias for __str__().

      -- Method: __eq__ (other)
          This method allows you to compare two Charset.
          instances for equality.

      -- Method: __ne__ (other)
          This method allows you to compare two Charset.
          instances for inequality.

  The email.charset module also provides the following
functions for adding new entries to the global character set, alias,
and codec registries:

 -- Function: email.charset.add_charset (charset[, header_enc[,
          body_enc[, output_charset]]])
     Add character properties to the global registry.

     _charset_ is the input character set, and must be the canonical
     name of a character set.

     Optional _header_enc_ and _body_enc_ is either `Charset.QP' for
     quoted-printable, `Charset.BASE64' for base64 encoding,
     `Charset.SHORTEST' for the shortest of quoted-printable or base64
     encoding, or `None' for no encoding.  `SHORTEST' is only valid for
     _header_enc_. The default is `None' for no encoding.

     Optional _output_charset_ is the character set that the output
     should be in.  Conversions will proceed from input charset, to
     Unicode, to the output charset when the method
     Charset.convert() is called.  The default is to output in
     the same character set as the input.

     Both _input_charset_ and _output_charset_ must have Unicode codec
     entries in the module's character set-to-codec mapping; use
     add_codec() to add codecs the module does not know about.
     See the codecs module's documentation for more
     information.

     The global character set registry is kept in the module global
     dictionary `CHARSETS'.

 -- Function: email.charset.add_alias (alias, canonical)
     Add a character set alias.  _alias_ is the alias name, e.g.
     `latin-1'.  _canonical_ is the character set's canonical name,
     e.g. `iso-8859-1'.

     The global charset alias registry is kept in the module global
     dictionary `ALIASES'.

 -- Function: email.charset.add_codec (charset, codecname)
     Add a codec that map characters in the given character set to and
     from Unicode.

     _charset_ is the canonical name of a character set. _codecname_ is
     the name of a Python codec, as appropriate for the second argument
     to the unicode().  built-in, or to the `encode()'
     method of a Unicode string.

5.18.1.10 `email': Encoders
...........................

When creating Message objects from scratch, you often need
to encode the payloads for transport through compliant mail servers.
This is especially true for `image/*' and `text/*' type messages
containing binary data.

  The email package provides some convenient encodings in its
`encoders' module.  These encoders are actually used by the
MIMEAudio and MIMEImage.  class constructors to
provide default encodings.  All encoder functions take exactly one
argument, the message object to encode.  They usually extract the
payload, encode it, and reset the payload to this newly encoded value.
They should also set the `Content-Transfer-Encoding' header as
appropriate.

  Here are the encoding functions provided:

 -- Function: email.encoders.encode_quopri (msg)
     Encodes the payload into quoted-printable form and sets the
     `Content-Transfer-Encoding' header to `quoted-printable' (1).
     This is a good encoding to use when most of your payload is normal
     printable data, but contains a few unprintable characters.

 -- Function: email.encoders.encode_base64 (msg)
     Encodes the payload into base64 form and sets the
     `Content-Transfer-Encoding' header to `base64'.  This is a good
     encoding to use when most of your payload is unprintable data
     since it is a more compact form than quoted-printable.  The
     drawback of base64 encoding is that it renders the text non-human
     readable.

 -- Function: email.encoders.encode_7or8bit (msg)
     This doesn't actually modify the message's payload, but it does
     set the `Content-Transfer-Encoding' header to either `7bit' or
     `8bit' as appropriate, based on the payload data.

 -- Function: email.encoders.encode_noop (msg)
     This does nothing; it doesn't even set the
     `Content-Transfer-Encoding' header.

Footnotes
.........

     (1) Note that encoding with encode_quopri() also
     encodes all tabs and space characters in the data.

5.18.1.11 `email': Exception and Defect classes
...............................................

The following exception classes are defined in the email.errors
module:

 -- Exception: email.errors.MessageError
     This is the base class for all exceptions that the email
     package can raise.  It is derived from the standard
     Exception class and defines no additional methods.

 -- Exception: email.errors.MessageParseError
     This is the base class for exceptions raised by the Parser.
     class.  It is derived from MessageError.

 -- Exception: email.errors.HeaderParseError
     Raised under some error conditions when parsing the RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) headers of a message,
     this class is derived from MessageParseError. It can
     be raised from the `Parser.parse()' or `Parser.parsestr()' methods.

     Situations where it can be raised include finding an envelope
     header after the first RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) header of the message,
     finding a continuation line before the first RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) header is found, or
     finding a line in the headers which is neither a header or a
     continuation line.

 -- Exception: email.errors.BoundaryError
     Raised under some error conditions when parsing the RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) headers of a message,
     this class is derived from MessageParseError. It can
     be raised from the `Parser.parse()' or `Parser.parsestr()' methods.

     Situations where it can be raised include not being able to find
     the starting or terminating boundary in a `multipart/*' message
     when strict parsing is used.

 -- Exception: email.errors.MultipartConversionError
     Raised when a payload is added to a `Message' object using
     `add_payload()', but the payload is already a scalar and the
     message's `Content-Type' main type is not either `multipart' or
     missing.  MultipartConversionError multiply inherits
     from MessageError and the built-in TypeError.

     Since `Message.add_payload()' is deprecated, this exception is
     rarely raised in practice.  However the exception may also be
     raised if the `attach()' method is called on an instance of a
     class derived from MIMENonMultipart. (e.g.  see
     MIMEImage).

  Here's the list of the defects that the `FeedParser' can find while
parsing messages.  Note that the defects are added to the message where
the problem was found, so for example, if a message nested inside a
`multipart/alternative' had a malformed header, that nested message
object would have a defect, but the containing messages would not.

  All defect classes are subclassed from `email.errors.MessageDefect',
but this class is _not_ an exception!

  New in version 2.4: All the defect classes were added.

   * `NoBoundaryInMultipartDefect' - A message claimed to be a
     multipart, but had no `boundary' parameter.

   * `StartBoundaryNotFoundDefect' - The start boundary claimed in the
     `Content-Type' header was never found.

   * `FirstHeaderLineIsContinuationDefect' - The message had a
     continuation line as its first header line.

   * `MisplacedEnvelopeHeaderDefect' - A "Unix From" header was found
     in the middle of a header block.

   * `MalformedHeaderDefect' - A header was found that was missing a
     colon, or was otherwise malformed.

   * `MultipartInvariantViolationDefect' - A message claimed to be a
     `multipart', but no subparts were found.  Note that when a message
     has this defect, its `is_multipart()' method may return false even
     though its content type claims to be `multipart'.

5.18.1.12 `email': Miscellaneous utilities
..........................................

There are several useful utilities provided in the email.utils
module:

 -- Function: email.utils.quote (str)
     Return a new string with backslashes in _str_ replaced by two
     backslashes, and double quotes replaced by backslash-double quote.

 -- Function: email.utils.unquote (str)
     Return a new string which is an _unquoted_ version of _str_. If
     _str_ ends and begins with double quotes, they are stripped off.
     Likewise if _str_ ends and begins with angle brackets, they are
     stripped off.

 -- Function: email.utils.parseaddr (address)
     Parse address - which should be the value of some
     address-containing field such as `To' or `Cc' - into its
     constituent _realname_ and _email address_ parts.  Returns a tuple
     of that information, unless the parse fails, in which case a
     2-tuple of `('', '')' is returned.

 -- Function: email.utils.formataddr (pair)
     The inverse of parseaddr(), this takes a 2-tuple of
     the form `(realname, email_address)' and returns the string value
     suitable for a `To' or `Cc' header.  If the first element of
     _pair_ is false, then the second element is returned unmodified.

 -- Function: email.utils.getaddresses (fieldvalues)
     This method returns a list of 2-tuples of the form returned by
     `parseaddr()'.  _fieldvalues_ is a sequence of header field values
     as might be returned by `Message.get_all()'.  Here's a simple
     example that gets all the recipients of a message:

         from email.utils import getaddresses

         tos = msg.get_all('to', [])
         ccs = msg.get_all('cc', [])
         resent_tos = msg.get_all('resent-to', [])
         resent_ccs = msg.get_all('resent-cc', [])
         all_recipients = getaddresses(tos + ccs + resent_tos + resent_ccs)

 -- Function: email.utils.parsedate (date)
     Attempts to parse a date according to the rules in RFC 2822
     (http://tools.ietf.org/html/rfc2822.html). however, some mailers
     don't follow that format as specified, so parsedate().
     tries to guess correctly in such cases.  _date_ is a string
     containing an RFC 2822 (http://tools.ietf.org/html/rfc2822.html)
     date, such as  `"Mon, 20 Nov 1995 19:12:08 -0500"'.  If it
     succeeds in parsing the date, see parsedate() returns a
     9-tuple that can be passed directly to time.mktime().;
     otherwise `None' will be returned.  Note that indexes 6, 7, and 8
     of the result tuple are not usable.

 -- Function: email.utils.parsedate_tz (date)
     Performs the same function as parsedate(), but returns
     either `None' or a 10-tuple; the first 9 elements make up a tuple
     that can be passed directly to time.mktime(), and the
     tenth is the offset of the date's timezone from UTC (which is the
     official term for Greenwich Mean Time) (1).  If the input string
     has no timezone, the last element of the tuple returned is `None'.
     Note that indexes 6, 7, and 8 of the result tuple are not usable.

 -- Function: email.utils.mktime_tz (tuple)
     Turn a 10-tuple as returned by parsedate_tz() into a
     UTC timestamp.  It the timezone item in the tuple is `None',
     assume local time.  Minor deficiency: mktime_tz().
     interprets the first 8 elements of _tuple_ as a local time and
     then compensates for the timezone difference.  This may yield a
     slight error around changes in daylight savings time, though not
     worth worrying about for common use.

 -- Function: email.utils.formatdate ([timeval[, localtime][, usegmt]])
     Returns a date string as per RFC 2822
     (http://tools.ietf.org/html/rfc2822.html), e.g.:

         Fri, 09 Nov 2001 01:08:47 -0000

     Optional _timeval_ if given is a floating point time value as
     accepted by time.gmtime() and time.localtime(),
     otherwise the current time is used.

     Optional _localtime_ is a flag that when `True', interprets
     _timeval_, and returns a date relative to the local timezone
     instead of UTC, properly taking daylight savings time into
     account. The default is `False' meaning UTC is used.

     Optional _usegmt_ is a flag that when `True', outputs a  date
     string with the timezone as an ascii string `GMT', rather than a
     numeric `-0000'. This is needed for some protocols (such as HTTP).
     This only applies when _localtime_ is `False'.  The default is
     `False'.

     New in version 2.4.

 -- Function: email.utils.make_msgid ([idstring])
     Returns a string suitable for an RFC 2822
     (http://tools.ietf.org/html/rfc2822.html)-compliant `Message-ID'
     header.  Optional _idstring_ if given, is a string used to
     strengthen the uniqueness of the message id.

 -- Function: email.utils.decode_rfc2231 (s)
     Decode the string _s_ according to RFC 2231
     (http://tools.ietf.org/html/rfc2231.html).

 -- Function: email.utils.encode_rfc2231 (s[, charset[, language]])
     Encode the string _s_ according to RFC 2231
     (http://tools.ietf.org/html/rfc2231.html).  Optional _charset_ and
     _language_, if given is the character set name and language name
     to use.  If neither is given, _s_ is returned as-is.  If _charset_
     is given but _language_ is not, the string is encoded using the
     empty string for _language_.

 -- Function: email.utils.collapse_rfc2231_value (value[, errors[,
          fallback_charset]])
     When a header parameter is encoded in RFC 2231
     (http://tools.ietf.org/html/rfc2231.html) format,
     `Message.get_param()' may return a 3-tuple containing the
     character set, language, and value.
     collapse_rfc2231_value() turns this into a unicode string.
     Optional _errors_ is passed to the _errors_ argument of the
     built-in unicode() function; it defaults to `replace'.
     Optional _fallback_charset_ specifies the character set to use if
     the one in the RFC 2231 (http://tools.ietf.org/html/rfc2231.html)
     header is not known by Python; it defaults to `us-ascii'.

     For convenience, if the _value_ passed to
     collapse_rfc2231_value() is not a tuple, it should be a
     string and it is returned unquoted.

 -- Function: email.utils.decode_params (params)
     Decode parameters list according to RFC 2231
     (http://tools.ietf.org/html/rfc2231.html).  _params_ is a sequence
     of 2-tuples containing elements of the form `(content-type,
     string-value)'.

  Changed in version 2.4: The `dump_address_pair()' function has been
removed; use formataddr().  instead.

  Changed in version 2.4: The `decode()' function has been removed; use
the `Header.decode_header()' method instead.

  Changed in version 2.4: The `encode()' function has been removed; use
the `Header.encode()' method instead.

Footnotes
.........

     (1) Note that the sign of the timezone offset is the opposite of
     the sign of the `time.timezone' variable for the same timezone;
     the latter variable follows the POSIX standard while this module
     follows RFC 2822 (http://tools.ietf.org/html/rfc2822.html).

5.18.1.13 `email': Iterators
............................

Iterating over a message object tree is fairly easy with the
`Message.walk()' method.  The email.iterators module
provides some useful higher level iterations over message object trees.

 -- Function: email.iterators.body_line_iterator (msg[, decode])
     This iterates over all the payloads in all the subparts of _msg_,
     returning the string payloads line-by-line.  It skips over all the
     subpart headers, and it skips over any subpart with a payload that
     isn't a Python string.  This is somewhat equivalent to reading the
     flat text representation of the message from a file using
     readline(), skipping over all the intervening headers.

     Optional _decode_ is passed through to `Message.get_payload()'.

 -- Function: email.iterators.typed_subpart_iterator (msg[, maintype[,
          subtype]])
     This iterates over all the subparts of _msg_, returning only those
     subparts that match the MIME type specified by _maintype_ and
     _subtype_.

     Note that _subtype_ is optional; if omitted, then subpart MIME
     type matching is done only with the main type.  _maintype_ is
     optional too; it defaults to `text'.

     Thus, by default typed_subpart_iterator() returns
     each subpart that has a MIME type of `text/*'.

  The following function has been added as a useful debugging tool.  It
should _not_ be considered part of the supported public interface for
the package.

 -- Function: email.iterators._structure (msg[, fp[, level]])
     Prints an indented representation of the content types of the
     message object structure.  For example:

         >>> msg = email.message_from_file(somefile)
         >>> _structure(msg)
         multipart/mixed
             text/plain
             text/plain
             multipart/digest
                 message/rfc822
                     text/plain
                 message/rfc822
                     text/plain
                 message/rfc822
                     text/plain
                 message/rfc822
                     text/plain
                 message/rfc822
                     text/plain
             text/plain

     Optional _fp_ is a file-like object to print the output to.  It
     must be suitable for Python's extended print statement.  _level_
     is used internally.

5.18.1.14 `email': Examples
...........................

Here are a few examples of how to use the email package to
read, write, and send simple email messages, as well as more complex
MIME messages.

  First, let's see how to create and send a simple text message:

    # Import smtplib for the actual sending function
    import smtplib

    # Import the email modules we'll need
    from email.mime.text import MIMEText

    # Open a plain text file for reading.  For this example, assume that
    # the text file contains only ASCII characters.
    fp = open(textfile, 'rb')
    # Create a text/plain message
    msg = MIMEText(fp.read())
    fp.close()

    # me == the sender's email address
    # you == the recipient's email address
    msg['Subject'] = 'The contents of %s' % textfile
    msg['From'] = me
    msg['To'] = you

    # Send the message via our own SMTP server, but don't include the
    # envelope header.
    s = smtplib.SMTP()
    s.sendmail(me, [you], msg.as_string())
    s.quit()

And parsing RFC822 headers can easily be done by the parse(filename) or
parsestr(message_as_string) methods of the Parser() class:

    # Import the email modules we'll need
    from email.parser import Parser

    #  If the e-mail headers are in a file, uncomment this line:
    #headers = Parser().parse(open(messagefile, 'r'))

    #  Or for parsing headers in a string, use:
    headers = Parser().parsestr('From: <user@example.com>\n'
            'To: <someone_else@example.com>\n'
            'Subject: Test message\n'
            '\n'
            'Body would go here\n')

    #  Now the header items can be accessed as a dictionary:
    print 'To: %s' % headers['to']
    print 'From: %s' % headers['from']
    print 'Subject: %s' % headers['subject']

Here's an example of how to send a MIME message containing a bunch of
family pictures that may be residing in a directory:

    # Import smtplib for the actual sending function
    import smtplib

    # Here are the email package modules we'll need
    from email.mime.image import MIMEImage
    from email.mime.multipart import MIMEMultipart

    COMMASPACE = ', '

    # Create the container (outer) email message.
    msg = MIMEMultipart()
    msg['Subject'] = 'Our family reunion'
    # me == the sender's email address
    # family = the list of all recipients' email addresses
    msg['From'] = me
    msg['To'] = COMMASPACE.join(family)
    msg.preamble = 'Our family reunion'

    # Assume we know that the image files are all in PNG format
    for file in pngfiles:
        # Open the files in binary mode.  Let the MIMEImage class automatically
        # guess the specific image type.
        fp = open(file, 'rb')
        img = MIMEImage(fp.read())
        fp.close()
        msg.attach(img)

    # Send the email via our own SMTP server.
    s = smtplib.SMTP()
    s.sendmail(me, family, msg.as_string())
    s.quit()

Here's an example of how to send the entire contents of a directory as
an email message: (1)

    #!/usr/bin/env python

    """Send the contents of a directory as a MIME message."""

    import os
    import sys
    import smtplib
    # For guessing MIME type based on file name extension
    import mimetypes

    from optparse import OptionParser

    from email import encoders
    from email.message import Message
    from email.mime.audio import MIMEAudio
    from email.mime.base import MIMEBase
    from email.mime.image import MIMEImage
    from email.mime.multipart import MIMEMultipart
    from email.mime.text import MIMEText

    COMMASPACE = ', '

    def main():
        parser = OptionParser(usage="""\
    Send the contents of a directory as a MIME message.

    Usage: %prog [options]

    Unless the -o option is given, the email is sent by forwarding to your local
    SMTP server, which then does the normal delivery process.  Your local machine
    must be running an SMTP server.
    """)
        parser.add_option('-d', '--directory',
                          type='string', action='store',
                          help="""Mail the contents of the specified directory,
                          otherwise use the current directory.  Only the regular
                          files in the directory are sent, and we don't recurse to
                          subdirectories.""")
        parser.add_option('-o', '--output',
                          type='string', action='store', metavar='FILE',
                          help="""Print the composed message to FILE instead of
                          sending the message to the SMTP server.""")
        parser.add_option('-s', '--sender',
                          type='string', action='store', metavar='SENDER',
                          help='The value of the From: header (required)')
        parser.add_option('-r', '--recipient',
                          type='string', action='append', metavar='RECIPIENT',
                          default=[], dest='recipients',
                          help='A To: header value (at least one required)')
        opts, args = parser.parse_args()
        if not opts.sender or not opts.recipients:
            parser.print_help()
            sys.exit(1)
        directory = opts.directory
        if not directory:
            directory = '.'
        # Create the enclosing (outer) message
        outer = MIMEMultipart()
        outer['Subject'] = 'Contents of directory %s' % os.path.abspath(directory)
        outer['To'] = COMMASPACE.join(opts.recipients)
        outer['From'] = opts.sender
        outer.preamble = 'You will not see this in a MIME-aware mail reader.\n'

        for filename in os.listdir(directory):
            path = os.path.join(directory, filename)
            if not os.path.isfile(path):
                continue
            # Guess the content type based on the file's extension.  Encoding
            # will be ignored, although we should check for simple things like
            # gzip'd or compressed files.
            ctype, encoding = mimetypes.guess_type(path)
            if ctype is None or encoding is not None:
                # No guess could be made, or the file is encoded (compressed), so
                # use a generic bag-of-bits type.
                ctype = 'application/octet-stream'
            maintype, subtype = ctype.split('/', 1)
            if maintype == 'text':
                fp = open(path)
                # Note: we should handle calculating the charset
                msg = MIMEText(fp.read(), _subtype=subtype)
                fp.close()
            elif maintype == 'image':
                fp = open(path, 'rb')
                msg = MIMEImage(fp.read(), _subtype=subtype)
                fp.close()
            elif maintype == 'audio':
                fp = open(path, 'rb')
                msg = MIMEAudio(fp.read(), _subtype=subtype)
                fp.close()
            else:
                fp = open(path, 'rb')
                msg = MIMEBase(maintype, subtype)
                msg.set_payload(fp.read())
                fp.close()
                # Encode the payload using Base64
                encoders.encode_base64(msg)
            # Set the filename parameter
            msg.add_header('Content-Disposition', 'attachment', filename=filename)
            outer.attach(msg)
        # Now send or store the message
        composed = outer.as_string()
        if opts.output:
            fp = open(opts.output, 'w')
            fp.write(composed)
            fp.close()
        else:
            s = smtplib.SMTP()
            s.sendmail(opts.sender, opts.recipients, composed)
            s.quit()

    if __name__ == '__main__':
        main()

Here's an example of how to unpack a MIME message like the one above,
into a directory of files:

    #!/usr/bin/env python

    """Unpack a MIME message into a directory of files."""

    import os
    import sys
    import email
    import errno
    import mimetypes

    from optparse import OptionParser

    def main():
        parser = OptionParser(usage="""\
    Unpack a MIME message into a directory of files.

    Usage: %prog [options] msgfile
    """)
        parser.add_option('-d', '--directory',
                          type='string', action='store',
                          help="""Unpack the MIME message into the named
                          directory, which will be created if it doesn't already
                          exist.""")
        opts, args = parser.parse_args()
        if not opts.directory:
            parser.print_help()
            sys.exit(1)

        try:
            msgfile = args[0]
        except IndexError:
            parser.print_help()
            sys.exit(1)

        try:
            os.mkdir(opts.directory)
        except OSError, e:
            # Ignore directory exists error
            if e.errno != errno.EEXIST:
                raise

        fp = open(msgfile)
        msg = email.message_from_file(fp)
        fp.close()

        counter = 1
        for part in msg.walk():
            # multipart/* are just containers
            if part.get_content_maintype() == 'multipart':
                continue
            # Applications should really sanitize the given filename so that an
            # email message can't be used to overwrite important files
            filename = part.get_filename()
            if not filename:
                ext = mimetypes.guess_extension(part.get_content_type())
                if not ext:
                    # Use a generic bag-of-bits extension
                    ext = '.bin'
                filename = 'part-%03d%s' % (counter, ext)
            counter += 1
            fp = open(os.path.join(opts.directory, filename), 'wb')
            fp.write(part.get_payload(decode=True))
            fp.close()

    if __name__ == '__main__':
        main()

Here's an example of how to create an HTML message with an alternative
plain text version: (2)

    #!/usr/bin/env python

    import smtplib

    from email.mime.multipart import MIMEMultipart
    from email.mime.text import MIMEText

    # me == my email address
    # you == recipient's email address
    me = "my@email.com"
    you = "your@email.com"

    # Create message container - the correct MIME type is multipart/alternative.
    msg = MIMEMultipart('alternative')
    msg['Subject'] = "Link"
    msg['From'] = me
    msg['To'] = you

    # Create the body of the message (a plain-text and an HTML version).
    text = "Hi!\nHow are you?\nHere is the link you wanted:\nhttp://www.python.org"
    html = """\
    <html>
      <head></head>
      <body>
        <p>Hi!<br>
           How are you?<br>
           Here is the <a href="http://www.python.org">link</a> you wanted.
        </p>
      </body>
    </html>
    """

    # Record the MIME types of both parts - text/plain and text/html.
    part1 = MIMEText(text, 'plain')
    part2 = MIMEText(html, 'html')

    # Attach parts into message container.
    # According to RFC 2046, the last part of a multipart message, in this case
    # the HTML message, is best and preferred.
    msg.attach(part1)
    msg.attach(part2)

    # Send the message via local SMTP server.
    s = smtplib.SMTP('localhost')
    # sendmail function takes 3 arguments: sender's address, recipient's address
    # and message to send - here it is sent as one string.
    s.sendmail(me, you, msg.as_string())
    s.quit()

Footnotes
.........

     (1) Thanks to Matthew Dixon Cowles for the original inspiration
     and examples.

     (2) Contributed by Martin Matejek.

Module smtplib.
     SMTP protocol client

Module nntplib.
     NNTP protocol client

5.18.1.15 Package History
.........................

This table describes the release history of the email package,
corresponding to the version of Python that the package was released
with.  For purposes of this document, when you see a note about change
or added versions, these refer to the Python version the change was
made in, _not_ the email package version.  This table also describes
the Python compatibility of each version of the package.

email version       distributed with                   compatible with
-----------------------------------------------------------------------------------
`1.x'               Python 2.2.0 to Python 2.2.1       _no longer supported_
`2.5'               Python 2.2.2+ and Python 2.3       Python 2.1 to 2.5
`3.0'               Python 2.4                         Python 2.3 to 2.5
`4.0'               Python 2.5                         Python 2.3 to 2.5

  Here are the major differences between email version 4 and
version 3:

   * All modules have been renamed according to PEP 8
     (http://www.python.org/dev/peps/pep-0008) standards.  For example,
     the version 3 module `email.Message' was renamed to
     email.message in version 4.

   * A new subpackage email.mime was added and all the
     version 3 `email.MIME*' modules were renamed and situated into the
     email.mime.  subpackage.  For example, the version 3
     module `email.MIMEText' was renamed to `email.mime.text'.

     _Note that the version 3 names will continue to work until Python
     2.6_.

   * The `email.mime.application' module was added, which contains the
     `MIMEApplication' class.

   * Methods that were deprecated in version 3 have been removed.
     These include `Generator.__call__()', `Message.get_type()',
     `Message.get_main_type()', `Message.get_subtype()'.

   * Fixes have been added for RFC 2231
     (http://tools.ietf.org/html/rfc2231.html) support which can change
     some of the return types for `Message.get_param()' and friends.
     Under some circumstances, values which used to return a 3-tuple
     now return simple strings (specifically, if all extended parameter
     segments were unencoded, there is no language and charset
     designation expected, so the return type is now a simple string).
     Also, %-decoding used to be done for both encoded and unencoded
     segments; this decoding is now done only for encoded segments.

  Here are the major differences between email version 3 and
version 2:

   * The `FeedParser' class was introduced, and the `Parser' class was
     implemented in terms of the `FeedParser'.  All parsing therefore is
     non-strict, and parsing will make a best effort never to raise an
     exception.  Problems found while parsing messages are stored in
     the message's _defect_ attribute.

   * All aspects of the API which raised DeprecationWarning.
     s in version 2 have been removed.  These include the
     __encoder_ argument to the `MIMEText' constructor, the
     `Message.add_payload()' method, the `Utils.dump_address_pair()'
     function, and the functions `Utils.decode()' and `Utils.encode()'.

   * New DeprecationWarning.s have been added to:
     `Generator.__call__()', `Message.get_type()',
     `Message.get_main_type()', `Message.get_subtype()', and the
     _strict_ argument to the `Parser' class.  These are expected to be
     removed in future versions.

   * Support for Pythons earlier than 2.3 has been removed.

  Here are the differences between email version 2 and
version 1:

   * The `email.Header' and `email.Charset' modules have been added.

   * The pickle format for `Message' instances has changed. Since this
     was never (and still isn't) formally defined, this isn't
     considered a backward incompatibility.  However if your
     application pickles and unpickles `Message' instances, be aware
     that in email version 2, `Message' instances now have
     private variables __charset_ and __default_type_.

   * Several methods in the `Message' class have been deprecated, or
     their signatures changed.  Also, many new methods have been added.
     See the documentation for the `Message' class for details.  The
     changes should be completely backward compatible.

   * The object structure has changed in the face of `message/rfc822'
     content types.  In email version 1, such a type would
     be represented by a scalar payload, i.e. the container message's
     `is_multipart()' returned false, `get_payload()' was not a list
     object, but a single `Message' instance.

     This structure was inconsistent with the rest of the package, so
     the object representation for `message/rfc822' content types was
     changed.  In email version 2, the container _does_
     return `True' from `is_multipart()', and `get_payload()' returns a
     list containing a single `Message' item.

     Note that this is one place that backward compatibility could not
     be completely maintained.  However, if you're already testing the
     return type of `get_payload()', you should be fine.  You just need
     to make sure your code doesn't do a `set_payload()' with a
     `Message' instance on a container with a content type of
     `message/rfc822'.

   * The `Parser' constructor's _strict_ argument was added, and its
     `parse()' and `parsestr()' methods grew a _headersonly_ argument.
     The _strict_ flag was also added to functions
     email.message_from_file() and
     email.message_from_string().

   * `Generator.__call__()' is deprecated; use `Generator.flatten()'
     instead.  The `Generator' class has also grown the `clone()'
     method.

   * The `DecodedGenerator' class in the `email.Generator' module was
     added.

   * The intermediate base classes `MIMENonMultipart' and
     `MIMEMultipart' have been added, and interposed in the class
     hierarchy for most of the other MIME-related derived classes.

   * The __encoder_ argument to the `MIMEText' constructor has been
     deprecated.  Encoding  now happens implicitly based on the
     __charset_ argument.

   * The following functions in the `email.Utils' module have been
     deprecated: `dump_address_pairs()', `decode()', and `encode()'.
     The following functions have been added to the module:
     `make_msgid()', `decode_rfc2231()', `encode_rfc2231()', and
     `decode_params()'.

   * The non-public function `email.Iterators._structure()' was added.

5.18.1.16 Differences from `mimelib'
....................................

The email package was originally prototyped as a separate
library called mimelib (http://mimelib.sf.net/). Changes have been made
so that method names are more consistent, and some methods or modules
have either been added or removed.  The semantics of some of the
methods have also changed.  For the most part, any functionality
available in `mimelib' is still available in the email.
package, albeit often in a different way.  Backward compatibility
between the `mimelib' package and the email package was not a
priority.

  Here is a brief description of the differences between the `mimelib'
and the email packages, along with hints on how to port your
applications.

  Of course, the most visible difference between the two packages is
that the package name has been changed to email.  In
addition, the top-level package has the following differences:

   * `messageFromString()' has been renamed to
     message_from_string().

   * `messageFromFile()' has been renamed to message_from_file().

  The `Message' class has the following differences:

   * The method `asString()' was renamed to `as_string()'.

   * The method `ismultipart()' was renamed to `is_multipart()'.

   * The `get_payload()' method has grown a _decode_ optional argument.

   * The method `getall()' was renamed to `get_all()'.

   * The method `addheader()' was renamed to `add_header()'.

   * The method `gettype()' was renamed to `get_type()'.

   * The method `getmaintype()' was renamed to `get_main_type()'.

   * The method `getsubtype()' was renamed to `get_subtype()'.

   * The method `getparams()' was renamed to `get_params()'. Also,
     whereas `getparams()' returned a list of strings, `get_params()'
     returns a list of 2-tuples, effectively the key/value pairs of the
     parameters, split on the `'='' sign.

   * The method `getparam()' was renamed to `get_param()'.

   * The method `getcharsets()' was renamed to `get_charsets()'.

   * The method `getfilename()' was renamed to `get_filename()'.

   * The method `getboundary()' was renamed to `get_boundary()'.

   * The method `setboundary()' was renamed to `set_boundary()'.

   * The method `getdecodedpayload()' was removed.  To get similar
     functionality, pass the value 1 to the _decode_ flag of the
     get_payload() method.

   * The method `getpayloadastext()' was removed.  Similar
     functionality is supported by the `DecodedGenerator' class in the
     email.generator.  module.

   * The method `getbodyastext()' was removed.  You can get similar
     functionality by creating an iterator with
     `typed_subpart_iterator()' in the email.iterators.
     module.

  The `Parser' class has no differences in its public interface. It does
have some additional smarts to recognize `message/delivery-status' type
messages, which it represents as a `Message' instance containing
separate `Message' subparts for each header block in the delivery status
notification (1).

  The `Generator' class has no differences in its public interface.
There is a new class in the email.generator module though,
called `DecodedGenerator' which provides most of the functionality
previously available in the `Message.getpayloadastext()' method.

  The following modules and classes have been changed:

   * The `MIMEBase' class constructor arguments __major_ and __minor_
     have changed to __maintype_ and __subtype_ respectively.

   * The `Image' class/module has been renamed to `MIMEImage'.  The
     __minor_ argument has been renamed to __subtype_.

   * The `Text' class/module has been renamed to `MIMEText'.  The
     __minor_ argument has been renamed to __subtype_.

   * The `MessageRFC822' class/module has been renamed to
     `MIMEMessage'.  Note that an earlier version of `mimelib' called
     this class/module `RFC822', but that clashed with the Python
     standard library module rfc822 on some
     case-insensitive file systems.

     Also, the `MIMEMessage' class now represents any kind of MIME
     message with main type `message'.  It takes an optional argument
     __subtype_ which is used to set the MIME subtype.  __subtype_
     defaults to `rfc822'.

  `mimelib' provided some utility functions in its `address' and `date'
modules.  All of these functions have been moved to the
email.utils module.

  The `MsgReader' class/module has been removed.  Its functionality is
most closely supported in the `body_line_iterator()' function in the
email.iterators module.

Footnotes
.........

     (1) Delivery Status Notifications (DSN) are defined in RFC 1894
     (http://tools.ietf.org/html/rfc1894.html).

5.18.2 `json' -- JSON encoder and decoder
-----------------------------------------

New in version 2.6.

  JSON (JavaScript Object Notation) <<http://json.org>> is a subset of
JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data
interchange format.

  json exposes an API familiar to users of the standard
library marshal and pickle modules.

  Encoding basic Python object hierarchies:

    >>> import json
    >>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
    '["foo", {"bar": ["baz", null, 1.0, 2]}]'
    >>> print json.dumps("\"foo\bar")
    "\"foo\bar"
    >>> print json.dumps(u'\u1234')
    "\u1234"
    >>> print json.dumps('\\')
    "\\"
    >>> print json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True)
    {"a": 0, "b": 0, "c": 0}
    >>> from StringIO import StringIO
    >>> io = StringIO()
    >>> json.dump(['streaming API'], io)
    >>> io.getvalue()
    '["streaming API"]'

Compact encoding:

    >>> import json
    >>> json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
    '[1,2,3,{"4":5,"6":7}]'

Pretty printing:

    >>> import json
    >>> print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
    {
        "4": 5,
        "6": 7
    }

Decoding JSON:

    >>> import json
    >>> json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
    [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]
    >>> json.loads('"\\"foo\\bar"')
    u'"foo\x08ar'
    >>> from StringIO import StringIO
    >>> io = StringIO('["streaming API"]')
    >>> json.load(io)
    [u'streaming API']

Specializing JSON object decoding:

    >>> import json
    >>> def as_complex(dct):
    ...     if '__complex__' in dct:
    ...         return complex(dct['real'], dct['imag'])
    ...     return dct
    ...
    >>> json.loads('{"__complex__": true, "real": 1, "imag": 2}',
    ...     object_hook=as_complex)
    (1+2j)
    >>> import decimal
    >>> json.loads('1.1', parse_float=decimal.Decimal)
    Decimal('1.1')

Extending JSONEncoder.:

    >>> import json
    >>> class ComplexEncoder(json.JSONEncoder):
    ...     def default(self, obj):
    ...         if isinstance(obj, complex):
    ...             return [obj.real, obj.imag]
    ...         return json.JSONEncoder.default(self, obj)
    ...
    >>> dumps(2 + 1j, cls=ComplexEncoder)
    '[2.0, 1.0]'
    >>> ComplexEncoder().encode(2 + 1j)
    '[2.0, 1.0]'
    >>> list(ComplexEncoder().iterencode(2 + 1j))
    ['[', '2.0', ', ', '1.0', ']']

Using json.tool from the shell to validate and pretty-print:

    $ echo '{"json":"obj"}' | python -mjson.tool
    {
        "json": "obj"
    }
    $ echo '{ 1.2:3.4}' | python -mjson.tool
    Expecting property name: line 1 column 2 (char 2)

     Note: The JSON produced by this module's default settings is a
     subset of YAML, so it may be used as a serializer for that as well.

5.18.2.1 Basic Usage
....................

 -- Function: json.dump (obj, fp[, skipkeys[, ensure_ascii[,
          check_circular[, allow_nan[, cls[, indent[, separators[,
          encoding[, default[, **kw]]]]]]]]]])
     Serialize _obj_ as a JSON formatted stream to _fp_ (a
     `.write()'-supporting file-like object).

     If _skipkeys_ is `True' (default: `False'), then dict keys that
     are not of a basic type (see str, unicode,
     int, long, float, bool,
     `None') will be skipped instead of raising a TypeError.

     If _ensure_ascii_ is `False' (default: `True'), then some chunks
     written to _fp_ may be unicode instances, subject to
     normal Python str to unicode coercion
     rules.  Unless `fp.write()' explicitly understands unicode.
     (as in codecs.getwriter()) this is likely to
     cause an error.

     If _check_circular_ is `False' (default: `True'), then the circular
     reference check for container types will be skipped and a circular
     reference will result in an OverflowError. (or worse).

     If _allow_nan_ is `False' (default: `True'), then it will be a
     ValueError to serialize out of range float.
     values (`nan', `inf', `-inf') in strict compliance of the JSON
     specification, instead of using the JavaScript equivalents (`NaN',
     `Infinity', `-Infinity').

     If _indent_ is a non-negative integer, then JSON array elements
     and object members will be pretty-printed with that indent level.
     An indent level of 0 will only insert newlines.  `None' (the
     default) selects the most compact representation.

     If _separators_ is an `(item_separator, dict_separator)' tuple,
     then it will be used instead of the default `(', ', ': ')'
     separators.  `(',', ':')' is the most compact JSON representation.

     _encoding_ is the character encoding for str instances, default is
     UTF-8.

     _default(obj)_ is a function that should return a serializable
     version of _obj_ or raise TypeError.  The default
     simply raises TypeError.

     To use a custom JSONEncoder subclass (e.g. one that
     overrides the `default()' method to serialize additional types),
     specify it with the _cls_ kwarg; otherwise JSONEncoder
     is used.

 -- Function: json.dumps (obj[, skipkeys[, ensure_ascii[,
          check_circular[, allow_nan[, cls[, indent[, separators[,
          encoding[, default[, **kw]]]]]]]]]])
     Serialize _obj_ to a JSON formatted str.

     If _ensure_ascii_ is `False', then the return value will be a
     unicode instance.  The other arguments have the same
     meaning as in dump().

 -- Function: json.load (fp[, encoding[, cls[, object_hook[,
          parse_float[, parse_int[, parse_constant[,
          object_pairs_hook[, **kw]]]]]]]])
     Deserialize _fp_ (a `.read()'-supporting file-like object
     containing a JSON document) to a Python object.

     If the contents of _fp_ are encoded with an ASCII based encoding
     other than UTF-8 (e.g. latin-1), then an appropriate _encoding_
     name must be specified.  Encodings that are not ASCII based (such
     as UCS-2) are not allowed, and should be wrapped with
     `codecs.getreader(encoding)(fp)', or simply decoded to a
     unicode object and passed to loads().

     _object_hook_ is an optional function that will be called with the
     result of any object literal decoded (a dict).  The
     return value of _object_hook_ will be used instead of the
     dict.  This feature can be used to implement custom decoders
     (e.g. JSON-RPC class hinting).

     _object_pairs_hook_ is an optional function that will be called
     with the result of any object literal decoded with an ordered list
     of pairs.  The return value of _object_pairs_hook_ will be used
     instead of the dict.  This feature can be used to
     implement custom decoders that rely on the order that the key and
     value pairs are decoded (for example, see
     collections.OrderedDict() will remember the order of
     insertion). If _object_hook_ is also defined, the
     _object_pairs_hook_ takes priority.

     Changed in version 2.7: Added support for _object_pairs_hook_.

     _parse_float_, if specified, will be called with the string of
     every JSON float to be decoded.  By default, this is equivalent to
     `float(num_str)'.  This can be used to use another datatype or
     parser for JSON floats (e.g. see decimal.Decimal).

     _parse_int_, if specified, will be called with the string of every
     JSON int to be decoded.  By default, this is equivalent to
     `int(num_str)'.  This can be used to use another datatype or
     parser for JSON integers (e.g. see float).

     _parse_constant_, if specified, will be called with one of the
     following strings: `'-Infinity'', `'Infinity'', `'NaN'', `'null'',
     `'true'', `'false''.  This can be used to raise an exception if
     invalid JSON numbers are encountered.

     To use a custom JSONDecoder subclass, specify it with
     the `cls' kwarg; otherwise JSONDecoder is used.
     Additional keyword arguments will be passed to the constructor of
     the class.

 -- Function: json.loads (s[, encoding[, cls[, object_hook[,
          parse_float[, parse_int[, parse_constant[,
          object_pairs_hook[, **kw]]]]]]]])
     Deserialize _s_ (a str or unicode instance
     containing a JSON document) to a Python object.

     If _s_ is a str instance and is encoded with an ASCII
     based encoding other than UTF-8 (e.g. latin-1), then an
     appropriate _encoding_ name must be specified.  Encodings that are
     not ASCII based (such as UCS-2) are not allowed and should be
     decoded to unicode first.

     The other arguments have the same meaning as in load().

5.18.2.2 Encoders and decoders
..............................

 -- Class: json.JSONDecoder ([encoding[, object_hook[, parse_float[,
          parse_int[, parse_constant[, strict[,
          object_pairs_hook]]]]]]])
     Simple JSON decoder.

     Performs the following translations in decoding by default:

     JSON                Python
     --------------------------------------------
     object              dict
     array               list
     string              unicode
     number (int)        int, long
     number (real)       float
     true                True
     false               False
     null                None

     It also understands `NaN', `Infinity', and `-Infinity' as their
     corresponding `float' values, which is outside the JSON spec.

     _encoding_ determines the encoding used to interpret any
     str objects decoded by this instance (UTF-8 by default).  It
     has no effect when decoding unicode objects.

     Note that currently only encodings that are a superset of ASCII
     work, strings of other encodings should be passed in as
     unicode.

     _object_hook_, if specified, will be called with the result of
     every JSON object decoded and its return value will be used in
     place of the given dict.  This can be used to provide
     custom deserializations (e.g. to support JSON-RPC class hinting).

     _object_pairs_hook_, if specified will be called with the result
     of every JSON object decoded with an ordered list of pairs.  The
     return value of _object_pairs_hook_ will be used instead of the
     dict.  This feature can be used to implement custom
     decoders that rely on the order that the key and value pairs are
     decoded (for example, see collections.OrderedDict() will
     remember the order of insertion). If _object_hook_ is also
     defined, the _object_pairs_hook_ takes priority.

     Changed in version 2.7: Added support for _object_pairs_hook_.

     _parse_float_, if specified, will be called with the string of
     every JSON float to be decoded.  By default, this is equivalent to
     `float(num_str)'.  This can be used to use another datatype or
     parser for JSON floats (e.g. see decimal.Decimal).

     _parse_int_, if specified, will be called with the string of every
     JSON int to be decoded.  By default, this is equivalent to
     `int(num_str)'.  This can be used to use another datatype or
     parser for JSON integers (e.g. see float).

     _parse_constant_, if specified, will be called with one of the
     following strings: `'-Infinity'', `'Infinity'', `'NaN'', `'null'',
     `'true'', `'false''.  This can be used to raise an exception if
     invalid JSON numbers are encountered.

     If _strict_ is `False' (`True' is the default), then control
     characters will be allowed inside strings.  Control characters in
     this context are those with character codes in the 0-31 range,
     including `'\t'' (tab), `'\n'', `'\r'' and `'\0''.

      -- Method: decode (s)
          Return the Python representation of _s_ (a str or
          unicode instance containing a JSON document)

      -- Method: raw_decode (s)
          Decode a JSON document from _s_ (a str or
          unicode.  beginning with a JSON document) and return a
          2-tuple of the Python representation and the index in _s_
          where the document ended.

          This can be used to decode a JSON document from a string that
          may have extraneous data at the end.

 -- Class: json.JSONEncoder ([skipkeys[, ensure_ascii[,
          check_circular[, allow_nan[, sort_keys[, indent[,
          separators[, encoding[, default]]]]]]]]])
     Extensible JSON encoder for Python data structures.

     Supports the following objects and types by default:

     Python                  JSON
     --------------------------------------------
     dict                    object
     list, tuple             array
     str, unicode            string
     int, long, float        number
     True                    true
     False                   false
     None                    null

     To extend this to recognize other objects, subclass and implement a
     default() method with another method that returns a
     serializable object for `o' if possible, otherwise it should call
     the superclass implementation (to raise TypeError).

     If _skipkeys_ is `False' (the default), then it is a
     TypeError to attempt encoding of keys that are not str, int,
     long, float or None.  If _skipkeys_ is `True', such items are
     simply skipped.

     If _ensure_ascii_ is `True' (the default), the output is
     guaranteed to be str objects with all incoming unicode
     characters escaped.  If _ensure_ascii_ is `False', the output will
     be a unicode object.

     If _check_circular_ is `True' (the default), then lists, dicts,
     and custom encoded objects will be checked for circular references
     during encoding to prevent an infinite recursion (which would
     cause an OverflowError).  Otherwise, no such check
     takes place.

     If _allow_nan_ is `True' (the default), then `NaN', `Infinity', and
     `-Infinity' will be encoded as such.  This behavior is not JSON
     specification compliant, but is consistent with most JavaScript
     based encoders and decoders.  Otherwise, it will be a
     ValueError to encode such floats.

     If _sort_keys_ is `True' (default `False'), then the output of
     dictionaries will be sorted by key; this is useful for regression
     tests to ensure that JSON serializations can be compared on a
     day-to-day basis.

     If _indent_ is a non-negative integer (it is `None' by default),
     then JSON array elements and object members will be pretty-printed
     with that indent level.  An indent level of 0 will only insert
     newlines.  `None' is the most compact representation.

     If specified, _separators_ should be an `(item_separator,
     key_separator)' tuple.  The default is `(', ', ': ')'.  To get the
     most compact JSON representation, you should specify `(',', ':')'
     to eliminate whitespace.

     If specified, _default_ is a function that gets called for objects
     that can't otherwise be serialized.  It should return a JSON
     encodable version of the object or raise a TypeError.

     If _encoding_ is not `None', then all input strings will be
     transformed into unicode using that encoding prior to
     JSON-encoding.  The default is UTF-8.

      -- Method: default (o)
          Implement this method in a subclass such that it returns a
          serializable object for _o_, or calls the base implementation
          (to raise a TypeError).

          For example, to support arbitrary iterators, you could
          implement default like this:

              def default(self, o):
                 try:
                     iterable = iter(o)
                 except TypeError:
                     pass
                 else:
                     return list(iterable)
                 return JSONEncoder.default(self, o)

      -- Method: encode (o)
          Return a JSON string representation of a Python data
          structure, _o_.  For example:

              >>> JSONEncoder().encode({"foo": ["bar", "baz"]})
              '{"foo": ["bar", "baz"]}'

      -- Method: iterencode (o)
          Encode the given object, _o_, and yield each string
          representation as available.  For example:

              for chunk in JSONEncoder().iterencode(bigobject):
                  mysocket.write(chunk)

5.18.3 `mailcap' -- Mailcap file handling
-----------------------------------------

Mailcap files are used to configure how MIME-aware applications such as
mail readers and Web browsers react to files with different MIME types.
(The name "mailcap" is derived from the phrase "mail capability".)  For
example, a mailcap file might contain a line like `video/mpeg; xmpeg
%s'.  Then, if the user encounters an email message or Web document
with the MIME type `video/mpeg', `%s' will be replaced by a filename
(usually one belonging to a temporary file) and the *xmpeg* program can
be automatically started to view the file.

  The mailcap format is documented in RFC 1524
(http://tools.ietf.org/html/rfc1524.html), "A User Agent Configuration
Mechanism For Multimedia Mail Format Information," but is not an
Internet standard.  However, mailcap files are supported on most Unix
systems.

 -- Function: mailcap.findmatch (caps, MIMEtype[, key[, filename[,
          plist]]])
     Return a 2-tuple; the first element is a string containing the
     command line to be executed (which can be passed to
     os.system()), and the second element is the mailcap entry
     for a given MIME type.  If no matching MIME type can be found,
     `(None, None)' is returned.

     _key_ is the name of the field desired, which represents the type
     of activity to be performed; the default value is 'view', since in
     the  most common case you simply want to view the body of the
     MIME-typed data.  Other possible values might be 'compose' and
     'edit', if you wanted to create a new body of the given MIME type
     or alter the existing body data.  See RFC 1524
     (http://tools.ietf.org/html/rfc1524.html) for a complete list of
     these fields.

     _filename_ is the filename to be substituted for `%s' in the
     command line; the default value is `'/dev/null'' which is almost
     certainly not what you want, so usually you'll override it by
     specifying a filename.

     _plist_ can be a list containing named parameters; the default
     value is simply an empty list.  Each entry in the list must be a
     string containing the parameter name, an equals sign (`'=''), and
     the parameter's value.  Mailcap entries can contain  named
     parameters like `%{foo}', which will be replaced by the value of
     the parameter named 'foo'.  For example, if the command line
     `showpartial %{id} %{number} %{total}' was in a mailcap file, and
     _plist_ was set to `['id=1', 'number=2', 'total=3']', the
     resulting command line would be `'showpartial 1 2 3''.

     In a mailcap file, the "test" field can optionally be specified to
     test some external condition (such as the machine architecture, or
     the window system in use) to determine whether or not the mailcap
     line applies.  See findmatch().  will automatically check
     such conditions and skip the entry if the check fails.

 -- Function: mailcap.getcaps ()
     Returns a dictionary mapping MIME types to a list of mailcap file
     entries. This dictionary must be passed to the findmatch()
     function.  An entry is stored as a list of dictionaries, but
     it shouldn't be necessary to know the details of this
     representation.

     The information is derived from all of the mailcap files found on
     the system.  Settings in the user's mailcap file `$HOME/.mailcap'
     will override settings in the system mailcap files `/etc/mailcap',
     `/usr/etc/mailcap', and `/usr/local/etc/mailcap'.

  An example usage:

    >>> import mailcap
    >>> d=mailcap.getcaps()
    >>> mailcap.findmatch(d, 'video/mpeg', filename='/tmp/tmp1223')
    ('xmpeg /tmp/tmp1223', {'view': 'xmpeg %s'})

5.18.4 `mailbox' -- Manipulate mailboxes in various formats
-----------------------------------------------------------

This module defines two classes, see Mailbox and
Message, for accessing and manipulating on-disk mailboxes and the
messages they contain.  Mailbox offers a dictionary-like
mapping from keys to messages.  Message extends the
`email.Message' module's Message.  class with
format-specific state and behavior. Supported mailbox formats are
Maildir, mbox, MH, Babyl, and MMDF.

Module email.
     Represent and manipulate messages.

5.18.4.1 `Mailbox' objects
..........................

 -- Class: mailbox.Mailbox
     A mailbox, which may be inspected and modified.

     The Mailbox class defines an interface and is not
     intended to be instantiated.  Instead, format-specific subclasses
     should inherit from Mailbox and your code should
     instantiate a particular subclass.

     The Mailbox interface is dictionary-like, with small
     keys corresponding to messages. Keys are issued by the
     Mailbox instance with which they will be used and are only
     meaningful to that Mailbox.  instance. A key continues
     to identify a message even if the corresponding message is
     modified, such as by replacing it with another message.

     Messages may be added to a Mailbox instance using the
     set-like method add() and removed using a `del'
     statement or the set-like methods remove() and
     discard().

     Mailbox interface semantics differ from dictionary
     semantics in some noteworthy ways. Each time a message is
     requested, a new representation (typically a Message.
     instance) is generated based upon the current state of the
     mailbox. Similarly, when a message is added to a Mailbox
     instance, the provided message representation's contents are
     copied. In neither case is a reference to the message
     representation kept by the Mailbox instance.

     The default Mailbox iterator iterates over message
     representations, not keys as the default dictionary iterator does.
     Moreover, modification of a mailbox during iteration is safe and
     well-defined. Messages added to the mailbox after an iterator is
     created will not be seen by the iterator. Messages removed from
     the mailbox before the iterator yields them will be silently
     skipped, though using a key from an iterator may result in a
     KeyError exception if the corresponding message is
     subsequently removed.

          Warning: Be very cautious when modifying mailboxes that might
          be simultaneously changed by some other process.  The safest
          mailbox format to use for such tasks is Maildir; try to avoid
          using single-file formats such as mbox for concurrent
          writing.  If you're modifying a mailbox, you _must_ lock it by
          calling the lock() and unlock().
          methods _before_ reading any messages in the file or making
          any changes by adding or deleting a message.  Failing to lock
          the mailbox runs the risk of losing messages or corrupting
          the entire mailbox.

     Mailbox instances have the following methods:

      -- Method: add (message)
          Add _message_ to the mailbox and return the key that has been
          assigned to it.

          Parameter _message_ may be a Message instance, an
          `email.Message.Message' instance, a string, or a file-like
          object (which should be open in text mode). If _message_ is
          an instance of the appropriate format-specific Message
          subclass (e.g., if it's an mboxMessage.
          instance and this is an mbox instance), its
          format-specific information is used. Otherwise, reasonable
          defaults for format-specific information are used.

      -- Method: remove (key)
      -- Method: __delitem__ (key)
      -- Method: discard (key)
          Delete the message corresponding to _key_ from the mailbox.

          If no such message exists, a KeyError exception
          is raised if the method was called as remove().
          or __delitem__() but no exception is raised if
          the method was called as discard(). The behavior
          of discard() may be preferred if the underlying
          mailbox format supports concurrent modification by other
          processes.

      -- Method: __setitem__ (key, message)
          Replace the message corresponding to _key_ with _message_.
          Raise a KeyError exception if no message already
          corresponds to _key_.

          As with add(), parameter _message_ may be a
          Message.  instance, an `email.Message.Message'
          instance, a string, or a file-like object (which should be
          open in text mode). If _message_ is an instance of the
          appropriate format-specific Message subclass
          (e.g., if it's an mboxMessage instance and this
          is an mbox instance), its format-specific
          information is used. Otherwise, the format-specific
          information of the message that currently corresponds to
          _key_ is left unchanged.

      -- Method: iterkeys ()
      -- Method: keys ()
          Return an iterator over all keys if called as
          iterkeys() or return a list of keys if called as
          keys().

      -- Method: itervalues ()
      -- Method: __iter__ ()
      -- Method: values ()
          Return an iterator over representations of all messages if
          called as itervalues() or __iter__()
          or return a list of such representations if called as
          values(). The messages are represented as
          instances of the appropriate format-specific Message
          subclass unless a custom message factory was specified
          when the Mailbox.  instance was initialized.

               Note: The behavior of __iter__() is unlike
               that of dictionaries, which iterate over keys.

      -- Method: iteritems ()
      -- Method: items ()
          Return an iterator over (_key_, _message_) pairs, where _key_
          is a key and _message_ is a message representation, if called
          as iteritems() or return a list of such pairs if
          called as items(). The messages are represented
          as instances of the appropriate format-specific
          Message subclass unless a custom message factory was
          specified when the Mailbox instance was
          initialized.

      -- Method: get (key[, default=None])
      -- Method: __getitem__ (key)
          Return a representation of the message corresponding to
          _key_. If no such message exists, _default_ is returned if
          the method was called as get() and a
          KeyError exception is raised if the method was called
          as __getitem__(). The message is represented as
          an instance of the appropriate format-specific Message
          subclass unless a custom message factory was specified
          when the Mailbox instance was initialized.

      -- Method: get_message (key)
          Return a representation of the message corresponding to _key_
          as an instance of the appropriate format-specific
          Message subclass, or raise a KeyError.
          exception if no such message exists.

      -- Method: get_string (key)
          Return a string representation of the message corresponding
          to _key_, or raise a KeyError exception if no
          such message exists.

      -- Method: get_file (key)
          Return a file-like representation of the message
          corresponding to _key_, or raise a KeyError.
          exception if no such message exists. The file-like object
          behaves as if open in binary mode. This file should be closed
          once it is no longer needed.

               Note: Unlike other representations of messages,
               file-like representations are not necessarily
               independent of the Mailbox instance that
               created them or of the underlying mailbox. More specific
               documentation is provided by each subclass.

      -- Method: has_key (key)
      -- Method: __contains__ (key)
          Return `True' if _key_ corresponds to a message, `False'
          otherwise.

      -- Method: __len__ ()
          Return a count of messages in the mailbox.

      -- Method: clear ()
          Delete all messages from the mailbox.

      -- Method: pop (key[, default])
          Return a representation of the message corresponding to _key_
          and delete the message. If no such message exists, return
          _default_ if it was supplied or else raise a KeyError
          exception. The message is represented as an instance of
          the appropriate format-specific Message subclass
          unless a custom message factory was specified when the
          Mailbox instance was initialized.

      -- Method: popitem ()
          Return an arbitrary (_key_, _message_) pair, where _key_ is a
          key and _message_ is a message representation, and delete the
          corresponding message. If the mailbox is empty, raise a
          KeyError exception. The message is represented as an
          instance of the appropriate format-specific Message
          subclass unless a custom message factory was specified
          when the Mailbox instance was initialized.

      -- Method: update (arg)
          Parameter _arg_ should be a _key_-to-_message_ mapping or an
          iterable of (_key_, _message_) pairs. Updates the mailbox so
          that, for each given _key_ and _message_, the message
          corresponding to _key_ is set to _message_ as if by using
          __setitem__(). As with __setitem__(),
          each _key_ must already correspond to a message in the
          mailbox or else a KeyError exception will be
          raised, so in general it is incorrect for _arg_ to be a
          Mailbox instance.

               Note: Unlike with dictionaries, keyword arguments are
               not supported.

      -- Method: flush ()
          Write any pending changes to the filesystem. For some
          Mailbox.  subclasses, changes are always written
          immediately and flush() does nothing, but you
          should still make a habit of calling this method.

      -- Method: lock ()
          Acquire an exclusive advisory lock on the mailbox so that
          other processes know not to modify it. An
          ExternalClashError is raised if the lock is not
          available. The particular locking mechanisms used depend upon
          the mailbox format.  You should _always_ lock the mailbox
          before making any modifications to its contents.

      -- Method: unlock ()
          Release the lock on the mailbox, if any.

      -- Method: close ()
          Flush the mailbox, unlock it if necessary, and close any open
          files. For some Mailbox subclasses, this method
          does nothing.

5.18.4.2 `Maildir'
..................

 -- Class: mailbox.Maildir (dirname[, factory=rfc822.Message[,
          create=True]])
     A subclass of Mailbox for mailboxes in Maildir
     format. Parameter _factory_ is a callable object that accepts a
     file-like message representation (which behaves as if opened in
     binary mode) and returns a custom representation.  If _factory_ is
     `None', see MaildirMessage is used as the default message
     representation. If _create_ is `True', the mailbox is created if
     it does not exist.

     It is for historical reasons that _factory_ defaults to
     rfc822.Message.  and that _dirname_ is named as such rather
     than _path_. For a Maildir.  instance that behaves like
     instances of other Mailbox subclasses, set _factory_
     to `None'.

     Maildir is a directory-based mailbox format invented for the qmail
     mail transfer agent and now widely supported by other programs.
     Messages in a Maildir mailbox are stored in separate files within
     a common directory structure. This design allows Maildir mailboxes
     to be accessed and modified by multiple unrelated programs without
     data corruption, so file locking is unnecessary.

     Maildir mailboxes contain three subdirectories, namely: `tmp',
     `new', and `cur'. Messages are created momentarily in the `tmp'
     subdirectory and then moved to the `new' subdirectory to finalize
     delivery. A mail user agent may subsequently move the message to
     the `cur' subdirectory and store information about the state of
     the message in a special "info" section appended to its file name.

     Folders of the style introduced by the Courier mail transfer agent
     are also supported. Any subdirectory of the main mailbox is
     considered a folder if `'.'' is the first character in its name.
     Folder names are represented by Maildir without the
     leading `'.''. Each folder is itself a Maildir mailbox but should
     not contain other folders. Instead, a logical nesting is indicated
     using `'.'' to delimit levels, e.g., "Archived.2005.07".

          Note: The Maildir specification requires the use of a colon
          (`':'') in certain message file names. However, some
          operating systems do not permit this character in file names,
          If you wish to use a Maildir-like format on such an operating
          system, you should specify another character to use instead.
          The exclamation point (`'!'') is a popular choice. For
          example:

              import mailbox
              mailbox.Maildir.colon = '!'

          The `colon' attribute may also be set on a per-instance basis.

     Maildir instances have all of the methods of
     Mailbox in addition to the following:

      -- Method: list_folders ()
          Return a list of the names of all folders.

      -- Method: get_folder (folder)
          Return a Maildir instance representing the folder
          whose name is _folder_. A NoSuchMailboxError.
          exception is raised if the folder does not exist.

      -- Method: add_folder (folder)
          Create a folder whose name is _folder_ and return a
          Maildir.  instance representing it.

      -- Method: remove_folder (folder)
          Delete the folder whose name is _folder_. If the folder
          contains any messages, a NotEmptyError exception
          will be raised and the folder will not be deleted.

      -- Method: clean ()
          Delete temporary files from the mailbox that have not been
          accessed in the last 36 hours. The Maildir specification says
          that mail-reading programs should do this occasionally.

     Some Mailbox methods implemented by Maildir
     deserve special remarks:

      -- Method: add (message)
      -- Method: __setitem__ (key, message)
      -- Method: update (arg)
               Warning: These methods generate unique file names based
               upon the current process ID. When using multiple
               threads, undetected name clashes may occur and cause
               corruption of the mailbox unless threads are coordinated
               to avoid using these methods to manipulate the same
               mailbox simultaneously.

      -- Method: flush ()
          All changes to Maildir mailboxes are immediately applied, so
          this method does nothing.

      -- Method: lock ()
      -- Method: unlock ()
          Maildir mailboxes do not support (or require) locking, so
          these methods do nothing.

      -- Method: close ()
          Maildir instances do not keep any open files and
          the underlying mailboxes do not support locking, so this
          method does nothing.

      -- Method: get_file (key)
          Depending upon the host platform, it may not be possible to
          modify or remove the underlying message while the returned
          file remains open.

maildir man page from qmail (http://www.qmail.org/man/man5/maildir.html)
     The original specification of the format.

Using maildir format (http://cr.yp.to/proto/maildir.html)
     Notes on Maildir by its inventor. Includes an updated
     name-creation scheme and details on "info" semantics.

maildir man page from Courier (http://www.courier-mta.org/maildir.html)
     Another specification of the format. Describes a common extension
     for supporting folders.

5.18.4.3 `mbox'
...............

 -- Class: mailbox.mbox (path[, factory=None[, create=True]])
     A subclass of Mailbox for mailboxes in mbox format.
     Parameter _factory_ is a callable object that accepts a file-like
     message representation (which behaves as if opened in binary mode)
     and returns a custom representation. If _factory_ is `None', see
     mboxMessage is used as the default message representation.
     If _create_ is `True', the mailbox is created if it does not exist.

     The mbox format is the classic format for storing mail on Unix
     systems. All messages in an mbox mailbox are stored in a single
     file with the beginning of each message indicated by a line whose
     first five characters are "From ".

     Several variations of the mbox format exist to address perceived
     shortcomings in the original. In the interest of compatibility,
     see mbox implements the original format, which is
     sometimes referred to as _mboxo_. This means that the
     `Content-Length' header, if present, is ignored and that any
     occurrences of "From " at the beginning of a line in a message
     body are transformed to ">From " when storing the message,
     although occurrences of ">From " are not transformed to "From "
     when reading the message.

     Some Mailbox methods implemented by mbox.
     deserve special remarks:

      -- Method: get_file (key)
          Using the file after calling `flush()' or `close()' on the
          mbox instance may yield unpredictable results or
          raise an exception.

      -- Method: lock ()
      -- Method: unlock ()
          Three locking mechanisms are used--dot locking and, if
          available, the `flock()' and `lockf()' system calls.

mbox man page from qmail (http://www.qmail.org/man/man5/mbox.html)
     A specification of the format and its variations.

mbox man page from tin (http://www.tin.org/bin/man.cgi?section=5&topic=mbox)
     Another specification of the format, with details on locking.

Configuring Netscape Mail on Unix: Why The Content-Length Format is Bad (http://www.jwz.org/doc/content-length.html)
     An argument for using the original mbox format rather than a
     variation.

"mbox" is a family of several mutually incompatible mailbox formats (http://homepages.tesco.net./~J.deBoynePollard/FGA/mail-mbox-formats.html)
     A history of mbox variations.

5.18.4.4 `MH'
.............

 -- Class: mailbox.MH (path[, factory=None[, create=True]])
     A subclass of Mailbox for mailboxes in MH format.
     Parameter _factory_ is a callable object that accepts a file-like
     message representation (which behaves as if opened in binary mode)
     and returns a custom representation. If _factory_ is `None', see
     MHMessage is used as the default message representation. If
     _create_ is `True', the mailbox is created if it does not exist.

     MH is a directory-based mailbox format invented for the MH Message
     Handling System, a mail user agent. Each message in an MH mailbox
     resides in its own file. An MH mailbox may contain other MH
     mailboxes (called _folders_) in addition to messages. Folders may
     be nested indefinitely. MH mailboxes also support _sequences_,
     which are named lists used to logically group messages without
     moving them to sub-folders. Sequences are defined in a file called
     `.mh_sequences' in each folder.

     The MH class manipulates MH mailboxes, but it does
     not attempt to emulate all of *mh*'s behaviors. In particular, it
     does not modify and is not affected by the `context' or
     `.mh_profile' files that are used by *mh* to store its state and
     configuration.

     MH instances have all of the methods of
     Mailbox in addition to the following:

      -- Method: list_folders ()
          Return a list of the names of all folders.

      -- Method: get_folder (folder)
          Return an MH instance representing the folder
          whose name is _folder_. A NoSuchMailboxError.
          exception is raised if the folder does not exist.

      -- Method: add_folder (folder)
          Create a folder whose name is _folder_ and return an
          MH instance representing it.

      -- Method: remove_folder (folder)
          Delete the folder whose name is _folder_. If the folder
          contains any messages, a NotEmptyError exception
          will be raised and the folder will not be deleted.

      -- Method: get_sequences ()
          Return a dictionary of sequence names mapped to key lists. If
          there are no sequences, the empty dictionary is returned.

      -- Method: set_sequences (sequences)
          Re-define the sequences that exist in the mailbox based upon
          _sequences_, a dictionary of names mapped to key lists, like
          returned by get_sequences().

      -- Method: pack ()
          Rename messages in the mailbox as necessary to eliminate gaps
          in numbering.  Entries in the sequences list are updated
          correspondingly.

               Note: Already-issued keys are invalidated by this
               operation and should not be subsequently used.

     Some Mailbox methods implemented by MH.
     deserve special remarks:

      -- Method: remove (key)
      -- Method: __delitem__ (key)
      -- Method: discard (key)
          These methods immediately delete the message. The MH
          convention of marking a message for deletion by prepending a
          comma to its name is not used.

      -- Method: lock ()
      -- Method: unlock ()
          Three locking mechanisms are used--dot locking and, if
          available, the `flock()' and `lockf()' system calls. For MH
          mailboxes, locking the mailbox means locking the
          `.mh_sequences' file and, only for the duration of any
          operations that affect them, locking individual message files.

      -- Method: get_file (key)
          Depending upon the host platform, it may not be possible to
          remove the underlying message while the returned file remains
          open.

      -- Method: flush ()
          All changes to MH mailboxes are immediately applied, so this
          method does nothing.

      -- Method: close ()
          MH instances do not keep any open files, so this
          method is equivalent to unlock().

nmh - Message Handling System (http://www.nongnu.org/nmh/)
     Home page of *nmh*, an updated version of the original *mh*.

MH & nmh: Email for Users & Programmers (http://rand-mh.sourceforge.net/book/)
     A GPL-licensed book on *mh* and *nmh*, with some information on
     the mailbox format.

5.18.4.5 `Babyl'
................

 -- Class: mailbox.Babyl (path[, factory=None[, create=True]])
     A subclass of Mailbox for mailboxes in Babyl format.
     Parameter _factory_ is a callable object that accepts a file-like
     message representation (which behaves as if opened in binary mode)
     and returns a custom representation.  If _factory_ is `None',
     see BabylMessage is used as the default message
     representation. If _create_ is `True', the mailbox is created if
     it does not exist.

     Babyl is a single-file mailbox format used by the Rmail mail user
     agent included with Emacs. The beginning of a message is indicated
     by a line containing the two characters Control-Underscore
     (`'\037'') and Control-L (`'\014''). The end of a message is
     indicated by the start of the next message or, in the case of the
     last message, a line containing a Control-Underscore (`'\037'')
     character.

     Messages in a Babyl mailbox have two sets of headers, original
     headers and so-called visible headers. Visible headers are
     typically a subset of the original headers that have been
     reformatted or abridged to be more attractive. Each message in a
     Babyl mailbox also has an accompanying list of _labels_, or short
     strings that record extra information about the message, and a
     list of all user-defined labels found in the mailbox is kept in
     the Babyl options section.

     Babyl instances have all of the methods of
     Mailbox in addition to the following:

      -- Method: get_labels ()
          Return a list of the names of all user-defined labels used in
          the mailbox.

               Note: The actual messages are inspected to determine
               which labels exist in the mailbox rather than consulting
               the list of labels in the Babyl options section, but the
               Babyl section is updated whenever the mailbox is
               modified.

     Some Mailbox methods implemented by Babyl
     deserve special remarks:

      -- Method: get_file (key)
          In Babyl mailboxes, the headers of a message are not stored
          contiguously with the body of the message. To generate a
          file-like representation, the headers and body are copied
          together into a StringIO instance (from the
          StringIO module), which has an API identical to that of
          a file. As a result, the file-like object is truly
          independent of the underlying mailbox but does not save
          memory compared to a string representation.

      -- Method: lock ()
      -- Method: unlock ()
          Three locking mechanisms are used--dot locking and, if
          available, the `flock()' and `lockf()' system calls.

Format of Version 5 Babyl Files (http://quimby.gnus.org/notes/BABYL)
     A specification of the Babyl format.

Reading Mail with Rmail (http://www.gnu.org/software/emacs/manual/html_node/emacs/Rmail.html)
     The Rmail manual, with some information on Babyl semantics.

5.18.4.6 `MMDF'
...............

 -- Class: mailbox.MMDF (path[, factory=None[, create=True]])
     A subclass of Mailbox for mailboxes in MMDF format.
     Parameter _factory_ is a callable object that accepts a file-like
     message representation (which behaves as if opened in binary mode)
     and returns a custom representation. If _factory_ is `None', see
     MMDFMessage is used as the default message representation.
     If _create_ is `True', the mailbox is created if it does not exist.

     MMDF is a single-file mailbox format invented for the Multichannel
     Memorandum Distribution Facility, a mail transfer agent. Each
     message is in the same form as an mbox message but is bracketed
     before and after by lines containing four Control-A (`'\001'')
     characters. As with the mbox format, the beginning of each message
     is indicated by a line whose first five characters are "From ",
     but additional occurrences of "From " are not transformed to
     ">From " when storing messages because the extra message separator
     lines prevent mistaking such occurrences for the starts of
     subsequent messages.

     Some Mailbox methods implemented by MMDF.
     deserve special remarks:

      -- Method: get_file (key)
          Using the file after calling `flush()' or `close()' on the
          MMDF instance may yield unpredictable results or
          raise an exception.

      -- Method: lock ()
      -- Method: unlock ()
          Three locking mechanisms are used--dot locking and, if
          available, the `flock()' and `lockf()' system calls.

mmdf man page from tin (http://www.tin.org/bin/man.cgi?section=5&topic=mmdf)
     A specification of MMDF format from the documentation of tin, a
     newsreader.

MMDF (http://en.wikipedia.org/wiki/MMDF)
     A Wikipedia article describing the Multichannel Memorandum
     Distribution Facility.

5.18.4.7 `Message' objects
..........................

 -- Class: mailbox.Message ([message])
     A subclass of the `email.Message' module's Message.
     Subclasses of mailbox.Message add
     mailbox-format-specific state and behavior.

     If _message_ is omitted, the new instance is created in a default,
     empty state.  If _message_ is an `email.Message.Message' instance,
     its contents are copied; furthermore, any format-specific
     information is converted insofar as possible if _message_ is a
     Message instance. If _message_ is a string or a file,
     it should contain an RFC 2822
     (http://tools.ietf.org/html/rfc2822.html)-compliant message, which
     is read and parsed.

     The format-specific state and behaviors offered by subclasses
     vary, but in general it is only the properties that are not
     specific to a particular mailbox that are supported (although
     presumably the properties are specific to a particular mailbox
     format). For example, file offsets for single-file mailbox formats
     and file names for directory-based mailbox formats are not
     retained, because they are only applicable to the original
     mailbox. But state such as whether a message has been read by the
     user or marked as important is retained, because it applies to the
     message itself.

     There is no requirement that Message instances be
     used to represent messages retrieved using Mailbox.
     instances. In some situations, the time and memory required to
     generate Message representations might not not
     acceptable. For such situations, see Mailbox instances
     also offer string and file-like representations, and a custom
     message factory may be specified when a Mailbox.
     instance is initialized.

5.18.4.8 `MaildirMessage'
.........................

 -- Class: mailbox.MaildirMessage ([message])
     A message with Maildir-specific behaviors. Parameter _message_ has
     the same meaning as with the Message constructor.

     Typically, a mail user agent application moves all of the messages
     in the `new' subdirectory to the `cur' subdirectory after the
     first time the user opens and closes the mailbox, recording that
     the messages are old whether or not they've actually been read.
     Each message in `cur' has an "info" section added to its file name
     to store information about its state.  (Some mail readers may also
     add an "info" section to messages in `new'.)  The "info" section
     may take one of two forms: it may contain "2," followed by a list
     of standardized flags (e.g., "2,FR") or it may contain "1,"
     followed by so-called experimental information. Standard flags for
     Maildir messages are as follows:

     Flag       Meaning       Explanation
     --------------------------------------------------------------
     D          Draft         Under composition
     F          Flagged       Marked as important
     P          Passed        Forwarded, resent, or bounced
     R          Replied       Replied to
     S          Seen          Read
     T          Trashed       Marked for subsequent deletion

     MaildirMessage instances offer the following methods:

      -- Method: get_subdir ()
          Return either "new" (if the message should be stored in the
          `new' subdirectory) or "cur" (if the message should be stored
          in the `cur' subdirectory).

               Note: A message is typically moved from `new' to `cur'
               after its mailbox has been accessed, whether or not the
               message is has been read. A message `msg' has been read
               if `"S" in msg.get_flags()' is `True'.

      -- Method: set_subdir (subdir)
          Set the subdirectory the message should be stored in.
          Parameter _subdir_ must be either "new" or "cur".

      -- Method: get_flags ()
          Return a string specifying the flags that are currently set.
          If the message complies with the standard Maildir format, the
          result is the concatenation in alphabetical order of zero or
          one occurrence of each of `'D'', `'F'', `'P'', `'R'', `'S'',
          and `'T''. The empty string is returned if no flags are set
          or if "info" contains experimental semantics.

      -- Method: set_flags (flags)
          Set the flags specified by _flags_ and unset all others.

      -- Method: add_flag (flag)
          Set the flag(s) specified by _flag_ without changing other
          flags. To add more than one flag at a time, _flag_ may be a
          string of more than one character. The current "info" is
          overwritten whether or not it contains experimental
          information rather than flags.

      -- Method: remove_flag (flag)
          Unset the flag(s) specified by _flag_ without changing other
          flags. To remove more than one flag at a time, _flag_ maybe a
          string of more than one character.  If "info" contains
          experimental information rather than flags, the current
          "info" is not modified.

      -- Method: get_date ()
          Return the delivery date of the message as a floating-point
          number representing seconds since the epoch.

      -- Method: set_date (date)
          Set the delivery date of the message to _date_, a
          floating-point number representing seconds since the epoch.

      -- Method: get_info ()
          Return a string containing the "info" for a message. This is
          useful for accessing and modifying "info" that is
          experimental (i.e., not a list of flags).

      -- Method: set_info (info)
          Set "info" to _info_, which should be a string.

When a MaildirMessage instance is created based upon an
mboxMessage or MMDFMessage instance, the
`Status' and `X-Status' headers are omitted and the following
conversions take place:

Resulting state          mboxMessage or MMDFMessage.
                         state
----------------------------------------------------------------------------
"cur" subdirectory       O flag
F flag                   F flag
R flag                   A flag
S flag                   R flag
T flag                   D flag

  When a MaildirMessage instance is created based upon an
MHMessage instance, the following conversions take place:

Resulting state                     MHMessage state
-------------------------------------------------------------------
"cur" subdirectory                  "unseen" sequence
"cur" subdirectory and S flag       no "unseen" sequence
F flag                              "flagged" sequence
R flag                              "replied" sequence

  When a MaildirMessage instance is created based upon a
BabylMessage instance, the following conversions take
place:

Resulting state                     BabylMessage state
------------------------------------------------------------------------
"cur" subdirectory                  "unseen" label
"cur" subdirectory and S flag       no "unseen" label
P flag                              "forwarded" or "resent" label
R flag                              "answered" label
T flag                              "deleted" label

5.18.4.9 `mboxMessage'
......................

 -- Class: mailbox.mboxMessage ([message])
     A message with mbox-specific behaviors. Parameter _message_ has
     the same meaning as with the Message constructor.

     Messages in an mbox mailbox are stored together in a single file.
     The sender's envelope address and the time of delivery are
     typically stored in a line beginning with "From " that is used to
     indicate the start of a message, though there is considerable
     variation in the exact format of this data among mbox
     implementations. Flags that indicate the state of the message,
     such as whether it has been read or marked as important, are
     typically stored in `Status' and `X-Status' headers.

     Conventional flags for mbox messages are as follows:

     Flag       Meaning        Explanation
     ---------------------------------------------------------------
     R          Read           Read
     O          Old            Previously detected by MUA
     D          Deleted        Marked for subsequent deletion
     F          Flagged        Marked as important
     A          Answered       Replied to

     The "R" and "O" flags are stored in the `Status' header, and the
     "D", "F", and "A" flags are stored in the `X-Status' header. The
     flags and headers typically appear in the order mentioned.

     mboxMessage instances offer the following methods:

      -- Method: get_from ()
          Return a string representing the "From " line that marks the
          start of the message in an mbox mailbox. The leading "From "
          and the trailing newline are excluded.

      -- Method: set_from (from_[, time_=None])
          Set the "From " line to _from__, which should be specified
          without a leading "From " or trailing newline. For
          convenience, _time__ may be specified and will be formatted
          appropriately and appended to _from__. If _time__ is
          specified, it should be a `struct_time' instance, a tuple
          suitable for passing to time.strftime(), or `True'
          (to use time.gmtime()).

      -- Method: get_flags ()
          Return a string specifying the flags that are currently set.
          If the message complies with the conventional format, the
          result is the concatenation in the following order of zero or
          one occurrence of each of `'R'', `'O'', `'D'', `'F'', and
          `'A''.

      -- Method: set_flags (flags)
          Set the flags specified by _flags_ and unset all others.
          Parameter _flags_ should be the concatenation in any order of
          zero or more occurrences of each of `'R'', `'O'', `'D'',
          `'F'', and `'A''.

      -- Method: add_flag (flag)
          Set the flag(s) specified by _flag_ without changing other
          flags. To add more than one flag at a time, _flag_ may be a
          string of more than one character.

      -- Method: remove_flag (flag)
          Unset the flag(s) specified by _flag_ without changing other
          flags. To remove more than one flag at a time, _flag_ maybe a
          string of more than one character.

When an mboxMessage instance is created based upon a
MaildirMessage instance, a "From " line is generated based upon
the MaildirMessage instance's delivery date, and the
following conversions take place:

Resulting state       MaildirMessage state
----------------------------------------------------------
R flag                S flag
O flag                "cur" subdirectory
D flag                T flag
F flag                F flag
A flag                R flag

  When an mboxMessage instance is created based upon an
MHMessage instance, the following conversions take place:

Resulting state         MHMessage state
-------------------------------------------------------
R flag and O flag       no "unseen" sequence
O flag                  "unseen" sequence
F flag                  "flagged" sequence
A flag                  "replied" sequence

  When an mboxMessage instance is created based upon a
BabylMessage instance, the following conversions take
place:

Resulting state         BabylMessage state
----------------------------------------------------------
R flag and O flag       no "unseen" label
O flag                  "unseen" label
D flag                  "deleted" label
A flag                  "answered" label

  When a Message instance is created based upon an
MMDFMessage.  instance, the "From " line is copied and all flags
directly correspond:

Resulting state       MMDFMessage state
-------------------------------------------------------
R flag                R flag
O flag                O flag
D flag                D flag
F flag                F flag
A flag                A flag

5.18.4.10 `MHMessage'
.....................

 -- Class: mailbox.MHMessage ([message])
     A message with MH-specific behaviors. Parameter _message_ has the
     same meaning as with the Message constructor.

     MH messages do not support marks or flags in the traditional
     sense, but they do support sequences, which are logical groupings
     of arbitrary messages. Some mail reading programs (although not
     the standard *mh* and *nmh*) use sequences in much the same way
     flags are used with other formats, as follows:

     Sequence       Explanation
     --------------------------------------------------------------
     unseen         Not read, but previously detected by MUA
     replied        Replied to
     flagged        Marked as important

     MHMessage instances offer the following methods:

      -- Method: get_sequences ()
          Return a list of the names of sequences that include this
          message.

      -- Method: set_sequences (sequences)
          Set the list of sequences that include this message.

      -- Method: add_sequence (sequence)
          Add _sequence_ to the list of sequences that include this
          message.

      -- Method: remove_sequence (sequence)
          Remove _sequence_ from the list of sequences that include
          this message.

When an MHMessage instance is created based upon a
MaildirMessage instance, the following conversions take place:

Resulting state          MaildirMessage state
-------------------------------------------------------------
"unseen" sequence        no S flag
"replied" sequence       R flag
"flagged" sequence       F flag

  When an MHMessage instance is created based upon an
mboxMessage or MMDFMessage instance, the
`Status' and `X-Status' headers are omitted and the following
conversions take place:

Resulting state          mboxMessage or MMDFMessage.
                         state
----------------------------------------------------------------------------
"unseen" sequence        no R flag
"replied" sequence       A flag
"flagged" sequence       F flag

  When an MHMessage instance is created based upon a
BabylMessage instance, the following conversions take place:

Resulting state          BabylMessage state
-----------------------------------------------------------
"unseen" sequence        "unseen" label
"replied" sequence       "answered" label

5.18.4.11 `BabylMessage'
........................

 -- Class: mailbox.BabylMessage ([message])
     A message with Babyl-specific behaviors. Parameter _message_ has
     the same meaning as with the Message constructor.

     Certain message labels, called _attributes_, are defined by
     convention to have special meanings. The attributes are as follows:

     Label           Explanation
     ---------------------------------------------------------------
     unseen          Not read, but previously detected by MUA
     deleted         Marked for subsequent deletion
     filed           Copied to another file or mailbox
     answered        Replied to
     forwarded       Forwarded
     edited          Modified by the user
     resent          Resent

     By default, Rmail displays only visible headers. The
     BabylMessage.  class, though, uses the original headers
     because they are more complete. Visible headers may be accessed
     explicitly if desired.

     BabylMessage instances offer the following methods:

      -- Method: get_labels ()
          Return a list of labels on the message.

      -- Method: set_labels (labels)
          Set the list of labels on the message to _labels_.

      -- Method: add_label (label)
          Add _label_ to the list of labels on the message.

      -- Method: remove_label (label)
          Remove _label_ from the list of labels on the message.

      -- Method: get_visible ()
          Return an Message instance whose headers are the
          message's visible headers and whose body is empty.

      -- Method: set_visible (visible)
          Set the message's visible headers to be the same as the
          headers in _message_.  Parameter _visible_ should be a
          Message instance, an `email.Message.Message' instance,
          a string, or a file-like object (which should be open in text
          mode).

      -- Method: update_visible ()
          When a BabylMessage instance's original headers
          are modified, the visible headers are not automatically
          modified to correspond. This method updates the visible
          headers as follows: each visible header with a corresponding
          original header is set to the value of the original header,
          each visible header without a corresponding original header
          is removed, and any of `Date', `From', `Reply-To', `To',
          `CC', and `Subject' that are present in the original headers
          but not the visible headers are added to the visible headers.

When a BabylMessage instance is created based upon a
MaildirMessage instance, the following conversions take place:

Resulting state         MaildirMessage state
------------------------------------------------------------
"unseen" label          no S flag
"deleted" label         T flag
"answered" label        R flag
"forwarded" label       P flag

  When a BabylMessage instance is created based upon an
mboxMessage or MMDFMessage instance, the
`Status' and `X-Status' headers are omitted and the following
conversions take place:

Resulting state        mboxMessage or MMDFMessage.
                       state
--------------------------------------------------------------------------
"unseen" label         no R flag
"deleted" label        D flag
"answered" label       A flag

  When a BabylMessage instance is created based upon an
MHMessage instance, the following conversions take place:

Resulting state        MHMessage state
------------------------------------------------------
"unseen" label         "unseen" sequence
"answered" label       "replied" sequence

5.18.4.12 `MMDFMessage'
.......................

 -- Class: mailbox.MMDFMessage ([message])
     A message with MMDF-specific behaviors. Parameter _message_ has
     the same meaning as with the Message constructor.

     As with message in an mbox mailbox, MMDF messages are stored with
     the sender's address and the delivery date in an initial line
     beginning with "From ".  Likewise, flags that indicate the state
     of the message are typically stored in `Status' and `X-Status'
     headers.

     Conventional flags for MMDF messages are identical to those of
     mbox message and are as follows:

     Flag       Meaning        Explanation
     ---------------------------------------------------------------
     R          Read           Read
     O          Old            Previously detected by MUA
     D          Deleted        Marked for subsequent deletion
     F          Flagged        Marked as important
     A          Answered       Replied to

     The "R" and "O" flags are stored in the `Status' header, and the
     "D", "F", and "A" flags are stored in the `X-Status' header. The
     flags and headers typically appear in the order mentioned.

     MMDFMessage instances offer the following methods,
     which are identical to those offered by mboxMessage.:

      -- Method: get_from ()
          Return a string representing the "From " line that marks the
          start of the message in an mbox mailbox. The leading "From "
          and the trailing newline are excluded.

      -- Method: set_from (from_[, time_=None])
          Set the "From " line to _from__, which should be specified
          without a leading "From " or trailing newline. For
          convenience, _time__ may be specified and will be formatted
          appropriately and appended to _from__. If _time__ is
          specified, it should be a `struct_time' instance, a tuple
          suitable for passing to time.strftime(), or `True'
          (to use time.gmtime()).

      -- Method: get_flags ()
          Return a string specifying the flags that are currently set.
          If the message complies with the conventional format, the
          result is the concatenation in the following order of zero or
          one occurrence of each of `'R'', `'O'', `'D'', `'F'', and
          `'A''.

      -- Method: set_flags (flags)
          Set the flags specified by _flags_ and unset all others.
          Parameter _flags_ should be the concatenation in any order of
          zero or more occurrences of each of `'R'', `'O'', `'D'',
          `'F'', and `'A''.

      -- Method: add_flag (flag)
          Set the flag(s) specified by _flag_ without changing other
          flags. To add more than one flag at a time, _flag_ may be a
          string of more than one character.

      -- Method: remove_flag (flag)
          Unset the flag(s) specified by _flag_ without changing other
          flags. To remove more than one flag at a time, _flag_ maybe a
          string of more than one character.

When an MMDFMessage instance is created based upon a
MaildirMessage instance, a "From " line is generated based upon
the MaildirMessage instance's delivery date, and the
following conversions take place:

Resulting state       MaildirMessage state
----------------------------------------------------------
R flag                S flag
O flag                "cur" subdirectory
D flag                T flag
F flag                F flag
A flag                R flag

  When an MMDFMessage instance is created based upon an
MHMessage instance, the following conversions take place:

Resulting state         MHMessage state
-------------------------------------------------------
R flag and O flag       no "unseen" sequence
O flag                  "unseen" sequence
F flag                  "flagged" sequence
A flag                  "replied" sequence

  When an MMDFMessage instance is created based upon a
BabylMessage instance, the following conversions take
place:

Resulting state         BabylMessage state
----------------------------------------------------------
R flag and O flag       no "unseen" label
O flag                  "unseen" label
D flag                  "deleted" label
A flag                  "answered" label

  When an MMDFMessage instance is created based upon an
mboxMessage instance, the "From " line is copied and all
flags directly correspond:

Resulting state       mboxMessage state
-------------------------------------------------------
R flag                R flag
O flag                O flag
D flag                D flag
F flag                F flag
A flag                A flag

5.18.4.13 Exceptions
....................

The following exception classes are defined in the mailbox.
module:

 -- Exception: mailbox.Error
     The based class for all other module-specific exceptions.

 -- Exception: mailbox.NoSuchMailboxError
     Raised when a mailbox is expected but is not found, such as when
     instantiating a Mailbox subclass with a path that
     does not exist (and with the _create_ parameter set to `False'),
     or when opening a folder that does not exist.

 -- Exception: mailbox.NotEmptyError
     Raised when a mailbox is not empty but is expected to be, such as
     when deleting a folder that contains messages.

 -- Exception: mailbox.ExternalClashError
     Raised when some mailbox-related condition beyond the control of
     the program causes it to be unable to proceed, such as when
     failing to acquire a lock that another program already holds a
     lock, or when a uniquely-generated file name already exists.

 -- Exception: mailbox.FormatError
     Raised when the data in a file cannot be parsed, such as when an
     MH.  instance attempts to read a corrupted
     `.mh_sequences' file.

5.18.4.14 Deprecated classes and methods
........................................

Deprecated since version 2.6.

  Older versions of the mailbox module do not support
modification of mailboxes, such as adding or removing message, and do
not provide classes to represent format-specific message properties.
For backward compatibility, the older mailbox classes are still
available, but the newer classes should be used in preference to them.
The old classes will be removed in Python 3.0.

  Older mailbox objects support only iteration and provide a single
public method:

 -- Method: oldmailbox.next ()
     Return the next message in the mailbox, created with the optional
     _factory_ argument passed into the mailbox object's constructor.
     By default this is an rfc822.Message object (see the
     rfc822 module).  Depending on the mailbox
     implementation the _fp_ attribute of this object may be a true file
     object or a class instance simulating a file object, taking care
     of things like message boundaries if multiple mail messages are
     contained in a single file, etc.  If no more messages are
     available, this method returns `None'.

  Most of the older mailbox classes have names that differ from the
current mailbox class names, except for Maildir. For this
reason, the new Maildir class defines a `next()' method and
its constructor differs slightly from those of the other new mailbox
classes.

  The older mailbox classes whose names are not the same as their newer
counterparts are as follows:

 -- Class: mailbox.UnixMailbox (fp[, factory])
     Access to a classic Unix-style mailbox, where all messages are
     contained in a single file and separated by `From' (a.k.a.
     `From_') lines.  The file object _fp_ points to the mailbox file.
     The optional _factory_ parameter is a callable that should create
     new message objects.  _factory_ is called with one argument, _fp_
     by the `next()' method of the mailbox object.  The default is the
     rfc822.Message class (see the rfc822.
     module - and the note below).

          Note: For reasons of this module's internal implementation,
          you will probably want to open the _fp_ object in binary
          mode.  This is especially important on Windows.

     For maximum portability, messages in a Unix-style mailbox are
     separated by any line that begins exactly with the string `'From
     '' (note the trailing space) if preceded by exactly two newlines.
     Because of the wide-range of variations in practice, nothing else
     on the `From_' line should be considered.  However, the current
     implementation doesn't check for the leading two newlines.  This is
     usually fine for most applications.

     The UnixMailbox class implements a more strict
     version of `From_' line checking, using a regular expression that
     usually correctly matched `From_' delimiters.  It considers
     delimiter line to be separated by `From name time' lines.  For
     maximum portability, use the PortableUnixMailbox.
     class instead.  This class is identical to UnixMailbox
     except that individual messages are separated by only `From'
     lines.

 -- Class: mailbox.PortableUnixMailbox (fp[, factory])
     A less-strict version of UnixMailbox, which considers
     only the `From' at the beginning of the line separating messages.
     The "_name_ _time_" portion of the From line is ignored, to
     protect against some variations that are observed in practice.
     This works since lines in the message which begin with `'From ''
     are quoted by mail handling software at delivery-time.

 -- Class: mailbox.MmdfMailbox (fp[, factory])
     Access an MMDF-style mailbox, where all messages are contained in
     a single file and separated by lines consisting of 4 control-A
     characters.  The file object _fp_ points to the mailbox file.
     Optional _factory_ is as with the UnixMailbox class.

 -- Class: mailbox.MHMailbox (dirname[, factory])
     Access an MH mailbox, a directory with each message in a separate
     file with a numeric name. The name of the mailbox directory is
     passed in _dirname_.  _factory_ is as with the UnixMailbox
     class.

 -- Class: mailbox.BabylMailbox (fp[, factory])
     Access a Babyl mailbox, which is similar to an MMDF mailbox.  In
     Babyl format, each message has two sets of headers, the _original_
     headers and the _visible_ headers.  The original headers appear
     before a line containing only `'*** EOOH ***''
     (End-Of-Original-Headers) and the visible headers appear after the
     `EOOH' line.  Babyl-compliant mail readers will show you only the
     visible headers, and BabylMailbox objects will return
     messages containing only the visible headers.  You'll have to do
     your own parsing of the mailbox file to get at the original
     headers.  Mail messages start with the EOOH line and end with a
     line containing only `'\037\014''.  _factory_ is as with the
     UnixMailbox class.

  If you wish to use the older mailbox classes with the email
module rather than the deprecated rfc822 module, you
can do so as follows:

    import email
    import email.Errors
    import mailbox

    def msgfactory(fp):
        try:
            return email.message_from_file(fp)
        except email.Errors.MessageParseError:
            # Don't return None since that will
            # stop the mailbox iterator
            return ''

    mbox = mailbox.UnixMailbox(fp, msgfactory)

Alternatively, if you know your mailbox contains only well-formed MIME
messages, you can simplify this to:

    import email
    import mailbox

    mbox = mailbox.UnixMailbox(fp, email.message_from_file)

5.18.4.15 Examples
..................

A simple example of printing the subjects of all messages in a mailbox
that seem interesting:

    import mailbox
    for message in mailbox.mbox('~/mbox'):
        subject = message['subject']       # Could possibly be None.
        if subject and 'python' in subject.lower():
            print subject

To copy all mail from a Babyl mailbox to an MH mailbox, converting all
of the format-specific information that can be converted:

    import mailbox
    destination = mailbox.MH('~/Mail')
    destination.lock()
    for message in mailbox.Babyl('~/RMAIL'):
        destination.add(mailbox.MHMessage(message))
    destination.flush()
    destination.unlock()

This example sorts mail from several mailing lists into different
mailboxes, being careful to avoid mail corruption due to concurrent
modification by other programs, mail loss due to interruption of the
program, or premature termination due to malformed messages in the
mailbox:

    import mailbox
    import email.Errors

    list_names = ('python-list', 'python-dev', 'python-bugs')

    boxes = dict((name, mailbox.mbox('~/email/%s' % name)) for name in list_names)
    inbox = mailbox.Maildir('~/Maildir', factory=None)

    for key in inbox.iterkeys():
        try:
            message = inbox[key]
        except email.Errors.MessageParseError:
            continue                # The message is malformed. Just leave it.

        for name in list_names:
            list_id = message['list-id']
            if list_id and name in list_id:
                # Get mailbox to use
                box = boxes[name]

                # Write copy to disk before removing original.
                # If there's a crash, you might duplicate a message, but
                # that's better than losing a message completely.
                box.lock()
                box.add(message)
                box.flush()
                box.unlock()

                # Remove original message
                inbox.lock()
                inbox.discard(key)
                inbox.flush()
                inbox.unlock()
                break               # Found destination, so stop looking.

    for box in boxes.itervalues():
        box.close()

5.18.5 `mhlib' -- Access to MH mailboxes
----------------------------------------

Deprecated since version 2.6: The mhlib module has been
removed in Python 3.0. Use the mailbox instead.

  The mhlib module provides a Python interface to MH
folders and their contents.

  The module contains three basic classes, see MH, which
represents a particular collection of folders, see Folder,
which represents a single folder, and Message, which
represents a single message.

 -- Class: mhlib.MH ([path[, profile]])
     MH represents a collection of MH folders.

 -- Class: mhlib.Folder (mh, name)
     The Folder class represents a single folder and its
     messages.

 -- Class: mhlib.Message (folder, number[, name])
     Message objects represent individual messages in a
     folder.  The Message class is derived from
     mimetools.Message.

5.18.5.1 MH Objects
...................

MH instances have the following methods:

 -- Method: MH.error (format[, ...])
     Print an error message - can be overridden.

 -- Method: MH.getprofile (key)
     Return a profile entry (`None' if not set).

 -- Method: MH.getpath ()
     Return the mailbox pathname.

 -- Method: MH.getcontext ()
     Return the current folder name.

 -- Method: MH.setcontext (name)
     Set the current folder name.

 -- Method: MH.listfolders ()
     Return a list of top-level folders.

 -- Method: MH.listallfolders ()
     Return a list of all folders.

 -- Method: MH.listsubfolders (name)
     Return a list of direct subfolders of the given folder.

 -- Method: MH.listallsubfolders (name)
     Return a list of all subfolders of the given folder.

 -- Method: MH.makefolder (name)
     Create a new folder.

 -- Method: MH.deletefolder (name)
     Delete a folder - must have no subfolders.

 -- Method: MH.openfolder (name)
     Return a new open folder object.

5.18.5.2 Folder Objects
.......................

Folder instances represent open folders and have the
following methods:

 -- Method: Folder.error (format[, ...])
     Print an error message - can be overridden.

 -- Method: Folder.getfullname ()
     Return the folder's full pathname.

 -- Method: Folder.getsequencesfilename ()
     Return the full pathname of the folder's sequences file.

 -- Method: Folder.getmessagefilename (n)
     Return the full pathname of message _n_ of the folder.

 -- Method: Folder.listmessages ()
     Return a list of messages in the folder (as numbers).

 -- Method: Folder.getcurrent ()
     Return the current message number.

 -- Method: Folder.setcurrent (n)
     Set the current message number to _n_.

 -- Method: Folder.parsesequence (seq)
     Parse msgs syntax into list of messages.

 -- Method: Folder.getlast ()
     Get last message, or `0' if no messages are in the folder.

 -- Method: Folder.setlast (n)
     Set last message (internal use only).

 -- Method: Folder.getsequences ()
     Return dictionary of sequences in folder.  The sequence names are
     used  as keys, and the values are the lists of message numbers in
     the sequences.

 -- Method: Folder.putsequences (dict)
     Return dictionary of sequences in folder name: list.

 -- Method: Folder.removemessages (list)
     Remove messages in list from folder.

 -- Method: Folder.refilemessages (list, tofolder)
     Move messages in list to other folder.

 -- Method: Folder.movemessage (n, tofolder, ton)
     Move one message to a given destination in another folder.

 -- Method: Folder.copymessage (n, tofolder, ton)
     Copy one message to a given destination in another folder.

5.18.5.3 Message Objects
........................

The Message class adds one method to those of
mimetools.Message.:

 -- Method: Message.openmessage (n)
     Return a new open message object (costs a file descriptor).

5.18.6 `mimetools' -- Tools for parsing MIME messages
-----------------------------------------------------

Deprecated since version 2.3: The email package should be
used in preference to the mimetools.  module.  This module
is present only to maintain backward compatibility, and it has been
removed in 3.x.

  This module defines a subclass of the rfc822 module's
Message.  class and a number of utility functions that are
useful for the manipulation for MIME multipart or encoded message.

  It defines the following items:

 -- Class: mimetools.Message (fp[, seekable])
     Return a new instance of the Message class.  This is
     a subclass of the rfc822.Message class, with some
     additional methods (see below).  The _seekable_ argument has the
     same meaning as for rfc822.Message.

 -- Function: mimetools.choose_boundary ()
     Return a unique string that has a high likelihood of being usable
     as a part boundary.  The string has the form
     `'hostipaddr.uid.pid.timestamp.random''.

 -- Function: mimetools.decode (input, output, encoding)
     Read data encoded using the allowed MIME _encoding_ from open file
     object _input_ and write the decoded data to open file object
     _output_.  Valid values for _encoding_ include `'base64'',
     `'quoted-printable'', `'uuencode'', `'x-uuencode'', `'uue'',
     `'x-uue'', `'7bit'', and  `'8bit''.  Decoding messages encoded in
     `'7bit'' or `'8bit'' has no effect.  The input is simply copied to
     the output.

 -- Function: mimetools.encode (input, output, encoding)
     Read data from open file object _input_ and write it encoded using
     the allowed MIME _encoding_ to open file object _output_. Valid
     values for _encoding_ are the same as for decode().

 -- Function: mimetools.copyliteral (input, output)
     Read lines from open file _input_ until EOF and write them to open
     file _output_.

 -- Function: mimetools.copybinary (input, output)
     Read blocks until EOF from open file _input_ and write them to
     open file _output_.  The block size is currently fixed at 8192.

Module email.
     Comprehensive email handling package; supersedes the
     mimetools module.

Module rfc822.
     Provides the base class for mimetools.Message.

Module multifile.
     Support for reading files which contain distinct parts, such as
     MIME data.

<http://faqs.cs.uu.nl/na-dir/mail/mime-faq/.html>
     The MIME Frequently Asked Questions document.  For an overview of
     MIME, see the answer to question 1.1 in Part 1 of this document.

5.18.6.1 Additional Methods of Message Objects
..............................................

The Message class defines the following methods in
addition to the rfc822.Message methods:

 -- Method: Message.getplist ()
     Return the parameter list of the `Content-Type' header. This is a
     list of strings.  For parameters of the form `key=value', _key_ is
     converted to lower case but _value_ is not.  For example, if the
     message contains the header `Content-type: text/html; spam=1;
     Spam=2; Spam' then getplist().  will return the Python
     list `['spam=1', 'spam=2', 'Spam']'.

 -- Method: Message.getparam (name)
     Return the _value_ of the first parameter (as returned by
     getplist()) of the form `name=value' for the given _name_.
     If _value_ is surrounded by quotes of the form '`<'...`>'' or
     '`"'...`"'', these are removed.

 -- Method: Message.getencoding ()
     Return the encoding specified in the `Content-Transfer-Encoding'
     message header.  If no such header exists, return `'7bit''.  The
     encoding is converted to lower case.

 -- Method: Message.gettype ()
     Return the message type (of the form `type/subtype') as specified
     in the `Content-Type' header.  If no such header exists, return
     `'text/plain''.  The type is converted to lower case.

 -- Method: Message.getmaintype ()
     Return the main type as specified in the `Content-Type' header.  If
     no such header exists, return `'text''.  The main type is
     converted to lower case.

 -- Method: Message.getsubtype ()
     Return the subtype as specified in the `Content-Type' header.  If
     no such header exists, return `'plain''.  The subtype is converted
     to lower case.

5.18.7 `mimetypes' -- Map filenames to MIME types
-------------------------------------------------

The mimetypes module converts between a filename or URL and
the MIME type associated with the filename extension.  Conversions are
provided from filename to MIME type and from MIME type to filename
extension; encodings are not supported for the latter conversion.

  The module provides one class and a number of convenience functions.
The functions are the normal interface to this module, but some
applications may be interested in the class as well.

  The functions described below provide the primary interface for this
module.  If the module has not been initialized, they will call
init() if they rely on the information init() sets
up.

 -- Function: mimetypes.guess_type (filename[, strict])
     Guess the type of a file based on its filename or URL, given by
     _filename_.  The return value is a tuple `(type, encoding)' where
     _type_ is `None' if the type can't be guessed (missing or unknown
     suffix) or a string of the form `'type/subtype'', usable for a
     MIME `content-type' header.

     _encoding_ is `None' for no encoding or the name of the program
     used to encode (e.g. *compress* or *gzip*). The encoding is
     suitable for use as a `Content-Encoding' header, _not_ as a
     `Content-Transfer-Encoding' header. The mappings are table driven.
     Encoding suffixes are case sensitive; type suffixes are first
     tried case sensitively, then case insensitively.

     Optional _strict_ is a flag specifying whether the list of known
     MIME types is limited to only the official types registered with
     IANA (http://www.iana.org/assignments/media-types/)are recognized.
     When _strict_ is true (the default), only the IANA types are
     supported; when _strict_ is false, some additional non-standard
     but commonly used MIME types are also recognized.

 -- Function: mimetypes.guess_all_extensions (type[, strict])
     Guess the extensions for a file based on its MIME type, given by
     _type_. The return value is a list of strings giving all possible
     filename extensions, including the leading dot (`'.'').  The
     extensions are not guaranteed to have been associated with any
     particular data stream, but would be mapped to the MIME type
     _type_ by guess_type().

     Optional _strict_ has the same meaning as with the
     guess_type() function.

 -- Function: mimetypes.guess_extension (type[, strict])
     Guess the extension for a file based on its MIME type, given by
     _type_. The return value is a string giving a filename extension,
     including the leading dot (`'.'').  The extension is not
     guaranteed to have been associated with any particular data
     stream, but would be mapped to the  MIME type _type_ by
     guess_type().  If no extension can be guessed for _type_,
     `None' is returned.

     Optional _strict_ has the same meaning as with the
     guess_type() function.

  Some additional functions and data items are available for
controlling the behavior of the module.

 -- Function: mimetypes.init ([files])
     Initialize the internal data structures.  If given, _files_ must
     be a sequence of file names which should be used to augment the
     default type map.  If omitted, the file names to use are taken
     from knownfiles.; on Windows, the current registry
     settings are loaded.  Each file named in _files_ or
     knownfiles takes precedence over those named before it.
     Calling init() repeatedly is allowed.

     Changed in version 2.7: Previously, Windows registry settings were
     ignored.

 -- Function: mimetypes.read_mime_types (filename)
     Load the type map given in the file _filename_, if it exists.  The
     type map is returned as a dictionary mapping filename extensions,
     including the leading dot (`'.''), to strings of the form
     `'type/subtype''.  If the file _filename_ does not exist or cannot
     be read, `None' is returned.

 -- Function: mimetypes.add_type (type, ext[, strict])
     Add a mapping from the mimetype _type_ to the extension _ext_.
     When the extension is already known, the new type will replace the
     old one. When the type is already known the extension will be
     added to the list of known extensions.

     When _strict_ is True (the default), the mapping will added to the
     official MIME types, otherwise to the non-standard ones.

 -- Data: mimetypes.inited
     Flag indicating whether or not the global data structures have
     been initialized.  This is set to true by init().

 -- Data: mimetypes.knownfiles
     List of type map file names commonly installed.  These files are
     typically named `mime.types' and are installed in different
     locations by different packages.

 -- Data: mimetypes.suffix_map
     Dictionary mapping suffixes to suffixes.  This is used to allow
     recognition of encoded files for which the encoding and the type
     are indicated by the same extension.  For example, the `.tgz'
     extension is mapped to `.tar.gz' to allow the encoding and type to
     be recognized separately.

 -- Data: mimetypes.encodings_map
     Dictionary mapping filename extensions to encoding types.

 -- Data: mimetypes.types_map
     Dictionary mapping filename extensions to MIME types.

 -- Data: mimetypes.common_types
     Dictionary mapping filename extensions to non-standard, but
     commonly found MIME types.

  The MimeTypes class may be useful for applications which
may want more than one MIME-type database:

 -- Class: mimetypes.MimeTypes ([filenames])
     This class represents a MIME-types database.  By default, it
     provides access to the same database as the rest of this module.
     The initial database is a copy of that provided by the module, and
     may be extended by loading additional `mime.types'-style files
     into the database using the read() or readfp()
     methods.  The mapping dictionaries may also be cleared before
     loading additional data if the default data is not desired.

     The optional _filenames_ parameter can be used to cause additional
     files to be loaded "on top" of the default database.

     New in version 2.2.

  An example usage of the module:

    >>> import mimetypes
    >>> mimetypes.init()
    >>> mimetypes.knownfiles
    ['/etc/mime.types', '/etc/httpd/mime.types', ... ]
    >>> mimetypes.suffix_map['.tgz']
    '.tar.gz'
    >>> mimetypes.encodings_map['.gz']
    'gzip'
    >>> mimetypes.types_map['.tgz']
    'application/x-tar-gz'

5.18.7.1 MimeTypes Objects
..........................

MimeTypes instances provide an interface which is very
like that of the mimetypes module.

 -- Attribute: MimeTypes.suffix_map
     Dictionary mapping suffixes to suffixes.  This is used to allow
     recognition of encoded files for which the encoding and the type
     are indicated by the same extension.  For example, the `.tgz'
     extension is mapped to `.tar.gz' to allow the encoding and type to
     be recognized separately.  This is initially a copy of the global
     `suffix_map' defined in the module.

 -- Attribute: MimeTypes.encodings_map
     Dictionary mapping filename extensions to encoding types.  This is
     initially a copy of the global `encodings_map' defined in the
     module.

 -- Attribute: MimeTypes.types_map
     Dictionary mapping filename extensions to MIME types.  This is
     initially a copy of the global `types_map' defined in the module.

 -- Attribute: MimeTypes.common_types
     Dictionary mapping filename extensions to non-standard, but
     commonly found MIME types.  This is initially a copy of the global
     `common_types' defined in the module.

 -- Method: MimeTypes.guess_extension (type[, strict])
     Similar to the guess_extension() function, using the
     tables stored as part of the object.

 -- Method: MimeTypes.guess_all_extensions (type[, strict])
     Similar to the guess_all_extensions() function, using
     the tables stored as part of the object.

 -- Method: MimeTypes.guess_type (url[, strict])
     Similar to the guess_type() function, using the
     tables stored as part of the object.

 -- Method: MimeTypes.read (path)
     Load MIME information from a file named _path_.  This uses
     readfp() to parse the file.

 -- Method: MimeTypes.readfp (file)
     Load MIME type information from an open file.  The file must have
     the format of the standard `mime.types' files.

 -- Method: MimeTypes.read_windows_registry ()
     Load MIME type information from the Windows registry.
     Availability: Windows.

     New in version 2.7.

5.18.8 `MimeWriter' -- Generic MIME file writer
-----------------------------------------------

Deprecated since version 2.3: The email package should be
used in preference to the MimeWriter.  module.  This module
is present only to maintain backward compatibility.

  This module defines the class MimeWriter.  The
MimeWriter.  class implements a basic formatter for creating MIME
multi-part files.  It doesn't seek around the output file nor does it
use large amounts of buffer space. You must write the parts out in the
order that they should occur in the final file. See MimeWriter.
does buffer the headers you add, allowing you to rearrange their order.

 -- Class: MimeWriter.MimeWriter (fp)
     Return a new instance of the MimeWriter class.  The
     only argument passed, _fp_, is a file object to be used for
     writing. Note that a StringIO object could also be
     used.

5.18.8.1 MimeWriter Objects
...........................

MimeWriter instances have the following methods:

 -- Method: MimeWriter.addheader (key, value[, prefix])
     Add a header line to the MIME message. The _key_ is the name of
     the header, where the _value_ obviously provides the value of the
     header. The optional argument _prefix_ determines where the header
     is inserted; `0' means append at the end, `1' is insert at the
     start. The default is to append.

 -- Method: MimeWriter.flushheaders ()
     Causes all headers accumulated so far to be written out (and
     forgotten). This is useful if you don't need a body part at all,
     e.g. for a subpart of type `message/rfc822' that's (mis)used to
     store some header-like information.

 -- Method: MimeWriter.startbody (ctype[, plist[, prefix]])
     Returns a file-like object which can be used to write to the body
     of the message.  The content-type is set to the provided _ctype_,
     and the optional parameter _plist_ provides additional parameters
     for the content-type declaration. _prefix_ functions as in
     addheader() except that the default is to insert at the
     start.

 -- Method: MimeWriter.startmultipartbody (subtype[, boundary[, plist[,
          prefix]]])
     Returns a file-like object which can be used to write to the body
     of the message.  Additionally, this method initializes the
     multi-part code, where _subtype_ provides the multipart subtype,
     _boundary_ may provide a user-defined boundary specification, and
     _plist_ provides optional parameters for the subtype. _prefix_
     functions as in startbody().  Subparts should be
     created using nextpart().

 -- Method: MimeWriter.nextpart ()
     Returns a new instance of MimeWriter which represents
     an individual part in a multipart message.  This may be used to
     write the  part as well as used for creating recursively complex
     multipart messages. The message must first be initialized with
     startmultipartbody() before using nextpart().

 -- Method: MimeWriter.lastpart ()
     This is used to designate the last part of a multipart message,
     and should _always_ be used when writing multipart messages.

5.18.9 `mimify' -- MIME processing of mail messages
---------------------------------------------------

Deprecated since version 2.3: The email package should be
used in preference to the mimify.  module.  This module is
present only to maintain backward compatibility.

  The mimify module defines two functions to convert mail
messages to and from MIME format.  The mail message can be either a
simple message or a so-called multipart message.  Each part is treated
separately. Mimifying (a part of) a message entails encoding the
message as quoted-printable if it contains any characters that cannot
be represented using 7-bit ASCII.  Unmimifying (a part of) a message
entails undoing the quoted-printable encoding.  Mimify and unmimify are
especially useful when a message has to be edited before being sent.
Typical use would be:

    unmimify message
    edit message
    mimify message
    send message

The modules defines the following user-callable functions and
user-settable variables:

 -- Function: mimify.mimify (infile, outfile)
     Copy the message in _infile_ to _outfile_, converting parts to
     quoted-printable and adding MIME mail headers when necessary.
     _infile_ and _outfile_ can be file objects (actually, any object
     that has a readline() method (for _infile_) or a
     `write()' method (for _outfile_)) or strings naming the files. If
     _infile_ and _outfile_ are both strings, they may have the same
     value.

 -- Function: mimify.unmimify (infile, outfile[, decode_base64])
     Copy the message in _infile_ to _outfile_, decoding all
     quoted-printable parts.  _infile_ and _outfile_ can be file
     objects (actually, any object that has a readline().
     method (for _infile_) or a `write()' method (for _outfile_)) or
     strings naming the files.  If _infile_ and _outfile_ are both
     strings, they may have the same value. If the _decode_base64_
     argument is provided and tests true, any parts that are coded in
     the base64 encoding are decoded as well.

 -- Function: mimify.mime_decode_header (line)
     Return a decoded version of the encoded header line in _line_.
     This only supports the ISO 8859-1 charset (Latin-1).

 -- Function: mimify.mime_encode_header (line)
     Return a MIME-encoded version of the header line in _line_.

 -- Data: mimify.MAXLEN
     By default, a part will be encoded as quoted-printable when it
     contains any non-ASCII characters (characters with the 8th bit
     set), or if there are any lines longer than MAXLEN.
     characters (default value 200).

 -- Data: mimify.CHARSET
     When not specified in the mail headers, a character set must be
     filled in.  The string used is stored in CHARSET, and
     the default value is ISO-8859-1 (also known as Latin1 (latin-one)).

  This module can also be used from the command line.  Usage is as
follows:

    mimify.py -e [-l length] [infile [outfile]]
    mimify.py -d [-b] [infile [outfile]]

to encode (mimify) and decode (unmimify) respectively.  _infile_
defaults to standard input, _outfile_ defaults to standard output. The
same file can be specified for input and output.

  If the *-l* option is given when encoding, if there are any lines
longer than the specified _length_, the containing part will be encoded.

  If the *-b* option is given when decoding, any base64 parts will be
decoded as well.

Module quopri.
     Encode and decode MIME quoted-printable files.

5.18.10 `multifile' -- Support for files containing distinct parts
------------------------------------------------------------------

Deprecated since version 2.5: The email package should be
used in preference to the multifile.  module. This module is
present only to maintain backward compatibility.

  The MultiFile object enables you to treat sections of a
text file as file-like input objects, with `''' being returned by
readline() when a given delimiter pattern is encountered.  The
defaults of this class are designed to make it useful for parsing MIME
multipart messages, but by subclassing it and overriding methods  it
can be easily adapted for more general use.

 -- Class: multifile.MultiFile (fp[, seekable])
     Create a multi-file.  You must instantiate this class with an
     input object argument for the MultiFile instance to
     get lines from, such as a file object returned by open().

     MultiFile only ever looks at the input object's
     readline(), seek() and tell().
     methods, and the latter two are only needed if you want random
     access to the individual MIME parts. To use MultiFile.
     on a non-seekable stream object, set the optional _seekable_
     argument to false; this will prevent using the input object's
     seek() and tell() methods.

  It will be useful to know that in MultiFile.'s view of
the world, text is composed of three kinds of lines: data,
section-dividers, and end-markers.  MultiFile is designed to support
parsing of messages that may have multiple nested message parts, each
with its own pattern for section-divider and end-marker lines.

Module email.
     Comprehensive email handling package; supersedes the
     multifile module.

5.18.10.1 MultiFile Objects
...........................

A MultiFile instance has the following methods:

 -- Method: MultiFile.readline (str)
     Read a line.  If the line is data (not a section-divider or
     end-marker or real EOF) return it.  If the line matches the
     most-recently-stacked boundary, return `''' and set `self.last' to
     1 or 0 according as the match is or is not an end-marker.  If the
     line matches any other stacked boundary, raise an error.  On
     encountering end-of-file on the underlying stream object, the
     method raises `Error' unless all boundaries have been popped.

 -- Method: MultiFile.readlines (str)
     Return all lines remaining in this part as a list of strings.

 -- Method: MultiFile.read ()
     Read all lines, up to the next section.  Return them as a single
     (multiline) string.  Note that this doesn't take a size argument!

 -- Method: MultiFile.seek (pos[, whence])
     Seek.  Seek indices are relative to the start of the current
     section. The _pos_ and _whence_ arguments are interpreted as for a
     file seek.

 -- Method: MultiFile.tell ()
     Return the file position relative to the start of the current
     section.

 -- Method: MultiFile.next ()
     Skip lines to the next section (that is, read lines until a
     section-divider or end-marker has been consumed).  Return true if
     there is such a section, false if an end-marker is seen.
     Re-enable the most-recently-pushed boundary.

 -- Method: MultiFile.is_data (str)
     Return true if _str_ is data and false if it might be a section
     boundary.  As written, it tests for a prefix other than `'-'`-''
     at start of line (which all MIME boundaries have) but it is
     declared so it can be overridden in derived classes.

     Note that this test is used intended as a fast guard for the real
     boundary tests; if it always returns false it will merely slow
     processing, not cause it to fail.

 -- Method: MultiFile.push (str)
     Push a boundary string.  When a decorated version of this boundary
     is found as an input line, it will be interpreted as a
     section-divider  or end-marker (depending on the decoration, see RFC
     2045 (http://tools.ietf.org/html/rfc2045.html)).  All subsequent
     reads will return the empty string to indicate end-of-file, until
     a call to pop().  removes the boundary a or
     next() call reenables it.

     It is possible to push more than one boundary.  Encountering the
     most-recently-pushed boundary will return EOF; encountering any
     other boundary will raise an error.

 -- Method: MultiFile.pop ()
     Pop a section boundary.  This boundary will no longer be
     interpreted as EOF.

 -- Method: MultiFile.section_divider (str)
     Turn a boundary into a section-divider line.  By default, this
     method prepends `'--'' (which MIME section boundaries have) but it
     is declared so it can be overridden in derived classes.  This
     method need not append LF or CR-LF, as comparison with the result
     ignores trailing whitespace.

 -- Method: MultiFile.end_marker (str)
     Turn a boundary string into an end-marker line.  By default, this
     method prepends `'--'' and appends `'--'' (like a MIME-multipart
     end-of-message marker) but it is declared so it can be overridden
     in derived classes.  This method need not append LF or CR-LF, as
     comparison with the result ignores trailing whitespace.

  Finally, see MultiFile instances have two public instance
variables:

 -- Attribute: MultiFile.level
     Nesting depth of the current part.

 -- Attribute: MultiFile.last
     True if the last end-of-file was for an end-of-message marker.

5.18.10.2 `MultiFile' Example
.............................

    import mimetools
    import multifile
    import StringIO

    def extract_mime_part_matching(stream, mimetype):
        """Return the first element in a multipart MIME message on stream
        matching mimetype."""

        msg = mimetools.Message(stream)
        msgtype = msg.gettype()
        params = msg.getplist()

        data = StringIO.StringIO()
        if msgtype[:10] == "multipart/":

            file = multifile.MultiFile(stream)
            file.push(msg.getparam("boundary"))
            while file.next():
                submsg = mimetools.Message(file)
                try:
                    data = StringIO.StringIO()
                    mimetools.decode(file, data, submsg.getencoding())
                except ValueError:
                    continue
                if submsg.gettype() == mimetype:
                    break
            file.pop()
        return data.getvalue()

5.18.11 `rfc822' -- Parse RFC 2822 mail headers
-----------------------------------------------

Deprecated since version 2.3: The email package should be
used in preference to the rfc822.  module.  This module is
present only to maintain backward compatibility, and has been removed
in 3.0.

  This module defines a class, see Message, which represents an
"email message" as defined by the Internet standard RFC 2822
(http://tools.ietf.org/html/rfc2822.html). (1)  Such messages consist
of a collection of message headers, and a message body.  This module
also defines a helper class AddressList for parsing RFC
2822 (http://tools.ietf.org/html/rfc2822.html) addresses.  Please refer
to the RFC for information on the specific syntax of RFC 2822
(http://tools.ietf.org/html/rfc2822.html) messages.

  The mailbox module provides classes  to read mailboxes
produced by various end-user mail programs.

 -- Class: rfc822.Message (file[, seekable])
     A Message instance is instantiated with an input
     object as parameter.  Message relies only on the input object
     having a readline() method; in particular, ordinary
     file objects qualify.  Instantiation reads headers from the input
     object up to a delimiter line (normally a blank line) and stores
     them in the instance.  The message body, following the headers, is
     not consumed.

     This class can work with any input object that supports a
     readline().  method.  If the input object has seek and tell
     capability, the rewindbody() method will work; also,
     illegal lines will be pushed back onto the input stream.  If the
     input object lacks seek but has an `unread()' method that can push
     back a line of input, see Message will use that to push
     back illegal lines.  Thus this class can be used to parse messages
     coming from a buffered stream.

     The optional _seekable_ argument is provided as a workaround for
     certain stdio libraries in which `tell()' discards buffered data
     before discovering that the `lseek()' system call doesn't work.
     For maximum portability, you should set the seekable argument to
     zero to prevent that initial `tell()' when passing in an
     unseekable object such as a file object created from a socket
     object.

     Input lines as read from the file may either be terminated by
     CR-LF or by a single linefeed; a terminating CR-LF is replaced by
     a single linefeed before the line is stored.

     All header matching is done independent of upper or lower case;
     e.g.  `m['From']', `m['from']' and `m['FROM']' all yield the same
     result.

 -- Class: rfc822.AddressList (field)
     You may instantiate the AddressList helper class
     using a single string parameter, a comma-separated list of RFC
     2822 (http://tools.ietf.org/html/rfc2822.html) addresses to be
     parsed.  (The parameter `None' yields an empty list.)

 -- Function: rfc822.quote (str)
     Return a new string with backslashes in _str_ replaced by two
     backslashes and double quotes replaced by backslash-double quote.

 -- Function: rfc822.unquote (str)
     Return a new string which is an _unquoted_ version of _str_. If
     _str_ ends and begins with double quotes, they are stripped off.
     Likewise if _str_ ends and begins with angle brackets, they are
     stripped off.

 -- Function: rfc822.parseaddr (address)
     Parse _address_, which should be the value of some
     address-containing field such as `To' or `Cc', into its
     constituent "realname" and "email address" parts. Returns a tuple
     of that information, unless the parse fails, in which case a
     2-tuple `(None, None)' is returned.

 -- Function: rfc822.dump_address_pair (pair)
     The inverse of parseaddr(), this takes a 2-tuple of
     the form `(realname, email_address)' and returns the string value
     suitable for a `To' or `Cc' header.  If the first element of
     _pair_ is false, then the second element is returned unmodified.

 -- Function: rfc822.parsedate (date)
     Attempts to parse a date according to the rules in RFC 2822
     (http://tools.ietf.org/html/rfc2822.html). however, some mailers
     don't follow that format as specified, so parsedate().
     tries to guess correctly in such cases.  _date_ is a string
     containing an RFC 2822 (http://tools.ietf.org/html/rfc2822.html)
     date, such as  `'Mon, 20 Nov 1995 19:12:08 -0500''.  If it
     succeeds in parsing the date, see parsedate() returns a
     9-tuple that can be passed directly to time.mktime().;
     otherwise `None' will be returned.  Note that indexes 6, 7, and 8
     of the result tuple are not usable.

 -- Function: rfc822.parsedate_tz (date)
     Performs the same function as parsedate(), but returns
     either `None' or a 10-tuple; the first 9 elements make up a tuple
     that can be passed directly to time.mktime(), and the
     tenth is the offset of the date's timezone from UTC (which is the
     official term for Greenwich Mean Time).  (Note that the sign of
     the timezone offset is the opposite of the sign of the
     `time.timezone' variable for the same timezone; the latter
     variable follows the POSIX standard while this module follows RFC
     2822 (http://tools.ietf.org/html/rfc2822.html).)  If the input
     string has no timezone, the last element of the tuple returned is
     `None'.  Note that indexes 6, 7, and 8 of the result tuple are not
     usable.

 -- Function: rfc822.mktime_tz (tuple)
     Turn a 10-tuple as returned by parsedate_tz() into a
     UTC timestamp.  If the timezone item in the tuple is `None',
     assume local time.  Minor deficiency: this first interprets the
     first 8 elements as a local time and then compensates for the
     timezone difference; this may yield a slight error around daylight
     savings time switch dates.  Not enough to worry about for common
     use.

Module email.
     Comprehensive email handling package; supersedes the rfc822
     module.

Module mailbox.
     Classes to read various mailbox formats produced  by end-user mail
     programs.

Module mimetools.
     Subclass of rfc822.Message that handles MIME encoded
     messages.

5.18.11.1 Message Objects
.........................

A Message instance has the following methods:

 -- Method: Message.rewindbody ()
     Seek to the start of the message body.  This only works if the
     file object is seekable.

 -- Method: Message.isheader (line)
     Returns a line's canonicalized fieldname (the dictionary key that
     will be used to index it) if the line is a legal RFC 2822
     (http://tools.ietf.org/html/rfc2822.html) header; otherwise returns
     `None' (implying that parsing should stop here and the line be
     pushed back on the input stream).  It is sometimes useful to
     override this method in a subclass.

 -- Method: Message.islast (line)
     Return true if the given line is a delimiter on which Message
     should stop.  The delimiter line is consumed, and the file
     object's read location positioned immediately after it.  By
     default this method just checks that the line is blank, but you
     can override it in a subclass.

 -- Method: Message.iscomment (line)
     Return `True' if the given line should be ignored entirely, just
     skipped. By default this is a stub that always returns `False',
     but you can override it in a subclass.

 -- Method: Message.getallmatchingheaders (name)
     Return a list of lines consisting of all headers matching _name_,
     if any.  Each physical line, whether it is a continuation line or
     not, is a separate list item.  Return the empty list if no header
     matches _name_.

 -- Method: Message.getfirstmatchingheader (name)
     Return a list of lines comprising the first header matching
     _name_, and its continuation line(s), if any.  Return `None' if
     there is no header matching _name_.

 -- Method: Message.getrawheader (name)
     Return a single string consisting of the text after the colon in
     the first header matching _name_.  This includes leading
     whitespace, the trailing linefeed, and internal linefeeds and
     whitespace if there any continuation line(s) were present.  Return
     `None' if there is no header matching _name_.

 -- Method: Message.getheader (name[, default])
     Return a single string consisting of the last header matching
     _name_, but strip leading and trailing whitespace.  Internal
     whitespace is not stripped.  The optional _default_ argument can be
     used to specify a different default to be returned when there is
     no header matching _name_; it defaults to `None'.  This is the
     preferred way to get parsed headers.

 -- Method: Message.get (name[, default])
     An alias for getheader(), to make the interface more
     compatible  with regular dictionaries.

 -- Method: Message.getaddr (name)
     Return a pair `(full name, email address)' parsed from the string
     returned by `getheader(name)'.  If no header matching _name_
     exists, return `(None, None)'; otherwise both the full name and
     the address are (possibly empty) strings.

     Example: If _m_'s first `From' header contains the string
     `'jack@cwi.nl (Jack Jansen)'', then `m.getaddr('From')' will yield
     the pair `('Jack Jansen', 'jack@cwi.nl')'. If the header contained
     `'Jack Jansen <jack@cwi.nl>'' instead, it would yield the exact
     same result.

 -- Method: Message.getaddrlist (name)
     This is similar to `getaddr(list)', but parses a header containing
     a list of email addresses (e.g. a `To' header) and returns a list
     of `(full name, email address)' pairs (even if there was only one
     address in the header).  If there is no header matching _name_,
     return an empty list.

     If multiple headers exist that match the named header (e.g. if
     there are several `Cc' headers), all are parsed for addresses. Any
     continuation lines the named headers contain are also parsed.

 -- Method: Message.getdate (name)
     Retrieve a header using getheader() and parse it into
     a 9-tuple compatible with time.mktime().; note that
     fields 6, 7, and 8  are not usable.  If there is no header
     matching _name_, or it is unparsable, return `None'.

     Date parsing appears to be a black art, and not all mailers adhere
     to the standard.  While it has been tested and found correct on a
     large collection of email from many sources, it is still possible
     that this function may occasionally yield an incorrect result.

 -- Method: Message.getdate_tz (name)
     Retrieve a header using getheader() and parse it into
     a 10-tuple; the first 9 elements will make a tuple compatible with
     time.mktime(), and the 10th is a number giving the
     offset of the date's timezone from UTC.  Note that fields 6, 7,
     and 8  are not usable.  Similarly to getdate(), if
     there is no header matching _name_, or it is unparsable, return
     `None'.

  Message instances also support a limited mapping
interface. In particular: `m[name]' is like `m.getheader(name)' but
raises KeyError.  if there is no matching header; and
`len(m)', `m.get(name[, default])', `name in m', `m.keys()',
`m.values()' `m.items()', and `m.setdefault(name[, default])' act as
expected, with the one difference that `setdefault()' uses an empty
string as the default value.  Message instances also
support the mapping writable interface `m[name] = value' and `del
m[name]'.  Message objects do not support the `clear()',
see copy(), `popitem()', or `update()' methods of the mapping
interface.  (Support for `get()' and `setdefault()' was only added in
Python 2.2.)

  Finally, see Message instances have some public instance
variables:

 -- Attribute: Message.headers
     A list containing the entire set of header lines, in the order in
     which they were read (except that setitem calls may disturb this
     order). Each line contains a trailing newline.  The blank line
     terminating the headers is not contained in the list.

 -- Attribute: Message.fp
     The file or file-like object passed at instantiation time.  This
     can be used to read the message content.

 -- Attribute: Message.unixfrom
     The Unix `From' line, if the message had one, or an empty string.
     This is needed to regenerate the message in some contexts, such as
     an `mbox'-style mailbox file.

5.18.11.2 AddressList Objects
.............................

An AddressList instance has the following methods:

 -- Method: AddressList.__len__ ()
     Return the number of addresses in the address list.

 -- Method: AddressList.__str__ ()
     Return a canonicalized string representation of the address list.
     Addresses are rendered in "name" <<host@domain>> form,
     comma-separated.

 -- Method: AddressList.__add__ (alist)
     Return a new AddressList instance that contains all
     addresses in both AddressList operands, with
     duplicates removed (set union).

 -- Method: AddressList.__iadd__ (alist)
     In-place version of __add__().; turns this
     AddressList instance into the union of itself and the
     right-hand instance, _alist_.

 -- Method: AddressList.__sub__ (alist)
     Return a new AddressList instance that contains every
     address in the left-hand AddressList operand that is
     not present in the right-hand address operand (set difference).

 -- Method: AddressList.__isub__ (alist)
     In-place version of __sub__(), removing addresses in
     this list which are also in _alist_.

  Finally, see AddressList instances have one public instance
variable:

 -- Attribute: AddressList.addresslist
     A list of tuple string pairs, one per address.  In each member,
     the first is the canonicalized name part, the second is the actual
     route-address (`'@''-separated username-host.domain pair).

Footnotes
.........

     (1) This module originally conformed to RFC 822
     (http://tools.ietf.org/html/rfc822.html), hence the name.  Since
     then, RFC 2822 (http://tools.ietf.org/html/rfc2822.html) has been
     released as an update to RFC 822
     (http://tools.ietf.org/html/rfc822.html).  This module should be
     considered RFC 2822
     (http://tools.ietf.org/html/rfc2822.html)-conformant, especially
     in cases where the syntax or semantics have changed since RFC 822
     (http://tools.ietf.org/html/rfc822.html).

5.18.12 `base64' -- RFC 3548: Base16, Base32, Base64 Data Encodings
-------------------------------------------------------------------

This module provides data encoding and decoding as specified in RFC
3548 (http://tools.ietf.org/html/rfc3548.html).  This standard defines
the Base16, Base32, and Base64 algorithms for encoding and decoding
arbitrary binary strings into text strings that can be safely sent by
email, used as parts of URLs, or included as part of an HTTP POST
request.  The encoding algorithm is not the same as the *uuencode*
program.

  There are two interfaces provided by this module.  The modern
interface supports encoding and decoding string objects using all three
alphabets.  The legacy interface provides for encoding and decoding to
and from file-like objects as well as strings, but only using the
Base64 standard alphabet.

  The modern interface, which was introduced in Python 2.4, provides:

 -- Function: base64.b64encode (s[, altchars])
     Encode a string use Base64.

     _s_ is the string to encode.  Optional _altchars_ must be a string
     of at least length 2 (additional characters are ignored) which
     specifies an alternative alphabet for the `+' and `/' characters.
     This allows an application to e.g.  generate URL or filesystem
     safe Base64 strings.  The default is `None', for which the
     standard Base64 alphabet is used.

     The encoded string is returned.

 -- Function: base64.b64decode (s[, altchars])
     Decode a Base64 encoded string.

     _s_ is the string to decode.  Optional _altchars_ must be a string
     of at least length 2 (additional characters are ignored) which
     specifies the alternative alphabet used instead of the `+' and `/'
     characters.

     The decoded string is returned.  A TypeError is raised
     if _s_ were incorrectly padded or if there are non-alphabet
     characters present in the string.

 -- Function: base64.standard_b64encode (s)
     Encode string _s_ using the standard Base64 alphabet.

 -- Function: base64.standard_b64decode (s)
     Decode string _s_ using the standard Base64 alphabet.

 -- Function: base64.urlsafe_b64encode (s)
     Encode string _s_ using a URL-safe alphabet, which substitutes `-'
     instead of `+' and `_' instead of `/' in the standard Base64
     alphabet.  The result can still contain `='.

 -- Function: base64.urlsafe_b64decode (s)
     Decode string _s_ using a URL-safe alphabet, which substitutes `-'
     instead of `+' and `_' instead of `/' in the standard Base64
     alphabet.

 -- Function: base64.b32encode (s)
     Encode a string using Base32.  _s_ is the string to encode.  The
     encoded string is returned.

 -- Function: base64.b32decode (s[, casefold[, map01]])
     Decode a Base32 encoded string.

     _s_ is the string to decode.  Optional _casefold_ is a flag
     specifying whether a lowercase alphabet is acceptable as input.
     For security purposes, the default is `False'.

     RFC 3548 (http://tools.ietf.org/html/rfc3548.html) allows for
     optional mapping of the digit 0 (zero) to the letter O (oh), and
     for optional mapping of the digit 1 (one) to either the letter I
     (eye) or letter L (el).  The optional argument _map01_ when not
     `None', specifies which letter the digit 1 should be mapped to
     (when _map01_ is not `None', the digit 0 is always mapped to the
     letter O).  For security purposes the default is `None', so that 0
     and 1 are not allowed in the input.

     The decoded string is returned.  A TypeError is raised
     if _s_ were incorrectly padded or if there are non-alphabet
     characters present in the string.

 -- Function: base64.b16encode (s)
     Encode a string using Base16.

     _s_ is the string to encode.  The encoded string is returned.

 -- Function: base64.b16decode (s[, casefold])
     Decode a Base16 encoded string.

     _s_ is the string to decode.  Optional _casefold_ is a flag
     specifying whether a lowercase alphabet is acceptable as input.
     For security purposes, the default is `False'.

     The decoded string is returned.  A TypeError is raised
     if _s_ were incorrectly padded or if there are non-alphabet
     characters present in the string.

  The legacy interface:

 -- Function: base64.decode (input, output)
     Decode the contents of the _input_ file and write the resulting
     binary data to the _output_ file. _input_ and _output_ must either
     be file objects or objects that mimic the file object interface.
     _input_ will be read until `input.read()' returns an empty string.

 -- Function: base64.decodestring (s)
     Decode the string _s_, which must contain one or more lines of
     base64 encoded data, and return a string containing the resulting
     binary data.

 -- Function: base64.encode (input, output)
     Encode the contents of the _input_ file and write the resulting
     base64 encoded data to the _output_ file. _input_ and _output_
     must either be file objects or objects that mimic the file object
     interface. _input_ will be read until `input.read()' returns an
     empty string.  encode() returns the encoded data plus
     a trailing newline character (`'\n'').

 -- Function: base64.encodestring (s)
     Encode the string _s_, which can contain arbitrary binary data,
     and return a string containing one or more lines of base64-encoded
     data.  encodestring() returns a string containing one
     or more lines of base64-encoded data always including an extra
     trailing newline (`'\n'').

  An example usage of the module:

    >>> import base64
    >>> encoded = base64.b64encode('data to be encoded')
    >>> encoded
    'ZGF0YSB0byBiZSBlbmNvZGVk'
    >>> data = base64.b64decode(encoded)
    >>> data
    'data to be encoded'

Module binascii.
     Support module containing ASCII-to-binary and binary-to-ASCII
     conversions.

RFC 1521 (http://tools.ietf.org/html/rfc1521.html) - MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies
     Section 5.2, "Base64 Content-Transfer-Encoding," provides the
     definition of the base64 encoding.

5.18.13 `binhex' -- Encode and decode binhex4 files
---------------------------------------------------

This module encodes and decodes files in binhex4 format, a format
allowing representation of Macintosh files in ASCII.  On the Macintosh,
both forks of a file and the finder information are encoded (or
decoded), on other platforms only the data fork is handled.

     Note: In Python 3.x, special Macintosh support has been removed.

  The binhex module defines the following functions:

 -- Function: binhex.binhex (input, output)
     Convert a binary file with filename _input_ to binhex file
     _output_. The _output_ parameter can either be a filename or a
     file-like object (any object supporting a `write()' and `close()'
     method).

 -- Function: binhex.hexbin (input[, output])
     Decode a binhex file _input_. _input_ may be a filename or a
     file-like object supporting `read()' and `close()' methods. The
     resulting file is written to a file named _output_, unless the
     argument is omitted in which case the output filename is read from
     the binhex file.

  The following exception is also defined:

 -- Exception: binhex.Error
     Exception raised when something can't be encoded using the binhex
     format (for example, a filename is too long to fit in the filename
     field), or when input is not properly encoded binhex data.

Module binascii.
     Support module containing ASCII-to-binary and binary-to-ASCII
     conversions.

5.18.13.1 Notes
...............

There is an alternative, more powerful interface to the coder and
decoder, see the source for details.

  If you code or decode textfiles on non-Macintosh platforms they will
still use the old Macintosh newline convention (carriage-return as end
of line).

  As of this writing, see hexbin() appears to not work in all
cases.

5.18.14 `binascii' -- Convert between binary and ASCII
------------------------------------------------------

The binascii module contains a number of methods to convert
between binary and various ASCII-encoded binary representations.
Normally, you will not use these functions directly but use wrapper
modules like uu, base64, or binhex.
instead. The binascii module contains low-level functions
written in C for greater speed that are used by the higher-level
modules.

  The binascii module defines the following functions:

 -- Function: binascii.a2b_uu (string)
     Convert a single line of uuencoded data back to binary and return
     the binary data. Lines normally contain 45 (binary) bytes, except
     for the last line. Line data may be followed by whitespace.

 -- Function: binascii.b2a_uu (data)
     Convert binary data to a line of ASCII characters, the return
     value is the converted line, including a newline char. The length
     of _data_ should be at most 45.

 -- Function: binascii.a2b_base64 (string)
     Convert a block of base64 data back to binary and return the
     binary data. More than one line may be passed at a time.

 -- Function: binascii.b2a_base64 (data)
     Convert binary data to a line of ASCII characters in base64
     coding. The return value is the converted line, including a
     newline char. The length of _data_ should be at most 57 to adhere
     to the base64 standard.

 -- Function: binascii.a2b_qp (string[, header])
     Convert a block of quoted-printable data back to binary and return
     the binary data. More than one line may be passed at a time. If
     the optional argument _header_ is present and true, underscores
     will be decoded as spaces.

 -- Function: binascii.b2a_qp (data[, quotetabs, istext, header])
     Convert binary data to a line(s) of ASCII characters in
     quoted-printable encoding.  The return value is the converted
     line(s). If the optional argument _quotetabs_ is present and true,
     all tabs and spaces will be encoded.   If the optional argument
     _istext_ is present and true, newlines are not encoded but
     trailing whitespace will be encoded. If the optional argument
     _header_ is present and true, spaces will be encoded as
     underscores per RFC1522. If the optional argument _header_ is
     present and false, newline characters will be encoded as well;
     otherwise linefeed conversion might corrupt the binary data stream.

 -- Function: binascii.a2b_hqx (string)
     Convert binhex4 formatted ASCII data to binary, without doing
     RLE-decompression.  The string should contain a complete number of
     binary bytes, or (in case of the last portion of the binhex4 data)
     have the remaining bits zero.

 -- Function: binascii.rledecode_hqx (data)
     Perform RLE-decompression on the data, as per the binhex4
     standard. The algorithm uses `0x90' after a byte as a repeat
     indicator, followed by a count.  A count of `0' specifies a byte
     value of `0x90'. The routine returns the decompressed data, unless
     data input data ends in an orphaned repeat indicator, in which
     case the Incomplete exception is raised.

 -- Function: binascii.rlecode_hqx (data)
     Perform binhex4 style RLE-compression on _data_ and return the
     result.

 -- Function: binascii.b2a_hqx (data)
     Perform hexbin4 binary-to-ASCII translation and return the
     resulting string. The argument should already be RLE-coded, and
     have a length divisible by 3 (except possibly the last fragment).

 -- Function: binascii.crc_hqx (data, crc)
     Compute the binhex4 crc value of _data_, starting with an initial
     _crc_ and returning the result.

 -- Function: binascii.crc32 (data[, crc])
     Compute CRC-32, the 32-bit checksum of data, starting with an
     initial crc.  This is consistent with the ZIP file checksum.
     Since the algorithm is designed for use as a checksum algorithm,
     it is not suitable for use as a general hash algorithm.  Use as
     follows:

         print binascii.crc32("hello world")
         # Or, in two pieces:
         crc = binascii.crc32("hello")
         crc = binascii.crc32(" world", crc) & 0xffffffff
         print 'crc32 = 0x%08x' % crc

     Note: To generate the same numeric value across all Python
     versions and platforms use crc32(data) & 0xffffffff.  If you are
     only using the checksum in packed binary format this is not
     necessary as the return value is the correct 32bit binary
     representation regardless of sign.

  Changed in version 2.6: The return value is in the range [-2**31,
2**31-1] regardless of platform.  In the past the value would be signed
on some platforms and unsigned on others.  Use & 0xffffffff on the
value if you want it to match 3.0 behavior.

  Changed in version 3.0: The return value is unsigned and in the range
[0, 2**32-1] regardless of platform.

 -- Function: binascii.b2a_hex (data)
 -- Function: binascii.hexlify (data)
     Return the hexadecimal representation of the binary _data_.  Every
     byte of _data_ is converted into the corresponding 2-digit hex
     representation.  The resulting string is therefore twice as long
     as the length of _data_.

 -- Function: binascii.a2b_hex (hexstr)
 -- Function: binascii.unhexlify (hexstr)
     Return the binary data represented by the hexadecimal string
     _hexstr_.  This function is the inverse of b2a_hex().
     _hexstr_ must contain an even number of hexadecimal digits (which
     can be upper or lower case), otherwise a TypeError is
     raised.

 -- Exception: binascii.Error
     Exception raised on errors. These are usually programming errors.

 -- Exception: binascii.Incomplete
     Exception raised on incomplete data. These are usually not
     programming errors, but may be handled by reading a little more
     data and trying again.

Module base64.
     Support for base64 encoding used in MIME email messages.

Module binhex.
     Support for the binhex format used on the Macintosh.

Module uu.
     Support for UU encoding used on Unix.

Module quopri.
     Support for quoted-printable encoding used in MIME email messages.

5.18.15 `quopri' -- Encode and decode MIME quoted-printable data
----------------------------------------------------------------

This module performs quoted-printable transport encoding and decoding,
as defined in RFC 1521 (http://tools.ietf.org/html/rfc1521.html): "MIME
(Multipurpose Internet Mail Extensions) Part One: Mechanisms for
Specifying and Describing the Format of Internet Message Bodies".  The
quoted-printable encoding is designed for data where there are
relatively few nonprintable characters; the base64 encoding scheme
available via the base64 module is more compact if there are
many such characters, as when sending a graphics file.

  Latest version of the quopri module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/quopri.py?view=markup)

 -- Function: quopri.decode (input, output[, header])
     Decode the contents of the _input_ file and write the resulting
     decoded binary data to the _output_ file. _input_ and _output_
     must either be file objects or objects that mimic the file object
     interface. _input_ will be read until `input.readline()' returns
     an empty string. If the optional argument _header_ is present and
     true, underscore will be decoded as space. This is used to decode
     "Q"-encoded headers as described in RFC 1522
     (http://tools.ietf.org/html/rfc1522.html): "MIME (Multipurpose
     Internet Mail Extensions) Part Two: Message Header Extensions for
     Non-ASCII Text".

 -- Function: quopri.encode (input, output, quotetabs)
     Encode the contents of the _input_ file and write the resulting
     quoted-printable data to the _output_ file. _input_ and _output_
     must either be file objects or objects that mimic the file object
     interface. _input_ will be read until `input.readline()' returns
     an empty string. _quotetabs_ is a flag which controls whether to
     encode embedded spaces and tabs; when true it encodes such
     embedded whitespace, and when false it leaves them unencoded.
     Note that spaces and tabs appearing at the end of lines are always
     encoded, as per RFC 1521 (http://tools.ietf.org/html/rfc1521.html).

 -- Function: quopri.decodestring (s[, header])
     Like decode(), except that it accepts a source string
     and returns the corresponding decoded string.

 -- Function: quopri.encodestring (s[, quotetabs])
     Like encode(), except that it accepts a source string
     and returns the corresponding encoded string.  _quotetabs_ is
     optional (defaulting to 0), and is passed straight through to
     encode().

Module mimify.
     General utilities for processing of MIME messages.

Module base64.
     Encode and decode MIME base64 data

5.18.16 `uu' -- Encode and decode uuencode files
------------------------------------------------

This module encodes and decodes files in uuencode format, allowing
arbitrary binary data to be transferred over ASCII-only connections.
Wherever a file argument is expected, the methods accept a file-like
object.  For backwards compatibility, a string containing a pathname is
also accepted, and the corresponding file will be opened for reading
and writing; the pathname `'-'' is understood to mean the standard
input or output.  However, this interface is deprecated; it's better
for the caller to open the file itself, and be sure that, when
required, the mode is `'rb'' or `'wb'' on Windows.

  This code was contributed by Lance Ellinghouse, and modified by Jack
Jansen.

  Latest version of the uu module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/uu.py?view=markup)The
uu module defines the following functions:

 -- Function: uu.encode (in_file, out_file[, name[, mode]])
     Uuencode file _in_file_ into file _out_file_.  The uuencoded file
     will have the header specifying _name_ and _mode_ as the defaults
     for the results of decoding the file. The default defaults are
     taken from _in_file_, or `'-'' and `0666' respectively.

 -- Function: uu.decode (in_file[, out_file[, mode[, quiet]]])
     This call decodes uuencoded file _in_file_ placing the result on
     file _out_file_. If _out_file_ is a pathname, _mode_ is used to
     set the permission bits if the file must be created. Defaults for
     _out_file_ and _mode_ are taken from the uuencode header.
     However, if the file specified in the header already exists, a
     uu.Error is raised.

     decode() may print a warning to standard error if the
     input was produced by an incorrect uuencoder and Python could
     recover from that error.  Setting _quiet_ to a true value silences
     this warning.

 -- Exception: uu.Error
     Subclass of Exception, this can be raised by
     uu.decode() under various situations, such as described
     above, but also including a badly formatted header, or truncated
     input file.

Module binascii.
     Support module containing ASCII-to-binary and binary-to-ASCII
     conversions.

5.19 Structured Markup Processing Tools
=======================================

Python supports a variety of modules to work with various forms of
structured data markup.  This includes modules to work with the
Standard Generalized Markup Language (SGML) and the Hypertext Markup
Language (HTML), and several interfaces for working with the Extensible
Markup Language (XML).

  It is important to note that modules in the `xml' package require that
there be at least one SAX-compliant XML parser available. Starting with
Python 2.3, the Expat parser is included with Python, so the
xml.parsers.expat.  module will always be available. You may still
want to be aware of the PyXML add-on package
(http://pyxml.sourceforge.net/); that package provides an extended set
of XML libraries for Python.

  The documentation for the xml.dom and xml.sax.
packages are the definition of the Python bindings for the DOM and SAX
interfaces.

5.19.1 `HTMLParser' -- Simple HTML and XHTML parser
---------------------------------------------------

     Note: The HTMLParser module has been renamed to
     `html.parser' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

New in version 2.2.

  This module defines a class HTMLParser which serves as the
basis for parsing text files formatted in HTML (HyperText Mark-up
Language) and XHTML.  Unlike the parser in htmllib, this
parser is not based on the SGML parser in sgmllib.

 -- Class: HTMLParser.HTMLParser
     The HTMLParser class is instantiated without arguments.

     An HTMLParser instance is fed HTML data and calls
     handler functions when tags begin and end.  The HTMLParser
     class is meant to be overridden by the user to provide a
     desired behavior.

     Unlike the parser in htmllib, this parser does not check
     that end tags match start tags or call the end-tag handler for
     elements which are closed implicitly by closing an outer element.

  An exception is defined as well:

 -- Exception: HTMLParser.HTMLParseError
     Exception raised by the HTMLParser class when it
     encounters an error while parsing.  This exception provides three
     attributes: `msg' is a brief message explaining the error,
     `lineno' is the number of the line on which the broken construct
     was detected, and `offset' is the number of characters into the
     line at which the construct starts.

  HTMLParser instances have the following methods:

 -- Method: HTMLParser.reset ()
     Reset the instance.  Loses all unprocessed data.  This is called
     implicitly at instantiation time.

 -- Method: HTMLParser.feed (data)
     Feed some text to the parser.  It is processed insofar as it
     consists of complete elements; incomplete data is buffered until
     more data is fed or close() is called.

 -- Method: HTMLParser.close ()
     Force processing of all buffered data as if it were followed by an
     end-of-file mark.  This method may be redefined by a derived class
     to define additional processing at the end of the input, but the
     redefined version should always call the HTMLParser.
     base class method close().

 -- Method: HTMLParser.getpos ()
     Return current line number and offset.

 -- Method: HTMLParser.get_starttag_text ()
     Return the text of the most recently opened start tag.  This
     should not normally be needed for structured processing, but may
     be useful in dealing with HTML "as deployed" or for re-generating
     input with minimal changes (whitespace between attributes can be
     preserved, etc.).

 -- Method: HTMLParser.handle_starttag (tag, attrs)
     This method is called to handle the start of a tag.  It is
     intended to be overridden by a derived class; the base class
     implementation does nothing.

     The _tag_ argument is the name of the tag converted to lower case.
     The _attrs_ argument is a list of `(name, value)' pairs containing
     the attributes found inside the tag's `<>' brackets.  The _name_
     will be translated to lower case, and quotes in the _value_ have
     been removed, and character and entity references have been
     replaced.  For instance, for the tag `<A
     HREF="http://www.cwi.nl/">', this method would be called as
     `handle_starttag('a', [('href', 'http://www.cwi.nl/')])'.

     Changed in version 2.6: All entity references from
     htmlentitydefs are now replaced in the attribute values.

 -- Method: HTMLParser.handle_startendtag (tag, attrs)
     Similar to handle_starttag(), but called when the
     parser encounters an XHTML-style empty tag (`<a .../>').  This
     method may be overridden by subclasses which require this
     particular lexical information; the default implementation simple
     calls handle_starttag() and handle_endtag().

 -- Method: HTMLParser.handle_endtag (tag)
     This method is called to handle the end tag of an element.  It is
     intended to be overridden by a derived class; the base class
     implementation does nothing.  The _tag_ argument is the name of
     the tag converted to lower case.

 -- Method: HTMLParser.handle_data (data)
     This method is called to process arbitrary data.  It is intended
     to be overridden by a derived class; the base class implementation
     does nothing.

 -- Method: HTMLParser.handle_charref (name)
     This method is called to process a character reference of the form
     `&#ref;'.  It is intended to be overridden by a derived class; the
     base class implementation does nothing.

 -- Method: HTMLParser.handle_entityref (name)
     This method is called to process a general entity reference of the
     form `&name;' where _name_ is an general entity reference.  It is
     intended to be overridden by a derived class; the base class
     implementation does nothing.

 -- Method: HTMLParser.handle_comment (data)
     This method is called when a comment is encountered.  The
     _comment_ argument is a string containing the text between the
     `--' and `--' delimiters, but not the delimiters themselves.  For
     example, the comment `<!--text-->' will cause this method to be
     called with the argument `'text''.  It is intended to be
     overridden by a derived class; the base class implementation does
     nothing.

 -- Method: HTMLParser.handle_decl (decl)
     Method called when an SGML `doctype' declaration is read by the
     parser.  The _decl_ parameter will be the entire contents of the
     declaration inside the `<!...>' markup.  It is intended to be
     overridden by a derived class; the base class implementation does
     nothing.

 -- Method: HTMLParser.unknown_decl (data)
     Method called when an unrecognized SGML declaration is read by the
     parser.  The _data_ parameter will be the entire contents of the
     declaration inside the `<!...>' markup.  It is sometimes useful to
     be be overridden by a derived class; the base class implementation
     throws an HTMLParseError.

 -- Method: HTMLParser.handle_pi (data)
     Method called when a processing instruction is encountered.  The
     _data_ parameter will contain the entire processing instruction.
     For example, for the processing instruction `<?proc color='red'>',
     this method would be called as `handle_pi("proc color='red'")'.
     It is intended to be overridden by a derived class; the base class
     implementation does nothing.

          Note: The HTMLParser class uses the SGML syntactic
          rules for processing instructions.  An XHTML processing
          instruction using the trailing `'?'' will cause the `'?'' to
          be included in _data_.

5.19.1.1 Example HTML Parser Application
........................................

As a basic example, below is a very basic HTML parser that uses the
HTMLParser class to print out tags as they are encountered:

    from HTMLParser import HTMLParser

    class MyHTMLParser(HTMLParser):

        def handle_starttag(self, tag, attrs):
            print "Encountered the beginning of a %s tag" % tag

        def handle_endtag(self, tag):
            print "Encountered the end of a %s tag" % tag

5.19.2 `sgmllib' -- Simple SGML parser
--------------------------------------

Deprecated since version 2.6: The sgmllib module has been
removed in Python 3.0.

  This module defines a class SGMLParser which serves as
the basis for parsing text files formatted in SGML (Standard
Generalized Mark-up Language).  In fact, it does not provide a full
SGML parser -- it only parses SGML insofar as it is used by HTML, and
the module only exists as a base for the htmllib module.
Another HTML parser which supports XHTML and offers a somewhat
different interface is available in the HTMLParser module.

 -- Class: sgmllib.SGMLParser
     The SGMLParser class is instantiated without
     arguments. The parser is hardcoded to recognize the following
     constructs:

        * Opening and closing tags of the form `<tag attr="value" ...>'
          and `</tag>', respectively.

        * Numeric character references of the form `&#name;'.

        * Entity references of the form `&name;'.

        * SGML comments of the form `<!--text-->'.  Note that spaces,
          tabs, and newlines are allowed between the trailing `>' and
          the immediately preceding `--'.

  A single exception is defined as well:

 -- Exception: sgmllib.SGMLParseError
     Exception raised by the SGMLParser class when it
     encounters an error while parsing.

     New in version 2.1.

  SGMLParser instances have the following methods:

 -- Method: SGMLParser.reset ()
     Reset the instance.  Loses all unprocessed data.  This is called
     implicitly at instantiation time.

 -- Method: SGMLParser.setnomoretags ()
     Stop processing tags.  Treat all following input as literal input
     (CDATA).  (This is only provided so the HTML tag `<PLAINTEXT>' can
     be implemented.)

 -- Method: SGMLParser.setliteral ()
     Enter literal mode (CDATA mode).

 -- Method: SGMLParser.feed (data)
     Feed some text to the parser.  It is processed insofar as it
     consists of complete elements; incomplete data is buffered until
     more data is fed or close() is called.

 -- Method: SGMLParser.close ()
     Force processing of all buffered data as if it were followed by an
     end-of-file mark.  This method may be redefined by a derived class
     to define additional processing at the end of the input, but the
     redefined version should always call close().

 -- Method: SGMLParser.get_starttag_text ()
     Return the text of the most recently opened start tag.  This
     should not normally be needed for structured processing, but may
     be useful in dealing with HTML "as deployed" or for re-generating
     input with minimal changes (whitespace between attributes can be
     preserved, etc.).

 -- Method: SGMLParser.handle_starttag (tag, method, attributes)
     This method is called to handle start tags for which either a
     `start_tag()' or `do_tag()' method has been defined.  The _tag_
     argument is the name of the tag converted to lower case, and the
     _method_ argument is the bound method which should be used to
     support semantic interpretation of the start tag. The _attributes_
     argument is a list of `(name, value)' pairs containing the
     attributes found inside the tag's `<>' brackets.

     The _name_ has been translated to lower case. Double quotes and
     backslashes in the _value_ have been interpreted, as well as known
     character references and known entity references terminated by a
     semicolon (normally, entity references can be terminated by any
     non-alphanumerical character, but this would break the very common
     case of `<A HREF="url?spam=1&eggs=2">' when `eggs' is a valid
     entity name).

     For instance, for the tag `<A HREF="http://www.cwi.nl/">', this
     method would be called as `unknown_starttag('a', [('href',
     'http://www.cwi.nl/')])'.  The base implementation simply calls
     _method_ with _attributes_ as the only argument.

     New in version 2.5: Handling of entity and character references
     within attribute values.

 -- Method: SGMLParser.handle_endtag (tag, method)
     This method is called to handle endtags for which an `end_tag()'
     method has been defined.  The _tag_ argument is the name of the
     tag converted to lower case, and the _method_ argument is the
     bound method which should be used to support semantic
     interpretation of the end tag.  If no `end_tag()' method is
     defined for the closing element, this handler is not called.  The
     base implementation simply calls _method_.

 -- Method: SGMLParser.handle_data (data)
     This method is called to process arbitrary data.  It is intended
     to be overridden by a derived class; the base class implementation
     does nothing.

 -- Method: SGMLParser.handle_charref (ref)
     This method is called to process a character reference of the form
     `&#ref;'.  The base implementation uses convert_charref()
     to convert the reference to a string.  If that method
     returns a string, it is passed to handle_data(),
     otherwise `unknown_charref(ref)' is called to handle the error.

     Changed in version 2.5: Use convert_charref() instead
     of hard-coding the conversion.

 -- Method: SGMLParser.convert_charref (ref)
     Convert a character reference to a string, or `None'.  _ref_ is
     the reference passed in as a string.  In the base implementation,
     _ref_ must be a decimal number in the range 0-255.  It converts
     the code point found using the convert_codepoint().
     method. If _ref_ is invalid or out of range, this method returns
     `None'.  This method is called by the default
     handle_charref() implementation and by the attribute value
     parser.

     New in version 2.5.

 -- Method: SGMLParser.convert_codepoint (codepoint)
     Convert a codepoint to a str value.  Encodings can be
     handled here if appropriate, though the rest of sgmllib
     is oblivious on this matter.

     New in version 2.5.

 -- Method: SGMLParser.handle_entityref (ref)
     This method is called to process a general entity reference of the
     form `&ref;' where _ref_ is an general entity reference.  It
     converts _ref_ by passing it to convert_entityref().
     If a translation is returned, it calls the method
     handle_data() with the translation; otherwise, it calls the
     method `unknown_entityref(ref)'. The default `entitydefs' defines
     translations for `&amp;', `&apos', `&gt;', `&lt;', and `&quot;'.

     Changed in version 2.5: Use convert_entityref().
     instead of hard-coding the conversion.

 -- Method: SGMLParser.convert_entityref (ref)
     Convert a named entity reference to a str value, or
     `None'.  The resulting value will not be parsed.  _ref_ will be
     only the name of the entity.  The default implementation looks for
     _ref_ in the instance (or class) variable `entitydefs' which
     should be a mapping from entity names to corresponding
     translations.  If no translation is available for _ref_, this
     method returns `None'.  This method is called by the default
     handle_entityref().  implementation and by the attribute
     value parser.

     New in version 2.5.

 -- Method: SGMLParser.handle_comment (comment)
     This method is called when a comment is encountered.  The
     _comment_ argument is a string containing the text between the
     `<!--' and `-->' delimiters, but not the delimiters themselves.
     For example, the comment `<!--text-->' will cause this method to
     be called with the argument `'text''.  The default method does
     nothing.

 -- Method: SGMLParser.handle_decl (data)
     Method called when an SGML declaration is read by the parser.  In
     practice, the `DOCTYPE' declaration is the only thing observed in
     HTML, but the parser does not discriminate among different (or
     broken) declarations.  Internal subsets in a `DOCTYPE' declaration
     are not supported.  The _data_ parameter will be the entire
     contents of the declaration inside the `<!'...`>' markup.  The
     default implementation does nothing.

 -- Method: SGMLParser.report_unbalanced (tag)
     This method is called when an end tag is found which does not
     correspond to any open element.

 -- Method: SGMLParser.unknown_starttag (tag, attributes)
     This method is called to process an unknown start tag.  It is
     intended to be overridden by a derived class; the base class
     implementation does nothing.

 -- Method: SGMLParser.unknown_endtag (tag)
     This method is called to process an unknown end tag.  It is
     intended to be overridden by a derived class; the base class
     implementation does nothing.

 -- Method: SGMLParser.unknown_charref (ref)
     This method is called to process unresolvable numeric character
     references.  Refer to handle_charref() to determine
     what is handled by default.  It is intended to be overridden by a
     derived class; the base class implementation does nothing.

 -- Method: SGMLParser.unknown_entityref (ref)
     This method is called to process an unknown entity reference.  It
     is intended to be overridden by a derived class; the base class
     implementation does nothing.

  Apart from overriding or extending the methods listed above, derived
classes may also define methods of the following form to define
processing of specific tags.  Tag names in the input stream are case
independent; the _tag_ occurring in method names must be in lower case:

 -- Method: SGMLParser.start_tag (attributes)
     This method is called to process an opening tag _tag_.  It has
     preference over `do_tag()'.  The _attributes_ argument has the
     same meaning as described for `handle_starttag()' above.

 -- Method: SGMLParser.do_tag (attributes)
     This method is called to process an opening tag _tag_  for which no
     `start_tag()' method is defined.   The _attributes_ argument has
     the same meaning as described for `handle_starttag()' above.

 -- Method: SGMLParser.end_tag ()
     This method is called to process a closing tag _tag_.

  Note that the parser maintains a stack of open elements for which no
end tag has been found yet.  Only tags processed by `start_tag()' are
pushed on this stack.  Definition of an `end_tag()' method is optional
for these tags.  For tags processed by `do_tag()' or by
`unknown_tag()', no `end_tag()' method must be defined; if defined, it
will not be used.  If both `start_tag()' and `do_tag()' methods exist
for a tag, the `start_tag()' method takes precedence.

5.19.3 `htmllib' -- A parser for HTML documents
-----------------------------------------------

Deprecated since version 2.6: The htmllib module has been
removed in Python 3.0.

  This module defines a class which can serve as a base for parsing
text files formatted in the HyperText Mark-up Language (HTML).  The
class is not directly concerned with I/O -- it must be provided with
input in string form via a method, and makes calls to methods of a
"formatter" object in order to produce output.  The HTMLParser
class is designed to be used as a base class for other classes in
order to add functionality, and allows most of its methods to be
extended or overridden.  In turn, this class is derived from and extends
the `SGMLParser' class defined in module sgmllib.  The
HTMLParser implementation supports the HTML 2.0 language as
described in RFC 1866 (http://tools.ietf.org/html/rfc1866.html).  Two
implementations of formatter objects are provided in the
formatter module; refer to the documentation for that module for
information on the formatter interface.

  The following is a summary of the interface defined by
sgmllib.SGMLParser.:

   * The interface to feed data to an instance is through the `feed()'
     method, which takes a string argument.  This can be called with as
     little or as much text at a time as desired; `p.feed(a);
     p.feed(b)' has the same effect as `p.feed(a+b)'.  When the data
     contains complete HTML markup constructs, these are processed
     immediately; incomplete constructs are saved in a buffer.  To
     force processing of all unprocessed data, call the `close()'
     method.

     For example, to parse the entire contents of a file, use:

         parser.feed(open('myfile.html').read())
         parser.close()

   * The interface to define semantics for HTML tags is very simple:
     derive a class and define methods called `start_tag()',
     `end_tag()', or `do_tag()'.  The parser will call these at
     appropriate moments: `start_tag()' or `do_tag()' is called when an
     opening tag of the form `<tag ...>' is encountered; `end_tag()' is
     called when a closing tag of the form `<tag>' is encountered.  If
     an opening tag requires a corresponding closing tag, like `<H1>'
     ... `</H1>', the class should define the `start_tag()' method; if
     a tag requires no closing tag, like `<P>', the class should define
     the `do_tag()' method.

  The module defines a parser class and an exception:

 -- Class: htmllib.HTMLParser (formatter)
     This is the basic HTML parser class.  It supports all entity names
     required by the XHTML 1.0 Recommendation
     (<http://www.w3.org/TR/xhtml1>).   It also defines handlers for
     all HTML 2.0 and many HTML 3.0 and 3.2 elements.

 -- Exception: htmllib.HTMLParseError
     Exception raised by the HTMLParser class when it
     encounters an error while parsing.

     New in version 2.4.

Module formatter.
     Interface definition for transforming an abstract flow of
     formatting events into specific output events on writer objects.

Module HTMLParser.
     Alternate HTML parser that offers a slightly lower-level view of
     the input, but is designed to work with XHTML, and does not
     implement some of the SGML syntax not used in "HTML as deployed"
     and which isn't legal for XHTML.

Module htmlentitydefs.
     Definition of replacement text for XHTML 1.0  entities.

Module sgmllib.
     Base class for HTMLParser.

5.19.3.1 HTMLParser Objects
...........................

In addition to tag methods, the HTMLParser class provides
some additional methods and instance variables for use within tag
methods.

 -- Attribute: HTMLParser.formatter
     This is the formatter instance associated with the parser.

 -- Attribute: HTMLParser.nofill
     Boolean flag which should be true when whitespace should not be
     collapsed, or false when it should be.  In general, this should
     only be true when character data is to be treated as
     "preformatted" text, as within a `<PRE>' element.  The default
     value is false.  This affects the operation of `handle_data()' and
     save_end().

 -- Method: HTMLParser.anchor_bgn (href, name, type)
     This method is called at the start of an anchor region.  The
     arguments correspond to the attributes of the `<A>' tag with the
     same names.  The default implementation maintains a list of
     hyperlinks (defined by the `HREF' attribute for `<A>' tags) within
     the document.  The list of hyperlinks is available as the data
     attribute `anchorlist'.

 -- Method: HTMLParser.anchor_end ()
     This method is called at the end of an anchor region.  The default
     implementation adds a textual footnote marker using an index into
     the list of hyperlinks created by anchor_bgn().

 -- Method: HTMLParser.handle_image (source, alt[, ismap[, align[,
          width[, height]]]])
     This method is called to handle images.  The default
     implementation simply passes the _alt_ value to the
     `handle_data()' method.

 -- Method: HTMLParser.save_bgn ()
     Begins saving character data in a buffer instead of sending it to
     the formatter object.  Retrieve the stored data via
     save_end(). Use of the save_bgn(). / see
     save_end() pair may not be nested.

 -- Method: HTMLParser.save_end ()
     Ends buffering character data and returns all data saved since the
     preceding call to save_bgn().  If the nofill
     flag is false, whitespace is collapsed to single spaces.  A
     call to this method without a preceding call to save_bgn()
     will raise a TypeError exception.

5.19.4 `htmlentitydefs' -- Definitions of HTML general entities
---------------------------------------------------------------

     Note: The htmlentitydefs module has been renamed to
     `html.entities' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

This module defines three dictionaries, `name2codepoint',
`codepoint2name', and `entitydefs'. `entitydefs' is used by the
htmllib module to provide the entitydefs member of
the HTMLParser class.  The definition provided here contains
all the entities defined by XHTML 1.0  that can be handled using simple
textual substitution in the Latin-1 character set (ISO-8859-1).

 -- Data: htmlentitydefs.entitydefs
     A dictionary mapping XHTML 1.0 entity definitions to their
     replacement text in ISO Latin-1.

 -- Data: htmlentitydefs.name2codepoint
     A dictionary that maps HTML entity names to the Unicode codepoints.

     New in version 2.3.

 -- Data: htmlentitydefs.codepoint2name
     A dictionary that maps Unicode codepoints to HTML entity names.

     New in version 2.3.

5.19.5 `xml.parsers.expat' -- Fast XML parsing using Expat
----------------------------------------------------------

New in version 2.0.

  The xml.parsers.expat module is a Python interface to the
Expat non-validating XML parser. The module provides a single extension
type, `xmlparser', that represents the current state of an XML parser.
After an `xmlparser' object has been created, various attributes of the
object can be set to handler functions.  When an XML document is then
fed to the parser, the handler functions are called for the character
data and markup in the XML document.

  This module uses the `pyexpat' module to provide access to the Expat
parser.  Direct use of the `pyexpat' module is deprecated.

  This module provides one exception and one type object:

 -- Exception: xml.parsers.expat.ExpatError
     The exception raised when Expat reports an error.  See section
     ExpatError Exceptions for more information on
     interpreting Expat errors.

 -- Exception: xml.parsers.expat.error
     Alias for ExpatError.

 -- Data: xml.parsers.expat.XMLParserType
     The type of the return values from the ParserCreate().
     function.

  The xml.parsers.expat module contains two functions:

 -- Function: xml.parsers.expat.ErrorString (errno)
     Returns an explanatory string for a given error number _errno_.

 -- Function: xml.parsers.expat.ParserCreate ([encoding[,
          namespace_separator]])
     Creates and returns a new `xmlparser' object.   _encoding_, if
     specified, must be a string naming the encoding  used by the XML
     data.  Expat doesn't support as many encodings as Python does, and
     its repertoire of encodings can't be extended; it supports UTF-8,
     UTF-16, ISO-8859-1 (Latin1), and ASCII.  If _encoding_ (1) is
     given it will override the implicit or explicit encoding of the
     document.

     Expat can optionally do XML namespace processing for you, enabled
     by providing a value for _namespace_separator_.  The value must be
     a one-character string; a ValueError will be raised if
     the string has an illegal length (`None' is considered the same as
     omission).  When namespace processing is enabled, element type
     names and attribute names that belong to a namespace will be
     expanded.  The element name passed to the element handlers
     `StartElementHandler' and `EndElementHandler' will be the
     concatenation of the namespace URI, the namespace separator
     character, and the local part of the name.  If the namespace
     separator is a zero byte (`chr(0)') then the namespace URI and the
     local part will be concatenated without any separator.

     For example, if _namespace_separator_ is set to a space character
     (`' '') and the following document is parsed:

         <?xml version="1.0"?>
         <root xmlns    = "http://default-namespace.org/"
               xmlns:py = "http://www.python.org/ns/">
           <py:elem1 />
           <elem2 xmlns="" />
         </root>

     `StartElementHandler' will receive the following strings for each
     element:

         http://default-namespace.org/ root
         http://www.python.org/ns/ elem1
         elem2

The Expat XML Parser (http://www.libexpat.org/)
     Home page of the Expat project.

5.19.5.1 XMLParser Objects
..........................

`xmlparser' objects have the following methods:

 -- Method: xmlparser.Parse (data[, isfinal])
     Parses the contents of the string _data_, calling the appropriate
     handler functions to process the parsed data.  _isfinal_ must be
     true on the final call to this method.  _data_ can be the empty
     string at any time.

 -- Method: xmlparser.ParseFile (file)
     Parse XML data reading from the object _file_.  _file_ only needs
     to provide the `read(nbytes)' method, returning the empty string
     when there's no more data.

 -- Method: xmlparser.SetBase (base)
     Sets the base to be used for resolving relative URIs in system
     identifiers in declarations.  Resolving relative identifiers is
     left to the application: this value will be passed through as the
     _base_ argument to the ExternalEntityRefHandler(),
     NotationDeclHandler(), and
     UnparsedEntityDeclHandler() functions.

 -- Method: xmlparser.GetBase ()
     Returns a string containing the base set by a previous call to
     SetBase(), or `None' if  SetBase() hasn't
     been called.

 -- Method: xmlparser.GetInputContext ()
     Returns the input data that generated the current event as a
     string. The data is in the encoding of the entity which contains
     the text. When called while an event handler is not active, the
     return value is `None'.

     New in version 2.1.

 -- Method: xmlparser.ExternalEntityParserCreate (context[, encoding])
     Create a "child" parser which can be used to parse an external
     parsed entity referred to by content parsed by the parent parser.
     The _context_ parameter should be the string passed to the
     ExternalEntityRefHandler() handler function, described
     below. The child parser is created with the
     ordered_attributes, returns_unicode and
     specified_attributes set to the values of this parser.

 -- Method: xmlparser.UseForeignDTD ([flag])
     Calling this with a true value for _flag_ (the default) will cause
     Expat to call the ExternalEntityRefHandler with
     None for all arguments to allow an alternate DTD to be
     loaded.  If the document does not contain a document type
     declaration, the ExternalEntityRefHandler will still
     be called, but the StartDoctypeDeclHandler and
     EndDoctypeDeclHandler will not be called.

     Passing a false value for _flag_ will cancel a previous call that
     passed a true value, but otherwise has no effect.

     This method can only be called before the Parse() or
     ParseFile().  methods are called; calling it after
     either of those have been called causes ExpatError to
     be raised with the code attribute set to
     `errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING'.

     New in version 2.3.

  `xmlparser' objects have the following attributes:

 -- Attribute: xmlparser.buffer_size
     The size of the buffer used when buffer_text is true.
     A new buffer size can be set by assigning a new integer value to
     this attribute.  When the size is changed, the buffer will be
     flushed.

     New in version 2.3.

     Changed in version 2.6: The buffer size can now be changed.

 -- Attribute: xmlparser.buffer_text
     Setting this to true causes the `xmlparser' object to buffer
     textual content returned by Expat to avoid multiple calls to the
     CharacterDataHandler() callback whenever possible.
     This can improve performance substantially since Expat normally
     breaks character data into chunks at every line ending.  This
     attribute is false by default, and may be changed at any time.

     New in version 2.3.

 -- Attribute: xmlparser.buffer_used
     If buffer_text is enabled, the number of bytes stored
     in the buffer.  These bytes represent UTF-8 encoded text.  This
     attribute has no meaningful interpretation when buffer_text
     is false.

     New in version 2.3.

 -- Attribute: xmlparser.ordered_attributes
     Setting this attribute to a non-zero integer causes the attributes
     to be reported as a list rather than a dictionary.  The attributes
     are presented in the order found in the document text.  For each
     attribute, two list entries are presented: the attribute name and
     the attribute value.  (Older versions of this module also used
     this format.)  By default, this attribute is false; it may be
     changed at any time.

     New in version 2.1.

 -- Attribute: xmlparser.returns_unicode
     If this attribute is set to a non-zero integer, the handler
     functions will be passed Unicode strings.  If
     returns_unicode is False, 8-bit strings
     containing UTF-8 encoded data will be passed to the handlers.
     This is True by default when Python is built with
     Unicode support.

     Changed in version 1.6: Can be changed at any time to affect the
     result type.

 -- Attribute: xmlparser.specified_attributes
     If set to a non-zero integer, the parser will report only those
     attributes which were specified in the document instance and not
     those which were derived from attribute declarations.
     Applications which set this need to be especially careful to use
     what additional information is available from the declarations as
     needed to comply with the standards for the behavior of XML
     processors.  By default, this attribute is false; it may be
     changed at any time.

     New in version 2.1.

  The following attributes contain values relating to the most recent
error encountered by an `xmlparser' object, and will only have correct
values once a call to `Parse()' or `ParseFile()' has raised a
xml.parsers.expat.ExpatError exception.

 -- Attribute: xmlparser.ErrorByteIndex
     Byte index at which an error occurred.

 -- Attribute: xmlparser.ErrorCode
     Numeric code specifying the problem.  This value can be passed to
     the ErrorString() function, or compared to one of the
     constants defined in the `errors' object.

 -- Attribute: xmlparser.ErrorColumnNumber
     Column number at which an error occurred.

 -- Attribute: xmlparser.ErrorLineNumber
     Line number at which an error occurred.

  The following attributes contain values relating to the current parse
location in an `xmlparser' object.  During a callback reporting a parse
event they indicate the location of the first of the sequence of
characters that generated the event.  When called outside of a
callback, the position indicated will be just past the last parse event
(regardless of whether there was an associated callback).

  New in version 2.4.

 -- Attribute: xmlparser.CurrentByteIndex
     Current byte index in the parser input.

 -- Attribute: xmlparser.CurrentColumnNumber
     Current column number in the parser input.

 -- Attribute: xmlparser.CurrentLineNumber
     Current line number in the parser input.

  Here is the list of handlers that can be set.  To set a handler on an
`xmlparser' object _o_, use `o.handlername = func'.  _handlername_ must
be taken from the following list, and _func_ must be a callable object
accepting the correct number of arguments.  The arguments are all
strings, unless otherwise stated.

 -- Method: xmlparser.XmlDeclHandler (version, encoding, standalone)
     Called when the XML declaration is parsed.  The XML declaration is
     the (optional) declaration of the applicable version of the XML
     recommendation, the encoding of the document text, and an optional
     "standalone" declaration.  _version_ and _encoding_ will be
     strings of the type dictated by the returns_unicode.
     attribute, and _standalone_ will be `1' if the document is
     declared standalone, `0' if it is declared not to be standalone,
     or `-1' if the standalone clause was omitted. This is only
     available with Expat version 1.95.0 or newer.

     New in version 2.1.

 -- Method: xmlparser.StartDoctypeDeclHandler (doctypeName, systemId,
          publicId, has_internal_subset)
     Called when Expat begins parsing the document type declaration
     (`<!DOCTYPE ...').  The _doctypeName_ is provided exactly as
     presented.  The _systemId_ and _publicId_ parameters give the
     system and public identifiers if specified, or `None' if omitted.
     _has_internal_subset_ will be true if the document contains and
     internal document declaration subset. This requires Expat version
     1.2 or newer.

 -- Method: xmlparser.EndDoctypeDeclHandler ()
     Called when Expat is done parsing the document type declaration.
     This requires Expat version 1.2 or newer.

 -- Method: xmlparser.ElementDeclHandler (name, model)
     Called once for each element type declaration.  _name_ is the name
     of the element type, and _model_ is a representation of the
     content model.

 -- Method: xmlparser.AttlistDeclHandler (elname, attname, type,
          default, required)
     Called for each declared attribute for an element type.  If an
     attribute list declaration declares three attributes, this handler
     is called three times, once for each attribute.  _elname_ is the
     name of the element to which the declaration applies and _attname_
     is the name of the attribute declared.  The attribute type is a
     string passed as _type_; the possible values are `'CDATA'',
     `'ID'', `'IDREF'', ... _default_ gives the default value for the
     attribute used when the attribute is not specified by the document
     instance, or `None' if there is no default value (`#IMPLIED'
     values).  If the attribute is required to be given in the document
     instance, _required_ will be true. This requires Expat version
     1.95.0 or newer.

 -- Method: xmlparser.StartElementHandler (name, attributes)
     Called for the start of every element.  _name_ is a string
     containing the element name, and _attributes_ is a dictionary
     mapping attribute names to their values.

 -- Method: xmlparser.EndElementHandler (name)
     Called for the end of every element.

 -- Method: xmlparser.ProcessingInstructionHandler (target, data)
     Called for every processing instruction.

 -- Method: xmlparser.CharacterDataHandler (data)
     Called for character data.  This will be called for normal
     character data, CDATA marked content, and ignorable whitespace.
     Applications which must distinguish these cases can use the
     StartCdataSectionHandler, EndCdataSectionHandler,
     and ElementDeclHandler callbacks to collect the
     required information.

 -- Method: xmlparser.UnparsedEntityDeclHandler (entityName, base,
          systemId, publicId, notationName)
     Called for unparsed (NDATA) entity declarations.  This is only
     present for version 1.2 of the Expat library; for more recent
     versions, use EntityDeclHandler instead.  (The
     underlying function in the Expat library has been declared
     obsolete.)

 -- Method: xmlparser.EntityDeclHandler (entityName,
          is_parameter_entity, value, base, systemId, publicId,
          notationName)
     Called for all entity declarations.  For parameter and internal
     entities, _value_ will be a string giving the declared contents of
     the entity; this will be `None' for external entities.  The
     _notationName_ parameter will be `None' for parsed entities, and
     the name of the notation for unparsed entities.
     _is_parameter_entity_ will be true if the entity is a parameter
     entity or false for general entities (most applications only need
     to be concerned with general entities). This is only available
     starting with version 1.95.0 of the Expat library.

     New in version 2.1.

 -- Method: xmlparser.NotationDeclHandler (notationName, base,
          systemId, publicId)
     Called for notation declarations.  _notationName_, _base_, and
     _systemId_, and _publicId_ are strings if given.  If the public
     identifier is omitted, _publicId_ will be `None'.

 -- Method: xmlparser.StartNamespaceDeclHandler (prefix, uri)
     Called when an element contains a namespace declaration.
     Namespace declarations are processed before the
     StartElementHandler is called for the element on which
     declarations are placed.

 -- Method: xmlparser.EndNamespaceDeclHandler (prefix)
     Called when the closing tag is reached for an element  that
     contained a namespace declaration.  This is called once for each
     namespace declaration on the element in the reverse of the order
     for which the StartNamespaceDeclHandler was called to
     indicate the start of each namespace declaration's scope.  Calls
     to this handler are made after the corresponding
     EndElementHandler for the end of the element.

 -- Method: xmlparser.CommentHandler (data)
     Called for comments.  _data_ is the text of the comment, excluding
     the leading '`<!-'`-'' and trailing '`-'`->''.

 -- Method: xmlparser.StartCdataSectionHandler ()
     Called at the start of a CDATA section.  This and
     EndCdataSectionHandler.  are needed to be able to identify
     the syntactical start and end for CDATA sections.

 -- Method: xmlparser.EndCdataSectionHandler ()
     Called at the end of a CDATA section.

 -- Method: xmlparser.DefaultHandler (data)
     Called for any characters in the XML document for which no
     applicable handler has been specified.  This means characters that
     are part of a construct which could be reported, but for which no
     handler has been supplied.

 -- Method: xmlparser.DefaultHandlerExpand (data)
     This is the same as the DefaultHandler(),  but doesn't
     inhibit expansion of internal entities. The entity reference will
     not be passed to the default handler.

 -- Method: xmlparser.NotStandaloneHandler ()
     Called if the XML document hasn't been declared as being a
     standalone document.  This happens when there is an external
     subset or a reference to a parameter entity, but the XML
     declaration does not set standalone to `yes' in an XML
     declaration.  If this handler returns `0', then the parser will
     raise an `XML_ERROR_NOT_STANDALONE' error.  If this handler is not
     set, no exception is raised by the parser for this condition.

 -- Method: xmlparser.ExternalEntityRefHandler (context, base,
          systemId, publicId)
     Called for references to external entities.  _base_ is the current
     base, as set by a previous call to SetBase().  The
     public and system identifiers, _systemId_ and _publicId_, are
     strings if given; if the public identifier is not given,
     _publicId_ will be `None'.  The _context_ value is opaque and
     should only be used as described below.

     For external entities to be parsed, this handler must be
     implemented. It is responsible for creating the sub-parser using
     `ExternalEntityParserCreate(context)', initializing it with the
     appropriate callbacks, and parsing the entity.  This handler
     should return an integer; if it returns `0', the parser will raise
     an `XML_ERROR_EXTERNAL_ENTITY_HANDLING' error, otherwise parsing
     will continue.

     If this handler is not provided, external entities are reported by
     the DefaultHandler callback, if provided.

5.19.5.2 ExpatError Exceptions
..............................

ExpatError exceptions have a number of interesting
attributes:

 -- Attribute: ExpatError.code
     Expat's internal error number for the specific error.  This will
     match one of the constants defined in the `errors' object from
     this module.

     New in version 2.1.

 -- Attribute: ExpatError.lineno
     Line number on which the error was detected.  The first line is
     numbered `1'.

     New in version 2.1.

 -- Attribute: ExpatError.offset
     Character offset into the line where the error occurred.  The
     first column is numbered `0'.

     New in version 2.1.

5.19.5.3 Example
................

The following program defines three handlers that just print out their
arguments.

    import xml.parsers.expat

    # 3 handler functions
    def start_element(name, attrs):
        print 'Start element:', name, attrs
    def end_element(name):
        print 'End element:', name
    def char_data(data):
        print 'Character data:', repr(data)

    p = xml.parsers.expat.ParserCreate()

    p.StartElementHandler = start_element
    p.EndElementHandler = end_element
    p.CharacterDataHandler = char_data

    p.Parse("""<?xml version="1.0"?>
    <parent id="top"><child1 name="paul">Text goes here</child1>
    <child2 name="fred">More text</child2>
    </parent>""", 1)

The output from this program is:

    Start element: parent {'id': 'top'}
    Start element: child1 {'name': 'paul'}
    Character data: 'Text goes here'
    End element: child1
    Character data: '\n'
    Start element: child2 {'name': 'fred'}
    Character data: 'More text'
    End element: child2
    Character data: '\n'
    End element: parent

5.19.5.4 Content Model Descriptions
...................................

Content modules are described using nested tuples.  Each tuple contains
four values: the type, the quantifier, the name, and a tuple of
children.  Children are simply additional content module descriptions.

  The values of the first two fields are constants defined in the
`model' object of the xml.parsers.expat module.  These
constants can be collected in two groups: the model type group and the
quantifier group.

  The constants in the model type group are:

 -- Data: xml.parsers.expat.XML_CTYPE_ANY
     The element named by the model name was declared to have a content
     model of `ANY'.

 -- Data: xml.parsers.expat.XML_CTYPE_CHOICE
     The named element allows a choice from a number of options; this
     is used for content models such as `(A | B | C)'.

 -- Data: xml.parsers.expat.XML_CTYPE_EMPTY
     Elements which are declared to be `EMPTY' have this model type.

 -- Data: xml.parsers.expat.XML_CTYPE_MIXED

 -- Data: xml.parsers.expat.XML_CTYPE_NAME

 -- Data: xml.parsers.expat.XML_CTYPE_SEQ
     Models which represent a series of models which follow one after
     the other are indicated with this model type.  This is used for
     models such as `(A, B, C)'.

  The constants in the quantifier group are:

 -- Data: xml.parsers.expat.XML_CQUANT_NONE
     No modifier is given, so it can appear exactly once, as for `A'.

 -- Data: xml.parsers.expat.XML_CQUANT_OPT
     The model is optional: it can appear once or not at all, as for
     `A?'.

 -- Data: xml.parsers.expat.XML_CQUANT_PLUS
     The model must occur one or more times (like `A+').

 -- Data: xml.parsers.expat.XML_CQUANT_REP
     The model must occur zero or more times, as for `A*'.

5.19.5.5 Expat error constants
..............................

The following constants are provided in the `errors' object of the
xml.parsers.expat module.  These constants are useful in
interpreting some of the attributes of the ExpatError.
exception objects raised when an error has occurred.

  The `errors' object has the following attributes:

 -- Data: xml.parsers.expat.XML_ERROR_ASYNC_ENTITY

 -- Data: xml.parsers.expat.XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF
     An entity reference in an attribute value referred to an external
     entity instead of an internal entity.

 -- Data: xml.parsers.expat.XML_ERROR_BAD_CHAR_REF
     A character reference referred to a character which is illegal in
     XML (for example, character `0', or '`&#0;'').

 -- Data: xml.parsers.expat.XML_ERROR_BINARY_ENTITY_REF
     An entity reference referred to an entity which was declared with
     a notation, so cannot be parsed.

 -- Data: xml.parsers.expat.XML_ERROR_DUPLICATE_ATTRIBUTE
     An attribute was used more than once in a start tag.

 -- Data: xml.parsers.expat.XML_ERROR_INCORRECT_ENCODING

 -- Data: xml.parsers.expat.XML_ERROR_INVALID_TOKEN
     Raised when an input byte could not properly be assigned to a
     character; for example, a NUL byte (value `0') in a UTF-8 input
     stream.

 -- Data: xml.parsers.expat.XML_ERROR_JUNK_AFTER_DOC_ELEMENT
     Something other than whitespace occurred after the document
     element.

 -- Data: xml.parsers.expat.XML_ERROR_MISPLACED_XML_PI
     An XML declaration was found somewhere other than the start of the
     input data.

 -- Data: xml.parsers.expat.XML_ERROR_NO_ELEMENTS
     The document contains no elements (XML requires all documents to
     contain exactly one top-level element)..

 -- Data: xml.parsers.expat.XML_ERROR_NO_MEMORY
     Expat was not able to allocate memory internally.

 -- Data: xml.parsers.expat.XML_ERROR_PARAM_ENTITY_REF
     A parameter entity reference was found where it was not allowed.

 -- Data: xml.parsers.expat.XML_ERROR_PARTIAL_CHAR
     An incomplete character was found in the input.

 -- Data: xml.parsers.expat.XML_ERROR_RECURSIVE_ENTITY_REF
     An entity reference contained another reference to the same
     entity; possibly via a different name, and possibly indirectly.

 -- Data: xml.parsers.expat.XML_ERROR_SYNTAX
     Some unspecified syntax error was encountered.

 -- Data: xml.parsers.expat.XML_ERROR_TAG_MISMATCH
     An end tag did not match the innermost open start tag.

 -- Data: xml.parsers.expat.XML_ERROR_UNCLOSED_TOKEN
     Some token (such as a start tag) was not closed before the end of
     the stream or the next token was encountered.

 -- Data: xml.parsers.expat.XML_ERROR_UNDEFINED_ENTITY
     A reference was made to a entity which was not defined.

 -- Data: xml.parsers.expat.XML_ERROR_UNKNOWN_ENCODING
     The document encoding is not supported by Expat.

 -- Data: xml.parsers.expat.XML_ERROR_UNCLOSED_CDATA_SECTION
     A CDATA marked section was not closed.

 -- Data: xml.parsers.expat.XML_ERROR_EXTERNAL_ENTITY_HANDLING

 -- Data: xml.parsers.expat.XML_ERROR_NOT_STANDALONE
     The parser determined that the document was not "standalone"
     though it declared itself to be in the XML declaration, and the
     `NotStandaloneHandler' was set and returned `0'.

 -- Data: xml.parsers.expat.XML_ERROR_UNEXPECTED_STATE

 -- Data: xml.parsers.expat.XML_ERROR_ENTITY_DECLARED_IN_PE

 -- Data: xml.parsers.expat.XML_ERROR_FEATURE_REQUIRES_XML_DTD
     An operation was requested that requires DTD support to be
     compiled in, but Expat was configured without DTD support.  This
     should never be reported by a standard build of the
     xml.parsers.expat module.

 -- Data: xml.parsers.expat.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING
     A behavioral change was requested after parsing started that can
     only be changed before parsing has started.  This is (currently)
     only raised by `UseForeignDTD()'.

 -- Data: xml.parsers.expat.XML_ERROR_UNBOUND_PREFIX
     An undeclared prefix was found when namespace processing was
     enabled.

 -- Data: xml.parsers.expat.XML_ERROR_UNDECLARING_PREFIX
     The document attempted to remove the namespace declaration
     associated with a prefix.

 -- Data: xml.parsers.expat.XML_ERROR_INCOMPLETE_PE
     A parameter entity contained incomplete markup.

 -- Data: xml.parsers.expat.XML_ERROR_XML_DECL
     The document contained no document element at all.

 -- Data: xml.parsers.expat.XML_ERROR_TEXT_DECL
     There was an error parsing a text declaration in an external
     entity.

 -- Data: xml.parsers.expat.XML_ERROR_PUBLICID
     Characters were found in the public id that are not allowed.

 -- Data: xml.parsers.expat.XML_ERROR_SUSPENDED
     The requested operation was made on a suspended parser, but isn't
     allowed.  This includes attempts to provide additional input or to
     stop the parser.

 -- Data: xml.parsers.expat.XML_ERROR_NOT_SUSPENDED
     An attempt to resume the parser was made when the parser had not
     been suspended.

 -- Data: xml.parsers.expat.XML_ERROR_ABORTED
     This should not be reported to Python applications.

 -- Data: xml.parsers.expat.XML_ERROR_FINISHED
     The requested operation was made on a parser which was finished
     parsing input, but isn't allowed.  This includes attempts to
     provide additional input or to stop the parser.

 -- Data: xml.parsers.expat.XML_ERROR_SUSPEND_PE

Footnotes
.........

     (1) The encoding string included in XML output should conform to
     the appropriate standards. For example, "UTF-8" is valid, but
     "UTF8" is not. See
     <http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl>
     and <http://www.iana.org/assignments/character-sets> .

5.19.6 `xml.dom' -- The Document Object Model API
-------------------------------------------------

New in version 2.0.

  The Document Object Model, or "DOM," is a cross-language API from the
World Wide Web Consortium (W3C) for accessing and modifying XML
documents.  A DOM implementation presents an XML document as a tree
structure, or allows client code to build such a structure from
scratch.  It then gives access to the structure through a set of
objects which provided well-known interfaces.

  The DOM is extremely useful for random-access applications.  SAX only
allows you a view of one bit of the document at a time.  If you are
looking at one SAX element, you have no access to another.  If you are
looking at a text node, you have no access to a containing element.
When you write a SAX application, you need to keep track of your
program's position in the document somewhere in your own code.  SAX
does not do it for you.  Also, if you need to look ahead in the XML
document, you are just out of luck.

  Some applications are simply impossible in an event driven model with
no access to a tree.  Of course you could build some sort of tree
yourself in SAX events, but the DOM allows you to avoid writing that
code.  The DOM is a standard tree representation for XML data.

  The Document Object Model is being defined by the W3C in stages, or
"levels" in their terminology.  The Python mapping of the API is
substantially based on the DOM Level 2 recommendation.

  DOM applications typically start by parsing some XML into a DOM.  How
this is accomplished is not covered at all by DOM Level 1, and Level 2
provides only limited improvements: There is a `DOMImplementation'
object class which provides access to `Document' creation methods, but
no way to access an XML reader/parser/Document builder in an
implementation-independent way. There is also no well-defined way to
access these methods without an existing `Document' object.  In Python,
each DOM implementation will provide a function
getDOMImplementation(). DOM Level 3 adds a Load/Store
specification, which defines an interface to the reader, but this is
not yet available in the Python standard library.

  Once you have a DOM document object, you can access the parts of your
XML document through its properties and methods.  These properties are
defined in the DOM specification; this portion of the reference manual
describes the interpretation of the specification in Python.

  The specification provided by the W3C defines the DOM API for Java,
ECMAScript, and OMG IDL.  The Python mapping defined here is based in
large part on the IDL version of the specification, but strict
compliance is not required (though implementations are free to support
the strict mapping from IDL).  See section Conformance for
a detailed discussion of mapping requirements.

Document Object Model (DOM) Level 2 Specification (http://www.w3.org/TR/DOM-Level-2-Core/)
     The W3C recommendation upon which the Python DOM API is based.

Document Object Model (DOM) Level 1 Specification (http://www.w3.org/TR/REC-DOM-Level-1/)
     The W3C recommendation for the DOM supported by
     xml.dom.minidom.

Python Language Mapping Specification (http://www.omg.org/spec/PYTH/1.2/PDF)
     This specifies the mapping from OMG IDL to Python.

5.19.6.1 Module Contents
........................

The xml.dom contains the following functions:

 -- Function: xml.dom.registerDOMImplementation (name, factory)
     Register the _factory_ function with the name _name_.  The factory
     function should return an object which implements the
     `DOMImplementation' interface.  The factory function can return
     the same object every time, or a new one for each call, as
     appropriate for the specific implementation (e.g. if that
     implementation supports some customization).

 -- Function: xml.dom.getDOMImplementation ([name[, features]])
     Return a suitable DOM implementation. The _name_ is either
     well-known, the module name of a DOM implementation, or `None'. If
     it is not `None', imports the corresponding module and returns a
     `DOMImplementation' object if the import succeeds.  If no name is
     given, and if the environment variable `PYTHON_DOM' is set, this
     variable is used to find the implementation.

     If name is not given, this examines the available implementations
     to find one with the required feature set.  If no implementation
     can be found, raise an ImportError.  The features list
     must be a sequence of `(feature, version)' pairs which are passed
     to the `hasFeature()' method on available `DOMImplementation'
     objects.

  Some convenience constants are also provided:

 -- Data: xml.dom.EMPTY_NAMESPACE
     The value used to indicate that no namespace is associated with a
     node in the DOM.  This is typically found as the `namespaceURI' of
     a node, or used as the _namespaceURI_ parameter to a
     namespaces-specific method.

     New in version 2.2.

 -- Data: xml.dom.XML_NAMESPACE
     The namespace URI associated with the reserved prefix `xml', as
     defined by Namespaces in XML
     (http://www.w3.org/TR/REC-xml-names/)(section 4).

     New in version 2.2.

 -- Data: xml.dom.XMLNS_NAMESPACE
     The namespace URI for namespace declarations, as defined by
     Document Object Model (DOM) Level 2 Core Specification
     (http://www.w3.org/TR/DOM-Level-2-Core/core.html)(section 1.1.8).

     New in version 2.2.

 -- Data: xml.dom.XHTML_NAMESPACE
     The URI of the XHTML namespace as defined by XHTML 1.0: The
     Extensible HyperText Markup Language
     (http://www.w3.org/TR/xhtml1/)(section 3.1.1).

     New in version 2.2.

  In addition, see xml.dom contains a base `Node' class and the
DOM exception classes.  The `Node' class provided by this module does
not implement any of the methods or attributes defined by the DOM
specification; concrete DOM implementations must provide those.  The
`Node' class provided as part of this module does provide the constants
used for the `nodeType' attribute on concrete `Node' objects; they are
located within the class rather than at the module level to conform
with the DOM specifications.

5.19.6.2 Objects in the DOM
...........................

The definitive documentation for the DOM is the DOM specification from
the W3C.

  Note that DOM attributes may also be manipulated as nodes instead of
as simple strings.  It is fairly rare that you must do this, however,
so this usage is not yet documented.

Interface                            Section                                 Purpose
-------------------------------------------------------------------------------------------------------------------
`DOMImplementation'                  DOMImplementation Objects.  Interface to the underlying
                                                                             implementation.
`Node'                               Node Objects.               Base interface for most objects in a
                                                                             document.
`NodeList'                           NodeList Objects.           Interface for a sequence of nodes.
`DocumentType'                       DocumentType Objects.       Information about the declarations
                                                                             needed to process a document.
`Document'                           Document Objects.           Object which represents an entire
                                                                             document.
`Element'                            Element Objects.            Element nodes in the document
                                                                             hierarchy.
`Attr'                               Attr Objects.               Attribute value nodes on element
                                                                             nodes.
`Comment'                            Comment Objects.            Representation of comments in the
                                                                             source document.
`Text'                               Text and CDATASection Objects.
                                                                from the document.
`ProcessingInstruction'              ProcessingInstruction Objects.
                                                                representation.

  An additional section describes the exceptions defined for working
with the DOM in Python.

5.19.6.3 DOMImplementation Objects
..................................

The `DOMImplementation' interface provides a way for applications to
determine the availability of particular features in the DOM they are
using.  DOM Level 2 added the ability to create new `Document' and
`DocumentType' objects using the `DOMImplementation' as well.

 -- Method: DOMImplementation.hasFeature (feature, version)
     Return true if the feature identified by the pair of strings
     _feature_ and _version_ is implemented.

 -- Method: DOMImplementation.createDocument (namespaceUri,
          qualifiedName, doctype)
     Return a new `Document' object (the root of the DOM), with a child
     `Element' object having the given _namespaceUri_ and
     _qualifiedName_. The _doctype_ must be a `DocumentType' object
     created by createDocumentType(), or `None'. In the
     Python DOM API, the first two arguments can also be `None' in
     order to indicate that no `Element' child is to be created.

 -- Method: DOMImplementation.createDocumentType (qualifiedName,
          publicId, systemId)
     Return a new `DocumentType' object that encapsulates the given
     _qualifiedName_, _publicId_, and _systemId_ strings, representing
     the information contained in an XML document type declaration.

5.19.6.4 Node Objects
.....................

All of the components of an XML document are subclasses of `Node'.

 -- Attribute: Node.nodeType
     An integer representing the node type.  Symbolic constants for the
     types are on the `Node' object: `ELEMENT_NODE', `ATTRIBUTE_NODE',
     `TEXT_NODE', `CDATA_SECTION_NODE', `ENTITY_NODE',
     `PROCESSING_INSTRUCTION_NODE', `COMMENT_NODE', `DOCUMENT_NODE',
     `DOCUMENT_TYPE_NODE', `NOTATION_NODE'.  This is a read-only
     attribute.

 -- Attribute: Node.parentNode
     The parent of the current node, or `None' for the document node.
     The value is always a `Node' object or `None'.  For `Element'
     nodes, this will be the parent element, except for the root
     element, in which case it will be the `Document' object. For
     `Attr' nodes, this is always `None'. This is a read-only attribute.

 -- Attribute: Node.attributes
     A `NamedNodeMap' of attribute objects.  Only elements have actual
     values for this; others provide `None' for this attribute. This is
     a read-only attribute.

 -- Attribute: Node.previousSibling
     The node that immediately precedes this one with the same parent.
     For instance the element with an end-tag that comes just before
     the _self_ element's start-tag.  Of course, XML documents are made
     up of more than just elements so the previous sibling could be
     text, a comment, or something else.  If this node is the first
     child of the parent, this attribute will be `None'. This is a
     read-only attribute.

 -- Attribute: Node.nextSibling
     The node that immediately follows this one with the same parent.
     See also previousSibling.  If this is the last child
     of the parent, this attribute will be `None'. This is a read-only
     attribute.

 -- Attribute: Node.childNodes
     A list of nodes contained within this node. This is a read-only
     attribute.

 -- Attribute: Node.firstChild
     The first child of the node, if there are any, or `None'. This is
     a read-only attribute.

 -- Attribute: Node.lastChild
     The last child of the node, if there are any, or `None'. This is a
     read-only attribute.

 -- Attribute: Node.localName
     The part of the `tagName' following the colon if there is one,
     else the entire `tagName'.  The value is a string.

 -- Attribute: Node.prefix
     The part of the `tagName' preceding the colon if there is one,
     else the empty string.  The value is a string, or `None'

 -- Attribute: Node.namespaceURI
     The namespace associated with the element name.  This will be a
     string or `None'.  This is a read-only attribute.

 -- Attribute: Node.nodeName
     This has a different meaning for each node type; see the DOM
     specification for details.  You can always get the information you
     would get here from another property such as the `tagName'
     property for elements or the `name' property for attributes. For
     all node types, the value of this attribute will be either a
     string or `None'.  This is a read-only attribute.

 -- Attribute: Node.nodeValue
     This has a different meaning for each node type; see the DOM
     specification for details.  The situation is similar to that with
     nodeName.  The value is a string or `None'.

 -- Method: Node.hasAttributes ()
     Returns true if the node has any attributes.

 -- Method: Node.hasChildNodes ()
     Returns true if the node has any child nodes.

 -- Method: Node.isSameNode (other)
     Returns true if _other_ refers to the same node as this node. This
     is especially useful for DOM implementations which use any sort of
     proxy architecture (because more than one object can refer to the
     same node).

          Note: This is based on a proposed DOM Level 3 API which is
          still in the "working draft" stage, but this particular
          interface appears uncontroversial.  Changes from the W3C will
          not necessarily affect this method in the Python DOM interface
          (though any new W3C API for this would also be supported).

 -- Method: Node.appendChild (newChild)
     Add a new child node to this node at the end of the list of
     children, returning _newChild_. If the node was already in in the
     tree, it is removed first.

 -- Method: Node.insertBefore (newChild, refChild)
     Insert a new child node before an existing child.  It must be the
     case that _refChild_ is a child of this node; if not, see
     ValueError is raised.  _newChild_ is returned. If _refChild_
     is `None', it inserts _newChild_ at the end of the children's list.

 -- Method: Node.removeChild (oldChild)
     Remove a child node.  _oldChild_ must be a child of this node; if
     not, see ValueError is raised.  _oldChild_ is returned on
     success.  If _oldChild_ will not be used further, its `unlink()'
     method should be called.

 -- Method: Node.replaceChild (newChild, oldChild)
     Replace an existing node with a new node. It must be the case that
     _oldChild_ is a child of this node; if not, see ValueError.
     is raised.

 -- Method: Node.normalize ()
     Join adjacent text nodes so that all stretches of text are stored
     as single `Text' instances.  This simplifies processing text from
     a DOM tree for many applications.

     New in version 2.1.

 -- Method: Node.cloneNode (deep)
     Clone this node.  Setting _deep_ means to clone all child nodes as
     well.  This returns the clone.

5.19.6.5 NodeList Objects
.........................

A `NodeList' represents a sequence of nodes.  These objects are used in
two ways in the DOM Core recommendation:  the `Element' objects provides
one as its list of child nodes, and the `getElementsByTagName()' and
`getElementsByTagNameNS()' methods of `Node' return objects with this
interface to represent query results.

  The DOM Level 2 recommendation defines one method and one attribute
for these objects:

 -- Method: NodeList.item (i)
     Return the _i_'th item from the sequence, if there is one, or
     `None'.  The index _i_ is not allowed to be less then zero or
     greater than or equal to the length of the sequence.

 -- Attribute: NodeList.length
     The number of nodes in the sequence.

  In addition, the Python DOM interface requires that some additional
support is provided to allow `NodeList' objects to be used as Python
sequences.  All `NodeList' implementations must include support for
__len__() and __getitem__().; this allows
iteration over the `NodeList' in for statements and proper
support for the len() built-in function.

  If a DOM implementation supports modification of the document, the
`NodeList' implementation must also support the __setitem__()
and __delitem__() methods.

5.19.6.6 DocumentType Objects
.............................

Information about the notations and entities declared by a document
(including the external subset if the parser uses it and can provide
the information) is available from a `DocumentType' object.  The
`DocumentType' for a document is available from the `Document' object's
`doctype' attribute; if there is no `DOCTYPE' declaration for the
document, the document's `doctype' attribute will be set to `None'
instead of an instance of this interface.

  `DocumentType' is a specialization of `Node', and adds the following
attributes:

 -- Attribute: DocumentType.publicId
     The public identifier for the external subset of the document type
     definition.  This will be a string or `None'.

 -- Attribute: DocumentType.systemId
     The system identifier for the external subset of the document type
     definition.  This will be a URI as a string, or `None'.

 -- Attribute: DocumentType.internalSubset
     A string giving the complete internal subset from the document.
     This does not include the brackets which enclose the subset.  If
     the document has no internal subset, this should be `None'.

 -- Attribute: DocumentType.name
     The name of the root element as given in the `DOCTYPE'
     declaration, if present.

 -- Attribute: DocumentType.entities
     This is a `NamedNodeMap' giving the definitions of external
     entities.  For entity names defined more than once, only the first
     definition is provided (others are ignored as required by the XML
     recommendation).  This may be `None' if the information is not
     provided by the parser, or if no entities are defined.

 -- Attribute: DocumentType.notations
     This is a `NamedNodeMap' giving the definitions of notations. For
     notation names defined more than once, only the first definition
     is provided (others are ignored as required by the XML
     recommendation).  This may be `None' if the information is not
     provided by the parser, or if no notations are defined.

5.19.6.7 Document Objects
.........................

A `Document' represents an entire XML document, including its
constituent elements, attributes, processing instructions, comments
etc.  Remember that it inherits properties from `Node'.

 -- Attribute: Document.documentElement
     The one and only root element of the document.

 -- Method: Document.createElement (tagName)
     Create and return a new element node.  The element is not inserted
     into the document when it is created.  You need to explicitly
     insert it with one of the other methods such as `insertBefore()'
     or `appendChild()'.

 -- Method: Document.createElementNS (namespaceURI, tagName)
     Create and return a new element with a namespace.  The _tagName_
     may have a prefix.  The element is not inserted into the document
     when it is created.  You need to explicitly insert it with one of
     the other methods such as `insertBefore()' or `appendChild()'.

 -- Method: Document.createTextNode (data)
     Create and return a text node containing the data passed as a
     parameter.  As with the other creation methods, this one does not
     insert the node into the tree.

 -- Method: Document.createComment (data)
     Create and return a comment node containing the data passed as a
     parameter.  As with the other creation methods, this one does not
     insert the node into the tree.

 -- Method: Document.createProcessingInstruction (target, data)
     Create and return a processing instruction node containing the
     _target_ and _data_ passed as parameters.  As with the other
     creation methods, this one does not insert the node into the tree.

 -- Method: Document.createAttribute (name)
     Create and return an attribute node.  This method does not
     associate the attribute node with any particular element.  You
     must use `setAttributeNode()' on the appropriate `Element' object
     to use the newly created attribute instance.

 -- Method: Document.createAttributeNS (namespaceURI, qualifiedName)
     Create and return an attribute node with a namespace.  The
     _tagName_ may have a prefix.  This method does not associate the
     attribute node with any particular element.  You must use
     `setAttributeNode()' on the appropriate `Element' object to use
     the newly created attribute instance.

 -- Method: Document.getElementsByTagName (tagName)
     Search for all descendants (direct children, children's children,
     etc.) with a particular element type name.

 -- Method: Document.getElementsByTagNameNS (namespaceURI, localName)
     Search for all descendants (direct children, children's children,
     etc.) with a particular namespace URI and localname.  The
     localname is the part of the namespace after the prefix.

5.19.6.8 Element Objects
........................

`Element' is a subclass of `Node', so inherits all the attributes of
that class.

 -- Attribute: Element.tagName
     The element type name.  In a namespace-using document it may have
     colons in it.  The value is a string.

 -- Method: Element.getElementsByTagName (tagName)
     Same as equivalent method in the `Document' class.

 -- Method: Element.getElementsByTagNameNS (namespaceURI, localName)
     Same as equivalent method in the `Document' class.

 -- Method: Element.hasAttribute (name)
     Returns true if the element has an attribute named by _name_.

 -- Method: Element.hasAttributeNS (namespaceURI, localName)
     Returns true if the element has an attribute named by
     _namespaceURI_ and _localName_.

 -- Method: Element.getAttribute (name)
     Return the value of the attribute named by _name_ as a string. If
     no such attribute exists, an empty string is returned, as if the
     attribute had no value.

 -- Method: Element.getAttributeNode (attrname)
     Return the `Attr' node for the attribute named by _attrname_.

 -- Method: Element.getAttributeNS (namespaceURI, localName)
     Return the value of the attribute named by _namespaceURI_ and
     _localName_ as a string. If no such attribute exists, an empty
     string is returned, as if the attribute had no value.

 -- Method: Element.getAttributeNodeNS (namespaceURI, localName)
     Return an attribute value as a node, given a _namespaceURI_ and
     _localName_.

 -- Method: Element.removeAttribute (name)
     Remove an attribute by name.  If there is no matching attribute, a
     NotFoundErr is raised.

 -- Method: Element.removeAttributeNode (oldAttr)
     Remove and return _oldAttr_ from the attribute list, if present.
     If _oldAttr_ is not present, see NotFoundErr is raised.

 -- Method: Element.removeAttributeNS (namespaceURI, localName)
     Remove an attribute by name.  Note that it uses a localName, not a
     qname.  No exception is raised if there is no matching attribute.

 -- Method: Element.setAttribute (name, value)
     Set an attribute value from a string.

 -- Method: Element.setAttributeNode (newAttr)
     Add a new attribute node to the element, replacing an existing
     attribute if necessary if the `name' attribute matches.  If a
     replacement occurs, the old attribute node will be returned.  If
     _newAttr_ is already in use, see InuseAttributeErr will
     be raised.

 -- Method: Element.setAttributeNodeNS (newAttr)
     Add a new attribute node to the element, replacing an existing
     attribute if necessary if the `namespaceURI' and `localName'
     attributes match.  If a replacement occurs, the old attribute node
     will be returned.  If _newAttr_ is already in use, see
     InuseAttributeErr will be raised.

 -- Method: Element.setAttributeNS (namespaceURI, qname, value)
     Set an attribute value from a string, given a _namespaceURI_ and a
     _qname_.  Note that a qname is the whole attribute name.  This is
     different than above.

5.19.6.9 Attr Objects
.....................

`Attr' inherits from `Node', so inherits all its attributes.

 -- Attribute: Attr.name
     The attribute name.  In a namespace-using document it may include
     a colon.

 -- Attribute: Attr.localName
     The part of the name following the colon if there is one, else the
     entire name.  This is a read-only attribute.

 -- Attribute: Attr.prefix
     The part of the name preceding the colon if there is one, else the
     empty string.

 -- Attribute: Attr.value
     The text value of the attribute.  This is a synonym for the
     `nodeValue' attribute.

5.19.6.10 NamedNodeMap Objects
..............................

`NamedNodeMap' does _not_ inherit from `Node'.

 -- Attribute: NamedNodeMap.length
     The length of the attribute list.

 -- Method: NamedNodeMap.item (index)
     Return an attribute with a particular index.  The order you get
     the attributes in is arbitrary but will be consistent for the life
     of a DOM.  Each item is an attribute node.  Get its value with the
     `value' attribute.

  There are also experimental methods that give this class more mapping
behavior.  You can use them or you can use the standardized
`getAttribute*()' family of methods on the `Element' objects.

5.19.6.11 Comment Objects
.........................

`Comment' represents a comment in the XML document.  It is a subclass of
`Node', but cannot have child nodes.

 -- Attribute: Comment.data
     The content of the comment as a string.  The attribute contains
     all characters between the leading `<!-'`-' and trailing `-'`->',
     but does not include them.

5.19.6.12 Text and CDATASection Objects
.......................................

The `Text' interface represents text in the XML document.  If the parser
and DOM implementation support the DOM's XML extension, portions of the
text enclosed in CDATA marked sections are stored in `CDATASection'
objects.  These two interfaces are identical, but provide different
values for the `nodeType' attribute.

  These interfaces extend the `Node' interface.  They cannot have child
nodes.

 -- Attribute: Text.data
     The content of the text node as a string.

     Note: The use of a `CDATASection' node does not indicate that the
     node represents a complete CDATA marked section, only that the
     content of the node was part of a CDATA section.  A single CDATA
     section may be represented by more than one node in the document
     tree.  There is no way to determine whether two adjacent
     `CDATASection' nodes represent different CDATA marked sections.

5.19.6.13 ProcessingInstruction Objects
.......................................

Represents a processing instruction in the XML document; this inherits
from the `Node' interface and cannot have child nodes.

 -- Attribute: ProcessingInstruction.target
     The content of the processing instruction up to the first
     whitespace character.  This is a read-only attribute.

 -- Attribute: ProcessingInstruction.data
     The content of the processing instruction following the first
     whitespace character.

5.19.6.14 Exceptions
....................

New in version 2.1.

  The DOM Level 2 recommendation defines a single exception, see
DOMException, and a number of constants that allow applications
to determine what sort of error occurred. See DOMException.
instances carry a code attribute that provides the
appropriate value for the specific exception.

  The Python DOM interface provides the constants, but also expands the
set of exceptions so that a specific exception exists for each of the
exception codes defined by the DOM.  The implementations must raise the
appropriate specific exception, each of which carries the appropriate
value for the code.  attribute.

 -- Exception: xml.dom.DOMException
     Base exception class used for all specific DOM exceptions.  This
     exception class cannot be directly instantiated.

 -- Exception: xml.dom.DomstringSizeErr
     Raised when a specified range of text does not fit into a string.
     This is not known to be used in the Python DOM implementations,
     but may be received from DOM implementations not written in Python.

 -- Exception: xml.dom.HierarchyRequestErr
     Raised when an attempt is made to insert a node where the node
     type is not allowed.

 -- Exception: xml.dom.IndexSizeErr
     Raised when an index or size parameter to a method is negative or
     exceeds the allowed values.

 -- Exception: xml.dom.InuseAttributeErr
     Raised when an attempt is made to insert an `Attr' node that is
     already present elsewhere in the document.

 -- Exception: xml.dom.InvalidAccessErr
     Raised if a parameter or an operation is not supported on the
     underlying object.

 -- Exception: xml.dom.InvalidCharacterErr
     This exception is raised when a string parameter contains a
     character that is not permitted in the context it's being used in
     by the XML 1.0 recommendation.  For example, attempting to create
     an `Element' node with a space in the element type name will cause
     this error to be raised.

 -- Exception: xml.dom.InvalidModificationErr
     Raised when an attempt is made to modify the type of a node.

 -- Exception: xml.dom.InvalidStateErr
     Raised when an attempt is made to use an object that is not
     defined or is no longer usable.

 -- Exception: xml.dom.NamespaceErr
     If an attempt is made to change any object in a way that is not
     permitted with regard to the Namespaces in XML
     (http://www.w3.org/TR/REC-xml-names/) recommendation, this
     exception is raised.

 -- Exception: xml.dom.NotFoundErr
     Exception when a node does not exist in the referenced context.
     For example, `NamedNodeMap.removeNamedItem()' will raise this if
     the node passed in does not exist in the map.

 -- Exception: xml.dom.NotSupportedErr
     Raised when the implementation does not support the requested type
     of object or operation.

 -- Exception: xml.dom.NoDataAllowedErr
     This is raised if data is specified for a node which does not
     support data.

 -- Exception: xml.dom.NoModificationAllowedErr
     Raised on attempts to modify an object where modifications are not
     allowed (such as for read-only nodes).

 -- Exception: xml.dom.SyntaxErr
     Raised when an invalid or illegal string is specified.

 -- Exception: xml.dom.WrongDocumentErr
     Raised when a node is inserted in a different document than it
     currently belongs to, and the implementation does not support
     migrating the node from one document to the other.

  The exception codes defined in the DOM recommendation map to the
exceptions described above according to this table:

Constant                                   Exception
---------------------------------------------------------------------------------
`DOMSTRING_SIZE_ERR'                       DomstringSizeErr.
`HIERARCHY_REQUEST_ERR'                    HierarchyRequestErr.
`INDEX_SIZE_ERR'                           IndexSizeErr.
`INUSE_ATTRIBUTE_ERR'                      InuseAttributeErr.
`INVALID_ACCESS_ERR'                       InvalidAccessErr.
`INVALID_CHARACTER_ERR'                    InvalidCharacterErr.
`INVALID_MODIFICATION_ERR'                 InvalidModificationErr.
`INVALID_STATE_ERR'                        InvalidStateErr.
`NAMESPACE_ERR'                            NamespaceErr.
`NOT_FOUND_ERR'                            NotFoundErr.
`NOT_SUPPORTED_ERR'                        NotSupportedErr.
`NO_DATA_ALLOWED_ERR'                      NoDataAllowedErr.
`NO_MODIFICATION_ALLOWED_ERR'              NoModificationAllowedErr.
`SYNTAX_ERR'                               SyntaxErr.
`WRONG_DOCUMENT_ERR'                       WrongDocumentErr.

5.19.6.15 Conformance
.....................

This section describes the conformance requirements and relationships
between the Python DOM API, the W3C DOM recommendations, and the OMG
IDL mapping for Python.

5.19.6.16 Type Mapping
......................

The primitive IDL types used in the DOM specification are mapped to
Python types according to the following table.

IDL Type               Python Type
-----------------------------------------------------------------------
`boolean'              `IntegerType' (with a value of `0' or `1')
`int'                  `IntegerType'
`long int'             `IntegerType'
`unsigned int'         `IntegerType'

  Additionally, the `DOMString' defined in the recommendation is mapped
to a Python string or Unicode string.  Applications should be able to
handle Unicode whenever a string is returned from the DOM.

  The IDL `null' value is mapped to `None', which may be accepted or
provided by the implementation whenever `null' is allowed by the API.

5.19.6.17 Accessor Methods
..........................

The mapping from OMG IDL to Python defines accessor functions for IDL
`attribute' declarations in much the way the Java mapping does.
Mapping the IDL declarations

    readonly attribute string someValue;
             attribute string anotherValue;

yields three accessor functions:  a "get" method for `someValue'
(`_get_someValue()'), and "get" and "set" methods for `anotherValue'
(`_get_anotherValue()' and `_set_anotherValue()').  The mapping, in
particular, does not require that the IDL attributes are accessible as
normal Python attributes:  `object.someValue' is _not_ required to
work, and may raise an AttributeError.

  The Python DOM API, however, _does_ require that normal attribute
access work.  This means that the typical surrogates generated by
Python IDL compilers are not likely to work, and wrapper objects may be
needed on the client if the DOM objects are accessed via CORBA. While
this does require some additional consideration for CORBA DOM clients,
the implementers with experience using DOM over CORBA from Python do
not consider this a problem.  Attributes that are declared `readonly'
may not restrict write access in all DOM implementations.

  In the Python DOM API, accessor functions are not required.  If
provided, they should take the form defined by the Python IDL mapping,
but these methods are considered unnecessary since the attributes are
accessible directly from Python.  "Set" accessors should never be
provided for `readonly' attributes.

  The IDL definitions do not fully embody the requirements of the W3C
DOM API, such as the notion of certain objects, such as the return
value of `getElementsByTagName()', being "live".  The Python DOM API
does not require implementations to enforce such requirements.

5.19.7 `xml.dom.minidom' -- Lightweight DOM implementation
----------------------------------------------------------

New in version 2.0.

  xml.dom.minidom is a light-weight implementation of the
Document Object Model interface.  It is intended to be simpler than the
full DOM and also significantly smaller.

  DOM applications typically start by parsing some XML into a DOM.  With
xml.dom.minidom, this is done through the parse functions:

    from xml.dom.minidom import parse, parseString

    dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

    datasource = open('c:\\temp\\mydata.xml')
    dom2 = parse(datasource)   # parse an open file

    dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')

The parse() function can take either a filename or an open
file object.

 -- Function: xml.dom.minidom.parse (filename_or_file[, parser[,
          bufsize]])
     Return a `Document' from the given input. _filename_or_file_ may be
     either a file name, or a file-like object. _parser_, if given,
     must be a SAX2 parser object. This function will change the
     document handler of the parser and activate namespace support;
     other parser configuration (like setting an entity resolver) must
     have been done in advance.

  If you have XML in a string, you can use the parseString()
function instead:

 -- Function: xml.dom.minidom.parseString (string[, parser])
     Return a `Document' that represents the _string_. This method
     creates a StringIO object for the string and passes
     that on to parse().

  Both functions return a `Document' object representing the content of
the document.

  What the parse() and parseString().
functions do is connect an XML parser with a "DOM builder" that can
accept parse events from any SAX parser and convert them into a DOM
tree.  The name of the functions are perhaps misleading, but are easy
to grasp when learning the interfaces.  The parsing of the document
will be completed before these functions return; it's simply that these
functions do not provide a parser implementation themselves.

  You can also create a `Document' by calling a method on a "DOM
Implementation" object.  You can get this object either by calling the
`getDOMImplementation()' function in the xml.dom package or
the xml.dom.minidom module. Using the implementation from
the xml.dom.minidom module will always return a `Document'
instance from the minidom implementation, while the version from
xml.dom may provide an alternate implementation (this is likely
if you have the PyXML package
(http://pyxml.sourceforge.net/)installed).  Once you have a `Document',
you can add child nodes to it to populate the DOM:

    from xml.dom.minidom import getDOMImplementation

    impl = getDOMImplementation()

    newdoc = impl.createDocument(None, "some_tag", None)
    top_element = newdoc.documentElement
    text = newdoc.createTextNode('Some textual content.')
    top_element.appendChild(text)

Once you have a DOM document object, you can access the parts of your
XML document through its properties and methods.  These properties are
defined in the DOM specification.  The main property of the document
object is the `documentElement' property.  It gives you the main
element in the XML document: the one that holds all others.  Here is an
example program:

    dom3 = parseString("<myxml>Some data</myxml>")
    assert dom3.documentElement.tagName == "myxml"

When you are finished with a DOM tree, you may optionally call the
`unlink()' method to encourage early cleanup of the now-unneeded
objects.  `unlink()' is a xml.dom.minidom.-specific
extension to the DOM API that renders the node and its descendants are
essentially useless.  Otherwise, Python's garbage collector will
eventually take care of the objects in the tree.

Document Object Model (DOM) Level 1 Specification (http://www.w3.org/TR/REC-DOM-Level-1/)
     The W3C recommendation for the DOM supported by
     xml.dom.minidom.

5.19.7.1 DOM Objects
....................

The definition of the DOM API for Python is given as part of the
xml.dom.  module documentation.  This section lists the
differences between the API and xml.dom.minidom.

 -- Method: Node.unlink ()
     Break internal references within the DOM so that it will be
     garbage collected on versions of Python without cyclic GC.  Even
     when cyclic GC is available, using this can make large amounts of
     memory available sooner, so calling this on DOM objects as soon as
     they are no longer needed is good practice.  This only needs to be
     called on the `Document' object, but may be called on child nodes
     to discard children of that node.

 -- Method: Node.writexml (writer[, indent=""[, addindent=""[,
          newl=""[, encoding=""]]]])
     Write XML to the writer object.  The writer should have a
     `write()' method which matches that of the file object interface.
     The _indent_ parameter is the indentation of the current node.
     The _addindent_ parameter is the incremental indentation to use
     for subnodes of the current one.  The _newl_ parameter specifies
     the string to use to terminate newlines.

     Changed in version 2.1: The optional keyword parameters _indent_,
     _addindent_, and _newl_ were added to support pretty output.

     Changed in version 2.3: For the `Document' node, an additional
     keyword argument _encoding_ can be used to specify the encoding
     field of the XML header.

 -- Method: Node.toxml ([encoding])
     Return the XML that the DOM represents as a string.

     With no argument, the XML header does not specify an encoding, and
     the result is Unicode string if the default encoding cannot
     represent all characters in the document. Encoding this string in
     an encoding other than UTF-8 is likely incorrect, since UTF-8 is
     the default encoding of XML.

     With an explicit _encoding_ (1) argument, the result is a byte
     string in the specified encoding. It is recommended that this
     argument is always specified. To avoid UnicodeError.
     exceptions in case of unrepresentable text data, the encoding
     argument should be specified as "utf-8".

     Changed in version 2.3: the _encoding_ argument was introduced;
     see writexml().

 -- Method: Node.toprettyxml ([indent=""[, newl=""[, encoding=""]]])
     Return a pretty-printed version of the document. _indent_
     specifies the indentation string and defaults to a tabulator;
     _newl_ specifies the string emitted at the end of each line and
     defaults to `\n'.

     New in version 2.1.

     Changed in version 2.3: the encoding argument was introduced; see
     writexml().

  The following standard DOM methods have special considerations with
xml.dom.minidom.:

 -- Method: Node.cloneNode (deep)
     Although this method was present in the version of
     xml.dom.minidom.  packaged with Python 2.0, it was seriously
     broken.  This has been corrected for subsequent releases.

5.19.7.2 DOM Example
....................

This example program is a fairly realistic example of a simple program.
In this particular case, we do not take much advantage of the
flexibility of the DOM.

    import xml.dom.minidom

    document = """\
    <slideshow>
    <title>Demo slideshow</title>
    <slide><title>Slide title</title>
    <point>This is a demo</point>
    <point>Of a program for processing slides</point>
    </slide>

    <slide><title>Another demo slide</title>
    <point>It is important</point>
    <point>To have more than</point>
    <point>one slide</point>
    </slide>
    </slideshow>
    """

    dom = xml.dom.minidom.parseString(document)

    def getText(nodelist):
        rc = []
        for node in nodelist:
            if node.nodeType == node.TEXT_NODE:
                rc.append(node.data)
        return ''.join(rc)

    def handleSlideshow(slideshow):
        print "<html>"
        handleSlideshowTitle(slideshow.getElementsByTagName("title")[0])
        slides = slideshow.getElementsByTagName("slide")
        handleToc(slides)
        handleSlides(slides)
        print "</html>"

    def handleSlides(slides):
        for slide in slides:
            handleSlide(slide)

    def handleSlide(slide):
        handleSlideTitle(slide.getElementsByTagName("title")[0])
        handlePoints(slide.getElementsByTagName("point"))

    def handleSlideshowTitle(title):
        print "<title>%s</title>" % getText(title.childNodes)

    def handleSlideTitle(title):
        print "<h2>%s</h2>" % getText(title.childNodes)

    def handlePoints(points):
        print "<ul>"
        for point in points:
            handlePoint(point)
        print "</ul>"

    def handlePoint(point):
        print "<li>%s</li>" % getText(point.childNodes)

    def handleToc(slides):
        for slide in slides:
            title = slide.getElementsByTagName("title")[0]
            print "<p>%s</p>" % getText(title.childNodes)

    handleSlideshow(dom)

5.19.7.3 minidom and the DOM standard
.....................................

The xml.dom.minidom module is essentially a DOM
1.0-compatible DOM with some DOM 2 features (primarily namespace
features).

  Usage of the DOM interface in Python is straight-forward.  The
following mapping rules apply:

   * Interfaces are accessed through instance objects. Applications
     should not instantiate the classes themselves; they should use the
     creator functions available on the `Document' object. Derived
     interfaces support all operations (and attributes) from the base
     interfaces, plus any new operations.

   * Operations are used as methods. Since the DOM uses only in.
     parameters, the arguments are passed in normal order (from
     left to right).  There are no optional arguments. `void'
     operations return `None'.

   * IDL attributes map to instance attributes. For compatibility with
     the OMG IDL language mapping for Python, an attribute `foo' can
     also be accessed through accessor methods `_get_foo()' and
     `_set_foo()'.  `readonly' attributes must not be changed; this is
     not enforced at runtime.

   * The types `short int', `unsigned int', `unsigned long long', and
     `boolean' all map to Python integer objects.

   * The type `DOMString' maps to Python strings.
     xml.dom.minidom supports either byte or Unicode strings, but
     will normally produce Unicode strings.  Values of type `DOMString'
     may also be `None' where allowed to have the IDL `null' value by
     the DOM specification from the W3C.

   * `const' declarations map to variables in their respective scope
     (e.g.  `xml.dom.minidom.Node.PROCESSING_INSTRUCTION_NODE'); they
     must not be changed.

   * `DOMException' is currently not supported in
     xml.dom.minidom.  Instead, see xml.dom.minidom uses
     standard Python exceptions such as TypeError and
     AttributeError.

   * `NodeList' objects are implemented using Python's built-in list
     type.  Starting with Python 2.2, these objects provide the
     interface defined in the DOM specification, but with earlier
     versions of Python they do not support the official API.  They
     are, however, much more "Pythonic" than the interface defined in
     the W3C recommendations.

  The following interfaces have no implementation in
xml.dom.minidom.:

   * `DOMTimeStamp'

   * `DocumentType' (added in Python 2.1)

   * `DOMImplementation' (added in Python 2.1)

   * `CharacterData'

   * `CDATASection'

   * `Notation'

   * `Entity'

   * `EntityReference'

   * `DocumentFragment'

  Most of these reflect information in the XML document that is not of
general utility to most DOM users.

Footnotes
.........

     (1) The encoding string included in XML output should conform to
     the appropriate standards. For example, "UTF-8" is valid, but
     "UTF8" is not. See
     <http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl>
     and <http://www.iana.org/assignments/character-sets> .

5.19.8 `xml.dom.pulldom' -- Support for building partial DOM trees
------------------------------------------------------------------

New in version 2.0.

  xml.dom.pulldom allows building only selected portions of
a Document Object Model representation of a document from SAX events.

 -- Class: xml.dom.pulldom.PullDOM ([documentFactory])
     xml.sax.handler.ContentHandler implementation that ...

 -- Class: xml.dom.pulldom.DOMEventStream (stream, parser, bufsize)
     ...

 -- Class: xml.dom.pulldom.SAX2DOM ([documentFactory])
     xml.sax.handler.ContentHandler implementation that ...

 -- Function: xml.dom.pulldom.parse (stream_or_string[, parser[,
          bufsize]])
     ...

 -- Function: xml.dom.pulldom.parseString (string[, parser])
     ...

 -- Data: xml.dom.pulldom.default_bufsize
     Default value for the _bufsize_ parameter to parse().

     Changed in version 2.1: The value of this variable can be changed
     before calling parse() and the new value will take
     effect.

5.19.8.1 DOMEventStream Objects
...............................

 -- Method: DOMEventStream.getEvent ()
     ...

 -- Method: DOMEventStream.expandNode (node)
     ...

 -- Method: DOMEventStream.reset ()
     ...

5.19.9 `xml.sax' -- Support for SAX2 parsers
--------------------------------------------

New in version 2.0.

  The xml.sax package provides a number of modules which
implement the Simple API for XML (SAX) interface for Python.  The
package itself provides the SAX exceptions and the convenience
functions which will be most used by users of the SAX API.

  The convenience functions are:

 -- Function: xml.sax.make_parser ([parser_list])
     Create and return a SAX `XMLReader' object.  The first parser
     found will be used.  If _parser_list_ is provided, it must be a
     sequence of strings which name modules that have a function named
     `create_parser()'.  Modules listed in _parser_list_ will be used
     before modules in the default list of parsers.

 -- Function: xml.sax.parse (filename_or_stream, handler[,
          error_handler])
     Create a SAX parser and use it to parse a document.  The document,
     passed in as _filename_or_stream_, can be a filename or a file
     object.  The _handler_ parameter needs to be a SAX
     `ContentHandler' instance.  If _error_handler_ is given, it must
     be a SAX `ErrorHandler' instance; if omitted,  see
     SAXParseException will be raised on all errors.  There is no
     return value; all work must be done by the _handler_ passed in.

 -- Function: xml.sax.parseString (string, handler[, error_handler])
     Similar to parse(), but parses from a buffer _string_
     received as a parameter.

  A typical SAX application uses three kinds of objects: readers,
handlers and input sources.  "Reader" in this context is another term
for parser, i.e. some piece of code that reads the bytes or characters
from the input source, and produces a sequence of events. The events
then get distributed to the handler objects, i.e. the reader invokes a
method on the handler.  A SAX application must therefore obtain a
reader object, create or open the input sources, create the handlers,
and connect these objects all together.  As the final step of
preparation, the reader is called to parse the input. During parsing,
methods on the handler objects are called based on structural and
syntactic events from the input data.

  For these objects, only the interfaces are relevant; they are
normally not instantiated by the application itself.  Since Python does
not have an explicit notion of interface, they are formally introduced
as classes, but applications may use implementations which do not
inherit from the provided classes.  The `InputSource', `Locator',
`Attributes', `AttributesNS', and `XMLReader' interfaces are defined in
the module xml.sax.xmlreader.  The handler interfaces are
defined in xml.sax.handler.  For convenience, `InputSource'
(which is often instantiated directly) and the handler classes are also
available from xml.sax.  These interfaces are described
below.

  In addition to these classes, see xml.sax provides the
following exception classes.

 -- Exception: xml.sax.SAXException (msg[, exception])
     Encapsulate an XML error or warning.  This class can contain basic
     error or warning information from either the XML parser or the
     application: it can be subclassed to provide additional
     functionality or to add localization.  Note that although the
     handlers defined in the `ErrorHandler' interface receive instances
     of this exception, it is not required to actually raise the
     exception -- it is also useful as a container for information.

     When instantiated, _msg_ should be a human-readable description of
     the error.  The optional _exception_ parameter, if given, should
     be `None' or an exception that was caught by the parsing code and
     is being passed along as information.

     This is the base class for the other SAX exception classes.

 -- Exception: xml.sax.SAXParseException (msg, exception, locator)
     Subclass of SAXException raised on parse errors.
     Instances of this class are passed to the methods of the SAX
     `ErrorHandler' interface to provide information about the parse
     error.  This class supports the SAX `Locator' interface as well as
     the SAXException interface.

 -- Exception: xml.sax.SAXNotRecognizedException (msg[, exception])
     Subclass of SAXException raised when a SAX
     `XMLReader' is confronted with an unrecognized feature or
     property.  SAX applications and extensions may use this class for
     similar purposes.

 -- Exception: xml.sax.SAXNotSupportedException (msg[, exception])
     Subclass of SAXException raised when a SAX
     `XMLReader' is asked to enable a feature that is not supported, or
     to set a property to a value that the implementation does not
     support.  SAX applications and extensions may use this class for
     similar purposes.

SAX: The Simple API for XML (http://www.saxproject.org/)
     This site is the focal point for the definition of the SAX API.
     It provides a Java implementation and online documentation.  Links
     to implementations and historical information are also available.

Module xml.sax.handler.
     Definitions of the interfaces for application-provided objects.

Module xml.sax.saxutils.
     Convenience functions for use in SAX applications.

Module xml.sax.xmlreader.
     Definitions of the interfaces for parser-provided objects.

5.19.9.1 SAXException Objects
.............................

The SAXException exception class supports the following
methods:

 -- Method: SAXException.getMessage ()
     Return a human-readable message describing the error condition.

 -- Method: SAXException.getException ()
     Return an encapsulated exception object, or `None'.

5.19.10 `xml.sax.handler' -- Base classes for SAX handlers
----------------------------------------------------------

New in version 2.0.

  The SAX API defines four kinds of handlers: content handlers, DTD
handlers, error handlers, and entity resolvers. Applications normally
only need to implement those interfaces whose events they are
interested in; they can implement the interfaces in a single object or
in multiple objects. Handler implementations should inherit from the
base classes provided in the module xml.sax.handler, so that
all methods get default implementations.

 -- Class: xml.sax.handler.ContentHandler
     This is the main callback interface in SAX, and the one most
     important to applications. The order of events in this interface
     mirrors the order of the information in the document.

 -- Class: xml.sax.handler.DTDHandler
     Handle DTD events.

     This interface specifies only those DTD events required for basic
     parsing (unparsed entities and attributes).

 -- Class: xml.sax.handler.EntityResolver
     Basic interface for resolving entities. If you create an object
     implementing this interface, then register the object with your
     Parser, the parser will call the method in your object to resolve
     all external entities.

 -- Class: xml.sax.handler.ErrorHandler
     Interface used by the parser to present error and warning messages
     to the application.  The methods of this object control whether
     errors are immediately converted to exceptions or are handled in
     some other way.

  In addition to these classes, see xml.sax.handler provides
symbolic constants for the feature and property names.

 -- Data: xml.sax.handler.feature_namespaces
         value: `"http://xml.org/sax/features/namespaces"'
         true: Perform Namespace processing.
         false: Optionally do not perform Namespace processing (implies namespace-prefixes; default).
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.feature_namespace_prefixes
         value: `"http://xml.org/sax/features/namespace-prefixes"'
         true: Report the original prefixed names and attributes used for Namespace declarations.
         false: Do not report attributes used for Namespace declarations, and optionally do not report original prefixed names (default).
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.feature_string_interning
         value: `"http://xml.org/sax/features/string-interning"'
         true: All element names, prefixes, attribute names, Namespace URIs, and local names are interned using the built-in intern function.
         false: Names are not necessarily interned, although they may be (default).
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.feature_validation
         value: `"http://xml.org/sax/features/validation"'
         true: Report all validation errors (implies external-general-entities and external-parameter-entities).
         false: Do not report validation errors.
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.feature_external_ges
         value: `"http://xml.org/sax/features/external-general-entities"'
         true: Include all external general (text) entities.
         false: Do not include external general entities.
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.feature_external_pes
         value: `"http://xml.org/sax/features/external-parameter-entities"'
         true: Include all external parameter entities, including the external DTD subset.
         false: Do not include any external parameter entities, even the external DTD subset.
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.all_features
     List of all features.

 -- Data: xml.sax.handler.property_lexical_handler
         value: `"http://xml.org/sax/properties/lexical-handler"'
         data type: xml.sax.sax2lib.LexicalHandler (not supported in Python 2)
         description: An optional extension handler for lexical events like comments.
         access: read/write

 -- Data: xml.sax.handler.property_declaration_handler
         value: `"http://xml.org/sax/properties/declaration-handler"'
         data type: xml.sax.sax2lib.DeclHandler (not supported in Python 2)
         description: An optional extension handler for DTD-related events other than notations and unparsed entities.
         access: read/write

 -- Data: xml.sax.handler.property_dom_node
         value: `"http://xml.org/sax/properties/dom-node"'
         data type: org.w3c.dom.Node (not supported in Python 2)
         description: When parsing, the current DOM node being visited if this is a DOM iterator; when not parsing, the root DOM node for iteration.
         access: (parsing) read-only; (not parsing) read/write

 -- Data: xml.sax.handler.property_xml_string
         value: `"http://xml.org/sax/properties/xml-string"'
         data type: String
         description: The literal string of characters that was the source for the current event.
         access: read-only

 -- Data: xml.sax.handler.all_properties
     List of all known property names.

5.19.10.1 ContentHandler Objects
................................

Users are expected to subclass ContentHandler to support
their application.  The following methods are called by the parser on
the appropriate events in the input document:

 -- Method: ContentHandler.setDocumentLocator (locator)
     Called by the parser to give the application a locator for
     locating the origin of document events.

     SAX parsers are strongly encouraged (though not absolutely
     required) to supply a locator: if it does so, it must supply the
     locator to the application by invoking this method before invoking
     any of the other methods in the DocumentHandler interface.

     The locator allows the application to determine the end position
     of any document-related event, even if the parser is not reporting
     an error. Typically, the application will use this information for
     reporting its own errors (such as character content that does not
     match an application's business rules). The information returned
     by the locator is probably not sufficient for use with a search
     engine.

     Note that the locator will return correct information only during
     the invocation of the events in this interface. The application
     should not attempt to use it at any other time.

 -- Method: ContentHandler.startDocument ()
     Receive notification of the beginning of a document.

     The SAX parser will invoke this method only once, before any other
     methods in this interface or in DTDHandler (except for
     setDocumentLocator()).

 -- Method: ContentHandler.endDocument ()
     Receive notification of the end of a document.

     The SAX parser will invoke this method only once, and it will be
     the last method invoked during the parse. The parser shall not
     invoke this method until it has either abandoned parsing (because
     of an unrecoverable error) or reached the end of input.

 -- Method: ContentHandler.startPrefixMapping (prefix, uri)
     Begin the scope of a prefix-URI Namespace mapping.

     The information from this event is not necessary for normal
     Namespace processing: the SAX XML reader will automatically
     replace prefixes for element and attribute names when the
     `feature_namespaces' feature is enabled (the default).

     There are cases, however, when applications need to use prefixes
     in character data or in attribute values, where they cannot safely
     be expanded automatically; the startPrefixMapping().
     and endPrefixMapping() events supply the information
     to the application to expand prefixes in those contexts itself, if
     necessary.

     Note that startPrefixMapping() and
     endPrefixMapping() events are not guaranteed to be properly
     nested relative to each-other: all startPrefixMapping()
     events will occur before the corresponding
     startElement() event, and all endPrefixMapping()
     events will occur after the corresponding
     endElement() event, but their order is not guaranteed.

 -- Method: ContentHandler.endPrefixMapping (prefix)
     End the scope of a prefix-URI mapping.

     See startPrefixMapping() for details. This event will
     always occur after the corresponding endElement().
     event, but the order of endPrefixMapping() events is
     not otherwise guaranteed.

 -- Method: ContentHandler.startElement (name, attrs)
     Signals the start of an element in non-namespace mode.

     The _name_ parameter contains the raw XML 1.0 name of the element
     type as a string and the _attrs_ parameter holds an object of the
     `Attributes' interface (see The Attributes Interface)
     containing the attributes of the element.  The object passed as
     _attrs_ may be re-used by the parser; holding on to a reference to
     it is not a reliable way to keep a copy of the attributes.  To
     keep a copy of the attributes, use the copy() method of
     the _attrs_ object.

 -- Method: ContentHandler.endElement (name)
     Signals the end of an element in non-namespace mode.

     The _name_ parameter contains the name of the element type, just
     as with the startElement() event.

 -- Method: ContentHandler.startElementNS (name, qname, attrs)
     Signals the start of an element in namespace mode.

     The _name_ parameter contains the name of the element type as a
     `(uri, localname)' tuple, the _qname_ parameter contains the raw
     XML 1.0 name used in the source document, and the _attrs_
     parameter holds an instance of the `AttributesNS' interface (see
     The AttributesNS Interface)  containing the
     attributes of the element.  If no namespace is associated with the
     element, the _uri_ component of _name_ will be `None'.  The object
     passed as _attrs_ may be re-used by the parser; holding on to a
     reference to it is not a reliable way to keep a copy of the
     attributes.  To keep a copy of the attributes, use the
     copy() method of the _attrs_ object.

     Parsers may set the _qname_ parameter to `None', unless the
     `feature_namespace_prefixes' feature is activated.

 -- Method: ContentHandler.endElementNS (name, qname)
     Signals the end of an element in namespace mode.

     The _name_ parameter contains the name of the element type, just
     as with the startElementNS() method, likewise the
     _qname_ parameter.

 -- Method: ContentHandler.characters (content)
     Receive notification of character data.

     The Parser will call this method to report each chunk of character
     data. SAX parsers may return all contiguous character data in a
     single chunk, or they may split it into several chunks; however,
     all of the characters in any single event must come from the same
     external entity so that the Locator provides useful information.

     _content_ may be a Unicode string or a byte string; the `expat'
     reader module produces always Unicode strings.

          Note: The earlier SAX 1 interface provided by the Python XML
          Special Interest Group used a more Java-like interface for
          this method.  Since most parsers used from Python did not
          take advantage of the older interface, the simpler signature
          was chosen to replace it.  To convert old code to the new
          interface, use _content_ instead of slicing content with the
          old _offset_ and _length_ parameters.

 -- Method: ContentHandler.ignorableWhitespace (whitespace)
     Receive notification of ignorable whitespace in element content.

     Validating Parsers must use this method to report each chunk of
     ignorable whitespace (see the W3C XML 1.0 recommendation, section
     2.10): non-validating parsers may also use this method if they are
     capable of parsing and using content models.

     SAX parsers may return all contiguous whitespace in a single
     chunk, or they may split it into several chunks; however, all of
     the characters in any single event must come from the same
     external entity, so that the Locator provides useful information.

 -- Method: ContentHandler.processingInstruction (target, data)
     Receive notification of a processing instruction.

     The Parser will invoke this method once for each processing
     instruction found: note that processing instructions may occur
     before or after the main document element.

     A SAX parser should never report an XML declaration (XML 1.0,
     section 2.8) or a text declaration (XML 1.0, section 4.3.1) using
     this method.

 -- Method: ContentHandler.skippedEntity (name)
     Receive notification of a skipped entity.

     The Parser will invoke this method once for each entity skipped.
     Non-validating processors may skip entities if they have not seen
     the declarations (because, for example, the entity was declared in
     an external DTD subset). All processors may skip external
     entities, depending on the values of the `feature_external_ges'
     and the `feature_external_pes' properties.

5.19.10.2 DTDHandler Objects
............................

DTDHandler instances provide the following methods:

 -- Method: DTDHandler.notationDecl (name, publicId, systemId)
     Handle a notation declaration event.

 -- Method: DTDHandler.unparsedEntityDecl (name, publicId, systemId,
          ndata)
     Handle an unparsed entity declaration event.

5.19.10.3 EntityResolver Objects
................................

 -- Method: EntityResolver.resolveEntity (publicId, systemId)
     Resolve the system identifier of an entity and return either the
     system identifier to read from as a string, or an InputSource to
     read from. The default implementation returns _systemId_.

5.19.10.4 ErrorHandler Objects
..............................

Objects with this interface are used to receive error and warning
information from the `XMLReader'.  If you create an object that
implements this interface, then register the object with your
`XMLReader', the parser will call the methods in your object to report
all warnings and errors. There are three levels of errors available:
warnings, (possibly) recoverable errors, and unrecoverable errors.  All
methods take a `SAXParseException' as the only parameter.  Errors and
warnings may be converted to an exception by raising the passed-in
exception object.

 -- Method: ErrorHandler.error (exception)
     Called when the parser encounters a recoverable error.  If this
     method does not raise an exception, parsing may continue, but
     further document information should not be expected by the
     application.  Allowing the parser to continue may allow additional
     errors to be discovered in the input document.

 -- Method: ErrorHandler.fatalError (exception)
     Called when the parser encounters an error it cannot recover from;
     parsing is expected to terminate when this method returns.

 -- Method: ErrorHandler.warning (exception)
     Called when the parser presents minor warning information to the
     application.  Parsing is expected to continue when this method
     returns, and document information will continue to be passed to
     the application. Raising an exception in this method will cause
     parsing to end.

5.19.11 `xml.sax.saxutils' -- SAX Utilities
-------------------------------------------

New in version 2.0.

  The module xml.sax.saxutils contains a number of classes
and functions that are commonly useful when creating SAX applications,
either in direct use, or as base classes.

 -- Function: xml.sax.saxutils.escape (data[, entities])
     Escape `'&'', `'<'', and `'>'' in a string of data.

     You can escape other strings of data by passing a dictionary as
     the optional _entities_ parameter.  The keys and values must all
     be strings; each key will be replaced with its corresponding
     value.  The characters `'&'', `'<'' and `'>'' are always escaped,
     even if _entities_ is provided.

 -- Function: xml.sax.saxutils.unescape (data[, entities])
     Unescape `'&amp;'', `'&lt;'', and `'&gt;'' in a string of data.

     You can unescape other strings of data by passing a dictionary as
     the optional _entities_ parameter.  The keys and values must all
     be strings; each key will be replaced with its corresponding
     value.  `'&amp'', `'&lt;'', and `'&gt;'' are always unescaped,
     even if _entities_ is provided.

     New in version 2.3.

 -- Function: xml.sax.saxutils.quoteattr (data[, entities])
     Similar to escape(), but also prepares _data_ to be
     used as an attribute value.  The return value is a quoted version
     of _data_ with any additional required replacements.
     quoteattr() will select a quote character based on the
     content of _data_, attempting to avoid encoding any quote
     characters in the string.  If both single- and double-quote
     characters are already in _data_, the double-quote characters will
     be encoded and _data_ will be wrapped in double-quotes.  The
     resulting string can be used directly as an attribute value:

         >>> print "<element attr=%s>" % quoteattr("ab ' cd \" ef")
         <element attr="ab ' cd &quot; ef">

     This function is useful when generating attribute values for HTML
     or any SGML using the reference concrete syntax.

     New in version 2.2.

 -- Class: xml.sax.saxutils.XMLGenerator ([out[, encoding]])
     This class implements the `ContentHandler' interface by writing SAX
     events back into an XML document. In other words, using an
     XMLGenerator.  as the content handler will reproduce the
     original document being parsed. _out_ should be a file-like object
     which will default to _sys.stdout_. _encoding_ is the encoding of
     the output stream which defaults to `'iso-8859-1''.

 -- Class: xml.sax.saxutils.XMLFilterBase (base)
     This class is designed to sit between an `XMLReader' and the client
     application's event handlers.  By default, it does nothing but
     pass requests up to the reader and events on to the handlers
     unmodified, but subclasses can override specific methods to modify
     the event stream or the configuration requests as they pass
     through.

 -- Function: xml.sax.saxutils.prepare_input_source (source[, base])
     This function takes an input source and an optional base URL and
     returns a fully resolved `InputSource' object ready for reading.
     The input source can be given as a string, a file-like object, or
     an `InputSource' object; parsers will use this function to
     implement the polymorphic _source_ argument to their `parse()'
     method.

5.19.12 `xml.sax.xmlreader' -- Interface for XML parsers
--------------------------------------------------------

New in version 2.0.

  SAX parsers implement the XMLReader interface. They are
implemented in a Python module, which must provide a function
`create_parser()'. This function is invoked by
xml.sax.make_parser() with no arguments to create a new  parser
object.

 -- Class: xml.sax.xmlreader.XMLReader
     Base class which can be inherited by SAX parsers.

 -- Class: xml.sax.xmlreader.IncrementalParser
     In some cases, it is desirable not to parse an input source at
     once, but to feed chunks of the document as they get available.
     Note that the reader will normally not read the entire file, but
     read it in chunks as well; still `parse()' won't return until the
     entire document is processed. So these interfaces should be used
     if the blocking behaviour of `parse()' is not desirable.

     When the parser is instantiated it is ready to begin accepting
     data from the feed method immediately. After parsing has been
     finished with a call to close the reset method must be called to
     make the parser ready to accept new data, either from feed or
     using the parse method.

     Note that these methods must _not_ be called during parsing, that
     is, after parse has been called and before it returns.

     By default, the class also implements the parse method of the
     XMLReader interface using the feed, close and reset methods of the
     IncrementalParser interface as a convenience to SAX 2.0 driver
     writers.

 -- Class: xml.sax.xmlreader.Locator
     Interface for associating a SAX event with a document location. A
     locator object will return valid results only during calls to
     DocumentHandler methods; at any other time, the results are
     unpredictable. If information is not available, methods may return
     `None'.

 -- Class: xml.sax.xmlreader.InputSource ([systemId])
     Encapsulation of the information needed by the XMLReader
     to read entities.

     This class may include information about the public identifier,
     system identifier, byte stream (possibly with character encoding
     information) and/or the character stream of an entity.

     Applications will create objects of this class for use in the
     XMLReader.parse() method and for returning from
     EntityResolver.resolveEntity.

     An InputSource belongs to the application, the
     XMLReader is not allowed to modify InputSource.
     objects passed to it from the application, although it may make
     copies and modify those.

 -- Class: xml.sax.xmlreader.AttributesImpl (attrs)
     This is an implementation of the `Attributes' interface (see
     section The Attributes Interface).  This is a
     dictionary-like object which represents the element attributes in
     a `startElement()' call. In addition to the most useful dictionary
     operations, it supports a number of other methods as described by
     the interface. Objects of this class should be instantiated by
     readers; _attrs_ must be a dictionary-like object containing a
     mapping from attribute names to attribute values.

 -- Class: xml.sax.xmlreader.AttributesNSImpl (attrs, qnames)
     Namespace-aware variant of AttributesImpl, which will
     be passed to `startElementNS()'. It is derived from
     AttributesImpl, but understands attribute names as
     two-tuples of _namespaceURI_ and _localname_. In addition, it
     provides a number of methods expecting qualified names as they
     appear in the original document.  This class implements the
     `AttributesNS' interface (see section The AttributesNS Interface).

5.19.12.1 XMLReader Objects
...........................

The XMLReader interface supports the following methods:

 -- Method: XMLReader.parse (source)
     Process an input source, producing SAX events. The _source_ object
     can be a system identifier (a string identifying the input source
     - typically a file name or an URL), a file-like object, or an
     InputSource object. When parse().
     returns, the input is completely processed, and the parser object
     can be discarded or reset. As a limitation, the current
     implementation only accepts byte streams; processing of character
     streams is for further study.

 -- Method: XMLReader.getContentHandler ()
     Return the current `ContentHandler'.

 -- Method: XMLReader.setContentHandler (handler)
     Set the current `ContentHandler'.  If no `ContentHandler' is set,
     content events will be discarded.

 -- Method: XMLReader.getDTDHandler ()
     Return the current `DTDHandler'.

 -- Method: XMLReader.setDTDHandler (handler)
     Set the current `DTDHandler'.  If no `DTDHandler' is set, DTD
     events will be discarded.

 -- Method: XMLReader.getEntityResolver ()
     Return the current `EntityResolver'.

 -- Method: XMLReader.setEntityResolver (handler)
     Set the current `EntityResolver'.  If no `EntityResolver' is set,
     attempts to resolve an external entity will result in opening the
     system identifier for the entity, and fail if it is not available.

 -- Method: XMLReader.getErrorHandler ()
     Return the current `ErrorHandler'.

 -- Method: XMLReader.setErrorHandler (handler)
     Set the current error handler.  If no `ErrorHandler' is set,
     errors will be raised as exceptions, and warnings will be printed.

 -- Method: XMLReader.setLocale (locale)
     Allow an application to set the locale for errors and warnings.

     SAX parsers are not required to provide localization for errors
     and warnings; if they cannot support the requested locale,
     however, they must raise a SAX exception.  Applications may
     request a locale change in the middle of a parse.

 -- Method: XMLReader.getFeature (featurename)
     Return the current setting for feature _featurename_.  If the
     feature is not recognized, `SAXNotRecognizedException' is raised.
     The well-known featurenames are listed in the module
     xml.sax.handler.

 -- Method: XMLReader.setFeature (featurename, value)
     Set the _featurename_ to _value_. If the feature is not recognized,
     `SAXNotRecognizedException' is raised. If the feature or its
     setting is not supported by the parser, _SAXNotSupportedException_
     is raised.

 -- Method: XMLReader.getProperty (propertyname)
     Return the current setting for property _propertyname_. If the
     property is not recognized, a `SAXNotRecognizedException' is
     raised. The well-known propertynames are listed in the module
     xml.sax.handler.

 -- Method: XMLReader.setProperty (propertyname, value)
     Set the _propertyname_ to _value_. If the property is not
     recognized, `SAXNotRecognizedException' is raised. If the property
     or its setting is not supported by the parser,
     _SAXNotSupportedException_ is raised.

5.19.12.2 IncrementalParser Objects
...................................

Instances of IncrementalParser offer the following
additional methods:

 -- Method: IncrementalParser.feed (data)
     Process a chunk of _data_.

 -- Method: IncrementalParser.close ()
     Assume the end of the document. That will check well-formedness
     conditions that can be checked only at the end, invoke handlers,
     and may clean up resources allocated during parsing.

 -- Method: IncrementalParser.reset ()
     This method is called after close has been called to reset the
     parser so that it is ready to parse new documents. The results of
     calling parse or feed after close without calling reset are
     undefined.

5.19.12.3 Locator Objects
.........................

Instances of Locator provide these methods:

 -- Method: Locator.getColumnNumber ()
     Return the column number where the current event ends.

 -- Method: Locator.getLineNumber ()
     Return the line number where the current event ends.

 -- Method: Locator.getPublicId ()
     Return the public identifier for the current event.

 -- Method: Locator.getSystemId ()
     Return the system identifier for the current event.

5.19.12.4 InputSource Objects
.............................

 -- Method: InputSource.setPublicId (id)
     Sets the public identifier of this InputSource.

 -- Method: InputSource.getPublicId ()
     Returns the public identifier of this InputSource.

 -- Method: InputSource.setSystemId (id)
     Sets the system identifier of this InputSource.

 -- Method: InputSource.getSystemId ()
     Returns the system identifier of this InputSource.

 -- Method: InputSource.setEncoding (encoding)
     Sets the character encoding of this InputSource.

     The encoding must be a string acceptable for an XML encoding
     declaration (see section 4.3.3 of the XML recommendation).

     The encoding attribute of the InputSource is ignored
     if the InputSource also contains a character stream.

 -- Method: InputSource.getEncoding ()
     Get the character encoding of this InputSource.

 -- Method: InputSource.setByteStream (bytefile)
     Set the byte stream (a Python file-like object which does not
     perform byte-to-character conversion) for this input source.

     The SAX parser will ignore this if there is also a character
     stream specified, but it will use a byte stream in preference to
     opening a URI connection itself.

     If the application knows the character encoding of the byte
     stream, it should set it with the setEncoding method.

 -- Method: InputSource.getByteStream ()
     Get the byte stream for this input source.

     The getEncoding method will return the character encoding for this
     byte stream, or None if unknown.

 -- Method: InputSource.setCharacterStream (charfile)
     Set the character stream for this input source. (The stream must
     be a Python 1.6 Unicode-wrapped file-like that performs conversion
     to Unicode strings.)

     If there is a character stream specified, the SAX parser will
     ignore any byte stream and will not attempt to open a URI
     connection to the system identifier.

 -- Method: InputSource.getCharacterStream ()
     Get the character stream for this input source.

5.19.12.5 The `Attributes' Interface
....................................

`Attributes' objects implement a portion of the mapping protocol,
including the methods copy(), `get()', `has_key()', `items()',
`keys()', and `values()'.  The following methods are also provided:

 -- Method: Attributes.getLength ()
     Return the number of attributes.

 -- Method: Attributes.getNames ()
     Return the names of the attributes.

 -- Method: Attributes.getType (name)
     Returns the type of the attribute _name_, which is normally
     `'CDATA''.

 -- Method: Attributes.getValue (name)
     Return the value of attribute _name_.

5.19.12.6 The `AttributesNS' Interface
......................................

This interface is a subtype of the `Attributes' interface (see section
The Attributes Interface).  All methods supported by that
interface are also available on `AttributesNS' objects.

  The following methods are also available:

 -- Method: AttributesNS.getValueByQName (name)
     Return the value for a qualified name.

 -- Method: AttributesNS.getNameByQName (name)
     Return the `(namespace, localname)' pair for a qualified _name_.

 -- Method: AttributesNS.getQNameByName (name)
     Return the qualified name for a `(namespace, localname)' pair.

 -- Method: AttributesNS.getQNames ()
     Return the qualified names of all attributes.

5.19.13 `xml.etree.ElementTree' -- The ElementTree XML API
----------------------------------------------------------

New in version 2.5.

  The Element type is a flexible container object,
designed to store hierarchical data structures in memory.  The type can
be described as a cross between a list and a dictionary.

  Each element has a number of properties associated with it:

   * a tag which is a string identifying what kind of data this element
     represents (the element type, in other words).

   * a number of attributes, stored in a Python dictionary.

   * a text string.

   * an optional tail string.

   * a number of child elements, stored in a Python sequence

  To create an element instance, use the Element.
constructor or the SubElement() factory function.

  The ElementTree class can be used to wrap an element
structure, and convert it from and to XML.

  A C implementation of this API is available as
`xml.etree.cElementTree'.

  See <http://effbot.org/zone/element-index.htm> for tutorials and
links to other docs.  Fredrik Lundh's page is also the location of the
development version of the xml.etree.ElementTree.

  Changed in version 2.7: The ElementTree API is updated to 1.3.  For
more information, see Introducing ElementTree 1.3
(http://effbot.org/zone/elementtree-13-intro.htm).

5.19.13.1 Functions
...................

 -- Function: xml.etree.ElementTree.Comment (text=None)
     Comment element factory.  This factory function creates a special
     element that will be serialized as an XML comment by the standard
     serializer.  The comment string can be either a bytestring or a
     Unicode string.  _text_ is a string containing the comment string.
     Returns an element instance representing a comment.

 -- Function: xml.etree.ElementTree.dump (elem)
     Writes an element tree or element structure to sys.stdout.  This
     function should be used for debugging only.

     The exact output format is implementation dependent.  In this
     version, it's written as an ordinary XML file.

     _elem_ is an element tree or an individual element.

 -- Function: xml.etree.ElementTree.fromstring (text)
     Parses an XML section from a string constant.  Same as
     XML().  _text_ is a string containing XML data.  Returns an
     Element instance.

 -- Function: xml.etree.ElementTree.fromstringlist (sequence,
          parser=None)
     Parses an XML document from a sequence of string fragments.
     _sequence_ is a list or other sequence containing XML data
     fragments.  _parser_ is an optional parser instance.  If not
     given, the standard XMLParser.  parser is used.
     Returns an Element instance.

     New in version 2.7.

 -- Function: xml.etree.ElementTree.iselement (element)
     Checks if an object appears to be a valid element object.
     _element_ is an element instance.  Returns a true value if this is
     an element object.

 -- Function: xml.etree.ElementTree.iterparse (source, events=None,
          parser=None)
     Parses an XML section into an element tree incrementally, and
     reports what's going on to the user.  _source_ is a filename or
     file object containing XML data.  _events_ is a list of events to
     report back.  If omitted, only "end" events are reported.
     _parser_ is an optional parser instance.  If not given, the
     standard XMLParser parser is used.  Returns an
     iterator providing `(event, elem)' pairs.

          Note: iterparse() only guarantees that it has
          seen the ">" character of a starting tag when it emits a
          "start" event, so the attributes are defined, but the
          contents of the text and tail attributes are undefined at
          that point.  The same applies to the element children; they
          may or may not be present.

          If you need a fully populated element, look for "end" events
          instead.

 -- Function: xml.etree.ElementTree.parse (source, parser=None)
     Parses an XML section into an element tree.  _source_ is a
     filename or file object containing XML data.  _parser_ is an
     optional parser instance.  If not given, the standard
     XMLParser parser is used.  Returns an ElementTree
     instance.

 -- Function: xml.etree.ElementTree.ProcessingInstruction (target,
          text=None)
     PI element factory.  This factory function creates a special
     element that will be serialized as an XML processing instruction.
     _target_ is a string containing the PI target.  _text_ is a string
     containing the PI contents, if given.  Returns an element
     instance, representing a processing instruction.

 -- Function: xml.etree.ElementTree.register_namespace (prefix, uri)
     Registers a namespace prefix.  The registry is global, and any
     existing mapping for either the given prefix or the namespace URI
     will be removed.  _prefix_ is a namespace prefix.  _uri_ is a
     namespace uri.  Tags and attributes in this namespace will be
     serialized with the given prefix, if at all possible.

     New in version 2.7.

 -- Function: xml.etree.ElementTree.SubElement (parent, tag, attrib={},
          **extra)
     Subelement factory.  This function creates an element instance,
     and appends it to an existing element.

     The element name, attribute names, and attribute values can be
     either bytestrings or Unicode strings.  _parent_ is the parent
     element.  _tag_ is the subelement name.  _attrib_ is an optional
     dictionary, containing element attributes.  _extra_ contains
     additional attributes, given as keyword arguments.  Returns an
     element instance.

 -- Function: xml.etree.ElementTree.tostring (element,
          encoding="us-ascii", method="xml")
     Generates a string representation of an XML element, including all
     subelements.  _element_ is an Element instance.
     _encoding_ (1) is the output encoding (default is US-ASCII).
     _method_ is either `"xml"', `"html"' or `"text"' (default is
     `"xml"').  Returns an encoded string containing the XML data.

 -- Function: xml.etree.ElementTree.tostringlist (element,
          encoding="us-ascii", method="xml")
     Generates a string representation of an XML element, including all
     subelements.  _element_ is an Element instance.
     _encoding_ (1) is the output encoding (default is US-ASCII).
     _method_ is either `"xml"', `"html"' or `"text"' (default is
     `"xml"').  Returns a list of encoded strings containing the XML
     data.  It does not guarantee any specific sequence, except that
     `"".join(tostringlist(element)) == tostring(element)'.

     New in version 2.7.

 -- Function: xml.etree.ElementTree.XML (text, parser=None)
     Parses an XML section from a string constant.  This function can
     be used to embed "XML literals" in Python code.  _text_ is a
     string containing XML data.  _parser_ is an optional parser
     instance.  If not given, the standard XMLParser.
     parser is used.  Returns an Element instance.

 -- Function: xml.etree.ElementTree.XMLID (text, parser=None)
     Parses an XML section from a string constant, and also returns a
     dictionary which maps from element id:s to elements.  _text_ is a
     string containing XML data.  _parser_ is an optional parser
     instance.  If not given, the standard XMLParser.
     parser is used.  Returns a tuple containing an Element
     instance and a dictionary.

5.19.13.2 Element Objects
.........................

 -- Class: xml.etree.ElementTree.Element (tag, attrib={}, **extra)
     Element class.  This class defines the Element interface, and
     provides a reference implementation of this interface.

     The element name, attribute names, and attribute values can be
     either bytestrings or Unicode strings.  _tag_ is the element name.
     _attrib_ is an optional dictionary, containing element attributes.
     _extra_ contains additional attributes, given as keyword arguments.

      -- Attribute: tag
          A string identifying what kind of data this element
          represents (the element type, in other words).

      -- Attribute: text
          The _text_ attribute can be used to hold additional data
          associated with the element.  As the name implies this
          attribute is usually a string but may be any
          application-specific object.  If the element is created from
          an XML file the attribute will contain any text found between
          the element tags.

      -- Attribute: tail
          The _tail_ attribute can be used to hold additional data
          associated with the element.  This attribute is usually a
          string but may be any application-specific object.  If the
          element is created from an XML file the attribute will
          contain any text found after the element's end tag and before
          the next tag.

      -- Attribute: attrib
          A dictionary containing the element's attributes.  Note that
          while the _attrib_ value is always a real mutable Python
          dictionary, an ElementTree implementation may choose to use
          another internal representation, and create the dictionary
          only if someone asks for it.  To take advantage of such
          implementations, use the dictionary methods below whenever
          possible.

     The following dictionary-like methods work on the element
     attributes.

      -- Method: clear ()
          Resets an element.  This function removes all subelements,
          clears all attributes, and sets the text and tail attributes
          to None.

      -- Method: get (key, default=None)
          Gets the element attribute named _key_.

          Returns the attribute value, or _default_ if the attribute
          was not found.

      -- Method: items ()
          Returns the element attributes as a sequence of (name, value)
          pairs.  The attributes are returned in an arbitrary order.

      -- Method: keys ()
          Returns the elements attribute names as a list.  The names
          are returned in an arbitrary order.

      -- Method: set (key, value)
          Set the attribute _key_ on the element to _value_.

     The following methods work on the element's children (subelements).

      -- Method: append (subelement)
          Adds the element _subelement_ to the end of this elements
          internal list of subelements.

      -- Method: extend (subelements)
          Appends _subelements_ from a sequence object with zero or
          more elements.  Raises AssertionError if a
          subelement is not a valid object.

          New in version 2.7.

      -- Method: find (match)
          Finds the first subelement matching _match_.  _match_ may be
          a tag name or path.  Returns an element instance or `None'.

      -- Method: findall (match)
          Finds all matching subelements, by tag name or path.  Returns
          a list containing all matching elements in document order.

      -- Method: findtext (match, default=None)
          Finds text for the first subelement matching _match_.
          _match_ may be a tag name or path.  Returns the text content
          of the first matching element, or _default_ if no element was
          found.  Note that if the matching element has no text content
          an empty string is returned.

      -- Method: getchildren ()
          Deprecated since version 2.7: Use `list(elem)' or iteration.

      -- Method: getiterator (tag=None)
          Deprecated since version 2.7: Use method
          Element.iter() instead.

      -- Method: insert (index, element)
          Inserts a subelement at the given position in this element.

      -- Method: iter (tag=None)
          Creates a tree iterator with the current element
          as the root.  The iterator iterates over this element and all
          elements below it, in document (depth first) order.  If _tag_
          is not `None' or `'*'', only elements whose tag equals _tag_
          are returned from the iterator.  If the tree structure is
          modified during iteration, the result is undefined.

      -- Method: iterfind (match)
          Finds all matching subelements, by tag name or path.  Returns
          an iterable yielding all matching elements in document order.

          New in version 2.7.

      -- Method: itertext ()
          Creates a text iterator.  The iterator loops over this
          element and all subelements, in document order, and returns
          all inner text.

          New in version 2.7.

      -- Method: makeelement (tag, attrib)
          Creates a new element object of the same type as this
          element.  Do not call this method, use the
          SubElement() factory function instead.

      -- Method: remove (subelement)
          Removes _subelement_ from the element.  Unlike the find*
          methods this method compares elements based on the instance
          identity, not on tag value or contents.

     Element objects also support the following sequence
     type methods for working with subelements: __delitem__(),
     see __getitem__(), __setitem__(),
     __len__().

     Caution: Elements with no subelements will test as `False'.  This
     behavior will change in future versions.  Use specific `len(elem)'
     or `elem is None' test instead.

         element = root.find('foo')

         if not element:  # careful!
             print "element not found, or element has no subelements"

         if element is None:
             print "element not found"

5.19.13.3 ElementTree Objects
.............................

 -- Class: xml.etree.ElementTree.ElementTree (element=None, file=None)
     ElementTree wrapper class.  This class represents an entire element
     hierarchy, and adds some extra support for serialization to and
     from standard XML.

     _element_ is the root element.  The tree is initialized with the
     contents of the XML _file_ if given.

      -- Method: _setroot (element)
          Replaces the root element for this tree.  This discards the
          current contents of the tree, and replaces it with the given
          element.  Use with care.  _element_ is an element instance.

      -- Method: find (match)
          Finds the first toplevel element matching _match_.  _match_
          may be a tag name or path.  Same as getroot().find(match).
          Returns the first matching element, or `None' if no element
          was found.

      -- Method: findall (match)
          Finds all matching subelements, by tag name or path.  Same as
          getroot().findall(match).  _match_ may be a tag name or path.
          Returns a list containing all matching elements, in document
          order.

      -- Method: findtext (match, default=None)
          Finds the element text for the first toplevel element with
          given tag.  Same as getroot().findtext(match).  _match_ may
          be a tag name or path.  _default_ is the value to return if
          the element was not found.  Returns the text content of the
          first matching element, or the default value no element was
          found.  Note that if the element is found, but has no text
          content, this method returns an empty string.

      -- Method: getiterator (tag=None)
          Deprecated since version 2.7: Use method
          ElementTree.iter() instead.

      -- Method: getroot ()
          Returns the root element for this tree.

      -- Method: iter (tag=None)
          Creates and returns a tree iterator for the root element.
          The iterator loops over all elements in this tree, in section
          order.  _tag_ is the tag to look for (default is to return
          all elements)

      -- Method: iterfind (match)
          Finds all matching subelements, by tag name or path.  Same as
          getroot().iterfind(match). Returns an iterable yielding all
          matching elements in document order.

          New in version 2.7.

      -- Method: parse (source, parser=None)
          Loads an external XML section into this element tree.
          _source_ is a file name or file object.  _parser_ is an
          optional parser instance.  If not given, the standard
          XMLParser parser is used.  Returns the section root element.

      -- Method: write (file, encoding="us-ascii",
               xml_declaration=None, method="xml")
          Writes the element tree to a file, as XML.  _file_ is a file
          name, or a file object opened for writing.  _encoding_ (1) is
          the output encoding (default is US-ASCII).  _xml_declaration_
          controls if an XML declaration should be added to the file.
          Use False for never, True for always, None for only if not
          US-ASCII or UTF-8 (default is None).  _method_ is either
          `"xml"', `"html"' or `"text"' (default is `"xml"').  Returns
          an encoded string.

This is the XML file that is going to be manipulated:

    <html>
        <head>
            <title>Example page</title>
        </head>
        <body>
            <p>Moved to <a href="http://example.org/">example.org</a>
            or <a href="http://example.com/">example.com</a>.</p>
        </body>
    </html>

Example of changing the attribute "target" of every link in first
paragraph:

    >>> from xml.etree.ElementTree import ElementTree
    >>> tree = ElementTree()
    >>> tree.parse("index.xhtml")
    <Element 'html' at 0xb77e6fac>
    >>> p = tree.find("body/p")     # Finds first occurrence of tag p in body
    >>> p
    <Element 'p' at 0xb77ec26c>
    >>> links = list(p.iter("a"))   # Returns list of all links
    >>> links
    [<Element 'a' at 0xb77ec2ac>, <Element 'a' at 0xb77ec1cc>]
    >>> for i in links:             # Iterates through all found links
    ...     i.attrib["target"] = "blank"
    >>> tree.write("output.xhtml")

5.19.13.4 QName Objects
.......................

 -- Class: xml.etree.ElementTree.QName (text_or_uri, tag=None)
     QName wrapper.  This can be used to wrap a QName attribute value,
     in order to get proper namespace handling on output.
     _text_or_uri_ is a string containing the QName value, in the form
     {uri}local, or, if the tag argument is given, the URI part of a
     QName.  If _tag_ is given, the first argument is interpreted as an
     URI, and this argument is interpreted as a local name.
     QName instances are opaque.

5.19.13.5 TreeBuilder Objects
.............................

 -- Class: xml.etree.ElementTree.TreeBuilder (element_factory=None)
     Generic element structure builder.  This builder converts a
     sequence of start, data, and end method calls to a well-formed
     element structure.  You can use this class to build an element
     structure using a custom XML parser, or a parser for some other
     XML-like format.  The _element_factory_ is called to create new
     Element instances when given.

      -- Method: close ()
          Flushes the builder buffers, and returns the toplevel document
          element.  Returns an Element instance.

      -- Method: data (data)
          Adds text to the current element.  _data_ is a string.  This
          should be either a bytestring, or a Unicode string.

      -- Method: end (tag)
          Closes the current element.  _tag_ is the element name.
          Returns the closed element.

      -- Method: start (tag, attrs)
          Opens a new element.  _tag_ is the element name.  _attrs_ is
          a dictionary containing element attributes.  Returns the
          opened element.

     In addition, a custom TreeBuilder object can provide
     the following method:

      -- Method: doctype (name, pubid, system)
          Handles a doctype declaration.  _name_ is the doctype name.
          _pubid_ is the public identifier.  _system_ is the system
          identifier.  This method does not exist on the default
          TreeBuilder class.

          New in version 2.7.

5.19.13.6 XMLParser Objects
...........................

 -- Class: xml.etree.ElementTree.XMLParser (html=0, target=None,
          encoding=None)
     Element structure builder for XML source data, based
     on the expat parser.  _html_ are predefined HTML entities.  This
     flag is not supported by the current implementation.  _target_ is
     the target object.  If omitted, the builder uses an instance of
     the standard TreeBuilder class.  _encoding_ (1) is optional.  If
     given, the value overrides the encoding specified in the XML file.

      -- Method: close ()
          Finishes feeding data to the parser.  Returns an element
          structure.

      -- Method: doctype (name, pubid, system)
          Deprecated since version 2.7: Define the
          TreeBuilder.doctype() method on a custom TreeBuilder
          target.

      -- Method: feed (data)
          Feeds data to the parser.  _data_ is encoded data.

XMLParser.feed() calls _target_'s `start()' method for
each opening tag, its `end()' method for each closing tag, and data is
processed by method `data()'.  See XMLParser.close().  calls
_target_'s method `close()'.  XMLParser can be used not
only for building a tree structure.  This is an example of counting the
maximum depth of an XML file:

    >>> from xml.etree.ElementTree import XMLParser
    >>> class MaxDepth:                     # The target object of the parser
    ...     maxDepth = 0
    ...     depth = 0
    ...     def start(self, tag, attrib):   # Called for each opening tag.
    ...         self.depth += 1
    ...         if self.depth > self.maxDepth:
    ...             self.maxDepth = self.depth
    ...     def end(self, tag):             # Called for each closing tag.
    ...         self.depth -= 1
    ...     def data(self, data):
    ...         pass            # We do not need to do anything with data.
    ...     def close(self):    # Called when all data has been parsed.
    ...         return self.maxDepth
    ...
    >>> target = MaxDepth()
    >>> parser = XMLParser(target=target)
    >>> exampleXml = """
    ... <a>
    ...   <b>
    ...   </b>
    ...   <b>
    ...     <c>
    ...       <d>
    ...       </d>
    ...     </c>
    ...   </b>
    ... </a>"""
    >>> parser.feed(exampleXml)
    >>> parser.close()
    4

Footnotes
.........

     (1) The encoding string included in XML output should conform to
     the appropriate standards.  For example, "UTF-8" is valid, but
     "UTF8" is not.  See
     <http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl>
     and <http://www.iana.org/assignments/character-sets>.

5.20 Internet Protocols and Support
===================================

The modules described in this chapter implement Internet protocols and
support for related technology.  They are all implemented in Python.
Most of these modules require the presence of the system-dependent
module socket, which is currently supported on most popular
platforms.  Here is an overview:

5.20.1 `webbrowser' -- Convenient Web-browser controller
--------------------------------------------------------

The webbrowser module provides a high-level interface to
allow displaying Web-based documents to users. Under most
circumstances, simply calling the open() function from
this module will do the right thing.

  Under Unix, graphical browsers are preferred under X11, but text-mode
browsers will be used if graphical browsers are not available or an X11
display isn't available.  If text-mode browsers are used, the calling
process will block until the user exits the browser.

  If the environment variable `BROWSER' exists, it is interpreted to
override the platform default list of browsers, as a os.pathsep.
-separated list of browsers to try in order.  When the value of a
list part contains the string `%s', then it is  interpreted as a
literal browser command line to be used with the argument URL
substituted for `%s'; if the part does not contain `%s', it is simply
interpreted as the name of the browser to launch. (1)

  For non-Unix platforms, or when a remote browser is available on
Unix, the controlling process will not wait for the user to finish with
the browser, but allow the remote browser to maintain its own windows
on the display.  If remote browsers are not available on Unix, the
controlling process will launch a new browser and wait.

  The script *webbrowser* can be used as a command-line interface for
the module. It accepts an URL as the argument. It accepts the following
optional parameters: `-n' opens the URL in a new browser window, if
possible; see -t opens the URL in a new browser page ("tab").
The options are, naturally, mutually exclusive.

  The following exception is defined:

 -- Exception: webbrowser.Error
     Exception raised when a browser control error occurs.

  The following functions are defined:

 -- Function: webbrowser.open (url[, new=0[, autoraise=True]])
     Display _url_ using the default browser. If _new_ is 0, the _url_
     is opened in the same browser window if possible.  If _new_ is 1,
     a new browser window is opened if possible.  If _new_ is 2, a new
     browser page ("tab") is opened if possible.  If _autoraise_ is
     `True', the window is raised if possible (note that under many
     window managers this will occur regardless of the setting of this
     variable).

     Note that on some platforms, trying to open a filename using this
     function, may work and start the operating system's associated
     program.  However, this is neither supported nor portable.

     Changed in version 2.5: _new_ can now be 2.

 -- Function: webbrowser.open_new (url)
     Open _url_ in a new window of the default browser, if possible,
     otherwise, open _url_ in the only browser window.

 -- Function: webbrowser.open_new_tab (url)
     Open _url_ in a new page ("tab") of the default browser, if
     possible, otherwise equivalent to open_new().

     New in version 2.5.

 -- Function: webbrowser.get ([name])
     Return a controller object for the browser type _name_.  If _name_
     is empty, return a controller for a default browser appropriate to
     the caller's environment.

 -- Function: webbrowser.register (name, constructor[, instance])
     Register the browser type _name_.  Once a browser type is
     registered, the get() function can return a
     controller for that browser type.  If _instance_ is not provided,
     or is `None', _constructor_ will be called without parameters to
     create an instance when needed.  If _instance_ is provided,
     _constructor_ will never be called, and may be `None'.

     This entry point is only useful if you plan to either set the `BROWSER'
     variable or call get() with a nonempty argument
     matching the name of a handler you declare.

  A number of browser types are predefined.  This table gives the type
names that may be passed to the get() function and the
corresponding instantiations for the controller classes, all defined in
this module.

Type Name                   Class Name                                    Notes
--------------------------------------------------------------------------------------
`'mozilla''                 `Mozilla('mozilla')'
`'firefox''                 `Mozilla('mozilla')'
`'netscape''                `Mozilla('netscape')'
`'galeon''                  `Galeon('galeon')'
`'epiphany''                `Galeon('epiphany')'
`'skipstone''               `BackgroundBrowser('skipstone')'
`'kfmclient''               `Konqueror()'                                 (1)
`'konqueror''               `Konqueror()'                                 (1)
`'kfm''                     `Konqueror()'                                 (1)
`'mosaic''                  `BackgroundBrowser('mosaic')'
`'opera''                   `Opera()'
`'grail''                   `Grail()'
`'links''                   `GenericBrowser('links')'
`'elinks''                  `Elinks('elinks')'
`'lynx''                    `GenericBrowser('lynx')'
`'w3m''                     `GenericBrowser('w3m')'
`'windows-default''         `WindowsDefault'                              (2)
`'internet-config''         `InternetConfig'                              (3)
`'macosx''                  `MacOSX('default')'                           (4)

  Notes:

  1. "Konqueror" is the file manager for the KDE desktop environment
     for Unix, and only makes sense to use if KDE is running.  Some way
     of reliably detecting KDE would be nice; the `KDEDIR' variable is
     not sufficient.  Note also that the name "kfm" is used even when
     using the *konqueror* command with KDE 2 -- the implementation
     selects the best strategy for running Konqueror.

  2. Only on Windows platforms.

  3. Only on Mac OS platforms; requires the standard MacPython
     ic module.

  4. Only on Mac OS X platform.

  Here are some simple examples:

    url = 'http://www.python.org/'

    # Open URL in a new tab, if a browser window is already open.
    webbrowser.open_new_tab(url + 'doc/')

    # Open URL in new window, raising the window if possible.
    webbrowser.open_new(url)

5.20.1.1 Browser Controller Objects
...................................

Browser controllers provide these methods which parallel three of the
module-level convenience functions:

 -- Method: controller.open (url[, new=0[, autoraise=True]])
     Display _url_ using the browser handled by this controller. If
     _new_ is 1, a new browser window is opened if possible. If _new_
     is 2, a new browser page ("tab") is opened if possible.

 -- Method: controller.open_new (url)
     Open _url_ in a new window of the browser handled by this
     controller, if possible, otherwise, open _url_ in the only browser
     window.  Alias open_new().

 -- Method: controller.open_new_tab (url)
     Open _url_ in a new page ("tab") of the browser handled by this
     controller, if possible, otherwise equivalent to open_new().

     New in version 2.5.

Footnotes
.........

     (1) Executables named here without a full path will be searched in
     the directories given in the `PATH' environment variable.

5.20.2 `cgi' -- Common Gateway Interface support
------------------------------------------------

Support module for Common Gateway Interface (CGI) scripts.

  This module defines a number of utilities for use by CGI scripts
written in Python.

5.20.2.1 Introduction
.....................

A CGI script is invoked by an HTTP server, usually to process user input
submitted through an HTML `<FORM>' or `<ISINDEX>' element.

  Most often, CGI scripts live in the server's special `cgi-bin'
directory.  The HTTP server places all sorts of information about the
request (such as the client's hostname, the requested URL, the query
string, and lots of other goodies) in the script's shell environment,
executes the script, and sends the script's output back to the client.

  The script's input is connected to the client too, and sometimes the
form data is read this way; at other times the form data is passed via
the "query string" part of the URL.  This module is intended to take
care of the different cases and provide a simpler interface to the
Python script.  It also provides a number of utilities that help in
debugging scripts, and the latest addition is support for file uploads
from a form (if your browser supports it).

  The output of a CGI script should consist of two sections, separated
by a blank line.  The first section contains a number of headers,
telling the client what kind of data is following.  Python code to
generate a minimal header section looks like this:

    print "Content-Type: text/html"     # HTML is following
    print                               # blank line, end of headers

The second section is usually HTML, which allows the client software to
display nicely formatted text with header, in-line images, etc. Here's
Python code that prints a simple piece of HTML:

    print "<TITLE>CGI script output</TITLE>"
    print "<H1>This is my first CGI script</H1>"
    print "Hello, world!"

5.20.2.2 Using the cgi module
.............................

Begin by writing `import cgi'.  Do not use `from cgi import *' -- the
module defines all sorts of names for its own use or for backward
compatibility that you don't want in your namespace.

  When you write a new script, consider adding these lines:

    import cgitb
    cgitb.enable()

This activates a special exception handler that will display detailed
reports in the Web browser if any errors occur.  If you'd rather not
show the guts of your program to users of your script, you can have the
reports saved to files instead, with code like this:

    import cgitb
    cgitb.enable(display=0, logdir="/tmp")

It's very helpful to use this feature during script development. The
reports produced by cgitb provide information that can save
you a lot of time in tracking down bugs.  You can always remove the
`cgitb' line later when you have tested your script and are confident
that it works correctly.

  To get at submitted form data, it's best to use the `FieldStorage'
class.  The other classes defined in this module are provided mostly
for backward compatibility. Instantiate it exactly once, without
arguments.  This reads the form contents from standard input or the
environment (depending on the value of various environment variables
set according to the CGI standard).  Since it may consume standard
input, it should be instantiated only once.

  The `FieldStorage' instance can be indexed like a Python dictionary.
It allows membership testing with the in operator, and also
supports the standard dictionary method `keys()' and the built-in
function len().  Form fields containing empty strings are
ignored and do not appear in the dictionary; to keep such values,
provide a true value for the optional _keep_blank_values_ keyword
parameter when creating the `FieldStorage' instance.

  For instance, the following code (which assumes that the
`Content-Type' header and blank line have already been printed) checks
that the fields `name' and `addr' are both set to a non-empty string:

    form = cgi.FieldStorage()
    if "name" not in form or "addr" not in form:
        print "<H1>Error</H1>"
        print "Please fill in the name and addr fields."
        return
    print "<p>name:", form["name"].value
    print "<p>addr:", form["addr"].value
    ...further form processing here...

Here the fields, accessed through `form[key]', are themselves instances
of `FieldStorage' (or `MiniFieldStorage', depending on the form
encoding). The `value' attribute of the instance yields the string value
of the field.  The `getvalue()' method returns this string value
directly; it also accepts an optional second argument as a default to
return if the requested key is not present.

  If the submitted form data contains more than one field with the same
name, the object retrieved by `form[key]' is not a `FieldStorage' or
`MiniFieldStorage' instance but a list of such instances.  Similarly, in
this situation, `form.getvalue(key)' would return a list of strings. If
you expect this possibility (when your HTML form contains multiple
fields with the same name), use the `getlist()' function, which always
returns a list of values (so that you do not need to special-case the
single item case).  For example, this code concatenates any number of
username fields, separated by commas:

    value = form.getlist("username")
    usernames = ",".join(value)

If a field represents an uploaded file, accessing the value via the
`value' attribute or the `getvalue()' method reads the entire file in
memory as a string.  This may not be what you want. You can test for an
uploaded file by testing either the `filename' attribute or the `file'
attribute.  You can then read the data at leisure from the `file'
attribute:

    fileitem = form["userfile"]
    if fileitem.file:
        # It's an uploaded file; count lines
        linecount = 0
        while 1:
            line = fileitem.file.readline()
            if not line: break
            linecount = linecount + 1

If an error is encountered when obtaining the contents of an uploaded
file (for example, when the user interrupts the form submission by
clicking on a Back or Cancel button) the `done' attribute of the object
for the field will be set to the value -1.

  The file upload draft standard entertains the possibility of
uploading multiple files from one field (using a recursive
`multipart/*' encoding).  When this occurs, the item will be a
dictionary-like `FieldStorage' item.  This can be determined by testing
its `type' attribute, which should be `multipart/form-data' (or perhaps
another MIME type matching `multipart/*').  In this case, it can be
iterated over recursively just like the top-level form object.

  When a form is submitted in the "old" format (as the query string or
as a single data part of type `application/x-www-form-urlencoded'), the
items will actually be instances of the class `MiniFieldStorage'.  In
this case, the `list', `file', and `filename' attributes are always
`None'.

  A form submitted via POST that also has a query string will contain
both `FieldStorage' and `MiniFieldStorage' items.

5.20.2.3 Higher Level Interface
...............................

New in version 2.2.

  The previous section explains how to read CGI form data using the
`FieldStorage' class.  This section describes a higher level interface
which was added to this class to allow one to do it in a more readable
and intuitive way.  The interface doesn't make the techniques described
in previous sections obsolete -- they are still useful to process file
uploads efficiently, for example.

  The interface consists of two simple methods. Using the methods you
can process form data in a generic way, without the need to worry
whether only one or more values were posted under one name.

  In the previous section, you learned to write following code anytime
you expected a user to post more than one value under one name:

    item = form.getvalue("item")
    if isinstance(item, list):
        # The user is requesting more than one item.
    else:
        # The user is requesting only one item.

This situation is common for example when a form contains a group of
multiple checkboxes with the same name:

    <input type="checkbox" name="item" value="1" />
    <input type="checkbox" name="item" value="2" />

In most situations, however, there's only one form control with a
particular name in a form and then you expect and need only one value
associated with this name.  So you write a script containing for
example this code:

    user = form.getvalue("user").upper()

The problem with the code is that you should never expect that a client
will provide valid input to your scripts.  For example, if a curious
user appends another `user=foo' pair to the query string, then the
script would crash, because in this situation the `getvalue("user")'
method call returns a list instead of a string.  Calling the
upper() method on a list is not valid (since lists do not have a
method of this name) and results in an AttributeError.
exception.

  Therefore, the appropriate way to read form data values was to always
use the code which checks whether the obtained value is a single value
or a list of values.  That's annoying and leads to less readable
scripts.

  A more convenient approach is to use the methods `getfirst()' and
`getlist()' provided by this higher level interface.

 -- Method: FieldStorage.getfirst (name[, default])
     This method always returns only one value associated with form
     field _name_.  The method returns only the first value in case
     that more values were posted under such name.  Please note that
     the order in which the values are received may vary from browser
     to browser and should not be counted on. (1)  If no such form
     field or value exists then the method returns the value specified
     by the optional parameter _default_.  This parameter defaults to
     `None' if not specified.

 -- Method: FieldStorage.getlist (name)
     This method always returns a list of values associated with form
     field _name_.  The method returns an empty list if no such form
     field or value exists for _name_.  It returns a list consisting of
     one item if only one such value exists.

  Using these methods you can write nice compact code:

    import cgi
    form = cgi.FieldStorage()
    user = form.getfirst("user", "").upper()    # This way it's safe.
    for item in form.getlist("item"):
        do_something(item)

5.20.2.4 Old classes
....................

Deprecated since version 2.6.

  `SvFormContentDict' stores single value form content as dictionary; it
assumes each field name occurs in the form only once.

  `FormContentDict' stores multiple value form content as a dictionary
(the form items are lists of values).  Useful if your form contains
multiple fields with the same name.

  Other classes (`FormContent', `InterpFormContentDict') are present
for backwards compatibility with really old applications only.

5.20.2.5 Functions
..................

These are useful if you want more control, or if you want to employ
some of the algorithms implemented in this module in other
circumstances.

 -- Function: cgi.parse (fp[, keep_blank_values[, strict_parsing]])
     Parse a query in the environment or from a file (the file defaults
     to `sys.stdin').  The _keep_blank_values_ and _strict_parsing_
     parameters are passed to urlparse.parse_qs().
     unchanged.

 -- Function: cgi.parse_qs (qs[, keep_blank_values[, strict_parsing]])
     This function is deprecated in this module. Use
     urlparse.parse_qs().  instead. It is maintained here only
     for backward compatiblity.

 -- Function: cgi.parse_qsl (qs[, keep_blank_values[, strict_parsing]])
     This function is deprecated in this module. Use
     urlparse.parse_qsl().  instead. It is maintained here only
     for backward compatiblity.

 -- Function: cgi.parse_multipart (fp, pdict)
     Parse input of type `multipart/form-data' (for  file uploads).
     Arguments are _fp_ for the input file and _pdict_ for a dictionary
     containing other parameters in the `Content-Type' header.

     Returns a dictionary just like urlparse.parse_qs().
     keys are the field names, each value is a list of values for that
     field.  This is easy to use but not much good if you are expecting
     megabytes to be uploaded -- in that case, use the `FieldStorage'
     class instead which is much more flexible.

     Note that this does not parse nested multipart parts -- use
     `FieldStorage' for that.

 -- Function: cgi.parse_header (string)
     Parse a MIME header (such as `Content-Type') into a main value and
     a dictionary of parameters.

 -- Function: cgi.test ()
     Robust test CGI script, usable as main program. Writes minimal
     HTTP headers and formats all information provided to the script in
     HTML form.

 -- Function: cgi.print_environ ()
     Format the shell environment in HTML.

 -- Function: cgi.print_form (form)
     Format a form in HTML.

 -- Function: cgi.print_directory ()
     Format the current directory in HTML.

 -- Function: cgi.print_environ_usage ()
     Print a list of useful (used by CGI) environment variables in HTML.

 -- Function: cgi.escape (s[, quote])
     Convert the characters `'&'', `'<'' and `'>'' in string _s_ to
     HTML-safe sequences.  Use this if you need to display text that
     might contain such characters in HTML.  If the optional flag
     _quote_ is true, the quotation mark character (`"') is also
     translated; this helps for inclusion in an HTML attribute value
     delimited by double quotes, as in `<a href="...">'.  Note that
     single quotes are never translated.

     If the value to be quoted might include single- or double-quote
     characters, or both, consider using the quoteattr().
     function in the xml.sax.saxutils module instead.

5.20.2.6 Caring about security
..............................

There's one important rule: if you invoke an external program (via the
os.system() or os.popen() functions. or others
with similar functionality), make very sure you don't pass arbitrary
strings received from the client to the shell.  This is a well-known
security hole whereby clever hackers anywhere on the Web can exploit a
gullible CGI script to invoke arbitrary shell commands.  Even parts of
the URL or field names cannot be trusted, since the request doesn't
have to come from your form!

  To be on the safe side, if you must pass a string gotten from a form
to a shell command, you should make sure the string contains only
alphanumeric characters, dashes, underscores, and periods.

5.20.2.7 Installing your CGI script on a Unix system
....................................................

Read the documentation for your HTTP server and check with your local
system administrator to find the directory where CGI scripts should be
installed; usually this is in a directory `cgi-bin' in the server tree.

  Make sure that your script is readable and executable by "others";
the Unix file mode should be `0755' octal (use `chmod 0755 filename').
Make sure that the first line of the script contains `#!' starting in
column 1 followed by the pathname of the Python interpreter, for
instance:

    #!/usr/local/bin/python

Make sure the Python interpreter exists and is executable by "others".

  Make sure that any files your script needs to read or write are
readable or writable, respectively, by "others" -- their mode should be
`0644' for readable and `0666' for writable.  This is because, for
security reasons, the HTTP server executes your script as user
"nobody", without any special privileges.  It can only read (write,
execute) files that everybody can read (write, execute).  The current
directory at execution time is also different (it is usually the
server's cgi-bin directory) and the set of environment variables is
also different from what you get when you log in.  In particular, don't
count on the shell's search path for executables ( `PATH') or the
Python module search path ( see PYTHONPATH) to be set to
anything interesting.

  If you need to load modules from a directory which is not on Python's
default module search path, you can change the path in your script,
before importing other modules.  For example:

    import sys
    sys.path.insert(0, "/usr/home/joe/lib/python")
    sys.path.insert(0, "/usr/local/lib/python")

(This way, the directory inserted last will be searched first!)

  Instructions for non-Unix systems will vary; check your HTTP server's
documentation (it will usually have a section on CGI scripts).

5.20.2.8 Testing your CGI script
................................

Unfortunately, a CGI script will generally not run when you try it from
the command line, and a script that works perfectly from the command
line may fail mysteriously when run from the server.  There's one
reason why you should still test your script from the command line: if
it contains a syntax error, the Python interpreter won't execute it at
all, and the HTTP server will most likely send a cryptic error to the
client.

  Assuming your script has no syntax errors, yet it does not work, you
have no choice but to read the next section.

5.20.2.9 Debugging CGI scripts
..............................

First of all, check for trivial installation errors -- reading the
section above on installing your CGI script carefully can save you a
lot of time.  If you wonder whether you have understood the
installation procedure correctly, try installing a copy of this module
file (`cgi.py') as a CGI script.  When invoked as a script, the file
will dump its environment and the contents of the form in HTML form.
Give it the right mode etc, and send it a request.  If it's installed
in the standard `cgi-bin' directory, it should be possible to send it a
request by entering a URL into your browser of the form:

    http://yourhostname/cgi-bin/cgi.py?name=Joe+Blow&addr=At+Home

If this gives an error of type 404, the server cannot find the script -
perhaps you need to install it in a different directory.  If it gives
another error, there's an installation problem that you should fix
before trying to go any further.  If you get a nicely formatted listing
of the environment and form content (in this example, the fields should
be listed as "addr" with value "At Home" and "name" with value "Joe
Blow"), the `cgi.py' script has been installed correctly.  If you
follow the same procedure for your own script, you should now be able
to debug it.

  The next step could be to call the cgi module's
test() function from your script: replace its main code with the
single statement

    cgi.test()

This should produce the same results as those gotten from installing the
`cgi.py' file itself.

  When an ordinary Python script raises an unhandled exception (for
whatever reason: of a typo in a module name, a file that can't be
opened, etc.), the Python interpreter prints a nice traceback and
exits.  While the Python interpreter will still do this when your CGI
script raises an exception, most likely the traceback will end up in
one of the HTTP server's log files, or be discarded altogether.

  Fortunately, once you have managed to get your script to execute
_some_ code, you can easily send tracebacks to the Web browser using
the cgitb module.  If you haven't done so already, just add
the lines:

    import cgitb
    cgitb.enable()

to the top of your script.  Then try running it again; when a problem
occurs, you should see a detailed report that will likely make apparent
the cause of the crash.

  If you suspect that there may be a problem in importing the
cgitb module, you can use an even more robust approach (which only
uses built-in modules):

    import sys
    sys.stderr = sys.stdout
    print "Content-Type: text/plain"
    print
    ...your code here...

This relies on the Python interpreter to print the traceback.  The
content type of the output is set to plain text, which disables all
HTML processing.  If your script works, the raw HTML will be displayed
by your client.  If it raises an exception, most likely after the first
two lines have been printed, a traceback will be displayed. Because no
HTML interpretation is going on, the traceback will be readable.

5.20.2.10 Common problems and solutions
.......................................

   * Most HTTP servers buffer the output from CGI scripts until the
     script is completed.  This means that it is not possible to
     display a progress report on the client's display while the script
     is running.

   * Check the installation instructions above.

   * Check the HTTP server's log files.  (`tail -f logfile' in a
     separate window may be useful!)

   * Always check a script for syntax errors first, by doing something
     like `python script.py'.

   * If your script does not have any syntax errors, try adding `import
     cgitb; cgitb.enable()' to the top of the script.

   * When invoking external programs, make sure they can be found.
     Usually, this means using absolute path names -- `PATH' is usually
     not set to a very useful value in a CGI script.

   * When reading or writing external files, make sure they can be read
     or written by the userid under which your CGI script will be
     running: this is typically the userid under which the web server
     is running, or some explicitly specified userid for a web server's
     `suexec' feature.

   * Don't try to give a CGI script a set-uid mode.  This doesn't work
     on most systems, and is a security liability as well.

Footnotes
.........

     (1) Note that some recent versions of the HTML specification do
     state what order the field values should be supplied in, but
     knowing whether a request was received from a conforming browser,
     or even from a browser at all, is tedious and error-prone.

5.20.3 `cgitb' -- Traceback manager for CGI scripts
---------------------------------------------------

New in version 2.2.

  The cgitb module provides a special exception handler for
Python scripts.  (Its name is a bit misleading.  It was originally
designed to display extensive traceback information in HTML for CGI
scripts.  It was later generalized to also display this information in
plain text.)  After this module is activated, if an uncaught exception
occurs, a detailed, formatted report will be displayed.  The report
includes a traceback showing excerpts of the source code for each level,
as well as the values of the arguments and local variables to currently
running functions, to help you debug the problem.  Optionally, you can
save this information to a file instead of sending it to the browser.

  To enable this feature, simply add this to the top of your CGI script:

    import cgitb
    cgitb.enable()

The options to the enable() function control whether the
report is displayed in the browser and whether the report is logged to
a file for later analysis.

 -- Function: cgitb.enable ([display[, logdir[, context[, format]]]])
     This function causes the cgitb module to take over the
     interpreter's default handling for exceptions by setting the value
     of sys.excepthook.

     The optional argument _display_ defaults to `1' and can be set to
     `0' to suppress sending the traceback to the browser. If the
     argument _logdir_ is present, the traceback reports are written to
     files.  The value of _logdir_ should be a directory where these
     files will be placed. The optional argument _context_ is the
     number of lines of context to display around the current line of
     source code in the traceback; this defaults to `5'. If the optional
     argument _format_ is `"html"', the output is formatted as HTML.
     Any other value forces plain text output.  The default value is
     `"html"'.

 -- Function: cgitb.handler ([info])
     This function handles an exception using the default settings
     (that is, show a report in the browser, but don't log to a file).
     This can be used when you've caught an exception and want to
     report it using cgitb.  The optional _info_ argument
     should be a 3-tuple containing an exception type, exception value,
     and traceback object, exactly like the tuple returned by
     sys.exc_info().  If the _info_ argument is not supplied, the
     current exception is obtained from sys.exc_info().

5.20.4 `wsgiref' -- WSGI Utilities and Reference Implementation
---------------------------------------------------------------

New in version 2.5.

  The Web Server Gateway Interface (WSGI) is a standard interface
between web server software and web applications written in Python.
Having a standard interface makes it easy to use an application that
supports WSGI with a number of different web servers.

  Only authors of web servers and programming frameworks need to know
every detail and corner case of the WSGI design.  You don't need to
understand every detail of WSGI just to install a WSGI application or
to write a web application using an existing framework.

  wsgiref is a reference implementation of the WSGI
specification that can be used to add WSGI support to a web server or
framework.  It provides utilities for manipulating WSGI environment
variables and response headers, base classes for implementing WSGI
servers, a demo HTTP server that serves WSGI applications, and a
validation tool that checks WSGI servers and applications for
conformance to the WSGI specification ( PEP 333
(http://www.python.org/dev/peps/pep-0333)).

  See <http://www.wsgi.org> for more information about WSGI, and links
to tutorials and other resources.

5.20.4.1 `wsgiref.util' - WSGI environment utilities
....................................................

This module provides a variety of utility functions for working with
WSGI environments.  A WSGI environment is a dictionary containing HTTP
request variables as described in PEP 333
(http://www.python.org/dev/peps/pep-0333).  All of the functions taking
an _environ_ parameter expect a WSGI-compliant dictionary to be
supplied; please see PEP 333 (http://www.python.org/dev/peps/pep-0333)
for a detailed specification.

 -- Function: wsgiref.util.guess_scheme (environ)
     Return a guess for whether `wsgi.url_scheme' should be "http" or
     "https", by checking for a `HTTPS' environment variable in the
     _environ_ dictionary.  The return value is a string.

     This function is useful when creating a gateway that wraps CGI or
     a CGI-like protocol such as FastCGI.  Typically, servers providing
     such protocols will include a `HTTPS' variable with a value of "1"
     "yes", or "on" when a request is received via SSL.  So, this
     function returns "https" if such a value is found, and "http"
     otherwise.

 -- Function: wsgiref.util.request_uri (environ[, include_query=1])
     Return the full request URI, optionally including the query
     string, using the algorithm found in the "URL Reconstruction"
     section of PEP 333 (http://www.python.org/dev/peps/pep-0333).  If
     _include_query_ is false, the query string is not included in the
     resulting URI.

 -- Function: wsgiref.util.application_uri (environ)
     Similar to request_uri(), except that the `PATH_INFO'
     and `QUERY_STRING' variables are ignored.  The result is the base
     URI of the application object addressed by the request.

 -- Function: wsgiref.util.shift_path_info (environ)
     Shift a single name from `PATH_INFO' to `SCRIPT_NAME' and return
     the name.  The _environ_ dictionary is _modified_ in-place; use a
     copy if you need to keep the original `PATH_INFO' or `SCRIPT_NAME'
     intact.

     If there are no remaining path segments in `PATH_INFO', `None' is
     returned.

     Typically, this routine is used to process each portion of a
     request URI path, for example to treat the path as a series of
     dictionary keys. This routine modifies the passed-in environment
     to make it suitable for invoking another WSGI application that is
     located at the target URI. For example, if there is a WSGI
     application at `/foo', and the request URI path is `/foo/bar/baz',
     and the WSGI application at `/foo' calls shift_path_info(),
     it will receive the string "bar", and the environment will
     be updated to be suitable for passing to a WSGI application at
     `/foo/bar'.  That is, `SCRIPT_NAME' will change from `/foo' to
     `/foo/bar', and `PATH_INFO' will change from `/bar/baz' to `/baz'.

     When `PATH_INFO' is just a "/", this routine returns an empty
     string and appends a trailing slash to `SCRIPT_NAME', even though
     empty path segments are normally ignored, and `SCRIPT_NAME'
     doesn't normally end in a slash.  This is intentional behavior, to
     ensure that an application can tell the difference between URIs
     ending in `/x' from ones ending in `/x/' when using this routine
     to do object traversal.

 -- Function: wsgiref.util.setup_testing_defaults (environ)
     Update _environ_ with trivial defaults for testing purposes.

     This routine adds various parameters required for WSGI, including
     `HTTP_HOST', `SERVER_NAME', `SERVER_PORT', `REQUEST_METHOD',
     `SCRIPT_NAME', `PATH_INFO', and all of the PEP 333
     (http://www.python.org/dev/peps/pep-0333)-defined `wsgi.*'
     variables.  It only supplies default values, and does not replace
     any existing settings for these variables.

     This routine is intended to make it easier for unit tests of WSGI
     servers and applications to set up dummy environments.  It should
     NOT be used by actual WSGI servers or applications, since the data
     is fake!

     Example usage:

         from wsgiref.util import setup_testing_defaults
         from wsgiref.simple_server import make_server

         # A relatively simple WSGI application. It's going to print out the
         # environment dictionary after being updated by setup_testing_defaults
         def simple_app(environ, start_response):
             setup_testing_defaults(environ)

             status = '200 OK'
             headers = [('Content-type', 'text/plain')]

             start_response(status, headers)

             ret = ["%s: %s\n" % (key, value)
                    for key, value in environ.iteritems()]
             return ret

         httpd = make_server('', 8000, simple_app)
         print "Serving on port 8000..."
         httpd.serve_forever()

  In addition to the environment functions above, the
wsgiref.util module also provides these miscellaneous utilities:

 -- Function: wsgiref.util.is_hop_by_hop (header_name)
     Return true if 'header_name' is an HTTP/1.1 "Hop-by-Hop" header,
     as defined by RFC 2616 (http://tools.ietf.org/html/rfc2616.html).

 -- Class: wsgiref.util.FileWrapper (filelike[, blksize=8192])
     A wrapper to convert a file-like object to an iterator.
     The resulting objects support both __getitem__() and
     __iter__() iteration styles, for compatibility with
     Python 2.1 and Jython. As the object is iterated over, the
     optional _blksize_ parameter will be repeatedly passed to the
     _filelike_ object's `read()' method to obtain strings to yield.
     When `read()' returns an empty string, iteration is ended and is
     not resumable.

     If _filelike_ has a `close()' method, the returned object will
     also have a `close()' method, and it will invoke the _filelike_
     object's `close()' method when called.

     Example usage:

         from StringIO import StringIO
         from wsgiref.util import FileWrapper

         # We're using a StringIO-buffer for as the file-like object
         filelike = StringIO("This is an example file-like object"*10)
         wrapper = FileWrapper(filelike, blksize=5)

         for chunk in wrapper:
             print chunk

5.20.4.2 `wsgiref.headers' - WSGI response header tools
.......................................................

This module provides a single class, see Headers, for convenient
manipulation of WSGI response headers using a mapping-like interface.

 -- Class: wsgiref.headers.Headers (headers)
     Create a mapping-like object wrapping _headers_, which must be a
     list of header name/value tuples as described in PEP 333
     (http://www.python.org/dev/peps/pep-0333).  Any changes made to
     the new Headers object will directly update the
     _headers_ list it was created with.

     Headers objects support typical mapping operations
     including __getitem__(), `get()', see __setitem__(),
     `setdefault()', see __delitem__(),
     __contains__() and `has_key()'.  For each of these methods,
     the key is the header name (treated case-insensitively), and the
     value is the first value associated with that header name.
     Setting a header deletes any existing values for that header, then
     adds a new value at the end of the wrapped header list.  Headers'
     existing order is generally maintained, with new headers added to
     the end of the wrapped list.

     Unlike a dictionary, see Headers objects do not raise an
     error when you try to get or delete a key that isn't in the
     wrapped header list. Getting a nonexistent header just returns
     `None', and deleting a nonexistent header does nothing.

     Headers objects also support `keys()', `values()', and
     `items()' methods.  The lists returned by `keys()' and `items()'
     can include the same key more than once if there is a multi-valued
     header.  The `len()' of a Headers object is the same
     as the length of its `items()', which is the same as the length of
     the wrapped header list.  In fact, the `items()' method just
     returns a copy of the wrapped header list.

     Calling `str()' on a Headers object returns a
     formatted string suitable for transmission as HTTP response
     headers.  Each header is placed on a line with its value,
     separated by a colon and a space. Each line is terminated by a
     carriage return and line feed, and the string is terminated with a
     blank line.

     In addition to their mapping interface and formatting features,
     see Headers.  objects also have the following methods for
     querying and adding multi-valued headers, and for adding headers
     with MIME parameters:

      -- Method: get_all (name)
          Return a list of all the values for the named header.

          The returned list will be sorted in the order they appeared
          in the original header list or were added to this instance,
          and may contain duplicates.  Any fields deleted and
          re-inserted are always appended to the header list.  If no
          fields exist with the given name, returns an empty list.

      -- Method: add_header (name, value, **_params)
          Add a (possibly multi-valued) header, with optional MIME
          parameters specified via keyword arguments.

          _name_ is the header field to add.  Keyword arguments can be
          used to set MIME parameters for the header field.  Each
          parameter must be a string or `None'.  Underscores in
          parameter names are converted to dashes, since dashes are
          illegal in Python identifiers, but many MIME parameter names
          include dashes.  If the parameter value is a string, it is
          added to the header value parameters in the form
          `name="value"'. If it is `None', only the parameter name is
          added.  (This is used for MIME parameters without a value.)
          Example usage:

              h.add_header('content-disposition', 'attachment', filename='bud.gif')

          The above will add a header that looks like this:

              Content-Disposition: attachment; filename="bud.gif"

5.20.4.3 `wsgiref.simple_server' - a simple WSGI HTTP server
............................................................

This module implements a simple HTTP server (based on
BaseHTTPServer)  that serves WSGI applications.  Each server
instance serves a single WSGI application on a given host and port.  If
you want to serve multiple applications on a single host and port, you
should create a WSGI application that parses `PATH_INFO' to select
which application to invoke for each request.  (E.g., using the
`shift_path_info()' function from wsgiref.util)

 -- Function: wsgiref.simple_server.make_server (host, port, app[,
          server_class=WSGIServer[, handler_class=WSGIRequestHandler]])
     Create a new WSGI server listening on _host_ and _port_, accepting
     connections for _app_.  The return value is an instance of the
     supplied _server_class_, and will process requests using the
     specified _handler_class_.  _app_ must be a WSGI application
     object, as defined by PEP 333
     (http://www.python.org/dev/peps/pep-0333).

     Example usage:

         from wsgiref.simple_server import make_server, demo_app

         httpd = make_server('', 8000, demo_app)
         print "Serving HTTP on port 8000..."

         # Respond to requests until process is killed
         httpd.serve_forever()

         # Alternative: serve one request, then exit
         httpd.handle_request()

 -- Function: wsgiref.simple_server.demo_app (environ, start_response)
     This function is a small but complete WSGI application that
     returns a text page containing the message "Hello world!" and a
     list of the key/value pairs provided in the _environ_ parameter.
     It's useful for verifying that a WSGI server (such as
     wsgiref.simple_server) is able to run a simple WSGI
     application correctly.

 -- Class: wsgiref.simple_server.WSGIServer (server_address,
          RequestHandlerClass)
     Create a WSGIServer instance.  _server_address_
     should be a `(host,port)' tuple, and _RequestHandlerClass_ should
     be the subclass of BaseHTTPServer.BaseHTTPRequestHandler
     that will be used to process requests.

     You do not normally need to call this constructor, as the
     make_server().  function can handle all the details for you.

     WSGIServer is a subclass of
     BaseHTTPServer.HTTPServer, so all of its methods (such as
     `serve_forever()' and `handle_request()') are available.
     WSGIServer also provides these WSGI-specific methods:

      -- Method: set_app (application)
          Sets the callable _application_ as the WSGI application that
          will receive requests.

      -- Method: get_app ()
          Returns the currently-set application callable.

     Normally, however, you do not need to use these additional
     methods, as set_app() is normally called by
     make_server(), and the get_app() exists mainly
     for the benefit of request handler instances.

 -- Class: wsgiref.simple_server.WSGIRequestHandler (request,
          client_address, server)
     Create an HTTP handler for the given _request_ (i.e. a socket),
     _client_address_ (a `(host,port)' tuple), and _server_ (see
     WSGIServer instance).

     You do not need to create instances of this class directly; they
     are automatically created as needed by WSGIServer.
     objects.  You can, however, subclass this class and supply it as a
     _handler_class_ to the make_server() function.  Some
     possibly relevant methods for overriding in subclasses:

      -- Method: get_environ ()
          Returns a dictionary containing the WSGI environment for a
          request.  The default implementation copies the contents of
          the WSGIServer object's `base_environ'
          dictionary attribute and then adds various headers derived
          from the HTTP request.  Each call to this method should
          return a new dictionary containing all of the relevant CGI
          environment variables as specified in PEP 333
          (http://www.python.org/dev/peps/pep-0333).

      -- Method: get_stderr ()
          Return the object that should be used as the `wsgi.errors'
          stream. The default implementation just returns `sys.stderr'.

      -- Method: handle ()
          Process the HTTP request.  The default implementation creates
          a handler instance using a wsgiref.handlers class
          to implement the actual WSGI application interface.

5.20.4.4 `wsgiref.validate' -- WSGI conformance checker
.......................................................

When creating new WSGI application objects, frameworks, servers, or
middleware, it can be useful to validate the new code's conformance
using wsgiref.validate.  This module provides a function
that creates WSGI application objects that validate communications
between a WSGI server or gateway and a WSGI application object, to
check both sides for protocol conformance.

  Note that this utility does not guarantee complete PEP 333
(http://www.python.org/dev/peps/pep-0333) compliance; an absence of
errors from this module does not necessarily mean that errors do not
exist.  However, if this module does produce an error, then it is
virtually certain that either the server or application is not 100%
compliant.

  This module is based on the `paste.lint' module from Ian Bicking's
"Python Paste" library.

 -- Function: wsgiref.validate.validator (application)
     Wrap _application_ and return a new WSGI application object.  The
     returned application will forward all requests to the original
     _application_, and will check that both the _application_ and the
     server invoking it are conforming to the WSGI specification and to
     RFC 2616.

     Any detected nonconformance results in an AssertionError
     being raised; note, however, that how these errors are
     handled is server-dependent.  For example, see
     wsgiref.simple_server and other servers based on
     wsgiref.handlers. (that don't override the error handling
     methods to do something else) will simply output a message that an
     error has occurred, and dump the traceback to `sys.stderr' or some
     other error stream.

     This wrapper may also generate output using the warnings
     module to indicate behaviors that are questionable but which
     may not actually be prohibited by PEP 333
     (http://www.python.org/dev/peps/pep-0333).  Unless they are
     suppressed using Python command-line options or the
     warnings. API, any such warnings will be written to
     `sys.stderr' (_not_ `wsgi.errors', unless they happen to be the
     same object).

     Example usage:

         from wsgiref.validate import validator
         from wsgiref.simple_server import make_server

         # Our callable object which is intentionally not compliant to the
         # standard, so the validator is going to break
         def simple_app(environ, start_response):
             status = '200 OK' # HTTP Status
             headers = [('Content-type', 'text/plain')] # HTTP Headers
             start_response(status, headers)

             # This is going to break because we need to return a list, and
             # the validator is going to inform us
             return "Hello World"

         # This is the application wrapped in a validator
         validator_app = validator(simple_app)

         httpd = make_server('', 8000, validator_app)
         print "Listening on port 8000...."
         httpd.serve_forever()

5.20.4.5 `wsgiref.handlers' - server/gateway base classes
.........................................................

This module provides base handler classes for implementing WSGI servers
and gateways.  These base classes handle most of the work of
communicating with a WSGI application, as long as they are given a
CGI-like environment, along with input, output, and error streams.

 -- Class: wsgiref.handlers.CGIHandler
     CGI-based invocation via `sys.stdin', `sys.stdout', `sys.stderr'
     and `os.environ'.  This is useful when you have a WSGI application
     and want to run it as a CGI script.  Simply invoke
     `CGIHandler().run(app)', where `app' is the WSGI application
     object you wish to invoke.

     This class is a subclass of BaseCGIHandler that sets
     `wsgi.run_once' to true, `wsgi.multithread' to false, and
     `wsgi.multiprocess' to true, and always uses sys and
     os to obtain the necessary CGI streams and environment.

 -- Class: wsgiref.handlers.BaseCGIHandler (stdin, stdout, stderr,
          environ[, multithread=True[, multiprocess=False]])
     Similar to CGIHandler, but instead of using the
     sys and os modules, the CGI environment and I/O
     streams are specified explicitly.  The _multithread_ and
     _multiprocess_ values are used to set the `wsgi.multithread' and
     `wsgi.multiprocess' flags for any applications run by the handler
     instance.

     This class is a subclass of SimpleHandler intended
     for use with software other than HTTP "origin servers".  If you
     are writing a gateway protocol implementation (such as CGI,
     FastCGI, SCGI, etc.) that uses a `Status:' header to send an HTTP
     status, you probably want to subclass this instead of
     SimpleHandler.

 -- Class: wsgiref.handlers.SimpleHandler (stdin, stdout, stderr,
          environ[, multithread=True[, multiprocess=False]])
     Similar to BaseCGIHandler, but designed for use with
     HTTP origin servers.  If you are writing an HTTP server
     implementation, you will probably want to subclass this instead of
     BaseCGIHandler.

     This class is a subclass of BaseHandler.  It overrides
     the __init__(), `get_stdin()', `get_stderr()',
     `add_cgi_vars()', `_write()', and `_flush()' methods to support
     explicitly setting the environment and streams via the
     constructor.  The supplied environment and streams are stored in
     the `stdin', `stdout', `stderr', and `environ' attributes.

 -- Class: wsgiref.handlers.BaseHandler
     This is an abstract base class for running WSGI applications.
     Each instance will handle a single HTTP request, although in
     principle you could create a subclass that was reusable for
     multiple requests.

     BaseHandler instances have only one method intended
     for external use:

      -- Method: run (app)
          Run the specified WSGI application, _app_.

     All of the other BaseHandler methods are invoked by
     this method in the process of running the application, and thus
     exist primarily to allow customizing the process.

     The following methods MUST be overridden in a subclass:

      -- Method: _write (data)
          Buffer the string _data_ for transmission to the client.
          It's okay if this method actually transmits the data; see
          BaseHandler just separates write and flush operations
          for greater efficiency when the underlying system actually
          has such a distinction.

      -- Method: _flush ()
          Force buffered data to be transmitted to the client.  It's
          okay if this method is a no-op (i.e., if _write()
          actually sends the data).

      -- Method: get_stdin ()
          Return an input stream object suitable for use as the
          `wsgi.input' of the request currently being processed.

      -- Method: get_stderr ()
          Return an output stream object suitable for use as the
          `wsgi.errors' of the request currently being processed.

      -- Method: add_cgi_vars ()
          Insert CGI variables for the current request into the
          `environ' attribute.

     Here are some other methods and attributes you may wish to
     override. This list is only a summary, however, and does not
     include every method that can be overridden.  You should consult
     the docstrings and source code for additional information before
     attempting to create a customized BaseHandler.
     subclass.

     Attributes and methods for customizing the WSGI environment:

      -- Attribute: wsgi_multithread
          The value to be used for the `wsgi.multithread' environment
          variable.  It defaults to true in BaseHandler,
          but may have a different default (or be set by the
          constructor) in the other subclasses.

      -- Attribute: wsgi_multiprocess
          The value to be used for the `wsgi.multiprocess' environment
          variable.  It defaults to true in BaseHandler,
          but may have a different default (or be set by the
          constructor) in the other subclasses.

      -- Attribute: wsgi_run_once
          The value to be used for the `wsgi.run_once' environment
          variable.  It defaults to false in BaseHandler,
          but CGIHandler sets it to true by default.

      -- Attribute: os_environ
          The default environment variables to be included in every
          request's WSGI environment.  By default, this is a copy of
          `os.environ' at the time that wsgiref.handlers.
          was imported, but subclasses can either create their own at
          the class or instance level.  Note that the dictionary should
          be considered read-only, since the default value is shared
          between multiple classes and instances.

      -- Attribute: server_software
          If the origin_server attribute is set, this
          attribute's value is used to set the default
          `SERVER_SOFTWARE' WSGI environment variable, and also to set a
          default `Server:' header in HTTP responses.  It is ignored
          for handlers (such as BaseCGIHandler and
          CGIHandler) that are not HTTP origin servers.

      -- Method: get_scheme ()
          Return the URL scheme being used for the current request.
          The default implementation uses the `guess_scheme()' function
          from wsgiref.util.  to guess whether the scheme
          should be "http" or "https", based on the current request's
          `environ' variables.

      -- Method: setup_environ ()
          Set the `environ' attribute to a fully-populated WSGI
          environment.  The default implementation uses all of the
          above methods and attributes, plus the get_stdin(),
          see get_stderr(), and add_cgi_vars()
          methods and the wsgi_file_wrapper.
          attribute.  It also inserts a `SERVER_SOFTWARE' key if not
          present, as long as the origin_server attribute
          is a true value and the server_software.
          attribute is set.

     Methods and attributes for customizing exception handling:

      -- Method: log_exception (exc_info)
          Log the _exc_info_ tuple in the server log.  _exc_info_ is a
          `(type, value, traceback)' tuple.  The default implementation
          simply writes the traceback to the request's `wsgi.errors'
          stream and flushes it.  Subclasses can override this method
          to change the format or retarget the output, mail the
          traceback to an administrator, or whatever other action may
          be deemed suitable.

      -- Attribute: traceback_limit
          The maximum number of frames to include in tracebacks output
          by the default log_exception() method.  If
          `None', all frames are included.

      -- Method: error_output (environ, start_response)
          This method is a WSGI application to generate an error page
          for the user.  It is only invoked if an error occurs before
          headers are sent to the client.

          This method can access the current error information using
          `sys.exc_info()', and should pass that information to
          _start_response_ when calling it (as described in the "Error
          Handling" section of PEP 333
          (http://www.python.org/dev/peps/pep-0333)).

          The default implementation just uses the error_status,
          see error_headers, and error_body.
          attributes to generate an output page.  Subclasses can
          override this to produce more dynamic error output.

          Note, however, that it's not recommended from a security
          perspective to spit out diagnostics to any old user; ideally,
          you should have to do something special to enable diagnostic
          output, which is why the default implementation doesn't
          include any.

      -- Attribute: error_status
          The HTTP status used for error responses.  This should be a
          status string as defined in PEP 333
          (http://www.python.org/dev/peps/pep-0333); it defaults to a
          500 code and message.

      -- Attribute: error_headers
          The HTTP headers used for error responses.  This should be a
          list of WSGI response headers (`(name, value)' tuples), as
          described in PEP 333
          (http://www.python.org/dev/peps/pep-0333).  The default list
          just sets the content type to `text/plain'.

      -- Attribute: error_body
          The error response body.  This should be an HTTP response
          body string. It defaults to the plain text, "A server error
          occurred.  Please contact the administrator."

     Methods and attributes for PEP 333
     (http://www.python.org/dev/peps/pep-0333)'s "Optional
     Platform-Specific File Handling" feature:

      -- Attribute: wsgi_file_wrapper
          A `wsgi.file_wrapper' factory, or `None'.  The default value
          of this attribute is the `FileWrapper' class from
          wsgiref.util.

      -- Method: sendfile ()
          Override to implement platform-specific file transmission.
          This method is called only if the application's return value
          is an instance of the class specified by the
          wsgi_file_wrapper attribute.  It should return a true
          value if it was able to successfully transmit the file, so
          that the default transmission code will not be executed. The
          default implementation of this method just returns a false
          value.

     Miscellaneous methods and attributes:

      -- Attribute: origin_server
          This attribute should be set to a true value if the handler's
          _write() and _flush() are being
          used to communicate directly to the client, rather than via a
          CGI-like gateway protocol that wants the HTTP status in a
          special `Status:' header.

          This attribute's default value is true in BaseHandler,
          but false in BaseCGIHandler and
          CGIHandler.

      -- Attribute: http_version
          If origin_server is true, this string attribute
          is used to set the HTTP version of the response set to the
          client.  It defaults to `"1.0"'.

5.20.4.6 Examples
.................

This is a working "Hello World" WSGI application:

    from wsgiref.simple_server import make_server

    # Every WSGI application must have an application object - a callable
    # object that accepts two arguments. For that purpose, we're going to
    # use a function (note that you're not limited to a function, you can
    # use a class for example). The first argument passed to the function
    # is a dictionary containing CGI-style envrironment variables and the
    # second variable is the callable object (see PEP 333).
    def hello_world_app(environ, start_response):
        status = '200 OK' # HTTP Status
        headers = [('Content-type', 'text/plain')] # HTTP Headers
        start_response(status, headers)

        # The returned object is going to be printed
        return ["Hello World"]

    httpd = make_server('', 8000, hello_world_app)
    print "Serving on port 8000..."

    # Serve until process is killed
    httpd.serve_forever()

5.20.5 `urllib' -- Open arbitrary resources by URL
--------------------------------------------------

     Note: The urllib module has been split into parts and
     renamed in Python 3.0 to `urllib.request', `urllib.parse', and
     `urllib.error'. The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.  Also note that the
     urllib.urlopen() function has been removed in Python
     3.0 in favor of urllib2.urlopen().

This module provides a high-level interface for fetching data across
the World Wide Web.  In particular, the urlopen() function
is similar to the built-in function open(), but accepts
Universal Resource Locators (URLs) instead of filenames.  Some
restrictions apply -- it can only open URLs for reading, and no seek
operations are available.

     Warning: When opening HTTPS URLs, it is not attempted to validate
     the server certificate.  Use at your own risk!

5.20.5.1 High-level interface
.............................

 -- Function: urllib.urlopen (url[, data[, proxies]])
     Open a network object denoted by a URL for reading.  If the URL
     does not have a scheme identifier, or if it has `file:' as its
     scheme identifier, this opens a local file (without universal
     newlines); otherwise it opens a socket to a server somewhere on
     the network.  If the connection cannot be made the IOError
     exception is raised.  If all went well, a file-like object is
     returned.  This supports the following methods: `read()', see
     readline(), `readlines()', `fileno()', `close()', `info()',
     `getcode()' and `geturl()'.  It also has proper support for the
     iterator protocol. One caveat: the `read()' method, if
     the size argument is omitted or negative, may not read until the
     end of the data stream; there is no good way to determine that the
     entire stream from a socket has been read in the general case.

     Except for the `info()', `getcode()' and `geturl()' methods, these
     methods have the same interface as for file objects -- see section
     File Objects in this manual.  (It is not a built-in
     file object, however, so it can't be used at those few places
     where a true built-in file object is required.)

     The `info()' method returns an instance of the class
     mimetools.Message containing meta-information associated
     with the URL.  When the method is HTTP, these headers are those
     returned by the server at the head of the retrieved HTML page
     (including Content-Length and Content-Type).  When the method is
     FTP, a Content-Length header will be present if (as is now usual)
     the server passed back a file length in response to the FTP
     retrieval request. A Content-Type header will be present if the
     MIME type can be guessed.  When the method is local-file, returned
     headers will include a Date representing the file's last-modified
     time, a Content-Length giving file size, and a Content-Type
     containing a guess at the file's type. See also the description of
     the mimetools module.

     The `geturl()' method returns the real URL of the page.  In some
     cases, the HTTP server redirects a client to another URL.  The
     urlopen() function handles this transparently, but in
     some cases the caller needs to know which URL the client was
     redirected to.  The `geturl()' method can be used to get at this
     redirected URL.

     The `getcode()' method returns the HTTP status code that was sent
     with the response, or `None' if the URL is no HTTP URL.

     If the _url_ uses the `http:' scheme identifier, the optional
     _data_ argument may be given to specify a `POST' request (normally
     the request type is `GET').  The _data_ argument must be in
     standard `application/x-www-form-urlencoded' format; see the
     urlencode().  function below.

     The urlopen() function works transparently with
     proxies which do not require authentication.  In a Unix or Windows
     environment, set the `http_proxy', or `ftp_proxy' environment
     variables to a URL that identifies the proxy server before
     starting the Python interpreter.  For example (the `'%'' is the
     command prompt):

         % http_proxy="http://www.someproxy.com:3128"
         % export http_proxy
         % python
         ...

     The `no_proxy' environment variable can be used to specify hosts
     which shouldn't be reached via proxy; if set, it should be a
     comma-separated list of hostname suffixes, optionally with `:port'
     appended, for example `cern.ch,ncsa.uiuc.edu,some.host:8080'.

     In a Windows environment, if no proxy environment variables are
     set, proxy settings are obtained from the registry's Internet
     Settings section.

     In a Mac OS X  environment, see urlopen() will retrieve
     proxy information from the OS X System Configuration Framework,
     which can be managed with Network System Preferences panel.

     Alternatively, the optional _proxies_ argument may be used to
     explicitly specify proxies.  It must be a dictionary mapping
     scheme names to proxy URLs, where an empty dictionary causes no
     proxies to be used, and `None' (the default value) causes
     environmental proxy settings to be used as discussed above.  For
     example:

         # Use http://www.someproxy.com:3128 for http proxying
         proxies = {'http': 'http://www.someproxy.com:3128'}
         filehandle = urllib.urlopen(some_url, proxies=proxies)
         # Don't use any proxies
         filehandle = urllib.urlopen(some_url, proxies={})
         # Use proxies from environment - both versions are equivalent
         filehandle = urllib.urlopen(some_url, proxies=None)
         filehandle = urllib.urlopen(some_url)

     Proxies which require authentication for use are not currently
     supported; this is considered an implementation limitation.

     Changed in version 2.3: Added the _proxies_ support.

     Changed in version 2.6: Added `getcode()' to returned object and
     support for the `no_proxy' environment variable.

     Deprecated since version 2.6: The urlopen() function
     has been removed in Python 3.0 in favor of
     urllib2.urlopen().

 -- Function: urllib.urlretrieve (url[, filename[, reporthook[, data]]])
     Copy a network object denoted by a URL to a local file, if
     necessary. If the URL points to a local file, or a valid cached
     copy of the object exists, the object is not copied.  Return a
     tuple `(filename, headers)' where _filename_ is the local file
     name under which the object can be found, and _headers_ is whatever
     the `info()' method of the object returned by urlopen()
     returned (for a remote object, possibly cached). Exceptions
     are the same as for urlopen().

     The second argument, if present, specifies the file location to
     copy to (if absent, the location will be a tempfile with a
     generated name). The third argument, if present, is a hook
     function that will be called once on establishment of the network
     connection and once after each block read thereafter.  The hook
     will be passed three arguments; a count of blocks transferred so
     far, a block size in bytes, and the total size of the file.  The
     third argument may be `-1' on older FTP servers which do not
     return a file size in response to a retrieval request.

     If the _url_ uses the `http:' scheme identifier, the optional
     _data_ argument may be given to specify a `POST' request (normally
     the request type is `GET').  The _data_ argument must in standard
     `application/x-www-form-urlencoded' format; see the
     urlencode().  function below.

     Changed in version 2.5: urlretrieve() will raise
     ContentTooShortError when it detects that the amount
     of data available  was less than the expected amount (which is the
     size reported by a  _Content-Length_ header). This can occur, for
     example, when the  download is interrupted.The _Content-Length_ is
     treated as a lower bound: if there's more data  to read,
     urlretrieve reads more data, but if less data is available,  it
     raises the exception.

     You can still retrieve the downloaded data in this case, it is
     stored  in the `content' attribute of the exception instance.

     If no _Content-Length_ header was supplied, urlretrieve can not
     check the size of the data it has downloaded, and just returns it.
     In this case you just have to assume that the download was
     successful.

 -- Data: urllib._urlopener
     The public functions urlopen() and
     urlretrieve() create an instance of the
     FancyURLopener class and use it to perform their requested
     actions.  To override this functionality, programmers can create a
     subclass of URLopener or FancyURLopener,
     then assign an instance of that class to the `urllib._urlopener'
     variable before calling the desired function.  For example,
     applications may want to specify a different `User-Agent' header
     than URLopener defines.  This can be accomplished
     with the following code:

         import urllib

         class AppURLopener(urllib.FancyURLopener):
             version = "App/1.7"

         urllib._urlopener = AppURLopener()

 -- Function: urllib.urlcleanup ()
     Clear the cache that may have been built up by previous calls to
     urlretrieve().

5.20.5.2 Utility functions
..........................

 -- Function: urllib.quote (string[, safe])
     Replace special characters in _string_ using the `%xx' escape.
     Letters, digits, and the characters `'_.-'' are never quoted. By
     default, this function is intended for quoting the path section of
     the URL.The optional _safe_ parameter specifies additional
     characters that should not be quoted -- its default value is `'/''.

     Example: `quote('/~connolly/')' yields `'/%7econnolly/''.

 -- Function: urllib.quote_plus (string[, safe])
     Like quote(), but also replaces spaces by plus signs,
     as required for quoting HTML form values when building up a query
     string to go into a URL.  Plus signs in the original string are
     escaped unless they are included in _safe_.  It also does not have
     _safe_ default to `'/''.

 -- Function: urllib.unquote (string)
     Replace `%xx' escapes by their single-character equivalent.

     Example: `unquote('/%7Econnolly/')' yields `'/~connolly/''.

 -- Function: urllib.unquote_plus (string)
     Like unquote(), but also replaces plus signs by
     spaces, as required for unquoting HTML form values.

 -- Function: urllib.urlencode (query[, doseq])
     Convert a mapping object or a sequence of two-element tuples to a
     "percent-encoded" string, suitable to pass to urlopen()
     above as the optional _data_ argument.  This is useful to
     pass a dictionary of form fields to a `POST' request.  The
     resulting string is a series of `key=value' pairs separated by
     `'&'' characters, where both _key_ and _value_ are quoted using
     quote_plus() above.  When a sequence of two-element
     tuples is used as the _query_ argument, the first element of each
     tuple is a key and the second is a value. The value element in
     itself can be a sequence and in that case, if the optional
     parameter _doseq_ is evaluates to _True_, individual `key=value'
     pairs separated by `'&'' are generated for each element of the
     value sequence for the key.  The order of parameters in the
     encoded string will match the order of parameter tuples in the
     sequence. The urlparse module provides the functions
     `parse_qs()' and `parse_qsl()' which are used to parse query
     strings into Python data structures.

 -- Function: urllib.pathname2url (path)
     Convert the pathname _path_ from the local syntax for a path to
     the form used in the path component of a URL.  This does not
     produce a complete URL.  The return value will already be quoted
     using the quote() function.

 -- Function: urllib.url2pathname (path)
     Convert the path component _path_ from an percent-encoded URL to
     the local syntax for a path.  This does not accept a complete URL.
     This function uses unquote().  to decode _path_.

 -- Function: urllib.getproxies ()
     This helper function returns a dictionary of scheme to proxy
     server URL mappings. It scans the environment for variables named
     `<scheme>_proxy' for all operating systems first, and when it
     cannot find it, looks for proxy information from Mac OSX System
     Configuration for Mac OS X and Windows Systems Registry for
     Windows.

5.20.5.3 URL Opener objects
...........................

 -- Class: urllib.URLopener ([proxies[, **x509]])
     Base class for opening and reading URLs.  Unless you need to
     support opening objects using schemes other than `http:', `ftp:',
     or `file:', you probably want to use FancyURLopener.

     By default, the URLopener class sends a `User-Agent'
     header of `urllib/VVV', where _VVV_ is the urllib.
     version number.  Applications can define their own `User-Agent'
     header by subclassing URLopener or
     FancyURLopener and setting the class attribute
     version to an appropriate string value in the subclass
     definition.

     The optional _proxies_ parameter should be a dictionary mapping
     scheme names to proxy URLs, where an empty dictionary turns
     proxies off completely.  Its default value is `None', in which
     case environmental proxy settings will be used if present, as
     discussed in the definition of urlopen(), above.

     Additional keyword parameters, collected in _x509_, may be used for
     authentication of the client when using the `https:' scheme.  The
     keywords _key_file_ and _cert_file_ are supported to provide an
     SSL key and certificate; both are needed to support client
     authentication.

     URLopener objects will raise an IOError.
     exception if the server returns an error code.

           -- Method: open (fullurl[, data])
               Open _fullurl_ using the appropriate protocol.  This
               method sets up cache and proxy information, then calls
               the appropriate open method with its input arguments.
               If the scheme is not recognized, see open_unknown()
               is called.  The _data_ argument has the same
               meaning as the _data_ argument of urlopen().

           -- Method: open_unknown (fullurl[, data])
               Overridable interface to open unknown URL types.

           -- Method: retrieve (url[, filename[, reporthook[, data]]])
               Retrieves the contents of _url_ and places it in
               _filename_.  The return value is a tuple consisting of a
               local filename and either a mimetools.Message
               object containing the response headers (for remote
               URLs) or `None' (for local URLs).  The caller must then
               open and read the contents of _filename_.  If _filename_
               is not given and the URL refers to a local file, the
               input filename is returned.  If the URL is non-local and
               _filename_ is not given, the filename is the output of
               tempfile.mktemp().  with a suffix that
               matches the suffix of the last path component of the
               input URL.  If _reporthook_ is given, it must be a
               function accepting three numeric parameters.  It will be
               called after each chunk of data is read from the
               network.  _reporthook_ is ignored for local URLs.

               If the _url_ uses the `http:' scheme identifier, the
               optional _data_ argument may be given to specify a
               `POST' request (normally the request type is `GET').
               The _data_ argument must in standard
               `application/x-www-form-urlencoded' format; see the
               urlencode().  function below.

           -- Attribute: version
               Variable that specifies the user agent of the opener
               object.  To get urllib to tell servers that
               it is a particular user agent, set this in a subclass as
               a class variable or in the constructor before calling
               the base constructor.

 -- Class: urllib.FancyURLopener (...)
     FancyURLopener subclasses URLopener.
     providing default handling for the following HTTP response codes:
     301, 302, 303, 307 and 401.  For the 30x response codes listed
     above, the `Location' header is used to fetch the actual URL.  For
     401 response codes (authentication required), basic HTTP
     authentication is performed.  For the 30x response codes,
     recursion is bounded by the value of the _maxtries_ attribute,
     which defaults to 10.

     For all other response codes, the method `http_error_default()' is
     called which you can override in subclasses to handle the error
     appropriately.

          Note: According to the letter of RFC 2616
          (http://tools.ietf.org/html/rfc2616.html), 301 and 302
          responses to POST requests must not be automatically
          redirected without confirmation by the user.  In reality,
          browsers do allow automatic redirection of these responses,
          changing the POST to a GET, and urllib reproduces
          this behaviour.

     The parameters to the constructor are the same as those for
     URLopener.

          Note:

               When performing basic authentication, a
               FancyURLopener instance calls its
               prompt_user_passwd() method.  The default
               implementation asks the users for the required
               information on the controlling terminal.  A subclass may
               override this method to support more appropriate
               behavior if needed.

          The FancyURLopener class offers one additional
          method that should be overloaded to provide the appropriate
          behavior:

           -- Method: prompt_user_passwd (host, realm)
               Return information needed to authenticate the user at
               the given host in the specified security realm.  The
               return value should be a tuple, `(user, password)',
               which can be used for basic authentication.

               The implementation prompts for this information on the
               terminal; an application should override this method to
               use an appropriate interaction model in the local
               environment.

 -- Exception: urllib.ContentTooShortError (msg[, content])
     This exception is raised when the urlretrieve().
     function detects that the amount of the downloaded data is less
     than the  expected amount (given by the _Content-Length_ header).
     The `content' attribute stores the downloaded (and supposedly
     truncated) data.

     New in version 2.5.

5.20.5.4 `urllib' Restrictions
..............................

   * Currently, only the following protocols are supported: HTTP,
     (versions 0.9 and 1.0),  FTP, and local files.

   * The caching feature of urlretrieve() has been
     disabled until I find the time to hack proper processing of
     Expiration time headers.

   * There should be a function to query whether a particular URL is in
     the cache.

   * For backward compatibility, if a URL appears to point to a local
     file but the file can't be opened, the URL is re-interpreted using
     the FTP protocol.  This can sometimes cause confusing error
     messages.

   * The urlopen() and urlretrieve() functions
     can cause arbitrarily long delays while waiting for a network
     connection to be set up.  This means that it is difficult to build
     an interactive Web client using these functions without using
     threads.

   * The data returned by urlopen() or urlretrieve()
     is the raw data returned by the server.  This may be binary
     data (such as an image), plain text or (for example) HTML.  The
     HTTP protocol provides type information in the reply header, which
     can be inspected by looking at the `Content-Type' header.  If the
     returned data is HTML, you can use the module htmllib to
     parse it.

   * The code handling the FTP protocol cannot differentiate between a
     file and a directory.  This can lead to unexpected behavior when
     attempting to read a URL that points to a file that is not
     accessible.  If the URL ends in a `/', it is assumed to refer to a
     directory and will be handled accordingly.  But if an attempt to
     read a file leads to a 550 error (meaning the URL cannot be found
     or is not accessible, often for permission reasons), then the path
     is treated as a directory in order to handle the case when a
     directory is specified by a URL but the trailing `/' has been left
     off.  This can cause misleading results when you try to fetch a
     file whose read permissions make it inaccessible; the FTP code
     will try to read it, fail with a 550 error, and then perform a
     directory listing for the unreadable file. If fine-grained control
     is needed, consider using the ftplib module,
     subclassing `FancyURLOpener', or changing __urlopener_ to meet
     your needs.

   * This module does not support the use of proxies which require
     authentication.  This may be implemented in the future.

   * Although the urllib module contains (undocumented)
     routines to parse and unparse URL strings, the recommended
     interface for URL manipulation is in module urlparse.

5.20.5.5 Examples
.................

Here is an example session that uses the `GET' method to retrieve a URL
containing parameters:

    >>> import urllib
    >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
    >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
    >>> print f.read()

The following example uses the `POST' method instead:

    >>> import urllib
    >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
    >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
    >>> print f.read()

The following example uses an explicitly specified HTTP proxy,
overriding environment settings:

    >>> import urllib
    >>> proxies = {'http': 'http://proxy.example.com:8080/'}
    >>> opener = urllib.FancyURLopener(proxies)
    >>> f = opener.open("http://www.python.org")
    >>> f.read()

The following example uses no proxies at all, overriding environment
settings:

    >>> import urllib
    >>> opener = urllib.FancyURLopener({})
    >>> f = opener.open("http://www.python.org/")
    >>> f.read()

5.20.6 `urllib2' -- extensible library for opening URLs
-------------------------------------------------------

     Note: The urllib2 module has been split across several
     modules in Python 3.0 named `urllib.request' and `urllib.error'.
     The 2to3 tool will automatically adapt imports when
     converting your sources to 3.0.

The urllib2 module defines functions and classes which help
in opening URLs (mostly HTTP) in a complex world -- basic and digest
authentication, redirections, cookies and more.

  The urllib2 module defines the following functions:

 -- Function: urllib2.urlopen (url[, data][, timeout])
     Open the URL _url_, which can be either a string or a
     Request object.

          Warning: HTTPS requests do not do any verification of the
          server's certificate.

     _data_ may be a string specifying additional data to send to the
     server, or `None' if no such data is needed.  Currently HTTP
     requests are the only ones that use _data_; the HTTP request will
     be a POST instead of a GET when the _data_ parameter is provided.
     _data_ should be a buffer in the standard
     `application/x-www-form-urlencoded' format.  The
     urllib.urlencode() function takes a mapping or sequence of
     2-tuples and returns a string in this format. urllib2 module sends
     HTTP/1.1 requests with `Connection:close' header included.

     The optional _timeout_ parameter specifies a timeout in seconds
     for blocking operations like the connection attempt (if not
     specified, the global default timeout setting will be used).  This
     actually only works for HTTP, HTTPS and FTP connections.

     This function returns a file-like object with two additional
     methods:

        * `geturl()' -- return the URL of the resource retrieved,
          commonly used to determine if a redirect was followed

        * `info()' -- return the meta-information of the page, such as
          headers, in the form of an mimetools.Message.
          instance (see Quick Reference to HTTP Headers
          (http://www.cs.tut.fi/~jkorpela/http.html))

     Raises URLError on errors.

     Note that `None' may be returned if no handler handles the request
     (though the default installed global OpenerDirector.
     uses UnknownHandler to ensure this never happens).

     In addition, default installed ProxyHandler makes
     sure the requests are handled through the proxy when they are set.

     Changed in version 2.6: _timeout_ was added.

 -- Function: urllib2.install_opener (opener)
     Install an OpenerDirector instance as the default
     global opener.  Installing an opener is only necessary if you want
     urlopen to use that opener; otherwise, simply call
     OpenerDirector.open() instead of urlopen().  The
     code does not check for a real OpenerDirector, and any
     class with the appropriate interface will work.

 -- Function: urllib2.build_opener ([handler, ...])
     Return an OpenerDirector instance, which chains the
     handlers in the order given. _handler_s can be either instances of
     BaseHandler, or subclasses of BaseHandler.
     (in which case it must be possible to call the constructor without
     any parameters).  Instances of the following classes will be in
     front of the _handler_s, unless the _handler_s contain them,
     instances of them or subclasses of them: ProxyHandler,
     UnknownHandler, HTTPHandler,
     HTTPDefaultErrorHandler, HTTPRedirectHandler,
     FTPHandler, FileHandler,
     `HTTPErrorProcessor'.

     If the Python installation has SSL support (i.e., if the
     ssl module can be imported), see HTTPSHandler will
     also be added.

     Beginning in Python 2.3, a BaseHandler subclass may
     also change its `handler_order' member variable to modify its
     position in the handlers list.

  The following exceptions are raised as appropriate:

 -- Exception: urllib2.URLError
     The handlers raise this exception (or derived exceptions) when
     they run into a problem.  It is a subclass of IOError.

      -- Attribute: reason
          The reason for this error.  It can be a message string or
          another exception instance (see socket.error for
          remote URLs, see OSError for local URLs).

 -- Exception: urllib2.HTTPError
     Though being an exception (a subclass of URLError),
     an HTTPError.  can also function as a non-exceptional
     file-like return value (the same thing that urlopen().
     returns).  This is useful when handling exotic HTTP errors, such
     as requests for authentication.

      -- Attribute: code
          An HTTP status code as defined in RFC 2616
          (http://www.faqs.org/rfcs/rfc2616.html).  This numeric value
          corresponds to a value found in the dictionary of codes as
          found in
          BaseHTTPServer.BaseHTTPRequestHandler.responses.

  The following classes are provided:

 -- Class: urllib2.Request (url[, data][, headers][, origin_req_host][,
          unverifiable])
     This class is an abstraction of a URL request.

     _url_ should be a string containing a valid URL.

     _data_ may be a string specifying additional data to send to the
     server, or `None' if no such data is needed.  Currently HTTP
     requests are the only ones that use _data_; the HTTP request will
     be a POST instead of a GET when the _data_ parameter is provided.
     _data_ should be a buffer in the standard
     `application/x-www-form-urlencoded' format.  The
     urllib.urlencode() function takes a mapping or sequence of
     2-tuples and returns a string in this format.

     _headers_ should be a dictionary, and will be treated as if
     add_header().  was called with each key and value as
     arguments.  This is often used to "spoof" the `User-Agent' header,
     which is used by a browser to identify itself - some HTTP servers
     only allow requests coming from common browsers as opposed to
     scripts.  For example, Mozilla Firefox may identify itself as
     `"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127
     Firefox/2.0.0.11"', while urllib2.'s default user agent
     string is `"Python-urllib/2.6"' (on Python 2.6).

     The final two arguments are only of interest for correct handling
     of third-party HTTP cookies:

     _origin_req_host_ should be the request-host of the origin
     transaction, as defined by RFC 2965
     (http://tools.ietf.org/html/rfc2965.html).  It defaults to
     `cookielib.request_host(self)'.  This is the host name or IP
     address of the original request that was initiated by the user.
     For example, if the request is for an image in an HTML document,
     this should be the request-host of the request for the page
     containing the image.

     _unverifiable_ should indicate whether the request is
     unverifiable, as defined by RFC 2965.  It defaults to False.  An
     unverifiable request is one whose URL the user did not have the
     option to approve.  For example, if the request is for an image in
     an HTML document, and the user had no option to approve the
     automatic fetching of the image, this should be true.

 -- Class: urllib2.OpenerDirector
     The OpenerDirector class opens URLs via
     BaseHandler.s chained together. It manages the chaining of
     handlers, and recovery from errors.

 -- Class: urllib2.BaseHandler
     This is the base class for all registered handlers -- and handles
     only the simple mechanics of registration.

 -- Class: urllib2.HTTPDefaultErrorHandler
     A class which defines a default handler for HTTP error responses;
     all responses are turned into HTTPError exceptions.

 -- Class: urllib2.HTTPRedirectHandler
     A class to handle redirections.

 -- Class: urllib2.HTTPCookieProcessor ([cookiejar])
     A class to handle HTTP Cookies.

 -- Class: urllib2.ProxyHandler ([proxies])
     Cause requests to go through a proxy. If _proxies_ is given, it
     must be a dictionary mapping protocol names to URLs of proxies.
     The default is to read the list of proxies from the environment
     variables `<protocol>_proxy'.  If no proxy environment variables
     are set, in a Windows environment, proxy settings are obtained
     from the registry's Internet Settings section and in a Mac OS X
     environment, proxy information is retrieved from the OS X System
     Configuration Framework.

     To disable autodetected proxy pass an empty dictionary.

 -- Class: urllib2.HTTPPasswordMgr
     Keep a database of  `(realm, uri) -> (user, password)' mappings.

 -- Class: urllib2.HTTPPasswordMgrWithDefaultRealm
     Keep a database of  `(realm, uri) -> (user, password)' mappings. A
     realm of `None' is considered a catch-all realm, which is searched
     if no other realm fits.

 -- Class: urllib2.AbstractBasicAuthHandler ([password_mgr])
     This is a mixin class that helps with HTTP authentication, both to
     the remote host and to a proxy. _password_mgr_, if given, should
     be something that is compatible with HTTPPasswordMgr.;
     refer to section HTTPPasswordMgr Objects for
     information on the interface that must be supported.

 -- Class: urllib2.HTTPBasicAuthHandler ([password_mgr])
     Handle authentication with the remote host. _password_mgr_, if
     given, should be something that is compatible with
     HTTPPasswordMgr.; refer to section HTTPPasswordMgr Objects
     for information on the interface that must be
     supported.

 -- Class: urllib2.ProxyBasicAuthHandler ([password_mgr])
     Handle authentication with the proxy. _password_mgr_, if given,
     should be something that is compatible with HTTPPasswordMgr.;
     refer to section HTTPPasswordMgr Objects for
     information on the interface that must be supported.

 -- Class: urllib2.AbstractDigestAuthHandler ([password_mgr])
     This is a mixin class that helps with HTTP authentication, both to
     the remote host and to a proxy. _password_mgr_, if given, should
     be something that is compatible with HTTPPasswordMgr.;
     refer to section HTTPPasswordMgr Objects for
     information on the interface that must be supported.

 -- Class: urllib2.HTTPDigestAuthHandler ([password_mgr])
     Handle authentication with the remote host. _password_mgr_, if
     given, should be something that is compatible with
     HTTPPasswordMgr.; refer to section HTTPPasswordMgr Objects
     for information on the interface that must be
     supported.

 -- Class: urllib2.ProxyDigestAuthHandler ([password_mgr])
     Handle authentication with the proxy. _password_mgr_, if given,
     should be something that is compatible with HTTPPasswordMgr.;
     refer to section HTTPPasswordMgr Objects for
     information on the interface that must be supported.

 -- Class: urllib2.HTTPHandler
     A class to handle opening of HTTP URLs.

 -- Class: urllib2.HTTPSHandler
     A class to handle opening of HTTPS URLs.

 -- Class: urllib2.FileHandler
     Open local files.

 -- Class: urllib2.FTPHandler
     Open FTP URLs.

 -- Class: urllib2.CacheFTPHandler
     Open FTP URLs, keeping a cache of open FTP connections to minimize
     delays.

 -- Class: urllib2.UnknownHandler
     A catch-all class to handle unknown URLs.

5.20.6.1 Request Objects
........................

The following methods describe all of Request.'s public
interface, and so all must be overridden in subclasses.

 -- Method: Request.add_data (data)
     Set the Request data to _data_.  This is ignored by
     all handlers except HTTP handlers -- and there it should be a byte
     string, and will change the request to be `POST' rather than `GET'.

 -- Method: Request.get_method ()
     Return a string indicating the HTTP request method.  This is only
     meaningful for HTTP requests, and currently always returns `'GET''
     or `'POST''.

 -- Method: Request.has_data ()
     Return whether the instance has a non-`None' data.

 -- Method: Request.get_data ()
     Return the instance's data.

 -- Method: Request.add_header (key, val)
     Add another header to the request.  Headers are currently ignored
     by all handlers except HTTP handlers, where they are added to the
     list of headers sent to the server.  Note that there cannot be
     more than one header with the same name, and later calls will
     overwrite previous calls in case the _key_ collides.  Currently,
     this is no loss of HTTP functionality, since all headers which have
     meaning when used more than once have a (header-specific) way of
     gaining the same functionality using only one header.

 -- Method: Request.add_unredirected_header (key, header)
     Add a header that will not be added to a redirected request.

     New in version 2.4.

 -- Method: Request.has_header (header)
     Return whether the instance has the named header (checks both
     regular and unredirected).

     New in version 2.4.

 -- Method: Request.get_full_url ()
     Return the URL given in the constructor.

 -- Method: Request.get_type ()
     Return the type of the URL -- also known as the scheme.

 -- Method: Request.get_host ()
     Return the host to which a connection will be made.

 -- Method: Request.get_selector ()
     Return the selector -- the part of the URL that is sent to the
     server.

 -- Method: Request.set_proxy (host, type)
     Prepare the request by connecting to a proxy server. The _host_
     and _type_ will replace those of the instance, and the instance's
     selector will be the original URL given in the constructor.

 -- Method: Request.get_origin_req_host ()
     Return the request-host of the origin transaction, as defined by RFC
     2965 (http://tools.ietf.org/html/rfc2965.html).  See the
     documentation for the Request constructor.

 -- Method: Request.is_unverifiable ()
     Return whether the request is unverifiable, as defined by RFC
     2965. See the documentation for the Request.
     constructor.

5.20.6.2 OpenerDirector Objects
...............................

OpenerDirector instances have the following methods:

 -- Method: OpenerDirector.add_handler (handler)
     _handler_ should be an instance of BaseHandler.  The
     following methods are searched, and added to the possible chains
     (note that HTTP errors are a special case).

        * `_protocol__open' -- signal that the handler knows how to open
          _protocol_ URLs.

        * `http_error__type_' -- signal that the handler knows how to
          handle HTTP errors with HTTP error code _type_.

        * `_protocol__error' -- signal that the handler knows how to
          handle errors from (non-`http') _protocol_.

        * `_protocol__request' -- signal that the handler knows how to
          pre-process _protocol_ requests.

        * `_protocol__response' -- signal that the handler knows how to
          post-process _protocol_ responses.

 -- Method: OpenerDirector.open (url[, data][, timeout])
     Open the given _url_ (which can be a request object or a string),
     optionally passing the given _data_. Arguments, return values and
     exceptions raised are the same as those of urlopen().
     (which simply calls the open().  method on the
     currently installed global OpenerDirector).  The
     optional _timeout_ parameter specifies a timeout in seconds for
     blocking operations like the connection attempt (if not specified,
     the global default timeout setting will be used). The timeout
     feature actually works only for HTTP, HTTPS and FTP connections).

     Changed in version 2.6: _timeout_ was added.

 -- Method: OpenerDirector.error (proto[, arg[, ...]])
     Handle an error of the given protocol.  This will call the
     registered error handlers for the given protocol with the given
     arguments (which are protocol specific).  The HTTP protocol is a
     special case which uses the HTTP response code to determine the
     specific error handler; refer to the `http_error_*()' methods of
     the handler classes.

     Return values and exceptions raised are the same as those of
     urlopen().

  OpenerDirector objects open URLs in three stages:

  The order in which these methods are called within each stage is
determined by sorting the handler instances.

  1. Every handler with a method named like `_protocol__request' has
     that method called to pre-process the request.

  2. Handlers with a method named like `_protocol__open' are called to
     handle the request. This stage ends when a handler either returns
     a non-see None.  value (ie. a response), or raises an
     exception (usually URLError).  Exceptions are allowed
     to propagate.

     In fact, the above algorithm is first tried for methods named
     `default_open()'.  If all such methods return None, the
     algorithm is repeated for methods named like `_protocol__open'.
     If all such methods return None, the algorithm is
     repeated for methods named `unknown_open()'.

     Note that the implementation of these methods may involve calls of
     the parent OpenerDirector instance's open()
     and error() methods.

  3. Every handler with a method named like `_protocol__response' has
     that method called to post-process the response.

5.20.6.3 BaseHandler Objects
............................

BaseHandler objects provide a couple of methods that are
directly useful, and others that are meant to be used by derived
classes.  These are intended for direct use:

 -- Method: BaseHandler.add_parent (director)
     Add a director as parent.

 -- Method: BaseHandler.close ()
     Remove any parents.

  The following members and methods should only be used by classes
derived from BaseHandler.

     Note: The convention has been adopted that subclasses defining
     `protocol_request()' or `protocol_response()' methods are named
     `*Processor'; all others are named `*Handler'.

 -- Attribute: BaseHandler.parent
     A valid OpenerDirector, which can be used to open
     using a different protocol, or handle errors.

 -- Method: BaseHandler.default_open (req)
     This method is _not_ defined in BaseHandler, but
     subclasses should define it if they want to catch all URLs.

     This method, if implemented, will be called by the parent
     OpenerDirector.  It should return a file-like object as
     described in the return value of the open() of
     OpenerDirector, or `None'.  It should raise URLError,
     unless a truly exceptional thing happens (for example, see
     MemoryError should not be mapped to URLError).

     This method will be called before any protocol-specific open
     method.

 -- Method: BaseHandler.protocol_open (req)
     ("protocol" is to be replaced by the protocol name.)

     This method is _not_ defined in BaseHandler, but
     subclasses should define it if they want to handle URLs with the
     given _protocol_.

     This method, if defined, will be called by the parent
     OpenerDirector.  Return values should be the same as for
     `default_open()'.

 -- Method: BaseHandler.unknown_open (req)
     This method is _not_ defined in BaseHandler, but
     subclasses should define it if they want to catch all URLs with no
     specific registered handler to open it.

     This method, if implemented, will be called by the parent.
     See OpenerDirector.  Return values should be the
     same as for default_open().

 -- Method: BaseHandler.http_error_default (req, fp, code, msg, hdrs)
     This method is _not_ defined in BaseHandler, but
     subclasses should override it if they intend to provide a
     catch-all for otherwise unhandled HTTP errors.  It will be called
     automatically by the  OpenerDirector getting the
     error, and should not normally be called in other circumstances.

     _req_ will be a Request object, _fp_ will be a
     file-like object with the HTTP error body, _code_ will be the
     three-digit code of the error, _msg_ will be the user-visible
     explanation of the code and _hdrs_ will be a mapping object with
     the headers of the error.

     Return values and exceptions raised should be the same as those of
     urlopen().

 -- Method: BaseHandler.http_error_nnn (req, fp, code, msg, hdrs)
     _nnn_ should be a three-digit HTTP error code.  This method is
     also not defined in BaseHandler, but will be called,
     if it exists, on an instance of a subclass, when an HTTP error
     with code _nnn_ occurs.

     Subclasses should override this method to handle specific HTTP
     errors.

     Arguments, return values and exceptions raised should be the same
     as for http_error_default().

 -- Method: BaseHandler.protocol_request (req)
     ("protocol" is to be replaced by the protocol name.)

     This method is _not_ defined in BaseHandler, but
     subclasses should define it if they want to pre-process requests
     of the given _protocol_.

     This method, if defined, will be called by the parent
     OpenerDirector.  _req_ will be a Request.
     object. The return value should be a Request object.

 -- Method: BaseHandler.protocol_response (req, response)
     ("protocol" is to be replaced by the protocol name.)

     This method is _not_ defined in BaseHandler, but
     subclasses should define it if they want to post-process responses
     of the given _protocol_.

     This method, if defined, will be called by the parent
     OpenerDirector.  _req_ will be a Request.
     object. _response_ will be an object implementing the same
     interface as the return value of urlopen().  The return
     value should implement the same interface as the return value of
     urlopen().

5.20.6.4 HTTPRedirectHandler Objects
....................................

     Note: Some HTTP redirections require action from this module's
     client code.  If this is the case, see HTTPError is
     raised.  See RFC 2616 (http://tools.ietf.org/html/rfc2616.html)
     for details of the precise meanings of the various redirection
     codes.

 -- Method: HTTPRedirectHandler.redirect_request (req, fp, code, msg,
          hdrs, newurl)
     Return a Request or `None' in response to a redirect.
     This is called by the default implementations of the
     `http_error_30*()' methods when a redirection is received from the
     server.  If a redirection should take place, return a new
     Request to allow `http_error_30*()' to perform the redirect
     to _newurl_.  Otherwise, raise HTTPError if no other
     handler should try to handle this URL, or return `None' if you
     can't but another handler might.

          Note: The default implementation of this method does not
          strictly follow RFC 2616
          (http://tools.ietf.org/html/rfc2616.html), which says that
          301 and 302 responses to `POST' requests must not be
          automatically redirected without confirmation by the user.
          In reality, browsers do allow automatic redirection of these
          responses, changing the POST to a `GET', and the default
          implementation reproduces this behavior.

 -- Method: HTTPRedirectHandler.http_error_301 (req, fp, code, msg,
          hdrs)
     Redirect to the `Location:' or `URI:' URL.  This method is called
     by the parent OpenerDirector when getting an HTTP
     'moved permanently' response.

 -- Method: HTTPRedirectHandler.http_error_302 (req, fp, code, msg,
          hdrs)
     The same as http_error_301(), but called for the
     'found' response.

 -- Method: HTTPRedirectHandler.http_error_303 (req, fp, code, msg,
          hdrs)
     The same as http_error_301(), but called for the 'see
     other' response.

 -- Method: HTTPRedirectHandler.http_error_307 (req, fp, code, msg,
          hdrs)
     The same as http_error_301(), but called for the
     'temporary redirect' response.

5.20.6.5 HTTPCookieProcessor Objects
....................................

New in version 2.4.

  HTTPCookieProcessor instances have one attribute:

 -- Attribute: HTTPCookieProcessor.cookiejar
     The cookielib.CookieJar in which cookies are stored.

5.20.6.6 ProxyHandler Objects
.............................

 -- Method: ProxyHandler.protocol_open (request)
     ("protocol" is to be replaced by the protocol name.)

     The ProxyHandler will have a method `_protocol__open'
     for every _protocol_ which has a proxy in the _proxies_ dictionary
     given in the constructor.  The method will modify requests to go
     through the proxy, by calling `request.set_proxy()', and call the
     next handler in the chain to actually execute the protocol.

5.20.6.7 HTTPPasswordMgr Objects
................................

These methods are available on HTTPPasswordMgr and
HTTPPasswordMgrWithDefaultRealm objects.

 -- Method: HTTPPasswordMgr.add_password (realm, uri, user, passwd)
     _uri_ can be either a single URI, or a sequence of URIs. _realm_,
     _user_ and _passwd_ must be strings. This causes `(user, passwd)'
     to be used as authentication tokens when authentication for
     _realm_ and a super-URI of any of the given URIs is given.

 -- Method: HTTPPasswordMgr.find_user_password (realm, authuri)
     Get user/password for given realm and URI, if any.  This method
     will return `(None, None)' if there is no matching user/password.

     For HTTPPasswordMgrWithDefaultRealm objects, the
     realm `None' will be searched if the given _realm_ has no matching
     user/password.

5.20.6.8 AbstractBasicAuthHandler Objects
.........................................

 -- Method: AbstractBasicAuthHandler.http_error_auth_reqed (authreq,
          host, req, headers)
     Handle an authentication request by getting a user/password pair,
     and re-trying the request.  _authreq_ should be the name of the
     header where the information about the realm is included in the
     request, _host_ specifies the URL and path to authenticate for,
     _req_ should be the (failed) Request object, and
     _headers_ should be the error headers.

     _host_ is either an authority (e.g. `"python.org"') or a URL
     containing an authority component (e.g. `"http://python.org/"').
     In either case, the authority must not contain a userinfo
     component (so, `"python.org"' and `"python.org:80"' are fine,
     `"joe:password@python.org"' is not).

5.20.6.9 HTTPBasicAuthHandler Objects
.....................................

 -- Method: HTTPBasicAuthHandler.http_error_401 (req, fp, code, msg,
          hdrs)
     Retry the request with authentication information, if available.

5.20.6.10 ProxyBasicAuthHandler Objects
.......................................

 -- Method: ProxyBasicAuthHandler.http_error_407 (req, fp, code, msg,
          hdrs)
     Retry the request with authentication information, if available.

5.20.6.11 AbstractDigestAuthHandler Objects
...........................................

 -- Method: AbstractDigestAuthHandler.http_error_auth_reqed (authreq,
          host, req, headers)
     _authreq_ should be the name of the header where the information
     about the realm is included in the request, _host_ should be the
     host to authenticate to, _req_ should be the (failed)
     Request object, and _headers_ should be the error headers.

5.20.6.12 HTTPDigestAuthHandler Objects
.......................................

 -- Method: HTTPDigestAuthHandler.http_error_401 (req, fp, code, msg,
          hdrs)
     Retry the request with authentication information, if available.

5.20.6.13 ProxyDigestAuthHandler Objects
........................................

 -- Method: ProxyDigestAuthHandler.http_error_407 (req, fp, code, msg,
          hdrs)
     Retry the request with authentication information, if available.

5.20.6.14 HTTPHandler Objects
.............................

 -- Method: HTTPHandler.http_open (req)
     Send an HTTP request, which can be either GET or POST, depending on
     `req.has_data()'.

5.20.6.15 HTTPSHandler Objects
..............................

 -- Method: HTTPSHandler.https_open (req)
     Send an HTTPS request, which can be either GET or POST, depending
     on `req.has_data()'.

5.20.6.16 FileHandler Objects
.............................

 -- Method: FileHandler.file_open (req)
     Open the file locally, if there is no host name, or the host name
     is `'localhost''. Change the protocol to `ftp' otherwise, and
     retry opening it using `parent'.

5.20.6.17 FTPHandler Objects
............................

 -- Method: FTPHandler.ftp_open (req)
     Open the FTP file indicated by _req_. The login is always done
     with empty username and password.

5.20.6.18 CacheFTPHandler Objects
.................................

CacheFTPHandler objects are FTPHandler.
objects with the following additional methods:

 -- Method: CacheFTPHandler.setTimeout (t)
     Set timeout of connections to _t_ seconds.

 -- Method: CacheFTPHandler.setMaxConns (m)
     Set maximum number of cached connections to _m_.

5.20.6.19 UnknownHandler Objects
................................

 -- Method: UnknownHandler.unknown_open ()
     Raise a URLError exception.

5.20.6.20 HTTPErrorProcessor Objects
....................................

New in version 2.4.

 -- Method: HTTPErrorProcessor.unknown_open ()
     Process HTTP error responses.

     For 200 error codes, the response object is returned immediately.

     For non-200 error codes, this simply passes the job on to the
     `_protocol__error_code' handler methods, via
     OpenerDirector.error().  Eventually, see
     urllib2.HTTPDefaultErrorHandler will raise an
     HTTPError if no other handler handles the error.

5.20.6.21 Examples
..................

This example gets the python.org main page and displays the first 100
bytes of it:

    >>> import urllib2
    >>> f = urllib2.urlopen('http://www.python.org/')
    >>> print f.read(100)
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
    <?xml-stylesheet href="./css/ht2html

Here we are sending a data-stream to the stdin of a CGI and reading the
data it returns to us. Note that this example will only work when the
Python installation supports SSL.

    >>> import urllib2
    >>> req = urllib2.Request(url='https://localhost/cgi-bin/test.cgi',
    ...                       data='This data is passed to stdin of the CGI')
    >>> f = urllib2.urlopen(req)
    >>> print f.read()
    Got Data: "This data is passed to stdin of the CGI"

The code for the sample CGI used in the above example is:

    #!/usr/bin/env python
    import sys
    data = sys.stdin.read()
    print 'Content-type: text-plain\n\nGot Data: "%s"' % data

Use of Basic HTTP Authentication:

    import urllib2
    # Create an OpenerDirector with support for Basic HTTP Authentication...
    auth_handler = urllib2.HTTPBasicAuthHandler()
    auth_handler.add_password(realm='PDQ Application',
                              uri='https://mahler:8092/site-updates.py',
                              user='klem',
                              passwd='kadidd!ehopper')
    opener = urllib2.build_opener(auth_handler)
    # ...and install it globally so it can be used with urlopen.
    urllib2.install_opener(opener)
    urllib2.urlopen('http://www.example.com/login.html')

build_opener() provides many handlers by default,
including a ProxyHandler.  By default, see ProxyHandler
uses the environment variables named `<scheme>_proxy', where
`<scheme>' is the URL scheme involved.  For example, the `http_proxy'
environment variable is read to obtain the HTTP proxy's URL.

  This example replaces the default ProxyHandler with one
that uses programmatically-supplied proxy URLs, and adds proxy
authorization support with ProxyBasicAuthHandler.

    proxy_handler = urllib2.ProxyHandler({'http': 'http://www.example.com:3128/'})
    proxy_auth_handler = urllib2.ProxyBasicAuthHandler()
    proxy_auth_handler.add_password('realm', 'host', 'username', 'password')

    opener = urllib2.build_opener(proxy_handler, proxy_auth_handler)
    # This time, rather than install the OpenerDirector, we use it directly:
    opener.open('http://www.example.com/login.html')

Adding HTTP headers:

  Use the _headers_ argument to the Request constructor,
or:

    import urllib2
    req = urllib2.Request('http://www.example.com/')
    req.add_header('Referer', 'http://www.python.org/')
    r = urllib2.urlopen(req)

OpenerDirector automatically adds a `User-Agent' header to
every Request.  To change this:

    import urllib2
    opener = urllib2.build_opener()
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    opener.open('http://www.example.com/')

Also, remember that a few standard headers (`Content-Length',
`Content-Type' and `Host') are added when the Request is
passed to urlopen(). (or OpenerDirector.open()).

5.20.7 `httplib' -- HTTP protocol client
----------------------------------------

     Note: The httplib module has been renamed to
     `http.client' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

This module defines classes which implement the client side of the HTTP
and HTTPS protocols.  It is normally not used directly -- the module
urllib.  uses it to handle URLs that use HTTP and HTTPS.

     Note: HTTPS support is only available if the socket.
     module was compiled with SSL support.

     Note: The public interface for this module changed substantially
     in Python 2.0.  The `HTTP' class is retained only for backward
     compatibility with 1.5.2.  It should not be used in new code.
     Refer to the online docstrings for usage.

  The module provides the following classes:

 -- Class: httplib.HTTPConnection (host[, port[, strict[, timeout[,
          source_address]]]])
     An HTTPConnection instance represents one transaction
     with an HTTP server.  It should be instantiated passing it a host
     and optional port number.  If no port number is passed, the port
     is extracted from the host string if it has the form `host:port',
     else the default HTTP port (80) is used.  When True, the optional
     parameter _strict_ (which defaults to a false value) causes
     `BadStatusLine' to be raised if the status line can't be parsed as
     a valid HTTP/1.0 or 1.1 status line.  If the optional _timeout_
     parameter is given, blocking operations (like connection attempts)
     will timeout after that many seconds (if it is not given, the
     global default timeout setting is used).  The optional
     _source_address_ parameter may be a tuple of a (host, port) to use
     as the source address the HTTP connection is made from.

     For example, the following calls all create instances that connect
     to the server at the same host and port:

         >>> h1 = httplib.HTTPConnection('www.cwi.nl')
         >>> h2 = httplib.HTTPConnection('www.cwi.nl:80')
         >>> h3 = httplib.HTTPConnection('www.cwi.nl', 80)
         >>> h3 = httplib.HTTPConnection('www.cwi.nl', 80, timeout=10)

     New in version 2.0.

     Changed in version 2.6: _timeout_ was added.

     Changed in version 2.7: _source_address_ was added.

 -- Class: httplib.HTTPSConnection (host[, port[, key_file[,
          cert_file[, strict[, timeout[, source_address]]]]]])
     A subclass of HTTPConnection that uses SSL for
     communication with secure servers.  Default port is `443'.
     _key_file_ is the name of a PEM formatted file that contains your
     private key. _cert_file_ is a PEM formatted certificate chain file.

          Warning: This does not do any verification of the server's
          certificate.

     New in version 2.0.

     Changed in version 2.6: _timeout_ was added.

     Changed in version 2.7: _source_address_ was added.

 -- Class: httplib.HTTPResponse (sock[, debuglevel=0][, strict=0])
     Class whose instances are returned upon successful connection.
     Not instantiated directly by user.

     New in version 2.0.

 -- Class: httplib.HTTPMessage
     An HTTPMessage instance is used to hold the headers
     from an HTTP response. It is implemented using the
     mimetools.Message class and provides utility functions to
     deal with HTTP Headers. It is not directly instantiated by the
     users.

  The following exceptions are raised as appropriate:

 -- Exception: httplib.HTTPException
     The base class of the other exceptions in this module.  It is a
     subclass of Exception.

     New in version 2.0.

 -- Exception: httplib.NotConnected
     A subclass of HTTPException.

     New in version 2.0.

 -- Exception: httplib.InvalidURL
     A subclass of HTTPException, raised if a port is given
     and is either non-numeric or empty.

     New in version 2.3.

 -- Exception: httplib.UnknownProtocol
     A subclass of HTTPException.

     New in version 2.0.

 -- Exception: httplib.UnknownTransferEncoding
     A subclass of HTTPException.

     New in version 2.0.

 -- Exception: httplib.UnimplementedFileMode
     A subclass of HTTPException.

     New in version 2.0.

 -- Exception: httplib.IncompleteRead
     A subclass of HTTPException.

     New in version 2.0.

 -- Exception: httplib.ImproperConnectionState
     A subclass of HTTPException.

     New in version 2.0.

 -- Exception: httplib.CannotSendRequest
     A subclass of ImproperConnectionState.

     New in version 2.0.

 -- Exception: httplib.CannotSendHeader
     A subclass of ImproperConnectionState.

     New in version 2.0.

 -- Exception: httplib.ResponseNotReady
     A subclass of ImproperConnectionState.

     New in version 2.0.

 -- Exception: httplib.BadStatusLine
     A subclass of HTTPException.  Raised if a server
     responds with a HTTP status code that we don't understand.

     New in version 2.0.

  The constants defined in this module are:

 -- Data: httplib.HTTP_PORT
     The default port for the HTTP protocol (always `80').

 -- Data: httplib.HTTPS_PORT
     The default port for the HTTPS protocol (always `443').

  and also the following constants for integer status codes:

Constant                                       Value         Definition
-----------------------------------------------------------------------------------------------------------------------------------------
`CONTINUE'                                     `100'         HTTP/1.1, RFC 2616, Section 10.1.1
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.1)
`SWITCHING_PROTOCOLS'                          `101'         HTTP/1.1, RFC 2616, Section 10.1.2
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.2)
`PROCESSING'                                   `102'         WEBDAV, RFC 2518, Section 10.1
                                                             (http://www.webdav.org/specs/rfc2518.html#STATUS_102)
`OK'                                           `200'         HTTP/1.1, RFC 2616, Section 10.2.1
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1)
`CREATED'                                      `201'         HTTP/1.1, RFC 2616, Section 10.2.2
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2)
`ACCEPTED'                                     `202'         HTTP/1.1, RFC 2616, Section 10.2.3
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3)
`NON_AUTHORITATIVE_INFORMATION'                `203'         HTTP/1.1, RFC 2616, Section 10.2.4
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.4)
`NO_CONTENT'                                   `204'         HTTP/1.1, RFC 2616, Section 10.2.5
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5)
`RESET_CONTENT'                                `205'         HTTP/1.1, RFC 2616, Section 10.2.6
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.6)
`PARTIAL_CONTENT'                              `206'         HTTP/1.1, RFC 2616, Section 10.2.7
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7)
`MULTI_STATUS'                                 `207'         WEBDAV RFC 2518, Section 10.2
                                                             (http://www.webdav.org/specs/rfc2518.html#STATUS_207)
`IM_USED'                                      `226'         Delta encoding in HTTP, RFC 3229
                                                             (http://tools.ietf.org/html/rfc3229.html), Section 10.4.1
`MULTIPLE_CHOICES'                             `300'         HTTP/1.1, RFC 2616, Section 10.3.1
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1)
`MOVED_PERMANENTLY'                            `301'         HTTP/1.1, RFC 2616, Section 10.3.2
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2)
`FOUND'                                        `302'         HTTP/1.1, RFC 2616, Section 10.3.3
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3)
`SEE_OTHER'                                    `303'         HTTP/1.1, RFC 2616, Section 10.3.4
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4)
`NOT_MODIFIED'                                 `304'         HTTP/1.1, RFC 2616, Section 10.3.5
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5)
`USE_PROXY'                                    `305'         HTTP/1.1, RFC 2616, Section 10.3.6
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.6)
`TEMPORARY_REDIRECT'                           `307'         HTTP/1.1, RFC 2616, Section 10.3.8
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.8)
`BAD_REQUEST'                                  `400'         HTTP/1.1, RFC 2616, Section 10.4.1
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1)
`UNAUTHORIZED'                                 `401'         HTTP/1.1, RFC 2616, Section 10.4.2
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2)
`PAYMENT_REQUIRED'                             `402'         HTTP/1.1, RFC 2616, Section 10.4.3
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.3)
`FORBIDDEN'                                    `403'         HTTP/1.1, RFC 2616, Section 10.4.4
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4)
`NOT_FOUND'                                    `404'         HTTP/1.1, RFC 2616, Section 10.4.5
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5)
`METHOD_NOT_ALLOWED'                           `405'         HTTP/1.1, RFC 2616, Section 10.4.6
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6)
`NOT_ACCEPTABLE'                               `406'         HTTP/1.1, RFC 2616, Section 10.4.7
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7)
`PROXY_AUTHENTICATION_REQUIRED'                `407'         HTTP/1.1, RFC 2616, Section 10.4.8
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.8)
`REQUEST_TIMEOUT'                              `408'         HTTP/1.1, RFC 2616, Section 10.4.9
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.9)
`CONFLICT'                                     `409'         HTTP/1.1, RFC 2616, Section 10.4.10
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10)
`GONE'                                         `410'         HTTP/1.1, RFC 2616, Section 10.4.11
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11)
`LENGTH_REQUIRED'                              `411'         HTTP/1.1, RFC 2616, Section 10.4.12
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.12)
`PRECONDITION_FAILED'                          `412'         HTTP/1.1, RFC 2616, Section 10.4.13
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13)
`REQUEST_ENTITY_TOO_LARGE'                     `413'         HTTP/1.1, RFC 2616, Section 10.4.14
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14)
`REQUEST_URI_TOO_LONG'                         `414'         HTTP/1.1, RFC 2616, Section 10.4.15
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.15)
`UNSUPPORTED_MEDIA_TYPE'                       `415'         HTTP/1.1, RFC 2616, Section 10.4.16
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16)
`REQUESTED_RANGE_NOT_SATISFIABLE'              `416'         HTTP/1.1, RFC 2616, Section 10.4.17
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.17)
`EXPECTATION_FAILED'                           `417'         HTTP/1.1, RFC 2616, Section 10.4.18
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.18)
`UNPROCESSABLE_ENTITY'                         `422'         WEBDAV, RFC 2518, Section 10.3
                                                             (http://www.webdav.org/specs/rfc2518.html#STATUS_422)
`LOCKED'                                       `423'         WEBDAV RFC 2518, Section 10.4
                                                             (http://www.webdav.org/specs/rfc2518.html#STATUS_423)
`FAILED_DEPENDENCY'                            `424'         WEBDAV, RFC 2518, Section 10.5
                                                             (http://www.webdav.org/specs/rfc2518.html#STATUS_424)
`UPGRADE_REQUIRED'                             `426'         HTTP Upgrade to TLS, RFC 2817 (http://tools.ietf.org/html/rfc2817.html),
                                                             Section 6
`INTERNAL_SERVER_ERROR'                        `500'         HTTP/1.1, RFC 2616, Section 10.5.1
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1)
`NOT_IMPLEMENTED'                              `501'         HTTP/1.1, RFC 2616, Section 10.5.2
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2)
`BAD_GATEWAY'                                  `502'         HTTP/1.1 RFC 2616, Section 10.5.3
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.3)
`SERVICE_UNAVAILABLE'                          `503'         HTTP/1.1, RFC 2616, Section 10.5.4
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4)
`GATEWAY_TIMEOUT'                              `504'         HTTP/1.1 RFC 2616, Section 10.5.5
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5)
`HTTP_VERSION_NOT_SUPPORTED'                   `505'         HTTP/1.1, RFC 2616, Section 10.5.6
                                                             (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6)
`INSUFFICIENT_STORAGE'                         `507'         WEBDAV, RFC 2518, Section 10.6
                                                             (http://www.webdav.org/specs/rfc2518.html#STATUS_507)
`NOT_EXTENDED'                                 `510'         An HTTP Extension Framework, RFC 2774
                                                             (http://tools.ietf.org/html/rfc2774.html), Section 7

 -- Data: httplib.responses
     This dictionary maps the HTTP 1.1 status codes to the W3C names.

     Example: `httplib.responses[httplib.NOT_FOUND]' is `'Not Found''.

     New in version 2.5.

5.20.7.1 HTTPConnection Objects
...............................

HTTPConnection instances have the following methods:

 -- Method: HTTPConnection.request (method, url[, body[, headers]])
     This will send a request to the server using the HTTP request
     method _method_ and the selector _url_.  If the _body_ argument is
     present, it should be a string of data to send after the headers
     are finished. Alternatively, it may be an open file object, in
     which case the contents of the file is sent; this file object
     should support `fileno()' and `read()' methods. The header
     Content-Length is automatically set to the correct value. The
     _headers_ argument should be a mapping of extra HTTP headers to
     send with the request.

     Changed in version 2.6: _body_ can be a file object.

 -- Method: HTTPConnection.getresponse ()
     Should be called after a request is sent to get the response from
     the server.  Returns an HTTPResponse instance.

          Note: Note that you must have read the whole response before
          you can send a new request to the server.

 -- Method: HTTPConnection.set_debuglevel (level)
     Set the debugging level (the amount of debugging output printed).
     The default debug level is `0', meaning no debugging output is
     printed.

 -- Method: HTTPConnection.set_tunnel (host, port=None, headers=None)
     Set the host and the port for HTTP Connect Tunnelling. Normally
     used when it is required to do HTTPS Conection through a proxy
     server.

     The headers argument should be a mapping of extra HTTP headers to
     to sent with the CONNECT request.

     New in version 2.7.

 -- Method: HTTPConnection.connect ()
     Connect to the server specified when the object was created.

 -- Method: HTTPConnection.close ()
     Close the connection to the server.

  As an alternative to using the `request()' method described above,
you can also send your request step by step, by using the four
functions below.

 -- Method: HTTPConnection.putrequest (request, selector[, skip_host[,
          skip_accept_encoding]])
     This should be the first call after the connection to the server
     has been made.  It sends a line to the server consisting of the
     _request_ string, the _selector_ string, and the HTTP version
     (`HTTP/1.1').  To disable automatic sending of `Host:' or
     `Accept-Encoding:' headers (for example to accept additional
     content encodings), specify _skip_host_ or _skip_accept_encoding_
     with non-False values.

     Changed in version 2.4: _skip_accept_encoding_ argument added.

 -- Method: HTTPConnection.putheader (header, argument[, ...])
     Send an RFC 822 (http://tools.ietf.org/html/rfc822.html)-style
     header to the server.  It sends a line to the server consisting of
     the header, a colon and a space, and the first argument.  If more
     arguments are given, continuation lines are sent, each consisting
     of a tab and an argument.

 -- Method: HTTPConnection.endheaders ()
     Send a blank line to the server, signalling the end of the headers.

 -- Method: HTTPConnection.send (data)
     Send data to the server.  This should be used directly only after
     the endheaders() method has been called and before
     getresponse() is called.

5.20.7.2 HTTPResponse Objects
.............................

HTTPResponse instances have the following methods and
attributes:

 -- Method: HTTPResponse.read ([amt])
     Reads and returns the response body, or up to the next _amt_ bytes.

 -- Method: HTTPResponse.getheader (name[, default])
     Get the contents of the header _name_, or _default_ if there is no
     matching header.

 -- Method: HTTPResponse.getheaders ()
     Return a list of (header, value) tuples.

     New in version 2.4.

 -- Method: HTTPResponse.fileno ()
     Returns the `fileno' of the underlying socket.

 -- Attribute: HTTPResponse.msg
     A mimetools.Message instance containing the response
     headers.

 -- Attribute: HTTPResponse.version
     HTTP protocol version used by server.  10 for HTTP/1.0, 11 for
     HTTP/1.1.

 -- Attribute: HTTPResponse.status
     Status code returned by server.

 -- Attribute: HTTPResponse.reason
     Reason phrase returned by server.

5.20.7.3 Examples
.................

Here is an example session that uses the `GET' method:

    >>> import httplib
    >>> conn = httplib.HTTPConnection("www.python.org")
    >>> conn.request("GET", "/index.html")
    >>> r1 = conn.getresponse()
    >>> print r1.status, r1.reason
    200 OK
    >>> data1 = r1.read()
    >>> conn.request("GET", "/parrot.spam")
    >>> r2 = conn.getresponse()
    >>> print r2.status, r2.reason
    404 Not Found
    >>> data2 = r2.read()
    >>> conn.close()

Here is an example session that uses the `HEAD' method.  Note that the
`HEAD' method never returns any data.

    >>> import httplib
    >>> conn = httplib.HTTPConnection("www.python.org")
    >>> conn.request("HEAD","/index.html")
    >>> res = conn.getresponse()
    >>> print res.status, res.reason
    200 OK
    >>> data = res.read()
    >>> print len(data)
    0
    >>> data == ''
    True

Here is an example session that shows how to `POST' requests:

    >>> import httplib, urllib
    >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
    >>> headers = {"Content-type": "application/x-www-form-urlencoded",
    ...            "Accept": "text/plain"}
    >>> conn = httplib.HTTPConnection("musi-cal.mojam.com:80")
    >>> conn.request("POST", "/cgi-bin/query", params, headers)
    >>> response = conn.getresponse()
    >>> print response.status, response.reason
    200 OK
    >>> data = response.read()
    >>> conn.close()

5.20.8 `ftplib' -- FTP protocol client
--------------------------------------

This module defines the class FTP and a few related items.
The FTP class implements the client side of the FTP
protocol.  You can use this to write Python programs that perform a
variety of automated FTP jobs, such as mirroring other ftp servers.  It
is also used by the module urllib to handle URLs that use
FTP.  For more information on FTP (File Transfer Protocol), see Internet RFC
959 (http://tools.ietf.org/html/rfc959.html).

  Here's a sample session using the ftplib module:

    >>> from ftplib import FTP
    >>> ftp = FTP('ftp.cwi.nl')   # connect to host, default port
    >>> ftp.login()               # user anonymous, passwd anonymous@
    >>> ftp.retrlines('LIST')     # list directory contents
    total 24418
    drwxrwsr-x   5 ftp-usr  pdmaint     1536 Mar 20 09:48 .
    dr-xr-srwt 105 ftp-usr  pdmaint     1536 Mar 21 14:32 ..
    -rw-r--r--   1 ftp-usr  pdmaint     5305 Mar 20 09:48 INDEX
     .
     .
     .
    >>> ftp.retrbinary('RETR README', open('README', 'wb').write)
    '226 Transfer complete.'
    >>> ftp.quit()

The module defines the following items:

 -- Class: ftplib.FTP ([host[, user[, passwd[, acct[, timeout]]]]])
     Return a new instance of the FTP class.  When _host_
     is given, the method call `connect(host)' is made.  When _user_ is
     given, additionally the method call `login(user, passwd, acct)' is
     made (where _passwd_ and _acct_ default to the empty string when
     not given).  The optional _timeout_ parameter specifies a timeout
     in seconds for blocking operations like the connection attempt (if
     is not specified, the global default timeout setting will be used).

     Changed in version 2.6: _timeout_ was added.

 -- Class: ftplib.FTP_TLS ([host[, user[, passwd[, acct[, keyfile[,
          certfile[, timeout]]]]]]])
     A FTP subclass which adds TLS support to FTP as
     described in RFC 4217 (http://tools.ietf.org/html/rfc4217.html).
     Connect as usual to port 21 implicitly securing the FTP control
     connection before authenticating. Securing the data connection
     requires the user to explicitly ask for it by calling the
     prot_p() method.  _keyfile_ and _certfile_ are optional -
     they can contain a PEM formatted private key and certificate chain
     file name for the SSL connection.

     New in version 2.7.

     Here's a sample session using the FTP_TLS class:

         >>> from ftplib import FTP_TLS
         >>> ftps = FTP_TLS('ftp.python.org')
         >>> ftps.login()           # login anonymously before securing control channel
         >>> ftps.prot_p()          # switch to secure data connection
         >>> ftps.retrlines('LIST') # list directory content securely
         total 9
         drwxr-xr-x   8 root     wheel        1024 Jan  3  1994 .
         drwxr-xr-x   8 root     wheel        1024 Jan  3  1994 ..
         drwxr-xr-x   2 root     wheel        1024 Jan  3  1994 bin
         drwxr-xr-x   2 root     wheel        1024 Jan  3  1994 etc
         d-wxrwxr-x   2 ftp      wheel        1024 Sep  5 13:43 incoming
         drwxr-xr-x   2 root     wheel        1024 Nov 17  1993 lib
         drwxr-xr-x   6 1094     wheel        1024 Sep 13 19:07 pub
         drwxr-xr-x   3 root     wheel        1024 Jan  3  1994 usr
         -rw-r--r--   1 root     root          312 Aug  1  1994 welcome.msg
         '226 Transfer complete.'
         >>> ftps.quit()
         >>>

 -- Exception: ftplib.error_reply
     Exception raised when an unexpected reply is received from the
     server.

 -- Exception: ftplib.error_temp
     Exception raised when an error code signifying a temporary error
     (response codes in the range 400-499) is received.

 -- Exception: ftplib.error_perm
     Exception raised when an error code signifying a permanent error
     (response codes in the range 500-599) is received.

 -- Exception: ftplib.error_proto
     Exception raised when a reply is received from the server that
     does not fit the response specifications of the File Transfer
     Protocol, i.e. begin with a digit in the range 1-5.

 -- Data: ftplib.all_errors
     The set of all exceptions (as a tuple) that methods of FTP.
     instances may raise as a result of problems with the FTP
     connection (as opposed to programming errors made by the caller).
     This set includes the four exceptions listed above as well as
     socket.error and IOError.

Module netrc.
     Parser for the `.netrc' file format.  The file `.netrc' is
     typically used by FTP clients to load user authentication
     information before prompting the user.

  The file `Tools/scripts/ftpmirror.py' in the Python source
distribution is a script that can mirror FTP sites, or portions
thereof, using the ftplib.  module. It can be used as an
extended example that applies this module.

5.20.8.1 FTP Objects
....................

Several methods are available in two flavors: one for handling text
files and another for binary files.  These are named for the command
which is used followed by `lines' for the text version or `binary' for
the binary version.

  FTP instances have the following methods:

 -- Method: FTP.set_debuglevel (level)
     Set the instance's debugging level.  This controls the amount of
     debugging output printed.  The default, `0', produces no debugging
     output.  A value of `1' produces a moderate amount of debugging
     output, generally a single line per request.  A value of `2' or
     higher produces the maximum amount of debugging output, logging
     each line sent and received on the control connection.

 -- Method: FTP.connect (host[, port[, timeout]])
     Connect to the given host and port.  The default port number is
     `21', as specified by the FTP protocol specification.  It is
     rarely needed to specify a different port number.  This function
     should be called only once for each instance; it should not be
     called at all if a host was given when the instance was created.
     All other methods can only be used after a connection has been
     made.

     The optional _timeout_ parameter specifies a timeout in seconds
     for the connection attempt. If no _timeout_ is passed, the global
     default timeout setting will be used.

     Changed in version 2.6: _timeout_ was added.

 -- Method: FTP.getwelcome ()
     Return the welcome message sent by the server in reply to the
     initial connection.  (This message sometimes contains disclaimers
     or help information that may be relevant to the user.)

 -- Method: FTP.login ([user[, passwd[, acct]]])
     Log in as the given _user_.  The _passwd_ and _acct_ parameters
     are optional and default to the empty string.  If no _user_ is
     specified, it defaults to `'anonymous''.  If _user_ is
     `'anonymous'', the default _passwd_ is `'anonymous@''.  This
     function should be called only once for each instance, after a
     connection has been established; it should not be called at all if
     a host and user were given when the instance was created.  Most
     FTP commands are only allowed after the client has logged in.  The
     _acct_ parameter supplies "accounting information"; few systems
     implement this.

 -- Method: FTP.abort ()
     Abort a file transfer that is in progress.  Using this does not
     always work, but it's worth a try.

 -- Method: FTP.sendcmd (command)
     Send a simple command string to the server and return the response
     string.

 -- Method: FTP.voidcmd (command)
     Send a simple command string to the server and handle the
     response.  Return nothing if a response code corresponding to
     success (codes in the range 200-299) is received.  Raise
     error_reply otherwise.

 -- Method: FTP.retrbinary (command, callback[, maxblocksize[, rest]])
     Retrieve a file in binary transfer mode.  _command_ should be an
     appropriate `RETR' command: `'RETR filename''. The _callback_
     function is called for each block of data received, with a single
     string argument giving the data block. The optional _maxblocksize_
     argument specifies the maximum chunk size to read on the low-level
     socket object created to do the actual transfer (which will also
     be the largest size of the data blocks passed to _callback_).  A
     reasonable default is chosen. _rest_ means the same thing as in the
     transfercmd() method.

 -- Method: FTP.retrlines (command[, callback])
     Retrieve a file or directory listing in ASCII transfer mode.
     _command_ should be an appropriate `RETR' command (see
     retrbinary()) or a command such as `LIST', `NLST' or `MLSD'
     (usually just the string `'LIST'').  `LIST' retrieves a list of
     files and information about those files.  `NLST' retrieves a list
     of file names.  On some servers, `MLSD' retrieves a machine
     readable list of files and information about those files.  The
     _callback_ function is called for each line with a string argument
     containing the line with the trailing CRLF stripped.  The default
     _callback_ prints the line to `sys.stdout'.

 -- Method: FTP.set_pasv (boolean)
     Enable "passive" mode if _boolean_ is true, other disable passive
     mode.  (In Python 2.0 and before, passive mode was off by default;
     in Python 2.1 and later, it is on by default.)

 -- Method: FTP.storbinary (command, file[, blocksize, callback, rest])
     Store a file in binary transfer mode.  _command_ should be an
     appropriate `STOR' command: `"STOR filename"'. _file_ is an open
     file object which is read until EOF using its `read()' method in
     blocks of size _blocksize_ to provide the data to be stored.  The
     _blocksize_ argument defaults to 8192.  _callback_ is an optional
     single parameter callable that is called on each block of data
     after it is sent. _rest_ means the same thing as in the
     transfercmd() method.

     Changed in version 2.1: default for _blocksize_ added.

     Changed in version 2.6: _callback_ parameter added.

     Changed in version 2.7: _rest_ parameter added.

 -- Method: FTP.storlines (command, file[, callback])
     Store a file in ASCII transfer mode.  _command_ should be an
     appropriate `STOR' command (see storbinary()).  Lines
     are read until EOF from the open file object _file_ using its
     readline() method to provide the data to be stored.
     _callback_ is an optional single parameter callable that is called
     on each line after it is sent.

     Changed in version 2.6: _callback_ parameter added.

 -- Method: FTP.transfercmd (cmd[, rest])
     Initiate a transfer over the data connection.  If the transfer is
     active, send a `EPRT' or  `PORT' command and the transfer command
     specified by _cmd_, and accept the connection.  If the server is
     passive, send a `EPSV' or `PASV' command, connect to it, and start
     the transfer command.  Either way, return the socket for the
     connection.

     If optional _rest_ is given, a `REST' command is sent to the
     server, passing _rest_ as an argument.  _rest_ is usually a byte
     offset into the requested file, telling the server to restart
     sending the file's bytes at the requested offset, skipping over
     the initial bytes.  Note however that RFC 959 requires only that
     _rest_ be a string containing characters in the printable range
     from ASCII code 33 to ASCII code 126.  The transfercmd()
     method, therefore, converts _rest_ to a string, but no check
     is performed on the string's contents.  If the server does not
     recognize the `REST' command, an error_reply exception
     will be raised.  If this happens, simply call transfercmd()
     without a _rest_ argument.

 -- Method: FTP.ntransfercmd (cmd[, rest])
     Like transfercmd(), but returns a tuple of the data
     connection and the expected size of the data.  If the expected
     size could not be computed, `None' will be returned as the
     expected size.  _cmd_ and _rest_ means the same thing as in
     transfercmd().

 -- Method: FTP.nlst (argument[, ...])
     Return a list of file names as returned by the `NLST' command.  The
     optional _argument_ is a directory to list (default is the current
     server directory).  Multiple arguments can be used to pass
     non-standard options to the `NLST' command.

 -- Method: FTP.dir (argument[, ...])
     Produce a directory listing as returned by the `LIST' command,
     printing it to standard output.  The optional _argument_ is a
     directory to list (default is the current server directory).
     Multiple arguments can be used to pass non-standard options to the
     `LIST' command.  If the last argument is a function, it is used as
     a _callback_ function as for retrlines().; the default
     prints to `sys.stdout'.  This method returns `None'.

 -- Method: FTP.rename (fromname, toname)
     Rename file _fromname_ on the server to _toname_.

 -- Method: FTP.delete (filename)
     Remove the file named _filename_ from the server.  If successful,
     returns the text of the response, otherwise raises
     error_perm on permission errors or error_reply.
     on other errors.

 -- Method: FTP.cwd (pathname)
     Set the current directory on the server.

 -- Method: FTP.mkd (pathname)
     Create a new directory on the server.

 -- Method: FTP.pwd ()
     Return the pathname of the current directory on the server.

 -- Method: FTP.rmd (dirname)
     Remove the directory named _dirname_ on the server.

 -- Method: FTP.size (filename)
     Request the size of the file named _filename_ on the server.  On
     success, the size of the file is returned as an integer, otherwise
     `None' is returned.  Note that the `SIZE' command is not
     standardized, but is supported by many common server
     implementations.

 -- Method: FTP.quit ()
     Send a `QUIT' command to the server and close the connection. This
     is the "polite" way to close a connection, but it may raise an
     exception if the server responds with an error to the `QUIT'
     command.  This implies a call to the close() method
     which renders the FTP instance useless for subsequent
     calls (see below).

 -- Method: FTP.close ()
     Close the connection unilaterally.  This should not be applied to
     an already closed connection such as after a successful call to
     quit().  After this call the FTP instance
     should not be used any more (after a call to close().
     or quit() you cannot reopen the connection by issuing
     another login() method).

5.20.8.2 FTP_TLS Objects
........................

FTP_TLS class inherits from FTP, defining these
additional objects:

 -- Attribute: FTP_TLS.ssl_version
     The SSL version to use (defaults to _TLSv1_).

 -- Method: FTP_TLS.auth ()
     Set up secure control connection by using TLS or SSL, depending on
     what specified in ssl_version() attribute.

 -- Method: FTP_TLS.prot_p ()
     Set up secure data connection.

 -- Method: FTP_TLS.prot_c ()
     Set up clear text data connection.

5.20.9 `poplib' -- POP3 protocol client
---------------------------------------

This module defines a class, see POP3, which encapsulates a
connection to a POP3 server and implements the protocol as defined in RFC
1725 (http://tools.ietf.org/html/rfc1725.html).  The POP3.
class supports both the minimal and optional command sets.
Additionally, this module provides a class POP3_SSL, which
provides support for connecting to POP3 servers that use SSL as an
underlying protocol layer.

  Note that POP3, though widely supported, is obsolescent.  The
implementation quality of POP3 servers varies widely, and too many are
quite poor. If your mailserver supports IMAP, you would be better off
using the imaplib.IMAP4 class, as IMAP servers tend to be
better implemented.

  A single class is provided by the poplib module:

 -- Class: poplib.POP3 (host[, port[, timeout]])
     This class implements the actual POP3 protocol.  The connection is
     created when the instance is initialized. If _port_ is omitted,
     the standard POP3 port (110) is used. The optional _timeout_
     parameter specifies a timeout in seconds for the connection
     attempt (if not specified, the global default timeout setting will
     be used).

     Changed in version 2.6: _timeout_ was added.

 -- Class: poplib.POP3_SSL (host[, port[, keyfile[, certfile]]])
     This is a subclass of POP3 that connects to the
     server over an SSL encrypted socket.  If _port_ is not specified,
     995, the standard POP3-over-SSL port is used.  _keyfile_ and
     _certfile_ are also optional - they can contain a PEM formatted
     private key and certificate chain file for the SSL connection.

     New in version 2.4.

  One exception is defined as an attribute of the poplib.
module:

 -- Exception: poplib.error_proto
     Exception raised on any errors from this module (errors from
     socket.  module are not caught). The reason for the exception
     is passed to the constructor as a string.

Module imaplib.
     The standard Python IMAP module.

Frequently Asked Questions About Fetchmail (http://www.catb.org/~esr/fetchmail/fetchmail-FAQ.html)
     The FAQ for the *fetchmail* POP/IMAP client collects information on
     POP3 server variations and RFC noncompliance that may be useful if
     you need to write an application based on the POP protocol.

5.20.9.1 POP3 Objects
.....................

All POP3 commands are represented by methods of the same name, in
lower-case; most return the response text sent by the server.

  An POP3 instance has the following methods:

 -- Method: POP3.set_debuglevel (level)
     Set the instance's debugging level.  This controls the amount of
     debugging output printed.  The default, `0', produces no debugging
     output.  A value of `1' produces a moderate amount of debugging
     output, generally a single line per request.  A value of `2' or
     higher produces the maximum amount of debugging output, logging
     each line sent and received on the control connection.

 -- Method: POP3.getwelcome ()
     Returns the greeting string sent by the POP3 server.

 -- Method: POP3.user (username)
     Send user command, response should indicate that a password is
     required.

 -- Method: POP3.pass_ (password)
     Send password, response includes message count and mailbox size.
     Note: the mailbox on the server is locked until quit().
     is called.

 -- Method: POP3.apop (user, secret)
     Use the more secure APOP authentication to log into the POP3
     server.

 -- Method: POP3.rpop (user)
     Use RPOP authentication (similar to UNIX r-commands) to log into
     POP3 server.

 -- Method: POP3.stat ()
     Get mailbox status.  The result is a tuple of 2 integers:
     `(message count, mailbox size)'.

 -- Method: POP3.list ([which])
     Request message list, result is in the form `(response, ['mesg_num
     octets', ...], octets)'. If _which_ is set, it is the message to
     list.

 -- Method: POP3.retr (which)
     Retrieve whole message number _which_, and set its seen flag.
     Result is in form `(response, ['line', ...], octets)'.

 -- Method: POP3.dele (which)
     Flag message number _which_ for deletion.  On most servers
     deletions are not actually performed until QUIT (the major
     exception is Eudora QPOP, which deliberately violates the RFCs by
     doing pending deletes on any disconnect).

 -- Method: POP3.rset ()
     Remove any deletion marks for the mailbox.

 -- Method: POP3.noop ()
     Do nothing.  Might be used as a keep-alive.

 -- Method: POP3.quit ()
     Signoff:  commit changes, unlock mailbox, drop connection.

 -- Method: POP3.top (which, howmuch)
     Retrieves the message header plus _howmuch_ lines of the message
     after the header of message number _which_. Result is in form
     `(response, ['line', ...], octets)'.

     The POP3 TOP command this method uses, unlike the RETR command,
     doesn't set the message's seen flag; unfortunately, TOP is poorly
     specified in the RFCs and is frequently broken in off-brand
     servers. Test this method by hand against the POP3 servers you
     will use before trusting it.

 -- Method: POP3.uidl ([which])
     Return message digest (unique id) list. If _which_ is specified,
     result contains the unique id for that message in the form
     `'response mesgnum uid', otherwise result is list `(response,
     ['mesgnum uid', ...], octets)'.

  Instances of POP3_SSL have no additional methods. The
interface of this subclass is identical to its parent.

5.20.9.2 POP3 Example
.....................

Here is a minimal example (without error checking) that opens a mailbox
and retrieves and prints all messages:

    import getpass, poplib

    M = poplib.POP3('localhost')
    M.user(getpass.getuser())
    M.pass_(getpass.getpass())
    numMessages = len(M.list()[1])
    for i in range(numMessages):
        for j in M.retr(i+1)[1]:
            print j

At the end of the module, there is a test section that contains a more
extensive example of usage.

5.20.10 `imaplib' -- IMAP4 protocol client
------------------------------------------

This module defines three classes, see IMAP4, IMAP4_SSL
and IMAP4_stream, which encapsulate a connection to
an IMAP4 server and implement a large subset of the IMAP4rev1 client
protocol as defined in RFC 2060
(http://tools.ietf.org/html/rfc2060.html). It is backward compatible
with IMAP4 ( RFC 1730 (http://tools.ietf.org/html/rfc1730.html))
servers, but note that the `STATUS' command is not supported in IMAP4.

  Three classes are provided by the imaplib module, see
IMAP4 is the base class:

 -- Class: imaplib.IMAP4 ([host[, port]])
     This class implements the actual IMAP4 protocol.  The connection
     is created and protocol version (IMAP4 or IMAP4rev1) is determined
     when the instance is initialized. If _host_ is not specified, `'''
     (the local host) is used. If _port_ is omitted, the standard IMAP4
     port (143) is used.

  Three exceptions are defined as attributes of the IMAP4.
class:

 -- Exception: IMAP4.error
     Exception raised on any errors.  The reason for the exception is
     passed to the constructor as a string.

 -- Exception: IMAP4.abort
     IMAP4 server errors cause this exception to be raised.  This is a
     sub-class of IMAP4.error.  Note that closing the
     instance and instantiating a new one will usually allow recovery
     from this exception.

 -- Exception: IMAP4.readonly
     This exception is raised when a writable mailbox has its status
     changed by the server.  This is a sub-class of IMAP4.error.
     Some other client now has write permission, and the mailbox
     will need to be re-opened to re-obtain write permission.

  There's also a subclass for secure connections:

 -- Class: imaplib.IMAP4_SSL ([host[, port[, keyfile[, certfile]]]])
     This is a subclass derived from IMAP4 that connects
     over an SSL encrypted socket (to use this class you need a socket
     module that was compiled with SSL support).  If _host_ is not
     specified, `''' (the local host) is used.  If _port_ is omitted,
     the standard IMAP4-over-SSL port (993) is used.  _keyfile_ and
     _certfile_ are also optional - they can contain a PEM formatted
     private key and certificate chain file for the SSL connection.

  The second subclass allows for connections created by a child process:

 -- Class: imaplib.IMAP4_stream (command)
     This is a subclass derived from IMAP4 that connects
     to the `stdin/stdout' file descriptors created by passing
     _command_ to `os.popen2()'.

     New in version 2.3.

  The following utility functions are defined:

 -- Function: imaplib.Internaldate2tuple (datestr)
     Converts an IMAP4 INTERNALDATE string to Coordinated Universal
     Time. Returns a time module tuple.

 -- Function: imaplib.Int2AP (num)
     Converts an integer into a string representation using characters
     from the set [`A' .. `P'].

 -- Function: imaplib.ParseFlags (flagstr)
     Converts an IMAP4 `FLAGS' response to a tuple of individual flags.

 -- Function: imaplib.Time2Internaldate (date_time)
     Converts a time module tuple to an IMAP4
     `INTERNALDATE' representation.  Returns a string in the form:
     `"DD-Mmm-YYYY HH:MM:SS +HHMM"' (including double-quotes).

  Note that IMAP4 message numbers change as the mailbox changes; in
particular, after an `EXPUNGE' command performs deletions the remaining
messages are renumbered. So it is highly advisable to use UIDs instead,
with the UID command.

  At the end of the module, there is a test section that contains a
more extensive example of usage.

  Documents describing the protocol, and sources and binaries  for
servers implementing it, can all be found at the University of
Washington's _IMAP Information Center_
(<http://www.washington.edu/imap/>).

5.20.10.1 IMAP4 Objects
.......................

All IMAP4rev1 commands are represented by methods of the same name,
either upper-case or lower-case.

  All arguments to commands are converted to strings, except for
`AUTHENTICATE', and the last argument to `APPEND' which is passed as an
IMAP4 literal.  If necessary (the string contains IMAP4
protocol-sensitive characters and isn't enclosed with either
parentheses or double quotes) each string is quoted.  However, the
_password_ argument to the `LOGIN' command is always quoted. If you
want to avoid having an argument string quoted (eg: the _flags_
argument to `STORE') then enclose the string in parentheses (eg:
`r'(\Deleted)'').

  Each command returns a tuple: `(type, [data, ...])' where _type_ is
usually `'OK'' or `'NO'', and _data_ is either the text from the
command response, or mandated results from the command. Each _data_ is
either a string, or a tuple. If a tuple, then the first part is the
header of the response, and the second part contains the data (ie:
'literal' value).

  The _message_set_ options to commands below is a string specifying
one or more messages to be acted upon.  It may be a simple message
number (`'1''), a range of message numbers (`'2:4''), or a group of
non-contiguous ranges separated by commas (`'1:3,6:9'').  A range can
contain an asterisk to indicate an infinite upper bound (`'3:*'').

  An IMAP4 instance has the following methods:

 -- Method: IMAP4.append (mailbox, flags, date_time, message)
     Append _message_ to named mailbox.

 -- Method: IMAP4.authenticate (mechanism, authobject)
     Authenticate command -- requires response processing.

     _mechanism_ specifies which authentication mechanism is to be used
     - it should appear in the instance variable `capabilities' in the
     form `AUTH=mechanism'.

     _authobject_ must be a callable object:

         data = authobject(response)

     It will be called to process server continuation responses. It
     should return `data' that will be encoded and sent to server. It
     should return `None' if the client abort response `*' should be
     sent instead.

 -- Method: IMAP4.check ()
     Checkpoint mailbox on server.

 -- Method: IMAP4.close ()
     Close currently selected mailbox. Deleted messages are removed
     from writable mailbox. This is the recommended command before
     `LOGOUT'.

 -- Method: IMAP4.copy (message_set, new_mailbox)
     Copy _message_set_ messages onto end of _new_mailbox_.

 -- Method: IMAP4.create (mailbox)
     Create new mailbox named _mailbox_.

 -- Method: IMAP4.delete (mailbox)
     Delete old mailbox named _mailbox_.

 -- Method: IMAP4.deleteacl (mailbox, who)
     Delete the ACLs (remove any rights) set for who on mailbox.

     New in version 2.4.

 -- Method: IMAP4.expunge ()
     Permanently remove deleted items from selected mailbox. Generates
     an `EXPUNGE' response for each deleted message. Returned data
     contains a list of `EXPUNGE' message numbers in order received.

 -- Method: IMAP4.fetch (message_set, message_parts)
     Fetch (parts of) messages.  _message_parts_ should be a string of
     message part names enclosed within parentheses, eg: `"(UID
     BODY[TEXT])"'.  Returned data are tuples of message part envelope
     and data.

 -- Method: IMAP4.getacl (mailbox)
     Get the `ACL's for _mailbox_. The method is non-standard, but is
     supported by the `Cyrus' server.

 -- Method: IMAP4.getannotation (mailbox, entry, attribute)
     Retrieve the specified `ANNOTATION's for _mailbox_. The method is
     non-standard, but is supported by the `Cyrus' server.

     New in version 2.5.

 -- Method: IMAP4.getquota (root)
     Get the `quota' _root_'s resource usage and limits. This method is
     part of the IMAP4 QUOTA extension defined in rfc2087.

     New in version 2.3.

 -- Method: IMAP4.getquotaroot (mailbox)
     Get the list of `quota' `roots' for the named _mailbox_. This
     method is part of the IMAP4 QUOTA extension defined in rfc2087.

     New in version 2.3.

 -- Method: IMAP4.list ([directory[, pattern]])
     List mailbox names in _directory_ matching _pattern_.  _directory_
     defaults to the top-level mail folder, and _pattern_ defaults to
     match anything.  Returned data contains a list of `LIST' responses.

 -- Method: IMAP4.login (user, password)
     Identify the client using a plaintext password. The _password_
     will be quoted.

 -- Method: IMAP4.login_cram_md5 (user, password)
     Force use of `CRAM-MD5' authentication when identifying the client
     to protect the password.  Will only work if the server
     `CAPABILITY' response includes the phrase `AUTH=CRAM-MD5'.

     New in version 2.3.

 -- Method: IMAP4.logout ()
     Shutdown connection to server. Returns server `BYE' response.

 -- Method: IMAP4.lsub ([directory[, pattern]])
     List subscribed mailbox names in directory matching pattern.
     _directory_ defaults to the top level directory and _pattern_
     defaults to match any mailbox.  Returned data are tuples of
     message part envelope and data.

 -- Method: IMAP4.myrights (mailbox)
     Show my ACLs for a mailbox (i.e. the rights that I have on
     mailbox).

     New in version 2.4.

 -- Method: IMAP4.namespace ()
     Returns IMAP namespaces as defined in RFC2342.

     New in version 2.3.

 -- Method: IMAP4.noop ()
     Send `NOOP' to server.

 -- Method: IMAP4.open (host, port)
     Opens socket to _port_ at _host_. The connection objects
     established by this method will be used in the `read', `readline',
     `send', and `shutdown' methods. You may override this method.

 -- Method: IMAP4.partial (message_num, message_part, start, length)
     Fetch truncated part of a message. Returned data is a tuple of
     message part envelope and data.

 -- Method: IMAP4.proxyauth (user)
     Assume authentication as _user_. Allows an authorised
     administrator to proxy into any user's mailbox.

     New in version 2.3.

 -- Method: IMAP4.read (size)
     Reads _size_ bytes from the remote server. You may override this
     method.

 -- Method: IMAP4.readline ()
     Reads one line from the remote server. You may override this
     method.

 -- Method: IMAP4.recent ()
     Prompt server for an update. Returned data is `None' if no new
     messages, else value of `RECENT' response.

 -- Method: IMAP4.rename (oldmailbox, newmailbox)
     Rename mailbox named _oldmailbox_ to _newmailbox_.

 -- Method: IMAP4.response (code)
     Return data for response _code_ if received, or `None'. Returns
     the given code, instead of the usual type.

 -- Method: IMAP4.search (charset, criterion[, ...])
     Search mailbox for matching messages.  _charset_ may be `None', in
     which case no `CHARSET' will be specified in the request to the
     server.  The IMAP protocol requires that at least one criterion be
     specified; an exception will be raised when the server returns an
     error.

     Example:

         # M is a connected IMAP4 instance...
         typ, msgnums = M.search(None, 'FROM', '"LDJ"')

         # or:
         typ, msgnums = M.search(None, '(FROM "LDJ")')

 -- Method: IMAP4.select ([mailbox[, readonly]])
     Select a mailbox. Returned data is the count of messages in
     _mailbox_ (`EXISTS' response).  The default _mailbox_ is
     `'INBOX''.  If the _readonly_ flag is set, modifications to the
     mailbox are not allowed.

 -- Method: IMAP4.send (data)
     Sends `data' to the remote server. You may override this method.

 -- Method: IMAP4.setacl (mailbox, who, what)
     Set an `ACL' for _mailbox_. The method is non-standard, but is
     supported by the `Cyrus' server.

 -- Method: IMAP4.setannotation (mailbox, entry, attribute[, ...])
     Set `ANNOTATION's for _mailbox_. The method is non-standard, but is
     supported by the `Cyrus' server.

     New in version 2.5.

 -- Method: IMAP4.setquota (root, limits)
     Set the `quota' _root_'s resource _limits_. This method is part of
     the IMAP4 QUOTA extension defined in rfc2087.

     New in version 2.3.

 -- Method: IMAP4.shutdown ()
     Close connection established in `open'. You may override this
     method.

 -- Method: IMAP4.socket ()
     Returns socket instance used to connect to server.

 -- Method: IMAP4.sort (sort_criteria, charset, search_criterion[, ...])
     The `sort' command is a variant of `search' with sorting semantics
     for the results.  Returned data contains a space separated list of
     matching message numbers.

     Sort has two arguments before the _search_criterion_ argument(s); a
     parenthesized list of _sort_criteria_, and the searching
     _charset_.  Note that unlike `search', the searching _charset_
     argument is mandatory.  There is also a `uid sort' command which
     corresponds to `sort' the way that `uid search' corresponds to
     `search'.  The `sort' command first searches the mailbox for
     messages that match the given searching criteria using the charset
     argument for the interpretation of strings in the searching
     criteria.  It then returns the numbers of matching messages.

     This is an `IMAP4rev1' extension command.

 -- Method: IMAP4.status (mailbox, names)
     Request named status conditions for _mailbox_.

 -- Method: IMAP4.store (message_set, command, flag_list)
     Alters flag dispositions for messages in mailbox.  _command_ is
     specified by section 6.4.6 of RFC 2060
     (http://tools.ietf.org/html/rfc2060.html) as being one of "FLAGS",
     "+FLAGS", or "-FLAGS", optionally with a suffix of ".SILENT".

     For example, to set the delete flag on all messages:

         typ, data = M.search(None, 'ALL')
         for num in data[0].split():
            M.store(num, '+FLAGS', '\\Deleted')
         M.expunge()

 -- Method: IMAP4.subscribe (mailbox)
     Subscribe to new mailbox.

 -- Method: IMAP4.thread (threading_algorithm, charset,
          search_criterion[, ...])
     The `thread' command is a variant of `search' with threading
     semantics for the results.  Returned data contains a space
     separated list of thread members.

     Thread members consist of zero or more messages numbers, delimited
     by spaces, indicating successive parent and child.

     Thread has two arguments before the _search_criterion_
     argument(s); a _threading_algorithm_, and the searching _charset_.
     Note that unlike `search', the searching _charset_ argument is
     mandatory.  There is also a `uid thread' command which corresponds
     to `thread' the way that `uid search' corresponds to `search'.
     The `thread' command first searches the mailbox for messages that
     match the given searching criteria using the charset argument for
     the interpretation of strings in the searching criteria. It then
     returns the matching messages threaded according to the specified
     threading algorithm.

     This is an `IMAP4rev1' extension command.

     New in version 2.4.

 -- Method: IMAP4.uid (command, arg[, ...])
     Execute command args with messages identified by UID, rather than
     message number.  Returns response appropriate to command.  At
     least one argument must be supplied; if none are provided, the
     server will return an error and an exception will be raised.

 -- Method: IMAP4.unsubscribe (mailbox)
     Unsubscribe from old mailbox.

 -- Method: IMAP4.xatom (name[, arg[, ...]])
     Allow simple extension commands notified by server in `CAPABILITY'
     response.

  Instances of IMAP4_SSL have just one additional method:

 -- Method: IMAP4_SSL.ssl ()
     Returns SSLObject instance used for the secure connection with the
     server.

  The following attributes are defined on instances of IMAP4.:

 -- Attribute: IMAP4.PROTOCOL_VERSION
     The most recent supported protocol in the `CAPABILITY' response
     from the server.

 -- Attribute: IMAP4.debug
     Integer value to control debugging output.  The initialize value
     is taken from the module variable `Debug'.  Values greater than
     three trace each command.

5.20.10.2 IMAP4 Example
.......................

Here is a minimal example (without error checking) that opens a mailbox
and retrieves and prints all messages:

    import getpass, imaplib

    M = imaplib.IMAP4()
    M.login(getpass.getuser(), getpass.getpass())
    M.select()
    typ, data = M.search(None, 'ALL')
    for num in data[0].split():
        typ, data = M.fetch(num, '(RFC822)')
        print 'Message %s\n%s\n' % (num, data[0][1])
    M.close()
    M.logout()

5.20.11 `nntplib' -- NNTP protocol client
-----------------------------------------

This module defines the class NNTP which implements the
client side of the NNTP protocol.  It can be used to implement a news
reader or poster, or automated news processors.  For more information
on NNTP (Network News Transfer Protocol), see Internet RFC 977
(http://tools.ietf.org/html/rfc977.html).

  Here are two small examples of how it can be used.  To list some
statistics about a newsgroup and print the subjects of the last 10
articles:

    >>> s = NNTP('news.gmane.org')
    >>> resp, count, first, last, name = s.group('gmane.comp.python.committers')
    >>> print 'Group', name, 'has', count, 'articles, range', first, 'to', last
    Group gmane.comp.python.committers has 1071 articles, range 1 to 1071
    >>> resp, subs = s.xhdr('subject', first + '-' + last)
    >>> for id, sub in subs[-10:]: print id, sub
    ...
    1062 Re: Mercurial Status?
    1063 Re: [python-committers]  (Windows) buildbots on 3.x
    1064 Re: Mercurial Status?
    1065 Re: Mercurial Status?
    1066 Python 2.6.6 status
    1067 Commit Privileges for Ask Solem
    1068 Re: Commit Privileges for Ask Solem
    1069 Re: Commit Privileges for Ask Solem
    1070 Re: Commit Privileges for Ask Solem
    1071 2.6.6 rc 2
    >>> s.quit()
    '205 Bye!'

To post an article from a file (this assumes that the article has valid
headers, and that you have right to post on the particular newsgroup):

    >>> s = NNTP('news.gmane.org')
    >>> f = open('/tmp/article')
    >>> s.post(f)
    '240 Article posted successfully.'
    >>> s.quit()
    '205 Bye!'

The module itself defines the following items:

 -- Class: nntplib.NNTP (host[, port [, user[, password [, readermode]
          [, usenetrc]]]])
     Return a new instance of the NNTP class, representing
     a connection to the NNTP server running on host _host_, listening
     at port _port_.  The default _port_ is 119.  If the optional
     _user_ and _password_ are provided, or if suitable credentials are
     present in `/.netrc' and the optional flag _usenetrc_ is true (the
     default), the `AUTHINFO USER' and `AUTHINFO PASS' commands are
     used to identify and authenticate the user to the server.  If the
     optional flag _readermode_ is true, then a `mode reader' command is
     sent before authentication is performed.  Reader mode is sometimes
     necessary if you are connecting to an NNTP server on the local
     machine and intend to call reader-specific commands, such as
     `group'.  If you get unexpected NNTPPermanentError.s,
     you might need to set _readermode_.  _readermode_ defaults to
     `None'. _usenetrc_ defaults to `True'.

     Changed in version 2.4: _usenetrc_ argument added.

 -- Exception: nntplib.NNTPError
     Derived from the standard exception Exception, this is
     the base class for all exceptions raised by the nntplib
     module.

 -- Exception: nntplib.NNTPReplyError
     Exception raised when an unexpected reply is received from the
     server.  For backwards compatibility, the exception `error_reply'
     is equivalent to this class.

 -- Exception: nntplib.NNTPTemporaryError
     Exception raised when an error code in the range 400-499 is
     received.  For backwards compatibility, the exception `error_temp'
     is equivalent to this class.

 -- Exception: nntplib.NNTPPermanentError
     Exception raised when an error code in the range 500-599 is
     received.  For backwards compatibility, the exception `error_perm'
     is equivalent to this class.

 -- Exception: nntplib.NNTPProtocolError
     Exception raised when a reply is received from the server that
     does not begin with a digit in the range 1-5.  For backwards
     compatibility, the exception `error_proto' is equivalent to this
     class.

 -- Exception: nntplib.NNTPDataError
     Exception raised when there is some error in the response data.
     For backwards compatibility, the exception `error_data' is
     equivalent to this class.

5.20.11.1 NNTP Objects
......................

NNTP instances have the following methods.  The _response_ that is
returned as the first item in the return tuple of almost all methods is
the server's response: a string beginning with a three-digit code. If
the server's response indicates an error, the method raises one of the
above exceptions.

 -- Method: NNTP.getwelcome ()
     Return the welcome message sent by the server in reply to the
     initial connection.  (This message sometimes contains disclaimers
     or help information that may be relevant to the user.)

 -- Method: NNTP.set_debuglevel (level)
     Set the instance's debugging level.  This controls the amount of
     debugging output printed.  The default, `0', produces no debugging
     output.  A value of `1' produces a moderate amount of debugging
     output, generally a single line per request or response.  A value
     of `2' or higher produces the maximum amount of debugging output,
     logging each line sent and received on the connection (including
     message text).

 -- Method: NNTP.newgroups (date, time[, file])
     Send a `NEWGROUPS' command.  The _date_ argument should be a
     string of the form `'yymmdd'' indicating the date, and _time_
     should be a string of the form `'hhmmss'' indicating the time.
     Return a pair `(response, groups)' where _groups_ is a list of
     group names that are new since the given date and time. If the
     _file_ parameter is supplied, then the output of the  `NEWGROUPS'
     command is stored in a file.  If _file_ is a string,  then the
     method will open a file object with that name, write to it  then
     close it.  If _file_ is a file object, then it will start calling
     `write()' on it to store the lines of the command output. If
     _file_ is supplied, then the returned _list_ is an empty list.

 -- Method: NNTP.newnews (group, date, time[, file])
     Send a `NEWNEWS' command.  Here, _group_ is a group name or `'*'',
     and _date_ and _time_ have the same meaning as for
     newgroups().  Return a pair `(response, articles)' where
     _articles_ is a list of message ids. If the _file_ parameter is
     supplied, then the output of the  `NEWNEWS' command is stored in a
     file.  If _file_ is a string,  then the method will open a file
     object with that name, write to it  then close it.  If _file_ is a
     file object, then it will start calling `write()' on it to store
     the lines of the command output. If _file_ is supplied, then the
     returned _list_ is an empty list.

 -- Method: NNTP.list ([file])
     Send a `LIST' command.  Return a pair `(response, list)' where
     _list_ is a list of tuples.  Each tuple has the form `(group,
     last, first, flag)', where _group_ is a group name, _last_ and
     _first_ are the last and first article numbers (as strings), and
     _flag_ is `'y'' if posting is allowed, `'n'' if not, and `'m'' if
     the newsgroup is moderated.  (Note the ordering: _last_, _first_.)
     If the _file_ parameter is supplied, then the output of the  `LIST'
     command is stored in a file.  If _file_ is a string,  then the
     method will open a file object with that name, write to it  then
     close it.  If _file_ is a file object, then it will start calling
     `write()' on it to store the lines of the command output. If
     _file_ is supplied, then the returned _list_ is an empty list.

 -- Method: NNTP.descriptions (grouppattern)
     Send a `LIST NEWSGROUPS' command, where _grouppattern_ is a
     wildmat string as specified in RFC2980 (it's essentially the same
     as DOS or UNIX shell wildcard strings).  Return a pair `(response,
     list)', where _list_ is a list of tuples containing `(name,
     title)'.

     New in version 2.4.

 -- Method: NNTP.description (group)
     Get a description for a single group _group_.  If more than one
     group matches (if 'group' is a real wildmat string), return the
     first match.   If no group matches, return an empty string.

     This elides the response code from the server.  If the response
     code is needed, use descriptions().

     New in version 2.4.

 -- Method: NNTP.group (name)
     Send a `GROUP' command, where _name_ is the group name. Return a
     tuple `(response, count, first, last, name)' where _count_ is the
     (estimated) number of articles in the group, _first_ is the first
     article number in the group, _last_ is the last article number in
     the group, and _name_ is the group name.  The numbers are returned
     as strings.

 -- Method: NNTP.help ([file])
     Send a `HELP' command.  Return a pair `(response, list)' where
     _list_ is a list of help strings. If the _file_ parameter is
     supplied, then the output of the  `HELP' command is stored in a
     file.  If _file_ is a string,  then the method will open a file
     object with that name, write to it  then close it.  If _file_ is a
     file object, then it will start calling `write()' on it to store
     the lines of the command output. If _file_ is supplied, then the
     returned _list_ is an empty list.

 -- Method: NNTP.stat (id)
     Send a `STAT' command, where _id_ is the message id (enclosed in
     `'<'' and `'>'') or an article number (as a string). Return a
     triple `(response, number, id)' where _number_ is the article
     number (as a string) and _id_ is the message id  (enclosed in
     `'<'' and `'>'').

 -- Method: NNTP.next ()
     Send a `NEXT' command.  Return as for stat().

 -- Method: NNTP.last ()
     Send a `LAST' command.  Return as for stat().

 -- Method: NNTP.head (id)
     Send a `HEAD' command, where _id_ has the same meaning as for
     stat().  Return a tuple `(response, number, id, list)'
     where the first three are the same as for stat(), and
     _list_ is a list of the article's headers (an uninterpreted list
     of lines, without trailing newlines).

 -- Method: NNTP.body (id[, file])
     Send a `BODY' command, where _id_ has the same meaning as for
     stat().  If the _file_ parameter is supplied, then the
     body is stored in a file.  If _file_ is a string, then the method
     will open a file object with that name, write to it then close it.
     If _file_ is a file object, then it will start calling `write()'
     on it to store the lines of the body. Return as for head().
     If _file_ is supplied, then the returned _list_ is an empty
     list.

 -- Method: NNTP.article (id)
     Send an `ARTICLE' command, where _id_ has the same meaning as for
     stat().  Return as for head().

 -- Method: NNTP.slave ()
     Send a `SLAVE' command.  Return the server's _response_.

 -- Method: NNTP.xhdr (header, string[, file])
     Send an `XHDR' command.  This command is not defined in the RFC
     but is a common extension.  The _header_ argument is a header
     keyword, e.g.  `'subject''.  The _string_ argument should have the
     form `'first-last'' where _first_ and _last_ are the first and
     last article numbers to search.  Return a pair `(response, list)',
     where _list_ is a list of pairs `(id, text)', where _id_ is an
     article number (as a string) and _text_ is the text of the
     requested header for that article. If the _file_ parameter is
     supplied, then the output of the  `XHDR' command is stored in a
     file.  If _file_ is a string, then the method will open a file
     object with that name, write to it  then close it.  If _file_ is a
     file object, then it will start calling `write()' on it to store
     the lines of the command output. If _file_ is supplied, then the
     returned _list_ is an empty list.

 -- Method: NNTP.post (file)
     Post an article using the `POST' command.  The _file_ argument is
     an open file object which is read until EOF using its
     readline() method.  It should be a well-formed news article,
     including the required headers.  The post().  method
     automatically escapes lines beginning with `.'.

 -- Method: NNTP.ihave (id, file)
     Send an `IHAVE' command. _id_ is a message id (enclosed in  `'<''
     and `'>''). If the response is not an error, treat _file_ exactly
     as for the post() method.

 -- Method: NNTP.date ()
     Return a triple `(response, date, time)', containing the current
     date and time in a form suitable for the newnews().
     and newgroups() methods. This is an optional NNTP
     extension, and may not be supported by all servers.

 -- Method: NNTP.xgtitle (name[, file])
     Process an `XGTITLE' command, returning a pair `(response, list)',
     where _list_ is a list of tuples containing `(name, title)'. If
     the _file_ parameter is supplied, then the output of the
     `XGTITLE' command is stored in a file.  If _file_ is a string,
     then the method will open a file object with that name, write to
     it  then close it.  If _file_ is a file object, then it will start
     calling `write()' on it to store the lines of the command output.
     If _file_ is supplied, then the returned _list_ is an empty list.
     This is an optional NNTP extension, and may not be supported by
     all servers.

     RFC2980 says "It is suggested that this extension be deprecated".
     Use descriptions() or description().
     instead.

 -- Method: NNTP.xover (start, end[, file])
     Return a pair `(resp, list)'.  _list_ is a list of tuples, one for
     each article in the range delimited by the _start_ and _end_
     article numbers.  Each tuple is of the form `(article number,
     subject, poster, date, id, references, size, lines)'. If the
     _file_ parameter is supplied, then the output of the `XOVER'
     command is stored in a file.  If _file_ is a string,  then the
     method will open a file object with that name, write to it  then
     close it.  If _file_ is a file object, then it will start calling
     `write()' on it to store the lines of the command output. If
     _file_ is supplied, then the returned _list_ is an empty list.
     This is an optional NNTP extension, and may not be supported by
     all servers.

 -- Method: NNTP.xpath (id)
     Return a pair `(resp, path)', where _path_ is the directory path
     to the article with message ID _id_.  This is an optional NNTP
     extension, and may not be supported by all servers.

 -- Method: NNTP.quit ()
     Send a `QUIT' command and close the connection.  Once this method
     has been called, no other methods of the NNTP object should be
     called.

5.20.12 `smtplib' -- SMTP protocol client
-----------------------------------------

The smtplib module defines an SMTP client session object
that can be used to send mail to any Internet machine with an SMTP or
ESMTP listener daemon.  For details of SMTP and ESMTP operation, consult RFC
821 (http://tools.ietf.org/html/rfc821.html) (Simple Mail Transfer
Protocol) and RFC 1869 (http://tools.ietf.org/html/rfc1869.html) (SMTP
Service Extensions).

 -- Class: smtplib.SMTP ([host[, port[, local_hostname[, timeout]]]])
     A SMTP instance encapsulates an SMTP connection.  It
     has methods that support a full repertoire of SMTP and ESMTP
     operations. If the optional host and port parameters are given,
     the SMTP connect() method is called with those
     parameters during initialization.  An SMTPConnectError
     is raised if the specified host doesn't respond correctly.
     The optional _timeout_ parameter specifies a timeout in seconds
     for blocking operations like the connection attempt (if not
     specified, the global default timeout setting will be used).

     For normal use, you should only require the initialization/connect,
     see sendmail(), and quit() methods.  An
     example is included below.

     Changed in version 2.6: _timeout_ was added.

 -- Class: smtplib.SMTP_SSL ([host[, port[, local_hostname[, keyfile[,
          certfile[, timeout]]]]]])
     A SMTP_SSL instance behaves exactly the same as
     instances of SMTP. SMTP_SSL should be
     used for situations where SSL is required from the beginning of
     the connection and using `starttls()' is not appropriate. If
     _host_ is not specified, the local host is used. If _port_ is
     omitted, the standard SMTP-over-SSL port (465) is used. _keyfile_
     and _certfile_ are also optional, and can contain a PEM formatted
     private key and certificate chain file for the SSL connection. The
     optional _timeout_ parameter specifies a timeout in seconds for
     blocking operations like the connection attempt (if not specified,
     the global default timeout setting will be used).

     New in version 2.6.

 -- Class: smtplib.LMTP ([host[, port[, local_hostname]]])
     The LMTP protocol, which is very similar to ESMTP, is heavily
     based on the standard SMTP client. It's common to use Unix sockets
     for LMTP, so our `connect()' method must support that as well as a
     regular host:port server. To specify a Unix socket, you must use
     an absolute path for _host_, starting with a '/'.

     Authentication is supported, using the regular SMTP mechanism.
     When using a Unix socket, LMTP generally don't support or require
     any authentication, but your mileage might vary.

     New in version 2.6.

  A nice selection of exceptions is defined as well:

 -- Exception: smtplib.SMTPException
     Base exception class for all exceptions raised by this module.

 -- Exception: smtplib.SMTPServerDisconnected
     This exception is raised when the server unexpectedly disconnects,
     or when an attempt is made to use the SMTP instance
     before connecting it to a server.

 -- Exception: smtplib.SMTPResponseException
     Base class for all exceptions that include an SMTP error code.
     These exceptions are generated in some instances when the SMTP
     server returns an error code.  The error code is stored in the
     `smtp_code' attribute of the error, and the `smtp_error' attribute
     is set to the error message.

 -- Exception: smtplib.SMTPSenderRefused
     Sender address refused.  In addition to the attributes set by on
     all SMTPResponseException exceptions, this sets
     'sender' to the string that the SMTP server refused.

 -- Exception: smtplib.SMTPRecipientsRefused
     All recipient addresses refused.  The errors for each recipient
     are accessible through the attribute `recipients', which is a
     dictionary of exactly the same sort as SMTP.sendmail()
     returns.

 -- Exception: smtplib.SMTPDataError
     The SMTP server refused to accept the message data.

 -- Exception: smtplib.SMTPConnectError
     Error occurred during establishment of a connection  with the
     server.

 -- Exception: smtplib.SMTPHeloError
     The server refused our `HELO' message.

 -- Exception: smtplib.SMTPAuthenticationError
     SMTP authentication went wrong.  Most probably the server didn't
     accept the username/password combination provided.

RFC 821 (http://tools.ietf.org/html/rfc821.html) - Simple Mail Transfer Protocol
     Protocol definition for SMTP.  This document covers the model,
     operating procedure, and protocol details for SMTP.

RFC 1869 (http://tools.ietf.org/html/rfc1869.html) - SMTP Service Extensions
     Definition of the ESMTP extensions for SMTP.  This describes a
     framework for extending SMTP with new commands, supporting dynamic
     discovery of the commands provided by the server, and defines a
     few additional commands.

5.20.12.1 SMTP Objects
......................

An SMTP instance has the following methods:

 -- Method: SMTP.set_debuglevel (level)
     Set the debug output level.  A true value for _level_ results in
     debug messages for connection and for all messages sent to and
     received from the server.

 -- Method: SMTP.connect ([host[, port]])
     Connect to a host on a given port.  The defaults are to connect to
     the local host at the standard SMTP port (25). If the hostname
     ends with a colon (`':'') followed by a number, that suffix will
     be stripped off and the number interpreted as the port number to
     use. This method is automatically invoked by the constructor if a
     host is specified during instantiation.

 -- Method: SMTP.docmd (cmd[, argstring])
     Send a command _cmd_ to the server.  The optional argument
     _argstring_ is simply concatenated to the command, separated by a
     space.

     This returns a 2-tuple composed of a numeric response code and the
     actual response line (multiline responses are joined into one long
     line.)

     In normal operation it should not be necessary to call this method
     explicitly.  It is used to implement other methods and may be
     useful for testing private extensions.

     If the connection to the server is lost while waiting for the
     reply, see SMTPServerDisconnected will be raised.

 -- Method: SMTP.helo ([hostname])
     Identify yourself to the SMTP server using `HELO'.  The hostname
     argument defaults to the fully qualified domain name of the local
     host.  The message returned by the server is stored as the
     `helo_resp' attribute of the object.

     In normal operation it should not be necessary to call this method
     explicitly.  It will be implicitly called by the sendmail()
     when necessary.

 -- Method: SMTP.ehlo ([hostname])
     Identify yourself to an ESMTP server using `EHLO'.  The hostname
     argument defaults to the fully qualified domain name of the local
     host.  Examine the response for ESMTP option and store them for
     use by has_extn().  Also sets several informational
     attributes: the message returned by the server is stored as the
     `ehlo_resp' attribute, `does_esmtp' is set to true or false
     depending on whether the server supports ESMTP, and
     `esmtp_features' will be a dictionary containing the names of the
     SMTP service extensions this server supports, and their parameters
     (if any).

     Unless you wish to use has_extn() before sending
     mail, it should not be necessary to call this method explicitly.
     It will be implicitly called by sendmail() when
     necessary.

 -- Method: SMTP.ehlo_or_helo_if_needed ()
     This method call ehlo() and or helo() if
     there has been no previous `EHLO' or `HELO' command this session.
     It tries ESMTP `EHLO' first.

    See SMTPHeloError.
          The server didn't reply properly to the `HELO' greeting.

     New in version 2.6.

 -- Method: SMTP.has_extn (name)
     Return True if _name_ is in the set of SMTP service
     extensions returned by the server, see False otherwise.
     Case is ignored.

 -- Method: SMTP.verify (address)
     Check the validity of an address on this server using SMTP `VRFY'.
     Returns a tuple consisting of code 250 and a full RFC 822
     (http://tools.ietf.org/html/rfc822.html) address (including human
     name) if the user address is valid. Otherwise returns an SMTP
     error code of 400 or greater and an error string.

          Note: Many sites disable SMTP `VRFY' in order to foil
          spammers.

 -- Method: SMTP.login (user, password)
     Log in on an SMTP server that requires authentication. The
     arguments are the username and the password to authenticate with.
     If there has been no previous `EHLO' or `HELO' command this
     session, this method tries ESMTP `EHLO' first. This method will
     return normally if the authentication was successful, or may raise
     the following exceptions:

    SMTPHeloError.
          The server didn't reply properly to the `HELO' greeting.

    See SMTPAuthenticationError.
          The server didn't accept the username/password combination.

    See SMTPException.
          No suitable authentication method was found.

 -- Method: SMTP.starttls ([keyfile[, certfile]])
     Put the SMTP connection in TLS (Transport Layer Security) mode.
     All SMTP commands that follow will be encrypted.  You should then
     call ehlo().  again.

     If _keyfile_ and _certfile_ are provided, these are passed to the
     socket.  module's ssl() function.

     If there has been no previous `EHLO' or `HELO' command this
     session, this method tries ESMTP `EHLO' first.

     Changed in version 2.6.

    See SMTPHeloError.
          The server didn't reply properly to the `HELO' greeting.

    See SMTPException.
          The server does not support the STARTTLS extension.

     Changed in version 2.6.

    See RuntimeError.
          SSL/TLS support is not available to your Python interpreter.

 -- Method: SMTP.sendmail (from_addr, to_addrs, msg[, mail_options,
          rcpt_options])
     Send mail.  The required arguments are an RFC 822
     (http://tools.ietf.org/html/rfc822.html) from-address string, a
     list of RFC 822 (http://tools.ietf.org/html/rfc822.html)
     to-address strings (a bare string will be treated as a list with 1
     address), and a message string.  The caller may pass a list of
     ESMTP options (such as `8bitmime') to be used in `MAIL FROM'
     commands as _mail_options_.  ESMTP options (such as `DSN'
     commands) that should be used with all `RCPT' commands can be
     passed as _rcpt_options_.  (If you need to use different ESMTP
     options to different recipients you have to use the low-level
     methods such as `mail()', `rcpt()' and `data()' to send the
     message.)

          Note: The _from_addr_ and _to_addrs_ parameters are used to
          construct the message envelope used by the transport agents.
          The SMTP does not modify the message headers in
          any way.

     If there has been no previous `EHLO' or `HELO' command this
     session, this method tries ESMTP `EHLO' first. If the server does
     ESMTP, message size and each of the specified options will be
     passed to it (if the option is in the feature set the server
     advertises).  If `EHLO' fails, `HELO' will be tried and ESMTP
     options suppressed.

     This method will return normally if the mail is accepted for at
     least one recipient. Otherwise it will raise an exception.  That
     is, if this method does not raise an exception, then someone
     should get your mail. If this method does not raise an exception,
     it returns a dictionary, with one entry for each recipient that
     was refused.  Each entry contains a tuple of the SMTP error code
     and the accompanying error message sent by the server.

     This method may raise the following exceptions:

    SMTPRecipientsRefused.
          All recipients were refused.  Nobody got the mail.  The
          `recipients' attribute of the exception object is a
          dictionary with information about the refused recipients
          (like the one returned when at least one recipient was
          accepted).

    See SMTPHeloError.
          The server didn't reply properly to the `HELO' greeting.

    See SMTPSenderRefused.
          The server didn't accept the _from_addr_.

    See SMTPDataError.
          The server replied with an unexpected error code (other than
          a refusal of a recipient).

     Unless otherwise noted, the connection will be open even after an
     exception is raised.

 -- Method: SMTP.quit ()
     Terminate the SMTP session and close the connection.  Return the
     result of the SMTP `QUIT' command.

     Changed in version 2.6: Return a value.

  Low-level methods corresponding to the standard SMTP/ESMTP commands
`HELP', `RSET', `NOOP', `MAIL', `RCPT', and `DATA' are also supported.
Normally these do not need to be called directly, so they are not
documented here.  For details, consult the module code.

5.20.12.2 SMTP Example
......................

This example prompts the user for addresses needed in the message
envelope ('To' and 'From' addresses), and the message to be delivered.
Note that the headers to be included with the message must be included
in the message as entered; this example doesn't do any processing of the RFC
822 (http://tools.ietf.org/html/rfc822.html) headers.  In particular,
the 'To' and 'From' addresses must be included in the message headers
explicitly.

    import smtplib

    def prompt(prompt):
        return raw_input(prompt).strip()

    fromaddr = prompt("From: ")
    toaddrs  = prompt("To: ").split()
    print "Enter message, end with ^D (Unix) or ^Z (Windows):"

    # Add the From: and To: headers at the start!
    msg = ("From: %s\r\nTo: %s\r\n\r\n"
           % (fromaddr, ", ".join(toaddrs)))
    while 1:
        try:
            line = raw_input()
        except EOFError:
            break
        if not line:
            break
        msg = msg + line

    print "Message length is " + repr(len(msg))

    server = smtplib.SMTP('localhost')
    server.set_debuglevel(1)
    server.sendmail(fromaddr, toaddrs, msg)
    server.quit()

     Note: In general, you will want to use the email.
     package's features to construct an email message, which you can
     then convert to a string and send via `sendmail()'; see
     email; Examples.

5.20.13 `smtpd' -- SMTP Server
------------------------------

This module offers several classes to implement SMTP servers.  One is a
generic do-nothing implementation, which can be overridden, while the
other two offer specific mail-sending strategies.

5.20.13.1 SMTPServer Objects
............................

 -- Class: smtpd.SMTPServer (localaddr, remoteaddr)
     Create a new SMTPServer object, which binds to local
     address _localaddr_.  It will treat _remoteaddr_ as an upstream
     SMTP relayer.  It inherits from asyncore.dispatcher,
     and so will insert itself into asyncore.'s event loop on
     instantiation.

      -- Method: process_message (peer, mailfrom, rcpttos, data)
          Raise NotImplementedError exception. Override
          this in subclasses to do something useful with this message.
          Whatever was passed in the constructor as _remoteaddr_ will
          be available as the `_remoteaddr' attribute. _peer_ is the
          remote host's address, _mailfrom_ is the envelope originator,
          _rcpttos_ are the envelope recipients and _data_ is a string
          containing the contents of the e-mail (which should be in RFC
          2822 (http://tools.ietf.org/html/rfc2822.html) format).

5.20.13.2 DebuggingServer Objects
.................................

 -- Class: smtpd.DebuggingServer (localaddr, remoteaddr)
     Create a new debugging server.  Arguments are as per
     SMTPServer.  Messages will be discarded, and printed on
     stdout.

5.20.13.3 PureProxy Objects
...........................

 -- Class: smtpd.PureProxy (localaddr, remoteaddr)
     Create a new pure proxy server. Arguments are as per
     SMTPServer.  Everything will be relayed to _remoteaddr_.
     Note that running this has a good chance to make you into an open
     relay, so please be careful.

5.20.13.4 MailmanProxy Objects
..............................

 -- Class: smtpd.MailmanProxy (localaddr, remoteaddr)
     Create a new pure proxy server. Arguments are as per
     SMTPServer.  Everything will be relayed to _remoteaddr_,
     unless local mailman configurations knows about an address, in
     which case it will be handled via mailman.  Note that running this
     has a good chance to make you into an open relay, so please be
     careful.

5.20.14 `telnetlib' -- Telnet client
------------------------------------

The telnetlib module provides a Telnet class
that implements the Telnet protocol.  See RFC 854
(http://tools.ietf.org/html/rfc854.html) for details about the
protocol. In addition, it provides symbolic constants for the protocol
characters (see below), and for the telnet options. The symbolic names
of the telnet options follow the definitions in `arpa/telnet.h', with
the leading `TELOPT_' removed. For symbolic names of options which are
traditionally not included in `arpa/telnet.h', see the module source
itself.

  The symbolic constants for the telnet commands are: IAC, DONT, DO,
WONT, WILL, SE (Subnegotiation End), NOP (No Operation), DM (Data
Mark), BRK (Break), IP (Interrupt process), AO (Abort output), AYT (Are
You There), EC (Erase Character), EL (Erase Line), GA (Go Ahead), SB
(Subnegotiation Begin).

 -- Class: telnetlib.Telnet ([host[, port[, timeout]]])
     Telnet represents a connection to a Telnet server. The
     instance is initially not connected by default; the open()
     method must be used to establish a connection.
     Alternatively, the host name and optional port number can be
     passed to the constructor, to, in which case the connection to the
     server will be established before the constructor returns.  The
     optional _timeout_ parameter specifies a timeout in seconds for
     blocking operations like the connection attempt (if not specified,
     the global default timeout setting will be used).

     Do not reopen an already connected instance.

     This class has many `read_*()' methods.  Note that some of them
     raise EOFError when the end of the connection is read,
     because they can return an empty string for other reasons.  See
     the individual descriptions below.

     Changed in version 2.6: _timeout_ was added.

RFC 854 (http://tools.ietf.org/html/rfc854.html) - Telnet Protocol Specification
     Definition of the Telnet protocol.

5.20.14.1 Telnet Objects
........................

Telnet instances have the following methods:

 -- Method: Telnet.read_until (expected[, timeout])
     Read until a given string, _expected_, is encountered or until
     _timeout_ seconds have passed.

     When no match is found, return whatever is available instead,
     possibly the empty string.  Raise EOFError if the
     connection is closed and no cooked data is available.

 -- Method: Telnet.read_all ()
     Read all data until EOF; block until connection closed.

 -- Method: Telnet.read_some ()
     Read at least one byte of cooked data unless EOF is hit. Return
     `''' if EOF is hit.  Block if no data is immediately available.

 -- Method: Telnet.read_very_eager ()
     Read everything that can be without blocking in I/O (eager).

     Raise EOFError if connection closed and no cooked data
     available.  Return `''' if no cooked data available otherwise. Do
     not block unless in the midst of an IAC sequence.

 -- Method: Telnet.read_eager ()
     Read readily available data.

     Raise EOFError if connection closed and no cooked data
     available.  Return `''' if no cooked data available otherwise. Do
     not block unless in the midst of an IAC sequence.

 -- Method: Telnet.read_lazy ()
     Process and return data already in the queues (lazy).

     Raise EOFError if connection closed and no data
     available. Return `''' if no cooked data available otherwise.  Do
     not block unless in the midst of an IAC sequence.

 -- Method: Telnet.read_very_lazy ()
     Return any data available in the cooked queue (very lazy).

     Raise EOFError if connection closed and no data
     available. Return `''' if no cooked data available otherwise.
     This method never blocks.

 -- Method: Telnet.read_sb_data ()
     Return the data collected between a SB/SE pair (suboption
     begin/end). The callback should access these data when it was
     invoked with a `SE' command.  This method never blocks.

     New in version 2.3.

 -- Method: Telnet.open (host[, port[, timeout]])
     Connect to a host. The optional second argument is the port
     number, which defaults to the standard Telnet port (23). The
     optional _timeout_ parameter specifies a timeout in seconds for
     blocking operations like the connection attempt (if not specified,
     the global default timeout setting will be used).

     Do not try to reopen an already connected instance.

     Changed in version 2.6: _timeout_ was added.

 -- Method: Telnet.msg (msg[, *args])
     Print a debug message when the debug level is `>' 0. If extra
     arguments are present, they are substituted in the message using
     the standard string formatting operator.

 -- Method: Telnet.set_debuglevel (debuglevel)
     Set the debug level.  The higher the value of _debuglevel_, the
     more debug output you get (on `sys.stdout').

 -- Method: Telnet.close ()
     Close the connection.

 -- Method: Telnet.get_socket ()
     Return the socket object used internally.

 -- Method: Telnet.fileno ()
     Return the file descriptor of the socket object used internally.

 -- Method: Telnet.write (buffer)
     Write a string to the socket, doubling any IAC characters. This
     can block if the connection is blocked.  May raise
     socket.error if the connection is closed.

 -- Method: Telnet.interact ()
     Interaction function, emulates a very dumb Telnet client.

 -- Method: Telnet.mt_interact ()
     Multithreaded version of interact().

 -- Method: Telnet.expect (list[, timeout])
     Read until one from a list of a regular expressions matches.

     The first argument is a list of regular expressions, either
     compiled (see re.RegexObject instances) or uncompiled
     (strings). The optional second argument is a timeout, in seconds;
     the default is to block indefinitely.

     Return a tuple of three items: the index in the list of the first
     regular expression that matches; the match object returned; and
     the text read up till and including the match.

     If end of file is found and no text was read, raise
     EOFError.  Otherwise, when nothing matches, return `(-1,
     None, text)' where _text_ is the text received so far (may be the
     empty string if a timeout happened).

     If a regular expression ends with a greedy match (such as `.*') or
     if more than one expression can match the same input, the results
     are non-deterministic, and may depend on the I/O timing.

 -- Method: Telnet.set_option_negotiation_callback (callback)
     Each time a telnet option is read on the input flow, this
     _callback_ (if set) is called with the following parameters :
     callback(telnet socket, command (DO/DONT/WILL/WONT), option).  No
     other action is done afterwards by telnetlib.

5.20.14.2 Telnet Example
........................

A simple example illustrating typical use:

    import getpass
    import sys
    import telnetlib

    HOST = "localhost"
    user = raw_input("Enter your remote account: ")
    password = getpass.getpass()

    tn = telnetlib.Telnet(HOST)

    tn.read_until("login: ")
    tn.write(user + "\n")
    if password:
        tn.read_until("Password: ")
        tn.write(password + "\n")

    tn.write("ls\n")
    tn.write("exit\n")

    print tn.read_all()

5.20.15 `uuid' -- UUID objects according to RFC 4122
----------------------------------------------------

New in version 2.5.

  This module provides immutable UUID objects (the
UUID class) and the functions uuid1(),
uuid3(), uuid4(), uuid5() for generating
version 1, 3, 4, and 5 UUIDs as specified in RFC 4122
(http://tools.ietf.org/html/rfc4122.html).

  If all you want is a unique ID, you should probably call
uuid1() or uuid4().  Note that uuid1().
may compromise privacy since it creates a UUID containing the
computer's network address.  uuid4() creates a random UUID.

 -- Class: uuid.UUID ([hex[, bytes[, bytes_le[, fields[, int[,
          version]]]]]])
     Create a UUID from either a string of 32 hexadecimal digits, a
     string of 16 bytes as the _bytes_ argument, a string of 16 bytes
     in little-endian order as the _bytes_le_ argument, a tuple of six
     integers (32-bit _time_low_, 16-bit _time_mid_, 16-bit
     _time_hi_version_, 8-bit _clock_seq_hi_variant_, 8-bit
     _clock_seq_low_, 48-bit _node_) as the _fields_ argument, or a
     single 128-bit integer as the _int_ argument.  When a string of
     hex digits is given, curly braces, hyphens, and a URN prefix are
     all optional.  For example, these expressions all yield the same
     UUID:

         UUID('{12345678-1234-5678-1234-567812345678}')
         UUID('12345678123456781234567812345678')
         UUID('urn:uuid:12345678-1234-5678-1234-567812345678')
         UUID(bytes='\x12\x34\x56\x78'*4)
         UUID(bytes_le='\x78\x56\x34\x12\x34\x12\x78\x56' +
                       '\x12\x34\x56\x78\x12\x34\x56\x78')
         UUID(fields=(0x12345678, 0x1234, 0x5678, 0x12, 0x34, 0x567812345678))
         UUID(int=0x12345678123456781234567812345678)

     Exactly one of _hex_, _bytes_, _bytes_le_, _fields_, or _int_ must
     be given.  The _version_ argument is optional; if given, the
     resulting UUID will have its variant and version number set
     according to RFC 4122, overriding bits in the given _hex_,
     _bytes_, _bytes_le_, _fields_, or _int_.

  UUID instances have these read-only attributes:

 -- Attribute: UUID.bytes
     The UUID as a 16-byte string (containing the six integer fields in
     big-endian byte order).

 -- Attribute: UUID.bytes_le
     The UUID as a 16-byte string (with _time_low_, _time_mid_, and
     _time_hi_version_ in little-endian byte order).

 -- Attribute: UUID.fields
     A tuple of the six integer fields of the UUID, which are also
     available as six individual attributes and two derived attributes:

     Field                              Meaning
     -----------------------------------------------------------------------
     `time_low'                         the first 32 bits of the UUID
     `time_mid'                         the next 16 bits of the UUID
     `time_hi_version'                  the next 16 bits of the UUID
     `clock_seq_hi_variant'             the next 8 bits of the UUID
     `clock_seq_low'                    the next 8 bits of the UUID
     `node'                             the last 48 bits of the UUID
     time.                   the 60-bit timestamp
     `clock_seq'                        the 14-bit sequence number

 -- Attribute: UUID.hex
     The UUID as a 32-character hexadecimal string.

 -- Attribute: UUID.int
     The UUID as a 128-bit integer.

 -- Attribute: UUID.urn
     The UUID as a URN as specified in RFC 4122.

 -- Attribute: UUID.variant
     The UUID variant, which determines the internal layout of the
     UUID. This will be one of the integer constants
     RESERVED_NCS, RFC_4122,
     RESERVED_MICROSOFT, or RESERVED_FUTURE.

 -- Attribute: UUID.version
     The UUID version number (1 through 5, meaningful only when the
     variant is RFC_4122).

  The uuid module defines the following functions:

 -- Function: uuid.getnode ()
     Get the hardware address as a 48-bit positive integer.  The first
     time this runs, it may launch a separate program, which could be
     quite slow.  If all attempts to obtain the hardware address fail,
     we choose a random 48-bit number with its eighth bit set to 1 as
     recommended in RFC 4122.  "Hardware address" means the MAC address
     of a network interface, and on a machine with multiple network
     interfaces the MAC address of any one of them may be returned.

 -- Function: uuid.uuid1 ([node[, clock_seq]])
     Generate a UUID from a host ID, sequence number, and the current
     time. If _node_ is not given, see getnode() is used to
     obtain the hardware address. If _clock_seq_ is given, it is used
     as the sequence number; otherwise a random 14-bit sequence number
     is chosen.

 -- Function: uuid.uuid3 (namespace, name)
     Generate a UUID based on the MD5 hash of a namespace identifier
     (which is a UUID) and a name (which is a string).

 -- Function: uuid.uuid4 ()
     Generate a random UUID.

 -- Function: uuid.uuid5 (namespace, name)
     Generate a UUID based on the SHA-1 hash of a namespace identifier
     (which is a UUID) and a name (which is a string).

  The uuid module defines the following namespace
identifiers for use with uuid3() or uuid5().

 -- Data: uuid.NAMESPACE_DNS
     When this namespace is specified, the _name_ string is a
     fully-qualified domain name.

 -- Data: uuid.NAMESPACE_URL
     When this namespace is specified, the _name_ string is a URL.

 -- Data: uuid.NAMESPACE_OID
     When this namespace is specified, the _name_ string is an ISO OID.

 -- Data: uuid.NAMESPACE_X500
     When this namespace is specified, the _name_ string is an X.500 DN
     in DER or a text output format.

  The uuid module defines the following constants for the
possible values of the `variant' attribute:

 -- Data: uuid.RESERVED_NCS
     Reserved for NCS compatibility.

 -- Data: uuid.RFC_4122
     Specifies the UUID layout given in RFC 4122
     (http://tools.ietf.org/html/rfc4122.html).

 -- Data: uuid.RESERVED_MICROSOFT
     Reserved for Microsoft compatibility.

 -- Data: uuid.RESERVED_FUTURE
     Reserved for future definition.

RFC 4122 (http://tools.ietf.org/html/rfc4122.html) - A Universally Unique IDentifier (UUID) URN Namespace
     This specification defines a Uniform Resource Name namespace for
     UUIDs, the internal format of UUIDs, and methods of generating
     UUIDs.

5.20.15.1 Example
.................

Here are some examples of typical usage of the uuid module:

    >>> import uuid

    # make a UUID based on the host ID and current time
    >>> uuid.uuid1()
    UUID('a8098c1a-f86e-11da-bd1a-00112444be1e')

    # make a UUID using an MD5 hash of a namespace UUID and a name
    >>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org')
    UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e')

    # make a random UUID
    >>> uuid.uuid4()
    UUID('16fd2706-8baf-433b-82eb-8c7fada847da')

    # make a UUID using a SHA-1 hash of a namespace UUID and a name
    >>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org')
    UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d')

    # make a UUID from a string of hex digits (braces and hyphens ignored)
    >>> x = uuid.UUID('{00010203-0405-0607-0809-0a0b0c0d0e0f}')

    # convert a UUID to a string of hex digits in standard form
    >>> str(x)
    '00010203-0405-0607-0809-0a0b0c0d0e0f'

    # get the raw 16 bytes of the UUID
    >>> x.bytes
    '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f'

    # make a UUID from a 16-byte string
    >>> uuid.UUID(bytes=x.bytes)
    UUID('00010203-0405-0607-0809-0a0b0c0d0e0f')

5.20.16 `urlparse' -- Parse URLs into components
------------------------------------------------

     Note: The urlparse module is renamed to `urllib.parse'
     in Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

This module defines a standard interface to break Uniform Resource
Locator (URL) strings up in components (addressing scheme, network
location, path etc.), to combine the components back into a URL string,
and to convert a "relative URL" to an absolute URL given a "base URL."

  The module has been designed to match the Internet RFC on Relative
Uniform Resource Locators (and discovered a bug in an earlier draft!).
It supports the following URL schemes: `file', `ftp', `gopher', `hdl',
`http', `https', `imap', `mailto', `mms', `news',  `nntp', `prospero',
`rsync', `rtsp', `rtspu',  `sftp', `shttp', `sip', `sips', `snews',
`svn',  `svn+ssh', `telnet', `wais'.

  New in version 2.5: Support for the `sftp' and `sips' schemes.

  Latest version of the urlparse module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/urlparse.py?view=markup)The
urlparse module defines the following functions:

 -- Function: urlparse.urlparse (urlstring[, scheme[, allow_fragments]])
     Parse a URL into six components, returning a 6-tuple.  This
     corresponds to the general structure of a URL:
     `scheme://netloc/path;parameters?query#fragment'.  Each tuple item
     is a string, possibly empty. The components are not broken up in
     smaller parts (for example, the network location is a single
     string), and % escapes are not expanded. The delimiters as shown
     above are not part of the result, except for a leading slash in
     the _path_ component, which is retained if present.  For example:

         >>> from urlparse import urlparse
         >>> o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html')
         >>> o   # doctest: +NORMALIZE_WHITESPACE
         ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
                     params='', query='', fragment='')
         >>> o.scheme
         'http'
         >>> o.port
         80
         >>> o.geturl()
         'http://www.cwi.nl:80/%7Eguido/Python.html'

     Following the syntax specifications in RFC 1808
     (http://tools.ietf.org/html/rfc1808.html), urlparse recognizes a
     netloc only if it is properly introduced by '//'.  Otherwise the
     input is presumed to be a relative URL and thus to start with a
     path component.

         >>> from urlparse import urlparse
         >>> urlparse('//www.cwi.nl:80/%7Eguido/Python.html')
         ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
                    params='', query='', fragment='')
         >>> urlparse('www.cwi.nl:80/%7Eguido/Python.html')
         ParseResult(scheme='', netloc='', path='www.cwi.nl:80/%7Eguido/Python.html',
                    params='', query='', fragment='')
         >>> urlparse('help/Python.html')
         ParseResult(scheme='', netloc='', path='help/Python.html', params='',
                    query='', fragment='')

     If the _scheme_ argument is specified, it gives the default
     addressing scheme, to be used only if the URL does not specify
     one.  The default value for this argument is the empty string.

     If the _allow_fragments_ argument is false, fragment identifiers
     are not allowed, even if the URL's addressing scheme normally does
     support them.  The default value for this argument is True.

     The return value is actually an instance of a subclass of
     tuple.  This class has the following additional read-only
     convenience attributes:

     Attribute              Index       Value                          Value if not present
     ---------------------------------------------------------------------------------------------
     `scheme'               0           URL scheme specifier           empty string
     `netloc'               1           Network location part          empty string
     `path'                 2           Hierarchical path              empty string
     `params'               3           Parameters for last path       empty string
                                        element
     `query'                4           Query component                empty string
     `fragment'             5           Fragment identifier            empty string
     `username'                         User name                      None.
     `password'                         Password                       None.
     `hostname'                         Host name (lower case)         None.
     `port'                             Port number as integer, if     None.
                                        present

     See section Results of urlparse() and urlsplit() for
     more information on the result object.

     Changed in version 2.5: Added attributes to return value.

     Changed in version 2.7: Added IPv6 URL parsing capabilities.

 -- Function: urlparse.parse_qs (qs[, keep_blank_values[,
          strict_parsing]])
     Parse a query string given as a string argument (data of type
     `application/x-www-form-urlencoded').  Data are returned as a
     dictionary.  The dictionary keys are the unique query variable
     names and the values are lists of values for each name.

     The optional argument _keep_blank_values_ is a flag indicating
     whether blank values in percent-encoded queries should be treated
     as blank strings.   A true value indicates that blanks should be
     retained as  blank strings.  The default false value indicates
     that blank values are to be ignored and treated as if they were
     not included.

     The optional argument _strict_parsing_ is a flag indicating what
     to do with parsing errors.  If false (the default), errors are
     silently ignored.  If true, errors raise a ValueError.
     exception.

     Use the urllib.urlencode() function to convert such
     dictionaries into query strings.

     New in version 2.6: Copied from the cgi module.

 -- Function: urlparse.parse_qsl (qs[, keep_blank_values[,
          strict_parsing]])
     Parse a query string given as a string argument (data of type
     `application/x-www-form-urlencoded').  Data are returned as a list
     of name, value pairs.

     The optional argument _keep_blank_values_ is a flag indicating
     whether blank values in percent-encoded queries should be treated
     as blank strings.   A true value indicates that blanks should be
     retained as  blank strings.  The default false value indicates
     that blank values are to be ignored and treated as if they were
     not included.

     The optional argument _strict_parsing_ is a flag indicating what
     to do with parsing errors.  If false (the default), errors are
     silently ignored.  If true, errors raise a ValueError.
     exception.

     Use the urllib.urlencode() function to convert such
     lists of pairs into query strings.

     New in version 2.6: Copied from the cgi module.

 -- Function: urlparse.urlunparse (parts)
     Construct a URL from a tuple as returned by `urlparse()'. The
     _parts_ argument can be any six-item iterable. This may result in
     a slightly different, but equivalent URL, if the URL that was
     parsed originally had unnecessary delimiters (for example, a ?
     with an empty query; the RFC states that these are equivalent).

 -- Function: urlparse.urlsplit (urlstring[, scheme[, allow_fragments]])
     This is similar to urlparse(), but does not split the
     params from the URL.  This should generally be used instead of
     urlparse() if the more recent URL syntax allowing
     parameters to be applied to each segment of the _path_ portion of
     the URL (see RFC 2396 (http://tools.ietf.org/html/rfc2396.html))
     is wanted.  A separate function is needed to separate the path
     segments and parameters.  This function returns a 5-tuple:
     (addressing scheme, network location, path, query, fragment
     identifier).

     The return value is actually an instance of a subclass of
     tuple.  This class has the following additional read-only
     convenience attributes:

     Attribute              Index       Value                         Value if not present
     --------------------------------------------------------------------------------------------
     `scheme'               0           URL scheme specifier          empty string
     `netloc'               1           Network location part         empty string
     `path'                 2           Hierarchical path             empty string
     `query'                3           Query component               empty string
     `fragment'             4           Fragment identifier           empty string
     `username'                         User name                     None.
     `password'                         Password                      None.
     `hostname'                         Host name (lower case)        None.
     `port'                             Port number as integer, if    None.
                                        present

     See section Results of urlparse() and urlsplit() for
     more information on the result object.

     New in version 2.2.

     Changed in version 2.5: Added attributes to return value.

 -- Function: urlparse.urlunsplit (parts)
     Combine the elements of a tuple as returned by urlsplit()
     into a complete URL as a string. The _parts_ argument can be
     any five-item iterable. This may result in a slightly different,
     but equivalent URL, if the URL that was parsed originally had
     unnecessary delimiters (for example, a ? with an empty query; the
     RFC states that these are equivalent).

     New in version 2.2.

 -- Function: urlparse.urljoin (base, url[, allow_fragments])
     Construct a full ("absolute") URL by combining a "base URL"
     (_base_) with another URL (_url_).  Informally, this uses
     components of the base URL, in particular the addressing scheme,
     the network location and (part of) the path, to provide missing
     components in the relative URL.  For example:

         >>> from urlparse import urljoin
         >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html')
         'http://www.cwi.nl/%7Eguido/FAQ.html'

     The _allow_fragments_ argument has the same meaning and default as
     for urlparse().

          Note: If _url_ is an absolute URL (that is, starting with
          `//' or `scheme://'), the _url_'s host name and/or scheme
          will be present in the result.  For example:

         >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html',
         ...         '//www.python.org/%7Eguido')
         'http://www.python.org/%7Eguido'

     If you do not want that behavior, preprocess the _url_ with
     urlsplit() and urlunsplit(), removing possible
     _scheme_ and _netloc_ parts.

 -- Function: urlparse.urldefrag (url)
     If _url_ contains a fragment identifier, returns a modified
     version of _url_ with no fragment identifier, and the fragment
     identifier as a separate string.  If there is no fragment
     identifier in _url_, returns _url_ unmodified and an empty string.

RFC 3986 (http://tools.ietf.org/html/rfc3986.html) - Uniform Resource Identifiers
     This is the current standard (STD66). Any changes to urlparse
     module should conform to this. Certain deviations could be
     observed, which are mostly due backward compatiblity purposes and
     for certain de-facto parsing requirements as commonly observed in
     major browsers.

RFC 2732 (http://tools.ietf.org/html/rfc2732.html) - Format for Literal IPv6 Addresses in URL's.
     This specifies the parsing requirements of IPv6 URLs.

RFC 2396 (http://tools.ietf.org/html/rfc2396.html) - Uniform Resource Identifiers (URI): Generic Syntax
     Document describing the generic syntactic requirements for both
     Uniform Resource Names (URNs) and Uniform Resource Locators (URLs).

RFC 2368 (http://tools.ietf.org/html/rfc2368.html) - The mailto URL scheme.
     Parsing requirements for mailto url schemes.

RFC 1808 (http://tools.ietf.org/html/rfc1808.html) - Relative Uniform Resource Locators
     This Request For Comments includes the rules for joining an
     absolute and a relative URL, including a fair number of "Abnormal
     Examples" which govern the treatment of border cases.

RFC 1738 (http://tools.ietf.org/html/rfc1738.html) - Uniform Resource Locators (URL)
     This specifies the formal syntax and semantics of absolute URLs.

5.20.16.1 Results of `urlparse()' and `urlsplit()'
..................................................

The result objects from the urlparse() and
urlsplit() functions are subclasses of the tuple.
type.  These subclasses add the attributes described in those
functions, as well as provide an additional method:

 -- Method: ParseResult.geturl ()
     Return the re-combined version of the original URL as a string.
     This may differ from the original URL in that the scheme will
     always be normalized to lower case and empty components may be
     dropped. Specifically, empty parameters, queries, and fragment
     identifiers will be removed.

     The result of this method is a fixpoint if passed back through the
     original parsing function:

         >>> import urlparse
         >>> url = 'HTTP://www.Python.org/doc/#'

         >>> r1 = urlparse.urlsplit(url)
         >>> r1.geturl()
         'http://www.Python.org/doc/'

         >>> r2 = urlparse.urlsplit(r1.geturl())
         >>> r2.geturl()
         'http://www.Python.org/doc/'

     New in version 2.5.

  The following classes provide the implementations of the parse
results:

 -- Class: urlparse.BaseResult
     Base class for the concrete result classes.  This provides most of
     the attribute definitions.  It does not provide a `geturl()'
     method.  It is derived from tuple, but does not
     override the __init__() or __new__().
     methods.

 -- Class: urlparse.ParseResult (scheme, netloc, path, params, query,
          fragment)
     Concrete class for urlparse() results.  The
     __new__() method is overridden to support checking that the
     right number of arguments are passed.

 -- Class: urlparse.SplitResult (scheme, netloc, path, query, fragment)
     Concrete class for urlsplit() results.  The
     __new__() method is overridden to support checking that the
     right number of arguments are passed.

5.20.17 `SocketServer' -- A framework for network servers
---------------------------------------------------------

     Note: The SocketServer module has been renamed to
     `socketserver' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

The SocketServer module simplifies the task of writing
network servers.

  There are four basic server classes: `TCPServer' uses the Internet TCP
protocol, which provides for continuous streams of data between the
client and server.  `UDPServer' uses datagrams, which are discrete
packets of information that may arrive out of order or be lost while in
transit.  The more infrequently used `UnixStreamServer' and
`UnixDatagramServer' classes are similar, but use Unix domain sockets;
they're not available on non-Unix platforms.  For more details on
network programming, consult a book such as W. Richard Steven's UNIX
Network Programming or Ralph Davis's Win32 Network Programming.

  These four classes process requests _synchronously_; each request
must be completed before the next request can be started.  This isn't
suitable if each request takes a long time to complete, because it
requires a lot of computation, or because it returns a lot of data
which the client is slow to process.  The solution is to create a
separate process or thread to handle each request; the `ForkingMixIn'
and `ThreadingMixIn' mix-in classes can be used to support asynchronous
behaviour.

  Creating a server requires several steps.  First, you must create a
request handler class by subclassing the `BaseRequestHandler' class and
overriding its `handle()' method; this method will process incoming
requests.  Second, you must instantiate one of the server classes,
passing it the server's address and the request handler class.
Finally, call the `handle_request()' or `serve_forever()' method of the
server object to process one or many requests.

  When inheriting from `ThreadingMixIn' for threaded connection
behavior, you should explicitly declare how you want your threads to
behave on an abrupt shutdown. The `ThreadingMixIn' class defines an
attribute _daemon_threads_, which indicates whether or not the server
should wait for thread termination. You should set the flag explicitly
if you would like threads to behave autonomously; the default is
False, meaning that Python will not exit until all threads created
by `ThreadingMixIn' have exited.

  Server classes have the same external methods and attributes, no
matter what network protocol they use.

5.20.17.1 Server Creation Notes
...............................

There are five classes in an inheritance diagram, four of which
represent synchronous servers of four types:

    +------------+
    | BaseServer |
    +------------+
          |
          v
    +-----------+        +------------------+
    | TCPServer |------->| UnixStreamServer |
    +-----------+        +------------------+
          |
          v
    +-----------+        +--------------------+
    | UDPServer |------->| UnixDatagramServer |
    +-----------+        +--------------------+

Note that `UnixDatagramServer' derives from `UDPServer', not from
`UnixStreamServer' -- the only difference between an IP and a Unix
stream server is the address family, which is simply repeated in both
Unix server classes.

  Forking and threading versions of each type of server can be created
using the `ForkingMixIn' and `ThreadingMixIn' mix-in classes.  For
instance, a threading UDP server class is created as follows:

    class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass

The mix-in class must come first, since it overrides a method defined in
`UDPServer'.  Setting the various member variables also changes the
behavior of the underlying server mechanism.

  To implement a service, you must derive a class from
`BaseRequestHandler' and redefine its `handle()' method.  You can then
run various versions of the service by combining one of the server
classes with your request handler class.  The request handler class
must be different for datagram or stream services.  This can be hidden
by using the handler subclasses `StreamRequestHandler' or
`DatagramRequestHandler'.

  Of course, you still have to use your head!  For instance, it makes
no sense to use a forking server if the service contains state in
memory that can be modified by different requests, since the
modifications in the child process would never reach the initial state
kept in the parent process and passed to each child.  In this case, you
can use a threading server, but you will probably have to use locks to
protect the integrity of the shared data.

  On the other hand, if you are building an HTTP server where all data
is stored externally (for instance, in the file system), a synchronous
class will essentially render the service "deaf" while one request is
being handled - which may be for a very long time if a client is slow
to receive all the data it has requested.  Here a threading or forking
server is appropriate.

  In some cases, it may be appropriate to process part of a request
synchronously, but to finish processing in a forked child depending on
the request data.  This can be implemented by using a synchronous
server and doing an explicit fork in the request handler class
`handle()' method.

  Another approach to handling multiple simultaneous requests in an
environment that supports neither threads nor `fork()' (or where these
are too expensive or inappropriate for the service) is to maintain an
explicit table of partially finished requests and to use
select() to decide which request to work on next (or whether to
handle a new incoming request).  This is particularly important for
stream services where each client can potentially be connected for a
long time (if threads or subprocesses cannot be used). See
asyncore for another way to manage this.

5.20.17.2 Server Objects
........................

 -- Class: SocketServer.BaseServer
     This is the superclass of all Server objects in the module.  It
     defines the interface, given below, but does not implement most of
     the methods, which is done in subclasses.

 -- Method: BaseServer.fileno ()
     Return an integer file descriptor for the socket on which the
     server is listening.  This function is most commonly passed to
     select.select(), to allow monitoring multiple servers
     in the same process.

 -- Method: BaseServer.handle_request ()
     Process a single request.  This function calls the following
     methods in order: get_request(),
     verify_request(), and process_request().  If the
     user-provided `handle()' method of the handler class raises an
     exception, the server's handle_error() method will be
     called.  If no request is received within `self.timeout' seconds,
     see handle_timeout() will be called and
     handle_request().  will return.

 -- Method: BaseServer.serve_forever (poll_interval=0.5)
     Handle requests until an explicit shutdown() request.
     Polls for shutdown every _poll_interval_ seconds.

 -- Method: BaseServer.shutdown ()
     Tells the serve_forever() loop to stop and waits
     until it does.

     New in version 2.6.

 -- Attribute: BaseServer.address_family
     The family of protocols to which the server's socket belongs.
     Common examples are socket.AF_INET and
     socket.AF_UNIX.

 -- Attribute: BaseServer.RequestHandlerClass
     The user-provided request handler class; an instance of this class
     is created for each request.

 -- Attribute: BaseServer.server_address
     The address on which the server is listening.  The format of
     addresses varies depending on the protocol family; see the
     documentation for the socket module for details.  For Internet
     protocols, this is a tuple containing a string giving the address,
     and an integer port number: `('127.0.0.1', 80)', for example.

 -- Attribute: BaseServer.socket
     The socket object on which the server will listen for incoming
     requests.

  The server classes support the following class variables:

 -- Attribute: BaseServer.allow_reuse_address
     Whether the server will allow the reuse of an address. This
     defaults to False, and can be set in subclasses to
     change the policy.

 -- Attribute: BaseServer.request_queue_size
     The size of the request queue.  If it takes a long time to process
     a single request, any requests that arrive while the server is
     busy are placed into a queue, up to request_queue_size
     requests.  Once the queue is full, further requests from
     clients will get a "Connection denied" error.  The default value
     is usually 5, but this can be overridden by subclasses.

 -- Attribute: BaseServer.socket_type
     The type of socket used by the server; see socket.SOCK_STREAM
     and socket.SOCK_DGRAM are two common values.

 -- Attribute: BaseServer.timeout
     Timeout duration, measured in seconds, or None if no
     timeout is desired.  If handle_request() receives no
     incoming requests within the timeout period, the
     handle_timeout() method is called.

  There are various server methods that can be overridden by subclasses
of base server classes like `TCPServer'; these methods aren't useful to
external users of the server object.

 -- Method: BaseServer.finish_request ()
     Actually processes the request by instantiating
     RequestHandlerClass and calling its `handle()' method.

 -- Method: BaseServer.get_request ()
     Must accept a request from the socket, and return a 2-tuple
     containing the _new_ socket object to be used to communicate with
     the client, and the client's address.

 -- Method: BaseServer.handle_error (request, client_address)
     This function is called if the RequestHandlerClass.'s
     `handle()' method raises an exception.  The default action is to
     print the traceback to standard output and continue handling
     further requests.

 -- Method: BaseServer.handle_timeout ()
     This function is called when the timeout attribute has
     been set to a value other than None and the timeout
     period has passed with no requests being received.  The default
     action for forking servers is to collect the status of any child
     processes that have exited, while in threading servers this method
     does nothing.

 -- Method: BaseServer.process_request (request, client_address)
     Calls finish_request() to create an instance of the
     RequestHandlerClass.  If desired, this function can
     create a new process or thread to handle the request; the
     `ForkingMixIn' and `ThreadingMixIn' classes do this.

 -- Method: BaseServer.server_activate ()
     Called by the server's constructor to activate the server.  The
     default behavior just `listen()'s to the server's socket. May be
     overridden.

 -- Method: BaseServer.server_bind ()
     Called by the server's constructor to bind the socket to the
     desired address.  May be overridden.

 -- Method: BaseServer.verify_request (request, client_address)
     Must return a Boolean value; if the value is True, the
     request will be processed, and if it's False, the
     request will be denied. This function can be overridden to
     implement access controls for a server. The default implementation
     always returns True.

5.20.17.3 RequestHandler Objects
................................

The request handler class must define a new `handle()' method, and can
override any of the following methods.  A new instance is created for
each request.

 -- Method: RequestHandler.finish ()
     Called after the handle() method to perform any
     clean-up actions required.  The default implementation does
     nothing.  If setup() or handle() raise
     an exception, this function will not be called.

 -- Method: RequestHandler.handle ()
     This function must do all the work required to service a request.
     The default implementation does nothing.  Several instance
     attributes are available to it; the request is available as
     `self.request'; the client address as `self.client_address'; and
     the server instance as `self.server', in case it needs access to
     per-server information.

     The type of `self.request' is different for datagram or stream
     services.  For stream services, `self.request' is a socket object;
     for datagram services, `self.request' is a pair of string and
     socket.  However, this can be hidden by using the request handler
     subclasses `StreamRequestHandler' or `DatagramRequestHandler',
     which override the setup() and finish().
     methods, and provide `self.rfile' and `self.wfile' attributes.
     `self.rfile' and `self.wfile' can be read or written,
     respectively, to get the request data or return data to the client.

 -- Method: RequestHandler.setup ()
     Called before the handle() method to perform any
     initialization actions required.  The default implementation does
     nothing.

5.20.17.4 Examples
..................

5.20.17.5 `SocketServer.TCPServer' Example
..........................................

This is the server side:

    import SocketServer

    class MyTCPHandler(SocketServer.BaseRequestHandler):
        """
        The RequestHandler class for our server.

        It is instantiated once per connection to the server, and must
        override the handle() method to implement communication to the
        client.
        """

        def handle(self):
            # self.request is the TCP socket connected to the client
            self.data = self.request.recv(1024).strip()
            print "%s wrote:" % self.client_address[0]
            print self.data
            # just send back the same data, but upper-cased
            self.request.send(self.data.upper())

    if __name__ == "__main__":
        HOST, PORT = "localhost", 9999

        # Create the server, binding to localhost on port 9999
        server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)

        # Activate the server; this will keep running until you
        # interrupt the program with Ctrl-C
        server.serve_forever()

An alternative request handler class that makes use of streams
(file-like objects that simplify communication by providing the
standard file interface):

    class MyTCPHandler(SocketServer.StreamRequestHandler):

        def handle(self):
            # self.rfile is a file-like object created by the handler;
            # we can now use e.g. readline() instead of raw recv() calls
            self.data = self.rfile.readline().strip()
            print "%s wrote:" % self.client_address[0]
            print self.data
            # Likewise, self.wfile is a file-like object used to write back
            # to the client
            self.wfile.write(self.data.upper())

The difference is that the `readline()' call in the second handler will
call `recv()' multiple times until it encounters a newline character,
while the single `recv()' call in the first handler will just return
what has been sent from the client in one `send()' call.

  This is the client side:

    import socket
    import sys

    HOST, PORT = "localhost", 9999
    data = " ".join(sys.argv[1:])

    # Create a socket (SOCK_STREAM means a TCP socket)
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect to server and send data
    sock.connect((HOST, PORT))
    sock.send(data + "\n")

    # Receive data from the server and shut down
    received = sock.recv(1024)
    sock.close()

    print "Sent:     %s" % data
    print "Received: %s" % received

The output of the example should look something like this:

  Server:

    $ python TCPServer.py
    127.0.0.1 wrote:
    hello world with TCP
    127.0.0.1 wrote:
    python is nice

Client:

    $ python TCPClient.py hello world with TCP
    Sent:     hello world with TCP
    Received: HELLO WORLD WITH TCP
    $ python TCPClient.py python is nice
    Sent:     python is nice
    Received: PYTHON IS NICE

5.20.17.6 `SocketServer.UDPServer' Example
..........................................

This is the server side:

    import SocketServer

    class MyUDPHandler(SocketServer.BaseRequestHandler):
        """
        This class works similar to the TCP handler class, except that
        self.request consists of a pair of data and client socket, and since
        there is no connection the client address must be given explicitly
        when sending data back via sendto().
        """

        def handle(self):
            data = self.request[0].strip()
            socket = self.request[1]
            print "%s wrote:" % self.client_address[0]
            print data
            socket.sendto(data.upper(), self.client_address)

    if __name__ == "__main__":
        HOST, PORT = "localhost", 9999
        server = SocketServer.UDPServer((HOST, PORT), MyUDPHandler)
        server.serve_forever()

This is the client side:

    import socket
    import sys

    HOST, PORT = "localhost", 9999
    data = " ".join(sys.argv[1:])

    # SOCK_DGRAM is the socket type to use for UDP sockets
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

    # As you can see, there is no connect() call; UDP has no connections.
    # Instead, data is directly sent to the recipient via sendto().
    sock.sendto(data + "\n", (HOST, PORT))
    received = sock.recv(1024)

    print "Sent:     %s" % data
    print "Received: %s" % received

The output of the example should look exactly like for the TCP server
example.

5.20.17.7 Asynchronous Mixins
.............................

To build asynchronous handlers, use the `ThreadingMixIn' and
`ForkingMixIn' classes.

  An example for the `ThreadingMixIn' class:

    import socket
    import threading
    import SocketServer

    class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):

        def handle(self):
            data = self.request.recv(1024)
            cur_thread = threading.currentThread()
            response = "%s: %s" % (cur_thread.getName(), data)
            self.request.send(response)

    class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
        pass

    def client(ip, port, message):
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.connect((ip, port))
        sock.send(message)
        response = sock.recv(1024)
        print "Received: %s" % response
        sock.close()

    if __name__ == "__main__":
        # Port 0 means to select an arbitrary unused port
        HOST, PORT = "localhost", 0

        server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
        ip, port = server.server_address

        # Start a thread with the server -- that thread will then start one
        # more thread for each request
        server_thread = threading.Thread(target=server.serve_forever)
        # Exit the server thread when the main thread terminates
        server_thread.setDaemon(True)
        server_thread.start()
        print "Server loop running in thread:", server_thread.getName()

        client(ip, port, "Hello World 1")
        client(ip, port, "Hello World 2")
        client(ip, port, "Hello World 3")

        server.shutdown()

The output of the example should look something like this:

    $ python ThreadedTCPServer.py
    Server loop running in thread: Thread-1
    Received: Thread-2: Hello World 1
    Received: Thread-3: Hello World 2
    Received: Thread-4: Hello World 3

The `ForkingMixIn' class is used in the same way, except that the server
will spawn a new process for each request.

5.20.18 `BaseHTTPServer' -- Basic HTTP server
---------------------------------------------

     Note: The BaseHTTPServer module has been merged into
     `http.server' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

This module defines two classes for implementing HTTP servers (Web
servers).  Usually, this module isn't used directly, but is used as a
basis for building functioning Web servers. See the
SimpleHTTPServer and CGIHTTPServer modules.

  The first class, see HTTPServer, is a `SocketServer.TCPServer'
subclass, and therefore implements the SocketServer.BaseServer.
interface.  It creates and listens at the HTTP socket,
dispatching the requests to a handler.  Code to create and run the
server looks like this:

    def run(server_class=BaseHTTPServer.HTTPServer,
            handler_class=BaseHTTPServer.BaseHTTPRequestHandler):
        server_address = ('', 8000)
        httpd = server_class(server_address, handler_class)
        httpd.serve_forever()

 -- Class: BaseHTTPServer.HTTPServer (server_address,
          RequestHandlerClass)
     This class builds on the `TCPServer' class by storing the server
     address as instance variables named `server_name' and
     `server_port'. The server is accessible by the handler, typically
     through the handler's `server' instance variable.

 -- Class: BaseHTTPServer.BaseHTTPRequestHandler (request,
          client_address, server)
     This class is used to handle the HTTP requests that arrive at the
     server. By itself, it cannot respond to any actual HTTP requests;
     it must be subclassed to handle each request method (e.g. GET or
     POST). BaseHTTPRequestHandler provides a number of
     class and instance variables, and methods for use by subclasses.

     The handler will parse the request and the headers, then call a
     method specific to the request type. The method name is
     constructed from the request. For example, for the request method
     `SPAM', the `do_SPAM()' method will be called with no arguments.
     All of the relevant information is stored in instance variables of
     the handler.  Subclasses should not need to override or extend the
     __init__() method.

     BaseHTTPRequestHandler has the following instance
     variables:

      -- Attribute: client_address
          Contains a tuple of the form `(host, port)' referring to the
          client's address.

      -- Attribute: server
          Contains the server instance.

      -- Attribute: command
          Contains the command (request type). For example, `'GET''.

      -- Attribute: path
          Contains the request path.

      -- Attribute: request_version
          Contains the version string from the request. For example,
          `'HTTP/1.0''.

      -- Attribute: headers
          Holds an instance of the class specified by the
          MessageClass class variable. This instance parses and
          manages the headers in the HTTP request.

      -- Attribute: rfile
          Contains an input stream, positioned at the start of the
          optional input data.

      -- Attribute: wfile
          Contains the output stream for writing a response back to the
          client. Proper adherence to the HTTP protocol must be used
          when writing to this stream.

     BaseHTTPRequestHandler has the following class
     variables:

      -- Attribute: server_version
          Specifies the server software version.  You may want to
          override this. The format is multiple whitespace-separated
          strings, where each string is of the form name[/version]. For
          example, `'BaseHTTP/0.2''.

      -- Attribute: sys_version
          Contains the Python system version, in a form usable by the
          version_string method and the
          server_version class variable. For example,
          `'Python/1.4''.

      -- Attribute: error_message_format
          Specifies a format string for building an error response to
          the client. It uses parenthesized, keyed format specifiers,
          so the format operand must be a dictionary. The _code_ key
          should be an integer, specifying the numeric HTTP error code
          value. _message_ should be a string containing a (detailed)
          error message of what occurred, and _explain_ should be an
          explanation of the error code number. Default _message_ and
          _explain_ values can found in the _responses_ class variable.

      -- Attribute: error_content_type
          Specifies the Content-Type HTTP header of error responses
          sent to the client.  The default value is `'text/html''.

          New in version 2.6: Previously, the content type was always
          `'text/html''.

      -- Attribute: protocol_version
          This specifies the HTTP protocol version used in responses.
          If set to `'HTTP/1.1'', the server will permit HTTP
          persistent connections; however, your server _must_ then
          include an accurate `Content-Length' header (using
          send_header()) in all of its responses to clients.
          For backwards compatibility, the setting defaults to
          `'HTTP/1.0''.

      -- Attribute: MessageClass
          Specifies a rfc822.Message.-like class to parse
          HTTP headers.  Typically, this is not overridden, and it
          defaults to mimetools.Message.

      -- Attribute: responses
          This variable contains a mapping of error code integers to
          two-element tuples containing a short and long message. For
          example, `{code: (shortmessage, longmessage)}'. The
          _shortmessage_ is usually used as the _message_ key in an
          error response, and _longmessage_ as the _explain_ key (see
          the error_message_format class variable).

     A BaseHTTPRequestHandler instance has the following
     methods:

      -- Method: handle ()
          Calls handle_one_request() once (or, if
          persistent connections are enabled, multiple times) to handle
          incoming HTTP requests. You should never need to override it;
          instead, implement appropriate `do_*()' methods.

      -- Method: handle_one_request ()
          This method will parse and dispatch the request to the
          appropriate `do_*()' method.  You should never need to
          override it.

      -- Method: send_error (code[, message])
          Sends and logs a complete error reply to the client. The
          numeric _code_ specifies the HTTP error code, with _message_
          as optional, more specific text. A complete set of headers is
          sent, followed by text composed using the
          error_message_format class variable.

      -- Method: send_response (code[, message])
          Sends a response header and logs the accepted request. The
          HTTP response line is sent, followed by _Server_ and _Date_
          headers. The values for these two headers are picked up from
          the version_string() and
          date_time_string() methods, respectively.

      -- Method: send_header (keyword, value)
          Writes a specific HTTP header to the output stream. _keyword_
          should specify the header keyword, with _value_ specifying
          its value.

      -- Method: end_headers ()
          Sends a blank line, indicating the end of the HTTP headers in
          the response.

      -- Method: log_request ([code[, size]])
          Logs an accepted (successful) request. _code_ should specify
          the numeric HTTP code associated with the response. If a size
          of the response is available, then it should be passed as the
          _size_ parameter.

      -- Method: log_error (...)
          Logs an error when a request cannot be fulfilled. By default,
          it passes the message to log_message(), so it
          takes the same arguments (_format_ and additional values).

      -- Method: log_message (format, ...)
          Logs an arbitrary message to `sys.stderr'. This is typically
          overridden to create custom error logging mechanisms. The
          _format_ argument is a standard printf-style format string,
          where the additional arguments to log_message().
          are applied as inputs to the formatting. The client address
          and current date and time are prefixed to every message
          logged.

      -- Method: version_string ()
          Returns the server software's version string. This is a
          combination of the server_version and
          sys_version class variables.

      -- Method: date_time_string ([timestamp])
          Returns the date and time given by _timestamp_ (which must be
          in the format returned by time.time()), formatted
          for a message header. If _timestamp_ is omitted, it uses the
          current date and time.

          The result looks like `'Sun, 06 Nov 1994 08:49:37 GMT''.

          New in version 2.5: The _timestamp_ parameter.

      -- Method: log_date_time_string ()
          Returns the current date and time, formatted for logging.

      -- Method: address_string ()
          Returns the client address, formatted for logging. A name
          lookup is performed on the client's IP address.

5.20.18.1 More examples
.......................

To create a server that doesn't run forever, but until some condition is
fulfilled:

    def run_while_true(server_class=BaseHTTPServer.HTTPServer,
                       handler_class=BaseHTTPServer.BaseHTTPRequestHandler):
        """
        This assumes that keep_running() is a function of no arguments which
        is tested initially and after each request.  If its return value
        is true, the server continues.
        """
        server_address = ('', 8000)
        httpd = server_class(server_address, handler_class)
        while keep_running():
            httpd.handle_request()

Module CGIHTTPServer.
     Extended request handler that supports CGI scripts.

Module SimpleHTTPServer.
     Basic request handler that limits response to files actually under
     the document root.

5.20.19 `SimpleHTTPServer' -- Simple HTTP request handler
---------------------------------------------------------

     Note: The SimpleHTTPServer module has been merged into
     `http.server' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

The SimpleHTTPServer module defines a single class, see
SimpleHTTPRequestHandler, which is interface-compatible with
BaseHTTPServer.BaseHTTPRequestHandler.

  The SimpleHTTPServer module defines the following class:

 -- Class: SimpleHTTPServer.SimpleHTTPRequestHandler (request,
          client_address, server)
     This class serves files from the current directory and below,
     directly mapping the directory structure to HTTP requests.

     A lot of the work, such as parsing the request, is done by the
     base class BaseHTTPServer.BaseHTTPRequestHandler.
     This class implements the do_GET() and
     do_HEAD() functions.

     The following are defined as class-level attributes of
     SimpleHTTPRequestHandler.:

      -- Attribute: server_version

     This will be `"SimpleHTTP/" + __version__', where `__version__' is
     defined at the module level.

      -- Attribute: extensions_map
          A dictionary mapping suffixes into MIME types. The default is
          signified by an empty string, and is considered to be
          `application/octet-stream'. The mapping is used
          case-insensitively, and so should contain only lower-cased
          keys.

     The SimpleHTTPRequestHandler class defines the
     following methods:

      -- Method: do_HEAD ()
          This method serves the `'HEAD'' request type: it sends the
          headers it would send for the equivalent `GET' request. See
          the do_GET().  method for a more complete
          explanation of the possible headers.

      -- Method: do_GET ()
          The request is mapped to a local file by interpreting the
          request as a path relative to the current working directory.

          If the request was mapped to a directory, the directory is
          checked for a file named `index.html' or `index.htm' (in that
          order). If found, the file's contents are returned; otherwise
          a directory listing is generated by calling the
          `list_directory()' method. This method uses
          os.listdir() to scan the directory, and returns a `404'
          error response if the `listdir()' fails.

          If the request was mapped to a file, it is opened and the
          contents are returned.  Any IOError exception in
          opening the requested file is mapped to a `404', `'File not
          found'' error. Otherwise, the content type is guessed by
          calling the `guess_type()' method, which in turn uses the
          _extensions_map_ variable.

          A `'Content-type:'' header with the guessed content type is
          output, followed by a `'Content-Length:'' header with the
          file's size and a `'Last-Modified:'' header with the file's
          modification time.

          Then follows a blank line signifying the end of the headers,
          and then the contents of the file are output. If the file's
          MIME type starts with `text/' the file is opened in text
          mode; otherwise binary mode is used.

          The test() function in the
          SimpleHTTPServer module is an example which creates a
          server using the SimpleHTTPRequestHandler.  as
          the Handler.

          New in version 2.5: The `'Last-Modified'' header.

  The SimpleHTTPServer module can be used in the following
manner in order to set up a very basic web server serving files
relative to the current directory.

    import SimpleHTTPServer
    import SocketServer

    PORT = 8000

    Handler = SimpleHTTPServer.SimpleHTTPRequestHandler

    httpd = SocketServer.TCPServer(("", PORT), Handler)

    print "serving at port", PORT
    httpd.serve_forever()

The SimpleHTTPServer module can also be invoked directly
using the -m switch of the interpreter with a `port number'
argument.  Similar to the previous example, this serves the files
relative to the current directory.

    python -m SimpleHTTPServer 8000

Module BaseHTTPServer.
     Base class implementation for Web server and request handler.

5.20.20 `CGIHTTPServer' -- CGI-capable HTTP request handler
-----------------------------------------------------------

     Note: The CGIHTTPServer module has been merged into
     `http.server' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

The CGIHTTPServer module defines a request-handler class,
interface compatible with BaseHTTPServer.BaseHTTPRequestHandler
and inherits behavior from
SimpleHTTPServer.SimpleHTTPRequestHandler but can also run CGI
scripts.

     Note: This module can run CGI scripts on Unix and Windows systems.

     Note: CGI scripts run by the CGIHTTPRequestHandler.
     class cannot execute redirects (HTTP code 302), because code 200
     (script output follows) is sent prior to execution of the CGI
     script.  This pre-empts the status code.

  The CGIHTTPServer module defines the following class:

 -- Class: CGIHTTPServer.CGIHTTPRequestHandler (request,
          client_address, server)
     This class is used to serve either files or output of CGI scripts
     from  the current directory and below. Note that mapping HTTP
     hierarchic structure to local directory structure is exactly as in
     SimpleHTTPServer.SimpleHTTPRequestHandler.

     The class will however, run the CGI script, instead of serving it
     as a file, if it guesses it to be a CGI script. Only
     directory-based CGI are used -- the other common server
     configuration is to treat special extensions as denoting CGI
     scripts.

     The `do_GET()' and `do_HEAD()' functions are modified to run CGI
     scripts and serve the output, instead of serving files, if the
     request leads to somewhere below the `cgi_directories' path.

     The CGIHTTPRequestHandler defines the following data
     member:

      -- Attribute: cgi_directories
          This defaults to `['/cgi-bin', '/htbin']' and describes
          directories to treat as containing CGI scripts.

     The CGIHTTPRequestHandler defines the following
     methods:

      -- Method: do_POST ()
          This method serves the `'POST'' request type, only allowed
          for CGI scripts.  Error 501, "Can only POST to CGI scripts",
          is output when trying to POST to a non-CGI url.

  Note that CGI scripts will be run with UID of user nobody, for
security reasons.  Problems with the CGI script will be translated to
error 403.

  For example usage, see the implementation of the test().
function.

Module BaseHTTPServer.
     Base class implementation for Web server and request handler.

5.20.21 `cookielib' -- Cookie handling for HTTP clients
-------------------------------------------------------

     Note: The cookielib module has been renamed to
     `http.cookiejar' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

New in version 2.4.

  The cookielib module defines classes for automatic
handling of HTTP cookies.  It is useful for accessing web sites that
require small pieces of data - _cookies_ - to be set on the client
machine by an HTTP response from a web server, and then returned to the
server in later HTTP requests.

  Both the regular Netscape cookie protocol and the protocol defined by RFC
2965 (http://tools.ietf.org/html/rfc2965.html) are handled.  RFC 2965
handling is switched off by default.  RFC 2109
(http://tools.ietf.org/html/rfc2109.html) cookies are parsed as
Netscape cookies and subsequently treated either as Netscape or RFC
2965 cookies according to the 'policy' in effect.  Note that the great
majority of cookies on the Internet are Netscape cookies.
cookielib attempts to follow the de-facto Netscape cookie protocol
(which differs substantially from that set out in the original Netscape
specification), including taking note of the `max-age' and `port'
cookie-attributes introduced with RFC 2965.

     Note: The various named parameters found in `Set-Cookie' and
     `Set-Cookie2' headers (eg. `domain' and `expires') are
     conventionally referred to as _attributes_.  To distinguish them
     from Python attributes, the documentation for this module uses the
     term _cookie-attribute_ instead.

  The module defines the following exception:

 -- Exception: cookielib.LoadError
     Instances of FileCookieJar raise this exception on
     failure to load cookies from a file.

          Note: For backwards-compatibility with Python 2.4 (which
          raised an IOError), see LoadError is a
          subclass of IOError.

  The following classes are provided:

 -- Class: cookielib.CookieJar (policy=None)
     _policy_ is an object implementing the CookiePolicy.
     interface.

     The CookieJar class stores HTTP cookies.  It extracts
     cookies from HTTP requests, and returns them in HTTP responses.
     CookieJar instances automatically expire contained
     cookies when necessary.  Subclasses are also responsible for
     storing and retrieving cookies from a file or database.

 -- Class: cookielib.FileCookieJar (filename, delayload=None,
          policy=None)
     _policy_ is an object implementing the CookiePolicy.
     interface.  For the other arguments, see the documentation for the
     corresponding attributes.

     A CookieJar which can load cookies from, and perhaps
     save cookies to, a file on disk.  Cookies are *NOT* loaded from
     the named file until either the load() or
     revert() method is called.  Subclasses of this class are
     documented in section FileCookieJar subclasses and co-operation with web browsers.

 -- Class: cookielib.CookiePolicy
     This class is responsible for deciding whether each cookie should
     be accepted from / returned to the server.

 -- Class: cookielib.DefaultCookiePolicy (blocked_domains=None,
          allowed_domains=None, netscape=True, rfc2965=False,
          rfc2109_as_netscape=None, hide_cookie2=False,
          strict_domain=False, strict_rfc2965_unverifiable=True,
          strict_ns_unverifiable=False,
          strict_ns_domain=DefaultCookiePolicy.DomainLiberal,
          strict_ns_set_initial_dollar=False, strict_ns_set_path=False)
     Constructor arguments should be passed as keyword arguments only.
     _blocked_domains_ is a sequence of domain names that we never
     accept cookies from, nor return cookies to. _allowed_domains_ if
     not None, this is a sequence of the only domains for
     which we accept and return cookies.  For all other arguments, see
     the documentation for CookiePolicy and
     DefaultCookiePolicy objects.

     DefaultCookiePolicy implements the standard accept /
     reject rules for Netscape and RFC 2965 cookies.  By default, RFC
     2109 cookies (ie. cookies received in a `Set-Cookie' header with a
     version cookie-attribute of 1) are treated according to the RFC
     2965 rules.  However, if RFC 2965 handling is turned off or
     rfc2109_as_netscape is True, RFC 2109 cookies are
     'downgraded' by the CookieJar instance to Netscape
     cookies, by setting the `version' attribute of the Cookie
     instance to 0.  DefaultCookiePolicy also provides
     some parameters to allow some fine-tuning of policy.

 -- Class: cookielib.Cookie
     This class represents Netscape, RFC 2109 and RFC 2965 cookies.  It
     is not expected that users of cookielib construct their
     own Cookie.  instances.  Instead, if necessary, call
     `make_cookies()' on a CookieJar instance.

Module urllib2.
     URL opening with automatic cookie handling.

Module Cookie.
     HTTP cookie classes, principally useful for server-side code.  The
     cookielib and Cookie modules do not depend
     on each other.

<http://wp.netscape.com/newsref/std/cookie_spec.html>
     The specification of the original Netscape cookie protocol.
     Though this is still the dominant protocol, the 'Netscape cookie
     protocol' implemented by all the major browsers (and
     cookielib) only bears a passing resemblance to the one
     sketched out in `cookie_spec.html'.

RFC 2109 (http://tools.ietf.org/html/rfc2109.html) - HTTP State Management Mechanism
     Obsoleted by RFC 2965. Uses `Set-Cookie' with version=1.

RFC 2965 (http://tools.ietf.org/html/rfc2965.html) - HTTP State Management Mechanism
     The Netscape protocol with the bugs fixed.  Uses `Set-Cookie2' in
     place of `Set-Cookie'.  Not widely used.

<http://kristol.org/cookie/errata.html>
     Unfinished errata to RFC 2965.

  RFC 2964 (http://tools.ietf.org/html/rfc2964.html) - Use of HTTP
State Management

5.20.21.1 CookieJar and FileCookieJar Objects
.............................................

CookieJar objects support the iterator.
protocol for iterating over contained Cookie objects.

  CookieJar has the following methods:

 -- Method: CookieJar.add_cookie_header (request)
     Add correct `Cookie' header to _request_.

     If policy allows (ie. the `rfc2965' and `hide_cookie2' attributes
     of the CookieJar.'s CookiePolicy instance
     are true and false respectively), the `Cookie2' header is also
     added when appropriate.

     The _request_ object (usually a urllib2.Request.
     instance) must support the methods `get_full_url()', `get_host()',
     `get_type()', `unverifiable()', `get_origin_req_host()',
     `has_header()', `get_header()', `header_items()', and
     `add_unredirected_header()',as documented by urllib2.

 -- Method: CookieJar.extract_cookies (response, request)
     Extract cookies from HTTP _response_ and store them in the
     CookieJar, where allowed by policy.

     The CookieJar will look for allowable `Set-Cookie' and
     `Set-Cookie2' headers in the _response_ argument, and store cookies
     as appropriate (subject to the CookiePolicy.set_ok().
     method's approval).

     The _response_ object (usually the result of a call to
     urllib2.urlopen(), or similar) should support an `info()'
     method, which returns an object with a `getallmatchingheaders()'
     method (usually a mimetools.Message.  instance).

     The _request_ object (usually a urllib2.Request.
     instance) must support the methods `get_full_url()', `get_host()',
     `unverifiable()', and `get_origin_req_host()', as documented by
     urllib2.  The request is used to set default values for
     cookie-attributes as well as for checking that the cookie is
     allowed to be set.

 -- Method: CookieJar.set_policy (policy)
     Set the CookiePolicy instance to be used.

 -- Method: CookieJar.make_cookies (response, request)
     Return sequence of Cookie objects extracted from
     _response_ object.

     See the documentation for extract_cookies() for the
     interfaces required of the _response_ and _request_ arguments.

 -- Method: CookieJar.set_cookie_if_ok (cookie, request)
     Set a Cookie if policy says it's OK to do so.

 -- Method: CookieJar.set_cookie (cookie)
     Set a Cookie, without checking with policy to see
     whether or not it should be set.

 -- Method: CookieJar.clear ([domain[, path[, name]]])
     Clear some cookies.

     If invoked without arguments, clear all cookies.  If given a
     single argument, only cookies belonging to that _domain_ will be
     removed. If given two arguments, cookies belonging to the
     specified _domain_ and URL _path_ are removed.  If given three
     arguments, then the cookie with the specified _domain_, _path_ and
     _name_ is removed.

     Raises KeyError if no matching cookie exists.

 -- Method: CookieJar.clear_session_cookies ()
     Discard all session cookies.

     Discards all contained cookies that have a true `discard' attribute
     (usually because they had either no `max-age' or `expires'
     cookie-attribute, or an explicit `discard' cookie-attribute).  For
     interactive browsers, the end of a session usually corresponds to
     closing the browser window.

     Note that the `save()' method won't save session cookies anyway,
     unless you ask otherwise by passing a true _ignore_discard_
     argument.

  FileCookieJar implements the following additional
methods:

 -- Method: FileCookieJar.save (filename=None, ignore_discard=False,
          ignore_expires=False)
     Save cookies to a file.

     This base class raises NotImplementedError.  Subclasses
     may leave this method unimplemented.

     _filename_ is the name of file in which to save cookies.  If
     _filename_ is not specified, `self.filename' is used (whose
     default is the value passed to the constructor, if any); if
     `self.filename' is None, ValueError is
     raised.

     _ignore_discard_: save even cookies set to be discarded.
     _ignore_expires_: save even cookies that have expired

     The file is overwritten if it already exists, thus wiping all the
     cookies it contains.  Saved cookies can be restored later using
     the load() or revert() methods.

 -- Method: FileCookieJar.load (filename=None, ignore_discard=False,
          ignore_expires=False)
     Load cookies from a file.

     Old cookies are kept unless overwritten by newly loaded ones.

     Arguments are as for save().

     The named file must be in the format understood by the class, or
     LoadError will be raised.  Also, see IOError.
     may be raised, for example if the file does not exist.

          Note: For backwards-compatibility with Python 2.4 (which
          raised an IOError), see LoadError is a
          subclass of IOError.

 -- Method: FileCookieJar.revert (filename=None, ignore_discard=False,
          ignore_expires=False)
     Clear all cookies and reload cookies from a saved file.

     revert() can raise the same exceptions as
     load(). If there is a failure, the object's state will not
     be altered.

  FileCookieJar instances have the following public
attributes:

 -- Attribute: FileCookieJar.filename
     Filename of default file in which to keep cookies.  This attribute
     may be assigned to.

 -- Attribute: FileCookieJar.delayload
     If true, load cookies lazily from disk.  This attribute should not
     be assigned to.  This is only a hint, since this only affects
     performance, not behaviour (unless the cookies on disk are
     changing). A CookieJar object may ignore it.  None of
     the FileCookieJar classes included in the standard
     library lazily loads cookies.

5.20.21.2 FileCookieJar subclasses and co-operation with web browsers
.....................................................................

The following CookieJar subclasses are provided for
reading and writing .

 -- Class: cookielib.MozillaCookieJar (filename, delayload=None,
          policy=None)
     A FileCookieJar that can load from and save cookies
     to disk in the Mozilla `cookies.txt' file format (which is also
     used by the Lynx and Netscape browsers).

          Note: Version 3 of the Firefox web browser no longer writes
          cookies in the `cookies.txt' file format.

          Note: This loses information about RFC 2965 cookies, and also
          about newer or non-standard cookie-attributes such as `port'.

          Warning: Back up your cookies before saving if you have
          cookies whose loss / corruption would be inconvenient (there
          are some subtleties which may lead to slight changes in the
          file over a load / save round-trip).

     Also note that cookies saved while Mozilla is running will get
     clobbered by Mozilla.

 -- Class: cookielib.LWPCookieJar (filename, delayload=None,
          policy=None)
     A FileCookieJar that can load from and save cookies
     to disk in format compatible with the libwww-perl library's
     `Set-Cookie3' file format.  This is convenient if you want to
     store cookies in a human-readable file.

5.20.21.3 CookiePolicy Objects
..............................

Objects implementing the CookiePolicy interface have the
following methods:

 -- Method: CookiePolicy.set_ok (cookie, request)
     Return boolean value indicating whether cookie should be accepted
     from server.

     _cookie_ is a cookielib.Cookie instance.  _request_
     is an object implementing the interface defined by the
     documentation for CookieJar.extract_cookies().

 -- Method: CookiePolicy.return_ok (cookie, request)
     Return boolean value indicating whether cookie should be returned
     to server.

     _cookie_ is a cookielib.Cookie instance.  _request_
     is an object implementing the interface defined by the
     documentation for CookieJar.add_cookie_header().

 -- Method: CookiePolicy.domain_return_ok (domain, request)
     Return false if cookies should not be returned, given cookie
     domain.

     This method is an optimization.  It removes the need for checking
     every cookie with a particular domain (which might involve reading
     many files).  Returning true from domain_return_ok().
     and path_return_ok() leaves all the work to
     return_ok().

     If domain_return_ok() returns true for the cookie
     domain, see path_return_ok() is called for the cookie
     path.  Otherwise, see path_return_ok() and
     return_ok() are never called for that cookie domain.  If
     path_return_ok() returns true, see return_ok()
     is called with the Cookie object itself for a
     full check.  Otherwise, see return_ok() is never called
     for that cookie path.

     Note that domain_return_ok() is called for every
     _cookie_ domain, not just for the _request_ domain.  For example,
     the function might be called with both `".example.com"' and
     `"www.example.com"' if the request domain is `"www.example.com"'.
     The same goes for path_return_ok().

     The _request_ argument is as documented for return_ok().

 -- Method: CookiePolicy.path_return_ok (path, request)
     Return false if cookies should not be returned, given cookie path.

     See the documentation for domain_return_ok().

  In addition to implementing the methods above, implementations of the
CookiePolicy interface must also supply the following
attributes, indicating which protocols should be used, and how.  All of
these attributes may be assigned to.

 -- Attribute: CookiePolicy.netscape
     Implement Netscape protocol.

 -- Attribute: CookiePolicy.rfc2965
     Implement RFC 2965 protocol.

 -- Attribute: CookiePolicy.hide_cookie2
     Don't add `Cookie2' header to requests (the presence of this header
     indicates to the server that we understand RFC 2965 cookies).

  The most useful way to define a CookiePolicy class is by
subclassing from DefaultCookiePolicy and overriding some
or all of the methods above.  CookiePolicy itself may be
used as a 'null policy' to allow setting and receiving any and all
cookies (this is unlikely to be useful).

5.20.21.4 DefaultCookiePolicy Objects
.....................................

Implements the standard rules for accepting and returning cookies.

  Both RFC 2965 and Netscape cookies are covered.  RFC 2965 handling is
switched off by default.

  The easiest way to provide your own policy is to override this class
and call its methods in your overridden implementations before adding
your own additional checks:

    import cookielib
    class MyCookiePolicy(cookielib.DefaultCookiePolicy):
        def set_ok(self, cookie, request):
            if not cookielib.DefaultCookiePolicy.set_ok(self, cookie, request):
                return False
            if i_dont_want_to_store_this_cookie(cookie):
                return False
            return True

In addition to the features required to implement the
CookiePolicy.  interface, this class allows you to block and
allow domains from setting and receiving cookies.  There are also some
strictness switches that allow you to tighten up the rather loose
Netscape protocol rules a little bit (at the cost of blocking some
benign cookies).

  A domain blacklist and whitelist is provided (both off by default).
Only domains not in the blacklist and present in the whitelist (if the
whitelist is active) participate in cookie setting and returning.  Use
the _blocked_domains_ constructor argument, and `blocked_domains()' and
`set_blocked_domains()' methods (and the corresponding argument and
methods for _allowed_domains_).  If you set a whitelist, you can turn
it off again by setting it to None.

  Domains in block or allow lists that do not start with a dot must
equal the cookie domain to be matched.  For example, `"example.com"'
matches a blacklist entry of `"example.com"', but `"www.example.com"'
does not.  Domains that do start with a dot are matched by more
specific domains too. For example, both `"www.example.com"' and
`"www.coyote.example.com"' match `".example.com"' (but `"example.com"'
itself does not).  IP addresses are an exception, and must match
exactly.  For example, if blocked_domains contains `"192.168.1.2"' and
`".168.1.2"', 192.168.1.2 is blocked, but 193.168.1.2 is not.

  DefaultCookiePolicy implements the following additional
methods:

 -- Method: DefaultCookiePolicy.blocked_domains ()
     Return the sequence of blocked domains (as a tuple).

 -- Method: DefaultCookiePolicy.set_blocked_domains (blocked_domains)
     Set the sequence of blocked domains.

 -- Method: DefaultCookiePolicy.is_blocked (domain)
     Return whether _domain_ is on the blacklist for setting or
     receiving cookies.

 -- Method: DefaultCookiePolicy.allowed_domains ()
     Return None, or the sequence of allowed domains (as a
     tuple).

 -- Method: DefaultCookiePolicy.set_allowed_domains (allowed_domains)
     Set the sequence of allowed domains, or None.

 -- Method: DefaultCookiePolicy.is_not_allowed (domain)
     Return whether _domain_ is not on the whitelist for setting or
     receiving cookies.

  DefaultCookiePolicy instances have the following
attributes, which are all initialised from the constructor arguments of
the same name, and which may all be assigned to.

 -- Attribute: DefaultCookiePolicy.rfc2109_as_netscape
     If true, request that the CookieJar instance
     downgrade RFC 2109 cookies (ie. cookies received in a `Set-Cookie'
     header with a version cookie-attribute of 1) to Netscape cookies
     by setting the version attribute of the Cookie instance
     to 0.  The default value is None, in which case RFC
     2109 cookies are downgraded if and only if RFC 2965 handling is
     turned off.  Therefore, RFC 2109 cookies are downgraded by default.

     New in version 2.5.

  General strictness switches:

 -- Attribute: DefaultCookiePolicy.strict_domain
     Don't allow sites to set two-component domains with country-code
     top-level domains like `.co.uk', `.gov.uk', `.co.nz'.etc.  This is
     far from perfect and isn't guaranteed to work!

  RFC 2965 protocol strictness switches:

 -- Attribute: DefaultCookiePolicy.strict_rfc2965_unverifiable
     Follow RFC 2965 rules on unverifiable transactions (usually, an
     unverifiable transaction is one resulting from a redirect or a
     request for an image hosted on another site).  If this is false,
     cookies are _never_ blocked on the basis of verifiability

  Netscape protocol strictness switches:

 -- Attribute: DefaultCookiePolicy.strict_ns_unverifiable
     apply RFC 2965 rules on unverifiable transactions even to Netscape
     cookies

 -- Attribute: DefaultCookiePolicy.strict_ns_domain
     Flags indicating how strict to be with domain-matching rules for
     Netscape cookies.  See below for acceptable values.

 -- Attribute: DefaultCookiePolicy.strict_ns_set_initial_dollar
     Ignore cookies in Set-Cookie: headers that have names starting
     with `'$''.

 -- Attribute: DefaultCookiePolicy.strict_ns_set_path
     Don't allow setting cookies whose path doesn't path-match request
     URI.

  `strict_ns_domain' is a collection of flags.  Its value is
constructed by or-ing together (for example,
`DomainStrictNoDots|DomainStrictNonDomain' means both flags are set).

 -- Attribute: DefaultCookiePolicy.DomainStrictNoDots
     When setting cookies, the 'host prefix' must not contain a dot (eg.
     `www.foo.bar.com' can't set a cookie for `.bar.com', because
     `www.foo' contains a dot).

 -- Attribute: DefaultCookiePolicy.DomainStrictNonDomain
     Cookies that did not explicitly specify a `domain'
     cookie-attribute can only be returned to a domain equal to the
     domain that set the cookie (eg.  `spam.example.com' won't be
     returned cookies from `example.com' that had no `domain'
     cookie-attribute).

 -- Attribute: DefaultCookiePolicy.DomainRFC2965Match
     When setting cookies, require a full RFC 2965 domain-match.

  The following attributes are provided for convenience, and are the
most useful combinations of the above flags:

 -- Attribute: DefaultCookiePolicy.DomainLiberal
     Equivalent to 0 (ie. all of the above Netscape domain strictness
     flags switched off).

 -- Attribute: DefaultCookiePolicy.DomainStrict
     Equivalent to `DomainStrictNoDots|DomainStrictNonDomain'.

5.20.21.5 Cookie Objects
........................

Cookie instances have Python attributes roughly
corresponding to the standard cookie-attributes specified in the
various cookie standards.  The correspondence is not one-to-one,
because there are complicated rules for assigning default values,
because the `max-age' and `expires' cookie-attributes contain
equivalent information, and because RFC 2109 cookies may be
'downgraded' by cookielib from version 1 to version 0
(Netscape) cookies.

  Assignment to these attributes should not be necessary other than in
rare circumstances in a CookiePolicy method.  The class
does not enforce internal consistency, so you should know what you're
doing if you do that.

 -- Attribute: Cookie.version
     Integer or None.  Netscape cookies have version.
     0. RFC 2965 and RFC 2109 cookies have a `version'
     cookie-attribute of 1.  However, note that cookielib.
     may 'downgrade' RFC 2109 cookies to Netscape cookies, in which
     case version is 0.

 -- Attribute: Cookie.name
     Cookie name (a string).

 -- Attribute: Cookie.value
     Cookie value (a string), or None.

 -- Attribute: Cookie.port
     String representing a port or a set of ports (eg. '80', or
     '80,8080'), or None.

 -- Attribute: Cookie.path
     Cookie path (a string, eg. `'/acme/rocket_launchers'').

 -- Attribute: Cookie.secure
     True if cookie should only be returned over a secure connection.

 -- Attribute: Cookie.expires
     Integer expiry date in seconds since epoch, or None.
     See also the is_expired() method.

 -- Attribute: Cookie.discard
     True if this is a session cookie.

 -- Attribute: Cookie.comment
     String comment from the server explaining the function of this
     cookie, or None.

 -- Attribute: Cookie.comment_url
     URL linking to a comment from the server explaining the function
     of this cookie, or None.

 -- Attribute: Cookie.rfc2109
     True if this cookie was received as an RFC 2109 cookie (ie. the
     cookie arrived in a `Set-Cookie' header, and the value of the
     Version cookie-attribute in that header was 1).  This attribute is
     provided because cookielib may 'downgrade' RFC 2109
     cookies to Netscape cookies, in which case version is
     0.

     New in version 2.5.

 -- Attribute: Cookie.port_specified
     True if a port or set of ports was explicitly specified by the
     server (in the `Set-Cookie' / `Set-Cookie2' header).

 -- Attribute: Cookie.domain_specified
     True if a domain was explicitly specified by the server.

 -- Attribute: Cookie.domain_initial_dot
     True if the domain explicitly specified by the server began with a
     dot (`'.'').

  Cookies may have additional non-standard cookie-attributes.  These
may be accessed using the following methods:

 -- Method: Cookie.has_nonstandard_attr (name)
     Return true if cookie has the named cookie-attribute.

 -- Method: Cookie.get_nonstandard_attr (name, default=None)
     If cookie has the named cookie-attribute, return its value.
     Otherwise, return _default_.

 -- Method: Cookie.set_nonstandard_attr (name, value)
     Set the value of the named cookie-attribute.

  The Cookie class also defines the following method:

 -- Method: Cookie.is_expired ([now=None])
     True if cookie has passed the time at which the server requested
     it should expire.  If _now_ is given (in seconds since the epoch),
     return whether the cookie has expired at the specified time.

5.20.21.6 Examples
..................

The first example shows the most common usage of cookielib.:

    import cookielib, urllib2
    cj = cookielib.CookieJar()
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    r = opener.open("http://example.com/")

This example illustrates how to open a URL using your Netscape,
Mozilla, or Lynx cookies (assumes Unix/Netscape convention for location
of the cookies file):

    import os, cookielib, urllib2
    cj = cookielib.MozillaCookieJar()
    cj.load(os.path.join(os.environ["HOME"], ".netscape/cookies.txt"))
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    r = opener.open("http://example.com/")

The next example illustrates the use of DefaultCookiePolicy.
Turn on RFC 2965 cookies, be more strict about domains when
setting and returning Netscape cookies, and block some domains from
setting cookies or having them returned:

    import urllib2
    from cookielib import CookieJar, DefaultCookiePolicy
    policy = DefaultCookiePolicy(
        rfc2965=True, strict_ns_domain=DefaultCookiePolicy.DomainStrict,
        blocked_domains=["ads.net", ".ads.net"])
    cj = CookieJar(policy)
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    r = opener.open("http://example.com/")

5.20.22 `Cookie' -- HTTP state management
-----------------------------------------

     Note: The Cookie module has been renamed to
     `http.cookies' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

The Cookie module defines classes for abstracting the
concept of cookies, an HTTP state management mechanism. It supports
both simple string-only cookies, and provides an abstraction for having
any serializable data-type as cookie value.

  The module formerly strictly applied the parsing rules described in
the RFC 2109 (http://tools.ietf.org/html/rfc2109.html) and RFC 2068
(http://tools.ietf.org/html/rfc2068.html) specifications.  It has since
been discovered that MSIE 3.0x doesn't follow the character rules
outlined in those specs.  As a result, the parsing rules used are a bit
less strict.

     Note: On encountering an invalid cookie, see CookieError.
     is raised, so if your cookie data comes from a browser you should
     always prepare for invalid data and catch CookieError.
     on parsing.

 -- Exception: Cookie.CookieError
     Exception failing because of RFC 2109
     (http://tools.ietf.org/html/rfc2109.html) invalidity: incorrect
     attributes, incorrect `Set-Cookie' header, etc.

 -- Class: Cookie.BaseCookie ([input])
     This class is a dictionary-like object whose keys are strings and
     whose values are Morsel instances. Note that upon
     setting a key to a value, the value is first converted to a
     Morsel containing the key and the value.

     If _input_ is given, it is passed to the load().
     method.

 -- Class: Cookie.SimpleCookie ([input])
     This class derives from BaseCookie and overrides
     `value_decode()' and `value_encode()' to be the identity and
     str() respectively.

 -- Class: Cookie.SerialCookie ([input])
     This class derives from BaseCookie and overrides
     `value_decode()' and `value_encode()' to be the
     pickle.loads() and pickle.dumps().

     Deprecated since version 2.3: Reading pickled values from
     untrusted cookie data is a huge security hole, as pickle strings
     can be crafted to cause arbitrary code to execute on your server.
     It is supported for backwards compatibility only, and may
     eventually go away.

 -- Class: Cookie.SmartCookie ([input])
     This class derives from BaseCookie. It overrides
     `value_decode()' to be pickle.loads() if it is a valid
     pickle, and otherwise the value itself. It overrides
     `value_encode()' to be pickle.dumps() unless it is a
     string, in which case it returns the value itself.

     Deprecated since version 2.3: The same security warning from
     SerialCookie applies here.

  A further security note is warranted.  For backwards compatibility,
the Cookie module exports a class named Cookie.
which is just an alias for SmartCookie.  This is probably a
mistake and will likely be removed in a future version.  You should not
use the Cookie class in your applications, for the same
reason why you should not use the SerialCookie class.

Module cookielib.
     HTTP cookie handling for web _clients_.  The cookielib.
     and Cookie.  modules do not depend on each other.

RFC 2109 (http://tools.ietf.org/html/rfc2109.html) - HTTP State Management Mechanism
     This is the state management specification implemented by this
     module.

5.20.22.1 Cookie Objects
........................

 -- Method: BaseCookie.value_decode (val)
     Return a decoded value from a string representation. Return value
     can be any type. This method does nothing in BaseCookie.
     -- it exists so it can be overridden.

 -- Method: BaseCookie.value_encode (val)
     Return an encoded value. _val_ can be any type, but return value
     must be a string. This method does nothing in BaseCookie.
     -- it exists so it can be overridden

     In general, it should be the case that value_encode().
     and value_decode() are inverses on the range of
     _value_decode_.

 -- Method: BaseCookie.output ([attrs[, header[, sep]]])
     Return a string representation suitable to be sent as HTTP
     headers. _attrs_ and _header_ are sent to each Morsel.'
     s output() method. _sep_ is used to join the
     headers together, and is by default the combination `'\r\n''
     (CRLF).

     Changed in version 2.5: The default separator has been changed
     from `'\n'' to match the cookie specification.

 -- Method: BaseCookie.js_output ([attrs])
     Return an embeddable JavaScript snippet, which, if run on a
     browser which supports JavaScript, will act the same as if the
     HTTP headers was sent.

     The meaning for _attrs_ is the same as in output().

 -- Method: BaseCookie.load (rawdata)
     If _rawdata_ is a string, parse it as an `HTTP_COOKIE' and add the
     values found there as Morsel.s. If it is a dictionary,
     it is equivalent to:

         for k, v in rawdata.items():
             cookie[k] = v

5.20.22.2 Morsel Objects
........................

 -- Class: Cookie.Morsel
     Abstract a key/value pair, which has some RFC 2109
     (http://tools.ietf.org/html/rfc2109.html) attributes.

     Morsels are dictionary-like objects, whose set of keys is constant
     -- the valid RFC 2109 (http://tools.ietf.org/html/rfc2109.html)
     attributes, which are

        * `expires'

        * `path'

        * `comment'

        * `domain'

        * `max-age'

        * `secure'

        * `version'

        * `httponly'

     The attribute `httponly' specifies that the cookie is only
     transfered in HTTP requests, and is not accessible through
     JavaScript. This is intended to mitigate some forms of cross-site
     scripting.

     The keys are case-insensitive.

     New in version 2.6: The `httponly' attribute was added.

 -- Attribute: Morsel.value
     The value of the cookie.

 -- Attribute: Morsel.coded_value
     The encoded value of the cookie -- this is what should be sent.

 -- Attribute: Morsel.key
     The name of the cookie.

 -- Method: Morsel.set (key, value, coded_value)
     Set the _key_, _value_ and _coded_value_ members.

 -- Method: Morsel.isReservedKey (K)
     Whether _K_ is a member of the set of keys of a Morsel.

 -- Method: Morsel.output ([attrs[, header]])
     Return a string representation of the Morsel, suitable to be sent
     as an HTTP header. By default, all the attributes are included,
     unless _attrs_ is given, in which case it should be a list of
     attributes to use. _header_ is by default `"Set-Cookie:"'.

 -- Method: Morsel.js_output ([attrs])
     Return an embeddable JavaScript snippet, which, if run on a
     browser which supports JavaScript, will act the same as if the
     HTTP header was sent.

     The meaning for _attrs_ is the same as in output().

 -- Method: Morsel.OutputString ([attrs])
     Return a string representing the Morsel, without any surrounding
     HTTP or JavaScript.

     The meaning for _attrs_ is the same as in output().

5.20.22.3 Example
.................

The following example demonstrates how to use the Cookie.
module.

    >>> import Cookie
    >>> C = Cookie.SimpleCookie()
    >>> C["fig"] = "newton"
    >>> C["sugar"] = "wafer"
    >>> print C # generate HTTP headers
    Set-Cookie: fig=newton
    Set-Cookie: sugar=wafer
    >>> print C.output() # same thing
    Set-Cookie: fig=newton
    Set-Cookie: sugar=wafer
    >>> C = Cookie.SimpleCookie()
    >>> C["rocky"] = "road"
    >>> C["rocky"]["path"] = "/cookie"
    >>> print C.output(header="Cookie:")
    Cookie: rocky=road; Path=/cookie
    >>> print C.output(attrs=[], header="Cookie:")
    Cookie: rocky=road
    >>> C = Cookie.SimpleCookie()
    >>> C.load("chips=ahoy; vienna=finger") # load from a string (HTTP header)
    >>> print C
    Set-Cookie: chips=ahoy
    Set-Cookie: vienna=finger
    >>> C = Cookie.SimpleCookie()
    >>> C.load('keebler="E=everybody; L=\\"Loves\\"; fudge=\\012;";')
    >>> print C
    Set-Cookie: keebler="E=everybody; L=\"Loves\"; fudge=\012;"
    >>> C = Cookie.SimpleCookie()
    >>> C["oreo"] = "doublestuff"
    >>> C["oreo"]["path"] = "/"
    >>> print C
    Set-Cookie: oreo=doublestuff; Path=/
    >>> C["twix"] = "none for you"
    >>> C["twix"].value
    'none for you'
    >>> C = Cookie.SimpleCookie()
    >>> C["number"] = 7 # equivalent to C["number"] = str(7)
    >>> C["string"] = "seven"
    >>> C["number"].value
    '7'
    >>> C["string"].value
    'seven'
    >>> print C
    Set-Cookie: number=7
    Set-Cookie: string=seven
    >>> # SerialCookie and SmartCookie are deprecated
    >>> # using it can cause security loopholes in your code.
    >>> C = Cookie.SerialCookie()
    >>> C["number"] = 7
    >>> C["string"] = "seven"
    >>> C["number"].value
    7
    >>> C["string"].value
    'seven'
    >>> print C
    Set-Cookie: number="I7\012."
    Set-Cookie: string="S'seven'\012p1\012."
    >>> C = Cookie.SmartCookie()
    >>> C["number"] = 7
    >>> C["string"] = "seven"
    >>> C["number"].value
    7
    >>> C["string"].value
    'seven'
    >>> print C
    Set-Cookie: number="I7\012."
    Set-Cookie: string=seven

5.20.23 `xmlrpclib' -- XML-RPC client access
--------------------------------------------

     Note: The xmlrpclib module has been renamed to
     `xmlrpc.client' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

New in version 2.2.

  XML-RPC is a Remote Procedure Call method that uses XML passed via
HTTP as a transport.  With it, a client can call methods with
parameters on a remote server (the server is named by a URI) and get
back structured data.  This module supports writing XML-RPC client
code; it handles all the details of translating between conformable
Python objects and XML on the wire.

 -- Class: xmlrpclib.ServerProxy (uri[, transport[, encoding[,
          verbose[, allow_none[, use_datetime]]]]])
     A ServerProxy instance is an object that manages
     communication with a remote XML-RPC server.  The required first
     argument is a URI (Uniform Resource Indicator), and will normally
     be the URL of the server.  The optional second argument is a
     transport factory instance; by default it is an internal
     `SafeTransport' instance for https: URLs and an internal HTTP
     `Transport' instance otherwise.  The optional third argument is an
     encoding, by default UTF-8. The optional fourth argument is a
     debugging flag.  If _allow_none_ is true,  the Python constant
     `None' will be translated into XML; the default behaviour is for
     `None' to raise a TypeError. This is a commonly-used
     extension to the XML-RPC specification, but isn't supported by all
     clients and servers; see
     <http://ontosys.com/xml-rpc/extensions.php> for a description.
     The _use_datetime_ flag can be used to cause date/time values to
     be presented as datetime.datetime objects; this is
     false by default.  datetime.datetime objects may be
     passed to calls.

     Both the HTTP and HTTPS transports support the URL syntax
     extension for HTTP Basic Authentication:
     `http://user:pass@host:port/path'.  The  `user:pass' portion will
     be base64-encoded as an HTTP 'Authorization' header, and sent to
     the remote server as part of the connection process when invoking
     an XML-RPC method.  You only need to use this if the remote server
     requires a Basic Authentication user and password.

     The returned instance is a proxy object with methods that can be
     used to invoke corresponding RPC calls on the remote server.  If
     the remote server supports the introspection API, the proxy can
     also be used to query the remote server for the methods it
     supports (service discovery) and fetch other server-associated
     metadata.

     ServerProxy instance methods take Python basic types
     and objects as arguments and return Python basic types and
     classes.  Types that are conformable (e.g. that can be marshalled
     through XML), include the following (and except where noted, they
     are unmarshalled as the same Python type):

     Name                                  Meaning
     ----------------------------------------------------------------------------------------
     see boolean.                  The True and False.
                                           constants
     `integers'                            Pass in directly
     `floating-point numbers'              Pass in directly
     `strings'                             Pass in directly
     `arrays'                              Any Python sequence type containing conformable
                                           elements. Arrays are returned as lists
     `structures'                          A Python dictionary. Keys must be strings,
                                           values may be any conformable type. Objects of
                                           user-defined classes can be passed in; only
                                           their ___dict___ attribute is transmitted.
     `dates'                               in seconds since the epoch (pass in an instance
                                           of the `DateTime' class) or a
                                           datetime.datetime instance.
     `binary data'                         pass in an instance of the `Binary' wrapper class

     This is the full set of data types supported by XML-RPC.  Method
     calls may also raise a special `Fault' instance, used to signal
     XML-RPC server errors, or `ProtocolError' used to signal an error
     in the HTTP/HTTPS transport layer.  Both `Fault' and
     `ProtocolError' derive from a base class called `Error'.  Note
     that even though starting with Python 2.2 you can subclass
     built-in types, the xmlrpclib module currently does not marshal
     instances of such subclasses.

     When passing strings, characters special to XML such as `<', `>',
     and `&' will be automatically escaped.  However, it's the caller's
     responsibility to ensure that the string is free of characters
     that aren't allowed in XML, such as the control characters with
     ASCII values between 0 and 31 (except, of course, tab, newline and
     carriage return); failing to do this will result in an XML-RPC
     request that isn't well-formed XML.  If you have to pass arbitrary
     strings via XML-RPC, use the `Binary' wrapper class described
     below.

     `Server' is retained as an alias for ServerProxy for
     backwards compatibility.  New code should use ServerProxy.

     Changed in version 2.5: The _use_datetime_ flag was added.

     Changed in version 2.6: Instances of new-style class.es
     can be passed in if they have an ___dict___ attribute and don't
     have a base class that is marshalled in a special way.

XML-RPC HOWTO (http://www.tldp.org/HOWTO/XML-RPC-HOWTO/index.html)
     A good description of XML-RPC operation and client software in
     several languages.  Contains pretty much everything an XML-RPC
     client developer needs to know.

XML-RPC Introspection (http://xmlrpc-c.sourceforge.net/introspection.html)
     Describes the XML-RPC protocol extension for introspection.

XML-RPC Specification (http://www.xmlrpc.com/spec)
     The official specification.

Unofficial XML-RPC Errata (http://effbot.org/zone/xmlrpc-errata.htm)
     Fredrik Lundh's "unofficial errata, intended to clarify certain
     details in the XML-RPC specification, as well as hint at 'best
     practices' to use when designing your own XML-RPC implementations."

5.20.23.1 ServerProxy Objects
.............................

A ServerProxy instance has a method corresponding to each
remote procedure call accepted by the XML-RPC server.  Calling the
method performs an RPC, dispatched by both name and argument signature
(e.g. the same method name can be overloaded with multiple argument
signatures).  The RPC finishes by returning a value, which may be
either returned data in a conformant type or a `Fault' or
`ProtocolError' object indicating an error.

  Servers that support the XML introspection API support some common
methods grouped under the reserved `system' member:

 -- Method: ServerProxy.system.listMethods ()
     This method returns a list of strings, one for each (non-system)
     method supported by the XML-RPC server.

 -- Method: ServerProxy.system.methodSignature (name)
     This method takes one parameter, the name of a method implemented
     by the XML-RPC server. It returns an array of possible signatures
     for this method. A signature is an array of types. The first of
     these types is the return type of the method, the rest are
     parameters.

     Because multiple signatures (ie. overloading) is permitted, this
     method returns a list of signatures rather than a singleton.

     Signatures themselves are restricted to the top level parameters
     expected by a method. For instance if a method expects one array
     of structs as a parameter, and it returns a string, its signature
     is simply "string, array". If it expects three integers and
     returns a string, its signature is "string, int, int, int".

     If no signature is defined for the method, a non-array value is
     returned. In Python this means that the type of the returned
     value will be something other than list.

 -- Method: ServerProxy.system.methodHelp (name)
     This method takes one parameter, the name of a method implemented
     by the XML-RPC server.  It returns a documentation string
     describing the use of that method. If no such string is available,
     an empty string is returned. The documentation string may contain
     HTML markup.

5.20.23.2 Boolean Objects
.........................

This class may be initialized from any Python value; the instance
returned depends only on its truth value.  It supports various Python
operators through __cmp__(), __repr__(),
__int__(), and __nonzero__().  methods, all implemented
in the obvious ways.

  It also has the following method, supported mainly for internal use
by the unmarshalling code:

 -- Method: Boolean.encode (out)
     Write the XML-RPC encoding of this Boolean item to the out stream
     object.

  A working example follows. The server code:

    import xmlrpclib
    from SimpleXMLRPCServer import SimpleXMLRPCServer

    def is_even(n):
        return n%2 == 0

    server = SimpleXMLRPCServer(("localhost", 8000))
    print "Listening on port 8000..."
    server.register_function(is_even, "is_even")
    server.serve_forever()

The client code for the preceding server:

    import xmlrpclib

    proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
    print "3 is even: %s" % str(proxy.is_even(3))
    print "100 is even: %s" % str(proxy.is_even(100))

5.20.23.3 DateTime Objects
..........................

This class may be initialized with seconds since the epoch, a time
tuple, an ISO 8601 time/date string, or a datetime.datetime.
instance.  It has the following methods, supported mainly for internal
use by the marshalling/unmarshalling code:

 -- Method: DateTime.decode (string)
     Accept a string as the instance's new time value.

 -- Method: DateTime.encode (out)
     Write the XML-RPC encoding of this `DateTime' item to the _out_
     stream object.

  It also supports certain of Python's built-in operators through
__cmp__().  and __repr__() methods.

  A working example follows. The server code:

    import datetime
    from SimpleXMLRPCServer import SimpleXMLRPCServer
    import xmlrpclib

    def today():
        today = datetime.datetime.today()
        return xmlrpclib.DateTime(today)

    server = SimpleXMLRPCServer(("localhost", 8000))
    print "Listening on port 8000..."
    server.register_function(today, "today")
    server.serve_forever()

The client code for the preceding server:

    import xmlrpclib
    import datetime

    proxy = xmlrpclib.ServerProxy("http://localhost:8000/")

    today = proxy.today()
    # convert the ISO8601 string to a datetime object
    converted = datetime.datetime.strptime(today.value, "%Y%m%dT%H:%M:%S")
    print "Today: %s" % converted.strftime("%d.%m.%Y, %H:%M")

5.20.23.4 Binary Objects
........................

This class may be initialized from string data (which may include
NULs). The primary access to the content of a `Binary' object is
provided by an attribute:

 -- Attribute: Binary.data
     The binary data encapsulated by the `Binary' instance.  The data is
     provided as an 8-bit string.

  `Binary' objects have the following methods, supported mainly for
internal use by the marshalling/unmarshalling code:

 -- Method: Binary.decode (string)
     Accept a base64 string and decode it as the instance's new data.

 -- Method: Binary.encode (out)
     Write the XML-RPC base 64 encoding of this binary item to the out
     stream object.

     The encoded data will have newlines every 76 characters as per RFC
     2045 section 6.8 (http://tools.ietf.org/html/rfc2045#section-6.8),
     which was the de facto standard base64 specification when the
     XML-RPC spec was written.

  It also supports certain of Python's built-in operators through a
__cmp__() method.

  Example usage of the binary objects.  We're going to transfer an
image over XMLRPC:

    from SimpleXMLRPCServer import SimpleXMLRPCServer
    import xmlrpclib

    def python_logo():
         with open("python_logo.jpg", "rb") as handle:
             return xmlrpclib.Binary(handle.read())

    server = SimpleXMLRPCServer(("localhost", 8000))
    print "Listening on port 8000..."
    server.register_function(python_logo, 'python_logo')

    server.serve_forever()

The client gets the image and saves it to a file:

    import xmlrpclib

    proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
    with open("fetched_python_logo.jpg", "wb") as handle:
        handle.write(proxy.python_logo().data)

5.20.23.5 Fault Objects
.......................

A `Fault' object encapsulates the content of an XML-RPC fault tag. Fault
objects have the following members:

 -- Attribute: Fault.faultCode
     A string indicating the fault type.

 -- Attribute: Fault.faultString
     A string containing a diagnostic message associated with the fault.

  In the following example we're going to intentionally cause a `Fault'
by returning a complex type object.  The server code:

    from SimpleXMLRPCServer import SimpleXMLRPCServer

    # A marshalling error is going to occur because we're returning a
    # complex number
    def add(x,y):
        return x+y+0j

    server = SimpleXMLRPCServer(("localhost", 8000))
    print "Listening on port 8000..."
    server.register_function(add, 'add')

    server.serve_forever()

The client code for the preceding server:

    import xmlrpclib

    proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
    try:
        proxy.add(2, 5)
    except xmlrpclib.Fault, err:
        print "A fault occurred"
        print "Fault code: %d" % err.faultCode
        print "Fault string: %s" % err.faultString

5.20.23.6 ProtocolError Objects
...............................

A `ProtocolError' object describes a protocol error in the underlying
transport layer (such as a 404 'not found' error if the server named by
the URI does not exist).  It has the following members:

 -- Attribute: ProtocolError.url
     The URI or URL that triggered the error.

 -- Attribute: ProtocolError.errcode
     The error code.

 -- Attribute: ProtocolError.errmsg
     The error message or diagnostic string.

 -- Attribute: ProtocolError.headers
     A string containing the headers of the HTTP/HTTPS request that
     triggered the error.

  In the following example we're going to intentionally cause a
`ProtocolError' by providing an URI that doesn't point to an XMLRPC
server:

    import xmlrpclib

    # create a ServerProxy with an URI that doesn't respond to XMLRPC requests
    proxy = xmlrpclib.ServerProxy("http://www.google.com/")

    try:
        proxy.some_method()
    except xmlrpclib.ProtocolError, err:
        print "A protocol error occurred"
        print "URL: %s" % err.url
        print "HTTP/HTTPS headers: %s" % err.headers
        print "Error code: %d" % err.errcode
        print "Error message: %s" % err.errmsg

5.20.23.7 MultiCall Objects
...........................

New in version 2.4.

  In <http://www.xmlrpc.com/discuss/msgReader%241208>, an approach is
presented to encapsulate multiple calls to a remote server into a
single request.

 -- Class: xmlrpclib.MultiCall (server)
     Create an object used to boxcar method calls. _server_ is the
     eventual target of the call. Calls can be made to the result
     object, but they will immediately return `None', and only store
     the call name and parameters in the MultiCall object.
     Calling the object itself causes all stored calls to be
     transmitted as a single `system.multicall' request. The result of
     this call is a generator.; iterating over this
     generator yields the individual results.

  A usage example of this class follows.  The server code

    from SimpleXMLRPCServer import SimpleXMLRPCServer

    def add(x,y):
        return x+y

    def subtract(x, y):
        return x-y

    def multiply(x, y):
        return x*y

    def divide(x, y):
        return x/y

    # A simple server with simple arithmetic functions
    server = SimpleXMLRPCServer(("localhost", 8000))
    print "Listening on port 8000..."
    server.register_multicall_functions()
    server.register_function(add, 'add')
    server.register_function(subtract, 'subtract')
    server.register_function(multiply, 'multiply')
    server.register_function(divide, 'divide')
    server.serve_forever()

The client code for the preceding server:

    import xmlrpclib

    proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
    multicall = xmlrpclib.MultiCall(proxy)
    multicall.add(7,3)
    multicall.subtract(7,3)
    multicall.multiply(7,3)
    multicall.divide(7,3)
    result = multicall()

    print "7+3=%d, 7-3=%d, 7*3=%d, 7/3=%d" % tuple(result)

5.20.23.8 Convenience Functions
...............................

 -- Function: xmlrpclib.boolean (value)
     Convert any Python value to one of the XML-RPC Boolean constants,
     `True' or `False'.

 -- Function: xmlrpclib.dumps (params[, methodname[, methodresponse[,
          encoding[, allow_none]]]])
     Convert _params_ into an XML-RPC request. or into a response if
     _methodresponse_ is true. _params_ can be either a tuple of
     arguments or an instance of the `Fault' exception class.  If
     _methodresponse_ is true, only a single value can be returned,
     meaning that _params_ must be of length 1. _encoding_, if
     supplied, is the encoding to use in the generated XML; the default
     is UTF-8.  Python's None value cannot be used in
     standard XML-RPC; to allow using it via an extension,  provide a
     true value for _allow_none_.

 -- Function: xmlrpclib.loads (data[, use_datetime])
     Convert an XML-RPC request or response into Python objects, a
     `(params, methodname)'.  _params_ is a tuple of argument;
     _methodname_ is a string, or `None' if no method name is present
     in the packet. If the XML-RPC packet represents a fault condition,
     this function will raise a `Fault' exception.  The _use_datetime_
     flag can be used to cause date/time values to be presented as
     datetime.datetime objects; this is false by default.

     Changed in version 2.5: The _use_datetime_ flag was added.

5.20.23.9 Example of Client Usage
.................................

    # simple test program (from the XML-RPC specification)
    from xmlrpclib import ServerProxy, Error

    # server = ServerProxy("http://localhost:8000") # local server
    server = ServerProxy("http://betty.userland.com")

    print server

    try:
        print server.examples.getStateName(41)
    except Error, v:
        print "ERROR", v

To access an XML-RPC server through a proxy, you need to define  a
custom transport.  The following example shows how:

    import xmlrpclib, httplib

    class ProxiedTransport(xmlrpclib.Transport):
        def set_proxy(self, proxy):
            self.proxy = proxy
        def make_connection(self, host):
            self.realhost = host
            h = httplib.HTTP(self.proxy)
            return h
        def send_request(self, connection, handler, request_body):
            connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
        def send_host(self, connection, host):
            connection.putheader('Host', self.realhost)

    p = ProxiedTransport()
    p.set_proxy('proxy-server:8080')
    server = xmlrpclib.Server('http://time.xmlrpc.com/RPC2', transport=p)
    print server.currentTime.getCurrentTime()

5.20.23.10 Example of Client and Server Usage
.............................................

See SimpleXMLRPCServer Example.

5.20.24 `SimpleXMLRPCServer' -- Basic XML-RPC server
----------------------------------------------------

     Note: The SimpleXMLRPCServer module has been merged
     into `xmlrpc.server' in Python 3.0.  The 2to3 tool
     will automatically adapt imports when converting your sources to
     3.0.

New in version 2.2.

  The SimpleXMLRPCServer module provides a basic server
framework for XML-RPC servers written in Python.  Servers can either be
free standing, using SimpleXMLRPCServer, or embedded in a
CGI environment, using CGIXMLRPCRequestHandler.

 -- Class: SimpleXMLRPCServer.SimpleXMLRPCServer (addr[,
          requestHandler[, logRequests[, allow_none[, encoding[,
          bind_and_activate]]]])
     Create a new server instance.  This class provides methods for
     registration of functions that can be called by the XML-RPC
     protocol.  The _requestHandler_ parameter should be a factory for
     request handler instances; it defaults to
     SimpleXMLRPCRequestHandler.  The _addr_ and _requestHandler_
     parameters are passed to the `SocketServer.TCPServer' constructor.
     If _logRequests_ is true (the default), requests will be logged;
     setting this parameter to false will turn off logging.   The
     _allow_none_ and _encoding_ parameters are passed on to
     xmlrpclib and control the XML-RPC responses that will be
     returned from the server. The _bind_and_activate_ parameter
     controls whether `server_bind()' and `server_activate()' are
     called immediately by the constructor; it defaults to true.
     Setting it to false allows code to manipulate the
     _allow_reuse_address_ class variable before the address is bound.

     Changed in version 2.5: The _allow_none_ and _encoding_ parameters
     were added.

     Changed in version 2.6: The _bind_and_activate_ parameter was
     added.

 -- Class: SimpleXMLRPCServer.CGIXMLRPCRequestHandler ([allow_none[,
          encoding]])
     Create a new instance to handle XML-RPC requests in a CGI
     environment.  The _allow_none_ and _encoding_ parameters are
     passed on to  xmlrpclib and control the XML-RPC
     responses that will be returned  from the server.

     New in version 2.3.

     Changed in version 2.5: The _allow_none_ and _encoding_ parameters
     were added.

 -- Class: SimpleXMLRPCServer.SimpleXMLRPCRequestHandler
     Create a new request handler instance.  This request handler
     supports `POST' requests and modifies logging so that the
     _logRequests_ parameter to the SimpleXMLRPCServer.
     constructor parameter is honored.

5.20.24.1 SimpleXMLRPCServer Objects
....................................

The SimpleXMLRPCServer class is based on
`SocketServer.TCPServer' and provides a means of creating simple, stand
alone XML-RPC servers.

 -- Method: SimpleXMLRPCServer.register_function (function[, name])
     Register a function that can respond to XML-RPC requests.  If
     _name_ is given, it will be the method name associated with
     _function_, otherwise `function.__name__' will be used.  _name_
     can be either a normal or Unicode string, and may contain
     characters not legal in Python identifiers, including the period
     character.

 -- Method: SimpleXMLRPCServer.register_instance (instance[,
          allow_dotted_names])
     Register an object which is used to expose method names which have
     not been registered using register_function().  If
     _instance_ contains a `_dispatch()' method, it is called with the
     requested method name and the parameters from the request.  Its
     API is `def _dispatch(self, method, params)' (note that _params_
     does not represent a variable argument list).  If it calls an
     underlying function to perform its task, that function is called as
     `func(*params)', expanding the parameter list. The return value
     from `_dispatch()' is returned to the client as the result.  If
     _instance_ does not have a `_dispatch()' method, it is searched
     for an attribute matching the name of the requested method.

     If the optional _allow_dotted_names_ argument is true and the
     instance does not have a `_dispatch()' method, then if the
     requested method name contains periods, each component of the
     method name is searched for individually, with the effect that a
     simple hierarchical search is performed.  The value found from
     this search is then called with the parameters from the request,
     and the return value is passed back to the client.

          Warning: Enabling the _allow_dotted_names_ option allows
          intruders to access your module's global variables and may
          allow intruders to execute arbitrary code on your machine.
          Only use this option on a secure, closed network.

     Changed in version 2.3.5,: 2.4.1 _allow_dotted_names_ was added to
     plug a security hole; prior versions are insecure.

 -- Method: SimpleXMLRPCServer.register_introspection_functions ()
     Registers the XML-RPC introspection functions `system.listMethods',
     `system.methodHelp' and `system.methodSignature'.

     New in version 2.3.

 -- Method: SimpleXMLRPCServer.register_multicall_functions ()
     Registers the XML-RPC multicall function system.multicall.

 -- Attribute: SimpleXMLRPCRequestHandler.rpc_paths
     An attribute value that must be a tuple listing valid path
     portions of the URL for receiving XML-RPC requests.  Requests
     posted to other paths will result in a 404 "no such page" HTTP
     error.  If this tuple is empty, all paths will be considered
     valid. The default value is `('/', '/RPC2')'.

     New in version 2.5.

 -- Attribute: SimpleXMLRPCRequestHandler.encode_threshold
     If this attribute is not `None', responses larger than this value
     will be encoded using the _gzip_ transfer encoding, if permitted by
     the client.  The default is `1400' which corresponds roughly to a
     single TCP packet.

     New in version 2.7.

5.20.24.2 SimpleXMLRPCServer Example
....................................

Server code:

    from SimpleXMLRPCServer import SimpleXMLRPCServer
    from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler

    # Restrict to a particular path.
    class RequestHandler(SimpleXMLRPCRequestHandler):
        rpc_paths = ('/RPC2',)

    # Create server
    server = SimpleXMLRPCServer(("localhost", 8000),
                                requestHandler=RequestHandler)
    server.register_introspection_functions()

    # Register pow() function; this will use the value of
    # pow.__name__ as the name, which is just 'pow'.
    server.register_function(pow)

    # Register a function under a different name
    def adder_function(x,y):
        return x + y
    server.register_function(adder_function, 'add')

    # Register an instance; all the methods of the instance are
    # published as XML-RPC methods (in this case, just 'div').
    class MyFuncs:
        def div(self, x, y):
            return x // y

    server.register_instance(MyFuncs())

    # Run the server's main loop
    server.serve_forever()

The following client code will call the methods made available by the
preceding server:

    import xmlrpclib

    s = xmlrpclib.ServerProxy('http://localhost:8000')
    print s.pow(2,3)  # Returns 2**3 = 8
    print s.add(2,3)  # Returns 5
    print s.div(5,2)  # Returns 5//2 = 2

    # Print list of available methods
    print s.system.listMethods()

5.20.24.3 CGIXMLRPCRequestHandler
.................................

The CGIXMLRPCRequestHandler class can be used to  handle
XML-RPC requests sent to Python CGI scripts.

 -- Method: CGIXMLRPCRequestHandler.register_function (function[, name])
     Register a function that can respond to XML-RPC requests. If
     _name_ is given, it will be the method name associated with
     function, otherwise _function.__name___ will be used. _name_ can
     be either a normal or Unicode string, and may contain  characters
     not legal in Python identifiers, including the period character.

 -- Method: CGIXMLRPCRequestHandler.register_instance (instance)
     Register an object which is used to expose method names  which
     have not been registered using register_function(). If
     instance contains a `_dispatch()' method, it is called with the
     requested method name and the parameters from the  request; the
     return value is returned to the client as the result. If instance
     does not have a `_dispatch()' method, it is searched for an
     attribute matching the name of the requested method; if  the
     requested method name contains periods, each  component of the
     method name is searched for individually,  with the effect that a
     simple hierarchical search is performed.  The value found from
     this search is then called with the  parameters from the request,
     and the return value is passed  back to the client.

 -- Method: CGIXMLRPCRequestHandler.register_introspection_functions ()
     Register the XML-RPC introspection functions  `system.listMethods',
     `system.methodHelp' and  `system.methodSignature'.

 -- Method: CGIXMLRPCRequestHandler.register_multicall_functions ()
     Register the XML-RPC multicall function `system.multicall'.

 -- Method: CGIXMLRPCRequestHandler.handle_request ([request_text =
          None])
     Handle a XML-RPC request. If _request_text_ is given, it  should
     be the POST data provided by the HTTP server,  otherwise the
     contents of stdin will be used.

  Example:

    class MyFuncs:
        def div(self, x, y) : return x // y

    handler = CGIXMLRPCRequestHandler()
    handler.register_function(pow)
    handler.register_function(lambda x,y: x+y, 'add')
    handler.register_introspection_functions()
    handler.register_instance(MyFuncs())
    handler.handle_request()

5.20.25 `DocXMLRPCServer' -- Self-documenting XML-RPC server
------------------------------------------------------------

     Note: The DocXMLRPCServer module has been merged into
     `xmlrpc.server' in Python 3.0.  The 2to3 tool will
     automatically adapt imports when converting your sources to 3.0.

New in version 2.3.

  The DocXMLRPCServer module extends the classes found in
SimpleXMLRPCServer to serve HTML documentation in response
to HTTP GET requests. Servers can either be free standing, using
DocXMLRPCServer, or embedded in a CGI environment, using
DocCGIXMLRPCRequestHandler.

 -- Class: DocXMLRPCServer.DocXMLRPCServer (addr[, requestHandler[,
          logRequests[, allow_none[, encoding[, bind_and_activate]]]]])
     Create a new server instance. All parameters have the same meaning
     as for SimpleXMLRPCServer.SimpleXMLRPCServer.;
     _requestHandler_ defaults to DocXMLRPCRequestHandler.

 -- Class: DocXMLRPCServer.DocCGIXMLRPCRequestHandler
     Create a new instance to handle XML-RPC requests in a CGI
     environment.

 -- Class: DocXMLRPCServer.DocXMLRPCRequestHandler
     Create a new request handler instance. This request handler
     supports XML-RPC POST requests, documentation GET requests, and
     modifies logging so that the _logRequests_ parameter to the
     DocXMLRPCServer constructor parameter is honored.

5.20.25.1 DocXMLRPCServer Objects
.................................

The DocXMLRPCServer class is derived from
SimpleXMLRPCServer.SimpleXMLRPCServer and provides a means of
creating self-documenting, stand alone XML-RPC servers. HTTP POST
requests are handled as XML-RPC method calls. HTTP GET requests are
handled by generating pydoc-style HTML documentation. This allows a
server to provide its own web-based documentation.

 -- Method: DocXMLRPCServer.set_server_title (server_title)
     Set the title used in the generated HTML documentation. This title
     will be used inside the HTML "title" element.

 -- Method: DocXMLRPCServer.set_server_name (server_name)
     Set the name used in the generated HTML documentation. This name
     will appear at the top of the generated documentation inside a
     "h1" element.

 -- Method: DocXMLRPCServer.set_server_documentation
          (server_documentation)
     Set the description used in the generated HTML documentation. This
     description will appear as a paragraph, below the server name, in
     the documentation.

5.20.25.2 DocCGIXMLRPCRequestHandler
....................................

The DocCGIXMLRPCRequestHandler class is derived from
SimpleXMLRPCServer.CGIXMLRPCRequestHandler and provides a means
of creating self-documenting, XML-RPC CGI scripts. HTTP POST requests
are handled as XML-RPC method calls. HTTP GET requests are handled by
generating pydoc-style HTML documentation. This allows a server to
provide its own web-based documentation.

 -- Method: DocCGIXMLRPCRequestHandler.set_server_title (server_title)
     Set the title used in the generated HTML documentation. This title
     will be used inside the HTML "title" element.

 -- Method: DocCGIXMLRPCRequestHandler.set_server_name (server_name)
     Set the name used in the generated HTML documentation. This name
     will appear at the top of the generated documentation inside a
     "h1" element.

 -- Method: DocCGIXMLRPCRequestHandler.set_server_documentation
          (server_documentation)
     Set the description used in the generated HTML documentation. This
     description will appear as a paragraph, below the server name, in
     the documentation.

5.21 Multimedia Services
========================

The modules described in this chapter implement various algorithms or
interfaces that are mainly useful for multimedia applications.  They
are available at the discretion of the installation.  Here's an
overview:

5.21.1 `audioop' -- Manipulate raw audio data
---------------------------------------------

The audioop module contains some useful operations on sound
fragments.  It operates on sound fragments consisting of signed integer
samples 8, 16 or 32 bits wide, stored in Python strings.  This is the
same format as used by the al and sunaudiodev.
modules.  All scalar items are integers, unless specified otherwise.

  This module provides support for a-LAW, u-LAW and Intel/DVI ADPCM
encodings.

  A few of the more complicated operations only take 16-bit samples,
otherwise the sample size (in bytes) is always a parameter of the
operation.

  The module defines the following variables and functions:

 -- Exception: audioop.error
     This exception is raised on all errors, such as unknown number of
     bytes per sample, etc.

 -- Function: audioop.add (fragment1, fragment2, width)
     Return a fragment which is the addition of the two samples passed
     as parameters.  _width_ is the sample width in bytes, either `1',
     `2' or `4'.  Both fragments should have the same length.

 -- Function: audioop.adpcm2lin (adpcmfragment, width, state)
     Decode an Intel/DVI ADPCM coded fragment to a linear fragment.
     See the description of lin2adpcm() for details on
     ADPCM coding. Return a tuple `(sample, newstate)' where the sample
     has the width specified in _width_.

 -- Function: audioop.alaw2lin (fragment, width)
     Convert sound fragments in a-LAW encoding to linearly encoded
     sound fragments.  a-LAW encoding always uses 8 bits samples, so
     _width_ refers only to the sample width of the output fragment
     here.

     New in version 2.5.

 -- Function: audioop.avg (fragment, width)
     Return the average over all samples in the fragment.

 -- Function: audioop.avgpp (fragment, width)
     Return the average peak-peak value over all samples in the
     fragment. No filtering is done, so the usefulness of this routine
     is questionable.

 -- Function: audioop.bias (fragment, width, bias)
     Return a fragment that is the original fragment with a bias added
     to each sample.

 -- Function: audioop.cross (fragment, width)
     Return the number of zero crossings in the fragment passed as an
     argument.

 -- Function: audioop.findfactor (fragment, reference)
     Return a factor _F_ such that `rms(add(fragment, mul(reference,
     -F)))' is minimal, i.e., return the factor with which you should
     multiply _reference_ to make it match as well as possible to
     _fragment_.  The fragments should both contain 2-byte samples.

     The time taken by this routine is proportional to `len(fragment)'.

 -- Function: audioop.findfit (fragment, reference)
     Try to match _reference_ as well as possible to a portion of
     _fragment_ (which should be the longer fragment).  This is
     (conceptually) done by taking slices out of _fragment_, using
     findfactor() to compute the best match, and
     minimizing the result.  The fragments should both contain 2-byte
     samples.  Return a tuple `(offset, factor)' where _offset_ is the
     (integer) offset into _fragment_ where the optimal match started
     and _factor_ is the (floating-point) factor as per
     findfactor().

 -- Function: audioop.findmax (fragment, length)
     Search _fragment_ for a slice of length _length_ samples (not
     bytes!) with maximum energy, i.e., return _i_ for which
     `rms(fragment[i*2:(i+length)*2])' is maximal.  The fragments
     should both contain 2-byte samples.

     The routine takes time proportional to `len(fragment)'.

 -- Function: audioop.getsample (fragment, width, index)
     Return the value of sample _index_ from the fragment.

 -- Function: audioop.lin2adpcm (fragment, width, state)
     Convert samples to 4 bit Intel/DVI ADPCM encoding.  ADPCM coding
     is an adaptive coding scheme, whereby each 4 bit number is the
     difference between one sample and the next, divided by a (varying)
     step.  The Intel/DVI ADPCM algorithm has been selected for use by
     the IMA, so it may well become a standard.

     _state_ is a tuple containing the state of the coder.  The coder
     returns a tuple `(adpcmfrag, newstate)', and the _newstate_ should
     be passed to the next call of lin2adpcm().  In the
     initial call, `None' can be passed as the state.  _adpcmfrag_ is
     the ADPCM coded fragment packed 2 4-bit values per byte.

 -- Function: audioop.lin2alaw (fragment, width)
     Convert samples in the audio fragment to a-LAW encoding and return
     this as a Python string.  a-LAW is an audio encoding format
     whereby you get a dynamic range of about 13 bits using only 8 bit
     samples.  It is used by the Sun audio hardware, among others.

     New in version 2.5.

 -- Function: audioop.lin2lin (fragment, width, newwidth)
     Convert samples between 1-, 2- and 4-byte formats.

          Note: In some audio formats, such as .WAV files, 16 and 32
          bit samples are signed, but 8 bit samples are unsigned.  So
          when converting to 8 bit wide samples for these formats, you
          need to also add 128 to the result:

              new_frames = audioop.lin2lin(frames, old_width, 1)
              new_frames = audioop.bias(new_frames, 1, 128)

          The same, in reverse, has to be applied when converting from
          8 to 16 or 32 bit width samples.

 -- Function: audioop.lin2ulaw (fragment, width)
     Convert samples in the audio fragment to u-LAW encoding and return
     this as a Python string.  u-LAW is an audio encoding format
     whereby you get a dynamic range of about 14 bits using only 8 bit
     samples.  It is used by the Sun audio hardware, among others.

 -- Function: audioop.minmax (fragment, width)
     Return a tuple consisting of the minimum and maximum values of all
     samples in the sound fragment.

 -- Function: audioop.max (fragment, width)
     Return the maximum of the _absolute value_ of all samples in a
     fragment.

 -- Function: audioop.maxpp (fragment, width)
     Return the maximum peak-peak value in the sound fragment.

 -- Function: audioop.mul (fragment, width, factor)
     Return a fragment that has all samples in the original fragment
     multiplied by the floating-point value _factor_.  Overflow is
     silently ignored.

 -- Function: audioop.ratecv (fragment, width, nchannels, inrate,
          outrate, state[, weightA[, weightB]])
     Convert the frame rate of the input fragment.

     _state_ is a tuple containing the state of the converter.  The
     converter returns a tuple `(newfragment, newstate)', and
     _newstate_ should be passed to the next call of ratecv().
     The initial call should pass `None' as the state.

     The _weightA_ and _weightB_ arguments are parameters for a simple
     digital filter and default to `1' and `0' respectively.

 -- Function: audioop.reverse (fragment, width)
     Reverse the samples in a fragment and returns the modified
     fragment.

 -- Function: audioop.rms (fragment, width)
     Return the root-mean-square of the fragment, i.e.
     `sqrt(sum(S_i^2)/n)'.

     This is a measure of the power in an audio signal.

 -- Function: audioop.tomono (fragment, width, lfactor, rfactor)
     Convert a stereo fragment to a mono fragment.  The left channel is
     multiplied by _lfactor_ and the right channel by _rfactor_ before
     adding the two channels to give a mono signal.

 -- Function: audioop.tostereo (fragment, width, lfactor, rfactor)
     Generate a stereo fragment from a mono fragment.  Each pair of
     samples in the stereo fragment are computed from the mono sample,
     whereby left channel samples are multiplied by _lfactor_ and right
     channel samples by _rfactor_.

 -- Function: audioop.ulaw2lin (fragment, width)
     Convert sound fragments in u-LAW encoding to linearly encoded
     sound fragments.  u-LAW encoding always uses 8 bits samples, so
     _width_ refers only to the sample width of the output fragment
     here.

  Note that operations such as mul() or max().
make no distinction between mono and stereo fragments, i.e. all samples
are treated equal.  If this is a problem the stereo fragment should be
split into two mono fragments first and recombined later.  Here is an
example of how to do that:

    def mul_stereo(sample, width, lfactor, rfactor):
        lsample = audioop.tomono(sample, width, 1, 0)
        rsample = audioop.tomono(sample, width, 0, 1)
        lsample = audioop.mul(lsample, width, lfactor)
        rsample = audioop.mul(rsample, width, rfactor)
        lsample = audioop.tostereo(lsample, width, 1, 0)
        rsample = audioop.tostereo(rsample, width, 0, 1)
        return audioop.add(lsample, rsample, width)

If you use the ADPCM coder to build network packets and you want your
protocol to be stateless (i.e. to be able to tolerate packet loss) you
should not only transmit the data but also the state.  Note that you
should send the _initial_ state (the one you passed to
lin2adpcm()) along to the decoder, not the final state (as
returned by the coder).  If you want to use `struct.struct()' to store
the state in binary you can code the first element (the predicted
value) in 16 bits and the second (the delta index) in 8.

  The ADPCM coders have never been tried against other ADPCM coders,
only against themselves.  It could well be that I misinterpreted the
standards in which case they will not be interoperable with the
respective standards.

  The `find*()' routines might look a bit funny at first sight. They are
primarily meant to do echo cancellation.  A reasonably fast way to do
this is to pick the most energetic piece of the output sample, locate
that in the input sample and subtract the whole output sample from the
input sample:

    def echocancel(outputdata, inputdata):
        pos = audioop.findmax(outputdata, 800)    # one tenth second
        out_test = outputdata[pos*2:]
        in_test = inputdata[pos*2:]
        ipos, factor = audioop.findfit(in_test, out_test)
        # Optional (for better cancellation):
        # factor = audioop.findfactor(in_test[ipos*2:ipos*2+len(out_test)],
        #              out_test)
        prefill = '\0'*(pos+ipos)*2
        postfill = '\0'*(len(inputdata)-len(prefill)-len(outputdata))
        outputdata = prefill + audioop.mul(outputdata,2,-factor) + postfill
        return audioop.add(inputdata, outputdata, 2)

5.21.2 `imageop' -- Manipulate raw image data
---------------------------------------------

Deprecated since version 2.6: The imageop module has been
removed in Python 3.0.

  The imageop module contains some useful operations on
images. It operates on images consisting of 8 or 32 bit pixels stored
in Python strings.  This is the same format as used by
`gl.lrectwrite()' and the imgfile module.

  The module defines the following variables and functions:

 -- Exception: imageop.error
     This exception is raised on all errors, such as unknown number of
     bits per pixel, etc.

 -- Function: imageop.crop (image, psize, width, height, x0, y0, x1, y1)
     Return the selected part of _image_, which should be _width_ by
     _height_ in size and consist of pixels of _psize_ bytes. _x0_,
     _y0_, _x1_ and _y1_ are like the `gl.lrectread()' parameters, i.e.
     the boundary is included in the new image.  The new boundaries
     need not be inside the picture.  Pixels that fall outside the old
     image will have their value set to zero.  If _x0_ is bigger than
     _x1_ the new image is mirrored.  The same holds for the y
     coordinates.

 -- Function: imageop.scale (image, psize, width, height, newwidth,
          newheight)
     Return _image_ scaled to size _newwidth_ by _newheight_. No
     interpolation is done, scaling is done by simple-minded pixel
     duplication or removal.  Therefore, computer-generated images or
     dithered images will not look nice after scaling.

 -- Function: imageop.tovideo (image, psize, width, height)
     Run a vertical low-pass filter over an image.  It does so by
     computing each destination pixel as the average of two
     vertically-aligned source pixels.  The main use of this routine is
     to forestall excessive flicker if the image is displayed on a
     video device that uses interlacing, hence the name.

 -- Function: imageop.grey2mono (image, width, height, threshold)
     Convert a 8-bit deep greyscale image to a 1-bit deep image by
     thresholding all the pixels.  The resulting image is tightly
     packed and is probably only useful as an argument to
     mono2grey().

 -- Function: imageop.dither2mono (image, width, height)
     Convert an 8-bit greyscale image to a 1-bit monochrome image using
     a (simple-minded) dithering algorithm.

 -- Function: imageop.mono2grey (image, width, height, p0, p1)
     Convert a 1-bit monochrome image to an 8 bit greyscale or color
     image. All pixels that are zero-valued on input get value _p0_ on
     output and all one-value input pixels get value _p1_ on output.
     To convert a monochrome black-and-white image to greyscale pass
     the values `0' and `255' respectively.

 -- Function: imageop.grey2grey4 (image, width, height)
     Convert an 8-bit greyscale image to a 4-bit greyscale image
     without dithering.

 -- Function: imageop.grey2grey2 (image, width, height)
     Convert an 8-bit greyscale image to a 2-bit greyscale image
     without dithering.

 -- Function: imageop.dither2grey2 (image, width, height)
     Convert an 8-bit greyscale image to a 2-bit greyscale image with
     dithering.  As for dither2mono(), the dithering
     algorithm is currently very simple.

 -- Function: imageop.grey42grey (image, width, height)
     Convert a 4-bit greyscale image to an 8-bit greyscale image.

 -- Function: imageop.grey22grey (image, width, height)
     Convert a 2-bit greyscale image to an 8-bit greyscale image.

 -- Data: imageop.backward_compatible
     If set to 0, the functions in this module use a non-backward
     compatible way of representing multi-byte pixels on little-endian
     systems.  The SGI for which this module was originally written is
     a big-endian system, so setting this variable will have no effect.
     However, the code wasn't originally intended to run on anything
     else, so it made assumptions about byte order which are not
     universal.  Setting this variable to 0 will cause the byte order
     to be reversed on little-endian systems, so that it then is the
     same as on big-endian systems.

5.21.3 `aifc' -- Read and write AIFF and AIFC files
---------------------------------------------------

This module provides support for reading and writing AIFF and AIFF-C
files.  AIFF is Audio Interchange File Format, a format for storing
digital audio samples in a file.  AIFF-C is a newer version of the
format that includes the ability to compress the audio data.

     Note: Some operations may only work under IRIX; these will raise
     ImportError.  when attempting to import the `cl'
     module, which is only available on IRIX.

  Audio files have a number of parameters that describe the audio data.
The sampling rate or frame rate is the number of times per second the
sound is sampled.  The number of channels indicate if the audio is
mono, stereo, or quadro.  Each frame consists of one sample per
channel.  The sample size is the size in bytes of each sample.  Thus a
frame consists of _nchannels_**samplesize* bytes, and a second's worth
of audio consists of _nchannels_**samplesize***framerate* bytes.

  For example, CD quality audio has a sample size of two bytes (16
bits), uses two channels (stereo) and has a frame rate of 44,100
frames/second.  This gives a frame size of 4 bytes (2*2), and a
second's worth occupies 2*2*44100 bytes (176,400 bytes).

  Module aifc defines the following function:

 -- Function: aifc.open (file[, mode])
     Open an AIFF or AIFF-C file and return an object instance with
     methods that are described below.  The argument _file_ is either a
     string naming a file or a file object.  _mode_ must be `'r'' or
     `'rb'' when the file must be opened for reading, or `'w''  or
     `'wb'' when the file must be opened for writing.  If omitted,
     `file.mode' is used if it exists, otherwise `'rb'' is used.  When
     used for writing, the file object should be seekable, unless you
     know ahead of time how many samples you are going to write in
     total and use `writeframesraw()' and `setnframes()'.

  Objects returned by open() when a file is opened for
reading have the following methods:

 -- Method: aifc.getnchannels ()
     Return the number of audio channels (1 for mono, 2 for stereo).

 -- Method: aifc.getsampwidth ()
     Return the size in bytes of individual samples.

 -- Method: aifc.getframerate ()
     Return the sampling rate (number of audio frames per second).

 -- Method: aifc.getnframes ()
     Return the number of audio frames in the file.

 -- Method: aifc.getcomptype ()
     Return a four-character string describing the type of compression
     used in the audio file.  For AIFF files, the returned value is
     `'NONE''.

 -- Method: aifc.getcompname ()
     Return a human-readable description of the type of compression
     used in the audio file.  For AIFF files, the returned value is
     `'not compressed''.

 -- Method: aifc.getparams ()
     Return a tuple consisting of all of the above values in the above
     order.

 -- Method: aifc.getmarkers ()
     Return a list of markers in the audio file.  A marker consists of
     a tuple of three elements.  The first is the mark ID (an integer),
     the second is the mark position in frames from the beginning of
     the data (an integer), the third is the name of the mark (a
     string).

 -- Method: aifc.getmark (id)
     Return the tuple as described in getmarkers() for the
     mark with the given _id_.

 -- Method: aifc.readframes (nframes)
     Read and return the next _nframes_ frames from the audio file.
     The returned data is a string containing for each frame the
     uncompressed samples of all channels.

 -- Method: aifc.rewind ()
     Rewind the read pointer.  The next readframes() will
     start from the beginning.

 -- Method: aifc.setpos (pos)
     Seek to the specified frame number.

 -- Method: aifc.tell ()
     Return the current frame number.

 -- Method: aifc.close ()
     Close the AIFF file.  After calling this method, the object can no
     longer be used.

  Objects returned by open() when a file is opened for
writing have all the above methods, except for `readframes()' and
`setpos()'.  In addition the following methods exist.  The `get*()'
methods can only be called after the corresponding `set*()' methods
have been called.  Before the first `writeframes()' or
`writeframesraw()', all parameters except for the number of frames must
be filled in.

 -- Method: aifc.aiff ()
     Create an AIFF file.  The default is that an AIFF-C file is
     created, unless the name of the file ends in `'.aiff'' in which
     case the default is an AIFF file.

 -- Method: aifc.aifc ()
     Create an AIFF-C file.  The default is that an AIFF-C file is
     created, unless the name of the file ends in `'.aiff'' in which
     case the default is an AIFF file.

 -- Method: aifc.setnchannels (nchannels)
     Specify the number of channels in the audio file.

 -- Method: aifc.setsampwidth (width)
     Specify the size in bytes of audio samples.

 -- Method: aifc.setframerate (rate)
     Specify the sampling frequency in frames per second.

 -- Method: aifc.setnframes (nframes)
     Specify the number of frames that are to be written to the audio
     file. If this parameter is not set, or not set correctly, the file
     needs to support seeking.

 -- Method: aifc.setcomptype (type, name)
     Specify the compression type.  If not specified, the audio data
     will not be compressed.  In AIFF files, compression is not
     possible.  The name parameter should be a human-readable
     description of the compression type, the type parameter should be
     a four-character string.  Currently the following compression
     types are supported: NONE, ULAW, ALAW, G722.

 -- Method: aifc.setparams (nchannels, sampwidth, framerate, comptype,
          compname)
     Set all the above parameters at once.  The argument is a tuple
     consisting of the various parameters.  This means that it is
     possible to use the result of a getparams() call as
     argument to setparams().

 -- Method: aifc.setmark (id, pos, name)
     Add a mark with the given id (larger than 0), and the given name
     at the given position.  This method can be called at any time
     before close().

 -- Method: aifc.tell ()
     Return the current write position in the output file.  Useful in
     combination with setmark().

 -- Method: aifc.writeframes (data)
     Write data to the output file.  This method can only be called
     after the audio file parameters have been set.

 -- Method: aifc.writeframesraw (data)
     Like writeframes(), except that the header of the
     audio file is not updated.

 -- Method: aifc.close ()
     Close the AIFF file.  The header of the file is updated to reflect
     the actual size of the audio data. After calling this method, the
     object can no longer be used.

5.21.4 `sunau' -- Read and write Sun AU files
---------------------------------------------

The sunau module provides a convenient interface to the Sun
AU sound format.  Note that this module is interface-compatible with
the modules aifc and wave.

  An audio file consists of a header followed by the data.  The fields
of the header are:

Field               Contents
------------------------------------------------------------------------
magic word          The four bytes `.snd'.
header size         Size of the header, including info, in bytes.
data size           Physical size of the data, in bytes.
encoding            Indicates how the audio samples are encoded.
sample rate         The sampling rate.
# of channels       The number of channels in the samples.
info                ASCII string giving a description of the audio
                    file (padded with null bytes).

  Apart from the info field, all header fields are 4 bytes in size.
They are all 32-bit unsigned integers encoded in big-endian byte order.

  The sunau module defines the following functions:

 -- Function: sunau.open (file, mode)
     If _file_ is a string, open the file by that name, otherwise treat
     it as a seekable file-like object. _mode_ can be any of

    `'r''
          Read only mode.

    `'w''
          Write only mode.

     Note that it does not allow read/write files.

     A _mode_ of `'r'' returns a `AU_read' object, while a _mode_ of
     `'w'' or `'wb'' returns a `AU_write' object.

 -- Function: sunau.openfp (file, mode)
     A synonym for open(), maintained for backwards
     compatibility.

  The sunau module defines the following exception:

 -- Exception: sunau.Error
     An error raised when something is impossible because of Sun AU
     specs or implementation deficiency.

  The sunau module defines the following data items:

 -- Data: sunau.AUDIO_FILE_MAGIC
     An integer every valid Sun AU file begins with, stored in
     big-endian form.  This is the string `.snd' interpreted as an
     integer.

 -- Data: sunau.AUDIO_FILE_ENCODING_MULAW_8
 -- Data: sunau.AUDIO_FILE_ENCODING_LINEAR_8
 -- Data: sunau.AUDIO_FILE_ENCODING_LINEAR_16
 -- Data: sunau.AUDIO_FILE_ENCODING_LINEAR_24
 -- Data: sunau.AUDIO_FILE_ENCODING_LINEAR_32
 -- Data: sunau.AUDIO_FILE_ENCODING_ALAW_8
     Values of the encoding field from the AU header which are
     supported by this module.

 -- Data: sunau.AUDIO_FILE_ENCODING_FLOAT
 -- Data: sunau.AUDIO_FILE_ENCODING_DOUBLE
 -- Data: sunau.AUDIO_FILE_ENCODING_ADPCM_G721
 -- Data: sunau.AUDIO_FILE_ENCODING_ADPCM_G722
 -- Data: sunau.AUDIO_FILE_ENCODING_ADPCM_G723_3
 -- Data: sunau.AUDIO_FILE_ENCODING_ADPCM_G723_5
     Additional known values of the encoding field from the AU header,
     but which are not supported by this module.

5.21.4.1 AU_read Objects
........................

AU_read objects, as returned by open() above, have the
following methods:

 -- Method: AU_read.close ()
     Close the stream, and make the instance unusable. (This is  called
     automatically on deletion.)

 -- Method: AU_read.getnchannels ()
     Returns number of audio channels (1 for mone, 2 for stereo).

 -- Method: AU_read.getsampwidth ()
     Returns sample width in bytes.

 -- Method: AU_read.getframerate ()
     Returns sampling frequency.

 -- Method: AU_read.getnframes ()
     Returns number of audio frames.

 -- Method: AU_read.getcomptype ()
     Returns compression type. Supported compression types are
     `'ULAW'', `'ALAW'' and `'NONE''.

 -- Method: AU_read.getcompname ()
     Human-readable version of getcomptype().  The
     supported types have the respective names `'CCITT G.711 u-law'',
     `'CCITT G.711 A-law'' and `'not compressed''.

 -- Method: AU_read.getparams ()
     Returns a tuple `(nchannels, sampwidth, framerate, nframes,
     comptype, compname)', equivalent to output of the `get*()' methods.

 -- Method: AU_read.readframes (n)
     Reads and returns at most _n_ frames of audio, as a string of
     bytes.  The data will be returned in linear format.  If the
     original data is in u-LAW format, it will be converted.

 -- Method: AU_read.rewind ()
     Rewind the file pointer to the beginning of the audio stream.

  The following two methods define a term "position" which is
compatible between them, and is otherwise implementation dependent.

 -- Method: AU_read.setpos (pos)
     Set the file pointer to the specified position.  Only values
     returned from tell() should be used for _pos_.

 -- Method: AU_read.tell ()
     Return current file pointer position.  Note that the returned
     value has nothing to do with the actual position in the file.

  The following two functions are defined for compatibility with the
aifc, and don't do anything interesting.

 -- Method: AU_read.getmarkers ()
     Returns `None'.

 -- Method: AU_read.getmark (id)
     Raise an error.

5.21.4.2 AU_write Objects
.........................

AU_write objects, as returned by open() above, have the
following methods:

 -- Method: AU_write.setnchannels (n)
     Set the number of channels.

 -- Method: AU_write.setsampwidth (n)
     Set the sample width (in bytes.)

 -- Method: AU_write.setframerate (n)
     Set the frame rate.

 -- Method: AU_write.setnframes (n)
     Set the number of frames. This can be later changed, when and if
     more  frames are written.

 -- Method: AU_write.setcomptype (type, name)
     Set the compression type and description. Only `'NONE'' and
     `'ULAW'' are supported on output.

 -- Method: AU_write.setparams (tuple)
     The _tuple_ should be `(nchannels, sampwidth, framerate, nframes,
     comptype, compname)', with values valid for the `set*()' methods.
     Set all parameters.

 -- Method: AU_write.tell ()
     Return current position in the file, with the same disclaimer for
     the AU_read.tell() and AU_read.setpos().
     methods.

 -- Method: AU_write.writeframesraw (data)
     Write audio frames, without correcting _nframes_.

 -- Method: AU_write.writeframes (data)
     Write audio frames and make sure _nframes_ is correct.

 -- Method: AU_write.close ()
     Make sure _nframes_ is correct, and close the file.

     This method is called upon deletion.

  Note that it is invalid to set any parameters after calling
`writeframes()' or `writeframesraw()'.

5.21.5 `wave' -- Read and write WAV files
-----------------------------------------

The wave module provides a convenient interface to the WAV
sound format.  It does not support compression/decompression, but it
does support mono/stereo.

  The wave module defines the following function and
exception:

 -- Function: wave.open (file[, mode])
     If _file_ is a string, open the file by that name, other treat it
     as a seekable file-like object. _mode_ can be any of

    `'r'', `'rb''
          Read only mode.

    `'w'', `'wb''
          Write only mode.

     Note that it does not allow read/write WAV files.

     A _mode_ of `'r'' or `'rb'' returns a `Wave_read' object, while a
     _mode_ of `'w'' or `'wb'' returns a `Wave_write' object.  If _mode_
     is omitted and a file-like  object is passed as _file_,
     `file.mode' is used as the default value for _mode_ (the `'b''
     flag is still added if  necessary).

 -- Function: wave.openfp (file, mode)
     A synonym for open(), maintained for backwards
     compatibility.

 -- Exception: wave.Error
     An error raised when something is impossible because it violates
     the WAV specification or hits an implementation deficiency.

5.21.5.1 Wave_read Objects
..........................

Wave_read objects, as returned by open(), have the
following methods:

 -- Method: Wave_read.close ()
     Close the stream, and make the instance unusable. This is called
     automatically on object collection.

 -- Method: Wave_read.getnchannels ()
     Returns number of audio channels (`1' for mono, `2' for stereo).

 -- Method: Wave_read.getsampwidth ()
     Returns sample width in bytes.

 -- Method: Wave_read.getframerate ()
     Returns sampling frequency.

 -- Method: Wave_read.getnframes ()
     Returns number of audio frames.

 -- Method: Wave_read.getcomptype ()
     Returns compression type (`'NONE'' is the only supported type).

 -- Method: Wave_read.getcompname ()
     Human-readable version of getcomptype(). Usually `'not
     compressed'' parallels `'NONE''.

 -- Method: Wave_read.getparams ()
     Returns a tuple `(nchannels, sampwidth, framerate, nframes,
     comptype, compname)', equivalent to output of the `get*()' methods.

 -- Method: Wave_read.readframes (n)
     Reads and returns at most _n_ frames of audio, as a string of
     bytes.

 -- Method: Wave_read.rewind ()
     Rewind the file pointer to the beginning of the audio stream.

  The following two methods are defined for compatibility with the
aifc.  module, and don't do anything interesting.

 -- Method: Wave_read.getmarkers ()
     Returns `None'.

 -- Method: Wave_read.getmark (id)
     Raise an error.

  The following two methods define a term "position" which is
compatible between them, and is otherwise implementation dependent.

 -- Method: Wave_read.setpos (pos)
     Set the file pointer to the specified position.

 -- Method: Wave_read.tell ()
     Return current file pointer position.

5.21.5.2 Wave_write Objects
...........................

Wave_write objects, as returned by open(), have the
following methods:

 -- Method: Wave_write.close ()
     Make sure _nframes_ is correct, and close the file. This method is
     called upon deletion.

 -- Method: Wave_write.setnchannels (n)
     Set the number of channels.

 -- Method: Wave_write.setsampwidth (n)
     Set the sample width to _n_ bytes.

 -- Method: Wave_write.setframerate (n)
     Set the frame rate to _n_.

 -- Method: Wave_write.setnframes (n)
     Set the number of frames to _n_. This will be changed later if
     more frames are written.

 -- Method: Wave_write.setcomptype (type, name)
     Set the compression type and description. At the moment, only
     compression type `NONE' is supported, meaning no compression.

 -- Method: Wave_write.setparams (tuple)
     The _tuple_ should be `(nchannels, sampwidth, framerate, nframes,
     comptype, compname)', with values valid for the `set*()' methods.
     Sets all parameters.

 -- Method: Wave_write.tell ()
     Return current position in the file, with the same disclaimer for
     the Wave_read.tell() and Wave_read.setpos()
     methods.

 -- Method: Wave_write.writeframesraw (data)
     Write audio frames, without correcting _nframes_.

 -- Method: Wave_write.writeframes (data)
     Write audio frames and make sure _nframes_ is correct.

  Note that it is invalid to set any parameters after calling
`writeframes()' or `writeframesraw()', and any attempt to do so will
raise wave.Error.

5.21.6 `chunk' -- Read IFF chunked data
---------------------------------------

This module provides an interface for reading files that use EA IFF 85
chunks.  (1)  This format is used in at least the Audio Interchange
File Format (AIFF/AIFF-C) and the Real Media File Format (RMFF).  The
WAVE audio file format is closely related and can also be read using
this module.

  A chunk has the following structure:

Offset        Length       Contents
---------------------------------------------------------------
0             4            Chunk ID
4             4            Size of chunk in big-endian byte
                           order, not including the header
8             _n_          Data bytes, where _n_ is the size
                           given in the preceding field
8 + _n_       0 or 1       Pad byte needed if _n_ is odd and
                           chunk alignment is used

  The ID is a 4-byte string which identifies the type of chunk.

  The size field (a 32-bit value, encoded using big-endian byte order)
gives the size of the chunk data, not including the 8-byte header.

  Usually an IFF-type file consists of one or more chunks.  The
proposed usage of the Chunk class defined here is to
instantiate an instance at the start of each chunk and read from the
instance until it reaches the end, after which a new instance can be
instantiated. At the end of the file, creating a new instance will fail
with a EOFError exception.

 -- Class: chunk.Chunk (file[, align, bigendian, inclheader])
     Class which represents a chunk.  The _file_ argument is expected
     to be a file-like object.  An instance of this class is
     specifically allowed.  The only method that is needed is
     read().  If the methods seek() and
     tell() are present and don't raise an exception, they are
     also used.  If these methods are present and raise an exception,
     they are expected to not have altered the object.  If the optional
     argument _align_ is true, chunks are assumed to be aligned on
     2-byte boundaries.  If _align_ is false, no alignment is assumed.
     The default value is true.  If the optional argument _bigendian_
     is false, the chunk size is assumed to be in little-endian order.
     This is needed for WAVE audio files. The default value is true.
     If the optional argument _inclheader_ is true, the size given in
     the chunk header includes the size of the header.  The default
     value is false.

     A Chunk object supports the following methods:

      -- Method: getname ()
          Returns the name (ID) of the chunk.  This is the first 4
          bytes of the chunk.

      -- Method: getsize ()
          Returns the size of the chunk.

      -- Method: close ()
          Close and skip to the end of the chunk.  This does not close
          the underlying file.

     The remaining methods will raise IOError if called
     after the close() method has been called.

      -- Method: isatty ()
          Returns `False'.

      -- Method: seek (pos[, whence])
          Set the chunk's current position.  The _whence_ argument is
          optional and defaults to `0' (absolute file positioning);
          other values are `1' (seek relative to the current position)
          and `2' (seek relative to the file's end).  There is no
          return value. If the underlying file does not allow seek,
          only forward seeks are allowed.

      -- Method: tell ()
          Return the current position into the chunk.

      -- Method: read ([size])
          Read at most _size_ bytes from the chunk (less if the read
          hits the end of the chunk before obtaining _size_ bytes).  If
          the _size_ argument is negative or omitted, read all data
          until the end of the chunk.  The bytes are returned as a
          string object.  An empty string is returned when the end of
          the chunk is encountered immediately.

      -- Method: skip ()
          Skip to the end of the chunk.  All further calls to
          read() for the chunk will return `'''.  If you are not
          interested in the contents of the chunk, this method should
          be called so that the file points to the start of the next
          chunk.

Footnotes
.........

     (1) "EA IFF 85" Standard for Interchange Format Files, Jerry
     Morrison, Electronic Arts, January 1985.

5.21.7 `colorsys' -- Conversions between color systems
------------------------------------------------------

The colorsys module defines bidirectional conversions of
color values between colors expressed in the RGB (Red Green Blue) color
space used in computer monitors and three other coordinate systems:
YIQ, HLS (Hue Lightness Saturation) and HSV (Hue Saturation Value).
Coordinates in all of these color spaces are floating point values.  In
the YIQ space, the Y coordinate is between 0 and 1, but the I and Q
coordinates can be positive or negative.  In all other spaces, the
coordinates are all between 0 and 1.

  More information about color spaces can be found at
<http://www.poynton.com/ColorFAQ.html> and
<http://www.cambridgeincolour.com/tutorials/color-spaces.htm>.

  The colorsys module defines the following functions:

 -- Function: colorsys.rgb_to_yiq (r, g, b)
     Convert the color from RGB coordinates to YIQ coordinates.

 -- Function: colorsys.yiq_to_rgb (y, i, q)
     Convert the color from YIQ coordinates to RGB coordinates.

 -- Function: colorsys.rgb_to_hls (r, g, b)
     Convert the color from RGB coordinates to HLS coordinates.

 -- Function: colorsys.hls_to_rgb (h, l, s)
     Convert the color from HLS coordinates to RGB coordinates.

 -- Function: colorsys.rgb_to_hsv (r, g, b)
     Convert the color from RGB coordinates to HSV coordinates.

 -- Function: colorsys.hsv_to_rgb (h, s, v)
     Convert the color from HSV coordinates to RGB coordinates.

  Example:

    >>> import colorsys
    >>> colorsys.rgb_to_hsv(.3, .4, .2)
    (0.25, 0.5, 0.4)
    >>> colorsys.hsv_to_rgb(0.25, 0.5, 0.4)
    (0.3, 0.4, 0.2)

5.21.8 `imghdr' -- Determine the type of an image
-------------------------------------------------

The imghdr module determines the type of image contained in
a file or byte stream.

  The imghdr module defines the following function:

 -- Function: imghdr.what (filename[, h])
     Tests the image data contained in the file named by _filename_,
     and returns a string describing the image type.  If optional _h_
     is provided, the _filename_ is ignored and _h_ is assumed to
     contain the byte stream to test.

  The following image types are recognized, as listed below with the
return value from what().:

Value            Image format
---------------------------------------------------------
`'rgb''          SGI ImgLib Files
`'gif''          GIF 87a and 89a Files
`'pbm''          Portable Bitmap Files
`'pgm''          Portable Graymap Files
`'ppm''          Portable Pixmap Files
`'tiff''         TIFF Files
`'rast''         Sun Raster Files
`'xbm''          X Bitmap Files
`'jpeg''         JPEG data in JFIF or Exif formats
`'bmp''          BMP files
`'png''          Portable Network Graphics

  New in version 2.5: Exif detection.

  You can extend the list of file types imghdr can recognize
by appending to this variable:

 -- Data: imghdr.tests
     A list of functions performing the individual tests.  Each
     function takes two arguments: the byte-stream and an open
     file-like object. When what() is called with a
     byte-stream, the file-like object will be `None'.

     The test function should return a string describing the image type
     if the test succeeded, or `None' if it failed.

  Example:

    >>> import imghdr
    >>> imghdr.what('/tmp/bass.gif')
    'gif'

5.21.9 `sndhdr' -- Determine type of sound file
-----------------------------------------------

The sndhdr provides utility functions which attempt to
determine the type of sound data which is in a file.  When these
functions are able to determine what type of sound data is stored in a
file, they return a tuple `(type, sampling_rate, channels, frames,
bits_per_sample)'.  The value for _type_ indicates the data type and
will be one of the strings `'aifc'', `'aiff'', `'au'', `'hcom'',
`'sndr'', `'sndt'', `'voc'', `'wav'', `'8svx'', `'sb'', `'ub'', or
`'ul''.  The _sampling_rate_ will be either the actual value or `0' if
unknown or difficult to decode.  Similarly, _channels_ will be either
the number of channels or `0' if it cannot be determined or if the
value is difficult to decode.  The value for _frames_ will be either
the number of frames or `-1'.  The last item in the tuple,
_bits_per_sample_, will either be the sample size in bits or `'A'' for
A-LAW or `'U'' for u-LAW.

 -- Function: sndhdr.what (filename)
     Determines the type of sound data stored in the file _filename_
     using whathdr().  If it succeeds, returns a tuple as
     described above, otherwise `None' is returned.

 -- Function: sndhdr.whathdr (filename)
     Determines the type of sound data stored in a file based on the
     file  header.  The name of the file is given by _filename_.  This
     function returns a tuple as described above on success, or `None'.

5.21.10 `ossaudiodev' -- Access to OSS-compatible audio devices
---------------------------------------------------------------

New in version 2.3.

  This module allows you to access the OSS (Open Sound System) audio
interface.  OSS is available for a wide range of open-source and
commercial Unices, and is the standard audio interface for Linux and
recent versions of FreeBSD.

Open Sound System Programmer's Guide (http://www.opensound.com/pguide/oss.pdf)
     the official documentation for the OSS C API

  The module defines a large number of constants supplied by the OSS
device driver; see `<sys/soundcard.h>' on either Linux or FreeBSD for a
listing .

  ossaudiodev defines the following variables and functions:

 -- Exception: ossaudiodev.OSSAudioError
     This exception is raised on certain errors.  The argument is a
     string describing what went wrong.

     (If ossaudiodev receives an error from a system call
     such as `open()', `write()', or `ioctl()', it raises
     IOError.  Errors detected directly by ossaudiodev.
     result in OSSAudioError)

     (For backwards compatibility, the exception class is also
     available as `ossaudiodev.error'.)

 -- Function: ossaudiodev.open ([device], mode)
     Open an audio device and return an OSS audio device object.  This
     object supports many file-like methods, such as `read()',
     `write()', and `fileno()' (although there are subtle differences
     between conventional Unix read/write semantics and those of OSS
     audio devices).  It also supports a number of audio-specific
     methods; see below for the complete list of methods.

     _device_ is the audio device filename to use.  If it is not
     specified, this module first looks in the environment variable `AUDIODEV'
     for a device to use.  If not found, it falls back to `/dev/dsp'.

     _mode_ is one of `'r'' for read-only (record) access, `'w'' for
     write-only (playback) access and `'rw'' for both. Since many sound
     cards only allow one process to have the recorder or player open
     at a time, it is a good idea to open the device only for the
     activity needed.  Further, some sound cards are half-duplex: they
     can be opened for reading or writing, but not both at once.

     Note the unusual calling syntax: the _first_ argument is optional,
     and the second is required.  This is a historical artifact for
     compatibility with the older `linuxaudiodev' module which
     ossaudiodev supersedes.

 -- Function: ossaudiodev.openmixer ([device])
     Open a mixer device and return an OSS mixer device object.
     _device_ is the mixer device filename to use.  If it is not
     specified, this module first looks in the environment variable `MIXERDEV'
     for a device to use.  If not found, it falls back to `/dev/mixer'.

5.21.10.1 Audio Device Objects
..............................

Before you can write to or read from an audio device, you must call
three methods in the correct order:

  1. `setfmt()' to set the output format

  2. `channels()' to set the number of channels

  3. `speed()' to set the sample rate

  Alternately, you can use the `setparameters()' method to set all
three audio parameters at once.  This is more convenient, but may not
be as flexible in all cases.

  The audio device objects returned by open() define the
following methods and (read-only) attributes:

 -- Method: oss_audio_device.close ()
     Explicitly close the audio device.  When you are done writing to
     or reading from an audio device, you should explicitly close it.
     A closed device cannot be used again.

 -- Method: oss_audio_device.fileno ()
     Return the file descriptor associated with the device.

 -- Method: oss_audio_device.read (size)
     Read _size_ bytes from the audio input and return them as a Python
     string.  Unlike most Unix device drivers, OSS audio devices in
     blocking mode (the default) will block read() until
     the entire requested amount of data is available.

 -- Method: oss_audio_device.write (data)
     Write the Python string _data_ to the audio device and return the
     number of bytes written.  If the audio device is in blocking mode
     (the default), the entire string is always written (again, this is
     different from usual Unix device semantics).  If the device is in
     non-blocking mode, some data may not be written --see
     writeall().

 -- Method: oss_audio_device.writeall (data)
     Write the entire Python string _data_ to the audio device: waits
     until the audio device is able to accept data, writes as much data
     as it will accept, and repeats until _data_ has been completely
     written. If the device is in blocking mode (the default), this has
     the same effect as write().; writeall().
     is only useful in non-blocking mode.  Has no return value, since
     the amount of data written is always equal to the amount of data
     supplied.

  The following methods each map to exactly one `ioctl()' system call.
The correspondence is obvious: for example, `setfmt()' corresponds to
the `SNDCTL_DSP_SETFMT' ioctl, and `sync()' to `SNDCTL_DSP_SYNC' (this
can be useful when consulting the OSS documentation).  If the underlying
`ioctl()' fails, they all raise IOError.

 -- Method: oss_audio_device.nonblock ()
     Put the device into non-blocking mode.  Once in non-blocking mode,
     there is no way to return it to blocking mode.

 -- Method: oss_audio_device.getfmts ()
     Return a bitmask of the audio output formats supported by the
     soundcard.  Some of the formats supported by OSS are:

     Format                        Description
     --------------------------------------------------------------------------------
     `AFMT_MU_LAW'                 a logarithmic encoding (used by Sun `.au' files
                                   and `/dev/audio')
     `AFMT_A_LAW'                  a logarithmic encoding
     `AFMT_IMA_ADPCM'              a 4:1 compressed format defined by the
                                   Interactive Multimedia Association
     `AFMT_U8'                     Unsigned, 8-bit audio
     `AFMT_S16_LE'                 Signed, 16-bit audio, little-endian byte order
                                   (as used by Intel processors)
     `AFMT_S16_BE'                 Signed, 16-bit audio, big-endian byte order (as
                                   used by 68k, PowerPC, Sparc)
     `AFMT_S8'                     Signed, 8 bit audio
     `AFMT_U16_LE'                 Unsigned, 16-bit little-endian audio
     `AFMT_U16_BE'                 Unsigned, 16-bit big-endian audio

     Consult the OSS documentation for a full list of audio formats,
     and note that most devices support only a subset of these formats.
     Some older devices only support `AFMT_U8'; the most common format
     used today is `AFMT_S16_LE'.

 -- Method: oss_audio_device.setfmt (format)
     Try to set the current audio format to _format_--see
     getfmts() for a list.  Returns the audio format that the
     device was set to, which may not be the requested format.  May
     also be used to return the current audio format--do this by
     passing an "audio format" of `AFMT_QUERY'.

 -- Method: oss_audio_device.channels (nchannels)
     Set the number of output channels to _nchannels_.  A value of 1
     indicates monophonic sound, 2 stereophonic.  Some devices may have
     more than 2 channels, and some high-end devices may not support
     mono. Returns the number of channels the device was set to.

 -- Method: oss_audio_device.speed (samplerate)
     Try to set the audio sampling rate to _samplerate_ samples per
     second.  Returns the rate actually set.  Most sound devices don't
     support arbitrary sampling rates.  Common rates are:

     Rate        Description
     ------------------------------------------------------------
     8000        default rate for `/dev/audio'
     11025       speech recording
     22050
     44100       CD quality audio (at 16 bits/sample and 2
                 channels)
     96000       DVD quality audio (at 24 bits/sample)

 -- Method: oss_audio_device.sync ()
     Wait until the sound device has played every byte in its buffer.
     (This happens implicitly when the device is closed.)  The OSS
     documentation recommends closing and re-opening the device rather
     than using sync().

 -- Method: oss_audio_device.reset ()
     Immediately stop playing or recording and return the device to a
     state where it can accept commands.  The OSS documentation
     recommends closing and re-opening the device after calling
     reset().

 -- Method: oss_audio_device.post ()
     Tell the driver that there is likely to be a pause in the output,
     making it possible for the device to handle the pause more
     intelligently.  You might use this after playing a spot sound
     effect, before waiting for user input, or before doing disk I/O.

  The following convenience methods combine several ioctls, or one
ioctl and some simple calculations.

 -- Method: oss_audio_device.setparameters (format, nchannels,
          samplerate[, strict=False])
     Set the key audio sampling parameters--sample format, number of
     channels, and sampling rate--in one method call.  _format_,
     _nchannels_, and _samplerate_ should be as specified in the
     setfmt(), channels(), and speed().
     methods.  If _strict_ is true, see setparameters() checks
     to see if each parameter was actually set to the requested value,
     and raises OSSAudioError if not.  Returns a tuple
     (_format_, _nchannels_, _samplerate_) indicating the parameter
     values that were actually set by the device driver (i.e., the same
     as the return values of setfmt(), channels(),
     and speed()).

     For example,

         (fmt, channels, rate) = dsp.setparameters(fmt, channels, rate)

     is equivalent to

         fmt = dsp.setfmt(fmt)
         channels = dsp.channels(channels)
         rate = dsp.rate(channels)

 -- Method: oss_audio_device.bufsize ()
     Returns the size of the hardware buffer, in samples.

 -- Method: oss_audio_device.obufcount ()
     Returns the number of samples that are in the hardware buffer yet
     to be played.

 -- Method: oss_audio_device.obuffree ()
     Returns the number of samples that could be queued into the
     hardware buffer to be played without blocking.

  Audio device objects also support several read-only attributes:

 -- Attribute: oss_audio_device.closed
     Boolean indicating whether the device has been closed.

 -- Attribute: oss_audio_device.name
     String containing the name of the device file.

 -- Attribute: oss_audio_device.mode
     The I/O mode for the file, either `"r"', `"rw"', or `"w"'.

5.21.10.2 Mixer Device Objects
..............................

The mixer object provides two file-like methods:

 -- Method: oss_mixer_device.close ()
     This method closes the open mixer device file.  Any further
     attempts to use the mixer after this file is closed will raise an
     IOError.

 -- Method: oss_mixer_device.fileno ()
     Returns the file handle number of the open mixer device file.

  The remaining methods are specific to audio mixing:

 -- Method: oss_mixer_device.controls ()
     This method returns a bitmask specifying the available mixer
     controls ("Control" being a specific mixable "channel", such as
     `SOUND_MIXER_PCM' or `SOUND_MIXER_SYNTH').  This bitmask indicates
     a subset of all available mixer controls--the `SOUND_MIXER_*'
     constants defined at module level.  To determine if, for example,
     the current mixer object supports a PCM mixer, use the following
     Python code:

         mixer=ossaudiodev.openmixer()
         if mixer.controls() & (1 << ossaudiodev.SOUND_MIXER_PCM):
             # PCM is supported
             ... code ...

     For most purposes, the `SOUND_MIXER_VOLUME' (master volume) and
     `SOUND_MIXER_PCM' controls should suffice--but code that uses the
     mixer should be flexible when it comes to choosing mixer controls.
     On the Gravis Ultrasound, for example, `SOUND_MIXER_VOLUME' does
     not exist.

 -- Method: oss_mixer_device.stereocontrols ()
     Returns a bitmask indicating stereo mixer controls.  If a bit is
     set, the corresponding control is stereo; if it is unset, the
     control is either monophonic or not supported by the mixer (use in
     combination with controls() to determine which).

     See the code example for the controls() function for
     an example of getting data from a bitmask.

 -- Method: oss_mixer_device.reccontrols ()
     Returns a bitmask specifying the mixer controls that may be used
     to record.  See the code example for controls() for
     an example of reading from a bitmask.

 -- Method: oss_mixer_device.get (control)
     Returns the volume of a given mixer control.  The returned volume
     is a 2-tuple `(left_volume,right_volume)'.  Volumes are specified
     as numbers from 0 (silent) to 100 (full volume).  If the control
     is monophonic, a 2-tuple is still returned, but both volumes are
     the same.

     Raises OSSAudioError if an invalid control was is
     specified, or IOError if an unsupported control is
     specified.

 -- Method: oss_mixer_device.set (control, (left, right))
     Sets the volume for a given mixer control to `(left,right)'.
     `left' and `right' must be ints and between 0 (silent) and 100
     (full volume).  On success, the new volume is returned as a
     2-tuple. Note that this may not be exactly the same as the volume
     specified, because of the limited resolution of some soundcard's
     mixers.

     Raises OSSAudioError if an invalid mixer control was
     specified, or if the specified volumes were out-of-range.

 -- Method: oss_mixer_device.get_recsrc ()
     This method returns a bitmask indicating which control(s) are
     currently being used as a recording source.

 -- Method: oss_mixer_device.set_recsrc (bitmask)
     Call this function to specify a recording source.  Returns a
     bitmask indicating the new recording source (or sources) if
     successful; raises IOError if an invalid source was
     specified.  To set the current recording source to the microphone
     input:

         mixer.setrecsrc (1 << ossaudiodev.SOUND_MIXER_MIC)

5.22 Internationalization
=========================

The modules described in this chapter help you write software that is
independent of language and locale by providing mechanisms for
selecting a language to be used in  program messages or by tailoring
output to match local conventions.

  The list of modules described in this chapter is:

5.22.1 `gettext' -- Multilingual internationalization services
--------------------------------------------------------------

The gettext module provides internationalization (I18N) and
localization (L10N) services for your Python modules and applications.
It supports both the GNU `gettext' message catalog API and a higher
level, class-based API that may be more appropriate for Python files.
The interface described below allows you to write your module and
application messages in one natural language, and provide a catalog of
translated messages for running under different natural languages.

  Some hints on localizing your Python modules and applications are
also given.

5.22.1.1 GNU *gettext* API
..........................

The gettext module defines the following API, which is very
similar to the GNU *gettext* API.  If you use this API you will affect
the translation of your entire application globally.  Often this is
what you want if your application is monolingual, with the choice of
language dependent on the locale of your user.  If you are localizing a
Python module, or if your application needs to switch languages on the
fly, you probably want to use the class-based API instead.

 -- Function: gettext.bindtextdomain (domain[, localedir])
     Bind the _domain_ to the locale directory _localedir_.  More
     concretely, see gettext will look for binary `.mo' files
     for the given domain using the path (on Unix):
     `localedir/language/LC_MESSAGES/domain.mo', where _languages_ is
     searched for in the environment variables `LANGUAGE', `LC_ALL', `LC_MESSAGES',
     and `LANG' respectively.

     If _localedir_ is omitted or `None', then the current binding for
     _domain_ is returned. (1)

 -- Function: gettext.bind_textdomain_codeset (domain[, codeset])
     Bind the _domain_ to _codeset_, changing the encoding of strings
     returned by the gettext() family of functions. If
     _codeset_ is omitted, then the current binding is returned.

     New in version 2.4.

 -- Function: gettext.textdomain ([domain])
     Change or query the current global domain.  If _domain_ is `None',
     then the current global domain is returned, otherwise the global
     domain is set to _domain_, which is returned.

 -- Function: gettext.gettext (message)
     Return the localized translation of _message_, based on the
     current global domain, language, and locale directory.  This
     function is usually aliased as `_()' in the local namespace (see
     examples below).

 -- Function: gettext.lgettext (message)
     Equivalent to gettext(), but the translation is returned
     in the preferred system encoding, if no other encoding was
     explicitly set with bind_textdomain_codeset().

     New in version 2.4.

 -- Function: gettext.dgettext (domain, message)
     Like gettext(), but look the message up in the specified
     _domain_.

 -- Function: gettext.ldgettext (domain, message)
     Equivalent to dgettext(), but the translation is
     returned in the preferred system encoding, if no other encoding
     was explicitly set with bind_textdomain_codeset().

     New in version 2.4.

 -- Function: gettext.ngettext (singular, plural, n)
     Like gettext(), but consider plural forms. If a
     translation is found, apply the plural formula to _n_, and return
     the resulting message (some languages have more than two plural
     forms). If no translation is found, return _singular_ if _n_ is 1;
     return _plural_ otherwise.

     The Plural formula is taken from the catalog header. It is a C or
     Python expression that has a free variable _n_; the expression
     evaluates to the index of the plural in the catalog. See the GNU
     gettext documentation for the precise syntax to be used in `.po'
     files and the formulas for a variety of languages.

     New in version 2.3.

 -- Function: gettext.lngettext (singular, plural, n)
     Equivalent to ngettext(), but the translation is
     returned in the preferred system encoding, if no other encoding
     was explicitly set with bind_textdomain_codeset().

     New in version 2.4.

 -- Function: gettext.dngettext (domain, singular, plural, n)
     Like ngettext(), but look the message up in the
     specified _domain_.

     New in version 2.3.

 -- Function: gettext.ldngettext (domain, singular, plural, n)
     Equivalent to dngettext(), but the translation is
     returned in the preferred system encoding, if no other encoding
     was explicitly set with bind_textdomain_codeset().

     New in version 2.4.

  Note that GNU *gettext* also defines a `dcgettext()' method, but this
was deemed not useful and so it is currently unimplemented.

  Here's an example of typical usage for this API:

    import gettext
    gettext.bindtextdomain('myapplication', '/path/to/my/language/directory')
    gettext.textdomain('myapplication')
    _ = gettext.gettext
    # ...
    print _('This is a translatable string.')

5.22.1.2 Class-based API
........................

The class-based API of the gettext module gives you more
flexibility and greater convenience than the GNU *gettext* API.  It is
the recommended way of localizing your Python applications and modules.
gettext defines a "translations" class which implements the
parsing of GNU `.mo' format files, and has methods for returning either
standard 8-bit strings or Unicode strings. Instances of this
"translations" class can also install themselves  in the built-in
namespace as the function `_()'.

 -- Function: gettext.find (domain[, localedir[, languages[, all]]])
     This function implements the standard `.mo' file search algorithm.
     It takes a _domain_, identical to what textdomain().
     takes.  Optional _localedir_ is as in bindtextdomain().
     Optional _languages_ is a list of strings, where each
     string is a language code.

     If _localedir_ is not given, then the default system locale
     directory is used.  (2)  If _languages_ is not given, then the
     following environment variables are searched: `LANGUAGE', `LC_ALL', `LC_MESSAGES',
     and `LANG'.  The first one returning a non-empty value is used for
     the _languages_ variable. The environment variables should contain
     a colon separated list of languages, which will be split on the
     colon to produce the expected list of language code strings.

     find() then expands and normalizes the languages, and
     then iterates through them, searching for an existing file built
     of these components:

     `localedir/language/LC_MESSAGES/domain.mo'

     The first such file name that exists is returned by find().
     If no such file is found, then `None' is returned. If _all_
     is given, it returns a list of all file names, in the order in
     which they appear in the languages list or the environment
     variables.

 -- Function: gettext.translation (domain[, localedir[, languages[,
          class_[, fallback[, codeset]]]]])
     Return a `Translations' instance based on the _domain_,
     _localedir_, and _languages_, which are first passed to
     find() to get a list of the associated `.mo' file paths.
     Instances with identical `.mo' file names are cached.  The actual
     class instantiated is either _class__ if provided, otherwise
     `GNUTranslations'.  The class's constructor must take a single
     file object argument. If provided, _codeset_ will change the
     charset used to encode translated strings.

     If multiple files are found, later files are used as fallbacks for
     earlier ones.  To allow setting the fallback, see copy.copy()
     is used to clone each translation object from the cache; the
     actual instance data is still shared with the cache.

     If no `.mo' file is found, this function raises IOError
     if _fallback_ is false (which is the default), and returns a
     NullTranslations instance if _fallback_ is true.

     Changed in version 2.4: Added the _codeset_ parameter.

 -- Function: gettext.install (domain[, localedir[, unicode[, codeset[,
          names]]]])
     This installs the function `_()' in Python's builtins namespace,
     based on _domain_, _localedir_, and _codeset_ which are passed to
     the function translation().  The _unicode_ flag is
     passed to the resulting translation object's install()
     method.

     For the _names_ parameter, please see the description of the
     translation object's install() method.

     As seen below, you usually mark the strings in your application
     that are candidates for translation, by wrapping them in a call to
     the `_()' function, like this:

         print _('This string will be translated.')

     For convenience, you want the `_()' function to be installed in
     Python's builtins namespace, so it is easily accessible in all
     modules of your application.

     Changed in version 2.4: Added the _codeset_ parameter.

     Changed in version 2.5: Added the _names_ parameter.

5.22.1.3 The `NullTranslations' class
.....................................

Translation classes are what actually implement the translation of
original source file message strings to translated message strings. The
base class used by all translation classes is NullTranslations.;
this provides the basic interface you can use to write your own
specialized translation classes.  Here are the methods of
NullTranslations.:

 -- Class: gettext.NullTranslations ([fp])
     Takes an optional file object _fp_, which is ignored by the base
     class.  Initializes "protected" instance variables __info_ and
     __charset_ which are set by derived classes, as well as
     __fallback_, which is set through add_fallback().  It
     then calls `self._parse(fp)' if _fp_ is not `None'.

      -- Method: _parse (fp)
          No-op'd in the base class, this method takes file object
          _fp_, and reads the data from the file, initializing its
          message catalog.  If you have an unsupported message catalog
          file format, you should override this method to parse your
          format.

      -- Method: add_fallback (fallback)
          Add _fallback_ as the fallback object for the current
          translation object. A translation object should consult the
          fallback if it cannot provide a translation for a given
          message.

      -- Method: gettext (message)
          If a fallback has been set, forward gettext() to
          the fallback. Otherwise, return the translated message.
          Overridden in derived classes.

      -- Method: lgettext (message)
          If a fallback has been set, forward lgettext().
          to the fallback. Otherwise, return the translated message.
          Overridden in derived classes.

          New in version 2.4.

      -- Method: ugettext (message)
          If a fallback has been set, forward ugettext().
          to the fallback. Otherwise, return the translated message as
          a Unicode string. Overridden in derived classes.

      -- Method: ngettext (singular, plural, n)
          If a fallback has been set, forward ngettext().
          to the fallback. Otherwise, return the translated message.
          Overridden in derived classes.

          New in version 2.3.

      -- Method: lngettext (singular, plural, n)
          If a fallback has been set, forward ngettext().
          to the fallback. Otherwise, return the translated message.
          Overridden in derived classes.

          New in version 2.4.

      -- Method: ungettext (singular, plural, n)
          If a fallback has been set, forward ungettext().
          to the fallback.  Otherwise, return the translated message as
          a Unicode string. Overridden in derived classes.

          New in version 2.3.

      -- Method: info ()
          Return the "protected" `_info' variable.

      -- Method: charset ()
          Return the "protected" `_charset' variable.

      -- Method: output_charset ()
          Return the "protected" `_output_charset' variable, which
          defines the encoding used to return translated messages.

          New in version 2.4.

      -- Method: set_output_charset (charset)
          Change the "protected" `_output_charset' variable, which
          defines the encoding used to return translated messages.

          New in version 2.4.

      -- Method: install ([unicode[, names]])
          If the _unicode_ flag is false, this method installs
          `self.gettext()' into the built-in namespace, binding it to
          `_'.  If _unicode_ is true, it binds `self.ugettext()'
          instead.  By default, _unicode_ is false.

          If the _names_ parameter is given, it must be a sequence
          containing the names of functions you want to install in the
          builtins namespace in addition to `_()'.  Supported names are
          `'gettext'' (bound to `self.gettext()' or `self.ugettext()'
          according to the _unicode_ flag), `'ngettext'' (bound to
          `self.ngettext()' or `self.ungettext()' according to the
          _unicode_ flag), `'lgettext'' and `'lngettext''.

          Note that this is only one way, albeit the most convenient
          way, to make the `_()' function available to your
          application.  Because it affects the entire application
          globally, and specifically the built-in namespace, localized
          modules should never install `_()'. Instead, they should use
          this code to make `_()' available to their module:

              import gettext
              t = gettext.translation('mymodule', ...)
              _ = t.gettext

          This puts `_()' only in the module's global namespace and so
          only affects calls within this module.

          Changed in version 2.5: Added the _names_ parameter.

5.22.1.4 The `GNUTranslations' class
....................................

The gettext module provides one additional class derived from
NullTranslations.: `GNUTranslations'.  This class overrides
`_parse()' to enable reading GNU *gettext* format `.mo' files in both
big-endian and little-endian format. It also coerces both message ids
and message strings to Unicode.

  `GNUTranslations' parses optional meta-data out of the translation
catalog.  It is convention with GNU *gettext* to include meta-data as
the translation for the empty string.  This meta-data is in RFC 822
(http://tools.ietf.org/html/rfc822.html)-style `key: value' pairs, and
should contain the `Project-Id-Version' key.  If the key `Content-Type'
is found, then the `charset' property is used to initialize the
"protected" `_charset' instance variable, defaulting to `None' if not
found.  If the charset encoding is specified, then all message ids and
message strings read from the catalog are converted to Unicode using
this encoding.  The `ugettext()' method always returns a Unicode, while
the gettext() returns an encoded 8-bit string.  For the
message id arguments of both methods, either Unicode strings or 8-bit
strings containing only US-ASCII characters are acceptable.  Note that
the Unicode version of the methods (i.e. `ugettext()' and
`ungettext()') are the recommended interface to use for
internationalized Python programs.

  The entire set of key/value pairs are placed into a dictionary and
set as the "protected" `_info' instance variable.

  If the `.mo' file's magic number is invalid, or if other problems
occur while reading the file, instantiating a `GNUTranslations' class
can raise IOError.

  The following methods are overridden from the base class
implementation:

 -- Method: GNUTranslations.gettext (message)
     Look up the _message_ id in the catalog and return the
     corresponding message string, as an 8-bit string encoded with the
     catalog's charset encoding, if known.  If there is no entry in the
     catalog for the _message_ id, and a fallback has been set, the
     look up is forwarded to the fallback's gettext() method.
     Otherwise, the _message_ id is returned.

 -- Method: GNUTranslations.lgettext (message)
     Equivalent to gettext(), but the translation is returned
     in the preferred system encoding, if no other encoding was
     explicitly set with `set_output_charset()'.

     New in version 2.4.

 -- Method: GNUTranslations.ugettext (message)
     Look up the _message_ id in the catalog and return the
     corresponding message string, as a Unicode string.  If there is no
     entry in the catalog for the _message_ id, and a fallback has been
     set, the look up is forwarded to the fallback's ugettext()
     method.  Otherwise, the _message_ id is returned.

 -- Method: GNUTranslations.ngettext (singular, plural, n)
     Do a plural-forms lookup of a message id.  _singular_ is used as
     the message id for purposes of lookup in the catalog, while _n_ is
     used to determine which plural form to use.  The returned message
     string is an 8-bit string encoded with the catalog's charset
     encoding, if known.

     If the message id is not found in the catalog, and a fallback is
     specified, the request is forwarded to the fallback's
     ngettext() method.  Otherwise, when _n_ is 1 _singular_ is
     returned, and _plural_ is returned in all other cases.

     New in version 2.3.

 -- Method: GNUTranslations.lngettext (singular, plural, n)
     Equivalent to gettext(), but the translation is returned
     in the preferred system encoding, if no other encoding was
     explicitly set with `set_output_charset()'.

     New in version 2.4.

 -- Method: GNUTranslations.ungettext (singular, plural, n)
     Do a plural-forms lookup of a message id.  _singular_ is used as
     the message id for purposes of lookup in the catalog, while _n_ is
     used to determine which plural form to use.  The returned message
     string is a Unicode string.

     If the message id is not found in the catalog, and a fallback is
     specified, the request is forwarded to the fallback's
     ungettext() method.  Otherwise, when _n_ is 1 _singular_ is
     returned, and _plural_ is returned in all other cases.

     Here is an example:

         n = len(os.listdir('.'))
         cat = GNUTranslations(somefile)
         message = cat.ungettext(
             'There is %(num)d file in this directory',
             'There are %(num)d files in this directory',
             n) % {'num': n}

     New in version 2.3.

5.22.1.5 Solaris message catalog support
........................................

The Solaris operating system defines its own binary `.mo' file format,
but since no documentation can be found on this format, it is not
supported at this time.

5.22.1.6 The Catalog constructor
................................

GNOME uses a version of the gettext module by James
Henstridge, but this version has a slightly different API.  Its
documented usage was:

    import gettext
    cat = gettext.Catalog(domain, localedir)
    _ = cat.gettext
    print _('hello world')

For compatibility with this older module, the function `Catalog()' is an
alias for the translation() function described above.

  One difference between this module and Henstridge's: his catalog
objects supported access through a mapping API, but this appears to be
unused and so is not currently supported.

5.22.1.7 Internationalizing your programs and modules
.....................................................

Internationalization (I18N) refers to the operation by which a program
is made aware of multiple languages.  Localization (L10N) refers to the
adaptation of your program, once internationalized, to the local
language and cultural habits.  In order to provide multilingual
messages for your Python programs, you need to take the following steps:

  1. prepare your program or module by specially marking translatable
     strings

  2. run a suite of tools over your marked files to generate raw
     messages catalogs

  3. create language specific translations of the message catalogs

  4. use the gettext module so that message strings are
     properly translated

  In order to prepare your code for I18N, you need to look at all the
strings in your files.  Any string that needs to be translated should
be marked by wrapping it in `_('...')' -- that is, a call to the
function `_()'.  For example:

    filename = 'mylog.txt'
    message = _('writing a log message')
    fp = open(filename, 'w')
    fp.write(message)
    fp.close()

In this example, the string `'writing a log message'' is marked as a
candidate for translation, while the strings `'mylog.txt'' and `'w''
are not.

  The Python distribution comes with two tools which help you generate
the message catalogs once you've prepared your source code.  These may
or may not be available from a binary distribution, but they can be
found in a source distribution, in the `Tools/i18n' directory.

  The *pygettext* (3) program scans all your Python source code looking
for the strings you previously marked as translatable.  It is similar
to the GNU *gettext* program except that it understands all the
intricacies of Python source code, but knows nothing about C or C++
source code.  You don't need GNU `gettext' unless you're also going to
be translating C code (such as C extension modules).

  *pygettext* generates textual Uniforum-style human readable message
catalog `.pot' files, essentially structured human readable files which
contain every marked string in the source code, along with a
placeholder for the translation strings. *pygettext* is a command line
script that supports a similar command line interface as *xgettext*;
for details on its use, run:

    pygettext.py --help

Copies of these `.pot' files are then handed over to the individual
human translators who write language-specific versions for every
supported natural language.  They send you back the filled in
language-specific versions as a `.po' file.  Using the *msgfmt.py* (4)
program (in the `Tools/i18n' directory), you take the `.po' files from
your translators and generate the machine-readable `.mo' binary catalog
files.  The `.mo' files are what the gettext module uses for
the actual translation processing during run-time.

  How you use the gettext module in your code depends on
whether you are internationalizing a single module or your entire
application. The next two sections will discuss each case.

5.22.1.8 Localizing your module
...............................

If you are localizing your module, you must take care not to make global
changes, e.g. to the built-in namespace.  You should not use the GNU
`gettext' API but instead the class-based API.

  Let's say your module is called "spam" and the module's various
natural language translation `.mo' files reside in `/usr/share/locale'
in GNU *gettext* format.  Here's what you would put at the top of your
module:

    import gettext
    t = gettext.translation('spam', '/usr/share/locale')
    _ = t.lgettext

If your translators were providing you with Unicode strings in their
`.po' files, you'd instead do:

    import gettext
    t = gettext.translation('spam', '/usr/share/locale')
    _ = t.ugettext

5.22.1.9 Localizing your application
....................................

If you are localizing your application, you can install the `_()'
function globally into the built-in namespace, usually in the main
driver file of your application.  This will let all your
application-specific files just use `_('...')' without having to
explicitly install it in each file.

  In the simple case then, you need only add the following bit of code
to the main driver file of your application:

    import gettext
    gettext.install('myapplication')

If you need to set the locale directory or the _unicode_ flag, you can
pass these into the install() function:

    import gettext
    gettext.install('myapplication', '/usr/share/locale', unicode=1)

5.22.1.10 Changing languages on the fly
.......................................

If your program needs to support many languages at the same time, you
may want to create multiple translation instances and then switch
between them explicitly, like so:

    import gettext

    lang1 = gettext.translation('myapplication', languages=['en'])
    lang2 = gettext.translation('myapplication', languages=['fr'])
    lang3 = gettext.translation('myapplication', languages=['de'])

    # start by using language1
    lang1.install()

    # ... time goes by, user selects language 2
    lang2.install()

    # ... more time goes by, user selects language 3
    lang3.install()

5.22.1.11 Deferred translations
...............................

In most coding situations, strings are translated where they are coded.
Occasionally however, you need to mark strings for translation, but
defer actual translation until later.  A classic example is:

    animals = ['mollusk',
               'albatross',
               'rat',
               'penguin',
               'python', ]
    # ...
    for a in animals:
        print a

Here, you want to mark the strings in the `animals' list as being
translatable, but you don't actually want to translate them until they
are printed.

  Here is one way you can handle this situation:

    def _(message): return message

    animals = [_('mollusk'),
               _('albatross'),
               _('rat'),
               _('penguin'),
               _('python'), ]

    del _

    # ...
    for a in animals:
        print _(a)

This works because the dummy definition of `_()' simply returns the
string unchanged.  And this dummy definition will temporarily override
any definition of `_()' in the built-in namespace (until the del
command). Take care, though if you have a previous definition of
`_()' in the local namespace.

  Note that the second use of `_()' will not identify "a" as being
translatable to the *pygettext* program, since it is not a string.

  Another way to handle this is with the following example:

    def N_(message): return message

    animals = [N_('mollusk'),
               N_('albatross'),
               N_('rat'),
               N_('penguin'),
               N_('python'), ]

    # ...
    for a in animals:
        print _(a)

In this case, you are marking translatable strings with the function
`N_()', (5) which won't conflict with any definition of `_()'.
However, you will need to teach your message extraction program to look
for translatable strings marked with `N_()'. *pygettext* and *xpot*
both support this through the use of command line switches.

5.22.1.12 `gettext()' vs. `lgettext()'
......................................

In Python 2.4 the lgettext() family of functions were
introduced. The intention of these functions is to provide an
alternative which is more compliant with the current implementation of
GNU gettext. Unlike gettext(), which returns strings encoded
with the same codeset used in the translation file, see lgettext()
will return strings encoded with the preferred system encoding,
as returned by locale.getpreferredencoding().  Also notice
that Python 2.4 introduces new functions to explicitly choose the
codeset used in translated strings. If a codeset is explicitly set, even
lgettext() will return translated strings in the requested
codeset, as would be expected in the GNU gettext implementation.

5.22.1.13 Acknowledgements
..........................

The following people contributed code, feedback, design suggestions,
previous implementations, and valuable experience to the creation of
this module:

   * Peter Funk

   * James Henstridge

   * Juan David Ibáñez Palomar

   * Marc-André Lemburg

   * Martin von Löwis

   * François Pinard

   * Barry Warsaw

   * Gustavo Niemeyer

Footnotes
.........

     (1) The default locale directory is system dependent; for example,
     on RedHat Linux it is `/usr/share/locale', but on Solaris it is
     `/usr/lib/locale'.  The gettext module does not try to
     support these system dependent defaults; instead its default is
     `sys.prefix/share/locale'. For this reason, it is always best to
     call bindtextdomain() with an explicit absolute path
     at the start of your application.

     (2) See the footnote for bindtextdomain() above.

     (3) François Pinard has written a program called *xpot* which does
     a similar job.  It is available as part of his *po-utils* package
     at http ://po-utils.progiciels-bpi.ca/.

     (4) *msgfmt.py* is binary compatible with GNU *msgfmt* except that
     it provides a simpler, all-Python implementation.  With this and
     *pygettext.py*, you generally won't need to install the GNU
     *gettext* package to internationalize your Python applications.

     (5) The choice of `N_()' here is totally arbitrary; it could have
     just as easily been `MarkThisStringForTranslation()'.

5.22.2 `locale' -- Internationalization services
------------------------------------------------

The locale module opens access to the POSIX locale database
and functionality. The POSIX locale mechanism allows programmers to
deal with certain cultural issues in an application, without requiring
the programmer to know all the specifics of each country where the
software is executed.

  The locale module is implemented on top of the `_locale'
module, which in turn uses an ANSI C locale implementation if available.

  The locale module defines the following exception and
functions:

 -- Exception: locale.Error
     Exception raised when setlocale() fails.

 -- Function: locale.setlocale (category[, locale])
     If _locale_ is specified, it may be a string, a tuple of the form
     `(language code, encoding)', or `None'. If it is a tuple, it is
     converted to a string using the locale aliasing engine.  If
     _locale_ is given and not `None', see setlocale() modifies
     the locale setting for the _category_.  The available categories
     are listed in the data description below.  The value is the name
     of a locale.  An empty string specifies the user's default
     settings. If the modification of the locale fails, the exception
     Error is raised.  If successful, the new locale
     setting is returned.

     If _locale_ is omitted or `None', the current setting for
     _category_ is returned.

     setlocale() is not thread-safe on most systems.
     Applications typically start with a call of

         import locale
         locale.setlocale(locale.LC_ALL, '')

     This sets the locale for all categories to the user's default
     setting (typically specified in the `LANG' environment variable).
     If the locale is not changed thereafter, using multithreading
     should not cause problems.

     Changed in version 2.0: Added support for tuple values of the
     _locale_ parameter.

 -- Function: locale.localeconv ()
     Returns the database of the local conventions as a dictionary.
     This dictionary has the following strings as keys:

     Category                   Key                                       Meaning
     ----------------------------------------------------------------------------------------------------------
     see LC_NUMERIC.    `'decimal_point''                         Decimal point character.
                                `'grouping''                              Sequence of numbers specifying
                                                                          which relative positions the
                                                                          `'thousands_sep'' is expected.  If
                                                                          the sequence is terminated with
                                                                          CHAR_MAX, no further
                                                                          grouping is performed. If the
                                                                          sequence terminates with a `0',
                                                                          the last group size is repeatedly
                                                                          used.
                                `'thousands_sep''                         Character used between groups.
     See LC_MONETARY.   `'int_curr_symbol''                       International currency symbol.
                                `'currency_symbol''                       Local currency symbol.
                                `'p_cs_precedes/n_cs_precedes''           Whether the currency symbol
                                                                          precedes the value (for positive
                                                                          resp. negative values).
                                `'p_sep_by_space/n_sep_by_space''         Whether the currency symbol is
                                                                          separated from the value  by a
                                                                          space (for positive resp.  negative
                                                                          values).
                                `'mon_decimal_point''                     Decimal point used for monetary
                                                                          values.
                                `'frac_digits''                           Number of fractional digits used in
                                                                          local formatting of monetary values.
                                `'int_frac_digits''                       Number of fractional digits used in
                                                                          international formatting of
                                                                          monetary values.
                                `'mon_thousands_sep''                     Group separator used for monetary
                                                                          values.
                                `'mon_grouping''                          Equivalent to `'grouping'', used
                                                                          for monetary values.
                                `'positive_sign''                         Symbol used to annotate a positive
                                                                          monetary value.
                                `'negative_sign''                         Symbol used to annotate a negative
                                                                          monetary value.
                                `'p_sign_posn/n_sign_posn''               The position of the sign (for
                                                                          positive resp. negative values),
                                                                          see below.

     All numeric values can be set to CHAR_MAX to indicate
     that there is no value specified in this locale.

     The possible values for `'p_sign_posn'' and `'n_sign_posn'' are
     given below.

     Value              Explanation
     -----------------------------------------------------------------
     `0'                Currency and value are surrounded by
                        parentheses.
     `1'                The sign should precede the value and
                        currency symbol.
     `2'                The sign should follow the value and
                        currency symbol.
     `3'                The sign should immediately precede the
                        value.
     `4'                The sign should immediately follow the value.
     `CHAR_MAX'         Nothing is specified in this locale.

 -- Function: locale.nl_langinfo (option)
     Return some locale-specific information as a string.  This
     function is not available on all systems, and the set of possible
     options might also vary across platforms.  The possible argument
     values are numbers, for which symbolic constants are available in
     the locale module.

     The nl_langinfo() function accepts one of the
     following keys.  Most descriptions are taken from the
     corresponding description in the GNU C library.

      -- Data: locale.CODESET
          Get a string with the name of the character encoding used in
          the selected locale.

      -- Data: locale.D_T_FMT
          Get a string that can be used as a format string for
          `strftime()' to represent time and date in a locale-specific
          way.

      -- Data: locale.D_FMT
          Get a string that can be used as a format string for
          `strftime()' to represent a date in a locale-specific way.

      -- Data: locale.T_FMT
          Get a string that can be used as a format string for
          `strftime()' to represent a time in a locale-specific way.

      -- Data: locale.T_FMT_AMPM
          Get a format string for `strftime()' to represent time in the
          am/pm format.

      -- Data: DAY_1 ... DAY_7
          Get the name of the n-th day of the week.

               Note: This follows the US convention of `DAY_1' being
               Sunday, not the international convention (ISO 8601) that
               Monday is the first day of the week.

      -- Data: ABDAY_1 ... ABDAY_7
          Get the abbreviated name of the n-th day of the week.

      -- Data: MON_1 ... MON_12
          Get the name of the n-th month.

      -- Data: ABMON_1 ... ABMON_12
          Get the abbreviated name of the n-th month.

      -- Data: locale.RADIXCHAR
          Get the radix character (decimal dot, decimal comma, etc.)

      -- Data: locale.THOUSEP
          Get the separator character for thousands (groups of three
          digits).

      -- Data: locale.YESEXPR
          Get a regular expression that can be used with the regex
          function to recognize a positive response to a yes/no
          question.

               Note: The expression is in the syntax suitable for the
               `regex()' function from the C library, which might
               differ from the syntax used in re.

      -- Data: locale.NOEXPR
          Get a regular expression that can be used with the regex(3)
          function to recognize a negative response to a yes/no
          question.

      -- Data: locale.CRNCYSTR
          Get the currency symbol, preceded by "-" if the symbol should
          appear before the value, "+" if the symbol should appear
          after the value, or "." if the symbol should replace the
          radix character.

      -- Data: locale.ERA
          Get a string that represents the era used in the current
          locale.

          Most locales do not define this value.  An example of a
          locale which does define this value is the Japanese one.  In
          Japan, the traditional representation of dates includes the
          name of the era corresponding to the then-emperor's reign.

          Normally it should not be necessary to use this value
          directly. Specifying the `E' modifier in their format strings
          causes the `strftime()' function to use this information.
          The format of the returned string is not specified, and
          therefore you should not assume knowledge of it on different
          systems.

      -- Data: locale.ERA_D_T_FMT
          Get a format string for `strftime()' to represent dates and
          times in a locale-specific era-based way.

      -- Data: locale.ERA_D_FMT
          Get a format string for `strftime()' to represent time in a
          locale-specific era-based way.

      -- Data: locale.ALT_DIGITS
          Get a representation of up to 100 values used to represent
          the values 0 to 99.

 -- Function: locale.getdefaultlocale ([envvars])
     Tries to determine the default locale settings and returns them as
     a tuple of the form `(language code, encoding)'.

     According to POSIX, a program which has not called
     `setlocale(LC_ALL, '')' runs using the portable `'C'' locale.
     Calling `setlocale(LC_ALL, '')' lets it use the default locale as
     defined by the `LANG' variable.  Since we do not want to interfere
     with the current locale setting we thus emulate the behavior in
     the way described above.

     To maintain compatibility with other platforms, not only the `LANG'
     variable is tested, but a list of variables given as envvars
     parameter.  The first found to be defined will be used.  _envvars_
     defaults to the search path used in GNU gettext; it must always
     contain the variable name `LANG'.  The GNU gettext search path
     contains `'LANGUAGE'', `'LC_ALL'', `'LC_CTYPE'', and `'LANG'', in
     that order.

     Except for the code `'C'', the language code corresponds to RFC
     1766 (http://tools.ietf.org/html/rfc1766.html).  _language code_
     and _encoding_ may be `None' if their values cannot be determined.

     New in version 2.0.

 -- Function: locale.getlocale ([category])
     Returns the current setting for the given locale category as
     sequence containing _language code_, _encoding_. _category_ may be
     one of the `LC_*' values except LC_ALL.  It defaults
     to LC_CTYPE.

     Except for the code `'C'', the language code corresponds to RFC
     1766 (http://tools.ietf.org/html/rfc1766.html).  _language code_
     and _encoding_ may be `None' if their values cannot be determined.

     New in version 2.0.

 -- Function: locale.getpreferredencoding ([do_setlocale])
     Return the encoding used for text data, according to user
     preferences.  User preferences are expressed differently on
     different systems, and might not be available programmatically on
     some systems, so this function only returns a guess.

     On some systems, it is necessary to invoke setlocale().
     to obtain the user preferences, so this function is not
     thread-safe. If invoking setlocale is not necessary or desired,
     _do_setlocale_ should be set to `False'.

     New in version 2.3.

 -- Function: locale.normalize (localename)
     Returns a normalized locale code for the given locale name.  The
     returned locale code is formatted for use with setlocale().
     If normalization fails, the original name is returned
     unchanged.

     If the given encoding is not known, the function defaults to the
     default encoding for the locale code just like setlocale().

     New in version 2.0.

 -- Function: locale.resetlocale ([category])
     Sets the locale for _category_ to the default setting.

     The default setting is determined by calling
     getdefaultlocale().  _category_ defaults to LC_ALL.

     New in version 2.0.

 -- Function: locale.strcoll (string1, string2)
     Compares two strings according to the current LC_COLLATE
     setting. As any other compare function, returns a negative,
     or a positive value, or `0', depending on whether _string1_
     collates before or after _string2_ or is equal to it.

 -- Function: locale.strxfrm (string)
     Transforms a string to one that can be used for the built-in
     function cmp(), and still returns locale-aware results.
     This function can be used when the same string is compared
     repeatedly, e.g. when collating a sequence of strings.

 -- Function: locale.format (format, val[, grouping[, monetary]])
     Formats a number _val_ according to the current LC_NUMERIC
     setting.  The format follows the conventions of the `%'
     operator.  For floating point values, the decimal point is
     modified if appropriate.  If _grouping_ is true, also takes the
     grouping into account.

     If _monetary_ is true, the conversion uses monetary thousands
     separator and grouping strings.

     Please note that this function will only work for exactly one
     %char specifier.  For whole format strings, use
     format_string().

     Changed in version 2.5: Added the _monetary_ parameter.

 -- Function: locale.format_string (format, val[, grouping])
     Processes formatting specifiers as in `format % val', but takes
     the current locale settings into account.

     New in version 2.5.

 -- Function: locale.currency (val[, symbol[, grouping[,
          international]]])
     Formats a number _val_ according to the current LC_MONETARY
     settings.

     The returned string includes the currency symbol if _symbol_ is
     true, which is the default. If _grouping_ is true (which is not
     the default), grouping is done with the value. If _international_
     is true (which is not the default), the international currency
     symbol is used.

     Note that this function will not work with the 'C' locale, so you
     have to set a locale via setlocale() first.

     New in version 2.5.

 -- Function: locale.str (float)
     Formats a floating point number using the same format as the
     built-in function `str(float)', but takes the decimal point into
     account.

 -- Function: locale.atof (string)
     Converts a string to a floating point number, following the
     LC_NUMERIC.  settings.

 -- Function: locale.atoi (string)
     Converts a string to an integer, following the LC_NUMERIC
     conventions.

 -- Data: locale.LC_CTYPE
     Locale category for the character type functions.  Depending on
     the settings of this category, the functions of module
     string dealing with case change their behaviour.

 -- Data: locale.LC_COLLATE
     Locale category for sorting strings.  The functions
     strcoll() and strxfrm() of the locale
     module are affected.

 -- Data: locale.LC_TIME
     Locale category for the formatting of time.  The function
     time.strftime().  follows these conventions.

 -- Data: locale.LC_MONETARY
     Locale category for formatting of monetary values.  The available
     options are available from the localeconv() function.

 -- Data: locale.LC_MESSAGES
     Locale category for message display. Python currently does not
     support application specific locale-aware messages.  Messages
     displayed by the operating system, like those returned by
     os.strerror() might be affected by this category.

 -- Data: locale.LC_NUMERIC
     Locale category for formatting numbers.  The functions
     format(), atoi(), atof() and
     str() of the locale module are affected by that
     category.  All other numeric formatting operations are not
     affected.

 -- Data: locale.LC_ALL
     Combination of all locale settings.  If this flag is used when the
     locale is changed, setting the locale for all categories is
     attempted. If that fails for any category, no category is changed
     at all.  When the locale is retrieved using this flag, a string
     indicating the setting for all categories is returned. This string
     can be later used to restore the settings.

 -- Data: locale.CHAR_MAX
     This is a symbolic constant used for different values returned by
     localeconv().

  Example:

    >>> import locale
    >>> loc = locale.getlocale() # get current locale
    # use German locale; name might vary with platform
    >>> locale.setlocale(locale.LC_ALL, 'de_DE')
    >>> locale.strcoll('f\xe4n', 'foo') # compare a string containing an umlaut
    >>> locale.setlocale(locale.LC_ALL, '') # use user's preferred locale
    >>> locale.setlocale(locale.LC_ALL, 'C') # use default (C) locale
    >>> locale.setlocale(locale.LC_ALL, loc) # restore saved locale

5.22.2.1 Background, details, hints, tips and caveats
.....................................................

The C standard defines the locale as a program-wide property that may be
relatively expensive to change.  On top of that, some implementation
are broken in such a way that frequent locale changes may cause core
dumps.  This makes the locale somewhat painful to use correctly.

  Initially, when a program is started, the locale is the `C' locale,
no matter what the user's preferred locale is.  The program must
explicitly say that it wants the user's preferred locale settings by
calling `setlocale(LC_ALL, '')'.

  It is generally a bad idea to call setlocale() in some
library routine, since as a side effect it affects the entire program.
Saving and restoring it is almost as bad: it is expensive and affects
other threads that happen to run before the settings have been restored.

  If, when coding a module for general use, you need a locale
independent version of an operation that is affected by the locale
(such as string.lower(), or certain formats used with
time.strftime()), you will have to find a way to do it without
using the standard library routine.  Even better is convincing yourself
that using locale settings is okay.  Only as a last resort should you
document that your module is not compatible with non-`C' locale
settings.

  The case conversion functions in the string module are
affected by the locale settings.  When a call to the setlocale()
function changes the LC_CTYPE settings, the variables
`string.lowercase', `string.uppercase' and `string.letters' are
recalculated.  Note that code that uses these variable through '
from... See import...', e.g. `from string import
letters', is not affected by subsequent setlocale() calls.

  The only way to perform numeric operations according to the locale is
to use the special functions defined by this module: atof(),
see atoi(), format(), str().

5.22.2.2 For extension writers and programs that embed Python
.............................................................

Extension modules should never call setlocale(), except to
find out what the current locale is.  But since the return value can
only be used portably to restore it, that is not very useful (except
perhaps to find out whether or not the locale is `C').

  When Python code uses the locale module to change the
locale, this also affects the embedding application.  If the embedding
application doesn't want this to happen, it should remove the `_locale'
extension module (which does all the work) from the table of built-in
modules in the `config.c' file, and make sure that the `_locale' module
is not accessible as a shared library.

5.22.2.3 Access to message catalogs
...................................

The locale module exposes the C library's gettext interface on systems
that provide this interface.  It consists of the functions
gettext(), `dgettext()', `dcgettext()', `textdomain()',
`bindtextdomain()', and `bind_textdomain_codeset()'.  These are similar
to the same functions in the gettext module, but use the C
library's binary format for message catalogs, and the C library's
search algorithms for locating message catalogs.

  Python applications should normally find no need to invoke these
functions, and should use gettext instead.  A known
exception to this rule are applications that link with additional C
libraries which internally invoke `gettext()' or `dcgettext()'.  For
these applications, it may be necessary to bind the text domain, so
that the libraries can properly locate their message catalogs.

5.23 Program Frameworks
=======================

The modules described in this chapter are frameworks that will largely
dictate the structure of your program.  Currently the modules described
here are all oriented toward writing command-line interfaces.

  The full list of modules described in this chapter is:

5.23.1 `cmd' -- Support for line-oriented command interpreters
--------------------------------------------------------------

The Cmd class provides a simple framework for writing
line-oriented command interpreters.  These are often useful for test
harnesses, administrative tools, and prototypes that will later be
wrapped in a more sophisticated interface.

  Latest version of the cmd module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/cmd.py?view=markup)

 -- Class: cmd.Cmd ([completekey[, stdin[, stdout]]])
     A Cmd instance or subclass instance is a
     line-oriented interpreter framework.  There is no good reason to
     instantiate Cmd itself; rather, it's useful as a
     superclass of an interpreter class you define yourself in order to
     inherit Cmd.'s methods and encapsulate action methods.

     The optional argument _completekey_ is the readline.
     name of a completion key; it defaults to `Tab'. If _completekey_
     is not None and readline is available,
     command completion is done automatically.

     The optional arguments _stdin_ and _stdout_ specify the  input and
     output file objects that the Cmd instance or subclass  instance
     will use for input and output. If not specified, they will default
     to sys.stdin and sys.stdout.

     If you want a given _stdin_ to be used, make sure to set the
     instance's use_rawinput attribute to `False',
     otherwise _stdin_ will be ignored.

     Changed in version 2.3: The _stdin_ and _stdout_ parameters were
     added.

5.23.1.1 Cmd Objects
....................

A Cmd instance has the following methods:

 -- Method: Cmd.cmdloop ([intro])
     Repeatedly issue a prompt, accept input, parse an initial prefix
     off the received input, and dispatch to action methods, passing
     them the remainder of the line as argument.

     The optional argument is a banner or intro string to be issued
     before the first prompt (this overrides the intro.
     class member).

     If the readline module is loaded, input will
     automatically inherit *bash*-like history-list editing (e.g.
     `Control-P' scrolls back to the last command, `Control-N' forward
     to the next one, `Control-F' moves the cursor to the right
     non-destructively, `Control-B' moves the cursor to the left
     non-destructively, etc.).

     An end-of-file on input is passed back as the string `'EOF''.

     An interpreter instance will recognize a command name `foo' if and
     only if it has a method `do_foo()'.  As a special case, a line
     beginning with the character `'?'' is dispatched to the method
     `do_help()'.  As another special case, a line beginning with the
     character `'!'' is dispatched to the method `do_shell()' (if such
     a method is defined).

     This method will return when the postcmd() method
     returns a true value.  The _stop_ argument to postcmd()
     is the return value from the command's corresponding
     `do_*()' method.

     If completion is enabled, completing commands will be done
     automatically, and completing of commands args is done by calling
     `complete_foo()' with arguments _text_, _line_, _begidx_, and
     _endidx_.  _text_ is the string prefix we are attempting to match:
     all returned matches must begin with it. _line_ is the current
     input line with leading whitespace removed, _begidx_ and _endidx_
     are the beginning and ending indexes of the prefix text, which
     could be used to provide different completion depending upon which
     position the argument is in.

     All subclasses of Cmd inherit a predefined
     `do_help()'.  This method, called with an argument `'bar'',
     invokes the corresponding method `help_bar()', and if that is not
     present, prints the docstring of `do_bar()', if available.  With
     no argument, `do_help()' lists all available help topics (that is,
     all commands with corresponding `help_*()' methods or commands
     that have docstrings), and also lists any undocumented commands.

 -- Method: Cmd.onecmd (str)
     Interpret the argument as though it had been typed in response to
     the prompt.  This may be overridden, but should not normally need
     to be; see the precmd() and postcmd().
     methods for useful execution hooks.  The return value is a flag
     indicating whether interpretation of commands by the interpreter
     should stop.  If there is a `do_*()' method for the command _str_,
     the return value of that method is returned, otherwise the return
     value from the default() method is returned.

 -- Method: Cmd.emptyline ()
     Method called when an empty line is entered in response to the
     prompt. If this method is not overridden, it repeats the last
     nonempty command entered.

 -- Method: Cmd.default (line)
     Method called on an input line when the command prefix is not
     recognized. If this method is not overridden, it prints an error
     message and returns.

 -- Method: Cmd.completedefault (text, line, begidx, endidx)
     Method called to complete an input line when no command-specific
     `complete_*()' method is available.  By default, it returns an
     empty list.

 -- Method: Cmd.precmd (line)
     Hook method executed just before the command line _line_ is
     interpreted, but after the input prompt is generated and issued.
     This method is a stub in Cmd.; it exists to be
     overridden by subclasses.  The return value is used as the command
     which will be executed by the onecmd() method; the
     precmd() implementation may re-write the command or
     simply return _line_ unchanged.

 -- Method: Cmd.postcmd (stop, line)
     Hook method executed just after a command dispatch is finished.
     This method is a stub in Cmd.; it exists to be
     overridden by subclasses.  _line_ is the command line which was
     executed, and _stop_ is a flag which indicates whether execution
     will be terminated after the call to postcmd().; this
     will be the return value of the onecmd() method.  The
     return value of this method will be used as the new value for the
     internal flag which corresponds to _stop_; returning false will
     cause interpretation to continue.

 -- Method: Cmd.preloop ()
     Hook method executed once when cmdloop() is called.
     This method is a stub in Cmd.; it exists to be
     overridden by subclasses.

 -- Method: Cmd.postloop ()
     Hook method executed once when cmdloop() is about to
     return. This method is a stub in Cmd.; it exists to be
     overridden by subclasses.

  Instances of Cmd subclasses have some public instance
variables:

 -- Attribute: Cmd.prompt
     The prompt issued to solicit input.

 -- Attribute: Cmd.identchars
     The string of characters accepted for the command prefix.

 -- Attribute: Cmd.lastcmd
     The last nonempty command prefix seen.

 -- Attribute: Cmd.intro
     A string to issue as an intro or banner.  May be overridden by
     giving the cmdloop() method an argument.

 -- Attribute: Cmd.doc_header
     The header to issue if the help output has a section for
     documented commands.

 -- Attribute: Cmd.misc_header
     The header to issue if the help output has a section for
     miscellaneous  help topics (that is, there are `help_*()' methods
     without corresponding `do_*()' methods).

 -- Attribute: Cmd.undoc_header
     The header to issue if the help output has a section for
     undocumented  commands (that is, there are `do_*()' methods
     without corresponding `help_*()' methods).

 -- Attribute: Cmd.ruler
     The character used to draw separator lines under the help-message
     headers.  If empty, no ruler line is drawn.  It defaults to `'=''.

 -- Attribute: Cmd.use_rawinput
     A flag, defaulting to true.  If true, see cmdloop() uses
     raw_input() to display a prompt and read the next
     command; if false, `sys.stdout.write()' and `sys.stdin.readline()'
     are used. (This means that by importing readline, on
     systems that support it, the interpreter will automatically
     support *Emacs*-like line editing  and command-history keystrokes.)

5.23.2 `shlex' -- Simple lexical analysis
-----------------------------------------

New in version 1.5.2.

  The shlex class makes it easy to write lexical analyzers
for simple syntaxes resembling that of the Unix shell.  This will often
be useful for writing minilanguages, (for example, in run control files
for Python applications) or for parsing quoted strings.

     Note: The shlex module currently does not support
     Unicode input.

  The shlex module defines the following functions:

 -- Function: shlex.split (s[, comments[, posix]])
     Split the string _s_ using shell-like syntax. If _comments_ is
     False.  (the default), the parsing of comments in the
     given string will be disabled (setting the `commenters' member of
     the shlex instance to the empty string).  This
     function operates in POSIX mode by default, but uses non-POSIX
     mode if the _posix_ argument is false.

     New in version 2.3.

     Changed in version 2.6: Added the _posix_ parameter.

          Note: Since the split() function instantiates a
          shlex instance, passing `None' for _s_ will read
          the string to split from standard input.

  The shlex module defines the following class:

 -- Class: shlex.shlex ([instream[, infile[, posix]]])
     A shlex instance or subclass instance is a lexical
     analyzer object.  The initialization argument, if present,
     specifies where to read characters from. It must be a
     file-/stream-like object with `read()' and readline().
     methods, or a string (strings are accepted since Python 2.3).  If
     no argument is given, input will be taken from `sys.stdin'.  The
     second optional argument is a filename string, which sets the
     initial value of the infile member.  If the
     _instream_ argument is omitted or equal to `sys.stdin', this
     second argument defaults to "stdin".  The _posix_ argument was
     introduced in Python 2.3, and defines the operational mode.  When
     _posix_ is not true (default), the shlex instance will
     operate in compatibility mode.  When operating in POSIX mode,
     see shlex will try to be as close as possible to the POSIX
     shell parsing rules.

Module ConfigParser.
     Parser for configuration files similar to the Windows `.ini' files.

5.23.2.1 shlex Objects
......................

A shlex instance has the following methods:

 -- Method: shlex.get_token ()
     Return a token.  If tokens have been stacked using
     push_token(), pop a token off the stack.  Otherwise, read
     one from the input stream.  If reading encounters an immediate
     end-of-file, `self.eof' is returned (the empty string (`''') in
     non-POSIX mode, and `None' in POSIX mode).

 -- Method: shlex.push_token (str)
     Push the argument onto the token stack.

 -- Method: shlex.read_token ()
     Read a raw token.  Ignore the pushback stack, and do not interpret
     source requests.  (This is not ordinarily a useful entry point,
     and is documented here only for the sake of completeness.)

 -- Method: shlex.sourcehook (filename)
     When shlex detects a source request (see source
     below) this method is given the following token as argument,
     and expected to return a tuple consisting of a filename and an
     open file-like object.

     Normally, this method first strips any quotes off the argument.
     If the result is an absolute pathname, or there was no previous
     source request in effect, or the previous source was a stream
     (such as `sys.stdin'), the result is left alone.  Otherwise, if
     the result is a relative pathname, the directory part of the name
     of the file immediately before it on the source inclusion stack is
     prepended (this behavior is like the way the C preprocessor
     handles `#include "file.h"').

     The result of the manipulations is treated as a filename, and
     returned as the first component of the tuple, with open()
     called on it to yield the second component. (Note: this is
     the reverse of the order of arguments in instance initialization!)

     This hook is exposed so that you can use it to implement directory
     search paths, addition of file extensions, and other namespace
     hacks. There is no corresponding 'close' hook, but a shlex
     instance will call the `close()' method of the sourced input
     stream when it returns EOF.

     For more explicit control of source stacking, use the
     push_source() and pop_source() methods.

 -- Method: shlex.push_source (stream[, filename])
     Push an input source stream onto the input stack.  If the filename
     argument is specified it will later be available for use in error
     messages.  This is the same method used internally by the
     sourcehook() method.

     New in version 2.1.

 -- Method: shlex.pop_source ()
     Pop the last-pushed input source from the input stack. This is the
     same method used internally when the lexer reaches EOF on a
     stacked input stream.

     New in version 2.1.

 -- Method: shlex.error_leader ([file[, line]])
     This method generates an error message leader in the format of a
     Unix C compiler error label; the format is `'"%s", line %d: '',
     where the `%s' is replaced with the name of the current source
     file and the `%d' with the current input line number (the optional
     arguments can be used to override these).

     This convenience is provided to encourage shlex users
     to generate error messages in the standard, parseable format
     understood by Emacs and other Unix tools.

  Instances of shlex subclasses have some public instance
variables which either control lexical analysis or can be used for
debugging:

 -- Attribute: shlex.commenters
     The string of characters that are recognized as comment beginners.
     All characters from the comment beginner to end of line are
     ignored. Includes just `'#'' by default.

 -- Attribute: shlex.wordchars
     The string of characters that will accumulate into multi-character
     tokens.  By default, includes all ASCII alphanumerics and
     underscore.

 -- Attribute: shlex.whitespace
     Characters that will be considered whitespace and skipped.
     Whitespace bounds tokens.  By default, includes space, tab,
     linefeed and carriage-return.

 -- Attribute: shlex.escape
     Characters that will be considered as escape. This will be only
     used in POSIX mode, and includes just `'\'' by default.

     New in version 2.3.

 -- Attribute: shlex.quotes
     Characters that will be considered string quotes.  The token
     accumulates until the same quote is encountered again (thus,
     different quote types protect each other as in the shell.)  By
     default, includes ASCII single and double quotes.

 -- Attribute: shlex.escapedquotes
     Characters in quotes that will interpret escape
     characters defined in escape.  This is only used in
     POSIX mode, and includes just `'"'' by default.

     New in version 2.3.

 -- Attribute: shlex.whitespace_split
     If `True', tokens will only be split in whitespaces. This is
     useful, for example, for parsing command lines with shlex,
     getting tokens in a similar way to shell arguments.

     New in version 2.3.

 -- Attribute: shlex.infile
     The name of the current input file, as initially set at class
     instantiation time or stacked by later source requests.  It may be
     useful to examine this when constructing error messages.

 -- Attribute: shlex.instream
     The input stream from which this shlex instance is
     reading characters.

 -- Attribute: shlex.source
     This member is `None' by default.  If you assign a string to it,
     that string will be recognized as a lexical-level inclusion
     request similar to the `source' keyword in various shells.  That
     is, the immediately following token will opened as a filename and
     input taken from that stream until EOF, at which point the
     `close()' method of that stream will be called and the input
     source will again become the original input stream. Source
     requests may be stacked any number of levels deep.

 -- Attribute: shlex.debug
     If this member is numeric and `1' or more, a shlex.
     instance will print verbose progress output on its behavior.  If
     you need to use this, you can read the module source code to learn
     the details.

 -- Attribute: shlex.lineno
     Source line number (count of newlines seen so far plus one).

 -- Attribute: shlex.token
     The token buffer.  It may be useful to examine this when catching
     exceptions.

 -- Attribute: shlex.eof
     Token used to determine end of file. This will be set to the empty
     string (`'''), in non-POSIX mode, and to `None' in POSIX mode.

     New in version 2.3.

5.23.2.2 Parsing Rules
......................

When operating in non-POSIX mode, see shlex will try to obey to
the following rules.

   * Quote characters are not recognized within words
     (`Do"Not"Separate' is parsed as the single word `Do"Not"Separate');

   * Escape characters are not recognized;

   * Enclosing characters in quotes preserve the literal value of all
     characters within the quotes;

   * Closing quotes separate words (`"Do"Separate' is parsed as `"Do"'
     and `Separate');

   * If `whitespace_split' is `False', any character not declared to be
     a word character, whitespace, or a quote will be returned as a
     single-character token. If it is `True', see shlex will
     only split words in whitespaces;

   * EOF is signaled with an empty string (`''');

   * It's not possible to parse empty strings, even if quoted.

  When operating in POSIX mode, see shlex will try to obey to
the following parsing rules.

   * Quotes are stripped out, and do not separate words
     (`"Do"Not"Separate"' is parsed as the single word `DoNotSeparate');

   * Non-quoted escape characters (e.g. `'\'') preserve the literal
     value of the next character that follows;

   * Enclosing characters in quotes which are not part of
     `escapedquotes' (e.g. `"'"') preserve the literal value of all
     characters within the quotes;

   * Enclosing characters in quotes which are part of `escapedquotes'
     (e.g.  `'"'') preserves the literal value of all characters within
     the quotes, with the exception of the characters mentioned in
     `escape'. The escape characters retain its special meaning only
     when followed by the quote in use, or the escape character itself.
     Otherwise the escape character will be considered a normal
     character.

   * EOF is signaled with a None value;

   * Quoted empty strings (`''') are allowed;

5.24 Graphical User Interfaces with Tk
======================================

Tk/Tcl has long been an integral part of Python.  It provides a robust
and platform independent windowing toolkit, that is available to Python
programmers using the Tkinter module, and its extensions,
the Tix and the ttk modules.

  The Tkinter module is a thin object-oriented layer on top
of Tcl/Tk. To use Tkinter, you don't need to write Tcl code,
but you will need to consult the Tk documentation, and occasionally the
Tcl documentation.  Tkinter is a set of wrappers that
implement the Tk widgets as Python classes.  In addition, the internal
module `_tkinter' provides a threadsafe mechanism which allows Python
and Tcl to interact.

  See Tkinter.'s chief virtues are that it is fast, and that it
usually comes bundled with Python. Although its standard documentation
is weak, good material is available, which includes: references,
tutorials, a book and others. Tkinter is also famous for
having an outdated look and feel, which has been vastly improved in Tk
8.5. Nevertheless, there are many other GUI libraries that you could be
interested in. For more information about alternatives, see the
Other Graphical User Interface Packages section.

5.24.1 `Tkinter' -- Python interface to Tcl/Tk
----------------------------------------------

The Tkinter module ("Tk interface") is the standard Python
interface to the Tk GUI toolkit.  Both Tk and Tkinter are
available on most Unix platforms, as well as on Windows systems.  (Tk
itself is not part of Python; it is maintained at ActiveState.)

     Note: Tkinter has been renamed to `tkinter' in Python
     3.0.  The 2to3 tool will automatically adapt imports
     when converting your sources to 3.0.

Python Tkinter Resources (http://www.python.org/topics/tkinter/)
     The Python Tkinter Topic Guide provides a great deal of
     information on using Tk from Python and links to other sources of
     information on Tk.

An Introduction to Tkinter (http://www.pythonware.com/library/an-introduction-to-tkinter.htm)
     Fredrik Lundh's on-line reference material.

Tkinter reference: a GUI for Python (http://infohost.nmt.edu/tcc/help/pubs/lang.html)
     On-line reference material.

Python and Tkinter Programming (http://www.amazon.com/exec/obidos/ASIN/1884777813)
     The book by John Grayson (ISBN 1-884777-81-3).

5.24.1.1 Tkinter Modules
........................

Most of the time, the Tkinter module is all you really
need, but a number of additional modules are available as well.  The Tk
interface is located in a binary module named `_tkinter'. This module
contains the low-level interface to Tk, and should never be used
directly by application programmers.  It is usually a shared library
(or DLL), but might in some cases be statically linked with the Python
interpreter.

  In addition to the Tk interface module, see Tkinter includes
a number of Python modules. The two most important modules are the
Tkinter module itself, and a module called `Tkconstants'.
The former automatically imports the latter, so to use Tkinter, all you
need to do is to import one module:

    import Tkinter

Or, more often:

    from Tkinter import *

 -- Class: Tkinter.Tk (screenName=None, baseName=None, className='Tk',
          useTk=1)
     The Tk class is instantiated without arguments. This
     creates a toplevel widget of Tk which usually is the main window
     of an application. Each instance has its own associated Tcl
     interpreter.

     Changed in version 2.4: The _useTk_ parameter was added.

 -- Function: Tkinter.Tcl (screenName=None, baseName=None,
          className='Tk', useTk=0)
     The Tcl() function is a factory function which
     creates an object much like that created by the Tk.
     class, except that it does not initialize the Tk subsystem.  This
     is most often useful when driving the Tcl interpreter in an
     environment where one doesn't want to create extraneous toplevel
     windows, or where one cannot (such as Unix/Linux systems without
     an X server).  An object created by the Tcl() object
     can have a Toplevel window created (and the Tk subsystem
     initialized) by calling its `loadtk()' method.

     New in version 2.4.

  Other modules that provide Tk support include:

ScrolledText.
     Text widget with a vertical scroll bar built in.

`tkColorChooser'
     Dialog to let the user choose a color.

`tkCommonDialog'
     Base class for the dialogs defined in the other modules listed
     here.

`tkFileDialog'
     Common dialogs to allow the user to specify a file to open or save.

`tkFont'
     Utilities to help work with fonts.

`tkMessageBox'
     Access to standard Tk dialog boxes.

`tkSimpleDialog'
     Basic dialogs and convenience functions.

`Tkdnd'
     Drag-and-drop support for Tkinter. This is experimental
     and should become deprecated when it is replaced  with the Tk DND.

See turtle.
     Turtle graphics in a Tk window.

  These have been renamed as well in Python 3.0; they were all made
submodules of the new `tkinter' package.

5.24.1.2 Tkinter Life Preserver
...............................

This section is not designed to be an exhaustive tutorial on either Tk
or Tkinter.  Rather, it is intended as a stop gap, providing some
introductory orientation on the system.

  Credits:

   * Tkinter was written by Steen Lumholt and Guido van Rossum.

   * Tk was written by John Ousterhout while at Berkeley.

   * This Life Preserver was written by Matt Conway at the University
     of Virginia.

   * The html rendering, and some liberal editing, was produced from a
     FrameMaker version by Ken Manheimer.

   * Fredrik Lundh elaborated and revised the class interface
     descriptions, to get them current with Tk 4.2.

   * Mike Clarkson converted the documentation to LaTeX, and compiled
     the  User Interface chapter of the reference manual.

5.24.1.3 How To Use This Section
................................

This section is designed in two parts: the first half (roughly) covers
background material, while the second half can be taken to the keyboard
as a handy reference.

  When trying to answer questions of the form "how do I do blah", it is
often best to find out how to do"blah" in straight Tk, and then convert
this back into the corresponding Tkinter call. Python
programmers can often guess at the correct Python command by looking at
the Tk documentation. This means that in order to use Tkinter, you will
have to know a little bit about Tk. This document can't fulfill that
role, so the best we can do is point you to the best documentation that
exists. Here are some hints:

   * The authors strongly suggest getting a copy of the Tk man pages.
     Specifically, the man pages in the `mann' directory are most
     useful. The `man3' man pages describe the C interface to the Tk
     library and thus are not especially helpful for script writers.

   * Addison-Wesley publishes a book called Tcl and the Tk Toolkit by
     John Ousterhout (ISBN 0-201-63337-X) which is a good introduction
     to Tcl and Tk for the novice.  The book is not exhaustive, and for
     many details it defers to the man pages.

   * `Tkinter.py' is a last resort for most, but can be a good place to
     go when nothing else makes sense.

ActiveState Tcl Home Page (http://tcl.activestate.com/)
     The Tk/Tcl development is largely taking place at ActiveState.

Tcl and the Tk Toolkit (http://www.amazon.com/exec/obidos/ASIN/020163337X)
     The book by John Ousterhout, the inventor of Tcl .

Practical Programming in Tcl and Tk (http://www.amazon.com/exec/obidos/ASIN/0130220280)
     Brent Welch's encyclopedic book.

5.24.1.4 A Simple Hello World Program
.....................................

    from Tkinter import *

    class Application(Frame):
        def say_hi(self):
            print "hi there, everyone!"

        def createWidgets(self):
            self.QUIT = Button(self)
            self.QUIT["text"] = "QUIT"
            self.QUIT["fg"]   = "red"
            self.QUIT["command"] =  self.quit

            self.QUIT.pack({"side": "left"})

            self.hi_there = Button(self)
            self.hi_there["text"] = "Hello",
            self.hi_there["command"] = self.say_hi

            self.hi_there.pack({"side": "left"})

        def __init__(self, master=None):
            Frame.__init__(self, master)
            self.pack()
            self.createWidgets()

    root = Tk()
    app = Application(master=root)
    app.mainloop()
    root.destroy()

5.24.1.5 A (Very) Quick Look at Tcl/Tk
......................................

The class hierarchy looks complicated, but in actual practice,
application programmers almost always refer to the classes at the very
bottom of the hierarchy.

  Notes:

   * These classes are provided for the purposes of organizing certain
     functions under one namespace. They aren't meant to be
     instantiated independently.

   * The Tk class is meant to be instantiated only once in
     an application.  Application programmers need not instantiate one
     explicitly, the system creates one whenever any of the other
     classes are instantiated.

   * The `Widget' class is not meant to be instantiated, it is meant
     only for subclassing to make "real" widgets (in C++, this is
     called an 'abstract class').

  To make use of this reference material, there will be times when you
will need to know how to read short passages of Tk and how to identify
the various parts of a Tk command.   (See section Mapping Basic Tk into Tkinter
for the Tkinter equivalents of
what's below.)

  Tk scripts are Tcl programs.  Like all Tcl programs, Tk scripts are
just lists of tokens separated by spaces.  A Tk widget is just its
_class_, the _options_ that help configure it, and the _actions_ that
make it do useful things.

  To make a widget in Tk, the command is always of the form:

    classCommand newPathname options

_classCommand_
     denotes which kind of widget to make (a button, a label, a menu...)

_newPathname_
     is the new name for this widget.  All names in Tk must be unique.
     To help enforce this, widgets in Tk are named with _pathnames_,
     just like files in a file system.  The top level widget, the
     _root_, is called `.' (period) and children are delimited by more
     periods.  For example, `.myApp.controlPanel.okButton' might be the
     name of a widget.

_options_
     configure the widget's appearance and in some cases, its behavior.
     The options come in the form of a list of flags and values. Flags
     are preceded by a '-', like Unix shell command flags, and values
     are put in quotes if they are more than one word.

  For example:

    button   .fred   -fg red -text "hi there"
       ^       ^     \_____________________/
       |       |                |
     class    new            options
    command  widget  (-opt val -opt val ...)

Once created, the pathname to the widget becomes a new command.  This
new _widget command_ is the programmer's handle for getting the new
widget to perform some _action_.  In C, you'd express this as
someAction(fred, someOptions), in C++, you would express this as
fred.someAction(someOptions), and in Tk, you say:

    .fred someAction someOptions

Note that the object name, `.fred', starts with a dot.

  As you'd expect, the legal values for _someAction_ will depend on the
widget's class: `.fred disable' works if fred is a button (fred gets
greyed out), but does not work if fred is a label (disabling of labels
is not supported in Tk).

  The legal values of _someOptions_ is action dependent.  Some actions,
like `disable', require no arguments, others, like a text-entry box's
`delete' command, would need arguments to specify what range of text to
delete.

5.24.1.6 Mapping Basic Tk into Tkinter
......................................

Class commands in Tk correspond to class constructors in Tkinter.

    button .fred                =====>  fred = Button()

The master of an object is implicit in the new name given to it at
creation time.  In Tkinter, masters are specified explicitly.

    button .panel.fred          =====>  fred = Button(panel)

The configuration options in Tk are given in lists of hyphened tags
followed by values.  In Tkinter, options are specified as
keyword-arguments in the instance constructor, and keyword-args for
configure calls or as instance indices, in dictionary style, for
established instances.  See section Setting Options on
setting options.

    button .fred -fg red        =====>  fred = Button(panel, fg = "red")
    .fred configure -fg red     =====>  fred["fg"] = red
                                OR ==>  fred.config(fg = "red")

In Tk, to perform an action on a widget, use the widget name as a
command, and follow it with an action name, possibly with arguments
(options).  In Tkinter, you call methods on the class instance to
invoke actions on the widget.  The actions (methods) that a given
widget can perform are listed in the Tkinter.py module.

    .fred invoke                =====>  fred.invoke()

To give a widget to the packer (geometry manager), you call pack with
optional arguments.  In Tkinter, the Pack class holds all this
functionality, and the various forms of the pack command are
implemented as methods.  All widgets in Tkinter are
subclassed from the Packer, and so inherit all the packing methods. See
the Tix module documentation for additional information on
the Form geometry manager.

    pack .fred -side left       =====>  fred.pack(side = "left")

5.24.1.7 How Tk and Tkinter are Related
.......................................

From the top down:

Your App Here (Python)
     A Python application makes a Tkinter call.

Tkinter (Python Module)
     This call (say, for example, creating a button widget), is
     implemented in the _Tkinter_ module, which is written in Python.
     This Python function will parse the commands and the arguments and
     convert them into a form that makes them look as if they had come
     from a Tk script instead of a Python script.

tkinter (C)
     These commands and their arguments will be passed to a C function
     in the _tkinter_ - note the lowercase - extension module.

Tk Widgets (C and Tcl)
     This C function is able to make calls into other C modules,
     including the C functions that make up the Tk library.  Tk is
     implemented in C and some Tcl.  The Tcl part of the Tk widgets is
     used to bind certain default behaviors to widgets, and is executed
     once at the point where the Python Tkinter.  module is
     imported. (The user never sees this stage).

Tk (C)
     The Tk part of the Tk Widgets implement the final mapping to ...

Xlib (C)
     the Xlib library to draw graphics on the screen.

5.24.1.8 Handy Reference
........................

5.24.1.9 Setting Options
........................

Options control things like the color and border width of a widget.
Options can be set in three ways:

At object creation time, using keyword arguments
         fred = Button(self, fg = "red", bg = "blue")

After object creation, treating the option name like a dictionary index
         fred["fg"] = "red"
         fred["bg"] = "blue"

Use the config() method to update multiple attrs subsequent to object creation
         fred.config(fg = "red", bg = "blue")

  For a complete explanation of a given option and its behavior, see
the Tk man pages for the widget in question.

  Note that the man pages list "STANDARD OPTIONS" and "WIDGET SPECIFIC
OPTIONS" for each widget.  The former is a list of options that are
common to many widgets, the latter are the options that are
idiosyncratic to that particular widget.  The Standard Options are
documented on the `options(3)' man page.

  No distinction between standard and widget-specific options is made
in this document.  Some options don't apply to some kinds of widgets.
Whether a given widget responds to a particular option depends on the
class of the widget; buttons have a `command' option, labels do not.

  The options supported by a given widget are listed in that widget's
man page, or can be queried at runtime by calling the `config()' method
without arguments, or by calling the `keys()' method on that widget.
The return value of these calls is a dictionary whose key is the name
of the option as a string (for example, `'relief'') and whose values
are 5-tuples.

  Some options, like `bg' are synonyms for common options with long
names (`bg' is shorthand for "background"). Passing the `config()'
method the name of a shorthand option will return a 2-tuple, not
5-tuple. The 2-tuple passed back will contain the name of the synonym
and the "real" option (such as `('bg', 'background')').

Index       Meaning                               Example
---------------------------------------------------------------------
0           option name                           `'relief''
1           option name for database lookup       `'relief''
2           option class for database lookup      `'Relief''
3           default value                         `'raised''
4           current value                         `'groove''

  Example:

    >>> print fred.config()
    {'relief' : ('relief', 'relief', 'Relief', 'raised', 'groove')}

Of course, the dictionary printed will include all the options
available and their values.  This is meant only as an example.

5.24.1.10 The Packer
....................

The packer is one of Tk's geometry-management mechanisms.    Geometry
managers are used to specify the relative positioning of the
positioning of widgets within their container - their mutual _master_.
In contrast to the more cumbersome _placer_ (which is used less
commonly, and we do not cover here), the packer takes qualitative
relationship specification - _above_, _to the left of_, _filling_, etc
- and works everything out to determine the exact placement coordinates
for you.

  The size of any _master_ widget is determined by the size of the
"slave widgets" inside.  The packer is used to control where slave
widgets appear inside the master into which they are packed.  You can
pack widgets into frames, and frames into other frames, in order to
achieve the kind of layout you desire.  Additionally, the arrangement
is dynamically adjusted to accommodate incremental changes to the
configuration, once it is packed.

  Note that widgets do not appear until they have had their geometry
specified with a geometry manager.  It's a common early mistake to
leave out the geometry specification, and then be surprised when the
widget is created but nothing appears.  A widget will appear only after
it has had, for example, the packer's `pack()' method applied to it.

  The pack() method can be called with keyword-option/value pairs that
control where the widget is to appear within its container, and how it
is to behave when the main application window is resized.  Here are
some examples:

    fred.pack()                     # defaults to side = "top"
    fred.pack(side = "left")
    fred.pack(expand = 1)

5.24.1.11 Packer Options
........................

For more extensive information on the packer and the options that it
can take, see the man pages and page 183 of John Ousterhout's book.

anchor
     Anchor type.  Denotes where the packer is to place each slave in
     its parcel.

expand
     Boolean, `0' or `1'.

fill
     Legal values: `'x'', `'y'', `'both'', `'none''.

ipadx and ipady
     A distance - designating internal padding on each side of the
     slave widget.

padx and pady
     A distance - designating external padding on each side of the
     slave widget.

side
     Legal values are: `'left'', `'right'', `'top'', `'bottom''.

5.24.1.12 Coupling Widget Variables
...................................

The current-value setting of some widgets (like text entry widgets) can
be connected directly to application variables by using special
options.  These options are `variable', `textvariable', `onvalue',
`offvalue', and `value'.  This connection works both ways: if the
variable changes for any reason, the widget it's connected to will be
updated to reflect the new value.

  Unfortunately, in the current implementation of Tkinter.
it is not possible to hand over an arbitrary Python variable to a
widget through a `variable' or `textvariable' option.  The only kinds
of variables for which this works are variables that are subclassed
from a class called Variable, defined in the Tkinter module.

  There are many useful subclasses of Variable already defined:
`StringVar', `IntVar', `DoubleVar', and `BooleanVar'.  To read the
current value of such a variable, call the `get()' method on it, and to
change its value you call the `set()' method.  If you follow this
protocol, the widget will always track the value of the variable, with
no further intervention on your part.

  For example:

    class App(Frame):
        def __init__(self, master=None):
            Frame.__init__(self, master)
            self.pack()

            self.entrythingy = Entry()
            self.entrythingy.pack()

            # here is the application variable
            self.contents = StringVar()
            # set it to some value
            self.contents.set("this is a variable")
            # tell the entry widget to watch this variable
            self.entrythingy["textvariable"] = self.contents

            # and here we get a callback when the user hits return.
            # we will have the program print out the value of the
            # application variable when the user hits return
            self.entrythingy.bind('<Key-Return>',
                                  self.print_contents)

        def print_contents(self, event):
            print "hi. contents of entry is now ---->", \
                  self.contents.get()

5.24.1.13 The Window Manager
............................

In Tk, there is a utility command, `wm', for interacting with the window
manager.  Options to the `wm' command allow you to control things like
titles, placement, icon bitmaps, and the like.  In Tkinter,
these commands have been implemented as methods on the `Wm' class.
Toplevel widgets are subclassed from the `Wm' class, and so can call
the `Wm' methods directly.

  To get at the toplevel window that contains a given widget, you can
often just refer to the widget's master.  Of course if the widget has
been packed inside of a frame, the master won't represent a toplevel
window.  To get at the toplevel window that contains an arbitrary
widget, you can call the `_root()' method.  This method begins with an
underscore to denote the fact that this function is part of the
implementation, and not an interface to Tk functionality.

  Here are some examples of typical usage:

    from Tkinter import *
    class App(Frame):
        def __init__(self, master=None):
            Frame.__init__(self, master)
            self.pack()

    # create the application
    myapp = App()

    #
    # here are method calls to the window manager class
    #
    myapp.master.title("My Do-Nothing Application")
    myapp.master.maxsize(1000, 400)

    # start the program
    myapp.mainloop()

5.24.1.14 Tk Option Data Types
..............................

anchor
     Legal values are points of the compass: `"n"', `"ne"', `"e"',
     `"se"', `"s"', `"sw"', `"w"', `"nw"', and also `"center"'.

bitmap
     There are eight built-in, named bitmaps: `'error'', `'gray25'',
     `'gray50'', `'hourglass'', `'info'', `'questhead'', `'question'',
     `'warning''.  To specify an X bitmap filename, give the full path
     to the file, preceded with an `@', as in
     `"@/usr/contrib/bitmap/gumby.bit"'.

boolean
     You can pass integers 0 or 1 or the strings `"yes"' or `"no"' .

callback
     This is any Python function that takes no arguments.  For example:

         def print_it():
                 print "hi there"
         fred["command"] = print_it

color
     Colors can be given as the names of X colors in the rgb.txt file,
     or as strings representing RGB values in 4 bit: `"#RGB"', 8 bit:
     `"#RRGGBB"', 12 bit" `"#RRRGGGBBB"', or 16 bit `"#RRRRGGGGBBBB"'
     ranges, where R,G,B here represent any legal hex digit.  See page
     160 of Ousterhout's book for details.

cursor
     The standard X cursor names from `cursorfont.h' can be used,
     without the `XC_' prefix.  For example to get a hand cursor
     (`XC_hand2'), use the string `"hand2"'.  You can also specify a
     bitmap and mask file of your own.  See page 179 of Ousterhout's
     book.

distance
     Screen distances can be specified in either pixels or absolute
     distances.  Pixels are given as numbers and absolute distances as
     strings, with the trailing character denoting units: `c' for
     centimetres, `i' for inches, `m' for millimetres, `p' for
     printer's points.  For example, 3.5 inches is expressed as
     `"3.5i"'.

font
     Tk uses a list font name format, such as `{courier 10 bold}'. Font
     sizes with positive numbers are measured in points; sizes with
     negative numbers are measured in pixels.

geometry
     This is a string of the form `widthxheight', where width and
     height are measured in pixels for most widgets (in characters for
     widgets displaying text).  For example: `fred["geometry"] =
     "200x100"'.

justify
     Legal values are the strings: `"left"', `"center"', `"right"', and
     `"fill"'.

region
     This is a string with four space-delimited elements, each of which
     is a legal distance (see above).  For example: `"2 3 4 5"' and
     `"3i 2i 4.5i 2i"' and `"3c 2c 4c 10.43c"'  are all legal regions.

relief
     Determines what the border style of a widget will be.  Legal
     values are: `"raised"', `"sunken"', `"flat"', `"groove"', and
     `"ridge"'.

scrollcommand
     This is almost always the `set()' method of some scrollbar widget,
     but can be any widget method that takes a single argument.   Refer
     to the file `Demo/tkinter/matt/canvas-with-scrollbars.py' in the
     Python source distribution for an example.

wrap:
     Must be one of: `"none"', `"char"', or `"word"'.

5.24.1.15 Bindings and Events
.............................

The bind method from the widget command allows you to watch for certain
events and to have a callback function trigger when that event type
occurs.  The form of the bind method is:

    def bind(self, sequence, func, add=''):

where:

sequence
     is a string that denotes the target kind of event.  (See the bind
     man page and page 201 of John Ousterhout's book for details).

func
     is a Python function, taking one argument, to be invoked when the
     event occurs.  An Event instance will be passed as the argument.
     (Functions deployed this way are commonly known as _callbacks_.)

add
     is optional, either `''' or `'+''.  Passing an empty string
     denotes that this binding is to replace any other bindings that
     this event is associated with.  Passing a `'+'' means that this
     function is to be added to the list of functions bound to this
     event type.

  For example:

    def turnRed(self, event):
        event.widget["activeforeground"] = "red"

    self.button.bind("<Enter>", self.turnRed)

Notice how the widget field of the event is being accessed in the
`turnRed()' callback.  This field contains the widget that caught the X
event.  The following table lists the other event fields you can
access, and how they are denoted in Tk, which can be useful when
referring to the Tk man pages.

    Tk      Tkinter Event Field             Tk      Tkinter Event Field
    --      -------------------             --      -------------------
    %f      focus                           %A      char
    %h      height                          %E      send_event
    %k      keycode                         %K      keysym
    %s      state                           %N      keysym_num
    %t      time                            %T      type
    %w      width                           %W      widget
    %x      x                               %X      x_root
    %y      y                               %Y      y_root

5.24.1.16 The index Parameter
.............................

A number of widgets require"index" parameters to be passed.  These are
used to point at a specific place in a Text widget, or to particular
characters in an Entry widget, or to particular menu items in a Menu
widget.

Entry widget indexes (index, view index, etc.)
     Entry widgets have options that refer to character positions in
     the text being displayed.  You can use these Tkinter.
     functions to access these special points in text widgets:

    AtEnd()
          refers to the last position in the text

    AtInsert()
          refers to the point where the text cursor is

    AtSelFirst()
          indicates the beginning point of the selected text

    AtSelLast()
          denotes the last point of the selected text and finally

    At(x[, y])
          refers to the character at pixel location _x_, _y_ (with _y_
          not used in the case of a text entry widget, which contains a
          single line of text).

Text widget indexes
     The index notation for Text widgets is very rich and is best
     described in the Tk man pages.

Menu indexes (menu.invoke(), menu.entryconfig(), etc.)
     Some options and methods for menus manipulate specific menu
     entries. Anytime a menu index is needed for an option or a
     parameter, you may pass in:

        * an integer which refers to the numeric position of the entry
          in the widget, counted from the top, starting with 0;

        * the string `'active'', which refers to the menu position that
          is currently under the cursor;

        * the string `"last"' which refers to the last menu item;

        * An integer preceded by `@', as in `@6', where the integer is
          interpreted as a y pixel coordinate in the menu's coordinate
          system;

        * the string `"none"', which indicates no menu entry at all,
          most often used with menu.activate() to deactivate all
          entries, and finally,

        * a text string that is pattern matched against the label of
          the menu entry, as scanned from the top of the menu to the
          bottom.  Note that this index type is considered after all
          the others, which means that matches for menu items labelled
          `last', `active', or `none' may be interpreted as the above
          literals, instead.

5.24.1.17 Images
................

Bitmap/Pixelmap images can be created through the subclasses of
`Tkinter.Image':

   * `BitmapImage' can be used for X11 bitmap data.

   * `PhotoImage' can be used for GIF and PPM/PGM color bitmaps.

  Either type of image is created through either the `file' or the
`data' option (other options are available as well).

  The image object can then be used wherever an `image' option is
supported by some widget (e.g. labels, buttons, menus). In these cases,
Tk will not keep a reference to the image. When the last Python
reference to the image object is deleted, the image data is deleted as
well, and Tk will display an empty box wherever the image was used.

5.24.2 `ttk' -- Tk themed widgets
---------------------------------

The ttk module provides access to the Tk themed widget set,
which has been introduced in Tk 8.5. If Python is not compiled against
Tk 8.5 code may still use this module as long as Tile is installed.
However, some features provided by the new Tk, like anti-aliased font
rendering under X11, window transparency (on X11 you will need a
composition window manager) will be missing.

  The basic idea of ttk is to separate, to the extent
possible, the code implementing a widget's behavior from the code
implementing its appearance.

Tk Widget Styling Support (http://www.tcl.tk/cgi-bin/tct/tip/48)
     The document which brought up theming support for Tk

5.24.2.1 Using Ttk
..................

To start using Ttk, import its module:

    import ttk

But code like this:

    from Tkinter import *

may optionally want to use this:

    from Tkinter import *
    from ttk import *

And then several ttk widgets (`Button', `Checkbutton',
`Entry', `Frame', `Label', `LabelFrame', `Menubutton', `PanedWindow',
`Radiobutton', `Scale' and `Scrollbar') will automatically substitute
for the Tk widgets.

  This has the direct benefit of using the new widgets, giving better
look & feel across platforms, but be aware that they are not totally
compatible. The main difference is that widget options such as "fg",
"bg" and others related to widget styling are no longer present in Ttk
widgets. Use ttk.Style to achieve the same (or better)
styling.

Converting existing applications to use the Tile widgets (http://tktable.sourceforge.net/tile/doc/converting.txt)
     A text which talks in Tcl terms about differences typically found
     when converting applications to use the new widgets.

5.24.2.2 Ttk Widgets
....................

Ttk comes with 17 widgets, 11 of which already exist in Tkinter:
`Button', `Checkbutton', `Entry', `Frame', `Label', `LabelFrame',
`Menubutton', `PanedWindow', `Radiobutton', `Scale' and `Scrollbar'.
The 6 new widget classes are: Combobox, Notebook,
see Progressbar, `Separator', `Sizegrip' and
Treeview.  All of these classes are subclasses of Widget.

  As said previously, you will notice changes in look-and-feel as well
in the styling code. To demonstrate the latter, a very simple example
is shown below.

  Tk code:

    l1 = Tkinter.Label(text="Test", fg="black", bg="white")
    l2 = Tkinter.Label(text="Test", fg="black", bg="white")

Corresponding Ttk code:

    style = ttk.Style()
    style.configure("BW.TLabel", foreground="black", background="white")

    l1 = ttk.Label(text="Test", style="BW.TLabel")
    l2 = ttk.Label(text="Test", style="BW.TLabel")

For more information about TtkStyling read the
Style class documentation.

5.24.2.3 Widget
...............

ttk.Widget defines standard options and methods supported
by Tk themed widgets and is not supposed to be directly instantiated.

5.24.2.4 Standard Options
.........................

All the ttk widgets accept the following options:

     Option          Description
     -----------------------------------------------------------------------------------
     class           Specifies the window class. The class is used when querying the
                     option database for the window's other options, to determine the
                     default bindtags for the window, and to select the widget's
                     default layout and style. This is a read-only option which may
                     only be specified when the window is created.
     cursor          Specifies the mouse cursor to be used for the widget. If set to
                     the empty string (the default), the cursor is inherited from the
                     parent widget.
     takefocus       Determines whether the window accepts the focus during keyboard
                     traversal. 0, 1 or an empty string is returned.  If 0, the window
                     should be skipped entirely during keyboard traversal. If 1, the
                     window should receive the input focus as long as it is viewable.
                     An empty string means that the traversal scripts make the
                     decision about whether or not to focus on the window.
     style           May be used to specify a custom widget style.

5.24.2.5 Scrollable Widget Options
..................................

The following options are supported by widgets that are controlled by a
scrollbar.

     option               description
     -----------------------------------------------------------------------------------
     xscrollcommand       Used to communicate with horizontal scrollbars.

                          When the view in the widget's window changes, the widget
                          will generate a Tcl command based on the scrollcommand.

                          Usually this option consists of the `Scrollbar.set()' method
                          of some scrollbar. This will cause the scrollbar to be
                          updated whenever the view in the window changes.
     yscrollcommand       Used to communicate with vertical scrollbars.  For more
                          information, see above.

5.24.2.6 Label Options
......................

The following options are supported by labels, buttons and other
button-like widgets.

     option             description
     -----------------------------------------------------------------------------------
     text               Specifies a text string to be displayed inside the widget.
     textvariable       Specifies a name whose value will be used in place of the text
                        option resource.
     underline          If set, specifies the index (0-based) of a character to
                        underline in the text string. The underline character is used
                        for mnemonic activation.
     image              Specifies an image to display. This is a list of 1 or more
                        elements. The first element is the default image name. The
                        rest of the list is a sequence of statespec/value pairs as
                        defined by Style.map(), specifying different images
                        to use when the widget is in a particular state or a
                        combination of states. All images in the list should have the
                        same size.
     compound           Specifies how to display the image relative to the text, in
                        the case both text and image options are present.  Valid
                        values are:

                           * text: display text only

                           * image: display image only

                           * top, bottom, left, right: display image above, below,
                             left of, or right of the text, respectively.

                           * none: the default. display the image if present,
                             otherwise the text.
     width              If greater than zero, specifies how much space, in character
                        widths, to allocate for the text label; if less than zero,
                        specifies a minimum width. If zero or unspecified, the natural
                        width of the text label is used.

5.24.2.7 Compatibility Options
..............................

     option       description
     ----------------------------------------------------------------------------------
     state        May be set to "normal" or "disabled" to control the "disabled"
                  state bit. This is a write-only option: setting it changes the
                  widget state, but the Widget.state() method does not
                  affect this option.

5.24.2.8 Widget States
......................

The widget state is a bitmap of independent state flags.

     flag             description
     -----------------------------------------------------------------------------------
     active           The mouse cursor is over the widget and pressing a mouse button
                      will cause some action to occur.
     disabled         Widget is disabled under program control.
     focus            Widget has keyboard focus.
     pressed          Widget is being pressed.
     selected         "On", "true", or "current" for things like Checkbuttons and
                      radiobuttons.
     background       Windows and Mac have a notion of an "active" or foreground
                      window. The _background_ state is set for widgets in a
                      background window, and cleared for those in the foreground
                      window.
     readonly         Widget should not allow user modification.
     alternate        A widget-specific alternate display format.
     invalid          The widget's value is invalid.

  A state specification is a sequence of state names, optionally
prefixed with an exclamation point indicating that the bit is off.

5.24.2.9 ttk.Widget
...................

Besides the methods described below, the ttk.Widget class
supports the `Tkinter.Widget.cget()' and `Tkinter.Widget.configure()'
methods.

 -- Class: ttk.Widget
      -- Method: identify (x, y)
          Returns the name of the element at position _x_ _y_, or the
          empty string if the point does not lie within any element.

          _x_ and _y_ are pixel coordinates relative to the widget.

      -- Method: instate (statespec[, callback=None[, *args[, **kw]]])
          Test the widget's state. If a callback is not specified,
          returns True if the widget state matches _statespec_ and
          False otherwise. If callback is specified then it is called
          with _args_ if widget state matches _statespec_.

      -- Method: state ([statespec=None])
          Modify or read widget state. If _statespec_ is specified,
          sets the widget state accordingly and returns a new
          _statespec_ indicating which flags were changed. If
          _statespec_ is not specified, returns the currently-enabled
          state flags.

     _statespec_ will usually be a list or a tuple.

5.24.2.10 Combobox
..................

The ttk.Combobox widget combines a text field with a
pop-down list of values. This widget is a subclass of `Entry'.

  Besides the methods inherited from Widget.
(`Widget.cget()', `Widget.configure()', see Widget.identify(),
Widget.instate().  and Widget.state()) and
those inherited from `Entry' (`Entry.bbox()', `Entry.delete()',
`Entry.icursor()', `Entry.index()', `Entry.inset()',
`Entry.selection()', `Entry.xview()'), this class has some other
methods, described at ttk.Combobox.

5.24.2.11 Options
.................

This widget accepts the following options:

     option                description
     -----------------------------------------------------------------------------------
     exportselection       Boolean value. If set, the widget selection is linked to
                           the Window Manager selection (which can be returned by
                           invoking `Misc.selection_get()', for example).
     justify               Specifies how the text is aligned within the widget.  One
                           of "left", "center", or "right".
     height                Specifies the height of the pop-down listbox, in rows.
     postcommand           A script (possibly registered with `Misc.register()') that
                           is called immediately before displaying the values. It may
                           specify which values to display.
     state                 One of "normal", "readonly", or "disabled". In the
                           "readonly" state, the value may not be edited directly, and
                           the user can only select one of the values from the
                           dropdown list. In the "normal" state, the text field is
                           directly editable. In the "disabled" state, no interaction
                           is possible.
     textvariable          Specifies a name whose value is linked to the widget value.
                           Whenever the value associated with that name changes, the
                           widget value is updated, and vice versa.  See
                           `Tkinter.StringVar'.
     values                Specifies the list of values to display in the drop-down
                           listbox.
     width                 Specifies an integer value indicating the desired width of
                           the entry window, in average-size characters of the
                           widget's font.

5.24.2.12 Virtual events
........................

The combobox widget generates a *<<ComboboxSelected>>* virtual event
when the user selects an element from the list of values.

5.24.2.13 ttk.Combobox
......................

 -- Class: ttk.Combobox
      -- Method: current ([newindex=None])
          If _newindex_ is specified, sets the combobox value to the
          element position _newindex_. Otherwise, returns the index of
          the current value or -1 if the current value is not in the
          values list.

      -- Method: get ()
          Returns the current value of the combobox.

      -- Method: set (value)
          Sets the value of the combobox to _value_.

5.24.2.14 Notebook
..................

The Ttk Notebook widget manages a collection of windows and displays a
single one at a time. Each child window is associated with a tab, which
the user may select to change the currently-displayed window.

5.24.2.15 Options
.................

This widget accepts the following specific options:

     option        description
     -----------------------------------------------------------------------------------
     height        If present and greater than zero, specifies the desired height of
                   the pane area (not including internal padding or tabs).  Otherwise,
                   the maximum height of all panes is used.
     padding       Specifies the amount of extra space to add around the outside of
                   the notebook. The padding is a list of up to four length
                   specifications: left top right bottom. If fewer than four elements
                   are specified, bottom defaults to top, right defaults to left, and
                   top defaults to left.
     width         If present and greater than zero, specifies the desired width of
                   the pane area (not including internal padding). Otherwise, the
                   maximum width of all panes is used.

5.24.2.16 Tab Options
.....................

There are also specific options for tabs:

     option          description
     -----------------------------------------------------------------------------------
     state           Either "normal", "disabled" or "hidden". If "disabled", then the
                     tab is not selectable. If "hidden", then the tab is not shown.
     sticky          Specifies how the child window is positioned within the pane
                     area. Value is a string containing zero or more of the characters
                     "n", "s", "e" or "w". Each letter refers to a side (north, south,
                     east or west) that the child window will stick to, as per the
                     `grid()' geometry manager.
     padding         Specifies the amount of extra space to add between the notebook
                     and this pane. Syntax is the same as for the option padding used
                     by this widget.
     text            Specifies a text to be displayed in the tab.
     image           Specifies an image to display in the tab. See the option image
                     described in Widget.
     compound        Specifies how to display the image relative to the text, in the
                     case both text and image options are present. See Label Options
                     for legal values.
     underline       Specifies the index (0-based) of a character to underline in the
                     text string. The underlined character is used for mnemonic
                     activation if Notebook.enable_traversal() is called.

5.24.2.17 Tab Identifiers
.........................

The _tab_id_ present in several methods of ttk.Notebook.
may take any of the following forms:

   * An integer between zero and the number of tabs.

   * The name of a child window.

   * A positional specification of the form "@x,y", which identifies
     the tab.

   * The literal string "current", which identifies the
     currently-selected tab.

   * The literal string "end", which returns the number of tabs (only
     valid for Notebook.index()).

5.24.2.18 Virtual Events
........................

This widget generates a *<<NotebookTabChanged>>* virtual event after a
new tab is selected.

5.24.2.19 ttk.Notebook
......................

 -- Class: ttk.Notebook
      -- Method: add (child, **kw)
          Adds a new tab to the notebook.

          If window is currently managed by the notebook but hidden, it
          is restored to its previous position.

          See Tab Options for the list of available
          options.

      -- Method: forget (tab_id)
          Removes the tab specified by _tab_id_, unmaps and unmanages
          the associated window.

      -- Method: hide (tab_id)
          Hides the tab specified by _tab_id_.

          The tab will not be displayed, but the associated window
          remains managed by the notebook and its configuration
          remembered. Hidden tabs may be restored with the add()
          command.

      -- Method: identify (x, y)
          Returns the name of the tab element at position _x_, _y_, or
          the empty string if none.

      -- Method: index (tab_id)
          Returns the numeric index of the tab specified by _tab_id_,
          or the total number of tabs if _tab_id_ is the string "end".

      -- Method: insert (pos, child, **kw)
          Inserts a pane at the specified position.

          _pos_ is either the string "end", an integer index, or the
          name of a managed child. If _child_ is already managed by the
          notebook, moves it to the specified position.

          See Tab Options for the list of available
          options.

      -- Method: select ([tab_id])
          Selects the specified _tab_id_.

          The associated child window will be displayed, and the
          previously-selected window (if different) is unmapped. If
          _tab_id_ is omitted, returns the widget name of the currently
          selected pane.

      -- Method: tab (tab_id[, option=None[, **kw]])
          Query or modify the options of the specific _tab_id_.

          If _kw_ is not given, returns a dictionary of the tab option
          values. If _option_ is specified, returns the value of that
          _option_. Otherwise, sets the options to the corresponding
          values.

      -- Method: tabs ()
          Returns a list of windows managed by the notebook.

      -- Method: enable_traversal ()
          Enable keyboard traversal for a toplevel window containing
          this notebook.

          This will extend the bindings for the toplevel window
          containing the notebook as follows:

             * Control-Tab: selects the tab following the currently
               selected one.

             * Shift-Control-Tab: selects the tab preceding the
               currently selected one.

             * Alt-K: where K is the mnemonic (underlined) character of
               any tab, will select that tab.

          Multiple notebooks in a single toplevel may be enabled for
          traversal, including nested notebooks. However, notebook
          traversal only works properly if all panes have the notebook
          they are in as master.

5.24.2.20 Progressbar
.....................

The ttk.Progressbar widget shows the status of a
long-running operation. It can operate in two modes: determinate mode
shows the amount completed relative to the total amount of work to be
done, and indeterminate mode provides an animated display to let the
user know that something is happening.

5.24.2.21 Options
.................

This widget accepts the following specific options:

     option         description
     -----------------------------------------------------------------------------------
     orient         One of "horizontal" or "vertical". Specifies the orientation of
                    the progress bar.
     length         Specifies the length of the long axis of the progress bar (width
                    if horizontal, height if vertical).
     mode           One of "determinate" or "indeterminate".
     maximum        A number specifying the maximum value. Defaults to 100.
     value          The current value of the progress bar. In "determinate" mode, this
                    represents the amount of work completed. In "indeterminate" mode,
                    it is interpreted as modulo _maximum_; that is, the progress bar
                    completes one "cycle" when its value increases by _maximum_.
     variable       A name which is linked to the option value. If specified, the
                    value of the progress bar is automatically set to the value of
                    this name whenever the latter is modified.
     phase          Read-only option. The widget periodically increments the value of
                    this option whenever its value is greater than 0 and, in
                    determinate mode, less than maximum. This option may be used by
                    the current theme to provide additional animation effects.

5.24.2.22 ttk.Progressbar
.........................

 -- Class: ttk.Progressbar
      -- Method: start ([interval])
          Begin autoincrement mode: schedules a recurring timer event
          that calls Progressbar.step() every _interval_
          milliseconds. If omitted, _interval_ defaults to 50
          milliseconds.

      -- Method: step ([amount])
          Increments the progress bar's value by _amount_.

          _amount_ defaults to 1.0 if omitted.

      -- Method: stop ()
          Stop autoincrement mode: cancels any recurring timer event
          initiated by Progressbar.start() for this
          progress bar.

5.24.2.23 Separator
...................

The `ttk.Separator' widget displays a horizontal or vertical separator
bar.

  It has no other methods besides the ones inherited from
ttk.Widget.

5.24.2.24 Options
.................

This widget accepts the following specific option:

     option       description
     ----------------------------------------------------------------------------------
     orient       One of "horizontal" or "vertical". Specifies the orientation of the
                  separator.

5.24.2.25 Sizegrip
..................

The `ttk.Sizegrip' widget (also known as a grow box) allows the user to
resize the containing toplevel window by pressing and dragging the grip.

  This widget has neither specific options nor specific methods,
besides the ones inherited from ttk.Widget.

5.24.2.26 Platform-specific notes
.................................

   * On Mac OS X, toplevel windows automatically include a built-in
     size grip by default. Adding a `Sizegrip' is harmless, since the
     built-in grip will just mask the widget.

5.24.2.27 Bugs
..............

   * If the containing toplevel's position was specified relative to
     the right or bottom of the screen (e.g. ....), the `Sizegrip'
     widget will not resize the window.

   * This widget supports only "southeast" resizing.

5.24.2.28 Treeview
..................

The ttk.Treeview widget displays a hierarchical collection
of items.  Each item has a textual label, an optional image, and an
optional list of data values. The data values are displayed in
successive columns after the tree label.

  The order in which data values are displayed may be controlled by
setting the widget option `displaycolumns'. The tree widget can also
display column headings. Columns may be accessed by number or symbolic
names listed in the widget option columns. See Column Identifiers.

  Each item is identified by an unique name. The widget will generate
item IDs if they are not supplied by the caller. There is a
distinguished root item, named `{}'. The root item itself is not
displayed; its children appear at the top level of the hierarchy.

  Each item also has a list of tags, which can be used to associate
event bindings with individual items and control the appearance of the
item.

  The Treeview widget supports horizontal and vertical scrolling,
according to the options described in Scrollable Widget Options
and the methods Treeview.xview() and
Treeview.yview().

5.24.2.29 Options
.................

This widget accepts the following specific options:

     option               description
     ----------------------------------------------------------------------------------
     columns              A list of column identifiers, specifying the number of
                          columns and their names.
     displaycolumns       A list of column identifiers (either symbolic or integer
                          indices) specifying which data columns are displayed and
                          the order in which they appear, or the string "#all".
     height               Specifies the number of rows which should be visible.
                          Note: the requested width is determined from the sum of the
                          column widths.
     padding              Specifies the internal padding for the widget. The padding
                          is a list of up to four length specifications.
     selectmode           Controls how the built-in class bindings manage the
                          selection. One of "extended", "browse" or "none".  If set
                          to "extended" (the default), multiple items may be
                          selected. If "browse", only a single item will be selected
                          at a time. If "none", the selection will not be changed.

                          Note that the application code and tag bindings can set the
                          selection however they wish, regardless of the value  of
                          this option.
     show                 A list containing zero or more of the following values,
                          specifying which elements of the tree to display.

                             * tree: display tree labels in column #0.

                             * headings: display the heading row.
                          The default is "tree headings", i.e., show all elements.

                          *Note*: Column #0 always refers to the tree column, even if
                          show="tree" is not specified.

5.24.2.30 Item Options
......................

The following item options may be specified for items in the insert and
item widget commands.

     option       description
     ---------------------------------------------------------------------------------
     text         The textual label to display for the item.
     image        A Tk Image, displayed to the left of the label.
     values       The list of values associated with the item.

                  Each item should have the same number of values as the widget
                  option columns. If there are fewer values than columns, the
                  remaining values are assumed empty. If there are more values than
                  columns, the extra values are ignored.
     open         True/False value indicating whether the item's children should be
                  displayed or hidden.
     tags         A list of tags associated with this item.

5.24.2.31 Tag Options
.....................

The following options may be specified on tags:

     option           description
     ---------------------------------------------------------------------------------
     foreground       Specifies the text foreground color.
     background       Specifies the cell or item background color.
     font             Specifies the font to use when drawing text.
     image            Specifies the item image, in case the item's image option is
                      empty.

5.24.2.32 Column Identifiers
............................

Column identifiers take any of the following forms:

   * A symbolic name from the list of columns option.

   * An integer n, specifying the nth data column.

   * A string of the form #n, where n is an integer, specifying the nth
     display column.

  Notes:

   * Item's option values may be displayed in a different order than
     the order in which they are stored.

   * Column #0 always refers to the tree column, even if show="tree" is
     not specified.

  A data column number is an index into an item's option values list; a
display column number is the column number in the tree where the values
are displayed.  Tree labels are displayed in column #0. If option
displaycolumns is not set, then data column n is displayed in column
#n+1. Again, *column #0 always refers to the tree column*.

5.24.2.33 Virtual Events
........................

The Treeview widget generates the following virtual events.

     event                    description
     --------------------------------------------------------------------------------
     <<TreeviewSelect>>       Generated whenever the selection changes.
     <<TreeviewOpen>>         Generated just before settings the focus item to
                              open=True.
     <<TreeviewClose>>        Generated just after setting the focus item to
                              open=False.

  The Treeview.focus() and Treeview.selection()
methods can be used to determine the affected item or items.

5.24.2.34 ttk.Treeview
......................

 -- Class: ttk.Treeview
      -- Method: bbox (item[, column=None])
          Returns the bounding box (relative to the treeview widget's
          window) of the specified _item_ in the form (x, y, width,
          height).

          If _column_ is specified, returns the bounding box of that
          cell. If the _item_ is not visible (i.e., if it is a
          descendant of a closed item or is scrolled offscreen),
          returns an empty string.

      -- Method: get_children ([item])
          Returns the list of children belonging to _item_.

          If _item_ is not specified, returns root children.

      -- Method: set_children (item, *newchildren)
          Replaces _item_'s child with _newchildren_.

          Children present in _item_ that are not present in
          _newchildren_ are detached from the tree. No items in
          _newchildren_ may be an ancestor of _item_. Note that not
          specifying _newchildren_ results in detaching _item_'s
          children.

      -- Method: column (column[, option=None[, **kw]])
          Query or modify the options for the specified _column_.

          If _kw_ is not given, returns a dict of the column option
          values. If _option_ is specified then the value for that
          _option_ is returned.  Otherwise, sets the options to the
          corresponding values.

          The valid options/values are:

             *
              id
                    Returns the column name. This is a read-only option.

             *
              anchor: One of the standard Tk anchor values.
                    Specifies how the text in this column should be
                    aligned with respect to the cell.

             *
              minwidth: width
                    The minimum width of the column in pixels. The
                    treeview widget will not make the column any
                    smaller than specified by this option when the
                    widget is resized or the user drags a column.

             *
              stretch: True/False
                    Specifies whether the column's width should be
                    adjusted when the widget is resized.

             *
              width: width
                    The width of the column in pixels.

          To configure the tree column, call this with column = "#0"

      -- Method: delete (*items)
          Delete all specified _items_ and all their descendants.

          The root item may not be deleted.

      -- Method: detach (*items)
          Unlinks all of the specified _items_ from the tree.

          The items and all of their descendants are still present, and
          may be reinserted at another point in the tree, but will not
          be displayed.

          The root item may not be detached.

      -- Method: exists (item)
          Returns True if the specified _item_ is present in the tree.

      -- Method: focus ([item=None])
          If _item_ is specified, sets the focus item to _item_.
          Otherwise, returns the current focus item, or " if there is
          none.

      -- Method: heading (column[, option=None[, **kw]])
          Query or modify the heading options for the specified
          _column_.

          If _kw_ is not given, returns a dict of the heading option
          values. If _option_ is specified then the value for that
          _option_ is returned.  Otherwise, sets the options to the
          corresponding values.

          The valid options/values are:

             *
              text: text
                    The text to display in the column heading.

             *
              image: imageName
                    Specifies an image to display to the right of the
                    column heading.

             *
              anchor: anchor
                    Specifies how the heading text should be aligned.
                    One of the standard Tk anchor values.

             *
              command: callback
                    A callback to be invoked when the heading label is
                    pressed.

          To configure the tree column heading, call this with column =
          "#0".

      -- Method: identify (component, x, y)
          Returns a description of the specified _component_ under the
          point given by _x_ and _y_, or the empty string if no such
          _component_ is present at that position.

      -- Method: identify_row (y)
          Returns the item ID of the item at position _y_.

      -- Method: identify_column (x)
          Returns the data column identifier of the cell at position
          _x_.

          The tree column has ID #0.

      -- Method: identify_region (x, y)
          Returns one of:

          region          meaning
          -----------------------------------------------------------
          heading         Tree heading area.
          separator       Space between two columns headings.
          tree            The tree area.
          cell            A data cell.

          Availability: Tk 8.6.

      -- Method: identify_element (x, y)
          Returns the element at position _x_, _y_.

          Availability: Tk 8.6.

      -- Method: index (item)
          Returns the integer index of _item_ within its parent's list
          of children.

      -- Method: insert (parent, index[, iid=None[, **kw]])
          Creates a new item and returns the item identifier of the
          newly created item.

          _parent_ is the item ID of the parent item, or the empty
          string to create a new top-level item. _index_ is an integer,
          or the value "end", specifying where in the list of parent's
          children to insert the new item.  If _index_ is less than or
          equal to zero, the new node is inserted at the beginning; if
          _index_ is greater than or equal to the current number of
          children, it is inserted at the end. If _iid_ is specified,
          it is used as the item identifier; _iid_ must not already
          exist in the tree.  Otherwise, a new unique identifier is
          generated.

          See Item Options for the list of available
          points.

      -- Method: item (item[, option[, **kw]])
          Query or modify the options for the specified _item_.

          If no options are given, a dict with options/values for the
          item is returned.  If _option_ is specified then the value
          for that option is returned.  Otherwise, sets the options to
          the corresponding values as given by _kw_.

      -- Method: move (item, parent, index)
          Moves _item_ to position _index_ in _parent_'s list of
          children.

          It is illegal to move an item under one of its descendants.
          If _index_ is less than or equal to zero, _item_ is moved to
          the beginning; if greater than or equal to the number of
          children, it is moved to the end. If _item_ was detached it
          is reattached.

      -- Method: next (item)
          Returns the identifier of _item_'s next sibling, or " if
          _item_ is the last child of its parent.

      -- Method: parent (item)
          Returns the ID of the parent of _item_, or " if _item_ is at
          the top level of the hierarchy.

      -- Method: prev (item)
          Returns the identifier of _item_'s previous sibling, or " if
          _item_ is the first child of its parent.

      -- Method: reattach (item, parent, index)
          An alias for Treeview.move().

      -- Method: see (item)
          Ensure that _item_ is visible.

          Sets all of _item_'s ancestors open option to True, and
          scrolls the widget if necessary so that _item_ is within the
          visible portion of the tree.

      -- Method: selection ([selop=None[, items=None]])
          If _selop_ is not specified, returns selected items.
          Otherwise, it will act according to the following selection
          methods.

      -- Method: selection_set (items)
          _items_ becomes the new selection.

      -- Method: selection_add (items)
          Add _items_ to the selection.

      -- Method: selection_remove (items)
          Remove _items_ from the selection.

      -- Method: selection_toggle (items)
          Toggle the selection state of each item in _items_.

      -- Method: set (item[, column=None[, value=None]])
          With one argument, returns a dictionary of column/value pairs
          for the specified _item_. With two arguments, returns the
          current value of the specified _column_. With three
          arguments, sets the value of given _column_ in given _item_
          to the specified _value_.

      -- Method: tag_bind (tagname[, sequence=None[, callback=None]])
          Bind a callback for the given event _sequence_ to the tag
          _tagname_.  When an event is delivered to an item, the
          callbacks for each of the item's tags option are called.

      -- Method: tag_configure (tagname[, option=None[, **kw]])
          Query or modify the options for the specified _tagname_.

          If _kw_ is not given, returns a dict of the option settings
          for _tagname_. If _option_ is specified, returns the value
          for that _option_ for the specified _tagname_. Otherwise,
          sets the options to the corresponding values for the given
          _tagname_.

      -- Method: tag_has (tagname[, item])
          If _item_ is specified, returns 1 or 0 depending on whether
          the specified _item_ has the given _tagname_. Otherwise,
          returns a list of all items that have the specified tag.

          Availability: Tk 8.6

      -- Method: xview (*args)
          Query or modify horizontal position of the treeview.

      -- Method: yview (*args)
          Query or modify vertical position of the treeview.

5.24.2.35 Ttk Styling
.....................

Each widget in ttk is assigned a style, which specifies the
set of elements making up the widget and how they are arranged, along
with dynamic and default settings for element options. By default the
style name is the same as the widget's class name, but it may be
overridden by the widget's style option. If the class name of a widget
is unknown, use the method `Misc.winfo_class()'
(somewidget.winfo_class()).

Tcl'2004 conference presentation (http://tktable.sourceforge.net/tile/tile-tcl2004.pdf)
     This document explains how the theme engine works

 -- Class: ttk.Style
     This class is used to manipulate the style database.

      -- Method: configure (style, query_opt=None, **kw)
          Query or set the default value of the specified option(s) in
          _style_.

          Each key in _kw_ is an option and each value is a string
          identifying the value for that option.

          For example, to change every default button to be a flat
          button with some padding and a different background color do:

              import ttk
              import Tkinter

              root = Tkinter.Tk()

              ttk.Style().configure("TButton", padding=6, relief="flat",
                 background="#ccc")

              btn = ttk.Button(text="Sample")
              btn.pack()

              root.mainloop()

      -- Method: map (style, query_opt=None, **kw)
          Query or sets dynamic values of the specified option(s) in
          _style_.

          Each key in _kw_ is an option and each value should be a list
          or a tuple (usually) containing statespecs grouped in tuples,
          lists, or something else of your preference. A statespec is a
          compound of one or more states and then a value.

          An example:

              import Tkinter
              import ttk

              root = Tkinter.Tk()

              style = ttk.Style()
              style.map("C.TButton",
                  foreground=[('pressed', 'red'), ('active', 'blue')],
                  background=[('pressed', '!disabled', 'black'), ('active', 'white')]
                  )

              colored_btn = ttk.Button(text="Test", style="C.TButton").pack()

              root.mainloop()

          Note that the order of the (states, value) sequences for an
          option matters.  In the previous example, if you change the
          order to `[('active', 'blue'), ('pressed', 'red')]' in the
          foreground option, for example, you would get a blue
          foreground when the widget is in the active or pressed states.

      -- Method: lookup (style, option[, state=None[, default=None]])
          Returns the value specified for _option_ in _style_.

          If _state_ is specified, it is expected to be a sequence of
          one or more states. If the _default_ argument is set, it is
          used as a fallback value in case no specification for option
          is found.

          To check what font a Button uses by default, do:

              import ttk

              print ttk.Style().lookup("TButton", "font")

      -- Method: layout (style[, layoutspec=None])
          Define the widget layout for given _style_. If _layoutspec_
          is omitted, return the layout specification for given style.

          _layoutspec_, if specified, is expected to be a list or some
          other sequence type (excluding strings), where each item
          should be a tuple and the first item is the layout name and
          the second item should have the format described described in
          Layouts.

          To understand the format, see the following example (it is not
          intended to do anything useful):

              import ttk
              import Tkinter

              root = Tkinter.Tk()

              style = ttk.Style()
              style.layout("TMenubutton", [
                 ("Menubutton.background", None),
                 ("Menubutton.button", {"children":
                     [("Menubutton.focus", {"children":
                         [("Menubutton.padding", {"children":
                             [("Menubutton.label", {"side": "left", "expand": 1})]
                         })]
                     })]
                 }),
              ])

              mbtn = ttk.Menubutton(text='Text')
              mbtn.pack()
              root.mainloop()

      -- Method: element_create (elementname, etype, *args, **kw)
          Create a new element in the current theme, of the given
          _etype_ which is expected to be either "image", "from" or
          "vsapi". The latter is only available in Tk 8.6a for Windows
          XP and Vista and is not described here.

          If "image" is used, _args_ should contain the default image
          name followed by statespec/value pairs (this is the
          imagespec), and _kw_ may have the following options:

                  *
                   border=padding
                         padding is a list of up to four integers,
                         specifying the left, top, right, and bottom
                         borders, respectively.

                  *
                   height=height
                         Specifies a minimum height for the element. If
                         less than zero, the base image's height is
                         used as a default.

                  *
                   padding=padding
                         Specifies the element's interior padding.
                         Defaults to border's value if not specified.

                  *
                   sticky=spec
                         Specifies how the image is placed within the
                         final parcel. spec contains zero or more
                         characters “n”, “s”, “w”, or “e”.

                  *
                   width=width
                         Specifies a minimum width for the element. If
                         less than zero, the base image's width is used
                         as a default.

          If "from" is used as the value of _etype_, see
          element_create() will clone an existing element.
          _args_ is expected to contain a themename, from which the
          element will be cloned, and optionally an element to clone
          from.  If this element to clone from is not specified, an
          empty element will be used. _kw_ is discarded.

      -- Method: element_names ()
          Returns the list of elements defined in the current theme.

      -- Method: element_options (elementname)
          Returns the list of _elementname_'s options.

      -- Method: theme_create (themename[, parent=None[,
               settings=None]])
          Create a new theme.

          It is an error if _themename_ already exists. If _parent_ is
          specified, the new theme will inherit styles, elements and
          layouts from the parent theme. If _settings_ are present they
          are expected to have the same syntax used for
          theme_settings().

      -- Method: theme_settings (themename, settings)
          Temporarily sets the current theme to _themename_, apply
          specified _settings_ and then restore the previous theme.

          Each key in _settings_ is a style and each value may contain
          the keys 'configure', 'map', 'layout' and 'element create'
          and they are expected to have the same format as specified by
          the methods Style.configure(), Style.map(),
          see Style.layout() and
          Style.element_create() respectively.

          As an example, let's change the Combobox for the default
          theme a bit:

              import ttk
              import Tkinter

              root = Tkinter.Tk()

              style = ttk.Style()
              style.theme_settings("default", {
                 "TCombobox": {
                     "configure": {"padding": 5},
                     "map": {
                         "background": [("active", "green2"),
                                        ("!disabled", "green4")],
                         "fieldbackground": [("!disabled", "green3")],
                         "foreground": [("focus", "OliveDrab1"),
                                        ("!disabled", "OliveDrab2")]
                     }
                 }
              })

              combo = ttk.Combobox().pack()

              root.mainloop()

      -- Method: theme_names ()
          Returns a list of all known themes.

      -- Method: theme_use ([themename])
          If _themename_ is not given, returns the theme in use.
          Otherwise, sets the current theme to _themename_, refreshes
          all widgets and emits a <<ThemeChanged>> event.

5.24.2.36 Layouts
.................

A layout can be just None, if it takes no options, or a dict of options
specifying how to arrange the element. The layout mechanism uses a
simplified version of the pack geometry manager: given an initial
cavity, each element is allocated a parcel. Valid options/values are:

        *
         side: whichside
               Specifies which side of the cavity to place the element;
               one of top, right, bottom or left. If omitted, the
               element occupies the entire cavity.

        *
         sticky: nswe
               Specifies where the element is placed inside its
               allocated parcel.

        *
         unit: 0 or 1
               If set to 1, causes the element and all of its
               descendants to be treated as a single element for the
               purposes of Widget.identify() et al. It's
               used for things like scrollbar thumbs with grips.

        *
         children: [sublayout... ]
               Specifies a list of elements to place inside the
               element. Each element is a tuple (or other sequence
               type) where the first item is the layout name, and the
               other is a Layout.

5.24.3 `Tix' -- Extension widgets for Tk
----------------------------------------

The Tix. (Tk Interface Extension) module provides an
additional rich set of widgets. Although the standard Tk library has
many useful widgets, they are far from complete. The Tix.
library provides most of the commonly needed widgets that are missing
from standard Tk: HList, ComboBox,
Control. (a.k.a. SpinBox) and an assortment of scrollable widgets.
Tix also includes many more widgets that are generally
useful in a wide range of applications: NoteBook,
FileEntry, PanedWindow, etc; there are more than 40
of them.

  With all these new widgets, you can introduce new interaction
techniques into applications, creating more useful and more intuitive
user interfaces. You can design your application by choosing the most
appropriate widgets to match the special needs of your application and
users.

     Note: Tix has been renamed to `tkinter.tix' in Python
     3.0.  The 2to3 tool will automatically adapt imports
     when converting your sources to 3.0.

Tix Homepage (http://tix.sourceforge.net/)
     The home page for Tix.  This includes links to
     additional documentation and downloads.

Tix Man Pages (http://tix.sourceforge.net/dist/current/man/)
     On-line version of the man pages and reference material.

Tix Programming Guide (http://tix.sourceforge.net/dist/current/docs/tix-book/tix.book.html)
     On-line version of the programmer's reference material.

Tix Development Applications (http://tix.sourceforge.net/Tixapps/src/Tide.html)
     Tix applications for development of Tix and Tkinter programs. Tide
     applications work under Tk or Tkinter, and include *TixInspect*,
     an inspector to remotely modify and debug Tix/Tk/Tkinter
     applications.

5.24.3.1 Using Tix
..................

 -- Class: Tix.Tix (screenName[, baseName[, className]])
     Toplevel widget of Tix which represents mostly the main window of
     an application. It has an associated Tcl interpreter.

     Classes in the Tix module subclasses the classes in
     the Tkinter.  module. The former imports the latter, so
     to use Tix with Tkinter, all you need to do is to
     import one module. In general, you can just import Tix,
     and replace the toplevel call to Tkinter.Tk with
     `Tix.Tk':

         import Tix
         from Tkconstants import *
         root = Tix.Tk()

  To use Tix, you must have the Tix widgets
installed, usually alongside your installation of the Tk widgets. To
test your installation, try the following:

    import Tix
    root = Tix.Tk()
    root.tk.eval('package require Tix')

If this fails, you have a Tk installation problem which must be
resolved before proceeding. Use the environment variable `TIX_LIBRARY'
to point to the installed Tix library directory, and make
sure you have the dynamic object library (`tix8183.dll' or
`libtix8183.so') in  the same directory that contains your Tk dynamic
object library (`tk8183.dll' or `libtk8183.so'). The directory with the
dynamic object library should also have a file called `pkgIndex.tcl'
(case sensitive), which contains the line:

    package ifneeded Tix 8.1 [list load "[file join $dir tix8183.dll]" Tix]

5.24.3.2 Tix Widgets
....................

Tix
(http://tix.sourceforge.net/dist/current/man/html/TixCmd/TixIntro.htm)introduces
over 40 widget classes to the Tkinter.  repertoire.  There
is a demo of all the Tix widgets in the `Demo/tix'
directory of the standard distribution.

5.24.3.3 Basic Widgets
......................

 -- Class: Tix.Balloon
     A Balloon
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixBalloon.htm)that
     pops up over a widget to provide help.  When the user moves the
     cursor inside a widget to which a Balloon widget has been bound, a
     small pop-up window with a descriptive message will be shown on
     the screen.

 -- Class: Tix.ButtonBox
     The ButtonBox
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixButtonBox.htm)widget
     creates a box of buttons, such as is commonly used for `Ok Cancel'.

 -- Class: Tix.ComboBox
     The ComboBox
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixComboBox.htm)
     widget is similar to the combo box control in MS Windows. The user
     can select a choice by either typing in the entry subwdget or
     selecting from the listbox subwidget.

 -- Class: Tix.Control
     The Control
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixControl.htm)widget
     is also known as the `SpinBox' widget. The user can adjust the
     value by pressing the two arrow buttons or by entering the value
     directly into the entry. The new value will be checked against the
     user-defined upper and lower limits.

 -- Class: Tix.LabelEntry
     The LabelEntry
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixLabelEntry.htm)widget
     packages an entry widget and a label into one mega widget. It can
     be used be used to simplify the creation of "entry-form" type of
     interface.

 -- Class: Tix.LabelFrame
     The LabelFrame
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixLabelFrame.htm)widget
     packages a frame widget and a label into one mega widget.  To
     create widgets inside a LabelFrame widget, one creates the new
     widgets relative to the `frame' subwidget and manage them inside
     the `frame' subwidget.

 -- Class: Tix.Meter
     The Meter
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixMeter.htm)widget
     can be used to show the progress of a background job which may
     take a long time to execute.

 -- Class: Tix.OptionMenu
     The OptionMenu
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixOptionMenu.htm)creates
     a menu button of options.

 -- Class: Tix.PopupMenu
     The PopupMenu
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixPopupMenu.htm)widget
     can be used as a replacement of the `tk_popup' command. The
     advantage of the Tix. PopupMenu widget is
     it requires less application code to manipulate.

 -- Class: Tix.Select
     The Select
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixSelect.htm)widget
     is a container of button subwidgets. It can be used to provide
     radio-box or check-box style of selection options for the user.

 -- Class: Tix.StdButtonBox
     The StdButtonBox
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixStdButtonBox.htm)widget
     is a group of standard buttons for Motif-like dialog boxes.

5.24.3.4 File Selectors
.......................

 -- Class: Tix.DirList
     The DirList
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixDirList.htm)widget
     displays a list view of a directory, its previous directories and
     its sub-directories. The user can choose one of the directories
     displayed in the list or change to another directory.

 -- Class: Tix.DirTree
     The DirTree
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixDirTree.htm)widget
     displays a tree view of a directory, its previous directories and
     its sub-directories. The user can choose one of the directories
     displayed in the list or change to another directory.

 -- Class: Tix.DirSelectDialog
     The DirSelectDialog
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixDirSelectDialog.htm)widget
     presents the directories in the file system in a dialog window.
     The user can use this dialog window to navigate through the file
     system to select the desired directory.

 -- Class: Tix.DirSelectBox
     The DirSelectBox is similar to the standard Motif(TM)
     directory-selection box. It is generally used for the user to
     choose a directory.  DirSelectBox stores the directories mostly
     recently selected into a ComboBox widget so that they can be
     quickly selected again.

 -- Class: Tix.ExFileSelectBox
     The ExFileSelectBox
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixExFileSelectBox.htm)widget
     is usually embedded in a tixExFileSelectDialog widget. It provides
     an convenient method for the user to select files. The style of the
     ExFileSelectBox widget is very similar to the
     standard file dialog on MS Windows 3.1.

 -- Class: Tix.FileSelectBox
     The FileSelectBox
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixFileSelectBox.htm)is
     similar to the standard Motif(TM) file-selection box. It is
     generally used for the user to choose a file. FileSelectBox stores
     the files mostly recently selected into a ComboBox.
     widget so that they can be quickly selected again.

 -- Class: Tix.FileEntry
     The FileEntry
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixFileEntry.htm)widget
     can be used to input a filename. The user can type in the filename
     manually. Alternatively, the user can press the button widget that
     sits next to the entry, which will bring up a file selection
     dialog.

5.24.3.5 Hierarchical ListBox
.............................

 -- Class: Tix.HList
     The HList
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixHList.htm)widget
     can be used to display any data that have a hierarchical
     structure, for example, file system directory trees. The list
     entries are indented and connected by branch lines according to
     their places in the hierarchy.

 -- Class: Tix.CheckList
     The CheckList
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixCheckList.htm)widget
     displays a list of items to be selected by the user. CheckList acts
     similarly to the Tk checkbutton or radiobutton widgets, except it
     is capable of handling many more items than checkbuttons or
     radiobuttons.

 -- Class: Tix.Tree
     The Tree
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixTree.htm)widget
     can be used to display hierarchical data in a tree form. The user
     can adjust the view of the tree by opening or closing parts of the
     tree.

5.24.3.6 Tabular ListBox
........................

 -- Class: Tix.TList
     The TList
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixTList.htm)widget
     can be used to display data in a tabular format. The list entries
     of a TList widget are similar to the entries in the
     Tk listbox widget.  The main differences are (1) the TList
     widget can display the list entries in a two dimensional
     format and (2) you can use graphical images as well as multiple
     colors and fonts for the list entries.

5.24.3.7 Manager Widgets
........................

 -- Class: Tix.PanedWindow
     The PanedWindow
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixPanedWindow.htm)widget
     allows the user to interactively manipulate the sizes of several
     panes.  The panes can be arranged either vertically or
     horizontally.  The user changes the sizes of the panes by dragging
     the resize handle between two panes.

 -- Class: Tix.ListNoteBook
     The ListNoteBook
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixListNoteBook.htm)widget
     is very similar to the `TixNoteBook' widget: it can be used to
     display many windows in a limited space using a notebook metaphor.
     The notebook is divided into a stack of pages (windows). At one
     time only one of these pages can be shown. The user can navigate
     through these pages by choosing the name of the desired page in
     the `hlist' subwidget.

 -- Class: Tix.NoteBook
     The NoteBook
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixNoteBook.htm)
     widget can be used to display many windows in a limited space
     using a notebook metaphor. The notebook is divided into a stack of
     pages. At one time only one of these pages can be shown. The user
     can navigate through these pages by choosing the visual "tabs" at
     the top of the NoteBook widget.

5.24.3.8 Image Types
....................

The Tix module adds:

   * pixmap
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/pixmap.htm)capabilities
     to all Tix and Tkinter widgets to create
     color images from XPM files.

   * Compound
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/compound.htm)
     image types can be used to create images that consists of multiple
     horizontal lines; each line is composed of a series of items
     (texts, bitmaps, images or spaces) arranged from left to right.
     For example, a compound image can be used to display a bitmap and
     a text string simultaneously in a Tk `Button' widget.

5.24.3.9 Miscellaneous Widgets
..............................

 -- Class: Tix.InputOnly
     The InputOnly
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixInputOnly.htm)widgets
     are to accept inputs from the user, which can be done with the
     `bind' command (Unix only).

5.24.3.10 Form Geometry Manager
...............................

In addition, see Tix augments Tkinter by providing:

 -- Class: Tix.Form
     The Form
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixForm.htm)geometry
     manager based on attachment rules for all Tk widgets.

5.24.3.11 Tix Commands
......................

 -- Class: Tix.tixCommand
     The tix commands
     (http://tix.sourceforge.net/dist/current/man/html/TixCmd/tix.htm)
     provide access to miscellaneous elements of Tix.'s
     internal state and the Tix application context.  Most
     of the information manipulated by these methods pertains to the
     application as a whole, or to a screen or display, rather than to
     a particular window.

     To view the current settings, the common usage is:

         import Tix
         root = Tix.Tk()
         print root.tix_configure()

 -- Method: tixCommand.tix_configure ([cnf], **kw)
     Query or modify the configuration options of the Tix application
     context. If no option is specified, returns a dictionary all of
     the available options.  If option is specified with no value, then
     the method returns a list describing the one named option (this
     list will be identical to the corresponding sublist of the value
     returned if no option is specified).  If one or more option-value
     pairs are specified, then the method modifies the given option(s)
     to have the given value(s); in this case the method returns an
     empty string. Option may be any of the configuration options.

 -- Method: tixCommand.tix_cget (option)
     Returns the current value of the configuration option given by
     _option_. Option may be any of the configuration options.

 -- Method: tixCommand.tix_getbitmap (name)
     Locates a bitmap file of the name `name.xpm' or `name' in one of
     the bitmap directories (see the tix_addbitmapdir().
     method).  By using tix_getbitmap(), you can avoid hard
     coding the pathnames of the bitmap files in your application. When
     successful, it returns the complete pathname of the bitmap file,
     prefixed with the character `@'.  The returned value can be used
     to configure the `bitmap' option of the Tk and Tix widgets.

 -- Method: tixCommand.tix_addbitmapdir (directory)
     Tix maintains a list of directories under which the
     tix_getimage() and tix_getbitmap() methods
     will search for image files.  The standard bitmap directory is
     `$TIX_LIBRARY/bitmaps'. The tix_addbitmapdir() method
     adds _directory_ into this list. By using this method, the image
     files of an applications can also be located using the
     tix_getimage() or tix_getbitmap() method.

 -- Method: tixCommand.tix_filedialog ([dlgclass])
     Returns the file selection dialog that may be shared among
     different calls from this application.  This method will create a
     file selection dialog widget when it is called the first time.
     This dialog will be returned by all subsequent calls to
     tix_filedialog().  An optional dlgclass parameter can be
     passed as a string to specified what type of file selection dialog
     widget is desired.  Possible options are `tix', `FileSelectDialog'
     or `tixExFileSelectDialog'.

 -- Method: tixCommand.tix_getimage (self, name)
     Locates an image file of the name `name.xpm', `name.xbm' or
     `name.ppm' in one of the bitmap directories (see the
     tix_addbitmapdir() method above). If more than one file
     with the same name (but different extensions) exist, then the
     image type is chosen according to the depth of the X display: xbm
     images are chosen on monochrome displays and color images are
     chosen on color displays. By using tix_getimage(), you
     can avoid hard coding the pathnames of the image files in your
     application. When successful, this method returns the name of the
     newly created image, which can be used to configure the `image'
     option of the Tk and Tix widgets.

 -- Method: tixCommand.tix_option_get (name)
     Gets the options maintained by the Tix scheme mechanism.

 -- Method: tixCommand.tix_resetoptions (newScheme, newFontSet[,
          newScmPrio])
     Resets the scheme and fontset of the Tix application to
     _newScheme_ and _newFontSet_, respectively.  This affects only
     those widgets created after this call.  Therefore, it is best to
     call the resetoptions method before the creation of any widgets in
     a Tix application.

     The optional parameter _newScmPrio_ can be given to reset the
     priority level of the Tk options set by the Tix schemes.

     Because of the way Tk handles the X option database, after Tix has
     been has imported and inited, it is not possible to reset the
     color schemes and font sets using the `tix_config()' method.
     Instead, the tix_resetoptions().  method must be used.

5.24.4 `ScrolledText' -- Scrolled Text Widget
---------------------------------------------

The ScrolledText module provides a class of the same name
which implements a basic text widget which has a vertical scroll bar
configured to do the "right thing."  Using the ScrolledText.
class is a lot easier than setting up a text widget and scroll bar
directly.  The constructor is the same as that of the `Tkinter.Text'
class.

     Note: ScrolledText has been renamed to
     `tkinter.scrolledtext' in Python 3.0.  The 2to3 tool
     will automatically adapt imports when converting your sources to
     3.0.

  The text widget and scrollbar are packed together in a `Frame', and
the methods of the `Grid' and `Pack' geometry managers are acquired
from the `Frame' object.  This allows the ScrolledText.
widget to be used directly to achieve most normal geometry management
behavior.

  Should more specific control be necessary, the following attributes
are available:

 -- Attribute: ScrolledText.frame
     The frame which surrounds the text and scroll bar widgets.

 -- Attribute: ScrolledText.vbar
     The scroll bar widget.

5.24.5 `turtle' -- Turtle graphics for Tk
-----------------------------------------

5.24.5.1 Introduction
.....................

Turtle graphics is a popular way for introducing programming to kids.
It was part of the original Logo programming language developed by
Wally Feurzig and Seymour Papert in 1966.

  Imagine a robotic turtle starting at (0, 0) in the x-y plane.  Give
it the command `turtle.forward(15)', and it moves (on-screen!) 15
pixels in the direction it is facing, drawing a line as it moves.  Give
it the command `turtle.left(25)', and it rotates in-place 25 degrees
clockwise.

  By combining together these and similar commands, intricate shapes
and pictures can easily be drawn.

  The turtle module is an extended reimplementation of the
same-named module from the Python standard distribution up to version
Python 2.5.

  It tries to keep the merits of the old turtle module and to be
(nearly) 100% compatible with it.  This means in the first place to
enable the learning programmer to use all the commands, classes and
methods interactively when using the module from within IDLE run with
the `-n' switch.

  The turtle module provides turtle graphics primitives, in both
object-oriented and procedure-oriented ways.  Because it uses
Tkinter for the underlying graphics, it needs a version of Python
installed with Tk support.

  The object-oriented interface uses essentially two+two classes:

  1. The TurtleScreen class defines graphics windows as a
     playground for the drawing turtles.  Its constructor needs a
     `Tkinter.Canvas' or a ScrolledCanvas as argument.  It
     should be used when turtle is used as part of some
     application.

     The function Screen() returns a singleton object of a
     TurtleScreen subclass. This function should be used
     when turtle is used as a standalone tool for doing
     graphics.  As a singleton object, inheriting from its class is not
     possible.

     All methods of TurtleScreen/Screen also exist as functions, i.e.
     as part of the procedure-oriented interface.

  2. RawTurtle. (alias: RawPen) defines Turtle
     objects which draw on a TurtleScreen.  Its constructor
     needs a Canvas, ScrolledCanvas or TurtleScreen as argument, so the
     RawTurtle objects know where to draw.

     Derived from RawTurtle is the subclass Turtle. (alias:
     `Pen'), which draws on "the" Screen. - instance which
     is automatically created, if not already present.

     All methods of RawTurtle/Turtle also exist as functions, i.e. part
     of the procedure-oriented interface.

  The procedural interface provides functions which are derived from
the methods of the classes Screen and Turtle.
They have the same names as the corresponding methods.  A screen object
is automatically created whenever a function derived from a Screen
method is called.  An (unnamed) turtle object is automatically created
whenever any of the functions derived from a Turtle method is called.

  To use multiple turtles an a screen one has to use the
object-oriented interface.

     Note: In the following documentation the argument list for
     functions is given.  Methods, of course, have the additional first
     argument _self_ which is omitted here.

5.24.5.2 Overview over available Turtle and Screen methods
..........................................................

5.24.5.3 Turtle methods
.......................

Turtle motion

    Move and draw
              See forward(). | See fd().
              See backward(). | See bk(). | See back().
              See right(). | See rt().
              See left(). | See lt().
              See goto(). | See setpos(). | See setposition().
              See setx().
              See sety().
              See setheading(). | See seth().
              See home().
              circle().
              dot().
              stamp().
              clearstamp().
              clearstamps().
              undo().
              speed().

    Tell Turtle's state
              See position(). | See pos().
              See towards().
              xcor().
              ycor().
              heading().
              distance().

    Setting and measurement
              See degrees().
              radians().

Pen control

    Drawing state
              See pendown(). | See pd(). | See down().
              See penup(). | See pu(). | See up().
              See pensize(). | See width().
              See pen().
              isdown().

    Color control
              See color().
              pencolor().
              fillcolor().

    Filling
              See fill().
              begin_fill().
              end_fill().

    More drawing control
              See reset().
              clear().
              write().

Turtle state

    Visibility
              See showturtle(). | See st().
              See hideturtle(). | See ht().
              See isvisible().

    Appearance
              See shape().
              See resizemode().
              See shapesize(). | See turtlesize().
              See settiltangle().
              tiltangle().
              tilt().

Using events
         See onclick().
         onrelease().
         ondrag().

Special Turtle methods
         See begin_poly().
         end_poly().
         get_poly().
         See clone().
         See getturtle(). | See getpen().
         See getscreen().
         setundobuffer().
         undobufferentries().
         tracer().
         window_width().
         window_height().

5.24.5.4 Methods of TurtleScreen/Screen
.......................................

Window control
         See bgcolor().
         See bgpic().
         See clear(). | See clearscreen().
         See reset(). | See resetscreen().
         See screensize().
         setworldcoordinates().

Animation control
         See delay().
         tracer().
         update().

Using screen events
         See listen().
         See onkey().
         See onclick(). | See onscreenclick().
         See ontimer().

Settings and special methods
         See mode().
         colormode().
         getcanvas().
         See getshapes().
         See register_shape(). | See addshape().
         See turtles().
         window_height().
         window_width().

Methods specific to Screen
         See bye().
         exitonclick().
         setup().
         title().

5.24.5.5 Methods of RawTurtle/Turtle and corresponding functions
................................................................

Most of the examples in this section refer to a Turtle instance called
`turtle'.

5.24.5.6 Turtle motion
......................

 -- Function: turtle.forward (distance)
 -- Function: turtle.fd (distance)
          Parameters: *distance* - a number (integer or float)

     Move the turtle forward by the specified _distance_, in the
     direction the turtle is headed.

         >>> turtle.position()
         (0.00,0.00)
         >>> turtle.forward(25)
         >>> turtle.position()
         (25.00,0.00)
         >>> turtle.forward(-75)
         >>> turtle.position()
         (-50.00,0.00)

 -- Function: turtle.back (distance)
 -- Function: turtle.bk (distance)
 -- Function: turtle.backward (distance)
          Parameters: *distance* - a number

     Move the turtle backward by _distance_, opposite to the direction
     the turtle is headed.  Do not change the turtle's heading.

         >>> turtle.position()
         (0.00,0.00)
         >>> turtle.backward(30)
         >>> turtle.position()
         (-30.00,0.00)

 -- Function: turtle.right (angle)
 -- Function: turtle.rt (angle)
          Parameters: *angle* - a number (integer or float)

     Turn turtle right by _angle_ units.  (Units are by default
     degrees, but can be set via the degrees() and
     radians() functions.)  Angle orientation depends on the
     turtle mode, see mode().

         >>> turtle.heading()
         22.0
         >>> turtle.right(45)
         >>> turtle.heading()
         337.0

 -- Function: turtle.left (angle)
 -- Function: turtle.lt (angle)
          Parameters: *angle* - a number (integer or float)

     Turn turtle left by _angle_ units.  (Units are by default degrees,
     but can be set via the degrees() and radians()
     functions.)  Angle orientation depends on the turtle mode,
     see mode().

         >>> turtle.heading()
         22.0
         >>> turtle.left(45)
         >>> turtle.heading()
         67.0

 -- Function: turtle.goto (x, y=None)
 -- Function: turtle.setpos (x, y=None)
 -- Function: turtle.setposition (x, y=None)
          Parameters:

             * *x* - a number or a pair/vector of numbers

             * *y* - a number or `None'

     If _y_ is `None', _x_ must be a pair of coordinates or a
     Vec2D.  (e.g. as returned by pos()).

     Move turtle to an absolute position.  If the pen is down, draw
     line.  Do not change the turtle's orientation.

         >>> tp = turtle.pos()
         >>> tp
         (0.00,0.00)
         >>> turtle.setpos(60,30)
         >>> turtle.pos()
         (60.00,30.00)
         >>> turtle.setpos((20,80))
         >>> turtle.pos()
         (20.00,80.00)
         >>> turtle.setpos(tp)
         >>> turtle.pos()
         (0.00,0.00)

 -- Function: turtle.setx (x)
          Parameters: *x* - a number (integer or float)

     Set the turtle's first coordinate to _x_, leave second coordinate
     unchanged.

         >>> turtle.position()
         (0.00,240.00)
         >>> turtle.setx(10)
         >>> turtle.position()
         (10.00,240.00)

 -- Function: turtle.sety (y)
          Parameters: *y* - a number (integer or float)

     Set the turtle's second coordinate to _y_, leave first coordinate
     unchanged.

         >>> turtle.position()
         (0.00,40.00)
         >>> turtle.sety(-10)
         >>> turtle.position()
         (0.00,-10.00)

 -- Function: turtle.setheading (to_angle)
 -- Function: turtle.seth (to_angle)
          Parameters: *to_angle* - a number (integer or float)

     Set the orientation of the turtle to _to_angle_.  Here are some
     common directions in degrees:

     standard mode           logo mode
     -------------------------------------------------
     0 - east                0 - north
     90 - north              90 - east
     180 - west              180 - south
     270 - south             270 - west

         >>> turtle.setheading(90)
         >>> turtle.heading()
         90.0

 -- Function: turtle.home ()
     Move turtle to the origin - coordinates (0,0) - and set its
     heading to its start-orientation (which depends on the mode, see
     mode()).

         >>> turtle.heading()
         90.0
         >>> turtle.position()
         (0.00,-10.00)
         >>> turtle.home()
         >>> turtle.position()
         (0.00,0.00)
         >>> turtle.heading()
         0.0

 -- Function: turtle.circle (radius, extent=None, steps=None)
          Parameters:

             * *radius* - a number

             * *extent* - a number (or `None')

             * *steps* - an integer (or `None')

     Draw a circle with given _radius_.  The center is _radius_ units
     left of the turtle; _extent_ - an angle - determines which part of
     the circle is drawn.  If _extent_ is not given, draw the entire
     circle.  If _extent_ is not a full circle, one endpoint of the arc
     is the current pen position.  Draw the arc in counterclockwise
     direction if _radius_ is positive, otherwise in clockwise
     direction.  Finally the direction of the turtle is changed by the
     amount of _extent_.

     As the circle is approximated by an inscribed regular polygon,
     _steps_ determines the number of steps to use.  If not given, it
     will be calculated automatically.  May be used to draw regular
     polygons.

         >>> turtle.home()
         >>> turtle.position()
         (0.00,0.00)
         >>> turtle.heading()
         0.0
         >>> turtle.circle(50)
         >>> turtle.position()
         (-0.00,0.00)
         >>> turtle.heading()
         0.0
         >>> turtle.circle(120, 180)  # draw a semicircle
         >>> turtle.position()
         (0.00,240.00)
         >>> turtle.heading()
         180.0

 -- Function: turtle.dot (size=None, *color)
          Parameters:

             * *size* - an integer >= 1 (if given)

             * *color* - a colorstring or a numeric color tuple

     Draw a circular dot with diameter _size_, using _color_.  If
     _size_ is not given, the maximum of pensize+4 and 2*pensize is
     used.

         >>> turtle.home()
         >>> turtle.dot()
         >>> turtle.fd(50); turtle.dot(20, "blue"); turtle.fd(50)
         >>> turtle.position()
         (100.00,-0.00)
         >>> turtle.heading()
         0.0

 -- Function: turtle.stamp ()
     Stamp a copy of the turtle shape onto the canvas at the current
     turtle position.  Return a stamp_id for that stamp, which can be
     used to delete it by calling `clearstamp(stamp_id)'.

         >>> turtle.color("blue")
         >>> turtle.stamp()
         11
         >>> turtle.fd(50)

 -- Function: turtle.clearstamp (stampid)
          Parameters: *stampid* - an integer, must be return value of
          previous stamp() call

     Delete stamp with given _stampid_.

         >>> turtle.position()
         (150.00,-0.00)
         >>> turtle.color("blue")
         >>> astamp = turtle.stamp()
         >>> turtle.fd(50)
         >>> turtle.position()
         (200.00,-0.00)
         >>> turtle.clearstamp(astamp)
         >>> turtle.position()
         (200.00,-0.00)

 -- Function: turtle.clearstamps (n=None)
          Parameters: *n* - an integer (or `None')

     Delete all or first/last _n_ of turtle's stamps.  If _n_ is None,
     delete all stamps, if _n_ > 0 delete first _n_ stamps, else if _n_
     < 0 delete last _n_ stamps.

         >>> for i in range(8):
         ...     turtle.stamp(); turtle.fd(30)
         13
         14
         15
         16
         17
         18
         19
         20
         >>> turtle.clearstamps(2)
         >>> turtle.clearstamps(-2)
         >>> turtle.clearstamps()

 -- Function: turtle.undo ()
     Undo (repeatedly) the last turtle action(s).  Number of available
     undo actions is determined by the size of the undobuffer.

         >>> for i in range(4):
         ...     turtle.fd(50); turtle.lt(80)
         ...
         >>> for i in range(8):
         ...     turtle.undo()

 -- Function: turtle.speed (speed=None)
          Parameters: *speed* - an integer in the range 0..10 or a
          speedstring (see below)

     Set the turtle's speed to an integer value in the range 0..10.  If
     no argument is given, return current speed.

     If input is a number greater than 10 or smaller than 0.5, speed is
     set to 0.  Speedstrings are mapped to speedvalues as follows:

        * "fastest":  0

        * "fast":  10

        * "normal":  6

        * "slow":  3

        * "slowest":  1

     Speeds from 1 to 10 enforce increasingly faster animation of line
     drawing and turtle turning.

     Attention: _speed_ = 0 means that _no_ animation takes place.
     forward/back makes turtle jump and likewise left/right make the
     turtle turn instantly.

         >>> turtle.speed()
         3
         >>> turtle.speed('normal')
         >>> turtle.speed()
         6
         >>> turtle.speed(9)
         >>> turtle.speed()
         9

5.24.5.7 Tell Turtle's state
............................

 -- Function: turtle.position ()
 -- Function: turtle.pos ()
     Return the turtle's current location (x,y) (as a Vec2D
     vector).

         >>> turtle.pos()
         (440.00,-0.00)

 -- Function: turtle.towards (x, y=None)
          Parameters:

             * *x* - a number or a pair/vector of numbers or a turtle
               instance

             * *y* - a number if _x_ is a number, else `None'

     Return the angle between the line from turtle position to position
     specified by (x,y), the vector or the other turtle.  This depends
     on the turtle's start orientation which depends on the mode -
     "standard"/"world" or "logo").

         >>> turtle.goto(10, 10)
         >>> turtle.towards(0,0)
         225.0

 -- Function: turtle.xcor ()
     Return the turtle's x coordinate.

         >>> turtle.home()
         >>> turtle.left(50)
         >>> turtle.forward(100)
         >>> turtle.pos()
         (64.28,76.60)
         >>> print turtle.xcor()
         64.2787609687

 -- Function: turtle.ycor ()
     Return the turtle's y coordinate.

         >>> turtle.home()
         >>> turtle.left(60)
         >>> turtle.forward(100)
         >>> print turtle.pos()
         (50.00,86.60)
         >>> print turtle.ycor()
         86.6025403784

 -- Function: turtle.heading ()
     Return the turtle's current heading (value depends on the turtle
     mode, see mode()).

         >>> turtle.home()
         >>> turtle.left(67)
         >>> turtle.heading()
         67.0

 -- Function: turtle.distance (x, y=None)
          Parameters:

             * *x* - a number or a pair/vector of numbers or a turtle
               instance

             * *y* - a number if _x_ is a number, else `None'

     Return the distance from the turtle to (x,y), the given vector, or
     the given other turtle, in turtle step units.

         >>> turtle.home()
         >>> turtle.distance(30,40)
         50.0
         >>> turtle.distance((30,40))
         50.0
         >>> joe = Turtle()
         >>> joe.forward(77)
         >>> turtle.distance(joe)
         77.0

5.24.5.8 Settings for measurement
.................................

 -- Function: turtle.degrees (fullcircle=360.0)
          Parameters: *fullcircle* - a number

     Set angle measurement units, i.e. set number of "degrees" for a
     full circle.  Default value is 360 degrees.

         >>> turtle.home()
         >>> turtle.left(90)
         >>> turtle.heading()
         90.0

         Change angle measurement unit to grad (also known as gon,
         grade, or gradian and equals 1/100-th of the right angle.)
         >>> turtle.degrees(400.0)
         >>> turtle.heading()
         100.0
         >>> turtle.degrees(360)
         >>> turtle.heading()
         90.0

 -- Function: turtle.radians ()
     Set the angle measurement units to radians.  Equivalent to
     `degrees(2*math.pi)'.

         >>> turtle.home()
         >>> turtle.left(90)
         >>> turtle.heading()
         90.0
         >>> turtle.radians()
         >>> turtle.heading()
         1.5707963267948966

5.24.5.9 Pen control
....................

5.24.5.10 Drawing state
.......................

 -- Function: turtle.pendown ()
 -- Function: turtle.pd ()
 -- Function: turtle.down ()
     Pull the pen down - drawing when moving.

 -- Function: turtle.penup ()
 -- Function: turtle.pu ()
 -- Function: turtle.up ()
     Pull the pen up - no drawing when moving.

 -- Function: turtle.pensize (width=None)
 -- Function: turtle.width (width=None)
          Parameters: *width* - a positive number

     Set the line thickness to _width_ or return it.  If resizemode is
     set to "auto" and turtleshape is a polygon, that polygon is drawn
     with the same line thickness.  If no argument is given, the
     current pensize is returned.

         >>> turtle.pensize()
         1
         >>> turtle.pensize(10)   # from here on lines of width 10 are drawn

 -- Function: turtle.pen (pen=None, **pendict)
          Parameters:

             * *pen* - a dictionary with some or all of the below
               listed keys

             * *pendict* - one or more keyword-arguments with the below
               listed keys as keywords

     Return or set the pen's attributes in a "pen-dictionary" with the
     following key/value pairs:

        * "shown": True/False

        * "pendown": True/False

        * "pencolor": color-string or color-tuple

        * "fillcolor": color-string or color-tuple

        * "pensize": positive number

        * "speed": number in range 0..10

        * "resizemode": "auto" or "user" or "noresize"

        * "stretchfactor": (positive number, positive number)

        * "outline": positive number

        * "tilt": number

     This dictionary can be used as argument for a subsequent call to
     pen().  to restore the former pen-state.  Moreover one
     or more of these attributes can be provided as keyword-arguments.
     This can be used to set several pen attributes in one statement.

         >>> turtle.pen(fillcolor="black", pencolor="red", pensize=10)
         >>> sorted(turtle.pen().items())
         [('fillcolor', 'black'), ('outline', 1), ('pencolor', 'red'),
          ('pendown', True), ('pensize', 10), ('resizemode', 'noresize'),
          ('shown', True), ('speed', 9), ('stretchfactor', (1, 1)), ('tilt', 0)]
         >>> penstate=turtle.pen()
         >>> turtle.color("yellow", "")
         >>> turtle.penup()
         >>> sorted(turtle.pen().items())
         [('fillcolor', ''), ('outline', 1), ('pencolor', 'yellow'),
          ('pendown', False), ('pensize', 10), ('resizemode', 'noresize'),
          ('shown', True), ('speed', 9), ('stretchfactor', (1, 1)), ('tilt', 0)]
         >>> turtle.pen(penstate, fillcolor="green")
         >>> sorted(turtle.pen().items())
         [('fillcolor', 'green'), ('outline', 1), ('pencolor', 'red'),
          ('pendown', True), ('pensize', 10), ('resizemode', 'noresize'),
          ('shown', True), ('speed', 9), ('stretchfactor', (1, 1)), ('tilt', 0)]

 -- Function: turtle.isdown ()
     Return `True' if pen is down, `False' if it's up.

         >>> turtle.penup()
         >>> turtle.isdown()
         False
         >>> turtle.pendown()
         >>> turtle.isdown()
         True

5.24.5.11 Color control
.......................

 -- Function: turtle.pencolor (*args)
     Return or set the pencolor.

     Four input formats are allowed:

    `pencolor()'
          Return the current pencolor as color specification string or
          as a tuple (see example).  May be used as input to another
          color/pencolor/fillcolor call.

    `pencolor(colorstring)'
          Set pencolor to _colorstring_, which is a Tk color
          specification string, such as `"red"', `"yellow"', or
          `"#33cc8c"'.

    `pencolor((r, g, b))'
          Set pencolor to the RGB color represented by the tuple of
          _r_, _g_, and _b_.  Each of _r_, _g_, and _b_ must be in the
          range 0..colormode, where colormode is either 1.0 or 255 (see
          colormode()).

    `pencolor(r, g, b)'
               Set pencolor to the RGB color represented by _r_, _g_,
               and _b_.  Each of _r_, _g_, and _b_ must be in the range
               0..colormode.

          If turtleshape is a polygon, the outline of that polygon is
          drawn with the newly set pencolor.

         >>> colormode()
         1.0
         >>> turtle.pencolor()
         'red'
         >>> turtle.pencolor("brown")
         >>> turtle.pencolor()
         'brown'
         >>> tup = (0.2, 0.8, 0.55)
         >>> turtle.pencolor(tup)
         >>> turtle.pencolor()
         (0.2, 0.8, 0.5490196078431373)
         >>> colormode(255)
         >>> turtle.pencolor()
         (51, 204, 140)
         >>> turtle.pencolor('#32c18f')
         >>> turtle.pencolor()
         (50, 193, 143)

 -- Function: turtle.fillcolor (*args)
     Return or set the fillcolor.

     Four input formats are allowed:

    `fillcolor()'
          Return the current fillcolor as color specification string,
          possibly in tuple format (see example).  May be used as input
          to another color/pencolor/fillcolor call.

    `fillcolor(colorstring)'
          Set fillcolor to _colorstring_, which is a Tk color
          specification string, such as `"red"', `"yellow"', or
          `"#33cc8c"'.

    `fillcolor((r, g, b))'
          Set fillcolor to the RGB color represented by the tuple of
          _r_, _g_, and _b_.  Each of _r_, _g_, and _b_ must be in the
          range 0..colormode, where colormode is either 1.0 or 255 (see
          colormode()).

    `fillcolor(r, g, b)'
               Set fillcolor to the RGB color represented by _r_, _g_,
               and _b_.  Each of _r_, _g_, and _b_ must be in the range
               0..colormode.

          If turtleshape is a polygon, the interior of that polygon is
          drawn with the newly set fillcolor.

         >>> turtle.fillcolor("violet")
         >>> turtle.fillcolor()
         'violet'
         >>> col = turtle.pencolor()
         >>> col
         (50, 193, 143)
         >>> turtle.fillcolor(col)
         >>> turtle.fillcolor()
         (50, 193, 143)
         >>> turtle.fillcolor('#ffffff')
         >>> turtle.fillcolor()
         (255, 255, 255)

 -- Function: turtle.color (*args)
     Return or set pencolor and fillcolor.

     Several input formats are allowed.  They use 0 to 3 arguments as
     follows:

    `color()'
          Return the current pencolor and the current fillcolor as a
          pair of color specification strings or tuples as returned by
          pencolor() and fillcolor().

    `color(colorstring)', `color((r,g,b))', `color(r,g,b)'
          Inputs as in pencolor(), set both, fillcolor and
          pencolor, to the given value.

    `color(colorstring1, colorstring2)', `color((r1,g1,b1), (r2,g2,b2))'
               Equivalent to `pencolor(colorstring1)' and
               `fillcolor(colorstring2)' and analogously if the other
               input format is used.

          If turtleshape is a polygon, outline and interior of that
          polygon is drawn with the newly set colors.

         >>> turtle.color("red", "green")
         >>> turtle.color()
         ('red', 'green')
         >>> color("#285078", "#a0c8f0")
         >>> color()
         ((40, 80, 120), (160, 200, 240))

  See also: Screen method colormode().

5.24.5.12 Filling
.................

 -- Function: turtle.fill (flag)
          Parameters: *flag* - True/False (or 1/0 respectively)

     Call `fill(True)' before drawing the shape you want to fill, and
     `fill(False)' when done.  When used without argument: return
     fillstate (`True' if filling, `False' else).

         >>> turtle.fill(True)
         >>> for _ in range(3):
         ...    turtle.forward(100)
         ...    turtle.left(120)
         ...
         >>> turtle.fill(False)

 -- Function: turtle.begin_fill ()
     Call just before drawing a shape to be filled.  Equivalent to
     `fill(True)'.

 -- Function: turtle.end_fill ()
     Fill the shape drawn after the last call to begin_fill().
     Equivalent to `fill(False)'.

         >>> turtle.color("black", "red")
         >>> turtle.begin_fill()
         >>> turtle.circle(80)
         >>> turtle.end_fill()

5.24.5.13 More drawing control
..............................

 -- Function: turtle.reset ()
     Delete the turtle's drawings from the screen, re-center the turtle
     and set variables to the default values.

         >>> turtle.goto(0,-22)
         >>> turtle.left(100)
         >>> turtle.position()
         (0.00,-22.00)
         >>> turtle.heading()
         100.0
         >>> turtle.reset()
         >>> turtle.position()
         (0.00,0.00)
         >>> turtle.heading()
         0.0

 -- Function: turtle.clear ()
     Delete the turtle's drawings from the screen.  Do not move turtle.
     State and position of the turtle as well as drawings of other
     turtles are not affected.

 -- Function: turtle.write (arg, move=False, align="left",
          font=("Arial", 8, "normal"))
          Parameters:

             * *arg* - object to be written to the TurtleScreen

             * *move* - True/False

             * *align* - one of the strings "left", "center" or right"

             * *font* - a triple (fontname, fontsize, fonttype)

     Write text - the string representation of _arg_ - at the current
     turtle position according to _align_ ("left", "center" or right")
     and with the given font.  If _move_ is True, the pen is moved to
     the bottom-right corner of the text.  By default, _move_ is False.

         >>> turtle.write("Home = ", True, align="center")
         >>> turtle.write((0,0), True)

5.24.5.14 Turtle state
......................

5.24.5.15 Visibility
....................

 -- Function: turtle.hideturtle ()
 -- Function: turtle.ht ()
     Make the turtle invisible.  It's a good idea to do this while
     you're in the middle of doing some complex drawing, because hiding
     the turtle speeds up the drawing observably.

         >>> turtle.hideturtle()

 -- Function: turtle.showturtle ()
 -- Function: turtle.st ()
     Make the turtle visible.

         >>> turtle.showturtle()

 -- Function: turtle.isvisible ()
     Return True if the Turtle is shown, False if it's hidden.

         >>> turtle.hideturtle()
         >>> turtle.isvisible()
         False
         >>> turtle.showturtle()
         >>> turtle.isvisible()
         True

5.24.5.16 Appearance
....................

 -- Function: turtle.shape (name=None)
          Parameters: *name* - a string which is a valid shapename

     Set turtle shape to shape with given _name_ or, if name is not
     given, return name of current shape.  Shape with _name_ must exist
     in the TurtleScreen's shape dictionary.  Initially there are the
     following polygon shapes: "arrow", "turtle", "circle", "square",
     "triangle", "classic".  To learn about how to deal with shapes see
     Screen method register_shape().

         >>> turtle.shape()
         'classic'
         >>> turtle.shape("turtle")
         >>> turtle.shape()
         'turtle'

 -- Function: turtle.resizemode (rmode=None)
          Parameters: *rmode* - one of the strings "auto", "user",
          "noresize"

     Set resizemode to one of the values: "auto", "user", "noresize".
     If _rmode_ is not given, return current resizemode.  Different
     resizemodes have the following effects:

        - "auto": adapts the appearance of the turtle corresponding to
          the value of pensize.

        - "user": adapts the appearance of the turtle according to the
          values of stretchfactor and outlinewidth (outline), which are
          set by shapesize().

        - "noresize": no adaption of the turtle's appearance takes
          place.

     resizemode("user") is called by shapesize() when used
     with arguments.

         >>> turtle.resizemode()
         'noresize'
         >>> turtle.resizemode("auto")
         >>> turtle.resizemode()
         'auto'

 -- Function: turtle.shapesize (stretch_wid=None, stretch_len=None,
          outline=None)
 -- Function: turtle.turtlesize (stretch_wid=None, stretch_len=None,
          outline=None)
          Parameters:

             * *stretch_wid* - positive number

             * *stretch_len* - positive number

             * *outline* - positive number

     Return or set the pen's attributes x/y-stretchfactors and/or
     outline.  Set resizemode to "user".  If and only if resizemode is
     set to "user", the turtle will be displayed stretched according to
     its stretchfactors: _stretch_wid_ is stretchfactor perpendicular
     to its orientation, _stretch_len_ is stretchfactor in direction of
     its orientation, _outline_ determines the width of the shapes's
     outline.

         >>> turtle.shapesize()
         (1, 1, 1)
         >>> turtle.resizemode("user")
         >>> turtle.shapesize(5, 5, 12)
         >>> turtle.shapesize()
         (5, 5, 12)
         >>> turtle.shapesize(outline=8)
         >>> turtle.shapesize()
         (5, 5, 8)

 -- Function: turtle.tilt (angle)
          Parameters: *angle* - a number

     Rotate the turtleshape by _angle_ from its current tilt-angle, but
     do _not_ change the turtle's heading (direction of movement).

         >>> turtle.reset()
         >>> turtle.shape("circle")
         >>> turtle.shapesize(5,2)
         >>> turtle.tilt(30)
         >>> turtle.fd(50)
         >>> turtle.tilt(30)
         >>> turtle.fd(50)

 -- Function: turtle.settiltangle (angle)
          Parameters: *angle* - a number

     Rotate the turtleshape to point in the direction specified by
     _angle_, regardless of its current tilt-angle.  _Do not_ change
     the turtle's heading (direction of movement).

         >>> turtle.reset()
         >>> turtle.shape("circle")
         >>> turtle.shapesize(5,2)
         >>> turtle.settiltangle(45)
         >>> turtle.fd(50)
         >>> turtle.settiltangle(-45)
         >>> turtle.fd(50)

 -- Function: turtle.tiltangle ()
     Return the current tilt-angle, i.e. the angle between the
     orientation of the turtleshape and the heading of the turtle (its
     direction of movement).

         >>> turtle.reset()
         >>> turtle.shape("circle")
         >>> turtle.shapesize(5,2)
         >>> turtle.tilt(45)
         >>> turtle.tiltangle()
         45.0

5.24.5.17 Using events
......................

 -- Function: turtle.onclick (fun, btn=1, add=None)
          Parameters:

             * *fun* - a function with two arguments which will be
               called with the coordinates of the clicked point on the
               canvas

             * *num* - number of the mouse-button, defaults to 1 (left
               mouse button)

             * *add* - `True' or `False' - if `True', a new binding
               will be added, otherwise it will replace a former binding

     Bind _fun_ to mouse-click events on this turtle.  If _fun_ is
     `None', existing bindings are removed.  Example for the anonymous
     turtle, i.e. the procedural way:

         >>> def turn(x, y):
         ...     left(180)
         ...
         >>> onclick(turn)  # Now clicking into the turtle will turn it.
         >>> onclick(None)  # event-binding will be removed

 -- Function: turtle.onrelease (fun, btn=1, add=None)
          Parameters:

             * *fun* - a function with two arguments which will be
               called with the coordinates of the clicked point on the
               canvas

             * *num* - number of the mouse-button, defaults to 1 (left
               mouse button)

             * *add* - `True' or `False' - if `True', a new binding
               will be added, otherwise it will replace a former binding

     Bind _fun_ to mouse-button-release events on this turtle.  If
     _fun_ is `None', existing bindings are removed.

         >>> class MyTurtle(Turtle):
         ...     def glow(self,x,y):
         ...         self.fillcolor("red")
         ...     def unglow(self,x,y):
         ...         self.fillcolor("")
         ...
         >>> turtle = MyTurtle()
         >>> turtle.onclick(turtle.glow)     # clicking on turtle turns fillcolor red,
         >>> turtle.onrelease(turtle.unglow) # releasing turns it to transparent.

 -- Function: turtle.ondrag (fun, btn=1, add=None)
          Parameters:

             * *fun* - a function with two arguments which will be
               called with the coordinates of the clicked point on the
               canvas

             * *num* - number of the mouse-button, defaults to 1 (left
               mouse button)

             * *add* - `True' or `False' - if `True', a new binding
               will be added, otherwise it will replace a former binding

     Bind _fun_ to mouse-move events on this turtle.  If _fun_ is
     `None', existing bindings are removed.

     Remark: Every sequence of mouse-move-events on a turtle is
     preceded by a mouse-click event on that turtle.

         >>> turtle.ondrag(turtle.goto)

     Subsequently, clicking and dragging the Turtle will move it across
     the screen thereby producing handdrawings (if pen is down).

5.24.5.18 Special Turtle methods
................................

 -- Function: turtle.begin_poly ()
     Start recording the vertices of a polygon.  Current turtle
     position is first vertex of polygon.

 -- Function: turtle.end_poly ()
     Stop recording the vertices of a polygon.  Current turtle position
     is last vertex of polygon.  This will be connected with the first
     vertex.

 -- Function: turtle.get_poly ()
     Return the last recorded polygon.

         >>> turtle.home()
         >>> turtle.begin_poly()
         >>> turtle.fd(100)
         >>> turtle.left(20)
         >>> turtle.fd(30)
         >>> turtle.left(60)
         >>> turtle.fd(50)
         >>> turtle.end_poly()
         >>> p = turtle.get_poly()
         >>> register_shape("myFavouriteShape", p)

 -- Function: turtle.clone ()
     Create and return a clone of the turtle with same position,
     heading and turtle properties.

         >>> mick = Turtle()
         >>> joe = mick.clone()

 -- Function: turtle.getturtle ()
 -- Function: turtle.getpen ()
     Return the Turtle object itself.  Only reasonable use: as a
     function to return the "anonymous turtle":

         >>> pet = getturtle()
         >>> pet.fd(50)
         >>> pet
         <turtle.Turtle object at 0x...>

 -- Function: turtle.getscreen ()
     Return the TurtleScreen object the turtle is drawing
     on.  TurtleScreen methods can then be called for that object.

         >>> ts = turtle.getscreen()
         >>> ts
         <turtle._Screen object at 0x...>
         >>> ts.bgcolor("pink")

 -- Function: turtle.setundobuffer (size)
          Parameters: *size* - an integer or `None'

     Set or disable undobuffer.  If _size_ is an integer an empty
     undobuffer of given size is installed.  _size_ gives the maximum
     number of turtle actions that can be undone by the undo()
     method/function.  If _size_ is `None', the undobuffer is
     disabled.

         >>> turtle.setundobuffer(42)

 -- Function: turtle.undobufferentries ()
     Return number of entries in the undobuffer.

         >>> while undobufferentries():
         ...     undo()

 -- Function: turtle.tracer (flag=None, delay=None)
     A replica of the corresponding TurtleScreen method.

     Deprecated since version 2.6.

 -- Function: turtle.window_width ()
 -- Function: turtle.window_height ()
     Both are replicas of the corresponding TurtleScreen methods.

     Deprecated since version 2.6.

5.24.5.19 Excursus about the use of compound shapes
...................................................

To use compound turtle shapes, which consist of several polygons of
different color, you must use the helper class Shape.
explicitly as described below:

  1. Create an empty Shape object of type "compound".

  2. Add as many components to this object as desired, using the
     `addcomponent()' method.

     For example:

         >>> s = Shape("compound")
         >>> poly1 = ((0,0),(10,-5),(0,10),(-10,-5))
         >>> s.addcomponent(poly1, "red", "blue")
         >>> poly2 = ((0,0),(10,-5),(-10,-5))
         >>> s.addcomponent(poly2, "blue", "red")

  3. Now add the Shape to the Screen's shapelist and use it:

         >>> register_shape("myshape", s)
         >>> shape("myshape")

     Note: The Shape class is used internally by the
     register_shape().  method in different ways.  The
     application programmer has to deal with the Shape class _only_
     when using compound shapes like shown above!

5.24.5.20 Methods of TurtleScreen/Screen and corresponding functions
....................................................................

Most of the examples in this section refer to a TurtleScreen instance
called `screen'.

5.24.5.21 Window control
........................

 -- Function: turtle.bgcolor (*args)
          Parameters: *args* - a color string or three numbers in the
          range 0..colormode or a 3-tuple of such numbers

     Set or return background color of the TurtleScreen.

         >>> screen.bgcolor("orange")
         >>> screen.bgcolor()
         'orange'
         >>> screen.bgcolor("#800080")
         >>> screen.bgcolor()
         (128, 0, 128)

 -- Function: turtle.bgpic (picname=None)
          Parameters: *picname* - a string, name of a gif-file or
          `"nopic"', or `None'

     Set background image or return name of current backgroundimage.
     If _picname_ is a filename, set the corresponding image as
     background.  If _picname_ is `"nopic"', delete background image,
     if present.  If _picname_ is `None', return the filename of the
     current backgroundimage.

         >>> screen.bgpic()
         'nopic'
         >>> screen.bgpic("landscape.gif")
         >>> screen.bgpic()
         "landscape.gif"

 -- Function: turtle.clear ()
 -- Function: turtle.clearscreen ()
     Delete all drawings and all turtles from the TurtleScreen.  Reset
     the now empty TurtleScreen to its initial state: white background,
     no background image, no event bindings and tracing on.

          Note: This TurtleScreen method is available as a global
          function only under the name `clearscreen'.  The global
          function `clear' is another one derived from the Turtle
          method `clear'.

 -- Function: turtle.reset ()
 -- Function: turtle.resetscreen ()
     Reset all Turtles on the Screen to their initial state.

          Note: This TurtleScreen method is available as a global
          function only under the name `resetscreen'.  The global
          function `reset' is another one derived from the Turtle
          method `reset'.

 -- Function: turtle.screensize (canvwidth=None, canvheight=None,
          bg=None)
          Parameters:

             * *canvwidth* - positive integer, new width of canvas in
               pixels

             * *canvheight* - positive integer, new height of canvas in
               pixels

             * *bg* - colorstring or color-tuple, new background color

     If no arguments are given, return current (canvaswidth,
     canvasheight).  Else resize the canvas the turtles are drawing on.
     Do not alter the drawing window.  To observe hidden parts of the
     canvas, use the scrollbars. With this method, one can make visible
     those parts of a drawing which were outside the canvas before.

         >>> screen.screensize()
         (400, 300)
         >>> screen.screensize(2000,1500)
         >>> screen.screensize()
         (2000, 1500)

     e.g. to search for an erroneously escaped turtle ;-)

 -- Function: turtle.setworldcoordinates (llx, lly, urx, ury)
          Parameters:

             * *llx* - a number, x-coordinate of lower left corner of
               canvas

             * *lly* - a number, y-coordinate of lower left corner of
               canvas

             * *urx* - a number, x-coordinate of upper right corner of
               canvas

             * *ury* - a number, y-coordinate of upper right corner of
               canvas

     Set up user-defined coordinate system and switch to mode "world" if
     necessary.  This performs a `screen.reset()'.  If mode "world" is
     already active, all drawings are redrawn according to the new
     coordinates.

     *ATTENTION*: in user-defined coordinate systems angles may appear
     distorted.

         >>> screen.reset()
         >>> screen.setworldcoordinates(-50,-7.5,50,7.5)
         >>> for _ in range(72):
         ...     left(10)
         ...
         >>> for _ in range(8):
         ...     left(45); fd(2)   # a regular octagon

5.24.5.22 Animation control
...........................

 -- Function: turtle.delay (delay=None)
          Parameters: *delay* - positive integer

     Set or return the drawing _delay_ in milliseconds.  (This is
     approximately the time interval between two consecutive canvas
     updates.)  The longer the drawing delay, the slower the animation.

     Optional argument:

         >>> screen.delay()
         10
         >>> screen.delay(5)
         >>> screen.delay()
         5

 -- Function: turtle.tracer (n=None, delay=None)
          Parameters:

             * *n* - nonnegative integer

             * *delay* - nonnegative integer

     Turn turtle animation on/off and set delay for update drawings.
     If _n_ is given, only each n-th regular screen update is really
     performed.  (Can be used to accelerate the drawing of complex
     graphics.)  Second argument sets delay value (see delay()).

         >>> screen.tracer(8, 25)
         >>> dist = 2
         >>> for i in range(200):
         ...     fd(dist)
         ...     rt(90)
         ...     dist += 2

 -- Function: turtle.update ()
     Perform a TurtleScreen update. To be used when tracer is turned
     off.

  See also the RawTurtle/Turtle method speed().

5.24.5.23 Using screen events
.............................

 -- Function: turtle.listen (xdummy=None, ydummy=None)
     Set focus on TurtleScreen (in order to collect key-events).  Dummy
     arguments are provided in order to be able to pass listen()
     to the onclick method.

 -- Function: turtle.onkey (fun, key)
          Parameters:

             * *fun* - a function with no arguments or `None'

             * *key* - a string: key (e.g. "a") or key-symbol (e.g.
               "space")

     Bind _fun_ to key-release event of key.  If _fun_ is `None', event
     bindings are removed. Remark: in order to be able to register
     key-events, TurtleScreen must have the focus. (See method
     listen())

         >>> def f():
         ...     fd(50)
         ...     lt(60)
         ...
         >>> screen.onkey(f, "Up")
         >>> screen.listen()

 -- Function: turtle.onclick (fun, btn=1, add=None)
 -- Function: turtle.onscreenclick (fun, btn=1, add=None)
          Parameters:

             * *fun* - a function with two arguments which will be
               called with the coordinates of the clicked point on the
               canvas

             * *num* - number of the mouse-button, defaults to 1 (left
               mouse button)

             * *add* - `True' or `False' - if `True', a new binding
               will be added, otherwise it will replace a former binding

     Bind _fun_ to mouse-click events on this screen.  If _fun_ is
     `None', existing bindings are removed.

     Example for a TurtleScreen instance named `screen' and a Turtle
     instance named turtle:

         >>> screen.onclick(turtle.goto) # Subsequently clicking into the TurtleScreen will
         >>>                             # make the turtle move to the clicked point.
         >>> screen.onclick(None)        # remove event binding again

          Note: This TurtleScreen method is available as a global
          function only under the name `onscreenclick'.  The global
          function `onclick' is another one derived from the Turtle
          method `onclick'.

 -- Function: turtle.ontimer (fun, t=0)
          Parameters:

             * *fun* - a function with no arguments

             * *t* - a number >= 0

     Install a timer that calls _fun_ after _t_ milliseconds.

         >>> running = True
         >>> def f():
         ...     if running:
         ...         fd(50)
         ...         lt(60)
         ...         screen.ontimer(f, 250)
         >>> f()   ### makes the turtle march around
         >>> running = False

5.24.5.24 Settings and special methods
......................................

 -- Function: turtle.mode (mode=None)
          Parameters: *mode* - one of the strings "standard", "logo" or
          "world"

     Set turtle mode ("standard", "logo" or "world") and perform reset.
     If mode is not given, current mode is returned.

     Mode "standard" is compatible with old turtle.  Mode
     "logo" is compatible with most Logo turtle graphics.  Mode "world"
     uses user-defined "world coordinates". *Attention*: in this mode
     angles appear distorted if `x/y' unit-ratio doesn't equal 1.

     Mode             Initial turtle heading        positive angles
     -----------------------------------------------------------------------
     "standard"       to the right (east)           counterclockwise
     "logo"           upward    (north)             clockwise

         >>> mode("logo")   # resets turtle heading to north
         >>> mode()
         'logo'

 -- Function: turtle.colormode (cmode=None)
          Parameters: *cmode* - one of the values 1.0 or 255

     Return the colormode or set it to 1.0 or 255.  Subsequently _r_,
     _g_, _b_ values of color triples have to be in the range
     0.._cmode_.

         >>> screen.colormode(1)
         >>> turtle.pencolor(240, 160, 80)
         Traceback (most recent call last):
              ...
         TurtleGraphicsError: bad color sequence: (240, 160, 80)
         >>> screen.colormode()
         1.0
         >>> screen.colormode(255)
         >>> screen.colormode()
         255
         >>> turtle.pencolor(240,160,80)

 -- Function: turtle.getcanvas ()
     Return the Canvas of this TurtleScreen.  Useful for insiders who
     know what to do with a Tkinter Canvas.

         >>> cv = screen.getcanvas()
         >>> cv
         <turtle.ScrolledCanvas instance at 0x...>

 -- Function: turtle.getshapes ()
     Return a list of names of all currently available turtle shapes.

         >>> screen.getshapes()
         ['arrow', 'blank', 'circle', ..., 'turtle']

 -- Function: turtle.register_shape (name, shape=None)
 -- Function: turtle.addshape (name, shape=None)
     There are three different ways to call this function:

       1. _name_ is the name of a gif-file and _shape_ is `None':
          Install the corresponding image shape.

              >>> screen.register_shape("turtle.gif")

               Note: Image shapes _do not_ rotate when turning the
               turtle, so they do not display the heading of the turtle!

       2. _name_ is an arbitrary string and _shape_ is a tuple of pairs
          of coordinates: Install the corresponding polygon shape.

              >>> screen.register_shape("triangle", ((5,-3), (0,5), (-5,-3)))

       3. _name_ is an arbitrary string and shape is a (compound)
          Shape.  object: Install the corresponding compound
          shape.

     Add a turtle shape to TurtleScreen's shapelist.  Only thusly
     registered shapes can be used by issuing the command
     `shape(shapename)'.

 -- Function: turtle.turtles ()
     Return the list of turtles on the screen.

         >>> for turtle in screen.turtles():
         ...     turtle.color("red")

 -- Function: turtle.window_height ()
     Return the height of the turtle window.

         >>> screen.window_height()
         480

 -- Function: turtle.window_width ()
     Return the width of the turtle window.

         >>> screen.window_width()
         640

5.24.5.25 Methods specific to Screen, not inherited from TurtleScreen
.....................................................................

 -- Function: turtle.bye ()
     Shut the turtlegraphics window.

 -- Function: turtle.exitonclick ()
     Bind bye() method to mouse clicks on the Screen.

     If the value "using_IDLE" in the configuration dictionary is
     `False' (default value), also enter mainloop.  Remark: If IDLE
     with the `-n' switch (no subprocess) is used, this value should be
     set to `True' in `turtle.cfg'.  In this case IDLE's own mainloop
     is active also for the client script.

 -- Function: turtle.setup (width=_CFG["width"], height=_CFG["height"],
          startx=_CFG["leftright"], starty=_CFG["topbottom"])
     Set the size and position of the main window.  Default values of
     arguments are stored in the configuration dictionary and can be
     changed via a `turtle.cfg' file.

          Parameters:

             * *width* - if an integer, a size in pixels, if a float, a
               fraction of the screen; default is 50% of screen

             * *height* - if an integer, the height in pixels, if a
               float, a fraction of the screen; default is 75% of screen

             * *startx* - if positive, starting position in pixels from
               the left edge of the screen, if negative from the right
               edge, if None, center window horizontally

             * *startx* - if positive, starting position in pixels from
               the top edge of the screen, if negative from the bottom
               edge, if None, center window vertically

         >>> screen.setup (width=200, height=200, startx=0, starty=0)
         >>>              # sets window to 200x200 pixels, in upper left of screen
         >>> screen.setup(width=.75, height=0.5, startx=None, starty=None)
         >>>              # sets window to 75% of screen by 50% of screen and centers

 -- Function: turtle.title (titlestring)
          Parameters: *titlestring* - a string that is shown in the
          titlebar of the turtle graphics window

     Set title of turtle window to _titlestring_.

         >>> screen.title("Welcome to the turtle zoo!")

5.24.5.26 The public classes of the module `turtle'
...................................................

 -- Class: turtle.RawTurtle (canvas)
 -- Class: turtle.RawPen (canvas)
          Parameters: *canvas* - a `Tkinter.Canvas', a
          ScrolledCanvas or a TurtleScreen.

     Create a turtle.  The turtle has all methods described above as
     "methods of Turtle/RawTurtle".

 -- Class: turtle.Turtle
     Subclass of RawTurtle, has the same interface but draws on a
     default Screen object created automatically when
     needed for the first time.

 -- Class: turtle.TurtleScreen (cv)
          Parameters: *cv* - a `Tkinter.Canvas'

     Provides screen oriented methods like `setbg()' etc. that are
     described above.

 -- Class: turtle.Screen
     Subclass of TurtleScreen, with four methods added.

 -- Class: turtle.ScrolledCanvas (master)
          Parameters: *master* - some Tkinter widget to contain the
          ScrolledCanvas, i.e.  a Tkinter-canvas with scrollbars added

     Used by class Screen, which thus automatically provides a
     ScrolledCanvas as playground for the turtles.

 -- Class: turtle.Shape (type_, data)
          Parameters: *type_* - one of the strings "polygon", "image",
          "compound"

     Data structure modeling shapes.  The pair `(type_, data)' must
     follow this specification:

     _type__         _data_
     -------------------------------------------------------------------------------
     "polygon"       a polygon-tuple, i.e. a tuple of pairs of coordinates
     "image"         an image  (in this form only used internally!)
     "compound"      `None' (a compound shape has to be constructed using the
                     addcomponent() method)

      -- Method: addcomponent (poly, fill, outline=None)
               Parameters:

                  * *poly* - a polygon, i.e. a tuple of pairs of numbers

                  * *fill* - a color the _poly_ will be filled with

                  * *outline* - a color for the poly's outline (if
                    given)

          Example:

              >>> poly = ((0,0),(10,-5),(0,10),(-10,-5))
              >>> s = Shape("compound")
              >>> s.addcomponent(poly, "red", "blue")
              >>> # ... add more components and then use register_shape()

          See Excursus about the use of compound shapes.

 -- Class: turtle.Vec2D (x, y)
     A two-dimensional vector class, used as a helper class for
     implementing turtle graphics.  May be useful for turtle graphics
     programs too.  Derived from tuple, so a vector is a tuple!

     Provides (for _a_, _b_ vectors, _k_ number):

        * `a + b' vector addition

        * `a - b' vector subtraction

        * `a * b' inner product

        * `k * a' and `a * k' multiplication with scalar

        * `abs(a)' absolute value of a

        * `a.rotate(angle)' rotation

5.24.5.27 Help and configuration
................................

5.24.5.28 How to use help
.........................

The public methods of the Screen and Turtle classes are documented
extensively via docstrings.  So these can be used as online-help via
the Python help facilities:

   - When using IDLE, tooltips show the signatures and first lines of
     the docstrings of typed in function-/method calls.

   - Calling help() on methods or functions displays the
     docstrings:

         >>> help(Screen.bgcolor)
         Help on method bgcolor in module turtle:

         bgcolor(self, *args) unbound turtle.Screen method
             Set or return backgroundcolor of the TurtleScreen.

             Arguments (if given): a color string or three numbers
             in the range 0..colormode or a 3-tuple of such numbers.

               >>> screen.bgcolor("orange")
               >>> screen.bgcolor()
               "orange"
               >>> screen.bgcolor(0.5,0,0.5)
               >>> screen.bgcolor()
               "#800080"

         >>> help(Turtle.penup)
         Help on method penup in module turtle:

         penup(self) unbound turtle.Turtle method
             Pull the pen up -- no drawing when moving.

             Aliases: penup | pu | up

             No argument

             >>> turtle.penup()

   - The docstrings of the functions which are derived from methods
     have a modified form:

         >>> help(bgcolor)
         Help on function bgcolor in module turtle:

         bgcolor(*args)
             Set or return backgroundcolor of the TurtleScreen.

             Arguments (if given): a color string or three numbers
             in the range 0..colormode or a 3-tuple of such numbers.

             Example::

               >>> bgcolor("orange")
               >>> bgcolor()
               "orange"
               >>> bgcolor(0.5,0,0.5)
               >>> bgcolor()
               "#800080"

         >>> help(penup)
         Help on function penup in module turtle:

         penup()
             Pull the pen up -- no drawing when moving.

             Aliases: penup | pu | up

             No argument

             Example:
             >>> penup()

  These modified docstrings are created automatically together with the
function definitions that are derived from the methods at import time.

5.24.5.29 Translation of docstrings into different languages
............................................................

There is a utility to create a dictionary the keys of which are the
method names and the values of which are the docstrings of the public
methods of the classes Screen and Turtle.

 -- Function: turtle.write_docstringdict
          (filename="turtle_docstringdict")
          Parameters: *filename* - a string, used as filename

     Create and write docstring-dictionary to a Python script with the
     given filename.  This function has to be called explicitly (it is
     not used by the turtle graphics classes).  The docstring
     dictionary will be written to the Python script `_filename_.py'.
     It is intended to serve as a template for translation of the
     docstrings into different languages.

  If you (or your students) want to use turtle with online
help in your native language, you have to translate the docstrings and
save the resulting file as e.g. `turtle_docstringdict_german.py'.

  If you have an appropriate entry in your `turtle.cfg' file this
dictionary will be read in at import time and will replace the original
English docstrings.

  At the time of this writing there are docstring dictionaries in
German and in Italian.  (Requests please to <glingl@aon.at>.)

5.24.5.30 How to configure Screen and Turtles
.............................................

The built-in default configuration mimics the appearance and behaviour
of the old turtle module in order to retain best possible compatibility
with it.

  If you want to use a different configuration which better reflects
the features of this module or which better fits to your needs, e.g.
for use in a classroom, you can prepare a configuration file
`turtle.cfg' which will be read at import time and modify the
configuration according to its settings.

  The built in configuration would correspond to the following
turtle.cfg:

    width = 0.5
    height = 0.75
    leftright = None
    topbottom = None
    canvwidth = 400
    canvheight = 300
    mode = standard
    colormode = 1.0
    delay = 10
    undobuffersize = 1000
    shape = classic
    pencolor = black
    fillcolor = black
    resizemode = noresize
    visible = True
    language = english
    exampleturtle = turtle
    examplescreen = screen
    title = Python Turtle Graphics
    using_IDLE = False

Short explanation of selected entries:

   - The first four lines correspond to the arguments of the
     `Screen.setup()' method.

   - Line 5 and 6 correspond to the arguments of the method
     `Screen.screensize()'.

   - _shape_ can be any of the built-in shapes, e.g: arrow, turtle,
     etc.  For more info try `help(shape)'.

   - If you want to use no fillcolor (i.e. make the turtle
     transparent), you have to write `fillcolor = ""' (but all nonempty
     strings must not have quotes in the cfg-file).

   - If you want to reflect the turtle its state, you have to use
     `resizemode = auto'.

   - If you set e.g. `language = italian' the docstringdict
     `turtle_docstringdict_italian.py' will be loaded at import time (if
     present on the import path, e.g. in the same directory as
     turtle.

   - The entries _exampleturtle_ and _examplescreen_ define the names
     of these objects as they occur in the docstrings.  The
     transformation of method-docstrings to function-docstrings will
     delete these names from the docstrings.

   - _using_IDLE_: Set this to `True' if you regularly work with IDLE
     and its -n switch ("no subprocess").  This will prevent
     exitonclick() to enter the mainloop.

  There can be a `turtle.cfg' file in the directory where turtle
is stored and an additional one in the current working directory.
The latter will override the settings of the first one.

  The `Demo/turtle' directory contains a `turtle.cfg' file.  You can
study it as an example and see its effects when running the demos
(preferably not from within the demo-viewer).

5.24.5.31 Demo scripts
......................

There is a set of demo scripts in the turtledemo directory located in
the `Demo/turtle' directory in the source distribution.

  It contains:

   - a set of 15 demo scripts demonstrating different features of the
     new module turtle.

   - a demo viewer `turtleDemo.py' which can be used to view the
     sourcecode of the scripts and run them at the same time. 14 of the
     examples can be accessed via the Examples menu; all of them can
     also be run standalone.

   - The example `turtledemo_two_canvases.py' demonstrates the
     simultaneous use of two canvases with the turtle module.
     Therefore it only can be run standalone.

   - There is a `turtle.cfg' file in this directory, which also serves
     as an example for how to write and use such files.

  The demoscripts are:

Name                 Description                        Features
bytedesign           complex classical turtlegraphics   tracer(),
                     pattern                            delay, see update().
chaos                graphs verhust dynamics, proves    world coordinates
                     that you must not trust
                     computers' computations
clock                analog clock showing time of your  turtles as clock's hands,
                     computer                           ontimer
colormixer           experiment with r, g, b            ondrag().
fractalcurves        Hilbert & Koch curves              recursion
lindenmayer          ethnomathematics (indian kolams)   L-System
minimal_hanoi        Towers of Hanoi                    Rectangular Turtles as
                                                        Hanoi discs (shape,
                                                        shapesize)
paint                super minimalistic drawing program onclick().
peace                elementary                         turtle: appearance and
                                                        animation
penrose              aperiodic tiling with kites and    stamp().
                     darts
planet_and_moon      simulation of gravitational system compound shapes, see
                                                        Vec2D.
tree                 a (graphical) breadth first tree   clone().
                     (using generators)
wikipedia            a pattern from the wikipedia       clone(),
                     article on turtle graphics undo().
yingyang             another elementary example         circle().

  Have fun!

5.24.6 IDLE
-----------

IDLE is the Python IDE built with the `tkinter' GUI toolkit.

  IDLE has the following features:

   * coded in 100% pure Python, using the `tkinter' GUI toolkit

   * cross-platform: works on Windows and Unix

   * multi-window text editor with multiple undo, Python colorizing and
     many other features, e.g. smart indent and call tips

   * Python shell window (a.k.a. interactive interpreter)

   * debugger (not complete, but you can set breakpoints, view  and
     step)

5.24.6.1 Menus
..............

5.24.6.2 File menu
..................

New window
     create a new editing window

Open...
     open an existing file

Open module...
     open an existing module (searches sys.path)

Class browser
     show classes and methods in current file

Path browser
     show sys.path directories, modules, classes and methods

Save
     save current window to the associated file (unsaved windows have a
     * before and after the window title)

Save As...
     save current window to new file, which becomes the associated file

Save Copy As...
     save current window to different file without changing the
     associated file

Close
     close current window (asks to save if unsaved)

Exit
     close all windows and quit IDLE (asks to save if unsaved)

5.24.6.3 Edit menu
..................

Undo
     Undo last change to current window (max 1000 changes)

Redo
     Redo last undone change to current window

Cut
     Copy selection into system-wide clipboard; then delete selection

Copy
     Copy selection into system-wide clipboard

Paste
     Insert system-wide clipboard into window

Select All
     Select the entire contents of the edit buffer

Find...
     Open a search dialog box with many options

Find again
     Repeat last search

Find selection
     Search for the string in the selection

Find in Files...
     Open a search dialog box for searching files

Replace...
     Open a search-and-replace dialog box

Go to line
     Ask for a line number and show that line

Indent region
     Shift selected lines right 4 spaces

Dedent region
     Shift selected lines left 4 spaces

Comment out region
     Insert ## in front of selected lines

Uncomment region
     Remove leading # or ## from selected lines

Tabify region
     Turns _leading_ stretches of spaces into tabs

Untabify region
     Turn _all_ tabs into the right number of spaces

Expand word
     Expand the word you have typed to match another word in the same
     buffer; repeat to get a different expansion

Format Paragraph
     Reformat the current blank-line-separated paragraph

Import module
     Import or reload the current module

Run script
     Execute the current file in the __main__ namespace

5.24.6.4 Windows menu
.....................

Zoom Height
     toggles the window between normal size (24x80) and maximum height.

  The rest of this menu lists the names of all open windows; select one
to bring it to the foreground (deiconifying it if necessary).

5.24.6.5 Debug menu (in the Python Shell window only)
.....................................................

Go to file/line
     look around the insert point for a filename and linenumber, open
     the file, and show the line.

Open stack viewer
     show the stack traceback of the last exception

Debugger toggle
     Run commands in the shell under the debugger

JIT Stack viewer toggle
     Open stack viewer on traceback

5.24.6.6 Basic editing and navigation
.....................................

   * `Backspace' deletes to the left; `Del' deletes to the right

   * Arrow keys and `Page Up'/`Page Down' to move around

   * `Home'/`End' go to begin/end of line

   * `C-Home'/`C-End' go to begin/end of file

   * Some *Emacs* bindings may also work, including `C-B', `C-P',
     `C-A', `C-E', `C-D', `C-L'

5.24.6.7 Automatic indentation
..............................

After a block-opening statement, the next line is indented by 4 spaces
(in the Python Shell window by one tab).  After certain keywords
(break, return etc.)  the next line is dedented.  In leading
indentation, `Backspace' deletes up to 4 spaces if they are there.
`Tab' inserts 1-4 spaces (in the Python Shell window one tab). See also
the indent/dedent region commands in the edit menu.

5.24.6.8 Python Shell window
............................

   * `C-C' interrupts executing command

   * `C-D' sends end-of-file; closes window if typed at a `>>>' prompt

   * `Alt-p' retrieves previous command matching what you have typed

   * `Alt-n' retrieves next

   * `Return' while on any previous command retrieves that command

   * `Alt-/' (Expand word) is also useful here

5.24.6.9 Syntax colors
......................

The coloring is applied in a background "thread," so you may
occasionally see uncolorized text.  To change the color scheme, edit
the `[Colors]' section in `config.txt'.

Python syntax colors:

    Keywords
          orange

    Strings
          green

    Comments
          red

    Definitions
          blue

Shell colors:

    Console output
          brown

    stdout
          blue

    stderr
          dark green

    stdin
          black

5.24.6.10 Startup
.................

Upon startup with the `-s' option, IDLE will execute the file
referenced by the environment variables `IDLESTARTUP' or
PYTHONSTARTUP.  Idle first checks for `IDLESTARTUP'; if
`IDLESTARTUP' is present the file referenced is run.  If `IDLESTARTUP'
is not present, Idle checks for `PYTHONSTARTUP'.  Files referenced by
these environment variables are convenient places to store functions
that are used frequently from the Idle shell, or for executing import
statements to import common modules.

  In addition, `Tk' also loads a startup file if it is present.  Note
that the Tk file is loaded unconditionally.  This additional file is
`.Idle.py' and is looked for in the user's home directory.  Statements
in this file will be executed in the Tk namespace, so this file is not
useful for importing functions to be used from Idle's Python shell.

5.24.6.11 Command line usage
............................

    idle.py [-c command] [-d] [-e] [-s] [-t title] [arg] ...

    -c command  run this command
    -d          enable debugger
    -e          edit mode; arguments are files to be edited
    -s          run $IDLESTARTUP or $PYTHONSTARTUP first
    -t title    set title of shell window

If there are arguments:

  1. If `-e' is used, arguments are files opened for editing and
     `sys.argv' reflects the arguments passed to IDLE itself.

  2. Otherwise, if -c is used, all arguments are placed in
     `sys.argv[1:...]', with `sys.argv[0]' set to `'-c''.

  3. Otherwise, if neither `-e' nor -c is used, the first
     argument is a script which is executed with the remaining
     arguments in `sys.argv[1:...]'  and `sys.argv[0]' set to the
     script name.  If the script name is '-', no script is executed but
     an interactive Python session is started; the arguments are still
     available in `sys.argv'.

5.24.7 Other Graphical User Interface Packages
----------------------------------------------

There are an number of extension widget sets to Tkinter.

Python megawidgets (http://pmw.sourceforge.net/)
     is a toolkit for building high-level compound widgets in Python
     using the Tkinter module.  It consists of a set of
     base classes and a library of flexible and extensible megawidgets
     built on this foundation. These megawidgets include notebooks,
     comboboxes, selection widgets, paned widgets, scrolled widgets,
     dialog windows, etc.  Also, with the Pmw.Blt interface to BLT, the
     busy, graph, stripchart, tabset and vector commands are be
     available.

     The initial ideas for Pmw were taken from the Tk `itcl' extensions
     `[incr Tk]' by Michael McLennan and `[incr Widgets]' by Mark
     Ulferts. Several of the megawidgets are direct translations from
     the itcl to Python. It offers most of the range of widgets that
     `[incr Widgets]' does, and is almost as complete as Tix, lacking
     however Tix's fast `HList' widget for drawing trees.

Tkinter3000 Widget Construction Kit (WCK) (http://tkinter.effbot.org/)
     is a library that allows you to write new Tkinter widgets in pure
     Python.  The WCK framework gives you full control over widget
     creation, configuration, screen appearance, and event handling.
     WCK widgets can be very fast and light-weight, since they can
     operate directly on Python data structures, without having to
     transfer data through the Tk/Tcl layer.

  The major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits
that are also available for Python:

PyGTK (http://www.pygtk.org/)
     is a set of bindings for the GTK (http://www.gtk.org/)widget set.
     It provides an object oriented interface that is slightly higher
     level than the C one. It comes with many more widgets than Tkinter
     provides, and has good Python-specific reference documentation.
     There are also bindings to GNOME (http://www.gnome.org).  One well
     known PyGTK application is PythonCAD (http://www.pythoncad.org/).
     An online tutorial
     (http://www.pygtk.org/pygtk2tutorial/index.html) is available.

PyQt (http://www.riverbankcomputing.co.uk/software/pyqt/)
     PyQt is a *sip*-wrapped binding to the Qt toolkit.  Qt is an
     extensive C++ GUI application development framework that is
     available for Unix, Windows and Mac OS X. *sip* is a tool for
     generating bindings for C++ libraries as Python classes, and is
     specifically designed for Python. The _PyQt3_ bindings have a
     book, GUI Programming with Python: QT Edition
     (http://www.commandprompt.com/community/pyqt/)by Boudewijn Rempt.
     The _PyQt4_ bindings also have a book, Rapid GUI Programming with
     Python and Qt (http://www.qtrac.eu/pyqtbook.html), by Mark
     Summerfield.

wxPython (http://www.wxpython.org)
     wxPython is a cross-platform GUI toolkit for Python that is built
     around the popular wxWidgets (http://www.wxwidgets.org/)(formerly
     wxWindows) C++ toolkit.  It provides a native look and feel for
     applications on Windows, Mac OS X, and Unix systems by using each
     platform's native widgets where ever possible, (GTK+ on Unix-like
     systems).  In addition to an extensive set of widgets, wxPython
     provides classes for online documentation and context sensitive
     help, printing, HTML viewing, low-level device context drawing,
     drag and drop, system clipboard access, an XML-based resource
     format and more, including an ever growing library of
     user-contributed modules.  wxPython has a book, wxPython in Action
     (http://www.amazon.com/exec/obidos/ASIN/1932394621), by Noel
     Rappin and Robin Dunn.

  PyGTK, PyQt, and wxPython, all have a modern look and feel and more
widgets than Tkinter. In addition, there are many other GUI toolkits for
Python, both cross-platform, and platform-specific. See the GUI
Programming (http://wiki.python.org/moin/GuiProgramming)page in the
Python Wiki for a much more complete list, and also for links to
documents where the different GUI toolkits are compared.

5.25 Development Tools
======================

The modules described in this chapter help you write software.  For
example, the pydoc module takes a module and generates
documentation based on the module's contents.  The doctest.
and unittest modules contains frameworks for writing unit
tests that automatically exercise code and verify that the expected
output is produced.  *2to3* can translate Python 2.x source code into
valid Python 3.x code.

  The list of modules described in this chapter is:

5.25.1 `pydoc' -- Documentation generator and online help system
----------------------------------------------------------------

New in version 2.1.

  The pydoc module automatically generates documentation
from Python modules.  The documentation can be presented as pages of
text on the console, served to a Web browser, or saved to HTML files.

  The built-in function help() invokes the online help
system in the interactive interpreter, which uses pydoc to
generate its documentation as text on the console.  The same text
documentation can also be viewed from outside the Python interpreter by
running *pydoc* as a script at the operating system's command prompt.
For example, running

    pydoc sys

at a shell prompt will display documentation on the sys.
module, in a style similar to the manual pages shown by the Unix *man*
command.  The argument to *pydoc* can be the name of a function,
module, or package, or a dotted reference to a class, method, or
function within a module or module in a package.  If the argument to
*pydoc* looks like a path (that is, it contains the path separator for
your operating system, such as a slash in Unix), and refers to an
existing Python source file, then documentation is produced for that
file.

     Note: In order to find objects and their documentation, see
     pydoc imports the module(s) to be documented.  Therefore,
     any code on module level will be executed on that occasion.  Use
     an `if __name__ == '__main__':' guard to only execute code when a
     file is invoked as a script and not just imported.

  Specifying a `-w' flag before the argument will cause HTML
documentation to be written out to a file in the current directory,
instead of displaying text on the console.

  Specifying a `-k' flag before the argument will search the synopsis
lines of all available modules for the keyword given as the argument,
again in a manner similar to the Unix *man* command.  The synopsis line
of a module is the first line of its documentation string.

  You can also use *pydoc* to start an HTTP server on the local machine
that will serve documentation to visiting Web browsers. *pydoc* `-p
1234' will start a HTTP server on port 1234, allowing you to browse the
documentation at `http://localhost:1234/' in your preferred Web browser.
*pydoc* `-g' will start the server and additionally bring up a small
Tkinter.-based graphical interface to help you search for
documentation pages.

  When *pydoc* generates documentation, it uses the current environment
and path to locate modules.  Thus, invoking *pydoc* `spam' documents
precisely the version of the module you would get if you started the
Python interpreter and typed `import spam'.

  Module docs for core modules are assumed to reside in
<http://docs.python.org/library/>.  This can be overridden by setting
the `PYTHONDOCS' environment variable to a different URL or to a local
directory containing the Library Reference Manual pages.

5.25.2 `doctest' -- Test interactive Python examples
----------------------------------------------------

The doctest module searches for pieces of text that look
like interactive Python sessions, and then executes those sessions to
verify that they work exactly as shown.  There are several common ways
to use doctest:

   * To check that a module's docstrings are up-to-date by verifying
     that all interactive examples still work as documented.

   * To perform regression testing by verifying that interactive
     examples from a test file or a test object work as expected.

   * To write tutorial documentation for a package, liberally
     illustrated with input-output examples.  Depending on whether the
     examples or the expository text are emphasized, this has the
     flavor of "literate testing" or "executable documentation".

  Here's a complete but small example module:

    """
    This is the "example" module.

    The example module supplies one function, factorial().  For example,

    >>> factorial(5)
    120
    """

    def factorial(n):
        """Return the factorial of n, an exact integer >= 0.

        If the result is small enough to fit in an int, return an int.
        Else return a long.

        >>> [factorial(n) for n in range(6)]
        [1, 1, 2, 6, 24, 120]
        >>> [factorial(long(n)) for n in range(6)]
        [1, 1, 2, 6, 24, 120]
        >>> factorial(30)
        265252859812191058636308480000000L
        >>> factorial(30L)
        265252859812191058636308480000000L
        >>> factorial(-1)
        Traceback (most recent call last):
            ...
        ValueError: n must be >= 0

        Factorials of floats are OK, but the float must be an exact integer:
        >>> factorial(30.1)
        Traceback (most recent call last):
            ...
        ValueError: n must be exact integer
        >>> factorial(30.0)
        265252859812191058636308480000000L

        It must also not be ridiculously large:
        >>> factorial(1e100)
        Traceback (most recent call last):
            ...
        OverflowError: n too large
        """

        import math
        if not n >= 0:
            raise ValueError("n must be >= 0")
        if math.floor(n) != n:
            raise ValueError("n must be exact integer")
        if n+1 == n:  # catch a value like 1e300
            raise OverflowError("n too large")
        result = 1
        factor = 2
        while factor <= n:
            result *= factor
            factor += 1
        return result

    if __name__ == "__main__":
        import doctest
        doctest.testmod()

If you run `example.py' directly from the command line, see doctest.
works its magic:

    $ python example.py
    $

There's no output!  That's normal, and it means all the examples
worked.  Pass -v to the script, and doctest.
prints a detailed log of what it's trying, and prints a summary at the
end:

    $ python example.py -v
    Trying:
        factorial(5)
    Expecting:
        120
    ok
    Trying:
        [factorial(n) for n in range(6)]
    Expecting:
        [1, 1, 2, 6, 24, 120]
    ok
    Trying:
        [factorial(long(n)) for n in range(6)]
    Expecting:
        [1, 1, 2, 6, 24, 120]
    ok

And so on, eventually ending with:

    Trying:
        factorial(1e100)
    Expecting:
        Traceback (most recent call last):
            ...
        OverflowError: n too large
    ok
    2 items passed all tests:
       1 tests in __main__
       8 tests in __main__.factorial
    9 tests in 2 items.
    9 passed and 0 failed.
    Test passed.
    $

That's all you need to know to start making productive use of
doctest.!  Jump in.  The following sections provide full details.
Note that there are many examples of doctests in the standard Python
test suite and libraries.  Especially useful examples can be found in
the standard test file `Lib/test/test_doctest.py'.

5.25.2.1 Simple Usage: Checking Examples in Docstrings
......................................................

The simplest way to start using doctest (but not necessarily the way
you'll continue to do it) is to end each module `M' with:

    if __name__ == "__main__":
        import doctest
        doctest.testmod()

doctest then examines docstrings in module `M'.

  Running the module as a script causes the examples in the docstrings
to get executed and verified:

    python M.py

This won't display anything unless an example fails, in which case the
failing example(s) and the cause(s) of the failure(s) are printed to
stdout, and the final line of output is `***Test Failed*** N
failures.', where _N_ is the number of examples that failed.

  Run it with the -v switch instead:

    python M.py -v

and a detailed report of all examples tried is printed to standard
output, along with assorted summaries at the end.

  You can force verbose mode by passing `verbose=True' to
testmod(), or prohibit it by passing `verbose=False'.  In either
of those cases, `sys.argv' is not examined by testmod(). (so
passing -v or not has no effect).

  Since Python 2.6, there is also a command line shortcut for running
testmod().  You can instruct the Python interpreter to run
the doctest module directly from the standard library and pass the
module name(s) on the command line:

    python -m doctest -v example.py

This will import `example.py' as a standalone module and run
testmod() on it.  Note that this may not work correctly if the
file is part of a package and imports other submodules from that
package.

  For more information on testmod(), see section Basic API.

5.25.2.2 Simple Usage: Checking Examples in a Text File
.......................................................

Another simple application of doctest is testing interactive examples
in a text file.  This can be done with the testfile().
function:

    import doctest
    doctest.testfile("example.txt")

That short script executes and verifies any interactive Python examples
contained in the file `example.txt'.  The file content is treated as if
it were a single giant docstring; the file doesn't need to contain a
Python program!   For example, perhaps `example.txt' contains this:

    The ``example`` module
    ======================

    Using ``factorial``
    -------------------

    This is an example text file in reStructuredText format.  First import
    ``factorial`` from the ``example`` module:

        >>> from example import factorial

    Now use it:

        >>> factorial(6)
        120

Running `doctest.testfile("example.txt")' then finds the error in this
documentation:

    File "./example.txt", line 14, in example.txt
    Failed example:
        factorial(6)
    Expected:
        120
    Got:
        720

As with testmod(), testfile() won't display
anything unless an example fails.  If an example does fail, then the
failing example(s) and the cause(s) of the failure(s) are printed to
stdout, using the same format as testmod().

  By default, see testfile() looks for files in the calling
module's directory.  See section Basic API for a
description of the optional arguments that can be used to tell it to
look for files in other locations.

  Like testmod(), testfile().'s verbosity can be
set with the -v command-line switch or with the optional
keyword argument _verbose_.

  Since Python 2.6, there is also a command line shortcut for running
testfile().  You can instruct the Python interpreter to run
the doctest module directly from the standard library and pass the file
name(s) on the command line:

    python -m doctest -v example.txt

Because the file name does not end with `.py', see doctest.
infers that it must be run with testfile(), not
testmod().

  For more information on testfile(), see section
Basic API.

5.25.2.3 How It Works
.....................

This section examines in detail how doctest works: which docstrings it
looks at, how it finds interactive examples, what execution context it
uses, how it handles exceptions, and how option flags can be used to
control its behavior.  This is the information that you need to know to
write doctest examples; for information about actually running doctest
on these examples, see the following sections.

5.25.2.4 Which Docstrings Are Examined?
.......................................

The module docstring, and all function, class and method docstrings are
searched.  Objects imported into the module are not searched.

  In addition, if `M.__test__' exists and "is true", it must be a dict,
and each entry maps a (string) name to a function object, class object,
or string.  Function and class object docstrings found from
`M.__test__' are searched, and strings are treated as if they were
docstrings.  In output, a key `K' in `M.__test__' appears with name

    <name of M>.__test__.K

Any classes found are recursively searched similarly, to test
docstrings in their contained methods and nested classes.

  Changed in version 2.4: A "private name" concept is deprecated and no
longer documented.

5.25.2.5 How are Docstring Examples Recognized?
...............................................

In most cases a copy-and-paste of an interactive console session works
fine, but doctest isn't trying to do an exact emulation of any specific
Python shell.

    >>> # comments are ignored
    >>> x = 12
    >>> x
    12
    >>> if x == 13:
    ...     print "yes"
    ... else:
    ...     print "no"
    ...     print "NO"
    ...     print "NO!!!"
    ...
    no
    NO
    NO!!!
    >>>

Any expected output must immediately follow the final `'>>> '' or `'...
'' line containing the code, and the expected output (if any) extends
to the next `'>>> '' or all-whitespace line.

  The fine print:

   * Expected output cannot contain an all-whitespace line, since such
     a line is taken to signal the end of expected output.  If expected
     output does contain a blank line, put `<BLANKLINE>' in your
     doctest example each place a blank line is expected.

     New in version 2.4: `<BLANKLINE>' was added; there was no way to
     use expected output containing empty lines in previous versions.

   * All hard tab characters are expanded to spaces, using 8-column tab
     stops.  Tabs in output generated by the tested code are not
     modified.  Because any hard tabs in the sample output _are_
     expanded, this means that if the code output includes hard tabs,
     the only way the doctest can pass is if the
     NORMALIZE_WHITESPACE option or directive is in effect.
     Alternatively, the test can be rewritten to capture the output and
     compare it to an expected value as part of the test.  This
     handling of tabs in the source was arrived at through trial and
     error, and has proven to be the least error prone way of handling
     them.  It is possible to use a different algorithm for handling
     tabs by writing a custom DocTestParser class.

     Changed in version 2.4: Expanding tabs to spaces is new; previous
     versions tried to preserve hard tabs, with confusing results.

   * Output to stdout is captured, but not output to stderr (exception
     tracebacks are captured via a different means).

   * If you continue a line via backslashing in an interactive session,
     or for any other reason use a backslash, you should use a raw
     docstring, which will preserve your backslashes exactly as you
     type them:

         >>> def f(x):
         ...     r'''Backslashes in a raw docstring: m\n'''
         >>> print f.__doc__
         Backslashes in a raw docstring: m\n

     Otherwise, the backslash will be interpreted as part of the
     string. For example, the "\" above would be interpreted as a
     newline character.  Alternatively, you can double each backslash
     in the doctest version (and not use a raw string):

         >>> def f(x):
         ...     '''Backslashes in a raw docstring: m\\n'''
         >>> print f.__doc__
         Backslashes in a raw docstring: m\n

   * The starting column doesn't matter:

         >>> assert "Easy!"
               >>> import math
                   >>> math.floor(1.9)
                   1.0

     and as many leading whitespace characters are stripped from the
     expected output as appeared in the initial `'>>> '' line that
     started the example.

5.25.2.6 What's the Execution Context?
......................................

By default, each time doctest finds a docstring to test, it
uses a _shallow copy_ of `M''s globals, so that running tests doesn't
change the module's real globals, and so that one test in `M' can't
leave behind crumbs that accidentally allow another test to work.  This
means examples can freely use any names defined at top-level in `M',
and names defined earlier in the docstring being run. Examples cannot
see names defined in other docstrings.

  You can force use of your own dict as the execution context by passing
`globs=your_dict' to testmod() or testfile().
instead.

5.25.2.7 What About Exceptions?
...............................

No problem, provided that the traceback is the only output produced by
the example:  just paste in the traceback. (1) Since tracebacks contain
details that are likely to change rapidly (for example, exact file
paths and line numbers), this is one case where doctest works hard to
be flexible in what it accepts.

  Simple example:

    >>> [1, 2, 3].remove(42)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: list.remove(x): x not in list

That doctest succeeds if ValueError is raised, with the
`list.remove(x): x not in list' detail as shown.

  The expected output for an exception must start with a traceback
header, which may be either of the following two lines, indented the
same as the first line of the example:

    Traceback (most recent call last):
    Traceback (innermost last):

The traceback header is followed by an optional traceback stack, whose
contents are ignored by doctest.  The traceback stack is typically
omitted, or copied verbatim from an interactive session.

  The traceback stack is followed by the most interesting part: the
line(s) containing the exception type and detail.  This is usually the
last line of a traceback, but can extend across multiple lines if the
exception has a multi-line detail:

    >>> raise ValueError('multi\n    line\ndetail')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: multi
        line
    detail

The last three lines (starting with ValueError) are
compared against the exception's type and detail, and the rest are
ignored.

  Changed in version 2.4: Previous versions were unable to handle
multi-line exception details.

  Best practice is to omit the traceback stack, unless it adds
significant documentation value to the example.  So the last example is
probably better as:

    >>> raise ValueError('multi\n    line\ndetail')
    Traceback (most recent call last):
        ...
    ValueError: multi
        line
    detail

Note that tracebacks are treated very specially.  In particular, in the
rewritten example, the use of `...' is independent of doctest's
ELLIPSIS option.  The ellipsis in that example could be left out,
or could just as well be three (or three hundred) commas or digits, or
an indented transcript of a Monty Python skit.

  Some details you should read once, but won't need to remember:

   * Doctest can't guess whether your expected output came from an
     exception traceback or from ordinary printing.  So, e.g., an
     example that expects `ValueError: 42 is prime' will pass whether
     ValueError is actually raised or if the example merely
     prints that traceback text.  In practice, ordinary output rarely
     begins with a traceback header line, so this doesn't create real
     problems.

   * Each line of the traceback stack (if present) must be indented
     further than the first line of the example, _or_ start with a
     non-alphanumeric character.  The first line following the
     traceback header indented the same and starting with an
     alphanumeric is taken to be the start of the exception detail.  Of
     course this does the right thing for genuine tracebacks.

   * When the IGNORE_EXCEPTION_DETAIL doctest option is
     specified, everything following the leftmost colon and any module
     information in the exception name is ignored.

   * The interactive shell omits the traceback header line for some
     SyntaxError.s.  But doctest uses the traceback header
     line to distinguish exceptions from non-exceptions.  So in the
     rare case where you need to test a SyntaxError that
     omits the traceback header, you will need to manually add the
     traceback header line to your test example.

   * For some SyntaxError.s, Python displays the character
     position of the syntax error, using a `^' marker:

         >>> 1 1
           File "<stdin>", line 1
             1 1
               ^
         SyntaxError: invalid syntax

     Since the lines showing the position of the error come before the
     exception type and detail, they are not checked by doctest.  For
     example, the following test would pass, even though it puts the
     `^' marker in the wrong location:

         >>> 1 1
         Traceback (most recent call last):
           File "<stdin>", line 1
             1 1
             ^
         SyntaxError: invalid syntax

5.25.2.8 Option Flags and Directives
....................................

A number of option flags control various aspects of doctest's behavior.
Symbolic names for the flags are supplied as module constants, which
can be or'ed together and passed to various functions.  The names can
also be used in doctest directives (see below).

  The first group of options define test semantics, controlling aspects
of how doctest decides whether actual output matches an example's
expected output:

 -- Data: doctest.DONT_ACCEPT_TRUE_FOR_1
     By default, if an expected output block contains just `1', an
     actual output block containing just `1' or just `True' is
     considered to be a match, and similarly for `0' versus `False'.
     When DONT_ACCEPT_TRUE_FOR_1 is specified, neither
     substitution is allowed.  The default behavior caters to that
     Python changed the return type of many functions from integer to
     boolean; doctests expecting "little integer" output still work in
     these cases.  This option will probably go away, but not for
     several years.

 -- Data: doctest.DONT_ACCEPT_BLANKLINE
     By default, if an expected output block contains a line containing
     only the string `<BLANKLINE>', then that line will match a blank
     line in the actual output.  Because a genuinely blank line
     delimits the expected output, this is the only way to communicate
     that a blank line is expected.  When DONT_ACCEPT_BLANKLINE
     is specified, this substitution is not allowed.

 -- Data: doctest.NORMALIZE_WHITESPACE
     When specified, all sequences of whitespace (blanks and newlines)
     are treated as equal.  Any sequence of whitespace within the
     expected output will match any sequence of whitespace within the
     actual output. By default, whitespace must match exactly.
     NORMALIZE_WHITESPACE is especially useful when a line of
     expected output is very long, and you want to wrap it across
     multiple lines in your source.

 -- Data: doctest.ELLIPSIS
     When specified, an ellipsis marker (`...') in the expected output
     can match any substring in the actual output.  This includes
     substrings that span line boundaries, and empty substrings, so
     it's best to keep usage of this simple.  Complicated uses can lead
     to the same kinds of "oops, it matched too much!"  surprises that
     `.*' is prone to in regular expressions.

 -- Data: doctest.IGNORE_EXCEPTION_DETAIL
     When specified, an example that expects an exception passes if an
     exception of the expected type is raised, even if the exception
     detail does not match.  For example, an example expecting
     `ValueError: 42' will pass if the actual exception raised is
     `ValueError: 3*14', but will fail, e.g., if TypeError.
     is raised.

     It will also ignore the module name used in Python 3 doctest
     reports. Hence both these variations will work regardless of
     whether the test is run under Python 2.7 or Python 3.2 (or later
     versions):

         >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL
         Traceback (most recent call last):
         CustomError: message

         >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL
         Traceback (most recent call last):
         my_module.CustomError: message

     Note that ELLIPSIS can also be used to ignore the
     details of the exception message, but such a test may still fail
     based on whether or not the module details are printed as part of
     the exception name. Using IGNORE_EXCEPTION_DETAIL and
     the details from Python 2.3 is also the only clear way to write a
     doctest that doesn't care about the exception detail yet continues
     to pass under Python 2.3 or earlier (those releases do not support
     doctest directives and ignore them as irrelevant comments). For
     example,

         >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL
         Traceback (most recent call last):
           File "<stdin>", line 1, in ?
         TypeError: object doesn't support item assignment

     passes under Python 2.3 and later Python versions, even though the
     detail changed in Python 2.4 to say "does not" instead of
     "doesn't".

     Changed in version 2.7: IGNORE_EXCEPTION_DETAIL now
     also ignores any information relating to the module containing the
     exception under test

 -- Data: doctest.SKIP
     When specified, do not run the example at all.  This can be useful
     in contexts where doctest examples serve as both documentation and
     test cases, and an example should be included for documentation
     purposes, but should not be checked.  E.g., the example's output
     might be random; or the example might depend on resources which
     would be unavailable to the test driver.

     The SKIP flag can also be used for temporarily "commenting out"
     examples.

  New in version 2.5.

 -- Data: doctest.COMPARISON_FLAGS
     A bitmask or'ing together all the comparison flags above.

  The second group of options controls how test failures are reported:

 -- Data: doctest.REPORT_UDIFF
     When specified, failures that involve multi-line expected and
     actual outputs are displayed using a unified diff.

 -- Data: doctest.REPORT_CDIFF
     When specified, failures that involve multi-line expected and
     actual outputs will be displayed using a context diff.

 -- Data: doctest.REPORT_NDIFF
     When specified, differences are computed by `difflib.Differ',
     using the same algorithm as the popular `ndiff.py' utility. This
     is the only method that marks differences within lines as well as
     across lines.  For example, if a line of expected output contains
     digit `1' where actual output contains letter `l', a line is
     inserted with a caret marking the mismatching column positions.

 -- Data: doctest.REPORT_ONLY_FIRST_FAILURE
     When specified, display the first failing example in each doctest,
     but suppress output for all remaining examples.  This will prevent
     doctest from reporting correct examples that break because of
     earlier failures; but it might also hide incorrect examples that
     fail independently of the first failure.  When
     REPORT_ONLY_FIRST_FAILURE is specified, the remaining
     examples are still run, and still count towards the total number
     of failures reported; only the output is suppressed.

 -- Data: doctest.REPORTING_FLAGS
     A bitmask or'ing together all the reporting flags above.

  "Doctest directives" may be used to modify the option flags for
individual examples.  Doctest directives are expressed as a special
Python comment following an example's source code:

    directive             ::= "#" "doctest:" directive_options
    directive_options     ::= directive_option ("," directive_option)\*
    directive_option      ::= on_or_off directive_option_name
    on_or_off             ::= "+" \| "-"
    directive_option_name ::= "DONT_ACCEPT_BLANKLINE" \| "NORMALIZE_WHITESPACE" \| ...

Whitespace is not allowed between the `+' or `-' and the directive
option name.  The directive option name can be any of the option flag
names explained above.

  An example's doctest directives modify doctest's behavior for that
single example.  Use `+' to enable the named behavior, or `-' to
disable it.

  For example, this test passes:

    >>> print range(20) #doctest: +NORMALIZE_WHITESPACE
    [0,   1,  2,  3,  4,  5,  6,  7,  8,  9,
    10,  11, 12, 13, 14, 15, 16, 17, 18, 19]

Without the directive it would fail, both because the actual output
doesn't have two blanks before the single-digit list elements, and
because the actual output is on a single line.  This test also passes,
and also requires a directive to do so:

    >>> print range(20) # doctest:+ELLIPSIS
    [0, 1, ..., 18, 19]

Multiple directives can be used on a single physical line, separated by
commas:

    >>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
    [0,    1, ...,   18,    19]

If multiple directive comments are used for a single example, then they
are combined:

    >>> print range(20) # doctest: +ELLIPSIS
    ...                 # doctest: +NORMALIZE_WHITESPACE
    [0,    1, ...,   18,    19]

As the previous example shows, you can add `...' lines to your example
containing only directives.  This can be useful when an example is too
long for a directive to comfortably fit on the same line:

    >>> print range(5) + range(10,20) + range(30,40) + range(50,60)
    ... # doctest: +ELLIPSIS
    [0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59]

Note that since all options are disabled by default, and directives
apply only to the example they appear in, enabling options (via `+' in
a directive) is usually the only meaningful choice.  However, option
flags can also be passed to functions that run doctests, establishing
different defaults.  In such cases, disabling an option via `-' in a
directive can be useful.

  New in version 2.4: Doctest directives and the associated constants
DONT_ACCEPT_BLANKLINE, NORMALIZE_WHITESPACE,
ELLIPSIS, IGNORE_EXCEPTION_DETAIL,
REPORT_UDIFF, REPORT_CDIFF, REPORT_NDIFF,
REPORT_ONLY_FIRST_FAILURE, COMPARISON_FLAGS and
REPORTING_FLAGS were added.

  There's also a way to register new option flag names, although this
isn't useful unless you intend to extend doctest internals
via subclassing:

 -- Function: doctest.register_optionflag (name)
     Create a new option flag with a given name, and return the new
     flag's integer value.  register_optionflag() can be
     used when subclassing OutputChecker or
     DocTestRunner to create new options that are supported by
     your subclasses.  register_optionflag() should always
     be called using the following idiom:

         MY_FLAG = register_optionflag('MY_FLAG')

     New in version 2.4.

5.25.2.9 Warnings
.................

doctest is serious about requiring exact matches in expected
output.  If even a single character doesn't match, the test fails.
This will probably surprise you a few times, as you learn exactly what
Python does and doesn't guarantee about output.  For example, when
printing a dict, Python doesn't guarantee that the key-value pairs will
be printed in any particular order, so a test like

    >>> foo()
    {"Hermione": "hippogryph", "Harry": "broomstick"}

is vulnerable!  One workaround is to do

    >>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"}
    True

instead.  Another is to do

    >>> d = foo().items()
    >>> d.sort()
    >>> d
    [('Harry', 'broomstick'), ('Hermione', 'hippogryph')]

There are others, but you get the idea.

  Another bad idea is to print things that embed an object address, like

    >>> id(1.0) # certain to fail some of the time
    7948648
    >>> class C: pass
    >>> C()   # the default repr() for instances embeds an address
    <__main__.C instance at 0x00AC18F0>

The ELLIPSIS directive gives a nice approach for the last
example:

    >>> C() #doctest: +ELLIPSIS
    <__main__.C instance at 0x...>

Floating-point numbers are also subject to small output variations
across platforms, because Python defers to the platform C library for
float formatting, and C libraries vary widely in quality here.

    >>> 1./7  # risky
    0.14285714285714285
    >>> print 1./7 # safer
    0.142857142857
    >>> print round(1./7, 6) # much safer
    0.142857

Numbers of the form `I/2.**J' are safe across all platforms, and I often
contrive doctest examples to produce numbers of that form:

    >>> 3./4  # utterly safe
    0.75

Simple fractions are also easier for people to understand, and that
makes for better documentation.

5.25.2.10 Basic API
...................

The functions testmod() and testfile() provide
a simple interface to doctest that should be sufficient for most basic
uses.  For a less formal introduction to these two functions, see
sections Simple Usage; Checking Examples in Docstrings.
and Simple Usage; Checking Examples in a Text File.

 -- Function: doctest.testfile (filename[, module_relative][, name][,
          package][, globs][, verbose][, report][, optionflags][,
          extraglobs][, raise_on_error][, parser][, encoding])
     All arguments except _filename_ are optional, and should be
     specified in keyword form.

     Test examples in the file named _filename_.  Return
     `(failure_count, test_count)'.

     Optional argument _module_relative_ specifies how the filename
     should be interpreted:

        * If _module_relative_ is `True' (the default), then _filename_
          specifies an OS-independent module-relative path.  By
          default, this path is relative to the calling module's
          directory; but if the _package_ argument is specified, then it
          is relative to that package.  To ensure OS-independence,
          _filename_ should use `/' characters to separate path
          segments, and may not be an absolute path (i.e., it may not
          begin with `/').

        * If _module_relative_ is `False', then _filename_ specifies an
          OS-specific path.  The path may be absolute or relative;
          relative paths are resolved with respect to the current
          working directory.

     Optional argument _name_ gives the name of the test; by default,
     or if `None', `os.path.basename(filename)' is used.

     Optional argument _package_ is a Python package or the name of a
     Python package whose directory should be used as the base
     directory for a module-relative filename.  If no package is
     specified, then the calling module's directory is used as the base
     directory for module-relative filenames.  It is an error to
     specify _package_ if _module_relative_ is `False'.

     Optional argument _globs_ gives a dict to be used as the globals
     when executing examples.  A new shallow copy of this dict is
     created for the doctest, so its examples start with a clean slate.
     By default, or if `None', a new empty dict is used.

     Optional argument _extraglobs_ gives a dict merged into the
     globals used to execute examples.  This works like
     dict.update().:  if _globs_ and _extraglobs_ have a common
     key, the associated value in _extraglobs_ appears in the combined
     dict.  By default, or if `None', no extra globals are used.  This
     is an advanced feature that allows parameterization of doctests.
     For example, a doctest can be written for a base class, using a
     generic name for the class, then reused to test any number of
     subclasses by passing an _extraglobs_ dict mapping the generic
     name to the subclass to be tested.

     Optional argument _verbose_ prints lots of stuff if true, and
     prints only failures if false; by default, or if `None', it's true
     if and only if `'-v'' is in `sys.argv'.

     Optional argument _report_ prints a summary at the end when true,
     else prints nothing at the end.  In verbose mode, the summary is
     detailed, else the summary is very brief (in fact, empty if all
     tests passed).

     Optional argument _optionflags_ or's together option flags.  See
     section Option Flags and Directives.

     Optional argument _raise_on_error_ defaults to false.  If true, an
     exception is raised upon the first failure or unexpected exception
     in an example.  This allows failures to be post-mortem debugged.
     Default behavior is to continue running examples.

     Optional argument _parser_ specifies a DocTestParser.
     (or subclass) that should be used to extract tests from the files.
     It defaults to a normal parser (i.e., `DocTestParser()').

     Optional argument _encoding_ specifies an encoding that should be
     used to convert the file to unicode.

     New in version 2.4.

     Changed in version 2.5: The parameter _encoding_ was added.

 -- Function: doctest.testmod ([m][, name][, globs][, verbose][,
          report][, optionflags][, extraglobs][, raise_on_error][,
          exclude_empty])
     All arguments are optional, and all except for _m_ should be
     specified in keyword form.

     Test examples in docstrings in functions and classes reachable
     from module _m_ (or module __main__ if _m_ is not
     supplied or is `None'), starting with `m.__doc__'.

     Also test examples reachable from dict `m.__test__', if it exists
     and is not `None'.  `m.__test__' maps names (strings) to
     functions, classes and strings; function and class docstrings are
     searched for examples; strings are searched directly, as if they
     were docstrings.

     Only docstrings attached to objects belonging to module _m_ are
     searched.

     Return `(failure_count, test_count)'.

     Optional argument _name_ gives the name of the module; by default,
     or if `None', `m.__name__' is used.

     Optional argument _exclude_empty_ defaults to false.  If true,
     objects for which no doctests are found are excluded from
     consideration. The default is a backward compatibility hack, so
     that code still using `doctest.master.summarize()' in conjunction
     with testmod() continues to get output for objects
     with no tests. The _exclude_empty_ argument to the newer
     DocTestFinder.  constructor defaults to true.

     Optional arguments _extraglobs_, _verbose_, _report_,
     _optionflags_, _raise_on_error_, and _globs_ are the same as for
     function testfile().  above, except that _globs_
     defaults to `m.__dict__'.

     Changed in version 2.3: The parameter _optionflags_ was added.

     Changed in version 2.4: The parameters _extraglobs_,
     _raise_on_error_ and _exclude_empty_ were added.

     Changed in version 2.5: The optional argument _isprivate_,
     deprecated in 2.4, was removed.

  There's also a function to run the doctests associated with a single
object.  This function is provided for backward compatibility.  There
are no plans to deprecate it, but it's rarely useful:

 -- Function: doctest.run_docstring_examples (f, globs[, verbose][,
          name][, compileflags][, optionflags])
     Test examples associated with object _f_; for example, _f_ may be
     a module, function, or class object.

     A shallow copy of dictionary argument _globs_ is used for the
     execution context.

     Optional argument _name_ is used in failure messages, and defaults
     to `"NoName"'.

     If optional argument _verbose_ is true, output is generated even
     if there are no failures.  By default, output is generated only in
     case of an example failure.

     Optional argument _compileflags_ gives the set of flags that
     should be used by the Python compiler when running the examples.
     By default, or if `None', flags are deduced corresponding to the
     set of future features found in _globs_.

     Optional argument _optionflags_ works as for function
     testfile() above.

5.25.2.11 Unittest API
......................

As your collection of doctest'ed modules grows, you'll want a way to
run all their doctests systematically.  Prior to Python 2.4, see
doctest had a barely documented `Tester' class that supplied a
rudimentary way to combine doctests from multiple modules. `Tester' was
feeble, and in practice most serious Python testing frameworks build on
the unittest module, which supplies many flexible ways to
combine tests from multiple sources.  So, in Python 2.4, see doctest.'
s `Tester' class is deprecated, and doctest provides two
functions that can be used to create unittest.  test suites
from modules and text files containing doctests.  To integrate with
unittest test discovery, include a `load_tests()' function
in your test module:

    import unittest
    import doctest
    import my_module_with_doctests

    def load_tests(loader, tests, ignore):
        tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
        return test

There are two main functions for creating unittest.TestSuite
instances from text files and modules with doctests:

 -- Function: doctest.DocFileSuite (*paths, [module_relative][,
          package][, setUp][, tearDown][, globs][, optionflags][,
          parser][, encoding])
     Convert doctest tests from one or more text files to a
     unittest.TestSuite.

     The returned unittest.TestSuite is to be run by the
     unittest framework and runs the interactive examples in each file.
     If an example in any file fails, then the synthesized unit test
     fails, and a `failureException' exception is raised showing the
     name of the file containing the test and a (sometimes approximate)
     line number.

     Pass one or more paths (as strings) to text files to be examined.

     Options may be provided as keyword arguments:

     Optional argument _module_relative_ specifies how the filenames in
     _paths_ should be interpreted:

        * If _module_relative_ is `True' (the default), then each
          filename in _paths_ specifies an OS-independent
          module-relative path.  By default, this path is relative to
          the calling module's directory; but if the _package_ argument
          is specified, then it is relative to that package.  To ensure
          OS-independence, each filename should use `/' characters to
          separate path segments, and may not be an absolute path
          (i.e., it may not begin with `/').

        * If _module_relative_ is `False', then each filename in
          _paths_ specifies an OS-specific path.  The path may be
          absolute or relative; relative paths are resolved with
          respect to the current working directory.

     Optional argument _package_ is a Python package or the name of a
     Python package whose directory should be used as the base
     directory for module-relative filenames in _paths_.  If no package
     is specified, then the calling module's directory is used as the
     base directory for module-relative filenames.  It is an error to
     specify _package_ if _module_relative_ is `False'.

     Optional argument _setUp_ specifies a set-up function for the test
     suite.  This is called before running the tests in each file.  The
     _setUp_ function will be passed a DocTest object.
     The setUp function can access the test globals as the _globs_
     attribute of the test passed.

     Optional argument _tearDown_ specifies a tear-down function for
     the test suite.  This is called after running the tests in each
     file.  The _tearDown_ function will be passed a DocTest
     object.  The setUp function can access the test globals as
     the _globs_ attribute of the test passed.

     Optional argument _globs_ is a dictionary containing the initial
     global variables for the tests.  A new copy of this dictionary is
     created for each test.  By default, _globs_ is a new empty
     dictionary.

     Optional argument _optionflags_ specifies the default doctest
     options for the tests, created by or-ing together individual
     option flags.  See section Option Flags and Directives.
     See function set_unittest_reportflags() below
     for a better way to set reporting options.

     Optional argument _parser_ specifies a DocTestParser.
     (or subclass) that should be used to extract tests from the files.
     It defaults to a normal parser (i.e., `DocTestParser()').

     Optional argument _encoding_ specifies an encoding that should be
     used to convert the file to unicode.

     New in version 2.4.

     Changed in version 2.5: The global `__file__' was added to the
     globals provided to doctests loaded from a text file using
     DocFileSuite().

     Changed in version 2.5: The parameter _encoding_ was added.

 -- Function: doctest.DocTestSuite ([module][, globs][, extraglobs][,
          test_finder][, setUp][, tearDown][, checker])
     Convert doctest tests for a module to a unittest.TestSuite.

     The returned unittest.TestSuite is to be run by the
     unittest framework and runs each doctest in the module.  If any of
     the doctests fail, then the synthesized unit test fails, and a
     `failureException' exception is raised showing the name of the
     file containing the test and a (sometimes approximate) line number.

     Optional argument _module_ provides the module to be tested.  It
     can be a module object or a (possibly dotted) module name.  If not
     specified, the module calling this function is used.

     Optional argument _globs_ is a dictionary containing the initial
     global variables for the tests.  A new copy of this dictionary is
     created for each test.  By default, _globs_ is a new empty
     dictionary.

     Optional argument _extraglobs_ specifies an extra set of global
     variables, which is merged into _globs_.  By default, no extra
     globals are used.

     Optional argument _test_finder_ is the DocTestFinder.
     object (or a drop-in replacement) that is used to extract doctests
     from the module.

     Optional arguments _setUp_, _tearDown_, and _optionflags_ are the
     same as for function DocFileSuite() above.

     New in version 2.3.

     Changed in version 2.4: The parameters _globs_, _extraglobs_,
     _test_finder_, _setUp_, _tearDown_, and _optionflags_ were added;
     this function now uses the same search technique as
     testmod().

  Under the covers, see DocTestSuite() creates a
unittest.TestSuite out of `doctest.DocTestCase' instances, and
`DocTestCase' is a subclass of unittest.TestCase.
`DocTestCase' isn't documented here (it's an internal detail), but
studying its code can answer questions about the exact details of
unittest integration.

  Similarly, see DocFileSuite() creates a
unittest.TestSuite out of `doctest.DocFileCase' instances, and
`DocFileCase' is a subclass of `DocTestCase'.

  So both ways of creating a unittest.TestSuite run
instances of `DocTestCase'.  This is important for a subtle reason:
when you run doctest functions yourself, you can control the
doctest options in use directly, by passing option flags to
doctest functions.  However, if you're writing a
unittest framework, see unittest ultimately controls when
and how tests get run.  The framework author typically wants to control
doctest reporting options (perhaps, e.g., specified by
command line options), but there's no way to pass options through
unittest to doctest test runners.

  For this reason, see doctest also supports a notion of
doctest.  reporting flags specific to unittest support,
via this function:

 -- Function: doctest.set_unittest_reportflags (flags)
     Set the doctest reporting flags to use.

     Argument _flags_ or's together option flags.  See section
     Option Flags and Directives.  Only "reporting flags" can be
     used.

     This is a module-global setting, and affects all future doctests
     run by module unittest.:  the `runTest()' method of
     `DocTestCase' looks at the option flags specified for the test
     case when the `DocTestCase' instance was constructed.  If no
     reporting flags were specified (which is the typical and expected
     case), see doctest.'s unittest reporting flags
     are or'ed into the option flags, and the option flags so augmented
     are passed to the DocTestRunner instance created to
     run the doctest.  If any reporting flags were specified when the
     `DocTestCase' instance was constructed, see doctest.'s
     unittest reporting flags are ignored.

     The value of the unittest reporting flags in effect
     before the function was called is returned by the function.

     New in version 2.4.

5.25.2.12 Advanced API
......................

The basic API is a simple wrapper that's intended to make doctest easy
to use.  It is fairly flexible, and should meet most users' needs;
however, if you require more fine-grained control over testing, or wish
to extend doctest's capabilities, then you should use the advanced API.

  The advanced API revolves around two container classes, which are
used to store the interactive examples extracted from doctest cases:

   * Example.: A single Python statement, paired
     with its expected output.

   * DocTest.: A collection of Example.s,
     typically extracted from a single docstring or text file.

  Additional processing classes are defined to find, parse, and run,
and check doctest examples:

   * DocTestFinder.: Finds all docstrings in a given
     module, and uses a DocTestParser to create a
     DocTest from every docstring that contains interactive
     examples.

   * DocTestParser.: Creates a DocTest object
     from a string (such as an object's docstring).

   * DocTestRunner.: Executes the examples in a
     DocTest, and uses an OutputChecker to verify
     their output.

   * OutputChecker.: Compares the actual output from a
     doctest example with the expected output, and decides whether they
     match.

  The relationships among these processing classes are summarized in
the following diagram:

                                list of:
    +------+                   +---------+
    |module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
    +------+    |        ^     +---------+     |       ^    (printed)
                |        |     | Example |     |       |
                v        |     |   ...   |     v       |
               DocTestParser   | Example |   OutputChecker
                               +---------+

5.25.2.13 DocTest Objects
.........................

 -- Class: doctest.DocTest (examples, globs, name, filename, lineno,
          docstring)
     A collection of doctest examples that should be run in a single
     namespace.  The constructor arguments are used to initialize the
     member variables of the same names.

     New in version 2.4.

     DocTest defines the following member variables.  They
     are initialized by the constructor, and should not be modified
     directly.

      -- Attribute: examples
          A list of Example objects encoding the
          individual interactive Python examples that should be run by
          this test.

      -- Attribute: globs
          The namespace (aka globals) that the examples should be run
          in. This is a dictionary mapping names to values.  Any
          changes to the namespace made by the examples (such as
          binding new variables) will be reflected in globs.
          after the test is run.

      -- Attribute: name
          A string name identifying the DocTest.
          Typically, this is the name of the object or file that the
          test was extracted from.

      -- Attribute: filename
          The name of the file that this DocTest was
          extracted from; or `None' if the filename is unknown, or if
          the DocTest was not extracted from a file.

      -- Attribute: lineno
          The line number within filename where this
          DocTest begins, or `None' if the line number is
          unavailable.  This line number is zero-based with respect to
          the beginning of the file.

      -- Attribute: docstring
          The string that the test was extracted from, or 'None' if the
          string is unavailable, or if the test was not extracted from
          a string.

5.25.2.14 Example Objects
.........................

 -- Class: doctest.Example (source, want[, exc_msg][, lineno][,
          indent][, options])
     A single interactive example, consisting of a Python statement and
     its expected output.  The constructor arguments are used to
     initialize the member variables of the same names.

     New in version 2.4.

     Example defines the following member variables.  They
     are initialized by the constructor, and should not be modified
     directly.

      -- Attribute: source
          A string containing the example's source code.  This source
          code consists of a single Python statement, and always ends
          with a newline; the constructor adds a newline when necessary.

      -- Attribute: want
          The expected output from running the example's source code
          (either from stdout, or a traceback in case of exception).
          want ends with a newline unless no output is
          expected, in which case it's an empty string.  The
          constructor adds a newline when necessary.

      -- Attribute: exc_msg
          The exception message generated by the example, if the
          example is expected to generate an exception; or `None' if it
          is not expected to generate an exception.  This exception
          message is compared against the return value of
          traceback.format_exception_only().  exc_msg
          ends with a newline unless it's `None'.  The
          constructor adds a newline if needed.

      -- Attribute: lineno
          The line number within the string containing this example
          where the example begins.  This line number is zero-based
          with respect to the beginning of the containing string.

      -- Attribute: indent
          The example's indentation in the containing string, i.e., the
          number of space characters that precede the example's first
          prompt.

      -- Attribute: options
          A dictionary mapping from option flags to `True' or `False',
          which is used to override default options for this example.
          Any option flags not contained in this dictionary are left at
          their default value (as specified by the DocTestRunner.'
          s `optionflags'). By default, no options are set.

5.25.2.15 DocTestFinder objects
...............................

 -- Class: doctest.DocTestFinder ([verbose][, parser][, recurse][,
          exclude_empty])
     A processing class used to extract the DocTest.s that
     are relevant to a given object, from its docstring and the
     docstrings of its contained objects.  See DocTest.s can
     currently be extracted from the following object types: modules,
     functions, classes, methods, staticmethods, classmethods, and
     properties.

     The optional argument _verbose_ can be used to display the objects
     searched by the finder.  It defaults to `False' (no output).

     The optional argument _parser_ specifies the DocTestParser
     object (or a drop-in replacement) that is used to extract
     doctests from docstrings.

     If the optional argument _recurse_ is false, then
     DocTestFinder.find().  will only examine the given object,
     and not any contained objects.

     If the optional argument _exclude_empty_ is false, then
     DocTestFinder.find() will include tests for objects with
     empty docstrings.

     New in version 2.4.

     DocTestFinder defines the following method:

      -- Method: find (obj[, name][, module][, globs][, extraglobs])
          Return a list of the DocTest.s that are defined
          by _obj_'s docstring, or by any of its contained objects'
          docstrings.

          The optional argument _name_ specifies the object's name;
          this name will be used to construct names for the returned
          DocTest.s.  If _name_ is not specified, then
          `obj.__name__' is used.

          The optional parameter _module_ is the module that contains
          the given object.  If the module is not specified or is None,
          then the test finder will attempt to automatically determine
          the correct module.  The object's module is used:

             * As a default namespace, if _globs_ is not specified.

             * To prevent the DocTestFinder from extracting DocTests
               from objects that are imported from other modules.
               (Contained objects with modules other than _module_ are
               ignored.)

             * To find the name of the file containing the object.

             * To help find the line number of the object within its
               file.

          If _module_ is `False', no attempt to find the module will be
          made.  This is obscure, of use mostly in testing doctest
          itself: if _module_ is `False', or is `None' but cannot be
          found automatically, then all objects are considered to
          belong to the (non-existent) module, so all contained objects
          will (recursively) be searched for doctests.

          The globals for each DocTest is formed by
          combining _globs_ and _extraglobs_ (bindings in _extraglobs_
          override bindings in _globs_).  A new shallow copy of the
          globals dictionary is created for each DocTest.
          If _globs_ is not specified, then it defaults to the module's
          ___dict___, if specified, or `{}' otherwise.  If _extraglobs_
          is not specified, then it defaults to `{}'.

5.25.2.16 DocTestParser objects
...............................

 -- Class: doctest.DocTestParser
     A processing class used to extract interactive examples from a
     string, and use them to create a DocTest object.

     New in version 2.4.

     DocTestParser defines the following methods:

      -- Method: get_doctest (string, globs, name, filename, lineno)
          Extract all doctest examples from the given string, and
          collect them into a DocTest object.

          _globs_, _name_, _filename_, and _lineno_ are attributes for
          the new DocTest object.  See the documentation
          for DocTest for more information.

      -- Method: get_examples (string[, name])
          Extract all doctest examples from the given string, and
          return them as a list of Example objects.  Line
          numbers are 0-based.  The optional argument _name_ is a name
          identifying this string, and is only used for error messages.

      -- Method: parse (string[, name])
          Divide the given string into examples and intervening text,
          and return them as a list of alternating Example.
          s and strings. Line numbers for the Example.
          s are 0-based.  The optional argument _name_ is a name
          identifying this string, and is only used for error messages.

5.25.2.17 DocTestRunner objects
...............................

 -- Class: doctest.DocTestRunner ([checker][, verbose][, optionflags])
     A processing class used to execute and verify the interactive
     examples in a DocTest.

     The comparison between expected outputs and actual outputs is done
     by an OutputChecker.  This comparison may be
     customized with a number of option flags; see section Option Flags and Directives
     for more information.  If the option
     flags are insufficient, then the comparison may also be customized
     by passing a subclass of OutputChecker to the
     constructor.

     The test runner's display output can be controlled in two ways.
     First, an output function can be passed to `TestRunner.run()';
     this function will be called with strings that should be
     displayed.  It defaults to `sys.stdout.write'.  If capturing the
     output is not sufficient, then the display output can be also
     customized by subclassing DocTestRunner, and overriding the methods
     report_start(), report_success(),
     report_unexpected_exception(), and report_failure().

     The optional keyword argument _checker_ specifies the
     OutputChecker.  object (or drop-in replacement) that should
     be used to compare the expected outputs to the actual outputs of
     doctest examples.

     The optional keyword argument _verbose_ controls the
     DocTestRunner.'s verbosity.  If _verbose_ is `True', then
     information is printed about each example, as it is run.  If
     _verbose_ is `False', then only failures are printed.  If
     _verbose_ is unspecified, or `None', then verbose output is used
     iff the command-line switch -v is used.

     The optional keyword argument _optionflags_ can be used to control
     how the test runner compares expected output to actual output, and
     how it displays failures.  For more information, see section
     Option Flags and Directives.

     New in version 2.4.

     DocTestParser defines the following methods:

      -- Method: report_start (out, test, example)
          Report that the test runner is about to process the given
          example. This method is provided to allow subclasses of
          DocTestRunner to customize their output; it should not
          be called directly.

          _example_ is the example about to be processed.  _test_ is
          the test _containing example_.  _out_ is the output function
          that was passed to DocTestRunner.run().

      -- Method: report_success (out, test, example, got)
          Report that the given example ran successfully.  This method
          is provided to allow subclasses of DocTestRunner.
          to customize their output; it should not be called directly.

          _example_ is the example about to be processed.  _got_ is the
          actual output from the example.  _test_ is the test
          containing _example_.  _out_ is the output function that was
          passed to DocTestRunner.run().

      -- Method: report_failure (out, test, example, got)
          Report that the given example failed.  This method is
          provided to allow subclasses of DocTestRunner to
          customize their output; it should not be called directly.

          _example_ is the example about to be processed.  _got_ is the
          actual output from the example.  _test_ is the test
          containing _example_.  _out_ is the output function that was
          passed to DocTestRunner.run().

      -- Method: report_unexpected_exception (out, test, example,
               exc_info)
          Report that the given example raised an unexpected exception.
          This method is provided to allow subclasses of
          DocTestRunner to customize their output; it should not
          be called directly.

          _example_ is the example about to be processed. _exc_info_ is
          a tuple containing information about the unexpected exception
          (as returned by sys.exc_info()). _test_ is the
          test containing _example_.  _out_ is the output function that
          was passed to DocTestRunner.run().

      -- Method: run (test[, compileflags][, out][, clear_globs])
          Run the examples in _test_ (a DocTest object),
          and display the results using the writer function _out_.

          The examples are run in the namespace `test.globs'.  If
          _clear_globs_ is true (the default), then this namespace will
          be cleared after the test runs, to help with garbage
          collection. If you would like to examine the namespace after
          the test completes, then use _clear_globs=False_.

          _compileflags_ gives the set of flags that should be used by
          the Python compiler when running the examples.  If not
          specified, then it will default to the set of future-import
          flags that apply to _globs_.

          The output of each example is checked using the
          DocTestRunner.'s output checker, and the results are
          formatted by the `DocTestRunner.report_*()' methods.

      -- Method: summarize ([verbose])
          Print a summary of all the test cases that have been run by
          this DocTestRunner, and return a named tuple.
          `TestResults(failed, attempted)'.

          The optional _verbose_ argument controls how detailed the
          summary is.  If the verbosity is not specified, then the
          DocTestRunner.'s verbosity is used.

          Changed in version 2.6: Use a named tuple.

5.25.2.18 OutputChecker objects
...............................

 -- Class: doctest.OutputChecker
     A class used to check the whether the actual output from a doctest
     example matches the expected output.  See OutputChecker.
     defines two methods: check_output(), which compares a
     given pair of outputs, and returns true if they match; and
     output_difference(), which returns a string describing the
     differences between two outputs.

     New in version 2.4.

     OutputChecker defines the following methods:

      -- Method: check_output (want, got, optionflags)
          Return `True' iff the actual output from an example (_got_)
          matches the expected output (_want_).  These strings are
          always considered to match if they are identical; but
          depending on what option flags the test runner is using,
          several non-exact match types are also possible.  See section
          Option Flags and Directives for more information
          about option flags.

      -- Method: output_difference (example, got, optionflags)
          Return a string describing the differences between the
          expected output for a given example (_example_) and the
          actual output (_got_).  _optionflags_ is the set of option
          flags used to compare _want_ and _got_.

5.25.2.19 Debugging
...................

Doctest provides several mechanisms for debugging doctest examples:

   * Several functions convert doctests to executable Python programs,
     which can be run under the Python debugger, see pdb.

   * The DebugRunner class is a subclass of
     DocTestRunner that raises an exception for the first
     failing example, containing information about that example. This
     information can be used to perform post-mortem debugging on the
     example.

   * The unittest cases generated by DocTestSuite()
     support the debug() method defined by
     unittest.TestCase.

   * You can add a call to pdb.set_trace() in a doctest
     example, and you'll drop into the Python debugger when that line
     is executed.  Then you can inspect current values of variables,
     and so on.  For example, suppose `a.py' contains just this module
     docstring:

         """
         >>> def f(x):
         ...     g(x*2)
         >>> def g(x):
         ...     print x+3
         ...     import pdb; pdb.set_trace()
         >>> f(3)
         9
         """

     Then an interactive Python session may look like this:

         >>> import a, doctest
         >>> doctest.testmod(a)
         --Return--
         > <doctest a[1]>(3)g()->None
         -> import pdb; pdb.set_trace()
         (Pdb) list
           1     def g(x):
           2         print x+3
           3  ->     import pdb; pdb.set_trace()
         [EOF]
         (Pdb) print x
         6
         (Pdb) step
         --Return--
         > <doctest a[0]>(2)f()->None
         -> g(x*2)
         (Pdb) list
           1     def f(x):
           2  ->     g(x*2)
         [EOF]
         (Pdb) print x
         3
         (Pdb) step
         --Return--
         > <doctest a[2]>(1)?()->None
         -> f(3)
         (Pdb) cont
         (0, 3)
         >>>

     Changed in version 2.4: The ability to use pdb.set_trace()
     usefully inside doctests was added.

  Functions that convert doctests to Python code, and possibly run the
synthesized code under the debugger:

 -- Function: doctest.script_from_examples (s)
     Convert text with examples to a script.

     Argument _s_ is a string containing doctest examples.  The string
     is converted to a Python script, where doctest examples in _s_ are
     converted to regular code, and everything else is converted to
     Python comments.  The generated script is returned as a string.
     For example,

         import doctest
         print doctest.script_from_examples(r"""
             Set x and y to 1 and 2.
             >>> x, y = 1, 2

             Print their sum:
             >>> print x+y
             3
         """)

     displays:

         # Set x and y to 1 and 2.
         x, y = 1, 2
         #
         # Print their sum:
         print x+y
         # Expected:
         ## 3

     This function is used internally by other functions (see below),
     but can also be useful when you want to transform an interactive
     Python session into a Python script.

     New in version 2.4.

 -- Function: doctest.testsource (module, name)
     Convert the doctest for an object to a script.

     Argument _module_ is a module object, or dotted name of a module,
     containing the object whose doctests are of interest.  Argument
     _name_ is the name (within the module) of the object with the
     doctests of interest.  The result is a string, containing the
     object's docstring converted to a Python script, as described for
     script_from_examples() above.  For example, if module
     `a.py' contains a top-level function `f()', then

         import a, doctest
         print doctest.testsource(a, "a.f")

     prints a script version of function `f()''s docstring, with
     doctests converted to code, and the rest placed in comments.

     New in version 2.3.

 -- Function: doctest.debug (module, name[, pm])
     Debug the doctests for an object.

     The _module_ and _name_ arguments are the same as for function
     testsource() above.  The synthesized Python script
     for the named object's docstring is written to a temporary file,
     and then that file is run under the control of the Python
     debugger, see pdb.

     A shallow copy of `module.__dict__' is used for both local and
     global execution context.

     Optional argument _pm_ controls whether post-mortem debugging is
     used.  If _pm_ has a true value, the script file is run directly,
     and the debugger gets involved only if the script terminates via
     raising an unhandled exception.  If it does, then post-mortem
     debugging is invoked, via pdb.post_mortem(), passing
     the traceback object from the unhandled exception.  If _pm_ is not
     specified, or is false, the script is run under the debugger from
     the start, via passing an appropriate execfile() call
     to pdb.run().

     New in version 2.3.

     Changed in version 2.4: The _pm_ argument was added.

 -- Function: doctest.debug_src (src[, pm][, globs])
     Debug the doctests in a string.

     This is like function debug() above, except that a
     string containing doctest examples is specified directly, via the
     _src_ argument.

     Optional argument _pm_ has the same meaning as in function
     debug() above.

     Optional argument _globs_ gives a dictionary to use as both local
     and global execution context.  If not specified, or `None', an
     empty dictionary is used.  If specified, a shallow copy of the
     dictionary is used.

     New in version 2.4.

  The DebugRunner class, and the special exceptions it may
raise, are of most interest to testing framework authors, and will only
be sketched here.  See the source code, and especially
DebugRunner.'s docstring (which is a doctest!) for more details:

 -- Class: doctest.DebugRunner ([checker][, verbose][, optionflags])
     A subclass of DocTestRunner that raises an exception
     as soon as a failure is encountered.  If an unexpected exception
     occurs, an UnexpectedException exception is raised,
     containing the test, the example, and the original exception.  If
     the output doesn't match, then a DocTestFailure.
     exception is raised, containing the test, the example, and the
     actual output.

     For information about the constructor parameters and methods, see
     the documentation for DocTestRunner in section
     Advanced API.

  There are two exceptions that may be raised by DebugRunner
instances:

 -- Exception: doctest.DocTestFailure (test, example, got)
     An exception raised by DocTestRunner to signal that a
     doctest example's actual output did not match its expected output.
     The constructor arguments are used to initialize the member
     variables of the same names.

  DocTestFailure defines the following member variables:

 -- Attribute: DocTestFailure.test
     The DocTest object that was being run when the
     example failed.

 -- Attribute: DocTestFailure.example
     The Example that failed.

 -- Attribute: DocTestFailure.got
     The example's actual output.

 -- Exception: doctest.UnexpectedException (test, example, exc_info)
     An exception raised by DocTestRunner to signal that a
     doctest example raised an unexpected exception.  The constructor
     arguments are used to initialize the member variables of the same
     names.

  UnexpectedException defines the following member
variables:

 -- Attribute: UnexpectedException.test
     The DocTest object that was being run when the
     example failed.

 -- Attribute: UnexpectedException.example
     The Example that failed.

 -- Attribute: UnexpectedException.exc_info
     A tuple containing information about the unexpected exception, as
     returned by sys.exc_info().

5.25.2.20 Soapbox
.................

As mentioned in the introduction, see doctest has grown to have
three primary uses:

  1. Checking examples in docstrings.

  2. Regression testing.

  3. Executable documentation / literate testing.

  These uses have different requirements, and it is important to
distinguish them.  In particular, filling your docstrings with obscure
test cases makes for bad documentation.

  When writing a docstring, choose docstring examples with care.
There's an art to this that needs to be learned--it may not be natural
at first.  Examples should add genuine value to the documentation.  A
good example can often be worth many words. If done with care, the
examples will be invaluable for your users, and will pay back the time
it takes to collect them many times over as the years go by and things
change.  I'm still amazed at how often one of my doctest.
examples stops working after a "harmless" change.

  Doctest also makes an excellent tool for regression testing,
especially if you don't skimp on explanatory text.  By interleaving
prose and examples, it becomes much easier to keep track of what's
actually being tested, and why.  When a test fails, good prose can make
it much easier to figure out what the problem is, and how it should be
fixed.  It's true that you could write extensive comments in code-based
testing, but few programmers do. Many have found that using doctest
approaches instead leads to much clearer tests.  Perhaps this is simply
because doctest makes writing prose a little easier than writing code,
while writing comments in code is a little harder.  I think it goes
deeper than just that: the natural attitude when writing a
doctest-based test is that you want to explain the fine points of your
software, and illustrate them with examples.  This in turn naturally
leads to test files that start with the simplest features, and
logically progress to complications and edge cases.  A coherent
narrative is the result, instead of a collection of isolated functions
that test isolated bits of functionality seemingly at random.  It's a
different attitude, and produces different results, blurring the
distinction between testing and explaining.

  Regression testing is best confined to dedicated objects or files.
There are several options for organizing tests:

   * Write text files containing test cases as interactive examples,
     and test the files using testfile() or
     DocFileSuite().  This is recommended, although is easiest to
     do for new projects, designed from the start to use doctest.

   * Define functions named `_regrtest_topic' that consist of single
     docstrings, containing test cases for the named topics.  These
     functions can be included in the same file as the module, or
     separated out into a separate test file.

   * Define a `__test__' dictionary mapping from regression test topics
     to docstrings containing test cases.

Footnotes
.........

     (1) Examples containing both expected output and an exception are
     not supported.  Trying to guess where one ends and the other
     begins is too error-prone, and that also makes for a confusing
     test.

5.25.3 `unittest' -- Unit testing framework
-------------------------------------------

New in version 2.1.

  The Python unit testing framework, sometimes referred to as "PyUnit,"
is a Python language version of JUnit, by Kent Beck and Erich Gamma.
JUnit is, in turn, a Java version of Kent's Smalltalk testing
framework.  Each is the de facto standard unit testing framework for
its respective language.

  unittest supports test automation, sharing of setup and
shutdown code for tests, aggregation of tests into collections, and
independence of the tests from the reporting framework.  The
unittest module provides classes that make it easy to support
these qualities for a set of tests.

  To achieve this, see unittest supports some important
concepts:

test fixture
     A _test fixture_ represents the preparation needed to perform one
     or more tests, and any associate cleanup actions.  This may
     involve, for example, creating temporary or proxy databases,
     directories, or starting a server process.

test case
     A _test case_ is the smallest unit of testing.  It checks for a
     specific response to a particular set of inputs.  unittest
     provides a base class, see TestCase, which may be used
     to create new test cases.

test suite
     A _test suite_ is a collection of test cases, test suites, or
     both.  It is used to aggregate tests that should be executed
     together.

test runner
     A _test runner_ is a component which orchestrates the execution of
     tests and provides the outcome to the user.  The runner may use a
     graphical interface, a textual interface, or return a special
     value to indicate the results of executing the tests.

  The test case and test fixture concepts are supported through the
TestCase and FunctionTestCase classes; the
former should be used when creating new tests, and the latter can be
used when integrating existing test code with a unittest.
-driven framework. When building test fixtures using
TestCase, the setUp() and tearDown().
methods can be overridden to provide initialization and cleanup for the
fixture.  With FunctionTestCase, existing functions can be
passed to the constructor for these purposes.  When the test is run, the
fixture initialization is run first; if it succeeds, the cleanup method
is run after the test has been executed, regardless of the outcome of
the test.  Each instance of the TestCase will only be used
to run a single test method, so a new fixture is created for each test.

  Test suites are implemented by the TestSuite class.  This
class allows individual tests and test suites to be aggregated; when
the suite is executed, all tests added directly to the suite and in
"child" test suites are run.

  A test runner is an object that provides a single method, `run()',
which accepts a TestCase or TestSuite.  object
as a parameter, and returns a result object.  The class
TestResult is provided for use as the result object. See
unittest.  provides the TextTestRunner as an example
test runner which reports test results on the standard error stream by
default.  Alternate runners can be implemented for other environments
(such as graphical environments) without any need to derive from a
specific class.

Module doctest.
     Another test-support module with a very different flavor.

unittest2: A backport of new unittest features for Python 2.4-2.6 (http://pypi.python.org/pypi/unittest2)
     Many new features were added to unittest in Python 2.7, including
     test discovery. unittest2 allows you to use these features with
     earlier versions of Python.

Simple Smalltalk Testing: With Patterns (http://www.XProgramming.com/testfram.htm)
     Kent Beck's original paper on testing frameworks using the pattern
     shared by unittest.

Nose (http://code.google.com/p/python-nose/)and py.test (http://pytest.org)
     Third-party unittest frameworks with a lighter-weight syntax for
     writing tests.  For example, `assert func(10) == 42'.

The Python Testing Tools Taxonomy (http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy)
     An extensive list of Python testing tools including functional
     testing frameworks and mock object libraries.

Testing in Python Mailing List (http://lists.idyll.org/listinfo/testing-in-python)
     A special-interest-group for discussion of testing, and testing
     tools, in Python.

5.25.3.1 Basic example
......................

The unittest module provides a rich set of tools for
constructing and running tests.  This section demonstrates that a small
subset of the tools suffice to meet the needs of most users.

  Here is a short script to test three functions from the random
module:

    import random
    import unittest

    class TestSequenceFunctions(unittest.TestCase):

        def setUp(self):
            self.seq = range(10)

        def test_shuffle(self):
            # make sure the shuffled sequence does not lose any elements
            random.shuffle(self.seq)
            self.seq.sort()
            self.assertEqual(self.seq, range(10))

            # should raise an exception for an immutable sequence
            self.assertRaises(TypeError, random.shuffle, (1,2,3))

        def test_choice(self):
            element = random.choice(self.seq)
            self.assertTrue(element in self.seq)

        def test_sample(self):
            with self.assertRaises(ValueError):
                random.sample(self.seq, 20)
            for element in random.sample(self.seq, 5):
                self.assertTrue(element in self.seq)

    if __name__ == '__main__':
        unittest.main()

A testcase is created by subclassing unittest.TestCase.  The
three individual tests are defined with methods whose names start with
the letters `test'.  This naming convention informs the test runner
about which methods represent tests.

  The crux of each test is a call to assertEqual() to check
for an expected result; see assertTrue() to verify a condition;
or assertRaises() to verify that an expected exception gets
raised.  These methods are used instead of the assert.
statement so the test runner can accumulate all test results and
produce a report.

  When a setUp() method is defined, the test runner will
run that method prior to each test.  Likewise, if a tearDown()
method is defined, the test runner will invoke that method after
each test.  In the example, see setUp() was used to create a
fresh sequence for each test.

  The final block shows a simple way to run the tests. See
unittest.main().  provides a command line interface to the test
script.  When run from the command line, the above script produces an
output that looks like this:

    ...
    ----------------------------------------------------------------------
    Ran 3 tests in 0.000s

    OK

Instead of unittest.main(), there are other ways to run the
tests with a finer level of control, less terse output, and no
requirement to be run from the command line.  For example, the last two
lines may be replaced with:

    suite = unittest.TestLoader().loadTestsFromTestCase(TestSequenceFunctions)
    unittest.TextTestRunner(verbosity=2).run(suite)

Running the revised script from the interpreter or another script
produces the following output:

    test_choice (__main__.TestSequenceFunctions) ... ok
    test_sample (__main__.TestSequenceFunctions) ... ok
    test_shuffle (__main__.TestSequenceFunctions) ... ok

    ----------------------------------------------------------------------
    Ran 3 tests in 0.110s

    OK

The above examples show the most commonly used unittest.
features which are sufficient to meet many everyday testing needs.  The
remainder of the documentation explores the full feature set from first
principles.

5.25.3.2 Command Line Interface
...............................

The unittest module can be used from the command line to run tests from
modules, classes or even individual test methods:

    python -m unittest test_module1 test_module2
    python -m unittest test_module.TestClass
    python -m unittest test_module.TestClass.test_method

You can pass in a list with any combination of module names, and fully
qualified class or method names.

  You can run tests with more detail (higher verbosity) by passing in
the -v flag:

    python -m unittest -v test_module

For a list of all the command line options:

    python -m unittest -h

Changed in version 2.7: In earlier versions it was only possible to run
individual test methods and not modules or classes.

5.25.3.3 failfast, catch and buffer command line options
........................................................

unittest supports three command options.

   * `-b' / `--buffer'

     The standard output and standard error streams are buffered during
     the test run. Output during a passing test is discarded. Output is
     echoed normally on test fail or error and is added to the failure
     messages.

   * -c. / `--catch'

     Control-C during the test run waits for the current test to end
     and then reports all the results so far. A second control-C raises
     the normal KeyboardInterrupt exception.

     See Signal Handling for the functions that provide
     this functionality.

   * `-f' / `--failfast'

     Stop the test run on the first error or failure.

  New in version 2.7: The command line options `-c', `-b' and `-f' were
added.

  The command line can also be used for test discovery, for running all
of the tests in a project or just a subset.

5.25.3.4 Test Discovery
.......................

New in version 2.7.

  Unittest supports simple test discovery. For a project's tests to be
compatible with test discovery they must all be importable from the top
level directory of the project (in other words, they must all be in
Python packages).

  Test discovery is implemented in TestLoader.discover(),
but can also be used from the command line. The basic command line
usage is:

    cd project_directory
    python -m unittest discover

The `discover' sub-command has the following options:

    `-v'
    `--verbose'
          Verbose output

    `-s directory'
          Directory to start discovery ('.' default)

    `-p pattern'
          Pattern to match test files ('test*.py' default)

    `-t directory'
          Top level directory of project (default to start directory)

  The -s, `-p', and -t options can be passed in
as positional arguments in that order. The following two command lines
are equivalent:

    python -m unittest discover -s project_directory -p '*_test.py'
    python -m unittest discover project_directory '*_test.py'

As well as being a path it is possible to pass a package name, for
example `myproject.subpackage.test', as the start directory. The
package name you supply will then be imported and its location on the
filesystem will be used as the start directory.

     Caution: Test discovery loads tests by importing them. Once test
     discovery has found all the test files from the start directory
     you specify it turns the paths into package names to import. For
     example `foo/bar/baz.py' will be imported as `foo.bar.baz'.

     If you have a package installed globally and attempt test
     discovery on a different copy of the package then the import
     _could_ happen from the wrong place. If this happens test
     discovery will warn you and exit.

     If you supply the start directory as a package name rather than a
     path to a directory then discover assumes that whichever location
     it imports from is the location you intended, so you will not get
     the warning.

  Test modules and packages can customize test loading and discovery by
through the load_tests protocol.

5.25.3.5 Organizing test code
.............................

The basic building blocks of unit testing are _test cases_ -- single
scenarios that must be set up and checked for correctness.  In
unittest, test cases are represented by instances of
unittest.'s TestCase.  class. To make your own test
cases you must write subclasses of TestCase, or use
FunctionTestCase.

  An instance of a TestCase.-derived class is an object that
can completely run a single test method, together with optional set-up
and tidy-up code.

  The testing code of a TestCase instance should be
entirely self contained, such that it can be run either in isolation or
in arbitrary combination with any number of other test cases.

  The simplest TestCase subclass will simply override the
`runTest()' method in order to perform specific testing code:

    import unittest

    class DefaultWidgetSizeTestCase(unittest.TestCase):
        def runTest(self):
            widget = Widget('The widget')
            self.assertEqual(widget.size(), (50, 50), 'incorrect default size')

Note that in order to test something, we use the one of the `assert*()'
methods provided by the TestCase base class.  If the test
fails, an exception will be raised, and unittest will
identify the test case as a _failure_.  Any other exceptions will be
treated as _errors_. This helps you identify where the problem is:
_failures_ are caused by incorrect results - a 5 where you expected a
6. _Errors_ are caused by incorrect code - e.g., a TypeError
caused by an incorrect function call.

  The way to run a test case will be described later.  For now, note
that to construct an instance of such a test case, we call its
constructor without arguments:

    testCase = DefaultWidgetSizeTestCase()

Now, such test cases can be numerous, and their set-up can be
repetitive.  In the above case, constructing a `Widget' in each of 100
Widget test case subclasses would mean unsightly duplication.

  Luckily, we can factor out such set-up code by implementing a method
called setUp(), which the testing framework will
automatically call for us when we run the test:

    import unittest

    class SimpleWidgetTestCase(unittest.TestCase):
        def setUp(self):
            self.widget = Widget('The widget')

    class DefaultWidgetSizeTestCase(SimpleWidgetTestCase):
        def runTest(self):
            self.assertEqual(self.widget.size(), (50,50),
                             'incorrect default size')

    class WidgetResizeTestCase(SimpleWidgetTestCase):
        def runTest(self):
            self.widget.resize(100,150)
            self.assertEqual(self.widget.size(), (100,150),
                             'wrong size after resize')

If the setUp() method raises an exception while the test is
running, the framework will consider the test to have suffered an
error, and the `runTest()' method will not be executed.

  Similarly, we can provide a tearDown() method that tidies
up after the `runTest()' method has been run:

    import unittest

    class SimpleWidgetTestCase(unittest.TestCase):
        def setUp(self):
            self.widget = Widget('The widget')

        def tearDown(self):
            self.widget.dispose()
            self.widget = None

If setUp() succeeded, the tearDown() method will
be run whether `runTest()' succeeded or not.

  Such a working environment for the testing code is called a _fixture_.

  Often, many small test cases will use the same fixture.  In this
case, we would end up subclassing `SimpleWidgetTestCase' into many
small one-method classes such as `DefaultWidgetSizeTestCase'.  This is
time-consuming and discouraging, so in the same vein as JUnit, see
unittest provides a simpler mechanism:

    import unittest

    class WidgetTestCase(unittest.TestCase):
        def setUp(self):
            self.widget = Widget('The widget')

        def tearDown(self):
            self.widget.dispose()
            self.widget = None

        def test_default_size(self):
            self.assertEqual(self.widget.size(), (50,50),
                             'incorrect default size')

        def test_resize(self):
            self.widget.resize(100,150)
            self.assertEqual(self.widget.size(), (100,150),
                             'wrong size after resize')

Here we have not provided a `runTest()' method, but have instead
provided two different test methods.  Class instances will now each run
one of the `test_*()' methods, with `self.widget' created and destroyed
separately for each instance.  When creating an instance we must
specify the test method it is to run.  We do this by passing the method
name in the constructor:

    defaultSizeTestCase = WidgetTestCase('test_default_size')
    resizeTestCase = WidgetTestCase('test_resize')

Test case instances are grouped together according to the features they
test.  unittest provides a mechanism for this: the _test
suite_, represented by unittest.'s TestSuite.
class:

    widgetTestSuite = unittest.TestSuite()
    widgetTestSuite.addTest(WidgetTestCase('test_default_size'))
    widgetTestSuite.addTest(WidgetTestCase('test_resize'))

For the ease of running tests, as we will see later, it is a good idea
to provide in each test module a callable object that returns a
pre-built test suite:

    def suite():
        suite = unittest.TestSuite()
        suite.addTest(WidgetTestCase('test_default_size'))
        suite.addTest(WidgetTestCase('test_resize'))
        return suite

or even:

    def suite():
        tests = ['test_default_size', 'test_resize']

        return unittest.TestSuite(map(WidgetTestCase, tests))

Since it is a common pattern to create a TestCase subclass
with many similarly named test functions, see unittest provides
a TestLoader.  class that can be used to automate the
process of creating a test suite and populating it with individual
tests. For example,

    suite = unittest.TestLoader().loadTestsFromTestCase(WidgetTestCase)

will create a test suite that will run
`WidgetTestCase.test_default_size()' and `WidgetTestCase.test_resize'.
TestLoader uses the `'test'' method name prefix to identify
test methods automatically.

  Note that the order in which the various test cases will be run is
determined by sorting the test function names with the built-in
cmp() function.

  Often it is desirable to group suites of test cases together, so as
to run tests for the whole system at once.  This is easy, since
TestSuite instances can be added to a TestSuite just
as TestCase instances can be added to a TestSuite.:

    suite1 = module1.TheTestSuite()
    suite2 = module2.TheTestSuite()
    alltests = unittest.TestSuite([suite1, suite2])

You can place the definitions of test cases and test suites in the same
modules as the code they are to test (such as `widget.py'), but there
are several advantages to placing the test code in a separate module,
such as `test_widget.py':

   * The test module can be run standalone from the command line.

   * The test code can more easily be separated from shipped code.

   * There is less temptation to change test code to fit the code it
     tests without a good reason.

   * Test code should be modified much less frequently than the code it
     tests.

   * Tested code can be refactored more easily.

   * Tests for modules written in C must be in separate modules anyway,
     so why not be consistent?

   * If the testing strategy changes, there is no need to change the
     source code.

5.25.3.6 Re-using old test code
...............................

Some users will find that they have existing test code that they would
like to run from unittest, without converting every old test
function to a TestCase subclass.

  For this reason, see unittest provides a
FunctionTestCase class.  This subclass of TestCase.
can be used to wrap an existing test function.  Set-up and tear-down
functions can also be provided.

  Given the following test function:

    def testSomething():
        something = makeSomething()
        assert something.name is not None
        # ...

one can create an equivalent test case instance as follows:

    testcase = unittest.FunctionTestCase(testSomething)

If there are additional set-up and tear-down methods that should be
called as part of the test case's operation, they can also be provided
like so:

    testcase = unittest.FunctionTestCase(testSomething,
                                         setUp=makeSomethingDB,
                                         tearDown=deleteSomethingDB)

To make migrating existing test suites easier, see unittest.
supports tests raising AssertionError to indicate test
failure. However, it is recommended that you use the explicit
`TestCase.fail*()' and `TestCase.assert*()' methods instead, as future
versions of unittest.  may treat AssertionError.
differently.

     Note: Even though FunctionTestCase can be used to
     quickly convert an existing test base over to a unittest.
     -based system, this approach is not recommended.  Taking the
     time to set up proper TestCase.  subclasses will make
     future test refactorings infinitely easier.

  In some cases, the existing tests may have been written using the
doctest.  module.  If so, see doctest provides a
`DocTestSuite' class that can automatically build
unittest.TestSuite instances from the existing doctest.
-based tests.

5.25.3.7 Skipping tests and expected failures
.............................................

New in version 2.7.

  Unittest supports skipping individual test methods and even whole
classes of tests.  In addition, it supports marking a test as a
"expected failure," a test that is broken and will fail, but shouldn't
be counted as a failure on a TestResult.

  Skipping a test is simply a matter of using the skip().
See decorator.  or one of its conditional variants.

  Basic skipping looks like this:

    class MyTestCase(unittest.TestCase):

        @unittest.skip("demonstrating skipping")
        def test_nothing(self):
            self.fail("shouldn't happen")

        @unittest.skipIf(mylib.__version__ < (1, 3),
                         "not supported in this library version")
        def test_format(self):
            # Tests that work for only a certain version of the library.
            pass

        @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
        def test_windows_support(self):
            # windows specific testing code
            pass

This is the output of running the example above in verbose mode:

    test_format (__main__.MyTestCase) ... skipped 'not supported in this library version'
    test_nothing (__main__.MyTestCase) ... skipped 'demonstrating skipping'
    test_windows_support (__main__.MyTestCase) ... skipped 'requires Windows'

    ----------------------------------------------------------------------
    Ran 3 tests in 0.005s

    OK (skipped=3)

Classes can be skipped just like methods:

    @skip("showing class skipping")
    class MySkippedTestCase(unittest.TestCase):
        def test_not_run(self):
            pass

TestCase.setUp() can also skip the test.  This is useful
when a resource that needs to be set up is not available.

  Expected failures use the expectedFailure() decorator.

    class ExpectedFailureTestCase(unittest.TestCase):
        @unittest.expectedFailure
        def test_fail(self):
            self.assertEqual(1, 0, "broken")

It's easy to roll your own skipping decorators by making a decorator
that calls skip() on the test when it wants it to be
skipped.  This decorator skips the test unless the passed object has a
certain attribute:

    def skipUnlessHasattr(obj, attr):
        if hasattr(obj, attr):
            return lambda func: func
        return unittest.skip("{0!r} doesn't have {1!r}".format(obj, attr))

The following decorators implement test skipping and expected failures:

 -- Function: unittest.skip (reason)
     Unconditionally skip the decorated test.  _reason_ should describe
     why the test is being skipped.

 -- Function: unittest.skipIf (condition, reason)
     Skip the decorated test if _condition_ is true.

 -- Function: unittest.skipUnless (condition, reason)
     Skip the decorated test unless _condition_ is true.

 -- Function: unittest.expectedFailure ()
     Mark the test as an expected failure.  If the test fails when run,
     the test is not counted as a failure.

  Skipped tests will not have `setUp()' or `tearDown()' run around them.
Skipped classes will not have `setUpClass()' or `tearDownClass()' run.

5.25.3.8 Classes and functions
..............................

This section describes in depth the API of unittest.

5.25.3.9 Test cases
...................

 -- Class: unittest.TestCase ([methodName])
     Instances of the TestCase class represent the smallest
     testable units in the unittest universe.  This class
     is intended to be used as a base class, with specific tests being
     implemented by concrete subclasses.  This class implements the
     interface needed by the test runner to allow it to drive the test,
     and methods that the test code can use to check for and report
     various kinds of failure.

     Each instance of TestCase will run a single test
     method: the method named _methodName_.  If you remember, we had an
     earlier example that went something like this:

         def suite():
             suite = unittest.TestSuite()
             suite.addTest(WidgetTestCase('test_default_size'))
             suite.addTest(WidgetTestCase('test_resize'))
             return suite

     Here, we create two instances of `WidgetTestCase', each of which
     runs a single test.

     _methodName_ defaults to `runTest()'.

     TestCase instances provide three groups of methods:
     one group used to run the test, another used by the test
     implementation to check conditions and report failures, and some
     inquiry methods allowing information about the test itself to be
     gathered.

     Methods in the first group (running the test) are:

      -- Method: setUp ()
          Method called to prepare the test fixture.  This is called
          immediately before calling the test method; any exception
          raised by this method will be considered an error rather than
          a test failure. The default implementation does nothing.

      -- Method: tearDown ()
          Method called immediately after the test method has been
          called and the result recorded.  This is called even if the
          test method raised an exception, so the implementation in
          subclasses may need to be particularly careful about checking
          internal state.  Any exception raised by this method will be
          considered an error rather than a test failure.  This method
          will only be called if the setUp() succeeds,
          regardless of the outcome of the test method. The default
          implementation does nothing.

      -- Method: setUpClass ()
          A class method called before tests in an individual class run.
          `setUpClass' is called with the class as the only argument
          and must be decorated as a classmethod().:

              @classmethod
              def setUpClass(cls):
                  ...

          See Class and Module Fixtures for more details.

          New in version 2.7.

      -- Method: tearDownClass ()
          A class method called after tests in an individual class have
          run.  `tearDownClass' is called with the class as the only
          argument and must be decorated as a classmethod().:

              @classmethod
              def tearDownClass(cls):
                  ...

          See Class and Module Fixtures for more details.

          New in version 2.7.

      -- Method: run ([result])
          Run the test, collecting the result into the test result
          object passed as _result_.  If _result_ is omitted or
          None, a temporary result object is created (by calling
          the defaultTestResult() method) and used. The
          result object is not returned to run().'s caller.

          The same effect may be had by simply calling the
          TestCase.  instance.

      -- Method: skipTest (reason)
          Calling this during a test method or setUp().
          skips the current test.  See Skipping tests and expected failures
          for more information.

          New in version 2.7.

      -- Method: debug ()
          Run the test without collecting the result.  This allows
          exceptions raised by the test to be propagated to the caller,
          and can be used to support running tests under a debugger.

     The test code can use any of the following methods to check for
     and report failures.

      -- Method: assertTrue (expr[, msg])
      -- Method: assert_ (expr[, msg])
      -- Method: failUnless (expr[, msg])
          Signal a test failure if _expr_ is false; the explanation for
          the failure will be _msg_ if given, otherwise it will be
          None.

          Deprecated since version 2.7: failUnless() and
          assert_().; use assertTrue().

      -- Method: assertEqual (first, second[, msg])
      -- Method: failUnlessEqual (first, second[, msg])
          Test that _first_ and _second_ are equal.  If the values do
          not compare equal, the test will fail with the explanation
          given by _msg_, or None.  Note that using
          assertEqual() improves upon doing the comparison as the
          first parameter to assertTrue().: the default
          value for _msg_ include representations of both _first_ and
          _second_.

          In addition, if _first_ and _second_ are the exact same type
          and one of list, tuple, dict, set, frozenset or unicode or
          any type that a subclass registers with
          addTypeEqualityFunc() the type specific equality
          function will be called in order to generate a more useful
          default error message.

          Changed in version 2.7: Added the automatic calling of type
          specific equality function.

          Deprecated since version 2.7: failUnlessEqual().;
          use assertEqual().

      -- Method: assertNotEqual (first, second[, msg])
      -- Method: failIfEqual (first, second[, msg])
          Test that _first_ and _second_ are not equal.  If the values
          do compare equal, the test will fail with the explanation
          given by _msg_, or None.  Note that using
          assertNotEqual() improves upon doing the comparison as
          the first parameter to assertTrue() is that the
          default value for _msg_ can be computed to include
          representations of both _first_ and _second_.

          Deprecated since version 2.7: failIfEqual().; use
          assertNotEqual().

      -- Method: assertAlmostEqual (first, second[, places[, msg[,
               delta]]])
      -- Method: failUnlessAlmostEqual (first, second[, places[, msg[,
               delta]]])
          Test that _first_ and _second_ are approximately equal by
          computing the difference, rounding to the given number of
          decimal _places_ (default 7), and comparing to zero.

          Note that comparing a given number of decimal places is not
          the same as comparing a given number of significant digits.
          If the values do not compare equal, the test will fail with
          the explanation given by _msg_, or None.

          If _delta_ is supplied instead of _places_ then the difference
          between _first_ and _second_ must be less than _delta_.

          Supplying both _delta_ and _places_ raises a `TypeError'.

          Changed in version 2.7: Objects that compare equal are
          automatically almost equal.  Added the `delta' keyword
          argument.

          Deprecated since version 2.7: failUnlessAlmostEqual().;
          use assertAlmostEqual().

      -- Method: assertNotAlmostEqual (first, second[, places[, msg[,
               delta]]])
      -- Method: failIfAlmostEqual (first, second[, places[, msg[,
               delta]]])
          Test that _first_ and _second_ are not approximately equal by
          computing the difference, rounding to the given number of
          decimal _places_ (default 7), and comparing to zero.

          Note that comparing a given number of decimal places is not
          the same as comparing a given number of significant digits.
          If the values do not compare equal, the test will fail with
          the explanation given by _msg_, or None.

          If _delta_ is supplied instead of _places_ then the difference
          between _first_ and _second_ must be more than _delta_.

          Supplying both _delta_ and _places_ raises a `TypeError'.

          Changed in version 2.7: Objects that compare equal
          automatically fail.  Added the `delta' keyword argument.

          Deprecated since version 2.7: failIfAlmostEqual().;
          use assertNotAlmostEqual().

      -- Method: assertGreater (first, second, msg=None)
      -- Method: assertGreaterEqual (first, second, msg=None)
      -- Method: assertLess (first, second, msg=None)
      -- Method: assertLessEqual (first, second, msg=None)
          Test that _first_ is respectively >, >=, < or <= than
          _second_ depending on the method name.  If not, the test will
          fail with an explanation or with the explanation given by
          _msg_:

              >>> self.assertGreaterEqual(3, 4)
              AssertionError: "3" unexpectedly not greater than or equal to "4"

          New in version 2.7.

      -- Method: assertMultiLineEqual (self, first, second, msg=None)
          Test that the multiline string _first_ is equal to the string
          _second_.  When not equal a diff of the two strings
          highlighting the differences will be included in the error
          message. This method is used by default when comparing
          Unicode strings with assertEqual().

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertRegexpMatches (text, regexp, msg=None)
          Verifies that a _regexp_ search matches _text_.  Fails with
          an error message including the pattern and the _text_.
          _regexp_ may be a regular expression object or a string
          containing a regular expression suitable for use by
          re.search().

          New in version 2.7.

      -- Method: assertNotRegexpMatches (text, regexp, msg=None)
          Verifies that a _regexp_ search does not match _text_.  Fails
          with an error message including the pattern and the part of
          _text_ that matches.  _regexp_ may be a regular expression
          object or a string containing a regular expression suitable
          for use by re.search().

          New in version 2.7.

      -- Method: assertIn (first, second, msg=None)
      -- Method: assertNotIn (first, second, msg=None)
          Tests that _first_ is or is not in _second_ with an
          explanatory error message as appropriate.

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertItemsEqual (actual, expected, msg=None)
          Test that sequence _expected_ contains the same elements as
          _actual_, regardless of their order. When they don't, an
          error message listing the differences between the sequences
          will be generated.

          Duplicate elements are _not_ ignored when comparing _actual_
          and _expected_. It verifies if each element has the same
          count in both sequences. It is the equivalent of
          `assertEqual(sorted(expected), sorted(actual))' but it works
          with sequences of unhashable objects as well.

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertSetEqual (set1, set2, msg=None)
          Tests that two sets are equal.  If not, an error message is
          constructed that lists the differences between the sets.
          This method is used by default when comparing sets or
          frozensets with assertEqual().

          Fails if either of _set1_ or _set2_ does not have a
          set.difference().  method.

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertDictEqual (expected, actual, msg=None)
          Test that two dictionaries are equal.  If not, an error
          message is constructed that shows the differences in the
          dictionaries. This method will be used by default to compare
          dictionaries in calls to assertEqual().

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertDictContainsSubset (expected, actual, msg=None)
          Tests whether the key/value pairs in dictionary _actual_ are a
          superset of those in _expected_.  If not, an error message
          listing the missing keys and mismatched values is generated.

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertListEqual (list1, list2, msg=None)
      -- Method: assertTupleEqual (tuple1, tuple2, msg=None)
          Tests that two lists or tuples are equal.  If not an error
          message is constructed that shows only the differences
          between the two.  An error is also raised if either of the
          parameters are of the wrong type.  These methods are used by
          default when comparing lists or tuples with
          assertEqual().

          If specified, _msg_ will be used as the error message on
          failure.

          New in version 2.7.

      -- Method: assertSequenceEqual (seq1, seq2, msg=None,
               seq_type=None)
          Tests that two sequences are equal.  If a _seq_type_ is
          supplied, both _seq1_ and _seq2_ must be instances of
          _seq_type_ or a failure will be raised.  If the sequences are
          different an error message is constructed that shows the
          difference between the two.

          If specified, _msg_ will be used as the error message on
          failure.

          This method is used to implement assertListEqual()
          and assertTupleEqual().

          New in version 2.7.

      -- Method: assertRaises (exception[, callable, ...])
      -- Method: failUnlessRaises (exception[, callable, ...])
          Test that an exception is raised when _callable_ is called
          with any positional or keyword arguments that are also passed
          to assertRaises().  The test passes if _exception_
          is raised, is an error if another exception is raised, or
          fails if no exception is raised.  To catch any of a group of
          exceptions, a tuple containing the exception classes may be
          passed as _exception_.

          If _callable_ is omitted or None, returns a context manager
          so that the code under test can be written inline rather than
          as a function:

              with self.assertRaises(SomeException):
                  do_something()

          The context manager will store the caught exception object in
          its `exception' attribute.  This can be useful if the
          intention is to perform additional checks on the exception
          raised:

              with self.assertRaises(SomeException) as cm:
                  do_something()

              the_exception = cm.exception
              self.assertEqual(the_exception.error_code, 3)

          Changed in version 2.7: Added the ability to use
          assertRaises() as a context manager.

          Deprecated since version 2.7: failUnlessRaises().;
          use assertRaises().

      -- Method: assertRaisesRegexp (exception, regexp[, callable, ...])
          Like assertRaises() but also tests that _regexp_
          matches on the string representation of the raised exception.
          _regexp_ may be a regular expression object or a string
          containing a regular expression suitable for use by
          re.search().  Examples:

              self.assertRaisesRegexp(ValueError, 'invalid literal for.*XYZ$',
                                      int, 'XYZ')

          or:

              with self.assertRaisesRegexp(ValueError, 'literal'):
                 int('XYZ')

          New in version 2.7.

      -- Method: assertIsNone (expr[, msg])
          This signals a test failure if _expr_ is not None.

          New in version 2.7.

      -- Method: assertIsNotNone (expr[, msg])
          The inverse of the assertIsNone() method.  This
          signals a test failure if _expr_ is None.

          New in version 2.7.

      -- Method: assertIs (expr1, expr2[, msg])
          This signals a test failure if _expr1_ and _expr2_ don't
          evaluate to the same object.

          New in version 2.7.

      -- Method: assertIsNot (expr1, expr2[, msg])
          The inverse of the assertIs() method.  This
          signals a test failure if _expr1_ and _expr2_ evaluate to the
          same object.

          New in version 2.7.

      -- Method: assertIsInstance (obj, cls[, msg])
          This signals a test failure if _obj_ is not an instance of
          _cls_ (which can be a class or a tuple of classes, as
          supported by isinstance()).

          New in version 2.7.

      -- Method: assertNotIsInstance (obj, cls[, msg])
          The inverse of the assertIsInstance() method.
          This signals a test failure if _obj_ is an instance of _cls_.

          New in version 2.7.

      -- Method: assertFalse (expr[, msg])
      -- Method: failIf (expr[, msg])
          The inverse of the assertTrue() method is the
          assertFalse() method.  This signals a test
          failure if _expr_ is true, with _msg_ or None.
          for the error message.

          Deprecated since version 2.7: failIf().; use
          assertFalse().

      -- Method: fail ([msg])
          Signals a test failure unconditionally, with _msg_ or
          None for the error message.

      -- Attribute: failureException
          This class attribute gives the exception raised by the test
          method.  If a test framework needs to use a specialized
          exception, possibly to carry additional information, it must
          subclass this exception in order to "play fair" with the
          framework.  The initial value of this attribute is
          AssertionError.

      -- Attribute: longMessage
          If set to True then any explicit failure message you pass in
          to the assert methods will be appended to the end of the
          normal failure message.  The normal messages contain useful
          information about the objects involved, for example the
          message from assertEqual shows you the repr of the two
          unequal objects. Setting this attribute to True allows you to
          have a custom error message in addition to the normal one.

          This attribute defaults to False, meaning that a custom
          message passed to an assert method will silence the normal
          message.

          The class setting can be overridden in individual tests by
          assigning an instance attribute to True or False before
          calling the assert methods.

          New in version 2.7.

      -- Attribute: maxDiff
          This attribute controls the maximum length of diffs output by
          assert methods that report diffs on failure. It defaults to
          80*8 characters.  Assert methods affected by this attribute
          are assertSequenceEqual(). (including all the
          sequence comparison methods that delegate to it), see
          assertDictEqual() and assertMultiLineEqual().

          Setting `maxDiff' to None means that there is no maximum
          length of diffs.

          New in version 2.7.

     Testing frameworks can use the following methods to collect
     information on the test:

      -- Method: countTestCases ()
          Return the number of tests represented by this test object.
          For TestCase instances, this will always be `1'.

      -- Method: defaultTestResult ()
          Return an instance of the test result class that should be
          used for this test case class (if no other result instance is
          provided to the run() method).

          For TestCase instances, this will always be an
          instance of TestResult.; subclasses of
          TestCase should override this as necessary.

      -- Method: id ()
          Return a string identifying the specific test case.  This is
          usually the full name of the test method, including the
          module and class name.

      -- Method: shortDescription ()
          Returns a description of the test, or None if no
          description has been provided.  The default implementation of
          this method returns the first line of the test method's
          docstring, if available, or None.

      -- Method: addTypeEqualityFunc (typeobj, function)
          Registers a type specific assertEqual() equality
          checking function to be called by assertEqual().
          when both objects it has been asked to compare are exactly
          _typeobj_ (not subclasses).  _function_ must take two
          positional arguments and a third msg=None keyword argument
          just as assertEqual() does.  It must raise
          `self.failureException' when inequality between the first two
          parameters is detected.

          One good use of custom equality checking functions for a type
          is to raise `self.failureException' with an error message
          useful for debugging the problem by explaining the
          inequalities in detail.

          New in version 2.7.

      -- Method: addCleanup (function[, *args[, **kwargs]])
          Add a function to be called after tearDown() to
          cleanup resources used during the test. Functions will be
          called in reverse order to the order they are added (LIFO).
          They are called with any arguments and keyword arguments
          passed into addCleanup() when they are added.

          If setUp() fails, meaning that tearDown()
          is not called, then any cleanup functions added will
          still be called.

          New in version 2.7.

      -- Method: doCleanups ()
          This method is called unconditionally after tearDown(),
          or after setUp() if setUp().
          raises an exception.

          It is responsible for calling all the cleanup functions added
          by addCleanup(). If you need cleanup functions to
          be called _prior_ to tearDown() then you can call
          doCleanups().  yourself.

          doCleanups() pops methods off the stack of cleanup
          functions one at a time, so it can be called at any time.

          New in version 2.7.

 -- Class: unittest.FunctionTestCase (testFunc[, setUp[, tearDown[,
          description]]])
     This class implements the portion of the TestCase.
     interface which allows the test runner to drive the test, but does
     not provide the methods which test code can use to check and
     report errors.  This is used to create test cases using legacy
     test code, allowing it to be integrated into a unittest.
     -based test framework.

5.25.3.10 Grouping tests
........................

 -- Class: unittest.TestSuite ([tests])
     This class represents an aggregation of individual tests cases and
     test suites.  The class presents the interface needed by the test
     runner to allow it to be run as any other test case.  Running a
     TestSuite instance is the same as iterating over the
     suite, running each test individually.

     If _tests_ is given, it must be an iterable of individual test
     cases or other test suites that will be used to build the suite
     initially. Additional methods are provided to add test cases and
     suites to the collection later on.

     TestSuite objects behave much like TestCase
     objects, except they do not actually implement a test.
     Instead, they are used to aggregate tests into groups of tests
     that should be run together. Some additional methods are available
     to add tests to TestSuite instances:

      -- Method: addTest (test)
          Add a TestCase or TestSuite to the
          suite.

      -- Method: addTests (tests)
          Add all the tests from an iterable of TestCase.
          and TestSuite.  instances to this test suite.

          This is equivalent to iterating over _tests_, calling
          addTest() for each element.

     TestSuite shares the following methods with
     TestCase.:

      -- Method: run (result)
          Run the tests associated with this suite, collecting the
          result into the test result object passed as _result_.  Note
          that unlike TestCase.run(),
          TestSuite.run() requires the result object to be
          passed in.

      -- Method: debug ()
          Run the tests associated with this suite without collecting
          the result. This allows exceptions raised by the test to be
          propagated to the caller and can be used to support running
          tests under a debugger.

      -- Method: countTestCases ()
          Return the number of tests represented by this test object,
          including all individual tests and sub-suites.

      -- Method: __iter__ ()
          Tests grouped by a TestSuite are always accessed
          by iteration.  Subclasses can lazily provide tests by
          overriding __iter__(). Note that this method
          maybe called several times on a single suite (for example
          when counting tests or comparing for equality) so the tests
          returned must be the same for repeated iterations.

          Changed in version 2.7: In earlier versions the
          TestSuite accessed tests directly rather than through
          iteration, so overriding __iter__() wasn't
          sufficient for providing tests.

     In the typical usage of a TestSuite object, the
     run() method is invoked by a `TestRunner' rather than by
     the end-user test harness.

5.25.3.11 Loading and running tests
...................................

 -- Class: unittest.TestLoader
     The TestLoader class is used to create test suites
     from classes and modules.  Normally, there is no need to create an
     instance of this class; the unittest module provides
     an instance that can be shared as `unittest.defaultTestLoader'.
     Using a subclass or instance, however, allows customization of
     some configurable properties.

     TestLoader objects have the following methods:

      -- Method: loadTestsFromTestCase (testCaseClass)
          Return a suite of all tests cases contained in the
          TestCase.-derived `testCaseClass'.

      -- Method: loadTestsFromModule (module)
          Return a suite of all tests cases contained in the given
          module. This method searches _module_ for classes derived
          from TestCase and creates an instance of the
          class for each test method defined for the class.

               Note: While using a hierarchy of TestCase.
               -derived classes can be convenient in sharing
               fixtures and helper functions, defining test methods on
               base classes that are not intended to be instantiated
               directly does not play well with this method.  Doing so,
               however, can be useful when the fixtures are different
               and defined in subclasses.

          If a module provides a `load_tests' function it will be
          called to load the tests. This allows modules to customize
          test loading.  This is the load_tests protocol.

          Changed in version 2.7: Support for `load_tests' added.

      -- Method: loadTestsFromName (name[, module])
          Return a suite of all tests cases given a string specifier.

          The specifier _name_ is a "dotted name" that may resolve
          either to a module, a test case class, a test method within a
          test case class, a TestSuite instance, or a
          callable object which returns a TestCase or
          TestSuite instance.  These checks are applied in the
          order listed here; that is, a method on a possible test case
          class will be picked up as "a test method within a test case
          class", rather than "a callable object".

          For example, if you have a module `SampleTests' containing a
          TestCase.-derived class `SampleTestCase' with
          three test methods (`test_one()', `test_two()', and
          `test_three()'), the specifier `'SampleTests.SampleTestCase''
          would cause this method to return a suite which will run all
          three test methods. Using the specifier
          `'SampleTests.SampleTestCase.test_two'' would cause it to
          return a test suite which will run only the `test_two()' test
          method. The specifier can refer to modules and packages which
          have not been imported; they will be imported as a
          side-effect.

          The method optionally resolves _name_ relative to the given
          _module_.

      -- Method: loadTestsFromNames (names[, module])
          Similar to loadTestsFromName(), but takes a
          sequence of names rather than a single name.  The return
          value is a test suite which supports all the tests defined
          for each name.

      -- Method: getTestCaseNames (testCaseClass)
          Return a sorted sequence of method names found within
          _testCaseClass_; this should be a subclass of TestCase.

      -- Method: discover (start_dir, pattern='test*.py',
               top_level_dir=None)
          Find and return all test modules from the specified start
          directory, recursing into subdirectories to find them. Only
          test files that match _pattern_ will be loaded. (Using shell
          style pattern matching.) Only module names that are
          importable (i.e. are valid Python identifiers) will be loaded.

          All test modules must be importable from the top level of the
          project. If the start directory is not the top level
          directory then the top level directory must be specified
          separately.

          If importing a module fails, for example due to a syntax
          error, then this will be recorded as a single error and
          discovery will continue.

          If a test package name (directory with `__init__.py') matches
          the pattern then the package will be checked for a
          `load_tests' function. If this exists then it will be called
          with _loader_, _tests_, _pattern_.

          If load_tests exists then discovery does _not_ recurse into
          the package, `load_tests' is responsible for loading all
          tests in the package.

          The pattern is deliberately not stored as a loader attribute
          so that packages can continue discovery themselves.
          _top_level_dir_ is stored so `load_tests' does not need to
          pass this argument in to `loader.discover()'.

          _start_dir_ can be a dotted module name as well as a
          directory.

          New in version 2.7.

     The following attributes of a TestLoader can be
     configured either by subclassing or assignment on an instance:

      -- Attribute: testMethodPrefix
          String giving the prefix of method names which will be
          interpreted as test methods.  The default value is `'test''.

          This affects getTestCaseNames() and all the
          `loadTestsFrom*()' methods.

      -- Attribute: sortTestMethodsUsing
          Function to be used to compare method names when sorting them
          in getTestCaseNames() and all the
          `loadTestsFrom*()' methods. The default value is the built-in
          cmp() function; the attribute can also be set to
          None to disable the sort.

      -- Attribute: suiteClass
          Callable object that constructs a test suite from a list of
          tests. No methods on the resulting object are needed.  The
          default value is the TestSuite class.

          This affects all the `loadTestsFrom*()' methods.

 -- Class: unittest.TestResult
     This class is used to compile information about which tests have
     succeeded and which have failed.

     A TestResult object stores the results of a set of
     tests.  The TestCase and TestSuite classes
     ensure that results are properly recorded; test authors do not
     need to worry about recording the outcome of tests.

     Testing frameworks built on top of unittest may want
     access to the TestResult object generated by running a
     set of tests for reporting purposes; a TestResult.
     instance is returned by the `TestRunner.run()' method for this
     purpose.

     TestResult instances have the following attributes
     that will be of interest when inspecting the results of running a
     set of tests:

      -- Attribute: errors
          A list containing 2-tuples of TestCase instances
          and strings holding formatted tracebacks. Each tuple
          represents a test which raised an unexpected exception.

          Changed in version 2.2: Contains formatted tracebacks instead
          of sys.exc_info() results.

      -- Attribute: failures
          A list containing 2-tuples of TestCase instances
          and strings holding formatted tracebacks. Each tuple
          represents a test where a failure was explicitly signalled
          using the `TestCase.fail*()' or `TestCase.assert*()' methods.

          Changed in version 2.2: Contains formatted tracebacks instead
          of sys.exc_info() results.

      -- Attribute: skipped
          A list containing 2-tuples of TestCase instances
          and strings holding the reason for skipping the test.

          New in version 2.7.

      -- Attribute: expectedFailures
          A list containing 2-tuples of TestCase instances
          and strings holding formatted tracebacks.  Each tuple
          represents an expected failure of the test case.

      -- Attribute: unexpectedSuccesses
          A list containing TestCase instances that were
          marked as expected failures, but succeeded.

      -- Attribute: shouldStop
          Set to `True' when the execution of tests should stop by
          stop().

      -- Attribute: testsRun
          The total number of tests run so far.

      -- Attribute: buffer
          If set to true, `sys.stdout' and `sys.stderr' will be
          buffered in between startTest() and
          stopTest() being called. Collected output will only be
          echoed onto the real `sys.stdout' and `sys.stderr' if the test
          fails or errors. Any output is also attached to the failure /
          error message.

          New in version 2.7.

      -- Attribute: failfast
          If set to true stop() will be called on the
          first failure or error, halting the test run.

          New in version 2.7.

      -- Method: wasSuccessful ()
          Return True if all tests run so far have passed,
          otherwise returns False.

      -- Method: stop ()
          This method can be called to signal that the set of tests
          being run should be aborted by setting the shouldStop
          attribute to True.  `TestRunner' objects
          should respect this flag and return without running any
          additional tests.

          For example, this feature is used by the
          TextTestRunner class to stop the test framework when
          the user signals an interrupt from the keyboard.  Interactive
          tools which provide `TestRunner' implementations can use this
          in a similar manner.

     The following methods of the TestResult class are used
     to maintain the internal data structures, and may be extended in
     subclasses to support additional reporting requirements.  This is
     particularly useful in building tools which support interactive
     reporting while tests are being run.

      -- Method: startTest (test)
          Called when the test case _test_ is about to be run.

      -- Method: stopTest (test)
          Called after the test case _test_ has been executed,
          regardless of the outcome.

      -- Method: startTestRun (test)
          Called once before any tests are executed.

          New in version 2.7.

      -- Method: stopTestRun (test)
          Called once after all tests are executed.

          New in version 2.7.

      -- Method: addError (test, err)
          Called when the test case _test_ raises an unexpected
          exception _err_ is a tuple of the form returned by
          sys.exc_info().: `(type, value, traceback)'.

          The default implementation appends a tuple `(test,
          formatted_err)' to the instance's errors.
          attribute, where _formatted_err_ is a formatted traceback
          derived from _err_.

      -- Method: addFailure (test, err)
          Called when the test case _test_ signals a failure. _err_ is
          a tuple of the form returned by sys.exc_info().:
          `(type, value, traceback)'.

          The default implementation appends a tuple `(test,
          formatted_err)' to the instance's failures.
          attribute, where _formatted_err_ is a formatted traceback
          derived from _err_.

      -- Method: addSuccess (test)
          Called when the test case _test_ succeeds.

          The default implementation does nothing.

      -- Method: addSkip (test, reason)
          Called when the test case _test_ is skipped.  _reason_ is the
          reason the test gave for skipping.

          The default implementation appends a tuple `(test, reason)'
          to the instance's skipped attribute.

      -- Method: addExpectedFailure (test, err)
          Called when the test case _test_ fails, but was marked with
          the expectedFailure() decorator.

          The default implementation appends a tuple `(test,
          formatted_err)' to the instance's expectedFailures
          attribute, where _formatted_err_ is a formatted
          traceback derived from _err_.

      -- Method: addUnexpectedSuccess (test)
          Called when the test case _test_ was marked with the
          expectedFailure() decorator, but succeeded.

          The default implementation appends the test to the instance's
          unexpectedSuccesses attribute.

 -- Class: unittest.TextTestResult (stream, descriptions, verbosity)
     A concrete implementation of TestResult used by the
     TextTestRunner.

     New in version 2.7: This class was previously named
     `_TextTestResult'. The old name still exists as an alias but is
     deprecated.

 -- Data: unittest.defaultTestLoader
     Instance of the TestLoader class intended to be
     shared.  If no customization of the TestLoader is
     needed, this instance can be used instead of repeatedly creating
     new instances.

 -- Class: unittest.TextTestRunner ([stream[, descriptions[,
          verbosity][, resultclass]]])
     A basic test runner implementation which prints results on
     standard error.  It has a few configurable parameters, but is
     essentially very simple.  Graphical applications which run test
     suites should provide alternate implementations.

      -- Method: _makeResult ()
          This method returns the instance of `TestResult' used by
          `run()'.  It is not intended to be called directly, but can
          be overridden in subclasses to provide a custom `TestResult'.

          `_makeResult()' instantiates the class or callable passed in
          the `TextTestRunner' constructor as the `resultclass'
          argument. It defaults to TextTestResult if no
          `resultclass' is provided.  The result class is instantiated
          with the following arguments:

              stream, descriptions, verbosity

 -- Function: unittest.main ([module[, defaultTest[, argv[,
          testRunner[, testLoader[, exit[, verbosity[, failfast[,
          catchbreak[, buffer]]]]]]]]]])
     A command-line program that runs a set of tests; this is primarily
     for making test modules conveniently executable.  The simplest use
     for this function is to include the following line at the end of a
     test script:

         if __name__ == '__main__':
             unittest.main()

     You can run tests with more detailed information by passing in the
     verbosity argument:

         if __name__ == '__main__':
             unittest.main(verbosity=2)

     The _testRunner_ argument can either be a test runner class or an
     already created instance of it. By default `main' calls
     sys.exit() with an exit code indicating success or failure
     of the tests run.

     `main' supports being used from the interactive interpreter by
     passing in the argument `exit=False'. This displays the result on
     standard output without calling sys.exit().:

         >>> from unittest import main
         >>> main(module='test_module', exit=False)

     The `failfast', `catchbreak' and `buffer' parameters have the same
     effect as the failfast, catch and buffer command line options.

     Calling `main' actually returns an instance of the `TestProgram'
     class.  This stores the result of the tests run as the `result'
     attribute.

     Changed in version 2.7: The `exit', `verbosity', `failfast',
     `catchbreak' and `buffer' parameters were added.

5.25.3.12 load_tests Protocol
.............................

New in version 2.7.

  Modules or packages can customize how tests are loaded from them
during normal test runs or test discovery by implementing a function
called `load_tests'.

  If a test module defines `load_tests' it will be called by
TestLoader.loadTestsFromModule() with the following arguments:

    load_tests(loader, standard_tests, None)

It should return a TestSuite.

  _loader_ is the instance of TestLoader doing the loading.
_standard_tests_ are the tests that would be loaded by default from the
module. It is common for test modules to only want to add or remove
tests from the standard set of tests.  The third argument is used when
loading packages as part of test discovery.

  A typical `load_tests' function that loads tests from a specific set
of TestCase classes may look like:

    test_cases = (TestCase1, TestCase2, TestCase3)

    def load_tests(loader, tests, pattern):
        suite = TestSuite()
        for test_class in test_cases:
            tests = loader.loadTestsFromTestCase(test_class)
            suite.addTests(tests)
        return suite

If discovery is started, either from the command line or by calling
TestLoader.discover(), with a pattern that matches a package
name then the package `__init__.py' will be checked for `load_tests'.

     Note: The default pattern is 'test*.py'. This matches all Python
     files that start with 'test' but _won't_ match any test
     directories.

     A pattern like 'test*' will match test packages as well as modules.

  If the package `__init__.py' defines `load_tests' then it will be
called and discovery not continued into the package. `load_tests' is
called with the following arguments:

    load_tests(loader, standard_tests, pattern)

This should return a TestSuite representing all the tests
from the package. (`standard_tests' will only contain tests collected
from `__init__.py'.)

  Because the pattern is passed into `load_tests' the package is free to
continue (and potentially modify) test discovery. A 'do nothing'
`load_tests' function for a test package would look like:

    def load_tests(loader, standard_tests, pattern):
        # top level directory cached on loader instance
        this_dir = os.path.dirname(__file__)
        package_tests = loader.discover(start_dir=this_dir, pattern=pattern)
        standard_tests.addTests(package_tests)
        return standard_tests

5.25.3.13 Class and Module Fixtures
...................................

Class and module level fixtures are implemented in TestSuite.
When the test suite encounters a test from a new class then
`tearDownClass()' from the previous class (if there is one) is called,
followed by `setUpClass()' from the new class.

  Similarly if a test is from a different module from the previous test
then `tearDownModule' from the previous module is run, followed by
`setUpModule' from the new module.

  After all the tests have run the final `tearDownClass' and
`tearDownModule' are run.

  Note that shared fixtures do not play well with [potential] features
like test parallelization and they break test isolation. They should be
used with care.

  The default ordering of tests created by the unittest test loaders is
to group all tests from the same modules and classes together. This
will lead to `setUpClass' / `setUpModule' (etc) being called exactly
once per class and module. If you randomize the order, so that tests
from different modules and classes are adjacent to each other, then
these shared fixture functions may be called multiple times in a single
test run.

  Shared fixtures are not intended to work with suites with non-standard
ordering. A `BaseTestSuite' still exists for frameworks that don't want
to support shared fixtures.

  If there are any exceptions raised during one of the shared fixture
functions the test is reported as an error. Because there is no
corresponding test instance an `_ErrorHolder' object (that has the same
interface as a TestCase) is created to represent the error.
If you are just using the standard unittest test runner then this
detail doesn't matter, but if you are a framework author it may be
relevant.

5.25.3.14 setUpClass and tearDownClass
......................................

These must be implemented as class methods:

    import unittest

    class Test(unittest.TestCase):
        @classmethod
        def setUpClass(cls):
            cls._connection = createExpensiveConnectionObject()

        @classmethod
        def tearDownClass(cls):
            cls._connection.destroy()

If you want the `setUpClass' and `tearDownClass' on base classes called
then you must call up to them yourself. The implementations in
TestCase are empty.

  If an exception is raised during a `setUpClass' then the tests in the
class are not run and the `tearDownClass' is not run. Skipped classes
will not have `setUpClass' or `tearDownClass' run. If the exception is a
`SkipTest' exception then the class will be reported as having been
skipped instead of as an error.

5.25.3.15 setUpModule and tearDownModule
........................................

These should be implemented as functions:

    def setUpModule():
        createConnection()

    def tearDownModule():
        closeConnection()

If an exception is raised in a `setUpModule' then none of the tests in
the module will be run and the `tearDownModule' will not be run. If the
exception is a `SkipTest' exception then the module will be reported as
having been skipped instead of as an error.

5.25.3.16 Signal Handling
.........................

The -c./`--catch' command line option to unittest, along
with the `catchbreak' parameter to unittest.main(), provide
more friendly handling of control-C during a test run. With catch break
behavior enabled control-C will allow the currently running test to
complete, and the test run will then end and report all the results so
far. A second control-c will raise a KeyboardInterrupt in
the usual way.

  The control-c handling signal handler attempts to remain compatible
with code or tests that install their own `signal.SIGINT' handler. If
the `unittest' handler is called but _isn't_ the installed
`signal.SIGINT' handler, i.e. it has been replaced by the system under
test and delegated to, then it calls the default handler. This will
normally be the expected behavior by code that replaces an installed
handler and delegates to it. For individual tests that need `unittest'
control-c handling disabled the removeHandler().  decorator
can be used.

  There are a few utility functions for framework authors to enable
control-c handling functionality within test frameworks.

 -- Function: unittest.installHandler ()
     Install the control-c handler. When a `signal.SIGINT' is received
     (usually in response to the user pressing control-c) all
     registered results have stop() called.

     New in version 2.7.

 -- Function: unittest.registerResult (result)
     Register a TestResult object for control-c handling.
     Registering a result stores a weak reference to it, so it doesn't
     prevent the result from being garbage collected.

     Registering a TestResult object has no side-effects if
     control-c handling is not enabled, so test frameworks can
     unconditionally register all results they create independently of
     whether or not handling is enabled.

     New in version 2.7.

 -- Function: unittest.removeResult (result)
     Remove a registered result. Once a result has been removed then
     stop() will no longer be called on that result object
     in response to a control-c.

     New in version 2.7.

 -- Function: unittest.removeHandler (function=None)
     When called without arguments this function removes the control-c
     handler if it has been installed. This function can also be used
     as a test decorator to temporarily remove the handler whilst the
     test is being executed:

         @unittest.removeHandler
         def test_signal_handling(self):
             ...

     New in version 2.7.

5.25.4 2to3 - Automated Python 2 to 3 code translation
------------------------------------------------------

2to3 is a Python program that reads Python 2.x source code and applies
a series of _fixers_ to transform it into valid Python 3.x code.  The
standard library contains a rich set of fixers that will handle almost
all code.  2to3 supporting library lib2to3 is, however, a
flexible and generic library, so it is possible to write your own
fixers for 2to3.  lib2to3 could also be adapted to custom
applications in which Python code needs to be edited automatically.

5.25.4.1 Using 2to3
...................

2to3 will usually be installed with the Python interpreter as a script.
It is also located in the `Tools/scripts' directory of the Python root.

  2to3's basic arguments are a list of files or directories to
transform.  The directories are to recursively traversed for Python
sources.

  Here is a sample Python 2.x source file, `example.py':

    def greet(name):
        print "Hello, {0}!".format(name)
    print "What's your name?"
    name = raw_input()
    greet(name)

It can be converted to Python 3.x code via 2to3 on the command line:

    $ 2to3 example.py

A diff against the original source file is printed.  2to3 can also
write the needed modifications right back to the source file.  (A
backup of the original file is made unless `-n' is also given.)
Writing the changes back is enabled with the `-w' flag:

    $ 2to3 -w example.py

After transformation, `example.py' looks like this:

    def greet(name):
        print("Hello, {0}!".format(name))
    print("What's your name?")
    name = input()
    greet(name)

Comments and exact indentation are preserved throughout the translation
process.

  By default, 2to3 runs a set of predefined fixers.  The
`-l' flag lists all available fixers.  An explicit set of fixers to run
can be given with `-f'.  Likewise the -x explicitly
disables a fixer.  The following example runs only the `imports' and
`has_key' fixers:

    $ 2to3 -f imports -f has_key example.py

This command runs every fixer except the `apply' fixer:

    $ 2to3 -x apply example.py

Some fixers are _explicit_, meaning they aren't run by default and must
be listed on the command line to be run.  Here, in addition to the
default fixers, the `idioms' fixer is run:

    $ 2to3 -f all -f idioms example.py

Notice how passing `all' enables all default fixers.

  Sometimes 2to3 will find a place in your source code that needs to be
changed, but 2to3 cannot fix automatically.  In this case, 2to3 will
print a warning beneath the diff for a file.  You should address the
warning in order to have compliant 3.x code.

  2to3 can also refactor doctests.  To enable this mode, use the
-d.  flag.  Note that _only_ doctests will be refactored.  This
also doesn't require the module to be valid Python.  For example,
doctest like examples in a reST document could also be refactored with
this option.

  The -v option enables output of more information on the
translation process.

  Since some print statements can be parsed as function calls or
statements, 2to3 cannot always read files containing the print
function.  When 2to3 detects the presence of the `from __future__
import print_function' compiler directive, it modifies its internal
grammar to interpret print() as a function.  This change
can also be enabled manually with the `-p' flag.  Use `-p' to run
fixers on code that already has had its print statements converted.

5.25.4.2 Fixers
...............

Each step of transforming code is encapsulated in a fixer.  The command
`2to3 -l' lists them.  As documented above, each can be
turned on and off individually.  They are described here in more detail.

 -- 2to3fixer: apply
     Removes usage of apply().  For example `apply(function,
     *args, **kwargs)' is converted to `function(*args, **kwargs)'.

 -- 2to3fixer: basestring
     Converts basestring to str.

 -- 2to3fixer: buffer
     Converts buffer to memoryview.  This fixer
     is optional because the memoryview. API is similar but
     not exactly the same as that of buffer.

 -- 2to3fixer: callable
     Converts `callable(x)' to `isinstance(x, collections.Callable)',
     adding an import to collections if needed.

 -- 2to3fixer: dict
     Fixes dictionary iteration methods.  See dict.iteritems().
     is converted to dict.items(), dict.iterkeys()
     to dict.keys(), and dict.itervalues().
     to dict.values().  Similarly, see dict.viewitems(),
     see dict.viewkeys() and dict.viewvalues().
     are converted respectively to dict.items(),
     dict.keys() and dict.values().  It also wraps
     existing usages of dict.items(), dict.keys(),
     and dict.values() in a call to list.

 -- 2to3fixer: except
     Converts `except X, T' to `except X as T'.

 -- 2to3fixer: exec
     Converts the exec statement to the `exec()' function.

 -- 2to3fixer: execfile
     Removes usage of execfile().  The argument to
     execfile() is wrapped in calls to open(),
     compile(), and `exec()'.

 -- 2to3fixer: exitfunc
     Changes assignment of sys.exitfunc to use of the
     atexit.  module.

 -- 2to3fixer: filter
     Wraps filter() usage in a list call.

 -- 2to3fixer: funcattrs
     Fixes function attributes that have been renamed.  For example,
     `my_function.func_closure' is converted to
     `my_function.__closure__'.

 -- 2to3fixer: future
     Removes `from __future__ import new_feature' statements.

 -- 2to3fixer: getcwdu
     Renames os.getcwdu() to os.getcwd().

 -- 2to3fixer: has_key
     Changes `dict.has_key(key)' to `key in dict'.

 -- 2to3fixer: idioms
     This optional fixer performs several transformations that make
     Python code more idiomatic.  Type comparisons like `type(x) is
     SomeClass' and `type(x) == SomeClass' are converted to
     `isinstance(x, SomeClass)'.  `while 1' becomes `while True'.  This
     fixer also tries to make use of sorted() in
     appropriate places.  For example, this block

         L = list(some_iterable)
         L.sort()

     is changed to

         L = sorted(some_iterable)

 -- 2to3fixer: import
     Detects sibling imports and converts them to relative imports.

 -- 2to3fixer: imports
     Handles module renames in the standard library.

 -- 2to3fixer: imports2
     Handles other modules renames in the standard library.  It is
     separate from the imports fixer only because of
     technical limitations.

 -- 2to3fixer: input
     Converts `input(prompt)' to `eval(input(prompt))'

 -- 2to3fixer: intern
     Converts intern() to `sys.intern()'.

 -- 2to3fixer: isinstance
     Fixes duplicate types in the second argument of
     isinstance().  For example, `isinstance(x, (int, int))' is
     converted to `isinstance(x, (int))'.

 -- 2to3fixer: itertools_imports
     Removes imports of itertools.ifilter(),
     itertools.izip(), and itertools.imap().  Imports
     of itertools.ifilterfalse() are also changed to
     `itertools.filterfalse()'.

 -- 2to3fixer: itertools
     Changes usage of itertools.ifilter(),
     itertools.izip(), and itertools.imap() to their
     built-in equivalents.  itertools.ifilterfalse() is
     changed to `itertools.filterfalse()'.

 -- 2to3fixer: long
     Strips the `L' prefix on long literals and renames long
     to int.

 -- 2to3fixer: map
     Wraps map() in a list call.  It also
     changes `map(None, x)' to `list(x)'.  Using `from future_builtins
     import map' disables this fixer.

 -- 2to3fixer: metaclass
     Converts the old metaclass syntax (`__metaclass__ = Meta' in the
     class body) to the new (`class X(metaclass=Meta)').

 -- 2to3fixer: methodattrs
     Fixes old method attribute names.  For example, `meth.im_func' is
     converted to `meth.__func__'.

 -- 2to3fixer: ne
     Converts the old not-equal syntax, `<>', to `!='.

 -- 2to3fixer: next
     Converts the use of iterator's next() methods to the
     next() function.  It also renames next().
     methods to `__next__()'.

 -- 2to3fixer: nonzero
     Renames __nonzero__() to `__bool__()'.

 -- 2to3fixer: numliterals
     Converts octal literals into the new syntax.

 -- 2to3fixer: paren
     Add extra parenthesis where they are required in list
     comprehensions.  For example, `[x for x in 1, 2]' becomes `[x for
     x in (1, 2)]'.

 -- 2to3fixer: print
     Converts the print statement to the print()
     function.

 -- 2to3fixer: raise
     Converts `raise E, V' to `raise E(V)', and `raise E, V, T' to
     `raise E(V).with_traceback(T)'.  If `E' is a tuple, the
     translation will be incorrect because substituting tuples for
     exceptions has been removed in 3.0.

 -- 2to3fixer: raw_input
     Converts raw_input() to input().

 -- 2to3fixer: reduce
     Handles the move of reduce() to
     functools.reduce().

 -- 2to3fixer: renames
     Changes sys.maxint to sys.maxsize.

 -- 2to3fixer: repr
     Replaces backtick repr with the repr() function.

 -- 2to3fixer: set_literal
     Replaces use of the set constructor with set literals.
     This fixer is optional.

 -- 2to3fixer: standard_error
     Renames StandardError to Exception.

 -- 2to3fixer: sys_exc
     Changes the deprecated sys.exc_value,
     sys.exc_type, sys.exc_traceback to use
     sys.exc_info().

 -- 2to3fixer: throw
     Fixes the API change in generator's `throw()' method.

 -- 2to3fixer: tuple_params
     Removes implicit tuple parameter unpacking.  This fixer inserts
     temporary variables.

 -- 2to3fixer: types
     Fixes code broken from the removal of some members in the
     types.  module.

 -- 2to3fixer: unicode
     Renames unicode to str.

 -- 2to3fixer: urllib
     Handles the rename of urllib and urllib2.
     to the urllib.  package.

 -- 2to3fixer: ws_comma
     Removes excess whitespace from comma separated items.  This fixer
     is optional.

 -- 2to3fixer: xrange
     Renames xrange() to range() and wraps
     existing range().  calls with list.

 -- 2to3fixer: xreadlines
     Changes `for x in file.xreadlines()' to `for x in file'.

 -- 2to3fixer: zip
     Wraps zip() usage in a list call.  This is
     disabled when `from future_builtins import zip' appears.

5.25.4.3 `lib2to3' - 2to3's library
...................................

     Note: The lib2to3. API should be considered unstable and
     may change drastically in the future.

5.25.5 `test' -- Regression tests package for Python
----------------------------------------------------

     Note: The test package is meant for internal use by
     Python only. It is documented for the benefit of the core
     developers of Python. Any use of this package outside of Python's
     standard library is discouraged as code mentioned here can change
     or be removed without notice between releases of Python.

The test package contains all regression tests for Python
as well as the modules test.test_support and
`test.regrtest'.  test.test_support is used to enhance your
tests while `test.regrtest' drives the testing suite.

  Each module in the test package whose name starts with
`test_' is a testing suite for a specific module or feature. All new
tests should be written using the unittest or
doctest module.  Some older tests are written using a
"traditional" testing style that compares output printed to
`sys.stdout'; this style of test is considered deprecated.

Module unittest.
     Writing PyUnit regression tests.

Module doctest.
     Tests embedded in documentation strings.

5.25.5.1 Writing Unit Tests for the `test' package
..................................................

It is preferred that tests that use the unittest module
follow a few guidelines. One is to name the test module by starting it
with `test_' and end it with the name of the module being tested. The
test methods in the test module should start with `test_' and end with
a description of what the method is testing. This is needed so that the
methods are recognized by the test driver as test methods. Also, no
documentation string for the method should be included. A comment (such
as `# Tests function returns only True or False') should be used to
provide documentation for test methods. This is done because
documentation strings get printed out if they exist and thus what test
is being run is not stated.

  A basic boilerplate is often used:

    import unittest
    from test import test_support

    class MyTestCase1(unittest.TestCase):

        # Only use setUp() and tearDown() if necessary

        def setUp(self):
            ... code to execute in preparation for tests ...

        def tearDown(self):
            ... code to execute to clean up after tests ...

        def test_feature_one(self):
            # Test feature one.
            ... testing code ...

        def test_feature_two(self):
            # Test feature two.
            ... testing code ...

        ... more test methods ...

    class MyTestCase2(unittest.TestCase):
        ... same structure as MyTestCase1 ...

    ... more test classes ...

    def test_main():
        test_support.run_unittest(MyTestCase1,
                                  MyTestCase2,
                                  ... list other tests ...
                                 )

    if __name__ == '__main__':
        test_main()

This boilerplate code allows the testing suite to be run by
`test.regrtest' as well as on its own as a script.

  The goal for regression testing is to try to break code. This leads
to a few guidelines to be followed:

   * The testing suite should exercise all classes, functions, and
     constants. This includes not just the external API that is to be
     presented to the outside world but also "private" code.

   * Whitebox testing (examining the code being tested when the tests
     are being written) is preferred. Blackbox testing (testing only
     the published user interface) is not complete enough to make sure
     all boundary and edge cases are tested.

   * Make sure all possible values are tested including invalid ones.
     This makes sure that not only all valid values are acceptable but
     also that improper values are handled correctly.

   * Exhaust as many code paths as possible. Test where branching
     occurs and thus tailor input to make sure as many different paths
     through the code are taken.

   * Add an explicit test for any bugs discovered for the tested code.
     This will make sure that the error does not crop up again if the
     code is changed in the future.

   * Make sure to clean up after your tests (such as close and remove
     all temporary files).

   * If a test is dependent on a specific condition of the operating
     system then verify the condition already exists before attempting
     the test.

   * Import as few modules as possible and do it as soon as possible.
     This minimizes external dependencies of tests and also minimizes
     possible anomalous behavior from side-effects of importing a
     module.

   * Try to maximize code reuse. On occasion, tests will vary by
     something as small as what type of input is used. Minimize code
     duplication by subclassing a basic test class with a class that
     specifies the input:

         class TestFuncAcceptsSequences(unittest.TestCase):

             func = mySuperWhammyFunction

             def test_func(self):
                 self.func(self.arg)

         class AcceptLists(TestFuncAcceptsSequences):
             arg = [1, 2, 3]

         class AcceptStrings(TestFuncAcceptsSequences):
             arg = 'abc'

         class AcceptTuples(TestFuncAcceptsSequences):
             arg = (1, 2, 3)

Test Driven Development
     A book by Kent Beck on writing tests before code.

5.25.5.2 Running tests using `test.regrtest'
............................................

`test.regrtest' can be used as a script to drive Python's regression
test suite. Running the script by itself automatically starts running
all regression tests in the test package. It does this by
finding all modules in the package whose name starts with `test_',
importing them, and executing the function `test_main()' if present.
The names of tests to execute may also be passed to the script.
Specifying a single regression test (*python regrtest.py*
`test_spam.py') will minimize output and only print whether the test
passed or failed and thus minimize output.

  Running `test.regrtest' directly allows what resources are available
for tests to use to be set. You do this by using the -u.
command-line option. Run *python regrtest.py* `-uall' to turn on all
resources; specifying `all' as an option for -u enables all
possible resources. If all but one resource is desired (a more common
case), a comma-separated list of resources that are not desired may be
listed after `all'. The command *python regrtest.py*
`-uall,-audio,-largefile' will run `test.regrtest' with all resources
except the `audio' and `largefile' resources. For a list of all
resources and more command-line options, run *python regrtest.py* See
-h.

  Some other ways to execute the regression tests depend on what
platform the tests are being executed on. On Unix, you can run *make*
`test' at the top-level directory where Python was built. On Windows,
executing *rt.bat* from your `PCBuild' directory will run all
regression tests.

5.25.6 `test.test_support' -- Utility functions for tests
---------------------------------------------------------

     Note: The test.test_support module has been renamed to
     `test.support' in Python 3.x.

The test.test_support module provides support for Python's
regression tests.

  This module defines the following exceptions:

 -- Exception: test.test_support.TestFailed
     Exception to be raised when a test fails. This is deprecated in
     favor of unittest.-based tests and
     unittest.TestCase.'s assertion methods.

 -- Exception: test.test_support.ResourceDenied
     Subclass of `unittest.SkipTest'. Raised when a resource (such as a
     network connection) is not available. Raised by the
     requires().  function.

  The test.test_support module defines the following
constants:

 -- Data: test.test_support.verbose
     True when verbose output is enabled. Should be checked
     when more detailed information is desired about a running test.
     _verbose_ is set by `test.regrtest'.

 -- Data: test.test_support.have_unicode
     True when Unicode support is available.

 -- Data: test.test_support.is_jython
     True if the running interpreter is Jython.

 -- Data: test.test_support.TESTFN
     Set to a name that is safe to use as the name of a temporary file.
     Any temporary file that is created should be closed and unlinked
     (removed).

  The test.test_support module defines the following
functions:

 -- Function: test.test_support.forget (module_name)
     Remove the module named _module_name_ from `sys.modules' and
     delete any byte-compiled files of the module.

 -- Function: test.test_support.is_resource_enabled (resource)
     Return True if _resource_ is enabled and available.
     The list of available resources is only set when `test.regrtest'
     is executing the tests.

 -- Function: test.test_support.requires (resource[, msg])
     Raise ResourceDenied if _resource_ is not available.
     _msg_ is the argument to ResourceDenied if it is
     raised. Always returns True if called by a function
     whose `__name__' is `'__main__''.  Used when tests are executed by
     `test.regrtest'.

 -- Function: test.test_support.findfile (filename)
     Return the path to the file named _filename_. If no match is found
     _filename_ is returned. This does not equal a failure since it
     could be the path to the file.

 -- Function: test.test_support.run_unittest (*classes)
     Execute unittest.TestCase subclasses passed to the
     function. The function scans the classes for methods starting with
     the prefix `test_' and executes the tests individually.

     It is also legal to pass strings as parameters; these should be
     keys in `sys.modules'. Each associated module will be scanned by
     `unittest.TestLoader.loadTestsFromModule()'. This is usually seen
     in the following `test_main()' function:

         def test_main():
             test_support.run_unittest(__name__)

     This will run all tests defined in the named module.

 -- Function: test.test_support.check_warnings (*filters, quiet=True)
     A convenience wrapper for warnings.catch_warnings().
     that makes it easier to test that a warning was correctly raised.
     It is approximately equivalent to calling
     `warnings.catch_warnings(record=True)' with
     warnings.simplefilter() set to `always' and with the option
     to automatically validate the results that are recorded.

     `check_warnings' accepts 2-tuples of the form `("message regexp",
     WarningCategory)' as positional arguments. If one or more
     _filters_ are provided, or if the optional keyword argument
     _quiet_ is False, it checks to make sure the warnings
     are as expected:  each specified filter must match at least one of
     the warnings raised by the enclosed code or the test fails, and if
     any warnings are raised that do not match any of the specified
     filters the test fails.  To disable the first of these checks, set
     _quiet_ to True.

     If no arguments are specified, it defaults to:

         check_warnings(("", Warning), quiet=True)

     In this case all warnings are caught and no errors are raised.

     On entry to the context manager, a `WarningRecorder' instance is
     returned. The underlying warnings list from
     catch_warnings() is available via the recorder object's
     warnings attribute.  As a convenience, the attributes
     of the object representing the most recent warning can also be
     accessed directly through the recorder object (see example below).
     If no warning has been raised, then any of the attributes that
     would otherwise be expected on an object representing a warning
     will return None.

     The recorder object also has a `reset()' method, which clears the
     warnings list.

     The context manager is designed to be used like this:

         with check_warnings(("assertion is always true", SyntaxWarning),
                             ("", UserWarning)):
             exec('assert(False, "Hey!")')
             warnings.warn(UserWarning("Hide me!"))

     In this case if either warning was not raised, or some other
     warning was raised, see check_warnings() would raise an
     error.

     When a test needs to look more deeply into the warnings, rather
     than just checking whether or not they occurred, code like this
     can be used:

         with check_warnings(quiet=True) as w:
             warnings.warn("foo")
             assert str(w.args[0]) == "foo"
             warnings.warn("bar")
             assert str(w.args[0]) == "bar"
             assert str(w.warnings[0].args[0]) == "foo"
             assert str(w.warnings[1].args[0]) == "bar"
             w.reset()
             assert len(w.warnings) == 0

     Here all warnings will be caught, and the test code tests the
     captured warnings directly.

     New in version 2.6.

     Changed in version 2.7: New optional arguments _filters_ and
     _quiet_.

 -- Function: test.test_support.check_py3k_warnings (*filters,
          quiet=False)
     Similar to check_warnings(), but for Python 3
     compatibility warnings.  If `sys.py3kwarning == 1', it checks if
     the warning is effectively raised.  If `sys.py3kwarning == 0', it
     checks that no warning is raised.  It accepts 2-tuples of the form
     `("message regexp", WarningCategory)' as positional arguments.
     When the optional keyword argument _quiet_ is True, it
     does not fail if a filter catches nothing.  Without arguments, it
     defaults to:

         check_py3k_warnings(("", DeprecationWarning), quiet=False)

     New in version 2.7.

 -- Function: test.test_support.captured_stdout ()
     This is a context manager that runs the with statement
     body using a StringIO.StringIO object as sys.stdout.
     That object can be retrieved using the `as' clause of the
     with statement.

     Example use:

         with captured_stdout() as s:
             print "hello"
         assert s.getvalue() == "hello"

     New in version 2.6.

 -- Function: test.test_support.import_module (name, deprecated=False)
     This function imports and returns the named module. Unlike a normal
     import, this function raises `unittest.SkipTest' if the module
     cannot be imported.

     Module and package deprecation messages are suppressed during this
     import if _deprecated_ is True.

     New in version 2.7.

 -- Function: test.test_support.import_fresh_module (name, fresh=(),
          blocked=(), deprecated=False)
     This function imports and returns a fresh copy of the named Python
     module by removing the named module from `sys.modules' before
     doing the import.  Note that unlike reload(), the
     original module is not affected by this operation.

     _fresh_ is an iterable of additional module names that are also
     removed from the `sys.modules' cache before doing the import.

     _blocked_ is an iterable of module names that are replaced with `0'
     in the module cache during the import to ensure that attempts to
     import them raise ImportError.

     The named module and any modules named in the _fresh_ and _blocked_
     parameters are saved before starting the import and then
     reinserted into `sys.modules' when the fresh import is complete.

     Module and package deprecation messages are suppressed during this
     import if _deprecated_ is True.

     This function will raise `unittest.SkipTest' is the named module
     cannot be imported.

     Example use:

         # Get copies of the warnings module for testing without
         # affecting the version being used by the rest of the test suite
         # One copy uses the C implementation, the other is forced to use
         # the pure Python fallback implementation
         py_warnings = import_fresh_module('warnings', blocked=['_warnings'])
         c_warnings = import_fresh_module('warnings', fresh=['_warnings'])

     New in version 2.7.

  The test.test_support module defines the following
classes:

 -- Class: test.test_support.TransientResource (exc[, **kwargs])
     Instances are a context manager that raises ResourceDenied
     if the specified exception type is raised.  Any keyword
     arguments are treated as attribute/value pairs to be compared
     against any exception raised within the with.
     statement.  Only if all pairs match properly against attributes on
     the exception is ResourceDenied raised.

     New in version 2.6.

 -- Class: test.test_support.EnvironmentVarGuard
     Class used to temporarily set or unset environment variables.
     Instances can be used as a context manager and have a complete
     dictionary interface for querying/modifying the underlying
     `os.environ'. After exit from the context manager all changes to
     environment variables done through this instance will be rolled
     back.

     New in version 2.6.

     Changed in version 2.7: Added dictionary interface.

 -- Method: EnvironmentVarGuard.set (envvar, value)
     Temporarily set the environment variable `envvar' to the value of
     `value'.

 -- Method: EnvironmentVarGuard.unset (envvar)
     Temporarily unset the environment variable `envvar'.

 -- Class: test.test_support.WarningsRecorder
     Class used to record warnings for unit tests. See documentation of
     check_warnings() above for more details.

     New in version 2.6.

5.26 Debugging and Profiling
============================

These libraries help you with Python development: the debugger enables
you to step through code, analyze stack frames and set breakpoints
etc., and the profilers run code and give you a detailed breakdown of
execution times, allowing you to identify bottlenecks in your programs.

5.26.1 `bdb' -- Debugger framework
----------------------------------

The bdb module handles basic debugger functions, like
setting breakpoints or managing execution via the debugger.

  The following exception is defined:

 -- Exception: bdb.BdbQuit
     Exception raised by the Bdb class for quitting the
     debugger.

  The bdb module also defines two classes:

 -- Class: bdb.Breakpoint (self, file, line[, temporary=0[, cond=None[,
          funcname=None]]])
     This class implements temporary breakpoints, ignore counts,
     disabling and (re-)enabling, and conditionals.

     Breakpoints are indexed by number through a list called
     `bpbynumber' and by `(file, line)' pairs through `bplist'.  The
     former points to a single instance of class Breakpoint.
     The latter points to a list of such instances since there
     may be more than one breakpoint per line.

     When creating a breakpoint, its associated filename should be in
     canonical form.  If a _funcname_ is defined, a breakpoint hit will
     be counted when the first line of that function is executed.  A
     conditional breakpoint always counts a hit.

     Breakpoint instances have the following methods:

      -- Method: deleteMe ()
          Delete the breakpoint from the list associated to a
          file/line.  If it is the last breakpoint in that position, it
          also deletes the entry for the file/line.

      -- Method: enable ()
          Mark the breakpoint as enabled.

      -- Method: disable ()
          Mark the breakpoint as disabled.

      -- Method: pprint ([out])
          Print all the information about the breakpoint:

             * The breakpoint number.

             * If it is temporary or not.

             * Its file,line position.

             * The condition that causes a break.

             * If it must be ignored the next N times.

             * The breakpoint hit count.

 -- Class: bdb.Bdb (skip=None)
     The Bdb class acts as a generic Python debugger base
     class.

     This class takes care of the details of the trace facility; a
     derived class should implement user interaction.  The standard
     debugger class (see pdb.Pdb) is an example.

     The _skip_ argument, if given, must be an iterable of glob-style
     module name patterns.  The debugger will not step into frames that
     originate in a module that matches one of these patterns. Whether a
     frame is considered to originate in a certain module is determined
     by the `__name__' in the frame globals.

     New in version 2.7: The _skip_ argument.

     The following methods of Bdb normally don't need to be
     overridden.

      -- Method: canonic (filename)
          Auxiliary method for getting a filename in a canonical form,
          that is, as a case-normalized (on case-insensitive
          filesystems) absolute path, stripped of surrounding angle
          brackets.

      -- Method: reset ()
          Set the `botframe', `stopframe', `returnframe' and `quitting'
          attributes with values ready to start debugging.

      -- Method: trace_dispatch (frame, event, arg)
          This function is installed as the trace function of debugged
          frames.  Its return value is the new trace function (in most
          cases, that is, itself).

          The default implementation decides how to dispatch a frame,
          depending on the type of event (passed as a string) that is
          about to be executed.  _event_ can be one of the following:

             * `"line"': A new line of code is going to be executed.

             * `"call"': A function is about to be called, or another
               code block entered.

             * `"return"': A function or other code block is about to
               return.

             * `"exception"': An exception has occurred.

             * `"c_call"': A C function is about to be called.

             * `"c_return"': A C function has returned.

             * `"c_exception"': A C function has raised an exception.

          For the Python events, specialized functions (see below) are
          called.  For the C events, no action is taken.

          The _arg_ parameter depends on the previous event.

          See the documentation for sys.settrace() for more
          information on the trace function.  For more information on
          code and frame objects, refer to The standard type hierarchy.

      -- Method: dispatch_line (frame)
          If the debugger should stop on the current line, invoke the
          user_line() method (which should be overridden
          in subclasses).  Raise a BdbQuit exception if
          the `Bdb.quitting' flag is set (which can be set from
          user_line()).  Return a reference to the
          trace_dispatch() method for further tracing in that
          scope.

      -- Method: dispatch_call (frame, arg)
          If the debugger should stop on this function call, invoke the
          user_call() method (which should be overridden
          in subclasses).  Raise a BdbQuit exception if
          the `Bdb.quitting' flag is set (which can be set from
          user_call()).  Return a reference to the
          trace_dispatch() method for further tracing in that
          scope.

      -- Method: dispatch_return (frame, arg)
          If the debugger should stop on this function return, invoke
          the user_return() method (which should be
          overridden in subclasses).  Raise a BdbQuit.
          exception if the `Bdb.quitting' flag is set (which can be set
          from user_return()).  Return a reference to the
          trace_dispatch() method for further tracing in
          that scope.

      -- Method: dispatch_exception (frame, arg)
          If the debugger should stop at this exception, invokes the
          user_exception() method (which should be
          overridden in subclasses).  Raise a BdbQuit.
          exception if the `Bdb.quitting' flag is set (which can be set
          from user_exception()).  Return a reference to
          the trace_dispatch() method for further tracing
          in that scope.

     Normally derived classes don't override the following methods, but
     they may if they want to redefine the definition of stopping and
     breakpoints.

      -- Method: stop_here (frame)
          This method checks if the _frame_ is somewhere below
          `botframe' in the call stack.  `botframe' is the frame in
          which debugging started.

      -- Method: break_here (frame)
          This method checks if there is a breakpoint in the filename
          and line belonging to _frame_ or, at least, in the current
          function.  If the breakpoint is a temporary one, this method
          deletes it.

      -- Method: break_anywhere (frame)
          This method checks if there is a breakpoint in the filename
          of the current frame.

     Derived classes should override these methods to gain control over
     debugger operation.

      -- Method: user_call (frame, argument_list)
          This method is called from dispatch_call() when
          there is the possibility that a break might be necessary
          anywhere inside the called function.

      -- Method: user_line (frame)
          This method is called from dispatch_line() when
          either stop_here() or break_here().
          yields True.

      -- Method: user_return (frame, return_value)
          This method is called from dispatch_return().
          when stop_here().  yields True.

      -- Method: user_exception (frame, exc_info)
          This method is called from dispatch_exception().
          when stop_here() yields True.

      -- Method: do_clear (arg)
          Handle how a breakpoint must be removed when it is a
          temporary one.

          This method must be implemented by derived classes.

     Derived classes and clients can call the following methods to
     affect the stepping state.

      -- Method: set_step ()
          Stop after one line of code.

      -- Method: set_next (frame)
          Stop on the next line in or below the given frame.

      -- Method: set_return (frame)
          Stop when returning from the given frame.

      -- Method: set_until (frame)
          Stop when the line with the line no greater than the current
          one is reached or when returning from current frame

      -- Method: set_trace ([frame])
          Start debugging from _frame_.  If _frame_ is not specified,
          debugging starts from caller's frame.

      -- Method: set_continue ()
          Stop only at breakpoints or when finished.  If there are no
          breakpoints, set the system trace function to None.

      -- Method: set_quit ()
          Set the `quitting' attribute to True.  This raises
          BdbQuit in the next call to one of the `dispatch_*()'
          methods.

     Derived classes and clients can call the following methods to
     manipulate breakpoints.  These methods return a string containing
     an error message if something went wrong, or `None' if all is well.

      -- Method: set_break (filename, lineno[, temporary=0[, cond[,
               funcname]]])
          Set a new breakpoint.  If the _lineno_ line doesn't exist for
          the _filename_ passed as argument, return an error message.
          The _filename_ should be in canonical form, as described in
          the canonic() method.

      -- Method: clear_break (filename, lineno)
          Delete the breakpoints in _filename_ and _lineno_.  If none
          were set, an error message is returned.

      -- Method: clear_bpbynumber (arg)
          Delete the breakpoint which has the index _arg_ in the
          `Breakpoint.bpbynumber'.  If _arg_ is not numeric or out of
          range, return an error message.

      -- Method: clear_all_file_breaks (filename)
          Delete all breakpoints in _filename_.  If none were set, an
          error message is returned.

      -- Method: clear_all_breaks ()
          Delete all existing breakpoints.

      -- Method: get_break (filename, lineno)
          Check if there is a breakpoint for _lineno_ of _filename_.

      -- Method: get_breaks (filename, lineno)
          Return all breakpoints for _lineno_ in _filename_, or an
          empty list if none are set.

      -- Method: get_file_breaks (filename)
          Return all breakpoints in _filename_, or an empty list if
          none are set.

      -- Method: get_all_breaks ()
          Return all breakpoints that are set.

     Derived classes and clients can call the following methods to get
     a data structure representing a stack trace.

      -- Method: get_stack (f, t)
          Get a list of records for a frame and all higher (calling)
          and lower frames, and the size of the higher part.

      -- Method: format_stack_entry (frame_lineno[, lprefix=': '])
          Return a string with information about a stack entry,
          identified by a `(frame, lineno)' tuple:

             * The canonical form of the filename which contains the
               frame.

             * The function name, or `"<lambda>"'.

             * The input arguments.

             * The return value.

             * The line of code (if it exists).

     The following two methods can be called by clients to use a
     debugger to debug a statement, given as a string.

      -- Method: run (cmd[, globals[, locals]])
          Debug a statement executed via the exec.
          statement.  _globals_ defaults to `__main__.__dict__',
          _locals_ defaults to _globals_.

      -- Method: runeval (expr[, globals[, locals]])
          Debug an expression executed via the eval().
          function.  _globals_ and _locals_ have the same meaning as in
          run().

      -- Method: runctx (cmd, globals, locals)
          For backwards compatibility.  Calls the run().
          method.

      -- Method: runcall (func, *args, **kwds)
          Debug a single function call, and return its result.

  Finally, the module defines the following functions:

 -- Function: bdb.checkfuncname (b, frame)
     Check whether we should break here, depending on the way the
     breakpoint _b_ was set.

     If it was set via line number, it checks if `b.line' is the same
     as the one in the frame also passed as argument.  If the
     breakpoint was set via function name, we have to check we are in
     the right frame (the right function) and if we are in its first
     executable line.

 -- Function: bdb.effective (file, line, frame)
     Determine if there is an effective (active) breakpoint at this
     line of code.  Return a tuple of the breakpoint and a boolean that
     indicates if it is ok to delete a temporary breakpoint.  Return
     `(None, None)' if there is no matching breakpoint.

 -- Function: bdb.set_trace ()
     Start debugging with a Bdb instance from caller's
     frame.

5.26.2 `pdb' -- The Python Debugger
-----------------------------------

The module pdb defines an interactive source code debugger
for Python programs.  It supports setting (conditional) breakpoints and
single stepping at the source line level, inspection of stack frames,
source code listing, and evaluation of arbitrary Python code in the
context of any stack frame.  It also supports post-mortem debugging and
can be called under program control.

  The debugger is extensible -- it is actually defined as the class
Pdb.  This is currently undocumented but easily understood
by reading the source.  The extension interface uses the modules
bdb and cmd.

  The debugger's prompt is `(Pdb)'. Typical usage to run a program
under control of the debugger is:

    >>> import pdb
    >>> import mymodule
    >>> pdb.run('mymodule.test()')
    > <string>(0)?()
    (Pdb) continue
    > <string>(1)?()
    (Pdb) continue
    NameError: 'spam'
    > <string>(1)?()
    (Pdb)

`pdb.py' can also be invoked as a script to debug other scripts.  For
example:

    python -m pdb myscript.py

When invoked as a script, pdb will automatically enter post-mortem
debugging if the program being debugged exits abnormally. After
post-mortem debugging (or after normal exit of the program), pdb will
restart the program. Automatic restarting preserves pdb's state (such
as breakpoints) and in most cases is more useful than quitting the
debugger upon program's exit.

  New in version 2.4: Restarting post-mortem behavior added.

  The typical usage to break into the debugger from a running program
is to insert

    import pdb; pdb.set_trace()

at the location you want to break into the debugger.  You can then step
through the code following this statement, and continue running without
the debugger using the `c' command.

  The typical usage to inspect a crashed program is:

    >>> import pdb
    >>> import mymodule
    >>> mymodule.test()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "./mymodule.py", line 4, in test
        test2()
      File "./mymodule.py", line 3, in test2
        print spam
    NameError: spam
    >>> pdb.pm()
    > ./mymodule.py(3)test2()
    -> print spam
    (Pdb)

The module defines the following functions; each enters the debugger in
a slightly different way:

 -- Function: pdb.run (statement[, globals[, locals]])
     Execute the _statement_ (given as a string) under debugger
     control.  The debugger prompt appears before any code is executed;
     you can set breakpoints and type `continue', or you can step
     through the statement using `step' or `next' (all these commands
     are explained below).  The optional _globals_ and _locals_
     arguments specify the environment in which the code is executed; by
     default the dictionary of the module __main__ is used.
     (See the explanation of the exec statement or the
     eval() built-in function.)

 -- Function: pdb.runeval (expression[, globals[, locals]])
     Evaluate the _expression_ (given as a string) under debugger
     control.  When runeval() returns, it returns the
     value of the expression.  Otherwise this function is similar to
     run().

 -- Function: pdb.runcall (function[, argument, ...])
     Call the _function_ (a function or method object, not a string)
     with the given arguments.  When runcall() returns, it
     returns whatever the function call returned.  The debugger prompt
     appears as soon as the function is entered.

 -- Function: pdb.set_trace ()
     Enter the debugger at the calling stack frame.  This is useful to
     hard-code a breakpoint at a given point in a program, even if the
     code is not otherwise being debugged (e.g. when an assertion
     fails).

 -- Function: pdb.post_mortem ([traceback])
     Enter post-mortem debugging of the given _traceback_ object.  If no
     _traceback_ is given, it uses the one of the exception that is
     currently being handled (an exception must be being handled if the
     default is to be used).

 -- Function: pdb.pm ()
     Enter post-mortem debugging of the traceback found in
     sys.last_traceback.

  The `run_*' functions and set_trace() are aliases for
instantiating the Pdb class and calling the method of the
same name.  If you want to access further features, you have to do this
yourself:

 -- Class: pdb.Pdb (completekey='tab', stdin=None, stdout=None,
          skip=None)
     Pdb is the debugger class.

     The _completekey_, _stdin_ and _stdout_ arguments are passed to the
     underlying cmd.Cmd class; see the description there.

     The _skip_ argument, if given, must be an iterable of glob-style
     module name patterns.  The debugger will not step into frames that
     originate in a module that matches one of these patterns. (1)

     Example call to enable tracing with _skip_:

         import pdb; pdb.Pdb(skip=['django.*']).set_trace()

     New in version 2.7: The _skip_ argument.

      -- Method: run (statement[, globals[, locals]])
      -- Method: runeval (expression[, globals[, locals]])
      -- Method: runcall (function[, argument, ...])
      -- Method: set_trace ()
          See the documentation for the functions explained above.

5.26.3 Debugger Commands
------------------------

The debugger recognizes the following commands.  Most commands can be
abbreviated to one or two letters; e.g. `h(elp)' means that either `h'
or `help' can be used to enter the help command (but not `he' or `hel',
nor `H' or `Help' or `HELP').  Arguments to commands must be separated
by whitespace (spaces or tabs).  Optional arguments are enclosed in
square brackets (`[]') in the command syntax; the square brackets must
not be typed.  Alternatives in the command syntax are separated by a
vertical bar (`|').

  Entering a blank line repeats the last command entered.  Exception:
if the last command was a `list' command, the next 11 lines are listed.

  Commands that the debugger doesn't recognize are assumed to be Python
statements and are executed in the context of the program being
debugged.  Python statements can also be prefixed with an exclamation
point (`!').  This is a powerful way to inspect the program being
debugged; it is even possible to change a variable or call a function.
When an exception occurs in such a statement, the exception name is
printed but the debugger's state is not changed.

  Multiple commands may be entered on a single line, separated by `;;'.
(A single `;' is not used as it is the separator for multiple commands
in a line that is passed to the Python parser.) No intelligence is
applied to separating the commands; the input is split at the first
`;;' pair, even if it is in the middle of a quoted string.

  The debugger supports aliases.  Aliases can have parameters which
allows one a certain level of adaptability to the context under
examination.

  If a file `.pdbrc'  exists in the user's home directory or in the
current directory, it is read in and executed as if it had been typed
at the debugger prompt. This is particularly useful for aliases.  If
both files exist, the one in the home directory is read first and
aliases defined there can be overridden by the local file.

h(elp) [_command_]
     Without argument, print the list of available commands.  With a
     _command_ as argument, print help about that command.  `help pdb'
     displays the full documentation file; if the environment variable `PAGER'
     is defined, the file is piped through that command instead.  Since
     the _command_ argument must be an identifier, `help exec' must be
     entered to get help on the `!' command.

w(here)
     Print a stack trace, with the most recent frame at the bottom.  An
     arrow indicates the current frame, which determines the context of
     most commands.

d(own)
     Move the current frame one level down in the stack trace (to a
     newer frame).

u(p)
     Move the current frame one level up in the stack trace (to an
     older frame).

b(reak) [[_filename_:]_lineno_ | _function_[, _condition_]]
     With a _lineno_ argument, set a break there in the current file.
     With a _function_ argument, set a break at the first executable
     statement within that function. The line number may be prefixed
     with a filename and a colon, to specify a breakpoint in another
     file (probably one that hasn't been loaded yet).  The file is
     searched on `sys.path'. Note that each breakpoint is assigned a
     number to which all the other breakpoint commands refer.

     If a second argument is present, it is an expression which must
     evaluate to true before the breakpoint is honored.

     Without argument, list all breaks, including for each breakpoint,
     the number of times that breakpoint has been hit, the current
     ignore count, and the associated condition if any.

tbreak [[_filename_:]_lineno_ | _function_[, _condition_]]
     Temporary breakpoint, which is removed automatically when it is
     first hit.  The arguments are the same as break.

cl(ear) [_bpnumber_ [_bpnumber ..._]]
     With a space separated list of breakpoint numbers, clear those
     breakpoints.  Without argument, clear all breaks (but first ask
     confirmation).

disable [_bpnumber_ [_bpnumber ..._]]
     Disables the breakpoints given as a space separated list of
     breakpoint numbers.  Disabling a breakpoint means it cannot cause
     the program to stop execution, but unlike clearing a breakpoint,
     it remains in the list of breakpoints and can be (re-)enabled.

enable [_bpnumber_ [_bpnumber ..._]]
     Enables the breakpoints specified.

ignore _bpnumber_ [_count_]
     Sets the ignore count for the given breakpoint number.  If count
     is omitted, the ignore count is set to 0.  A breakpoint becomes
     active when the ignore count is zero.  When non-zero, the count is
     decremented each time the breakpoint is reached and the breakpoint
     is not disabled and any associated condition evaluates to true.

condition _bpnumber_ [_condition_]
     Condition is an expression which must evaluate to true before the
     breakpoint is honored.  If condition is absent, any existing
     condition is removed; i.e., the breakpoint is made unconditional.

commands [_bpnumber_]
     Specify a list of commands for breakpoint number _bpnumber_.  The
     commands themselves appear on the following lines.  Type a line
     containing just 'end' to terminate the commands. An example:

         (Pdb) commands 1
         (com) print some_variable
         (com) end
         (Pdb)

     To remove all commands from a breakpoint, type commands and follow
     it immediately with  end; that is, give no commands.

     With no _bpnumber_ argument, commands refers to the last
     breakpoint set.

     You can use breakpoint commands to start your program up again.
     Simply use the continue command, or step, or any other command
     that resumes execution.

     Specifying any command resuming execution (currently continue,
     step, next, return, jump, quit and their abbreviations) terminates
     the command list (as if that command was immediately followed by
     end). This is because any time you resume execution (even with a
     simple next or step), you may encounter another breakpoint-which
     could have its own command list, leading to ambiguities about
     which list to execute.

     If you use the 'silent' command in the command list, the usual
     message about stopping at a breakpoint is not printed.  This may
     be desirable for breakpoints that are to print a specific message
     and then continue.  If none of the other commands print anything,
     you see no sign that the breakpoint was reached.

     New in version 2.5.

s(tep)
     Execute the current line, stop at the first possible occasion
     (either in a function that is called or on the next line in the
     current function).

n(ext)
     Continue execution until the next line in the current function is
     reached or it returns.  (The difference between `next' and `step'
     is that `step' stops inside a called function, while `next'
     executes called functions at (nearly) full speed, only stopping at
     the next line in the current function.)

unt(il)
     Continue execution until the line with the line number greater
     than the current one is reached or when returning from current
     frame.

     New in version 2.6.

r(eturn)
     Continue execution until the current function returns.

c(ont(inue))
     Continue execution, only stop when a breakpoint is encountered.

j(ump) _lineno_
     Set the next line that will be executed.  Only available in the
     bottom-most frame.  This lets you jump back and execute code
     again, or jump forward to skip code that you don't want to run.

     It should be noted that not all jumps are allowed -- for instance
     it is not possible to jump into the middle of a for.
     loop or out of a finally clause.

l(ist) [_first_[, _last_]]
     List source code for the current file.  Without arguments, list 11
     lines around the current line or continue the previous listing.
     With one argument, list 11 lines around at that line.  With two
     arguments, list the given range; if the second argument is less
     than the first, it is interpreted as a count.

a(rgs)
     Print the argument list of the current function.

p _expression_
     Evaluate the _expression_ in the current context and print its
     value.

          Note: `print' can also be used, but is not a debugger command
          -- this executes the Python print statement.

pp _expression_
     Like the `p' command, except the value of the expression is
     pretty-printed using the pprint module.

alias [_name_ [command]]
     Creates an alias called _name_ that executes _command_.  The
     command must _not_ be enclosed in quotes.  Replaceable parameters
     can be indicated by `%1', `%2', and so on, while `%*' is replaced
     by all the parameters.  If no command is given, the current alias
     for _name_ is shown. If no arguments are given, all aliases are
     listed.

     Aliases may be nested and can contain anything that can be legally
     typed at the pdb prompt.  Note that internal pdb commands _can_ be
     overridden by aliases.  Such a command is then hidden until the
     alias is removed.  Aliasing is recursively applied to the first
     word of the command line; all other words in the line are left
     alone.

     As an example, here are two useful aliases (especially when placed
     in the `.pdbrc' file):

         #Print instance variables (usage "pi classInst")
         alias pi for k in %1.__dict__.keys(): print "%1.",k,"=",%1.__dict__[k]
         #Print instance variables in self
         alias ps pi self

unalias _name_
     Deletes the specified alias.

[!]_statement_
     Execute the (one-line) _statement_ in the context of the current
     stack frame.  The exclamation point can be omitted unless the
     first word of the statement resembles a debugger command. To set a
     global variable, you can prefix the assignment command with a
     `global' command on the same line, e.g.:

         (Pdb) global list_options; list_options = ['-l']
         (Pdb)

run [_args_ ...]
     Restart the debugged Python program. If an argument is supplied,
     it is split with "shlex" and the result is used as the new
     sys.argv. History, breakpoints, actions and debugger options are
     preserved. "restart" is an alias for "run".

     New in version 2.6.

q(uit)
     Quit from the debugger. The program being executed is aborted.

Footnotes
.........

     (1) Whether a frame is considered to originate in a certain module
     is determined by the `__name__' in the frame globals.

5.26.4 The Python Profilers
---------------------------

Copyright © 1994, by InfoSeek Corporation, all rights reserved.

  Written by James Roskind. (1)

  Permission to use, copy, modify, and distribute this Python software
and its associated documentation for any purpose (subject to the
restriction in the following sentence) without fee is hereby granted,
provided that the above copyright notice appears in all copies, and
that both that copyright notice and this permission notice appear in
supporting documentation, and that the name of InfoSeek not be used in
advertising or publicity pertaining to distribution of the software
without specific, written prior permission.  This permission is
explicitly restricted to the copying and modification of the software
to remain in Python, compiled Python, or other languages (such as C)
wherein the modified or derived code is exclusively imported into a
Python module.

  INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY
SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

5.26.4.1 Introduction to the profilers
......................................

A _profiler_ is a program that describes the run time performance of a
program, providing a variety of statistics.  This documentation
describes the profiler functionality provided in the modules
cProfile, profile and pstats.  This profiler
provides _deterministic profiling_ of Python programs.  It also
provides a series of report generation tools to allow users to rapidly
examine the results of a profile operation.

  The Python standard library provides three different profilers:

  1. cProfile is recommended for most users; it's a C
     extension with reasonable overhead that makes it suitable for
     profiling long-running programs.  Based on `lsprof', contributed
     by Brett Rosen and Ted Czotter.

     New in version 2.5.

  2. profile, a pure Python module whose interface is
     imitated by cProfile.  Adds significant overhead to
     profiled programs.  If you're trying to extend the profiler in
     some way, the task might be easier with this module.  Copyright ©
     1994, by InfoSeek Corporation.

     Changed in version 2.4: Now also reports the time spent in calls
     to built-in functions and methods.

  3. hotshot was an experimental C module that focused on
     minimizing the overhead of profiling, at the expense of longer data
     post-processing times.  It is no longer maintained and may be
     dropped in a future version of Python.

     Changed in version 2.5: The results should be more meaningful than
     in the past: the timing core contained a critical bug.

  The profile and cProfile modules export the
same interface, so they are mostly interchangeable; see cProfile.
has a much lower overhead but is newer and might not be available on
all systems.  cProfile is really a compatibility layer on
top of the internal `_lsprof' module.  The hotshot module is
reserved for specialized usage.

5.26.4.2 Instant User's Manual
..............................

This section is provided for users that "don't want to read the
manual." It provides a very brief overview, and allows a user to
rapidly perform profiling on an existing application.

  To profile an application with a main entry point of `foo()', you
would add the following to your module:

    import cProfile
    cProfile.run('foo()')

(Use profile instead of cProfile if the latter
is not available on your system.)

  The above action would cause `foo()' to be run, and a series of
informative lines (the profile) to be printed.  The above approach is
most useful when working with the interpreter.  If you would like to
save the results of a profile into a file for later examination, you
can supply a file name as the second argument to the `run()' function:

    import cProfile
    cProfile.run('foo()', 'fooprof')

The file `cProfile.py' can also be invoked as a script to profile
another script.  For example:

    python -m cProfile myscript.py

`cProfile.py' accepts two optional arguments on the command line:

    cProfile.py [-o output_file] [-s sort_order]

`-s' only applies to standard output (`-o' is not supplied).  Look in
the `Stats' documentation for valid sort values.

  When you wish to review the profile, you should use the methods in the
pstats module.  Typically you would load the statistics
data as follows:

    import pstats
    p = pstats.Stats('fooprof')

The class `Stats' (the above code just created an instance of this
class) has a variety of methods for manipulating and printing the data
that was just read into `p'.  When you ran cProfile.run().
above, what was printed was the result of three method calls:

    p.strip_dirs().sort_stats(-1).print_stats()

The first method removed the extraneous path from all the module names.
The second method sorted all the entries according to the standard
module/line/name string that is printed. The third method printed out
all the statistics.  You might try the following sort calls:

    p.sort_stats('name')
    p.print_stats()

The first call will actually sort the list by function name, and the
second call will print out the statistics.  The following are some
interesting calls to experiment with:

    p.sort_stats('cumulative').print_stats(10)

This sorts the profile by cumulative time in a function, and then only
prints the ten most significant lines.  If you want to understand what
algorithms are taking time, the above line is what you would use.

  If you were looking to see what functions were looping a lot, and
taking a lot of time, you would do:

    p.sort_stats('time').print_stats(10)

to sort according to time spent within each function, and then print the
statistics for the top ten functions.

  You might also try:

    p.sort_stats('file').print_stats('__init__')

This will sort all the statistics by file name, and then print out
statistics for only the class init methods (since they are spelled with
`__init__' in them).  As one final example, you could try:

    p.sort_stats('time', 'cum').print_stats(.5, 'init')

This line sorts statistics with a primary key of time, and a secondary
key of cumulative time, and then prints out some of the statistics. To
be specific, the list is first culled down to 50% (re: `.5') of its
original size, then only lines containing `init' are maintained, and
that sub-sub-list is printed.

  If you wondered what functions called the above functions, you could
now (`p' is still sorted according to the last criteria) do:

    p.print_callers(.5, 'init')

and you would get a list of callers for each of the listed functions.

  If you want more functionality, you're going to have to read the
manual, or guess what the following functions do:

    p.print_callees()
    p.add('fooprof')

Invoked as a script, the pstats module is a statistics
browser for reading and examining profile dumps.  It has a simple
line-oriented interface (implemented using cmd) and
interactive help.

5.26.4.3 What Is Deterministic Profiling?
.........................................

_Deterministic profiling_ is meant to reflect the fact that all
_function call_, _function return_, and _exception_ events are
monitored, and precise timings are made for the intervals between these
events (during which time the user's code is executing).  In contrast,
_statistical profiling_ (which is not done by this module) randomly
samples the effective instruction pointer, and deduces where time is
being spent.  The latter technique traditionally involves less overhead
(as the code does not need to be instrumented), but provides only
relative indications of where time is being spent.

  In Python, since there is an interpreter active during execution, the
presence of instrumented code is not required to do deterministic
profiling.  Python automatically provides a _hook_ (optional callback)
for each event.  In addition, the interpreted nature of Python tends to
add so much overhead to execution, that deterministic profiling tends
to only add small processing overhead in typical applications.  The
result is that deterministic profiling is not that expensive, yet
provides extensive run time statistics about the execution of a Python
program.

  Call count statistics can be used to identify bugs in code
(surprising counts), and to identify possible inline-expansion points
(high call counts).  Internal time statistics can be used to identify
"hot loops" that should be carefully optimized.  Cumulative time
statistics should be used to identify high level errors in the
selection of algorithms.  Note that the unusual handling of cumulative
times in this profiler allows statistics for recursive implementations
of algorithms to be directly compared to iterative implementations.

5.26.4.4 Reference Manual - `profile' and `cProfile'
....................................................

The primary entry point for the profiler is the global function
`profile.run()' (resp. see cProfile.run()). It is typically
used to create any profile information.  The reports are formatted and
printed using methods of the class pstats.Stats.  The
following is a description of all of these standard entry points and
functions.  For a more in-depth view of some of the code, consider
reading the later section on Profiler Extensions, which includes
discussion of how to derive "better" profilers from the classes
presented, or reading the source code for these modules.

 -- Function: cProfile.run (command[, filename])
     This function takes a single argument that can be passed to the
     exec statement, and an optional file name.  In all
     cases this routine attempts to exec its first
     argument, and gather profiling statistics from the execution. If
     no file name is present, then this function automatically prints a
     simple profiling report, sorted by the standard name string
     (file/line/function-name) that is presented in each line.  The
     following is a typical output from such a call:

               2706 function calls (2004 primitive calls) in 4.504 CPU seconds

         Ordered by: standard name

         ncalls  tottime  percall  cumtime  percall filename:lineno(function)
              2    0.006    0.003    0.953    0.477 pobject.py:75(save_objects)
           43/3    0.533    0.012    0.749    0.250 pobject.py:99(evaluate)
          ...

     The first line indicates that 2706 calls were monitored.  Of those
     calls, 2004 were _primitive_.  We define _primitive_ to mean that
     the call was not induced via recursion. The next line: `Ordered
     by: standard name', indicates that the text string in the far
     right column was used to sort the output. The column headings
     include:

    ncalls
          for the number of calls,

    tottime
          for the total time spent in the given function (and excluding
          time made in calls to sub-functions),

    percall
          is the quotient of `tottime' divided by `ncalls'

    cumtime
          is the total time spent in this and all subfunctions (from
          invocation till exit). This figure is accurate _even_ for
          recursive functions.

    percall
          is the quotient of `cumtime' divided by primitive calls

    filename:lineno(function)
          provides the respective data of each function

     When there are two numbers in the first column (for example,
     `43/3'), then the latter is the number of primitive calls, and the
     former is the actual number of calls.  Note that when the function
     does not recurse, these two values are the same, and only the
     single figure is printed.

 -- Function: cProfile.runctx (command, globals, locals[, filename])
     This function is similar to run(), with added
     arguments to supply the globals and locals dictionaries for the
     _command_ string.

  Analysis of the profiler data is done using the `Stats' class.

     Note: The `Stats' class is defined in the pstats.
     module.

 -- Class: pstats.Stats (filename[, stream=sys.stdout[, ...]])
     This class constructor creates an instance of a "statistics
     object" from a _filename_ (or set of filenames).  Stats
     objects are manipulated by methods, in order to print useful
     reports.  You may specify an alternate output stream by giving the
     keyword argument, `stream'.

     The file selected by the above constructor must have been created
     by the corresponding version of profile or
     cProfile.  To be specific, there is _no_ file compatibility
     guaranteed with future versions of this profiler, and there is no
     compatibility with files produced by other profilers.  If several
     files are provided, all the statistics for identical functions will
     be coalesced, so that an overall view of several processes can be
     considered in a single report.  If additional files need to be
     combined with data in an existing Stats object, the
     add() method can be used.

     Changed in version 2.5: The _stream_ parameter was added.

5.26.4.5 The `Stats' Class
..........................

Stats objects have the following methods:

 -- Method: Stats.strip_dirs ()
     This method for the Stats class removes all leading
     path information from file names.  It is very useful in reducing
     the size of the printout to fit within (close to) 80 columns.
     This method modifies the object, and the stripped information is
     lost.  After performing a strip operation, the object is
     considered to have its entries in a "random" order, as it was just
     after object initialization and loading.  If strip_dirs()
     causes two function names to be indistinguishable (they are
     on the same line of the same filename, and have the same function
     name), then the statistics for these two entries are accumulated
     into a single entry.

 -- Method: Stats.add (filename[, ...])
     This method of the Stats class accumulates additional
     profiling information into the current profiling object.  Its
     arguments should refer to filenames created by the corresponding
     version of `profile.run()' or cProfile.run().
     Statistics for identically named (re: file, line, name) functions
     are automatically accumulated into single function statistics.

 -- Method: Stats.dump_stats (filename)
     Save the data loaded into the Stats object to a file
     named _filename_.  The file is created if it does not exist, and
     is overwritten if it already exists.  This is equivalent to the
     method of the same name on the `profile.Profile' and
     `cProfile.Profile' classes.

     New in version 2.3.

 -- Method: Stats.sort_stats (key[, ...])
     This method modifies the Stats object by sorting it
     according to the supplied criteria.  The argument is typically a
     string identifying the basis of a sort (example: `'time'' or
     `'name'').

     When more than one key is provided, then additional keys are used
     as secondary criteria when there is equality in all keys selected
     before them.  For example, `sort_stats('name', 'file')' will sort
     all the entries according to their function name, and resolve all
     ties (identical function names) by sorting by file name.

     Abbreviations can be used for any key names, as long as the
     abbreviation is unambiguous.  The following are the keys currently
     defined:

     Valid Arg              Meaning
     --------------------------------------------------
     `'calls''              call count
     `'cumulative''         cumulative time
     `'file''               file name
     `'module''             file name
     `'pcalls''             primitive call count
     `'line''               line number
     `'name''               function name
     `'nfl''                name/file/line
     `'stdname''            standard name
     `'time''               internal time

     Note that all sorts on statistics are in descending order (placing
     most time consuming items first), where as name, file, and line
     number searches are in ascending order (alphabetical). The subtle
     distinction between `'nfl'' and `'stdname'' is that the standard
     name is a sort of the name as printed, which means that the
     embedded line numbers get compared in an odd way.  For example,
     lines 3, 20, and 40 would (if the file names were the same) appear
     in the string order 20, 3 and 40.  In contrast, `'nfl'' does a
     numeric compare of the line numbers.  In fact, `sort_stats('nfl')'
     is the same as `sort_stats('name', 'file', 'line')'.

     For backward-compatibility reasons, the numeric arguments `-1',
     `0', `1', and `2' are permitted.  They are interpreted as
     `'stdname'', `'calls'', `'time'', and `'cumulative'' respectively.
     If this old style format (numeric) is used, only one sort key (the
     numeric key) will be used, and additional arguments will be
     silently ignored.

 -- Method: Stats.reverse_order ()
     This method for the Stats class reverses the ordering
     of the basic list within the object.  Note that by default
     ascending vs descending order is properly selected based on the
     sort key of choice.

 -- Method: Stats.print_stats ([restriction, ...])
     This method for the Stats class prints out a report
     as described in the `profile.run()' definition.

     The order of the printing is based on the last sort_stats()
     operation done on the object (subject to caveats in
     add() and strip_dirs()).

     The arguments provided (if any) can be used to limit the list down
     to the significant entries.  Initially, the list is taken to be
     the complete set of profiled functions.  Each restriction is
     either an integer (to select a count of lines), or a decimal
     fraction between 0.0 and 1.0 inclusive (to select a percentage of
     lines), or a regular expression (to pattern match the standard
     name that is printed; as of Python 1.5b1, this uses the Perl-style
     regular expression syntax defined by the re module).
     If several restrictions are provided, then they are applied
     sequentially.  For example:

         print_stats(.1, 'foo:')

     would first limit the printing to first 10% of list, and then only
     print functions that were part of filename `.*foo:'.  In contrast,
     the command:

         print_stats('foo:', .1)

     would limit the list to all functions having file names `.*foo:',
     and then proceed to only print the first 10% of them.

 -- Method: Stats.print_callers ([restriction, ...])
     This method for the Stats class prints a list of all
     functions that called each function in the profiled database.  The
     ordering is identical to that provided by print_stats(),
     and the definition of the restricting argument is also
     identical.  Each caller is reported on its own line.  The format
     differs slightly depending on the profiler that produced the stats:

        * With profile, a number is shown in parentheses
          after each caller to show how many times this specific call
          was made.  For convenience, a second non-parenthesized number
          repeats the cumulative time spent in the function at the
          right.

        * With cProfile, each caller is preceded by three
          numbers: the number of times this specific call was made, and
          the total and cumulative times spent in the current function
          while it was invoked by this specific caller.

 -- Method: Stats.print_callees ([restriction, ...])
     This method for the Stats class prints a list of all
     function that were called by the indicated function.  Aside from
     this reversal of direction of calls (re: called vs was called by),
     the arguments and ordering are identical to the
     print_callers() method.

5.26.4.6 Limitations
....................

One limitation has to do with accuracy of timing information. There is a
fundamental problem with deterministic profilers involving accuracy.
The most obvious restriction is that the underlying "clock" is only
ticking at a rate (typically) of about .001 seconds.  Hence no
measurements will be more accurate than the underlying clock.  If
enough measurements are taken, then the "error" will tend to average
out. Unfortunately, removing this first error induces a second source
of error.

  The second problem is that it "takes a while" from when an event is
dispatched until the profiler's call to get the time actually _gets_
the state of the clock.  Similarly, there is a certain lag when exiting
the profiler event handler from the time that the clock's value was
obtained (and then squirreled away), until the user's code is once
again executing.  As a result, functions that are called many times, or
call many functions, will typically accumulate this error. The error
that accumulates in this fashion is typically less than the accuracy of
the clock (less than one clock tick), but it _can_ accumulate and
become very significant.

  The problem is more important with profile than with the
lower-overhead cProfile.  For this reason, see profile
provides a means of calibrating itself for a given platform so
that this error can be probabilistically (on the average) removed.
After the profiler is calibrated, it will be more accurate (in a least
square sense), but it will sometimes produce negative numbers (when
call counts are exceptionally low, and the gods of probability work
against you :-). )  Do _not_ be alarmed by negative numbers in the
profile.  They should _only_ appear if you have calibrated your
profiler, and the results are actually better than without calibration.

5.26.4.7 Calibration
....................

The profiler of the profile module subtracts a constant
from each event handling time to compensate for the overhead of calling
the time function, and socking away the results.  By default, the
constant is 0. The following procedure can be used to obtain a better
constant for a given platform (see discussion in section Limitations
above).

    import profile
    pr = profile.Profile()
    for i in range(5):
        print pr.calibrate(10000)

The method executes the number of Python calls given by the argument,
directly and again under the profiler, measuring the time for both. It
then computes the hidden overhead per profiler event, and returns that
as a float.  For example, on an 800 MHz Pentium running Windows 2000,
and using Python's time.clock() as the timer, the magical number is
about 12.5e-6.

  The object of this exercise is to get a fairly consistent result. If
your computer is _very_ fast, or your timer function has poor
resolution, you might have to pass 100000, or even 1000000, to get
consistent results.

  When you have a consistent answer, there are three ways you can use
it: (2)

    import profile

    # 1. Apply computed bias to all Profile instances created hereafter.
    profile.Profile.bias = your_computed_bias

    # 2. Apply computed bias to a specific Profile instance.
    pr = profile.Profile()
    pr.bias = your_computed_bias

    # 3. Specify computed bias in instance constructor.
    pr = profile.Profile(bias=your_computed_bias)

If you have a choice, you are better off choosing a smaller constant,
and then your results will "less often" show up as negative in profile
statistics.

5.26.4.8 Extensions -- Deriving Better Profilers
................................................

The `Profile' class of both modules, see profile and
cProfile, were written so that derived classes could be developed
to extend the profiler.  The details are not described here, as doing
this successfully requires an expert understanding of how the `Profile'
class works internally.  Study the source code of the module carefully
if you want to pursue this.

  If all you want to do is change how current time is determined (for
example, to force use of wall-clock time or elapsed process time), pass
the timing function you want to the `Profile' class constructor:

    pr = profile.Profile(your_time_func)

The resulting profiler will then call `your_time_func()'.

`profile.Profile'
     `your_time_func()' should return a single number, or a list of
     numbers whose sum is the current time (like what os.times()
     returns).  If the function returns a single time number, or
     the list of returned numbers has length 2, then you will get an
     especially fast version of the dispatch routine.

     Be warned that you should calibrate the profiler class for the
     timer function that you choose.  For most machines, a timer that
     returns a lone integer value will provide the best results in
     terms of low overhead during profiling.  (see os.times().
     is _pretty_ bad, as it returns a tuple of floating point values).
     If you want to substitute a better timer in the cleanest fashion,
     derive a class and hardwire a replacement dispatch method that
     best handles your timer call, along with the appropriate
     calibration constant.

`cProfile.Profile'
     `your_time_func()' should return a single number.  If it returns
     plain integers, you can also invoke the class constructor with a
     second argument specifying the real duration of one unit of time.
     For example, if `your_integer_time_func()' returns times measured
     in thousands of seconds, you would construct the `Profile'
     instance as follows:

         pr = profile.Profile(your_integer_time_func, 0.001)

     As the `cProfile.Profile' class cannot be calibrated, custom timer
     functions should be used with care and should be as fast as
     possible.  For the best results with a custom timer, it might be
     necessary to hard-code it in the C source of the internal
     `_lsprof' module.

Footnotes
.........

     (1) Updated and converted to LaTeX by Guido van Rossum. Further
     updated by Armin Rigo to integrate the documentation for the new
     cProfile module of Python 2.5.

     (2) Prior to Python 2.2, it was necessary to edit the profiler
     source code to embed the bias as a literal number.  You still can,
     but that method is no longer described, because no longer needed.

5.26.5 `hotshot' -- High performance logging profiler
-----------------------------------------------------

New in version 2.2.

  This module provides a nicer interface to the `_hotshot' C module.
Hotshot is a replacement for the existing profile module.
As it's written mostly in C, it should result in a much smaller
performance impact than the existing profile module.

     Note: The hotshot module focuses on minimizing the
     overhead while profiling, at the expense of long data
     post-processing times. For common usage it is recommended to use
     cProfile instead. hotshot is not maintained
     and might be removed from the standard library in the future.

  Changed in version 2.5: The results should be more meaningful than in
the past: the timing core contained a critical bug.

     Note: The hotshot profiler does not yet work well with
     threads. It is useful to use an unthreaded script to run the
     profiler over the code you're interested in measuring if at all
     possible.

 -- Class: hotshot.Profile (logfile[, lineevents[, linetimings]])
     The profiler object. The argument _logfile_ is the name of a log
     file to use for logged profile data. The argument _lineevents_
     specifies whether to generate events for every source line, or
     just on function call/return. It defaults to `0' (only log
     function call/return). The argument _linetimings_ specifies
     whether to record timing information. It defaults to `1' (store
     timing information).

5.26.5.1 Profile Objects
........................

Profile objects have the following methods:

 -- Method: Profile.addinfo (key, value)
     Add an arbitrary labelled value to the profile output.

 -- Method: Profile.close ()
     Close the logfile and terminate the profiler.

 -- Method: Profile.fileno ()
     Return the file descriptor of the profiler's log file.

 -- Method: Profile.run (cmd)
     Profile an exec.-compatible string in the script
     environment. The globals from the __main__ module are
     used as both the globals and locals for the script.

 -- Method: Profile.runcall (func, *args, **keywords)
     Profile a single call of a callable. Additional positional and
     keyword arguments may be passed along; the result of the call is
     returned, and exceptions are allowed to propagate cleanly, while
     ensuring that profiling is disabled on the way out.

 -- Method: Profile.runctx (cmd, globals, locals)
     Evaluate an exec.-compatible string in a specific
     environment. The string is compiled before profiling begins.

 -- Method: Profile.start ()
     Start the profiler.

 -- Method: Profile.stop ()
     Stop the profiler.

5.26.5.2 Using hotshot data
...........................

New in version 2.2.

  This module loads hotshot profiling data into the standard
pstats. Stats objects.

 -- Function: hotshot.stats.load (filename)
     Load hotshot data from _filename_. Returns an instance of the
     pstats.Stats class.

Module profile.
     The profile module's `Stats' class

5.26.5.3 Example Usage
......................

Note that this example runs the Python "benchmark" pystones.  It can
take some time to run, and will produce large output files.

    >>> import hotshot, hotshot.stats, test.pystone
    >>> prof = hotshot.Profile("stones.prof")
    >>> benchtime, stones = prof.runcall(test.pystone.pystones)
    >>> prof.close()
    >>> stats = hotshot.stats.load("stones.prof")
    >>> stats.strip_dirs()
    >>> stats.sort_stats('time', 'calls')
    >>> stats.print_stats(20)
             850004 function calls in 10.090 CPU seconds

       Ordered by: internal time, call count

       ncalls  tottime  percall  cumtime  percall filename:lineno(function)
            1    3.295    3.295   10.090   10.090 pystone.py:79(Proc0)
       150000    1.315    0.000    1.315    0.000 pystone.py:203(Proc7)
        50000    1.313    0.000    1.463    0.000 pystone.py:229(Func2)
     .
     .
     .

5.26.6 `timeit' -- Measure execution time of small code snippets
----------------------------------------------------------------

New in version 2.3.

  This module provides a simple way to time small bits of Python code.
It has both command line as well as callable interfaces.  It avoids a
number of common traps for measuring execution times.  See also Tim
Peters' introduction to the "Algorithms" chapter in the Python
Cookbook, published by O'Reilly.

  The module defines the following public class:

 -- Class: timeit.Timer ([stmt='pass'[, setup='pass'[, timer=<timer
          function>]]])
     Class for timing execution speed of small code snippets.

     The constructor takes a statement to be timed, an additional
     statement used for setup, and a timer function.  Both statements
     default to `'pass''; the timer function is platform-dependent (see
     the module doc string).  _stmt_ and _setup_ may also contain
     multiple statements separated by `;' or newlines, as long as they
     don't contain multi-line string literals.

     To measure the execution time of the first statement, use the
     timeit().  method.  The repeat() method is
     a convenience to call timeit().  multiple times and
     return a list of results.

     Changed in version 2.6: The _stmt_ and _setup_ parameters can now
     also take objects that are callable without arguments. This will
     embed calls to them in a timer function that will then be executed
     by timeit().  Note that the timing overhead is a little
     larger in this case because of the extra function calls.

 -- Method: Timer.print_exc ([file=None])
     Helper to print a traceback from the timed code.

     Typical use:

         t = Timer(...)       # outside the try/except
         try:
             t.timeit(...)    # or t.repeat(...)
         except:
             t.print_exc()

     The advantage over the standard traceback is that source lines in
     the compiled template will be displayed. The optional _file_
     argument directs where the traceback is sent; it defaults to
     `sys.stderr'.

 -- Method: Timer.repeat ([repeat=3[, number=1000000]])
     Call timeit() a few times.

     This is a convenience function that calls the timeit().
     repeatedly, returning a list of results.  The first argument
     specifies how many times to call timeit().  The second
     argument specifies the _number_ argument for timeit().

          Note: It's tempting to calculate mean and standard deviation
          from the result vector and report these.  However, this is
          not very useful.  In a typical case, the lowest value gives a
          lower bound for how fast your machine can run the given code
          snippet; higher values in the result vector are typically not
          caused by variability in Python's speed, but by other
          processes interfering with your timing accuracy.  So the
          min() of the result is probably the only number
          you should be interested in.  After that, you should look at
          the entire vector and apply common sense rather than
          statistics.

 -- Method: Timer.timeit ([number=1000000])
     Time _number_ executions of the main statement. This executes the
     setup statement once, and then returns the time it takes to
     execute the main statement a number of times, measured in seconds
     as a float.  The argument is the number of times through the loop,
     defaulting to one million.  The main statement, the setup
     statement and the timer function to be used are passed to the
     constructor.

          Note: By default, see timeit() temporarily turns off
          garbage collection.  during the timing.  The
          advantage of this approach is that it makes independent
          timings more comparable.  This disadvantage is that GC may be
          an important component of the performance of the function
          being measured.  If so, GC can be re-enabled as the first
          statement in the _setup_ string.  For example:

              timeit.Timer('for i in xrange(10): oct(i)', 'gc.enable()').timeit()

  Starting with version 2.6, the module also defines two convenience
functions:

 -- Function: timeit.repeat (stmt[, setup[, timer[, repeat=3[,
          number=1000000]]]])
     Create a Timer instance with the given statement,
     setup code and timer function and run its repeat().
     method with the given repeat count and _number_ executions.

     New in version 2.6.

 -- Function: timeit.timeit (stmt[, setup[, timer[, number=1000000]]])
     Create a Timer instance with the given statement,
     setup code and timer function and run its timeit().
     method with _number_ executions.

     New in version 2.6.

5.26.6.1 Command Line Interface
...............................

When called as a program from the command line, the following form is
used:

    python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...]

where the following options are understood:

-n N/`--number=N'
     how many times to execute 'statement'

-r N/`--repeat=N'
     how many times to repeat the timer (default 3)

-s S/`--setup=S'
     statement to be executed once initially (default `'pass'')

-t/`--time'
     use time.time(). (default on all platforms but Windows)

-c/`--clock'
     use time.clock(). (default on Windows)

-v/`--verbose'
     print raw timing results; repeat for more digits precision

-h/see -help.
     print a short usage message and exit

  A multi-line statement may be given by specifying each line as a
separate statement argument; indented lines are possible by enclosing
an argument in quotes and using leading spaces.  Multiple -s
options are treated similarly.

  If `-n' is not given, a suitable number of loops is calculated by
trying successive powers of 10 until the total time is at least 0.2
seconds.

  The default timer function is platform dependent.  On Windows, see
time.clock() has microsecond granularity but time.time().'
s granularity is 1/60th of a second; on Unix, see time.clock()
has 1/100th of a second granularity and time.time().
is much more precise.  On either platform, the default timer functions
measure wall clock time, not the CPU time.  This means that other
processes running on the same computer may interfere with the timing.
The best thing to do when accurate timing is necessary is to repeat the
timing a few times and use the best time.  The `-r' option is good for
this; the default of 3 repetitions is probably enough in most cases.  On
Unix, you can use time.clock() to measure CPU time.

     Note: There is a certain baseline overhead associated with
     executing a pass statement.  The code here doesn't try to hide it,
     but you should be aware of it.  The baseline overhead can be
     measured by invoking the program without arguments.

  The baseline overhead differs between Python versions!  Also, to
fairly compare older Python versions to Python 2.3, you may want to use
Python's -O.  option for the older versions to avoid timing
`SET_LINENO' instructions.

5.26.6.2 Examples
.................

Here are two example sessions (one using the command line, one using
the module interface) that compare the cost of using hasattr()
vs.  see try./see except to test for missing and
present object attributes.

    % timeit.py 'try:' '  str.__nonzero__' 'except AttributeError:' '  pass'
    100000 loops, best of 3: 15.7 usec per loop
    % timeit.py 'if hasattr(str, "__nonzero__"): pass'
    100000 loops, best of 3: 4.26 usec per loop
    % timeit.py 'try:' '  int.__nonzero__' 'except AttributeError:' '  pass'
    1000000 loops, best of 3: 1.43 usec per loop
    % timeit.py 'if hasattr(int, "__nonzero__"): pass'
    100000 loops, best of 3: 2.23 usec per loop

    >>> import timeit
    >>> s = """\
    ... try:
    ...     str.__nonzero__
    ... except AttributeError:
    ...     pass
    ... """
    >>> t = timeit.Timer(stmt=s)
    >>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
    17.09 usec/pass
    >>> s = """\
    ... if hasattr(str, '__nonzero__'): pass
    ... """
    >>> t = timeit.Timer(stmt=s)
    >>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
    4.85 usec/pass
    >>> s = """\
    ... try:
    ...     int.__nonzero__
    ... except AttributeError:
    ...     pass
    ... """
    >>> t = timeit.Timer(stmt=s)
    >>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
    1.97 usec/pass
    >>> s = """\
    ... if hasattr(int, '__nonzero__'): pass
    ... """
    >>> t = timeit.Timer(stmt=s)
    >>> print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)
    3.15 usec/pass

To give the timeit module access to functions you define,
you can pass a `setup' parameter which contains an import statement:

    def test():
        "Stupid test function"
        L = []
        for i in range(100):
            L.append(i)

    if __name__=='__main__':
        from timeit import Timer
        t = Timer("test()", "from __main__ import test")
        print t.timeit()

5.26.7 `trace' -- Trace or track Python statement execution
-----------------------------------------------------------

The trace module allows you to trace program execution,
generate annotated statement coverage listings, print caller/callee
relationships and list functions executed during a program run.  It can
be used in another program or from the command line.

  Latest version of the trace module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/trace.py?view=markup)

5.26.7.1 Command Line Usage
...........................

The trace module can be invoked from the command line.  It
can be as simple as

    python -m trace --count somefile.py ...

The above will generate annotated listings of all Python modules
imported during the execution of `somefile.py'.

  The following command-line arguments are supported:

`--trace', see -t.
     Display lines as they are executed.

`--count', see -c.
     Produce a set of  annotated listing files upon program completion
     that shows how many times each statement was executed.

`--report', `-r'
     Produce an annotated list from an earlier program run that used the
     `--count' and `--file' arguments.

`--no-report', `-R'
     Do not generate annotated listings.  This is useful if you intend
     to make several runs with `--count' then produce a single set of
     annotated listings at the end.

`--listfuncs', `-l'
     List the functions executed by running the program.

`--trackcalls', `-T'
     Generate calling relationships exposed by running the program.

`--file', `-f'
     Name a file containing (or to contain) counts.

`--coverdir', `-C'
     Name a directory in which to save annotated listing files.

`--missing', see -m.
     When generating annotated listings, mark lines which were not
     executed with '`>>>>>>''.

`--summary', see -s.
     When using `--count' or `--report', write a brief summary to
     stdout for each file processed.

`--ignore-module'
     Accepts comma separated list of module names. Ignore each of the
     named module and its submodules (if it is a package).  May be given
     multiple times.

`--ignore-dir'
     Ignore all modules and packages in the named directory and
     subdirectories (multiple directories can be joined by os.pathsep).
     May be given multiple times.

5.26.7.2 Programming Interface
..............................

 -- Class: trace.Trace ([count=1[, trace=1[, countfuncs=0[,
          countcallers=0[, ignoremods=()[, ignoredirs=()[,
          infile=None[, outfile=None[, timing=False]]]]]]]]])
     Create an object to trace execution of a single statement or
     expression. All parameters are optional.  _count_ enables counting
     of line numbers. _trace_ enables line execution tracing.
     _countfuncs_ enables listing of the functions called during the
     run.  _countcallers_ enables call relationship tracking.
     _ignoremods_ is a list of modules or packages to ignore.
     _ignoredirs_ is a list of directories whose modules or packages
     should be ignored.  _infile_ is the file from which to read stored
     count information.  _outfile_ is a file in which to write updated
     count information. _timing_ enables a timestamp relative to when
     tracing was started to be displayed.

 -- Method: Trace.run (cmd)
     Run _cmd_ under control of the Trace object with the current
     tracing parameters.

 -- Method: Trace.runctx (cmd[, globals=None[, locals=None]])
     Run _cmd_ under control of the Trace object with the current
     tracing parameters in the defined global and local environments.
     If not defined, _globals_ and _locals_ default to empty
     dictionaries.

 -- Method: Trace.runfunc (func, *args, **kwds)
     Call _func_ with the given arguments under control of the
     Trace object with the current tracing parameters.

  This is a simple example showing the use of this module:

    import sys
    import trace

    # create a Trace object, telling it what to ignore, and whether to
    # do tracing or line-counting or both.
    tracer = trace.Trace(
        ignoredirs=[sys.prefix, sys.exec_prefix],
        trace=0,
        count=1)

    # run the new command using the given tracer
    tracer.run('main()')

    # make a report, placing output in /tmp
    r = tracer.results()
    r.write_results(show_missing=True, coverdir="/tmp")

5.27 Python Runtime Services
============================

The modules described in this chapter provide a wide range of services
related to the Python interpreter and its interaction with its
environment.  Here's an overview:

5.27.1 `sys' -- System-specific parameters and functions
--------------------------------------------------------

This module provides access to some variables used or maintained by the
interpreter and to functions that interact strongly with the
interpreter. It is always available.

 -- Data: sys.argv
     The list of command line arguments passed to a Python script.
     `argv[0]' is the script name (it is operating system dependent
     whether this is a full pathname or not).  If the command was
     executed using the -c command line option to the
     interpreter, `argv[0]' is set to the string `'-c''.  If no script
     name was passed to the Python interpreter, `argv[0]' is the empty
     string.

     To loop over the standard input, or the list of files given on the
     command line, see the fileinput module.

 -- Data: sys.byteorder
     An indicator of the native byte order.  This will have the value
     `'big'' on big-endian (most-significant byte first) platforms, and
     `'little'' on little-endian (least-significant byte first)
     platforms.

     New in version 2.0.

 -- Data: sys.subversion
     A triple (repo, branch, version) representing the Subversion
     information of the Python interpreter. _repo_ is the name of the
     repository, `'CPython''.  _branch_ is a string of one of the forms
     `'trunk'', `'branches/name'' or `'tags/name''. _version_ is the
     output of `svnversion', if the interpreter was built from a
     Subversion checkout; it contains the revision number (range) and
     possibly a trailing 'M' if there were local modifications. If the
     tree was exported (or svnversion was not available), it is the
     revision of `Include/patchlevel.h' if the branch is a tag.
     Otherwise, it is `None'.

     New in version 2.5.

 -- Data: sys.builtin_module_names
     A tuple of strings giving the names of all modules that are
     compiled into this Python interpreter.  (This information is not
     available in any other way -- `modules.keys()' only lists the
     imported modules.)

 -- Function: sys.call_tracing (func, args)
     Call `func(*args)', while tracing is enabled.  The tracing state
     is saved, and restored afterwards.  This is intended to be called
     from a debugger from a checkpoint, to recursively debug some other
     code.

 -- Data: sys.copyright
     A string containing the copyright pertaining to the Python
     interpreter.

 -- Function: sys._clear_type_cache ()
     Clear the internal type cache. The type cache is used to speed up
     attribute and method lookups. Use the function _only_ to drop
     unnecessary references during reference leak debugging.

     This function should be used for internal and specialized purposes
     only.

     New in version 2.6.

 -- Function: sys._current_frames ()
     Return a dictionary mapping each thread's identifier to the
     topmost stack frame currently active in that thread at the time
     the function is called. Note that functions in the
     traceback module can build the call stack given such a frame.

     This is most useful for debugging deadlock:  this function does
     not require the deadlocked threads' cooperation, and such threads'
     call stacks are frozen for as long as they remain deadlocked.  The
     frame returned for a non-deadlocked thread may bear no
     relationship to that thread's current activity by the time calling
     code examines the frame.

     This function should be used for internal and specialized purposes
     only.

     New in version 2.5.

 -- Data: sys.dllhandle
     Integer specifying the handle of the Python DLL. Availability:
     Windows.

 -- Function: sys.displayhook (value)
     If _value_ is not `None', this function prints it to `sys.stdout',
     and saves it in `__builtin__._'.

     `sys.displayhook' is called on the result of evaluating an
     expression.  entered in an interactive Python session.  The
     display of these values can be customized by assigning another
     one-argument function to `sys.displayhook'.

 -- Function: sys.excepthook (type, value, traceback)
     This function prints out a given traceback and exception to
     `sys.stderr'.

     When an exception is raised and uncaught, the interpreter calls
     `sys.excepthook' with three arguments, the exception class,
     exception instance, and a traceback object.  In an interactive
     session this happens just before control is returned to the
     prompt; in a Python program this happens just before the program
     exits.  The handling of such top-level exceptions can be
     customized by assigning another three-argument function to
     `sys.excepthook'.

 -- Data: sys.__displayhook__
 -- Data: sys.__excepthook__
     These objects contain the original values of `displayhook' and
     `excepthook' at the start of the program.  They are saved so that
     `displayhook' and `excepthook' can be restored in case they happen
     to get replaced with broken objects.

 -- Function: sys.exc_info ()
     This function returns a tuple of three values that give
     information about the exception that is currently being handled.
     The information returned is specific both to the current thread
     and to the current stack frame.  If the current stack frame is not
     handling an exception, the information is taken from the calling
     stack frame, or its caller, and so on until a stack frame is found
     that is handling an exception.  Here, "handling an exception" is
     defined as "executing or having executed an except clause."  For
     any stack frame, only information about the most recently handled
     exception is accessible.

     If no exception is being handled anywhere on the stack, a tuple
     containing three `None' values is returned.  Otherwise, the values
     returned are `(type, value, traceback)'.  Their meaning is: _type_
     gets the exception type of the exception being handled (a class
     object); _value_ gets the exception parameter (its _associated
     value_ or the second argument to raise, which is always
     a class instance if the exception type is a class object);
     _traceback_ gets a traceback object (see the Reference Manual)
     which encapsulates the call stack at the point where the exception
     originally occurred.

     If exc_clear() is called, this function will return
     three `None' values until either another exception is raised in
     the current thread or the execution stack returns to a frame where
     another exception is being handled.

          Warning: Assigning the _traceback_ return value to a local
          variable in a function that is handling an exception will
          cause a circular reference.  This will prevent anything
          referenced by a local variable in the same function or by the
          traceback from being garbage collected.  Since most functions
          don't need access to the traceback, the best solution is to
          use something like `exctype, value = sys.exc_info()[:2]' to
          extract only the exception type and value.  If you do need
          the traceback, make sure to delete it after use (best done
          with a try... finally statement) or
          to call exc_info() in a function that does not
          itself handle an exception.

          Note: Beginning with Python 2.2, such cycles are
          automatically reclaimed when garbage collection is enabled
          and they become unreachable, but it remains more efficient to
          avoid creating cycles.

 -- Function: sys.exc_clear ()
     This function clears all information relating to the current or
     last exception that occurred in the current thread.  After calling
     this function, see exc_info() will return three `None'
     values until another exception is raised in the current thread or
     the execution stack returns to a frame where another exception is
     being handled.

     This function is only needed in only a few obscure situations.
     These include logging and error handling systems that report
     information on the last or current exception.  This function can
     also be used to try to free resources and trigger object
     finalization, though no guarantee is made as to what objects will
     be freed, if any.

     New in version 2.3.

 -- Data: sys.exc_type
 -- Data: sys.exc_value
 -- Data: sys.exc_traceback
     Deprecated since version 1.5: Use exc_info() instead.

     Since they are global variables, they are not specific to the
     current thread, so their use is not safe in a multi-threaded
     program.  When no exception is being handled, `exc_type' is set to
     `None' and the other two are undefined.

 -- Data: sys.exec_prefix
     A string giving the site-specific directory prefix where the
     platform-dependent Python files are installed; by default, this is
     also `'/usr/local''.  This can be set at build time with the
     `--exec-prefix' argument to the *configure* script.  Specifically,
     all configuration files (e.g. the `pyconfig.h' header file) are
     installed in the directory `exec_prefix +
     '/lib/pythonversion/config'', and shared library modules are
     installed in `exec_prefix + '/lib/pythonversion/lib-dynload'',
     where _version_ is equal to `version[:3]'.

 -- Data: sys.executable
     A string giving the name of the executable binary for the Python
     interpreter, on systems where this makes sense.

 -- Function: sys.exit ([arg])
     Exit from Python.  This is implemented by raising the
     SystemExit.  exception, so cleanup actions specified by
     finally clauses of try.  statements are honored, and it
     is possible to intercept the exit attempt at an outer level.

     The optional argument _arg_ can be an integer giving the exit
     status (defaulting to zero), or another type of object.  If it is
     an integer, zero is considered "successful termination" and any
     nonzero value is considered "abnormal termination" by shells and
     the like.  Most systems require it to be in the range 0-127, and
     produce undefined results otherwise.  Some systems have a
     convention for assigning specific meanings to specific exit codes,
     but these are generally underdeveloped; Unix programs generally
     use 2 for command line syntax errors and 1 for all other kind of
     errors.  If another type of object is passed, `None' is equivalent
     to passing zero, and any other object is printed to stderr
     and results in an exit code of 1.  In particular,
     `sys.exit("some error message")' is a quick way to exit a program
     when an error occurs.

     Since exit() ultimately "only" raises an exception, it
     will only exit the process when called from the main thread, and
     the exception is not intercepted.

 -- Data: sys.exitfunc
     This value is not actually defined by the module, but can be set
     by the user (or by a program) to specify a clean-up action at
     program exit.  When set, it should be a parameterless function.
     This function will be called when the interpreter exits.  Only one
     function may be installed in this way; to allow multiple functions
     which will be called at termination, use the atexit.
     module.

          Note: The exit function is not called when the program is
          killed by a signal, when a Python fatal internal error is
          detected, or when `os._exit()' is called.

     Deprecated since version 2.4: Use atexit instead.

 -- Data: sys.flags
     The struct sequence _flags_ exposes the status of command line
     flags. The attributes are read only.

     attribute                          flag
     ----------------------------------------------------------------------------------
     `debug'                            -d
     `py3k_warning'                     -3
     `division_warning'                 -Q
     `division_new'                     -Qnew
     inspect.                 -i
     `interactive'                      -i
     `optimize'                         -O or -OO
     dont_write_bytecode.   -B
     `no_user_site'                     -s
     `no_site'                          -S
     `ignore_environment'               -E
     `tabcheck'                         -t or -tt
     `verbose'                          -v
     unicode.                -U
     `bytes_warning'                    -b

     New in version 2.6.

 -- Data: sys.float_info
     A structseq holding information about the float type. It contains
     low level information about the precision and internal
     representation.  The values correspond to the various
     floating-point constants defined in the standard header file
     `float.h' for the 'C' programming language; see section 5.2.4.2.2
     of the 1999 ISO/IEC C standard [C99], 'Characteristics
     of floating types', for details.

     attribute                 float.h macro        explanation
     ------------------------------------------------------------------------------------------------------
     `epsilon'                 DBL_EPSILON          difference between 1 and the least value greater than
                                                    1 that is representable as a float
     `dig'                     DBL_DIG              maximum number of decimal digits that can be
                                                    faithfully represented in a float;  see below
     `mant_dig'                DBL_MANT_DIG         float precision: the number of base-`radix' digits in
                                                    the significand of a float
     max.           DBL_MAX              maximum representable finite float
     `max_exp'                 DBL_MAX_EXP          maximum integer e such that `radix**(e-1)' is a
                                                    representable finite float
     `max_10_exp'              DBL_MAX_10_EXP       maximum integer e such that `10**e' is in the range
                                                    of representable finite floats
     min.           DBL_MIN              minimum positive normalized float
     `min_exp'                 DBL_MIN_EXP          minimum integer e such that `radix**(e-1)' is a
                                                    normalized float
     `min_10_exp'              DBL_MIN_10_EXP       minimum integer e such that `10**e' is a normalized
                                                    float
     `radix'                   FLT_RADIX            radix of exponent representation
     `rounds'                  FLT_ROUNDS           constant representing rounding mode used for
                                                    arithmetic operations

     The attribute `sys.float_info.dig' needs further explanation.  If
     `s' is any string representing a decimal number with at most
     `sys.float_info.dig' significant digits, then converting `s' to a
     float and back again will recover a string representing the same
     decimal value:

         >>> import sys
         >>> sys.float_info.dig
         15
         >>> s = '3.14159265358979'    # decimal string with 15 significant digits
         >>> format(float(s), '.15g')  # convert to float and back -> same value
         '3.14159265358979'

     But for strings with more than `sys.float_info.dig' significant
     digits, this isn't always true:

         >>> s = '9876543211234567'    # 16 significant digits is too many!
         >>> format(float(s), '.16g')  # conversion changes value
         '9876543211234568'

     New in version 2.6.

 -- Data: sys.float_repr_style
     A string indicating how the repr() function behaves for
     floats.  If the string has value `'short'' then for a finite float
     `x', `repr(x)' aims to produce a short string with the property
     that `float(repr(x)) == x'.  This is the usual behaviour in Python
     2.7 and later.  Otherwise, `float_repr_style' has value `'legacy''
     and `repr(x)' behaves in the same way as it did in versions of
     Python prior to 2.7.

     New in version 2.7.

 -- Function: sys.getcheckinterval ()
     Return the interpreter's "check interval"; see
     setcheckinterval().

     New in version 2.3.

 -- Function: sys.getdefaultencoding ()
     Return the name of the current default string encoding used by the
     Unicode implementation.

     New in version 2.0.

 -- Function: sys.getdlopenflags ()
     Return the current value of the flags that are used for `dlopen()'
     calls.  The flag constants are defined in the dl and
     `DLFCN' modules.  Availability: Unix.

     New in version 2.2.

 -- Function: sys.getfilesystemencoding ()
     Return the name of the encoding used to convert Unicode filenames
     into system file names, or `None' if the system default encoding
     is used. The result value depends on the operating system:

        * On Mac OS X, the encoding is `'utf-8''.

        * On Unix, the encoding is the user's preference according to
          the result of nl_langinfo(CODESET), or `None' if the
          `nl_langinfo(CODESET)' failed.

        * On Windows NT+, file names are Unicode natively, so no
          conversion is performed. See getfilesystemencoding().
          still returns `'mbcs'', as this is the encoding that
          applications should use when they explicitly want to convert
          Unicode strings to byte strings that are equivalent when used
          as file names.

        * On Windows 9x, the encoding is `'mbcs''.

     New in version 2.3.

 -- Function: sys.getrefcount (object)
     Return the reference count of the _object_.  The count returned is
     generally one higher than you might expect, because it includes
     the (temporary) reference as an argument to getrefcount().

 -- Function: sys.getrecursionlimit ()
     Return the current value of the recursion limit, the maximum depth
     of the Python interpreter stack.  This limit prevents infinite
     recursion from causing an overflow of the C stack and crashing
     Python.  It can be set by setrecursionlimit().

 -- Function: sys.getsizeof (object[, default])
     Return the size of an object in bytes. The object can be any type
     of object. All built-in objects will return correct results, but
     this does not have to hold true for third-party extensions as it
     is implementation specific.

     If given, _default_ will be returned if the object does not
     provide means to retrieve the size.  Otherwise a TypeError
     will be raised.

     getsizeof() calls the object's `__sizeof__' method
     and adds an additional garbage collector overhead if the object is
     managed by the garbage collector.

     New in version 2.6.

 -- Function: sys._getframe ([depth])
     Return a frame object from the call stack.  If optional integer
     _depth_ is given, return the frame object that many calls below
     the top of the stack.  If that is deeper than the call stack,
     see ValueError is raised.  The default for _depth_ is
     zero, returning the frame at the top of the call stack.

     *CPython implementation detail:* This function should be used for
     internal and specialized purposes only.  It is not guaranteed to
     exist in all implementations of Python.

 -- Function: sys.getprofile ()
     Get the profiler function as set by setprofile().

     New in version 2.6.

 -- Function: sys.gettrace ()
     Get the trace function as set by settrace().

     *CPython implementation detail:* The gettrace().
     function is intended only for implementing debuggers, profilers,
     coverage tools and the like.  Its behavior is part of the
     implementation platform, rather than part of the language
     definition, and thus may not be available in all Python
     implementations.

     New in version 2.6.

 -- Function: sys.getwindowsversion ()
     Return a named tuple describing the Windows version currently
     running.  The named elements are _major_, _minor_, _build_,
     _platform_, _service_pack_, _service_pack_minor_,
     _service_pack_major_, _suite_mask_, and _product_type_.
     _service_pack_ contains a string while all other values are
     integers. The components can also be accessed by name, so
     `sys.getwindowsversion()[0]' is equivalent to
     `sys.getwindowsversion().major'. For compatibility with prior
     versions, only the first 5 elements are retrievable by indexing.

     _platform_ may be one of the following values:

     Constant                                      Platform
     ----------------------------------------------------------------------------
     `0 (VER_PLATFORM_WIN32s)'                     Win32s on Windows 3.1
     `1 (VER_PLATFORM_WIN32_WINDOWS)'              Windows 95/98/ME
     `2 (VER_PLATFORM_WIN32_NT)'                   Windows NT/2000/XP/x64
     `3 (VER_PLATFORM_WIN32_CE)'                   Windows CE

     _product_type_ may be one of the following values:

     Constant                                    Meaning
     ----------------------------------------------------------------------------------
     `1 (VER_NT_WORKSTATION)'                    The system is a workstation.
     `2 (VER_NT_DOMAIN_CONTROLLER)'              The system is a domain controller.
     `3 (VER_NT_SERVER)'                         The system is a server, but not a
                                                 domain controller.

     This function wraps the Win32 `GetVersionEx()' function; see the
     Microsoft documentation on `OSVERSIONINFOEX()' for more information
     about these fields.

     Availability: Windows.

     New in version 2.3.

     Changed in version 2.7: Changed to a named tuple and added
     _service_pack_minor_, _service_pack_major_, _suite_mask_, and
     _product_type_.

 -- Data: sys.hexversion
     The version number encoded as a single integer.  This is
     guaranteed to increase with each version, including proper support
     for non-production releases.  For example, to test that the Python
     interpreter is at least version 1.5.2, use:

         if sys.hexversion >= 0x010502F0:
             # use some advanced feature
             ...
         else:
             # use an alternative implementation or warn the user
             ...

     This is called `hexversion' since it only really looks meaningful
     when viewed as the result of passing it to the built-in
     hex() function.  The `version_info' value may be used for a
     more human-friendly encoding of the same information.

     New in version 1.5.2.

 -- Data: sys.long_info
     A struct sequence that holds information about Python's internal
     representation of integers.  The attributes are read only.

     attribute                     explanation
     ---------------------------------------------------------------------------------
     `bits_per_digit'              number of bits held in each digit.  Python
                                   integers are stored internally in base
                                   `2**long_info.bits_per_digit'
     `sizeof_digit'                size in bytes of the C type used to represent a
                                   digit

     New in version 2.7.

 -- Data: sys.last_type
 -- Data: sys.last_value
 -- Data: sys.last_traceback
     These three variables are not always defined; they are set when an
     exception is not handled and the interpreter prints an error
     message and a stack traceback.  Their intended use is to allow an
     interactive user to import a debugger module and engage in
     post-mortem debugging without having to re-execute the command
     that caused the error.  (Typical use is `import pdb; pdb.pm()' to
     enter the post-mortem debugger; see chapter pdb -- The Python Debugger
     for more information.)

     The meaning of the variables is the same as that of the return
     values from exc_info() above.  (Since there is only
     one interactive thread, thread-safety is not a concern for these
     variables, unlike for `exc_type' etc.)

 -- Data: sys.maxint
     The largest positive integer supported by Python's regular integer
     type.  This is at least 2**31-1.  The largest negative integer is
     `-maxint-1' -- the asymmetry results from the use of 2's
     complement binary arithmetic.

 -- Data: sys.maxsize
     The largest positive integer supported by the platform's
     Py_ssize_t type, and thus the maximum size lists, strings, dicts,
     and many other containers can have.

 -- Data: sys.maxunicode
     An integer giving the largest supported code point for a Unicode
     character.  The value of this depends on the configuration option
     that specifies whether Unicode characters are stored as UCS-2 or
     UCS-4.

 -- Data: sys.meta_path
     A list of finder objects that have their
     `find_module()' methods called to see if one of the objects can
     find the module to be imported. The `find_module()' method is
     called at least with the absolute name of the module being
     imported. If the module to be imported is contained in package
     then the parent package's `__path__' attribute is passed in as a
     second argument. The method returns `None' if the module cannot be
     found, else returns a loader.

     sys.meta_path is searched before any implicit default
     finders or sys.path.

     See PEP 302 (http://www.python.org/dev/peps/pep-0302) for the
     original specification.

 -- Data: sys.modules
     This is a dictionary that maps module names to modules which have
     already been loaded.  This can be manipulated to force reloading
     of modules and other tricks.  Note that removing a module from
     this dictionary is _not_ the same as calling reload().
     on the corresponding module object.

 -- Data: sys.path
     A list of strings that specifies the search path for modules.
     Initialized from the environment variable PYTHONPATH,
     plus an installation-dependent default.

     As initialized upon program startup, the first item of this list,
     `path[0]', is the directory containing the script that was used to
     invoke the Python interpreter.  If the script directory is not
     available (e.g.  if the interpreter is invoked interactively or if
     the script is read from standard input), `path[0]' is the empty
     string, which directs Python to search modules in the current
     directory first.  Notice that the script directory is inserted
     _before_ the entries inserted as a result of PYTHONPATH.

     A program is free to modify this list for its own purposes.

     Changed in version 2.3: Unicode strings are no longer ignored.

     Module site. This describes how to use .pth files to
     extend sys.path.

 -- Data: sys.path_hooks
     A list of callables that take a path argument to try to create a
     finder for the path. If a finder can be created, it is
     to be returned by the callable, else raise ImportError.

     Originally specified in PEP 302
     (http://www.python.org/dev/peps/pep-0302).

 -- Data: sys.path_importer_cache
     A dictionary acting as a cache for finder objects. The
     keys are paths that have been passed to sys.path_hooks.
     and the values are the finders that are found. If a path is a
     valid file system path but no explicit finder is found on
     sys.path_hooks then `None' is stored to represent the
     implicit default finder should be used. If the path is not an
     existing path then imp.NullImporter is set.

     Originally specified in PEP 302
     (http://www.python.org/dev/peps/pep-0302).

 -- Data: sys.platform
     This string contains a platform identifier that can be used to
     append platform-specific components to sys.path, for
     instance.

     For Unix systems, this is the lowercased OS name as returned by
     `uname -s' with the first part of the version as returned by
     `uname -r' appended, e.g. `'sunos5'' or `'linux2'', _at the time
     when Python was built_.  For other systems, the values are:

     System               platform value
     -----------------------------------------------------
     Windows              `'win32''
     Windows/Cygwin       `'cygwin''
     Mac OS X             `'darwin''
     OS/2                 `'os2''
     OS/2 EMX             `'os2emx''
     RiscOS               `'riscos''
     AtheOS               `'atheos''

 -- Data: sys.prefix
     A string giving the site-specific directory prefix where the
     platform independent Python files are installed; by default, this
     is the string `'/usr/local''.  This can be set at build time with
     the `--prefix' argument to the *configure* script.  The main
     collection of Python library modules is installed in the directory
     `prefix + '/lib/pythonversion'' while the platform independent
     header files (all except `pyconfig.h') are stored in `prefix +
     '/include/pythonversion'', where _version_ is equal to
     `version[:3]'.

 -- Data: sys.ps1
 -- Data: sys.ps2
     Strings specifying the primary and secondary prompt of the
     interpreter.  These are only defined if the interpreter is in
     interactive mode.  Their initial values in this case are `'>>> ''
     and `'... ''.  If a non-string object is assigned to either
     variable, its str() is re-evaluated each time the
     interpreter prepares to read a new interactive command; this can
     be used to implement a dynamic prompt.

 -- Data: sys.py3kwarning
     Bool containing the status of the Python 3.0 warning flag. It's
     `True' when Python is started with the -3 option.  (This should be
     considered read-only; setting it to a different value doesn't have
     an effect on Python 3.0 warnings.)

     New in version 2.6.

 -- Data: sys.dont_write_bytecode
     If this is true, Python won't try to write `.pyc' or `.pyo' files
     on the import of source modules.  This value is initially set to
     `True' or `False' depending on the `-B' command line option and
     the `PYTHONDONTWRITEBYTECODE' environment variable, but you can
     set it yourself to control bytecode file generation.

     New in version 2.6.

 -- Function: sys.setcheckinterval (interval)
     Set the interpreter's "check interval".  This integer value
     determines how often the interpreter checks for periodic things
     such as thread switches and signal handlers.  The default is
     `100', meaning the check is performed every 100 Python virtual
     instructions. Setting it to a larger value may increase
     performance for programs using threads.  Setting it to a value
     `<=' 0 checks every virtual instruction, maximizing responsiveness
     as well as overhead.

 -- Function: sys.setdefaultencoding (name)
     Set the current default string encoding used by the Unicode
     implementation.  If _name_ does not match any available encoding,
     see LookupError is raised.  This function is only intended
     to be used by the site module implementation and,
     where needed, by `sitecustomize'.  Once used by the site
     module, it is removed from the sys module's
     namespace.

     New in version 2.0.

 -- Function: sys.setdlopenflags (n)
     Set the flags used by the interpreter for `dlopen()' calls, such
     as when the interpreter loads extension modules.  Among other
     things, this will enable a lazy resolving of symbols when
     importing a module, if called as `sys.setdlopenflags(0)'.  To
     share symbols across extension modules, call as
     `sys.setdlopenflags(dl.RTLD_NOW | dl.RTLD_GLOBAL)'.  Symbolic
     names for the flag modules can be either found in the dl
     module, or in the `DLFCN' module. If `DLFCN' is not available,
     it can be generated from `/usr/include/dlfcn.h' using the *h2py*
     script. Availability: Unix.

     New in version 2.2.

 -- Function: sys.setprofile (profilefunc)
     Set the system's profile function, which allows you to implement a
     Python source code profiler in Python.  See chapter The Python Profilers
     for more information on the Python
     profiler.  The system's profile function is called similarly to the
     system's trace function (see settrace()), but it isn't
     called for each executed line of code (only on call and return,
     but the return event is reported even when an exception has been
     set).  The function is thread-specific, but there is no way for
     the profiler to know about context switches between threads, so it
     does not make sense to use this in the presence of multiple
     threads. Also, its return value is not used, so it can simply
     return `None'.

 -- Function: sys.setrecursionlimit (limit)
     Set the maximum depth of the Python interpreter stack to _limit_.
     This limit prevents infinite recursion from causing an overflow of
     the C stack and crashing Python.

     The highest possible limit is platform-dependent.  A user may need
     to set the limit higher when she has a program that requires deep
     recursion and a platform that supports a higher limit.  This
     should be done with care, because a too-high limit can lead to a
     crash.

 -- Function: sys.settrace (tracefunc)
     Set the system's trace function, which allows you to implement a
     Python source code debugger in Python.  The function is
     thread-specific; for a debugger to support multiple threads, it
     must be registered using settrace() for each thread
     being debugged.

     Trace functions should have three arguments: _frame_, _event_, and
     _arg_. _frame_ is the current stack frame.  _event_ is a string:
     `'call'', `'line'', `'return'', `'exception'', `'c_call'',
     `'c_return'', or `'c_exception''. _arg_ depends on the event type.

     The trace function is invoked (with _event_ set to `'call'')
     whenever a new local scope is entered; it should return a
     reference to a local trace function to be used that scope, or
     `None' if the scope shouldn't be traced.

     The local trace function should return a reference to itself (or
     to another function for further tracing in that scope), or `None'
     to turn off tracing in that scope.

     The events have the following meaning:

    `'call''
          A function is called (or some other code block entered).  The
          global trace function is called; _arg_ is `None'; the return
          value specifies the local trace function.

    `'line''
          The interpreter is about to execute a new line of code or
          re-execute the condition of a loop.  The local trace function
          is called; _arg_ is `None'; the return value specifies the
          new local trace function.  See `Objects/lnotab_notes.txt' for
          a detailed explanation of how this works.

    `'return''
          A function (or other code block) is about to return.  The
          local trace function is called; _arg_ is the value that will
          be returned, or `None' if the event is caused by an exception
          being raised.  The trace function's return value is ignored.

    `'exception''
          An exception has occurred.  The local trace function is
          called; _arg_ is a tuple `(exception, value, traceback)'; the
          return value specifies the new local trace function.

    `'c_call''
          A C function is about to be called.  This may be an extension
          function or a built-in.  _arg_ is the C function object.

    `'c_return''
          A C function has returned. _arg_ is the C function object.

    `'c_exception''
          A C function has raised an exception.  _arg_ is the C
          function object.

     Note that as an exception is propagated down the chain of callers,
     an `'exception'' event is generated at each level.

     For more information on code and frame objects, refer to The standard type hierarchy.

     *CPython implementation detail:* The settrace().
     function is intended only for implementing debuggers, profilers,
     coverage tools and the like.  Its behavior is part of the
     implementation platform, rather than part of the language
     definition, and thus may not be available in all Python
     implementations.

 -- Function: sys.settscdump (on_flag)
     Activate dumping of VM measurements using the Pentium timestamp
     counter, if _on_flag_ is true. Deactivate these dumps if _on_flag_
     is off. The function is available only if Python was compiled with
     `--with-tsc'. To understand the output of this dump, read
     `Python/ceval.c' in the Python sources.

     New in version 2.4.

     *CPython implementation detail:* This function is intimately bound
     to CPython implementation details and thus not likely to be
     implemented elsewhere.

 -- Data: sys.stdin
 -- Data: sys.stdout
 -- Data: sys.stderr
     File objects corresponding to the interpreter's standard input,
     output and error streams.  `stdin' is used for all interpreter
     input except for scripts but including calls to input()
     and raw_input().  `stdout' is used for the output
     of print and expression statements and
     for the prompts of input() and raw_input().
     The interpreter's own prompts and (almost all of) its error
     messages go to `stderr'.  `stdout' and `stderr' needn't be
     built-in file objects: any object is acceptable as long as it has
     a `write()' method that takes a string argument.  (Changing these
     objects doesn't affect the standard I/O streams of processes
     executed by os.popen(), os.system() or the
     `exec*()' family of functions in the os module.)

 -- Data: sys.__stdin__
 -- Data: sys.__stdout__
 -- Data: sys.__stderr__
     These objects contain the original values of `stdin', `stderr' and
     `stdout' at the start of the program.  They are used during
     finalization, and could be useful to print to the actual standard
     stream no matter if the `sys.std*' object has been redirected.

     It can also be used to restore the actual files to known working
     file objects in case they have been overwritten with a broken
     object.  However, the preferred way to do this is to explicitly
     save the previous stream before replacing it, and restore the
     saved object.

 -- Data: sys.tracebacklimit
     When this variable is set to an integer value, it determines the
     maximum number of levels of traceback information printed when an
     unhandled exception occurs.  The default is `1000'.  When set to
     `0' or less, all traceback information is suppressed and only the
     exception type and value are printed.

 -- Data: sys.version
     A string containing the version number of the Python interpreter
     plus additional information on the build number and compiler used.
     This string is displayed when the interactive interpreter is
     started.  Do not extract version information out of it, rather,
     use version_info and the functions provided by the
     platform module.

 -- Data: sys.api_version
     The C API version for this interpreter.  Programmers may find this
     useful when debugging version conflicts between Python and
     extension modules.

     New in version 2.3.

 -- Data: sys.version_info
     A tuple containing the five components of the version number:
     _major_, _minor_, _micro_, _releaselevel_, and _serial_.  All
     values except _releaselevel_ are integers; the release level is
     `'alpha'', `'beta'', `'candidate'', or `'final''.  The
     `version_info' value corresponding to the Python version 2.0 is
     `(2, 0, 0, 'final', 0)'.  The components can also be accessed by
     name, so `sys.version_info[0]' is equivalent to
     `sys.version_info.major' and so on.

     New in version 2.0.

     Changed in version 2.7: Added named component attributes

 -- Data: sys.warnoptions
     This is an implementation detail of the warnings framework; do not
     modify this value.  Refer to the warnings module for
     more information on the warnings framework.

 -- Data: sys.winver
     The version number used to form registry keys on Windows
     platforms. This is stored as string resource 1000 in the Python
     DLL.  The value is normally the first three characters of
     version.  It is provided in the sys.  module for
     informational purposes; modifying this value has no effect on the
     registry keys used by Python. Availability: Windows.

Citations
.........

     (C99) ISO/IEC 9899:1999.  "Programming languages - C."  A public
     draft of this standard is available at
     <http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf> .

5.27.2 `sysconfig' -- Provide access to Python's configuration information
--------------------------------------------------------------------------

New in version 2.7.

  The sysconfig module provides access to Python's
configuration information like the list of installation paths and the
configuration variables relevant for the current platform.

5.27.2.1 Configuration variables
................................

A Python distribution contains a `Makefile' and a `pyconfig.h' header
file that are necessary to build both the Python binary itself and
third-party C extensions compiled using distutils.

  sysconfig puts all variables found in these files in a
dictionary that can be accessed using get_config_vars() or
get_config_var().

  Notice that on Windows, it's a much smaller set.

 -- Function: sysconfig.get_config_vars (*args)
     With no arguments, return a dictionary of all configuration
     variables relevant for the current platform.

     With arguments, return a list of values that result from looking
     up each argument in the configuration variable dictionary.

     For each argument, if the value is not found, return `None'.

 -- Function: sysconfig.get_config_var (name)
     Return the value of a single variable _name_. Equivalent to
     `get_config_vars().get(name)'.

     If _name_ is not found, return `None'.

  Example of usage:

    >>> import sysconfig
    >>> sysconfig.get_config_var('Py_ENABLE_SHARED')
    0
    >>> sysconfig.get_config_var('LIBDIR')
    '/usr/local/lib'
    >>> sysconfig.get_config_vars('AR', 'CXX')
    ['ar', 'g++']

5.27.2.2 Installation paths
...........................

Python uses an installation scheme that differs depending on the
platform and on the installation options.  These schemes are stored in
sysconfig under unique identifiers based on the value
returned by os.name.

  Every new component that is installed using distutils or a
Distutils-based system will follow the same scheme to copy its file in
the right places.

  Python currently supports seven schemes:

   - _posix_prefix_: scheme for Posix platforms like Linux or Mac OS X.
     This is the default scheme used when Python or a component is
     installed.

   - _posix_home_: scheme for Posix platforms used when a _home_ option
     is used upon installation.  This scheme is used when a component
     is installed through Distutils with a specific home prefix.

   - _posix_user_: scheme for Posix platforms used when a component is
     installed through Distutils and the _user_ option is used.  This
     scheme defines paths located under the user home directory.

   - _nt_: scheme for NT platforms like Windows.

   - _nt_user_: scheme for NT platforms, when the _user_ option is used.

   - _os2_: scheme for OS/2 platforms.

   - _os2_home_: scheme for OS/2 patforms, when the _user_ option is
     used.

  Each scheme is itself composed of a series of paths and each path has
a unique identifier.  Python currently uses eight paths:

   - _stdlib_: directory containing the standard Python library files
     that are not platform-specific.

   - _platstdlib_: directory containing the standard Python library
     files that are platform-specific.

   - _platlib_: directory for site-specific, platform-specific files.

   - _purelib_: directory for site-specific, non-platform-specific
     files.

   - _include_: directory for non-platform-specific header files.

   - _platinclude_: directory for platform-specific header files.

   - _scripts_: directory for script files.

   - _data_: directory for data files.

  sysconfig provides some functions to determine these
paths.

 -- Function: sysconfig.get_scheme_names ()
     Return a tuple containing all schemes currently supported in
     sysconfig.

 -- Function: sysconfig.get_path_names ()
     Return a tuple containing all path names currently supported in
     sysconfig.

 -- Function: sysconfig.get_path (name[, scheme[, vars[, expand]]])
     Return an installation path corresponding to the path _name_, from
     the install scheme named _scheme_.

     _name_ has to be a value from the list returned by
     get_path_names().

     sysconfig stores installation paths corresponding to
     each path name, for each platform, with variables to be expanded.
     For instance the _stdlib_ path for the _nt_ scheme is:
     `{base}/Lib'.

     get_path() will use the variables returned by
     get_config_vars().  to expand the path.  All variables have
     default values for each platform so one may call this function and
     get the default value.

     If _scheme_ is provided, it must be a value from the list returned
     by get_path_names().  Otherwise, the default scheme
     for the current platform is used.

     If _vars_ is provided, it must be a dictionary of variables that
     will update the dictionary return by get_config_vars().

     If _expand_ is set to `False', the path will not be expanded using
     the variables.

     If _name_ is not found, return `None'.

 -- Function: sysconfig.get_paths ([scheme[, vars[, expand]]])
     Return a dictionary containing all installation paths
     corresponding to an installation scheme. See get_path()
     for more information.

     If _scheme_ is not provided, will use the default scheme for the
     current platform.

     If _vars_ is provided, it must be a dictionary of variables that
     will update the dictionary used to expand the paths.

     If _expand_ is set to False, the paths will not be expanded.

     If _scheme_ is not an existing scheme, see get_paths().
     will raise a KeyError.

5.27.2.3 Other functions
........................

 -- Function: sysconfig.get_python_version ()
     Return the `MAJOR.MINOR' Python version number as a string.
     Similar to `sys.version[:3]'.

 -- Function: sysconfig.get_platform ()
     Return a string that identifies the current platform.

     This is used mainly to distinguish platform-specific build
     directories and platform-specific built distributions.  Typically
     includes the OS name and version and the architecture (as supplied
     by os.uname()), although the exact information
     included depends on the OS; e.g. for IRIX the architecture isn't
     particularly important (IRIX only runs on SGI hardware), but for
     Linux the kernel version isn't particularly important.

     Examples of returned values:

        - linux-i586

        - linux-alpha (?)

        - solaris-2.6-sun4u

        - irix-5.3

        - irix64-6.2

     Windows will return one of:

        - win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64,
          EM64T, etc)

        - win-ia64 (64bit Windows on Itanium)

        - win32 (all others - specifically, sys.platform is returned)

     Mac OS X can return:

        - macosx-10.6-ppc

        - macosx-10.4-ppc64

        - macosx-10.3-i386

        - macosx-10.4-fat

     For other non-POSIX platforms, currently just returns
     sys.platform.

 -- Function: sysconfig.is_python_build ()
     Return `True' if the current Python installation was built from
     source.

 -- Function: sysconfig.parse_config_h (fp[, vars])
     Parse a `config.h'-style file.

     _fp_ is a file-like object pointing to the `config.h'-like file.

     A dictionary containing name/value pairs is returned.  If an
     optional dictionary is passed in as the second argument, it is
     used instead of a new dictionary, and updated with the values read
     in the file.

 -- Function: sysconfig.get_config_h_filename ()
     Return the path of `pyconfig.h'.

5.27.3 `__builtin__' -- Built-in objects
----------------------------------------

This module provides direct access to all 'built-in' identifiers of
Python; for example, `__builtin__.open' is the full name for the
built-in function open().

  This module is not normally accessed explicitly by most applications,
but can be useful in modules that provide objects with the same name as
a built-in value, but in which the built-in of that name is also
needed.  For example, in a module that wants to implement an
open() function that wraps the built-in open(), this
module can be used directly:

    import __builtin__

    def open(path):
        f = __builtin__.open(path, 'r')
        return UpperCaser(f)

    class UpperCaser:
        '''Wrapper around a file that converts output to upper-case.'''

        def __init__(self, f):
            self._f = f

        def read(self, count=-1):
            return self._f.read(count).upper()

        # ...

*CPython implementation detail:* Most modules have the name
`__builtins__' (note the `'s'') made available as part of their
globals.  The value of `__builtins__' is normally either this module or
the value of this modules's `__dict__' attribute.  Since this is an
implementation detail, it may not be used by alternate implementations
of Python.

5.27.4 `future_builtins' -- Python 3 builtins
---------------------------------------------

New in version 2.6.

  This module provides functions that exist in 2.x, but have different
behavior in Python 3, so they cannot be put into the 2.x builtins
namespace.

  Instead, if you want to write code compatible with Python 3 builtins,
import them from this module, like this:

    from future_builtins import map, filter

    ... code using Python 3-style map and filter ...

The 2to3 tool that ports Python 2 code to Python 3 will
recognize this usage and leave the new builtins alone.

     Note: The Python 3 print() function is already in the
     builtins, but cannot be accessed from Python 2 code unless you use
     the appropriate future statement:

         from __future__ import print_function

  Available builtins are:

 -- Function: future_builtins.ascii (object)
     Returns the same as repr().  In Python 3, see repr()
     will return printable Unicode characters unescaped, while
     ascii() will always backslash-escape them.  Using
     future_builtins.ascii() instead of repr().
     in 2.6 code makes it clear that you need a pure ASCII return value.

 -- Function: future_builtins.filter (function, iterable)
     Works like itertools.ifilter().

 -- Function: future_builtins.hex (object)
     Works like the built-in hex(), but instead of
     __hex__() it will use the __index__() method on
     its argument to get an integer that is then converted to
     hexadecimal.

 -- Function: future_builtins.map (function, iterable, ...)
     Works like itertools.imap().

 -- Function: future_builtins.oct (object)
     Works like the built-in oct(), but instead of
     __oct__() it will use the __index__() method on
     its argument to get an integer that is then converted to octal.

 -- Function: future_builtins.zip (*iterables)
     Works like itertools.izip().

5.27.5 `__main__' -- Top-level script environment
-------------------------------------------------

This module represents the (otherwise anonymous) scope in which the
interpreter's main program executes -- commands read either from
standard input, from a script file, or from an interactive prompt.  It
is this environment in which the idiomatic "conditional script" stanza
causes a script to run:

    if __name__ == "__main__":
        main()

5.27.6 `warnings' -- Warning control
------------------------------------

New in version 2.1.

  Warning messages are typically issued in situations where it is
useful to alert the user of some condition in a program, where that
condition (normally) doesn't warrant raising an exception and
terminating the program.  For example, one might want to issue a
warning when a program uses an obsolete module.

  Python programmers issue warnings by calling the warn().
function defined in this module.  (C programmers use
PyErr_WarnEx().; see Exception Handling for details).

  Warning messages are normally written to `sys.stderr', but their
disposition can be changed flexibly, from ignoring all warnings to
turning them into exceptions.  The disposition of warnings can vary
based on the warning category (see below), the text of the warning
message, and the source location where it is issued.  Repetitions of a
particular warning for the same source location are typically
suppressed.

  There are two stages in warning control: first, each time a warning
is issued, a determination is made whether a message should be issued
or not; next, if a message is to be issued, it is formatted and printed
using a user-settable hook.

  The determination whether to issue a warning message is controlled by
the warning filter, which is a sequence of matching rules and actions.
Rules can be added to the filter by calling filterwarnings()
and reset to its default state by calling resetwarnings().

  The printing of warning messages is done by calling
showwarning(), which may be overridden; the default
implementation of this function formats the message by calling
formatwarning(), which is also available for use by custom
implementations.

5.27.6.1 Warning Categories
...........................

There are a number of built-in exceptions that represent warning
categories.  This categorization is useful to be able to filter out
groups of warnings.  The following warnings category classes are
currently defined:

Class                                  Description
-------------------------------------------------------------------------------------------
see Warning.                    This is the base class of all warning category
                                       classes.  It is a subclass of Exception.
See UserWarning.                The default category for warn().
See DeprecationWarning.         Base category for warnings about deprecated
                                       features (ignored by default).
See SyntaxWarning.              Base category for warnings about dubious syntactic
                                       features.
See RuntimeWarning.             Base category for warnings about dubious runtime
                                       features.
See FutureWarning.              Base category for warnings about constructs that
                                       will change semantically in the future.
See PendingDeprecationWarning.  Base category for warnings about features that
                                       will be deprecated in the future (ignored by
                                       default).
See ImportWarning.              Base category for warnings triggered during the
                                       process of importing a module (ignored by default).
See UnicodeWarning.             Base category for warnings related to Unicode.

  While these are technically built-in exceptions, they are documented
here, because conceptually they belong to the warnings mechanism.

  User code can define additional warning categories by subclassing one
of the standard warning categories.  A warning category must always be
a subclass of the Warning class.

  Changed in version 2.7: DeprecationWarning is ignored by
default.

5.27.6.2 The Warnings Filter
............................

The warnings filter controls whether warnings are ignored, displayed,
or turned into errors (raising an exception).

  Conceptually, the warnings filter maintains an ordered list of filter
specifications; any specific warning is matched against each filter
specification in the list in turn until a match is found; the match
determines the disposition of the match.  Each entry is a tuple of the
form (_action_, _message_, _category_, _module_, _lineno_), where:

   * _action_ is one of the following strings:

     Value               Disposition
     -----------------------------------------------------------------------
     `"error"'           turn matching warnings into exceptions
     `"ignore"'          never print matching warnings
     `"always"'          always print matching warnings
     `"default"'         print the first occurrence of matching warnings
                         for each location where the warning is issued
     `"module"'          print the first occurrence of matching warnings
                         for each module where the warning is issued
     `"once"'            print only the first occurrence of matching
                         warnings, regardless of location

   * _message_ is a string containing a regular expression that the
     warning message must match (the match is compiled to always be
     case-insensitive).

   * _category_ is a class (a subclass of Warning) of which
     the warning category must be a subclass in order to match.

   * _module_ is a string containing a regular expression that the
     module name must match (the match is compiled to be
     case-sensitive).

   * _lineno_ is an integer that the line number where the warning
     occurred must match, or `0' to match all line numbers.

  Since the Warning class is derived from the built-in
Exception.  class, to turn a warning into an error we simply
raise `category(message)'.

  The warnings filter is initialized by -W options passed
to the Python interpreter command line.  The interpreter saves the
arguments for all -W options without interpretation in
`sys.warnoptions'; the warnings module parses these when it
is first imported (invalid options are ignored, after printing a
message to `sys.stderr').

5.27.6.3 Default Warning Filters
................................

By default, Python installs several warning filters, which can be
overridden by the command-line options passed to -W and
calls to filterwarnings().

   * PendingDeprecationWarning, and ImportWarning
     are ignored.

   * `BytesWarning' is ignored unless the `-b' option is given once or
     twice; in this case this warning is either printed (`-b') or
     turned into an exception (`-bb').

5.27.6.4 Temporarily Suppressing Warnings
.........................................

If you are using code that you know will raise a warning, such as a
deprecated function, but do not want to see the warning, then it is
possible to suppress the warning using the catch_warnings.
context manager:

    import warnings

    def fxn():
        warnings.warn("deprecated", DeprecationWarning)

    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        fxn()

While within the context manager all warnings will simply be ignored.
This allows you to use known-deprecated code without having to see the
warning while not suppressing the warning for other code that might not
be aware of its use of deprecated code.  Note: this can only be
guaranteed in a single-threaded application. If two or more threads use
the catch_warnings context manager at the same time, the
behavior is undefined.

5.27.6.5 Testing Warnings
.........................

To test warnings raised by code, use the catch_warnings.
context manager. With it you can temporarily mutate the warnings filter
to facilitate your testing. For instance, do the following to capture
all raised warnings to check:

    import warnings

    def fxn():
        warnings.warn("deprecated", DeprecationWarning)

    with warnings.catch_warnings(record=True) as w:
        # Cause all warnings to always be triggered.
        warnings.simplefilter("always")
        # Trigger a warning.
        fxn()
        # Verify some things
        assert len(w) == 1
        assert issubclass(w[-1].category, DeprecationWarning)
        assert "deprecated" in str(w[-1].message)

One can also cause all warnings to be exceptions by using `error'
instead of `always'. One thing to be aware of is that if a warning has
already been raised because of a `once'/`default' rule, then no matter
what filters are set the warning will not be seen again unless the
warnings registry related to the warning has been cleared.

  Once the context manager exits, the warnings filter is restored to
its state when the context was entered. This prevents tests from
changing the warnings filter in unexpected ways between tests and
leading to indeterminate test results. The showwarning().
function in the module is also restored to its original value.  Note:
this can only be guaranteed in a single-threaded application. If two or
more threads use the catch_warnings context manager at the
same time, the behavior is undefined.

  When testing multiple operations that raise the same kind of warning,
it is important to test them in a manner that confirms each operation
is raising a new warning (e.g. set warnings to be raised as exceptions
and check the operations raise exceptions, check that the length of the
warning list continues to increase after each operation, or else delete
the previous entries from the warnings list before each new operation).

5.27.6.6 Updating Code For New Versions of Python
.................................................

Warnings that are only of interest to the developer are ignored by
default. As such you should make sure to test your code with typically
ignored warnings made visible. You can do this from the command-line by
passing `-Wd' to the interpreter (this is shorthand for `-W default').
This enables default handling for all warnings, including those that
are ignored by default.  To change what action is taken for encountered
warnings you simply change what argument is passed to -W,
e.g. `-W error'. See the -W flag for more details on what
is possible.

  To programmatically do the same as `-Wd', use:

    warnings.simplefilter('default')

Make sure to execute this code as soon as possible. This prevents the
registering of what warnings have been raised from unexpectedly
influencing how future warnings are treated.

  Having certain warnings ignored by default is done to prevent a user
from seeing warnings that are only of interest to the developer. As you
do not necessarily have control over what interpreter a user uses to
run their code, it is possible that a new version of Python will be
released between your release cycles.  The new interpreter release
could trigger new warnings in your code that were not there in an older
interpreter, e.g.  see DeprecationWarning for a module that you
are using. While you as a developer want to be notified that your code
is using a deprecated module, to a user this information is essentially
noise and provides no benefit to them.

5.27.6.7 Available Functions
............................

 -- Function: warnings.warn (message[, category[, stacklevel]])
     Issue a warning, or maybe ignore it or raise an exception.  The
     _category_ argument, if given, must be a warning category class
     (see above); it defaults to UserWarning.  Alternatively
     _message_ can be a Warning instance, in which case
     _category_ will be ignored and `message.__class__' will be used.
     In this case the message text will be `str(message)'. This
     function raises an exception if the particular warning issued is
     changed into an error by the warnings filter see above.  The
     _stacklevel_ argument can be used by wrapper functions written in
     Python, like this:

         def deprecation(message):
             warnings.warn(message, DeprecationWarning, stacklevel=2)

     This makes the warning refer to `deprecation()''s caller, rather
     than to the source of `deprecation()' itself (since the latter
     would defeat the purpose of the warning message).

 -- Function: warnings.warn_explicit (message, category, filename,
          lineno[, module[, registry[, module_globals]]])
     This is a low-level interface to the functionality of
     warn(), passing in explicitly the message, category, filename
     and line number, and optionally the module name and the registry
     (which should be the `__warningregistry__' dictionary of the
     module).  The module name defaults to the filename with `.py'
     stripped; if no registry is passed, the warning is never
     suppressed.  _message_ must be a string and _category_ a subclass
     of Warning or _message_ may be a Warning.
     instance, in which case _category_ will be ignored.

     _module_globals_, if supplied, should be the global namespace in
     use by the code for which the warning is issued.  (This argument
     is used to support displaying source for modules found in zipfiles
     or other non-filesystem import sources).

     Changed in version 2.5: Added the _module_globals_ parameter.

 -- Function: warnings.warnpy3k (message[, category[, stacklevel]])
     Issue a warning related to Python 3.x deprecation. Warnings are
     only shown when Python is started with the -3 option. Like
     warn(). _message_ must be a string and _category_ a subclass
     of Warning. See warnpy3k().  is using
     DeprecationWarning as default warning class.

     New in version 2.6.

 -- Function: warnings.showwarning (message, category, filename,
          lineno[, file[, line]])
     Write a warning to a file.  The default implementation calls
     `formatwarning(message, category, filename, lineno, line)' and
     writes the resulting string to _file_, which defaults to
     `sys.stderr'.  You may replace this function with an alternative
     implementation by assigning to `warnings.showwarning'.  _line_ is
     a line of source code to be included in the warning message; if
     _line_ is not supplied, see showwarning() will try to
     read the line specified by _filename_ and _lineno_.

     Changed in version 2.7: The _line_ argument is required to be
     supported.

 -- Function: warnings.formatwarning (message, category, filename,
          lineno[, line])
     Format a warning the standard way.  This returns a string which
     may contain embedded newlines and ends in a newline.  _line_ is a
     line of source code to be included in the warning message; if
     _line_ is not supplied, see formatwarning() will try to
     read the line specified by _filename_ and _lineno_.

     Changed in version 2.6: Added the _line_ argument.

 -- Function: warnings.filterwarnings (action[, message[, category[,
          module[, lineno[, append]]]]])
     Insert an entry into the list of warnings filter specifications.
     The entry is inserted at the front by
     default; if _append_ is true, it is inserted at the end.  This
     checks the types of the arguments, compiles the _message_ and
     _module_ regular expressions, and inserts them as a tuple in the
     list of warnings filters.  Entries closer to the front of the list
     override entries later in the list, if both match a particular
     warning.  Omitted arguments default to a value that matches
     everything.

 -- Function: warnings.simplefilter (action[, category[, lineno[,
          append]]])
     Insert a simple entry into the list of warnings filter specifications.
     The meaning of the function parameters is
     as for filterwarnings(), but regular expressions are
     not needed as the filter inserted always matches any message in
     any module as long as the category and line number match.

 -- Function: warnings.resetwarnings ()
     Reset the warnings filter.  This discards the effect of all
     previous calls to filterwarnings(), including that of
     the -W command line options and calls to
     simplefilter().

5.27.6.8 Available Context Managers
...................................

 -- Class: warnings.catch_warnings ([*, record=False, module=None])
     A context manager that copies and, upon exit, restores the
     warnings filter and the showwarning() function.  If
     the _record_ argument is False. (the default) the
     context manager returns None on entry. If _record_ is
     True, a list is returned that is progressively
     populated with objects as seen by a custom showwarning()
     function (which also suppresses output to `sys.stdout').
     Each object in the list has attributes with the same names as the
     arguments to showwarning().

     The _module_ argument takes a module that will be used instead of
     the module returned when you import warnings whose
     filter will be protected. This argument exists primarily for
     testing the warnings.  module itself.

          Note: The catch_warnings manager works by
          replacing and then later restoring the module's
          showwarning() function and internal list of filter
          specifications.  This means the context manager is modifying
          global state and therefore is not thread-safe.

          Note: In Python 3.0, the arguments to the constructor for
          catch_warnings are keyword-only arguments.

     New in version 2.6.

5.27.7 `contextlib' -- Utilities for `with'-statement contexts
--------------------------------------------------------------

New in version 2.5.

  This module provides utilities for common tasks involving the
with.  statement. For more information see also Context Manager Types
and With Statement Context Managers.

  Latest version of the contextlib Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/contextlib.py?view=markup)Functions
provided:

 -- Function: contextlib.contextmanager (func)
     This function is a decorator that can be used to
     define a factory function for with statement context
     managers, without needing to create a class or separate
     __enter__() and __exit__() methods.

     A simple example (this is not recommended as a real way of
     generating HTML!):

         from contextlib import contextmanager

         @contextmanager
         def tag(name):
             print "<%s>" % name
             yield
             print "</%s>" % name

         >>> with tag("h1"):
         ...    print "foo"
         ...
         <h1>
         foo
         </h1>

     The function being decorated must return a generator.
     -iterator when called. This iterator must yield exactly one
     value, which will be bound to the targets in the with.
     statement's as clause, if any.

     At the point where the generator yields, the block nested in the
     with.  statement is executed.  The generator is then
     resumed after the block is exited.  If an unhandled exception
     occurs in the block, it is reraised inside the generator at the
     point where the yield occurred.  Thus, you can use a try...
     See except...finally statement to trap
     the error (if any), or ensure that some cleanup takes place. If an
     exception is trapped merely in order to log it or to perform some
     action (rather than to suppress it entirely), the generator must
     reraise that exception. Otherwise the generator context manager
     will indicate to the with statement that the exception
     has been handled, and execution will resume with the statement
     immediately following the with statement.

 -- Function: contextlib.nested (mgr1[, mgr2[, ...]])
     Combine multiple context managers into a single nested context
     manager.

     This function has been deprecated in favour of the multiple
     manager form of the with statement.

     The one advantage of this function over the multiple manager form
     of the with statement is that argument unpacking
     allows it to be used with a variable number of context managers as
     follows:

         from contextlib import nested

         with nested(*managers):
             do_something()

     Note that if the __exit__() method of one of the
     nested context managers indicates an exception should be
     suppressed, no exception information will be passed to any
     remaining outer context managers. Similarly, if the
     __exit__() method of one of the nested managers raises an
     exception, any previous exception state will be lost; the new
     exception will be passed to the __exit__() methods of
     any remaining outer context managers. In general, see
     __exit__() methods should avoid raising exceptions, and in
     particular they should not re-raise a passed-in exception.

     This function has two major quirks that have led to it being
     deprecated. Firstly, as the context managers are all constructed
     before the function is invoked, the __new__() and
     __init__() methods of the inner context managers are
     not actually covered by the scope of the outer context managers.
     That means, for example, that using nested() to open
     two files is a programming error as the first file will not be
     closed promptly if an exception is thrown when opening the second
     file.

     Secondly, if the __enter__() method of one of the
     inner context managers raises an exception that is caught and
     suppressed by the __exit__() method of one of the
     outer context managers, this construct will raise
     RuntimeError rather than skipping the body of the
     with.  statement.

     Developers that need to support nesting of a variable number of
     context managers can either use the warnings module to
     suppress the DeprecationWarning raised by this function or else
     use this function as a model for an application specific
     implementation.

     Deprecated since version 2.7: The with-statement now supports this
     functionality directly (without the confusing error prone quirks).

 -- Function: contextlib.closing (thing)
     Return a context manager that closes _thing_ upon completion of
     the block.  This is basically equivalent to:

         from contextlib import contextmanager

         @contextmanager
         def closing(thing):
             try:
                 yield thing
             finally:
                 thing.close()

     And lets you write code like this:

         from contextlib import closing
         import urllib

         with closing(urllib.urlopen('http://www.python.org')) as page:
             for line in page:
                 print line

     without needing to explicitly close `page'.  Even if an error
     occurs, `page.close()' will be called when the with.
     block is exited.

PEP 0343 (http://www.python.org/dev/peps/pep-0343) - The "with" statement
     The specification, background, and examples for the Python
     with.  statement.

5.27.8 `abc' -- Abstract Base Classes
-------------------------------------

New in version 2.6.

  This module provides the infrastructure for defining an
abstract base class. (ABCs) in Python, as outlined in PEP 3119
(http://www.python.org/dev/peps/pep-3119); see the PEP for why this was
added to Python. (See also PEP 3141
(http://www.python.org/dev/peps/pep-3141) and the numbers.
module regarding a type hierarchy for numbers based on ABCs.)

  The collections module has some concrete classes that
derive from ABCs; these can, of course, be further derived. In addition
the collections module has some ABCs that can be used to
test whether a class or instance provides a particular interface, for
example, is it hashable or a mapping.

  This module provides the following class:

 -- Class: abc.ABCMeta
     Metaclass for defining Abstract Base Classes (ABCs).

     Use this metaclass to create an ABC.  An ABC can be subclassed
     directly, and then acts as a mix-in class.  You can also register
     unrelated concrete classes (even built-in classes) and unrelated
     ABCs as "virtual subclasses" - these and their descendants will be
     considered subclasses of the registering ABC by the built-in
     issubclass() function, but the registering ABC won't show up
     in their MRO (Method Resolution Order) nor will method
     implementations defined by the registering ABC be callable (not
     even via super()). (1)

     Classes created with a metaclass of ABCMeta have the
     following method:

      -- Method: register (subclass)
          Register _subclass_ as a "virtual subclass" of this ABC. For
          example:

              from abc import ABCMeta

              class MyABC:
                  __metaclass__ = ABCMeta

              MyABC.register(tuple)

              assert issubclass(tuple, MyABC)
              assert isinstance((), MyABC)

     You can also override this method in an abstract base class:

      -- Method: __subclasshook__ (subclass)
          (Must be defined as a class method.)

          Check whether _subclass_ is considered a subclass of this
          ABC.  This means that you can customize the behavior of
          `issubclass' further without the need to call
          register() on every class you want to consider a
          subclass of the ABC.  (This class method is called from the
          `__subclasscheck__()' method of the ABC.)

          This method should return `True', `False' or
          `NotImplemented'.  If it returns `True', the _subclass_ is
          considered a subclass of this ABC.  If it returns `False',
          the _subclass_ is not considered a subclass of this ABC, even
          if it would normally be one.  If it returns `NotImplemented',
          the subclass check is continued with the usual mechanism.

     For a demonstration of these concepts, look at this example ABC
     definition:

         class Foo(object):
             def __getitem__(self, index):
                 ...
             def __len__(self):
                 ...
             def get_iterator(self):
                 return iter(self)

         class MyIterable:
             __metaclass__ = ABCMeta

             @abstractmethod
             def __iter__(self):
                 while False:
                     yield None

             def get_iterator(self):
                 return self.__iter__()

             @classmethod
             def __subclasshook__(cls, C):
                 if cls is MyIterable:
                     if any("__iter__" in B.__dict__ for B in C.__mro__):
                         return True
                 return NotImplemented

         MyIterable.register(Foo)

     The ABC `MyIterable' defines the standard iterable method, see
     __iter__(), as an abstract method.  The implementation given
     here can still be called from subclasses.  The `get_iterator()'
     method is also part of the `MyIterable' abstract base class, but
     it does not have to be overridden in non-abstract derived classes.

     The __subclasshook__() class method defined here says
     that any class that has an __iter__() method in its
     `__dict__' (or in that of one of its base classes, accessed via
     the `__mro__' list) is considered a `MyIterable' too.

     Finally, the last line makes `Foo' a virtual subclass of
     `MyIterable', even though it does not define an __iter__()
     method (it uses the old-style iterable protocol, defined in
     terms of __len__() and __getitem__()).
     Note that this will not make `get_iterator' available as a method
     of `Foo', so it is provided separately.

  It also provides the following decorators:

 -- Function: abc.abstractmethod (function)
     A decorator indicating abstract methods.

     Using this decorator requires that the class's metaclass is
     ABCMeta or is derived from it.  A class that has a metaclass
     derived from ABCMeta.  cannot be instantiated unless
     all of its abstract methods and properties are overridden.  The
     abstract methods can be called using any of the normal 'super' call
     mechanisms.

     Dynamically adding abstract methods to a class, or attempting to
     modify the abstraction status of a method or class once it is
     created, are not supported.  The abstractmethod().
     only affects subclasses derived using regular inheritance;
     "virtual subclasses" registered with the ABC's `register()' method
     are not affected.

     Usage:

         class C:
             __metaclass__ = ABCMeta
             @abstractmethod
             def my_abstract_method(self, ...):
                 ...

          Note: Unlike Java abstract methods, these abstract methods
          may have an implementation. This implementation can be called
          via the super() mechanism from the class that
          overrides it.  This could be useful as an end-point for a
          super-call in a framework that uses cooperative
          multiple-inheritance.

 -- Function: abc.abstractproperty ([fget[, fset[, fdel[, doc]]]])
     A subclass of the built-in property(), indicating an
     abstract property.

     Using this function requires that the class's metaclass is
     ABCMeta or is derived from it.  A class that has a metaclass
     derived from ABCMeta cannot be instantiated unless all
     of its abstract methods and properties are overridden.  The
     abstract properties can be called using any of the normal 'super'
     call mechanisms.

     Usage:

         class C:
             __metaclass__ = ABCMeta
             @abstractproperty
             def my_abstract_property(self):
                 ...

     This defines a read-only property; you can also define a
     read-write abstract property using the 'long' form of property
     declaration:

         class C:
             __metaclass__ = ABCMeta
             def getx(self): ...
             def setx(self, value): ...
             x = abstractproperty(getx, setx)

Footnotes
.........

     (1) C++ programmers should note that Python's virtual base class
     concept is not the same as C++'s.

5.27.9 `atexit' -- Exit handlers
--------------------------------

New in version 2.0.

  The atexit module defines a single function to register
cleanup functions.  Functions thus registered are automatically
executed upon normal interpreter termination.

  Latest version of the atexit Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/atexit.py?view=markup)Note:
the functions registered via this module are not called when the program
is killed by a signal not handled by Python, when a Python fatal
internal error is detected, or when os._exit() is called.

  This is an alternate interface to the functionality provided by the
`sys.exitfunc' variable.

  Note: This module is unlikely to work correctly when used with other
code that sets `sys.exitfunc'.  In particular, other core Python
modules are free to use atexit without the programmer's
knowledge.  Authors who use `sys.exitfunc' should convert their code to
use atexit instead.  The simplest way to convert code that
sets `sys.exitfunc' is to import atexit and register the
function that had been bound to `sys.exitfunc'.

 -- Function: atexit.register (func[, *args[, **kargs]])
     Register _func_ as a function to be executed at termination.  Any
     optional arguments that are to be passed to _func_ must be passed
     as arguments to register().

     At normal program termination (for instance, if sys.exit()
     is called or the main module's execution completes), all
     functions registered are called in last in, first out order.  The
     assumption is that lower level modules will normally be imported
     before higher level modules and thus must be cleaned up later.

     If an exception is raised during execution of the exit handlers, a
     traceback is printed (unless SystemExit is raised) and
     the exception information is saved.  After all exit handlers have
     had a chance to run the last exception to be raised is re-raised.

     Changed in version 2.6: This function now returns _func_ which
     makes it possible to use it as a decorator without binding the
     original name to `None'.

Module readline.
     Useful example of atexit to read and write
     readline history files.

5.27.9.1 `atexit' Example
.........................

The following simple example demonstrates how a module can initialize a
counter from a file when it is imported and save the counter's updated
value automatically when the program terminates without relying on the
application making an explicit call into this module at termination.

    try:
        _count = int(open("/tmp/counter").read())
    except IOError:
        _count = 0

    def incrcounter(n):
        global _count
        _count = _count + n

    def savecounter():
        open("/tmp/counter", "w").write("%d" % _count)

    import atexit
    atexit.register(savecounter)

Positional and keyword arguments may also be passed to
register() to be passed along to the registered function when it
is called:

    def goodbye(name, adjective):
        print 'Goodbye, %s, it was %s to meet you.' % (name, adjective)

    import atexit
    atexit.register(goodbye, 'Donny', 'nice')

    # or:
    atexit.register(goodbye, adjective='nice', name='Donny')

Usage as a decorator.:

    import atexit

    @atexit.register
    def goodbye():
        print "You are now leaving the Python sector."

This obviously only works with functions that don't take arguments.

5.27.10 `traceback' -- Print or retrieve a stack traceback
----------------------------------------------------------

This module provides a standard interface to extract, format and print
stack traces of Python programs.  It exactly mimics the behavior of the
Python interpreter when it prints a stack trace.  This is useful when
you want to print stack traces under program control, such as in a
"wrapper" around the interpreter.

  The module uses traceback objects -- this is the object type that is
stored in the variables sys.exc_traceback. (deprecated) and
sys.last_traceback and returned as the third item from
sys.exc_info().

  The module defines the following functions:

 -- Function: traceback.print_tb (traceback[, limit[, file]])
     Print up to _limit_ stack trace entries from _traceback_.  If
     _limit_ is omitted or `None', all entries are printed. If _file_
     is omitted or `None', the output goes to `sys.stderr'; otherwise
     it should be an open file or file-like object to receive the
     output.

 -- Function: traceback.print_exception (type, value, traceback[,
          limit[, file]])
     Print exception information and up to _limit_ stack trace entries
     from _traceback_ to _file_. This differs from print_tb()
     in the following ways: (1) if _traceback_ is not `None', it
     prints a header `Traceback (most recent call last):'; (2) it
     prints the exception _type_ and _value_ after the stack trace; (3)
     if _type_ is SyntaxError and _value_ has the
     appropriate format, it prints the line where the syntax error
     occurred with a caret indicating the approximate position of the
     error.

 -- Function: traceback.print_exc ([limit[, file]])
     This is a shorthand for `print_exception(sys.exc_type,
     sys.exc_value, sys.exc_traceback, limit, file)'.  (In fact, it
     uses sys.exc_info() to retrieve the same information
     in a thread-safe way instead of using the deprecated variables.)

 -- Function: traceback.format_exc ([limit])
     This is like `print_exc(limit)' but returns a string instead of
     printing to a file.

     New in version 2.4.

 -- Function: traceback.print_last ([limit[, file]])
     This is a shorthand for `print_exception(sys.last_type,
     sys.last_value, sys.last_traceback, limit, file)'.  In general it
     will work only after an exception has reached an interactive
     prompt (see sys.last_type).

 -- Function: traceback.print_stack ([f[, limit[, file]]])
     This function prints a stack trace from its invocation point.  The
     optional _f_ argument can be used to specify an alternate stack
     frame to start.  The optional _limit_ and _file_ arguments have
     the same meaning as for print_exception().

 -- Function: traceback.extract_tb (traceback[, limit])
     Return a list of up to _limit_ "pre-processed" stack trace entries
     extracted from the traceback object _traceback_.  It is useful for
     alternate formatting of stack traces.  If _limit_ is omitted or
     `None', all entries are extracted.  A "pre-processed" stack trace
     entry is a quadruple (_filename_, _line number_, _function name_,
     _text_) representing the information that is usually printed for a
     stack trace.  The _text_ is a string with leading and trailing
     whitespace stripped; if the source is not available it is `None'.

 -- Function: traceback.extract_stack ([f[, limit]])
     Extract the raw traceback from the current stack frame.  The
     return value has the same format as for extract_tb().
     The optional _f_ and _limit_ arguments have the same meaning as
     for print_stack().

 -- Function: traceback.format_list (list)
     Given a list of tuples as returned by extract_tb() or
     extract_stack(), return a list of strings ready for
     printing.  Each string in the resulting list corresponds to the
     item with the same index in the argument list.  Each string ends
     in a newline; the strings may contain internal newlines as well,
     for those items whose source text line is not `None'.

 -- Function: traceback.format_exception_only (type, value)
     Format the exception part of a traceback.  The arguments are the
     exception type and value such as given by `sys.last_type' and
     `sys.last_value'.  The return value is a list of strings, each
     ending in a newline.  Normally, the list contains a single string;
     however, for SyntaxError exceptions, it contains
     several lines that (when printed) display detailed information
     about where the syntax error occurred.  The message indicating
     which exception occurred is the always last string in the list.

 -- Function: traceback.format_exception (type, value, tb[, limit])
     Format a stack trace and the exception information.  The arguments
     have the same meaning as the corresponding arguments to
     print_exception().  The return value is a list of strings,
     each ending in a newline and some containing internal newlines.
     When these lines are concatenated and printed, exactly the same
     text is printed as does print_exception().

 -- Function: traceback.format_tb (tb[, limit])
     A shorthand for `format_list(extract_tb(tb, limit))'.

 -- Function: traceback.format_stack ([f[, limit]])
     A shorthand for `format_list(extract_stack(f, limit))'.

 -- Function: traceback.tb_lineno (tb)
     This function returns the current line number set in the traceback
     object.  This function was necessary because in versions of Python
     prior to 2.3 when the -O flag was passed to Python the
     `tb.tb_lineno' was not updated correctly.  This function has no
     use in versions past 2.3.

5.27.10.1 Traceback Examples
............................

This simple example implements a basic read-eval-print loop, similar to
(but less useful than) the standard Python interactive interpreter
loop.  For a more complete implementation of the interpreter loop,
refer to the code.  module.

    import sys, traceback

    def run_user_code(envdir):
        source = raw_input(">>> ")
        try:
            exec source in envdir
        except:
            print "Exception in user code:"
            print '-'*60
            traceback.print_exc(file=sys.stdout)
            print '-'*60

    envdir = {}
    while 1:
        run_user_code(envdir)

The following example demonstrates the different ways to print and
format the exception and traceback:

    import sys, traceback

    def lumberjack():
        bright_side_of_death()

    def bright_side_of_death():
        return tuple()[0]

    try:
        lumberjack()
    except IndexError:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        print "*** print_tb:"
        traceback.print_tb(exc_traceback, limit=1, file=sys.stdout)
        print "*** print_exception:"
        traceback.print_exception(exc_type, exc_value, exc_traceback,
                                  limit=2, file=sys.stdout)
        print "*** print_exc:"
        traceback.print_exc()
        print "*** format_exc, first and last line:"
        formatted_lines = traceback.format_exc().splitlines()
        print formatted_lines[0]
        print formatted_lines[-1]
        print "*** format_exception:"
        print repr(traceback.format_exception(exc_type, exc_value,
                                              exc_traceback))
        print "*** extract_tb:"
        print repr(traceback.extract_tb(exc_traceback))
        print "*** format_tb:"
        print repr(traceback.format_tb(exc_traceback))
        print "*** tb_lineno:", exc_traceback.tb_lineno

The output for the example would look similar to this:

    *** print_tb:
      File "<doctest...>", line 10, in <module>
        lumberjack()
    *** print_exception:
    Traceback (most recent call last):
      File "<doctest...>", line 10, in <module>
        lumberjack()
      File "<doctest...>", line 4, in lumberjack
        bright_side_of_death()
    IndexError: tuple index out of range
    *** print_exc:
    Traceback (most recent call last):
      File "<doctest...>", line 10, in <module>
        lumberjack()
      File "<doctest...>", line 4, in lumberjack
        bright_side_of_death()
    IndexError: tuple index out of range
    *** format_exc, first and last line:
    Traceback (most recent call last):
    IndexError: tuple index out of range
    *** format_exception:
    ['Traceback (most recent call last):\n',
     '  File "<doctest...>", line 10, in <module>\n    lumberjack()\n',
     '  File "<doctest...>", line 4, in lumberjack\n    bright_side_of_death()\n',
     '  File "<doctest...>", line 7, in bright_side_of_death\n    return tuple()[0]\n',
     'IndexError: tuple index out of range\n']
    *** extract_tb:
    [('<doctest...>', 10, '<module>', 'lumberjack()'),
     ('<doctest...>', 4, 'lumberjack', 'bright_side_of_death()'),
     ('<doctest...>', 7, 'bright_side_of_death', 'return tuple()[0]')]
    *** format_tb:
    ['  File "<doctest...>", line 10, in <module>\n    lumberjack()\n',
     '  File "<doctest...>", line 4, in lumberjack\n    bright_side_of_death()\n',
     '  File "<doctest...>", line 7, in bright_side_of_death\n    return tuple()[0]\n']
    *** tb_lineno: 10

The following example shows the different ways to print and format the
stack:

    >>> import traceback
    >>> def another_function():
    ...     lumberstack()
    ...
    >>> def lumberstack():
    ...     traceback.print_stack()
    ...     print repr(traceback.extract_stack())
    ...     print repr(traceback.format_stack())
    ...
    >>> another_function()
      File "<doctest>", line 10, in <module>
        another_function()
      File "<doctest>", line 3, in another_function
        lumberstack()
      File "<doctest>", line 6, in lumberstack
        traceback.print_stack()
    [('<doctest>', 10, '<module>', 'another_function()'),
     ('<doctest>', 3, 'another_function', 'lumberstack()'),
     ('<doctest>', 7, 'lumberstack', 'print repr(traceback.extract_stack())')]
    ['  File "<doctest>", line 10, in <module>\n    another_function()\n',
     '  File "<doctest>", line 3, in another_function\n    lumberstack()\n',
     '  File "<doctest>", line 8, in lumberstack\n    print repr(traceback.format_stack())\n']

This last example demonstrates the final few formatting functions:

    >>> import traceback
    >>> traceback.format_list([('spam.py', 3, '<module>', 'spam.eggs()'),
    ...                        ('eggs.py', 42, 'eggs', 'return "bacon"')])
    ['  File "spam.py", line 3, in <module>\n    spam.eggs()\n',
     '  File "eggs.py", line 42, in eggs\n    return "bacon"\n']
    >>> an_error = IndexError('tuple index out of range')
    >>> traceback.format_exception_only(type(an_error), an_error)
    ['IndexError: tuple index out of range\n']

5.27.11 `__future__' -- Future statement definitions
----------------------------------------------------

__future__ is a real module, and serves three purposes:

   * To avoid confusing existing tools that analyze import statements
     and expect to find the modules they're importing.

   * To ensure that future statements run under releases
     prior to 2.1 at least yield runtime exceptions (the import of
     __future__ will fail, because there was no module of
     that name prior to 2.1).

   * To document when incompatible changes were introduced, and when
     they will be -- or were -- made mandatory.  This is a form of
     executable documentation, and can be inspected programmatically
     via importing __future__ and examining its contents.

  Each statement in `__future__.py' is of the form:

    FeatureName = _Feature(OptionalRelease, MandatoryRelease,
                           CompilerFlag)

where, normally, _OptionalRelease_ is less than _MandatoryRelease_, and
both are 5-tuples of the same form as `sys.version_info':

    (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int
     PY_MINOR_VERSION, # the 1; an int
     PY_MICRO_VERSION, # the 0; an int
     PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string
     PY_RELEASE_SERIAL # the 3; an int
    )

_OptionalRelease_ records the first release in which the feature was
accepted.

  In the case of a _MandatoryRelease_ that has not yet occurred,
_MandatoryRelease_ predicts the release in which the feature will
become part of the language.

  Else _MandatoryRelease_ records when the feature became part of the
language; in releases at or after that, modules no longer need a future
statement to use the feature in question, but may continue to use such
imports.

  _MandatoryRelease_ may also be `None', meaning that a planned feature
got dropped.

  Instances of class `_Feature' have two corresponding methods,
`getOptionalRelease()' and `getMandatoryRelease()'.

  _CompilerFlag_ is the (bitfield) flag that should be passed in the
fourth argument to the built-in function compile() to
enable the feature in dynamically compiled code.  This flag is stored
in the `compiler_flag' attribute on `_Feature' instances.

  No feature description will ever be deleted from __future__.
Since its introduction in Python 2.1 the following features have found
their way into the language using this mechanism:

feature                optional in       mandatory in       effect
--------------------------------------------------------------------------------------------------------------
nested_scopes          2.1.0b1           2.2                PEP 227
                                                            (http://www.python.org/dev/peps/pep-0227):
                                                            _Statically Nested Scopes_
generators             2.2.0a1           2.3                PEP 255
                                                            (http://www.python.org/dev/peps/pep-0255):
                                                            _Simple Generators_
division               2.2.0a2           3.0                PEP 238
                                                            (http://www.python.org/dev/peps/pep-0238):
                                                            _Changing the Division Operator_
absolute_import        2.5.0a1           2.7                PEP 328
                                                            (http://www.python.org/dev/peps/pep-0328):
                                                            _Imports: Multi-Line and Absolute/Relative_
with_statement         2.5.0a1           2.6                PEP 343
                                                            (http://www.python.org/dev/peps/pep-0343): _The
                                                            "with" Statement_
print_function         2.6.0a2           3.0                PEP 3105
                                                            (http://www.python.org/dev/peps/pep-3105): _Make
                                                            print a function_
unicode_literals       2.6.0a2           3.0                PEP 3112
                                                            (http://www.python.org/dev/peps/pep-3112):
                                                            _Bytes literals in Python 3000_

See Future statements.
     How the compiler treats future imports.

5.27.12 `gc' -- Garbage Collector interface
-------------------------------------------

This module provides an interface to the optional garbage collector.  It
provides the ability to disable the collector, tune the collection
frequency, and set debugging options.  It also provides access to
unreachable objects that the collector found but cannot free.  Since
the collector supplements the reference counting already used in
Python, you can disable the collector if you are sure your program does
not create reference cycles.  Automatic collection can be disabled by
calling `gc.disable()'.  To debug a leaking program call
`gc.set_debug(gc.DEBUG_LEAK)'. Notice that this includes
`gc.DEBUG_SAVEALL', causing garbage-collected objects to be saved in
gc.garbage for inspection.

  The gc module provides the following functions:

 -- Function: gc.enable ()
     Enable automatic garbage collection.

 -- Function: gc.disable ()
     Disable automatic garbage collection.

 -- Function: gc.isenabled ()
     Returns true if automatic collection is enabled.

 -- Function: gc.collect ([generation])
     With no arguments, run a full collection.  The optional argument
     _generation_ may be an integer specifying which generation to
     collect (from 0 to 2).  A ValueError is raised if the
     generation number  is invalid. The number of unreachable objects
     found is returned.

     Changed in version 2.5: The optional _generation_ argument was
     added.

     Changed in version 2.6: The free lists maintained for a number of
     built-in types are cleared whenever a full collection or
     collection of the highest generation (2) is run.  Not all items in
     some free lists may be freed due to the particular implementation,
     in particular int and float.

 -- Function: gc.set_debug (flags)
     Set the garbage collection debugging flags. Debugging information
     will be written to `sys.stderr'.  See below for a list of
     debugging flags which can be combined using bit operations to
     control debugging.

 -- Function: gc.get_debug ()
     Return the debugging flags currently set.

 -- Function: gc.get_objects ()
     Returns a list of all objects tracked by the collector, excluding
     the list returned.

     New in version 2.2.

 -- Function: gc.set_threshold (threshold0[, threshold1[, threshold2]])
     Set the garbage collection thresholds (the collection frequency).
     Setting _threshold0_ to zero disables collection.

     The GC classifies objects into three generations depending on how
     many collection sweeps they have survived.  New objects are placed
     in the youngest generation (generation `0').  If an object
     survives a collection it is moved into the next older generation.
     Since generation `2' is the oldest generation, objects in that
     generation remain there after a collection.  In order to decide
     when to run, the collector keeps track of the number object
     allocations and deallocations since the last collection.  When the
     number of allocations minus the number of deallocations exceeds
     _threshold0_, collection starts.  Initially only generation `0' is
     examined.  If generation `0' has been examined more than
     _threshold1_ times since generation `1' has been examined, then
     generation `1' is examined as well.  Similarly, _threshold2_
     controls the number of collections of generation `1' before
     collecting generation `2'.

 -- Function: gc.get_count ()
     Return the current collection  counts as a tuple of `(count0,
     count1, count2)'.

     New in version 2.5.

 -- Function: gc.get_threshold ()
     Return the current collection thresholds as a tuple of
     `(threshold0, threshold1, threshold2)'.

 -- Function: gc.get_referrers (*objs)
     Return the list of objects that directly refer to any of objs.
     This function will only locate those containers which support
     garbage collection; extension types which do refer to other
     objects but do not support garbage collection will not be found.

     Note that objects which have already been dereferenced, but which
     live in cycles and have not yet been collected by the garbage
     collector can be listed among the resulting referrers.  To get
     only currently live objects, call collect().  before
     calling get_referrers().

     Care must be taken when using objects returned by
     get_referrers() because some of them could still be under
     construction and hence in a temporarily invalid state. Avoid using
     get_referrers() for any purpose other than debugging.

     New in version 2.2.

 -- Function: gc.get_referents (*objs)
     Return a list of objects directly referred to by any of the
     arguments. The referents returned are those objects visited by the
     arguments' C-level `tp_traverse' methods (if any), and may not be
     all objects actually directly reachable.  `tp_traverse' methods
     are supported only by objects that support garbage collection, and
     are only required to visit objects that may be involved in a
     cycle.  So, for example, if an integer is directly reachable from
     an argument, that integer object may or may not appear in the
     result list.

     New in version 2.3.

 -- Function: gc.is_tracked (obj)
     Returns True if the object is currently tracked by the garbage
     collector, False otherwise.  As a general rule, instances of
     atomic types aren't tracked and instances of non-atomic types
     (containers, user-defined objects...) are.  However, some
     type-specific optimizations can be present in order to suppress
     the garbage collector footprint of simple instances (e.g. dicts
     containing only atomic keys and values):

         >>> gc.is_tracked(0)
         False
         >>> gc.is_tracked("a")
         False
         >>> gc.is_tracked([])
         True
         >>> gc.is_tracked({})
         False
         >>> gc.is_tracked({"a": 1})
         False
         >>> gc.is_tracked({"a": []})
         True

     New in version 2.7.

  The following variable is provided for read-only access (you can
mutate its value but should not rebind it):

 -- Data: gc.garbage
     A list of objects which the collector found to be unreachable but
     could not be freed (uncollectable objects).  By default, this list
     contains only objects with __del__() methods. (1)
     Objects that have __del__() methods and are part of a
     reference cycle cause the entire reference cycle to be
     uncollectable, including objects not necessarily in the cycle but
     reachable only from it.  Python doesn't collect such cycles
     automatically because, in general, it isn't possible for Python to
     guess a safe order in which to run the __del__().
     methods.  If you know a safe order, you can force the issue by
     examining the _garbage_ list, and explicitly breaking cycles due
     to your objects within the list.  Note that these objects are kept
     alive even so by virtue of being in the _garbage_ list, so they
     should be removed from _garbage_ too.  For example, after breaking
     cycles, do `del gc.garbage[:]' to empty the list.  It's generally
     better to avoid the issue by not creating cycles containing objects
     with __del__() methods, and _garbage_ can be examined
     in that case to verify that no such cycles are being created.

     If DEBUG_SAVEALL is set, then all unreachable objects
     will be added to this list rather than freed.

  The following constants are provided for use with set_debug().:

 -- Data: gc.DEBUG_STATS
     Print statistics during collection.  This information can be
     useful when tuning the collection frequency.

 -- Data: gc.DEBUG_COLLECTABLE
     Print information on collectable objects found.

 -- Data: gc.DEBUG_UNCOLLECTABLE
     Print information of uncollectable objects found (objects which
     are not reachable but cannot be freed by the collector).  These
     objects will be added to the `garbage' list.

 -- Data: gc.DEBUG_INSTANCES
     When DEBUG_COLLECTABLE or DEBUG_UNCOLLECTABLE
     is set, print information about instance objects found.

 -- Data: gc.DEBUG_OBJECTS
     When DEBUG_COLLECTABLE or DEBUG_UNCOLLECTABLE
     is set, print information about objects other than instance
     objects found.

 -- Data: gc.DEBUG_SAVEALL
     When set, all unreachable objects found will be appended to
     _garbage_ rather than being freed.  This can be useful for
     debugging a leaking program.

 -- Data: gc.DEBUG_LEAK
     The debugging flags necessary for the collector to print
     information about a leaking program (equal to `DEBUG_COLLECTABLE |
     DEBUG_UNCOLLECTABLE | DEBUG_INSTANCES | DEBUG_OBJECTS |
     DEBUG_SAVEALL').

Footnotes
.........

     (1) Prior to Python 2.2, the list contained all instance objects
     in unreachable cycles,  not only those with __del__().
     methods.

5.27.13 `inspect' -- Inspect live objects
-----------------------------------------

New in version 2.1.

  The inspect module provides several useful functions to
help get information about live objects such as modules, classes,
methods, functions, tracebacks, frame objects, and code objects.  For
example, it can help you examine the contents of a class, retrieve the
source code of a method, extract and format the argument list for a
function, or get all the information you need to display a detailed
traceback.

  There are four main kinds of services provided by this module: type
checking, getting source code, inspecting classes and functions, and
examining the interpreter stack.

5.27.13.1 Types and members
...........................

The getmembers() function retrieves the members of an
object such as a class or module. The sixteen functions whose names
begin with "is" are mainly provided as convenient choices for the
second argument to getmembers().  They also help you
determine when you can expect to find the following special attributes:

Type            Attribute             Description                     Notes
----------------------------------------------------------------------------------
module          __doc__               documentation string
                __file__              filename (missing for built-in
                                      modules)
class           __doc__               documentation string
                __module__            name of module in which this
                                      class was defined
method          __doc__               documentation string
                __name__              name with which this method
                                      was defined
                im_class              class object that asked for     (1)
                                      this method
                im_func or __func__   function object containing
                                      implementation of method
                im_self or __self__   instance to which this method
                                      is bound, or `None'
function        __doc__               documentation string
                __name__              name with which this function
                                      was defined
                func_code             code object containing
                                      compiled function
                                      bytecode.
                func_defaults         tuple of any default values
                                      for arguments
                func_doc              (same as __doc__)
                func_globals          global namespace in which this
                                      function was defined
                func_name             (same as __name__)
generator       __iter__              defined to support iteration
                                      over container
                close                 raises new GeneratorExit
                                      exception inside the generator
                                      to terminate the iteration
                gi_code               code object
                gi_frame              frame object or possibly None
                                      once the generator has been
                                      exhausted
                gi_running            set to 1 when generator is
                                      executing, 0 otherwise
                next                  return the next item from the
                                      container
                send                  resumes the generator and
                                      "sends" a value that becomes
                                      the result of the current
                                      yield-expression
                throw                 used to raise an exception
                                      inside the generator
traceback       tb_frame              frame object at this level
                tb_lasti              index of last attempted
                                      instruction in bytecode
                tb_lineno             current line number in Python
                                      source code
                tb_next               next inner traceback object
                                      (called by this level)
frame           f_back                next outer frame object (this
                                      frame's caller)
                f_builtins            builtins namespace seen by
                                      this frame
                f_code                code object being executed in
                                      this frame
                f_exc_traceback       traceback if raised in this
                                      frame, or `None'
                f_exc_type            exception type if raised in
                                      this frame, or `None'
                f_exc_value           exception value if raised in
                                      this frame, or `None'
                f_globals             global namespace seen by this
                                      frame
                f_lasti               index of last attempted
                                      instruction in bytecode
                f_lineno              current line number in Python
                                      source code
                f_locals              local namespace seen by this
                                      frame
                f_restricted          0 or 1 if frame is in
                                      restricted execution mode
                f_trace               tracing function for this
                                      frame, or `None'
code            co_argcount           number of arguments (not
                                      including * or ** args)
                co_code               string of raw compiled bytecode
                co_consts             tuple of constants used in the
                                      bytecode
                co_filename           name of file in which this
                                      code object was created
                co_firstlineno        number of first line in Python
                                      source code
                co_flags              bitmap: 1=optimized `|'
                                      2=newlocals `|' 4=*arg `|'
                                      8=**arg
                co_lnotab             encoded mapping of line
                                      numbers to bytecode indices
                co_name               name with which this code
                                      object was defined
                co_names              tuple of names of local
                                      variables
                co_nlocals            number of local variables
                co_stacksize          virtual machine stack space
                                      required
                co_varnames           tuple of names of arguments
                                      and local variables
builtin         __doc__               documentation string
                __name__              original name of this function
                                      or method
                __self__              instance to which a method is
                                      bound, or `None'

  Note:

  1. Changed in version 2.2: `im_class' used to refer to the class that
     defined the method.

 -- Function: inspect.getmembers (object[, predicate])
     Return all the members of an object in a list of (name, value)
     pairs sorted by name.  If the optional _predicate_ argument is
     supplied, only members for which the predicate returns a true
     value are included.

          Note: getmembers() does not return metaclass
          attributes when the argument is a class (this behavior is
          inherited from the dir() function).

 -- Function: inspect.getmoduleinfo (path)
     Return a tuple of values that describe how Python will interpret
     the file identified by _path_ if it is a module, or `None' if it
     would not be identified as a module.  The return tuple is `(name,
     suffix, mode, mtype)', where _name_ is the name of the module
     without the name of any enclosing package, _suffix_ is the
     trailing part of the file name (which may not be a dot-delimited
     extension), _mode_ is the open() mode that would be
     used (`'r'' or `'rb''), and _mtype_ is an integer giving the type
     of the module.  _mtype_ will have a value which can be compared to
     the constants defined in the imp module; see the
     documentation for that module for more information on module types.

     Changed in version 2.6: Returns a named tuple.
     `ModuleInfo(name, suffix, mode, module_type)'.

 -- Function: inspect.getmodulename (path)
     Return the name of the module named by the file _path_, without
     including the names of enclosing packages.  This uses the same
     algorithm as the interpreter uses when searching for modules.  If
     the name cannot be matched according to the interpreter's rules,
     `None' is returned.

 -- Function: inspect.ismodule (object)
     Return true if the object is a module.

 -- Function: inspect.isclass (object)
     Return true if the object is a class, whether built-in or created
     in Python code.

 -- Function: inspect.ismethod (object)
     Return true if the object is a bound method written in Python.

 -- Function: inspect.isfunction (object)
     Return true if the object is a Python function, which includes
     functions created by a lambda expression.

 -- Function: inspect.isgeneratorfunction (object)
     Return true if the object is a Python generator function.

     New in version 2.6.

 -- Function: inspect.isgenerator (object)
     Return true if the object is a generator.

     New in version 2.6.

 -- Function: inspect.istraceback (object)
     Return true if the object is a traceback.

 -- Function: inspect.isframe (object)
     Return true if the object is a frame.

 -- Function: inspect.iscode (object)
     Return true if the object is a code.

 -- Function: inspect.isbuiltin (object)
     Return true if the object is a built-in function or a bound
     built-in method.

 -- Function: inspect.isroutine (object)
     Return true if the object is a user-defined or built-in function
     or method.

 -- Function: inspect.isabstract (object)
     Return true if the object is an abstract base class.

     New in version 2.6.

 -- Function: inspect.ismethoddescriptor (object)
     Return true if the object is a method descriptor, but not if
     ismethod(), isclass(), isfunction().
     or isbuiltin().  are true.

     This is new as of Python 2.2, and, for example, is true of
     `int.__add__'. An object passing this test has a `__get__'
     attribute but not a `__set__' attribute, but beyond that the set
     of attributes varies.  `__name__' is usually sensible, and
     `__doc__' often is.

     Methods implemented via descriptors that also pass one of the
     other tests return false from the ismethoddescriptor()
     test, simply because the other tests promise more - you can,
     e.g., count on having the `im_func' attribute (etc) when an object
     passes ismethod().

 -- Function: inspect.isdatadescriptor (object)
     Return true if the object is a data descriptor.

     Data descriptors have both a `__get__' and a `__set__' attribute.
     Examples are properties (defined in Python), getsets, and members.
     The latter two are defined in C and there are more specific tests
     available for those types, which is robust across Python
     implementations.  Typically, data descriptors will also have
     `__name__' and `__doc__' attributes (properties, getsets, and
     members have both of these attributes), but this is not guaranteed.

     New in version 2.3.

 -- Function: inspect.isgetsetdescriptor (object)
     Return true if the object is a getset descriptor.

     *CPython implementation detail:* getsets are attributes defined in
     extension modules via `PyGetSetDef' structures.  For Python
     implementations without such types, this method will always return
     `False'.

     New in version 2.5.

 -- Function: inspect.ismemberdescriptor (object)
     Return true if the object is a member descriptor.

     *CPython implementation detail:* Member descriptors are attributes
     defined in extension modules via PyMemberDef.
     structures.  For Python implementations without such types, this
     method will always return `False'.

     New in version 2.5.

5.27.13.2 Retrieving source code
................................

 -- Function: inspect.getdoc (object)
     Get the documentation string for an object, cleaned up with
     cleandoc().

 -- Function: inspect.getcomments (object)
     Return in a single string any lines of comments immediately
     preceding the object's source code (for a class, function, or
     method), or at the top of the Python source file (if the object is
     a module).

 -- Function: inspect.getfile (object)
     Return the name of the (text or binary) file in which an object
     was defined.  This will fail with a TypeError if the
     object is a built-in module, class, or function.

 -- Function: inspect.getmodule (object)
     Try to guess which module an object was defined in.

 -- Function: inspect.getsourcefile (object)
     Return the name of the Python source file in which an object was
     defined.  This will fail with a TypeError if the
     object is a built-in module, class, or function.

 -- Function: inspect.getsourcelines (object)
     Return a list of source lines and starting line number for an
     object. The argument may be a module, class, method, function,
     traceback, frame, or code object.  The source code is returned as
     a list of the lines corresponding to the object and the line
     number indicates where in the original source file the first line
     of code was found.  An IOError is raised if the source
     code cannot be retrieved.

 -- Function: inspect.getsource (object)
     Return the text of the source code for an object. The argument may
     be a module, class, method, function, traceback, frame, or code
     object.  The source code is returned as a single string.  An
     IOError is raised if the source code cannot be retrieved.

 -- Function: inspect.cleandoc (doc)
     Clean up indentation from docstrings that are indented to line up
     with blocks of code.  Any whitespace that can be uniformly removed
     from the second line onwards is removed.  Also, all tabs are
     expanded to spaces.

     New in version 2.6.

5.27.13.3 Classes and functions
...............................

 -- Function: inspect.getclasstree (classes[, unique])
     Arrange the given list of classes into a hierarchy of nested
     lists. Where a nested list appears, it contains classes derived
     from the class whose entry immediately precedes the list.  Each
     entry is a 2-tuple containing a class and a tuple of its base
     classes.  If the _unique_ argument is true, exactly one entry
     appears in the returned structure for each class in the given
     list.  Otherwise, classes using multiple inheritance and their
     descendants will appear multiple times.

 -- Function: inspect.getargspec (func)
     Get the names and default values of a Python function's arguments.
     A tuple of four things is returned: `(args, varargs, varkw,
     defaults)'. _args_ is a list of the argument names (it may contain
     nested lists). _varargs_ and _varkw_ are the names of the `*' and
     `**' arguments or `None'. _defaults_ is a tuple of default
     argument values or None if there are no default arguments; if this
     tuple has _n_ elements, they correspond to the last _n_ elements
     listed in _args_.

     Changed in version 2.6: Returns a named tuple.
     `ArgSpec(args, varargs, keywords, defaults)'.

 -- Function: inspect.getargvalues (frame)
     Get information about arguments passed into a particular frame. A
     tuple of four things is returned: `(args, varargs, varkw,
     locals)'. _args_ is a list of the argument names (it may contain
     nested lists). _varargs_ and _varkw_ are the names of the `*' and
     `**' arguments or `None'. _locals_ is the locals dictionary of the
     given frame.

     Changed in version 2.6: Returns a named tuple.
     `ArgInfo(args, varargs, keywords, locals)'.

 -- Function: inspect.formatargspec (args[, varargs, varkw, defaults,
          formatarg, formatvarargs, formatvarkw, formatvalue, join])
     Format a pretty argument spec from the four values returned by
     getargspec().  The format* arguments are the
     corresponding optional formatting functions that are called to
     turn names and values into strings.

 -- Function: inspect.formatargvalues (args[, varargs, varkw, locals,
          formatarg, formatvarargs, formatvarkw, formatvalue, join])
     Format a pretty argument spec from the four values returned by
     getargvalues().  The format* arguments are the
     corresponding optional formatting functions that are called to
     turn names and values into strings.

 -- Function: inspect.getmro (cls)
     Return a tuple of class cls's base classes, including cls, in
     method resolution order.  No class appears more than once in this
     tuple. Note that the method resolution order depends on cls's
     type.  Unless a very peculiar user-defined metatype is in use, cls
     will be the first element of the tuple.

 -- Function: inspect.getcallargs (func[, *args][, **kwds])
     Bind the _args_ and _kwds_ to the argument names of the Python
     function or method _func_, as if it was called with them. For
     bound methods, bind also the first argument (typically named
     `self') to the associated instance. A dict is returned, mapping
     the argument names (including the names of the `*' and `**'
     arguments, if any) to their values from _args_ and _kwds_. In case
     of invoking _func_ incorrectly, i.e. whenever `func(*args,
     **kwds)' would raise an exception because of incompatible
     signature, an exception of the same type and the same or similar
     message is raised. For example:

         >>> from inspect import getcallargs
         >>> def f(a, b=1, *pos, **named):
         ...     pass
         >>> getcallargs(f, 1, 2, 3)
         {'a': 1, 'named': {}, 'b': 2, 'pos': (3,)}
         >>> getcallargs(f, a=2, x=4)
         {'a': 2, 'named': {'x': 4}, 'b': 1, 'pos': ()}
         >>> getcallargs(f)
         Traceback (most recent call last):
         ...
         TypeError: f() takes at least 1 argument (0 given)

     New in version 2.7.

5.27.13.4 The interpreter stack
...............................

When the following functions return "frame records," each record is a
tuple of six items: the frame object, the filename, the line number of
the current line, the function name, a list of lines of context from
the source code, and the index of the current line within that list.

     Note: Keeping references to frame objects, as found in the first
     element of the frame records these functions return, can cause
     your program to create reference cycles.  Once a reference cycle
     has been created, the lifespan of all objects which can be
     accessed from the objects which form the cycle can become much
     longer even if Python's optional cycle detector is enabled.  If
     such cycles must be created, it is important to ensure they are
     explicitly broken to avoid the delayed destruction of objects and
     increased memory consumption which occurs.

     Though the cycle detector will catch these, destruction of the
     frames (and local variables) can be made deterministic by removing
     the cycle in a finally clause.  This is also important
     if the cycle detector was disabled when Python was compiled or
     using gc.disable().  For example:

         def handle_stackframe_without_leak():
             frame = inspect.currentframe()
             try:
                 # do something with the frame
             finally:
                 del frame

  The optional _context_ argument supported by most of these functions
specifies the number of lines of context to return, which are centered
around the current line.

 -- Function: inspect.getframeinfo (frame[, context])
     Get information about a frame or traceback object.  A 5-tuple is
     returned, the last five elements of the frame's frame record.

     Changed in version 2.6: Returns a named tuple.
     `Traceback(filename, lineno, function, code_context, index)'.

 -- Function: inspect.getouterframes (frame[, context])
     Get a list of frame records for a frame and all outer frames.
     These frames represent the calls that lead to the creation of
     _frame_. The first entry in the returned list represents _frame_;
     the last entry represents the outermost call on _frame_'s stack.

 -- Function: inspect.getinnerframes (traceback[, context])
     Get a list of frame records for a traceback's frame and all inner
     frames.  These frames represent calls made as a consequence of
     _frame_.  The first entry in the list represents _traceback_; the
     last entry represents where the exception was raised.

 -- Function: inspect.currentframe ()
     Return the frame object for the caller's stack frame.

     *CPython implementation detail:* This function relies on Python
     stack frame support in the interpreter, which isn't guaranteed to
     exist in all implementations of Python.  If running in an
     implementation without Python stack frame support this function
     returns `None'.

 -- Function: inspect.stack ([context])
     Return a list of frame records for the caller's stack.  The first
     entry in the returned list represents the caller; the last entry
     represents the outermost call on the stack.

 -- Function: inspect.trace ([context])
     Return a list of frame records for the stack between the current
     frame and the frame in which an exception currently being handled
     was raised in.  The first entry in the list represents the caller;
     the last entry represents where the exception was raised.

5.27.14 `site' -- Site-specific configuration hook
--------------------------------------------------

*This module is automatically imported during initialization.* The
automatic import can be suppressed using the interpreter's -S
option.

  Importing this module will append site-specific paths to the module
search path.

  It starts by constructing up to four directories from a head and a
tail part.  For the head part, it uses `sys.prefix' and
`sys.exec_prefix'; empty heads are skipped.  For the tail part, it uses
the empty string and then `lib/site-packages' (on Windows) or
`lib/python|version|/site-packages' and then `lib/site-python' (on Unix
and Macintosh).  For each of the distinct head-tail combinations, it
sees if it refers to an existing directory, and if so, adds it to
`sys.path' and also inspects the newly added path for configuration
files.

  A path configuration file is a file whose name has the form
`package.pth' and exists in one of the four directories mentioned
above; its contents are additional items (one per line) to be added to
`sys.path'.  Non-existing items are never added to `sys.path', but no
check is made that the item refers to a directory (rather than a file).
No item is added to `sys.path' more than once.  Blank lines and lines
beginning with `#' are skipped.  Lines starting with `import' (followed
by space or tab) are executed.

  Changed in version 2.6: A space or tab is now required after the
import keyword.

  For example, suppose `sys.prefix' and `sys.exec_prefix' are set to
`/usr/local'.  The Python X.Y library is then installed in
`/usr/local/lib/python_X.Y_' (where only the first three characters of
`sys.version' are used to form the installation path name).  Suppose
this has a subdirectory `/usr/local/lib/python_X.Y_/site-packages' with
three subsubdirectories, `foo', `bar' and `spam', and two path
configuration files, `foo.pth' and `bar.pth'.  Assume `foo.pth'
contains the following:

    # foo package configuration

    foo
    bar
    bletch

and `bar.pth' contains:

    # bar package configuration

    bar

Then the following version-specific directories are added to
`sys.path', in this order:

    /usr/local/lib/pythonX.Y/site-packages/bar
    /usr/local/lib/pythonX.Y/site-packages/foo

Note that `bletch' is omitted because it doesn't exist; the `bar'
directory precedes the `foo' directory because `bar.pth' comes
alphabetically before `foo.pth'; and `spam' is omitted because it is
not mentioned in either path configuration file.

  After these path manipulations, an attempt is made to import a module
named `sitecustomize', which can perform arbitrary site-specific
customizations.  If this import fails with an ImportError.
exception, it is silently ignored.

  Note that for some non-Unix systems, `sys.prefix' and
`sys.exec_prefix' are empty, and the path manipulations are skipped;
however the import of `sitecustomize' is still attempted.

 -- Data: site.PREFIXES
     A list of prefixes for site package directories

     New in version 2.6.

 -- Data: site.ENABLE_USER_SITE
     Flag showing the status of the user site directory. True means the
     user site directory is enabled and added to sys.path. When the flag
     is None the user site directory is disabled for security reasons.

     New in version 2.6.

 -- Data: site.USER_SITE
     Path to the user site directory for the current Python version or
     None

     New in version 2.6.

 -- Data: site.USER_BASE
     Path to the base directory for user site directories

     New in version 2.6.

 -- envvar: PYTHONNOUSERSITE
     New in version 2.6.

 -- envvar: PYTHONUSERBASE
     New in version 2.6.

 -- Function: site.addsitedir (sitedir, known_paths=None)
     Adds a directory to sys.path and processes its pth files.

 -- Function: site.getsitepackages ()
     Returns a list containing all global site-packages directories
     (and possibly site-python).

     New in version 2.7.

 -- Function: site.getuserbase ()
     Returns the "user base" directory path.

     The "user base" directory can be used to store data. If the global
     variable `USER_BASE' is not initialized yet, this function will
     also set it.

     New in version 2.7.

 -- Function: site.getusersitepackages ()
     Returns the user-specific site-packages directory path.

     If the global variable `USER_SITE' is not initialized yet, this
     function will also set it.

     New in version 2.7.

5.27.15 `user' -- User-specific configuration hook
--------------------------------------------------

Deprecated since version 2.6: The user module has been
removed in Python 3.0.

  As a policy, Python doesn't run user-specified code on startup of
Python programs.  (Only interactive sessions execute the script
specified in the PYTHONSTARTUP environment variable if it
exists).

  However, some programs or sites may find it convenient to allow users
to have a standard customization file, which gets run when a program
requests it.  This module implements such a mechanism.  A program that
wishes to use the mechanism must execute the statement

    import user

The user module looks for a file `.pythonrc.py' in the
user's home directory and if it can be opened, executes it (using
execfile()) in its own (the module user.'s) global
namespace.  Errors during this phase are not caught; that's up to the
program that imports the user module, if it wishes.  The
home directory is assumed to be named by the `HOME' environment
variable; if this is not set, the current directory is used.

  The user's `.pythonrc.py' could conceivably test for `sys.version' if
it wishes to do different things depending on the Python version.

  A warning to users: be very conservative in what you place in your
`.pythonrc.py' file.  Since you don't know which programs will use it,
changing the behavior of standard modules or functions is generally not
a good idea.

  A suggestion for programmers who wish to use this mechanism: a simple
way to let users specify options for your package is to have them
define variables in their `.pythonrc.py' file that you test in your
module.  For example, a module `spam' that has a verbosity level can
look for a variable `user.spam_verbose', as follows:

    import user

    verbose = bool(getattr(user, "spam_verbose", 0))

(The three-argument form of getattr() is used in case the
user has not defined `spam_verbose' in their `.pythonrc.py' file.)

  Programs with extensive customization needs are better off reading a
program-specific customization file.

  Programs with security or privacy concerns should _not_ import this
module; a user can easily break into a program by placing arbitrary
code in the `.pythonrc.py' file.

  Modules for general use should _not_ import this module; it may
interfere with the operation of the importing program.

Module site.
     Site-wide customization mechanism.

5.27.16 `fpectl' -- Floating point exception control
----------------------------------------------------

     Note: The fpectl module is not built by default, and
     its usage is discouraged and may be dangerous except in the hands
     of experts.  See also the section Limitations and other considerations
     on limitations for more details.

Most computers carry out floating point operations in conformance with
the so-called IEEE-754 standard. On any real computer, some floating
point operations produce results that cannot be expressed as a normal
floating point value. For example, try

    >>> import math
    >>> math.exp(1000)
    inf
    >>> math.exp(1000) / math.exp(1000)
    nan

(The example above will work on many platforms. DEC Alpha may be one
exception.)  "Inf" is a special, non-numeric value in IEEE-754 that
stands for "infinity", and "nan" means "not a number." Note that, other
than the non-numeric results, nothing special happened when you asked
Python to carry out those calculations.  That is in fact the default
behaviour prescribed in the IEEE-754 standard, and if it works for you,
stop reading now.

  In some circumstances, it would be better to raise an exception and
stop processing at the point where the faulty operation was attempted.
The fpectl module is for use in that situation. It provides
control over floating point units from several hardware manufacturers,
allowing the user to turn on the generation of `SIGFPE' whenever any of
the IEEE-754 exceptions Division by Zero, Overflow, or Invalid
Operation occurs. In tandem with a pair of wrapper macros that are
inserted into the C code comprising your python system, `SIGFPE' is
trapped and converted into the Python FloatingPointError.
exception.

  The fpectl module defines the following functions and may
raise the given exception:

 -- Function: fpectl.turnon_sigfpe ()
     Turn on the generation of `SIGFPE', and set up an appropriate
     signal handler.

 -- Function: fpectl.turnoff_sigfpe ()
     Reset default handling of floating point exceptions.

 -- Exception: fpectl.FloatingPointError
     After turnon_sigfpe() has been executed, a floating
     point operation that raises one of the IEEE-754 exceptions
     Division by Zero, Overflow, or Invalid operation will in turn
     raise this standard Python exception.

5.27.16.1 Example
.................

The following example demonstrates how to start up and test operation
of the fpectl module.

    >>> import fpectl
    >>> import fpetest
    >>> fpectl.turnon_sigfpe()
    >>> fpetest.test()
    overflow        PASS
    FloatingPointError: Overflow

    div by 0        PASS
    FloatingPointError: Division by zero
      [ more output from test elided ]
    >>> import math
    >>> math.exp(1000)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    FloatingPointError: in math_1

5.27.16.2 Limitations and other considerations
..............................................

Setting up a given processor to trap IEEE-754 floating point errors
currently requires custom code on a per-architecture basis. You may
have to modify fpectl to control your particular hardware.

  Conversion of an IEEE-754 exception to a Python exception requires
that the wrapper macros `PyFPE_START_PROTECT' and `PyFPE_END_PROTECT'
be inserted into your code in an appropriate fashion.  Python itself
has been modified to support the fpectl module, but many
other codes of interest to numerical analysts have not.

  The fpectl module is not thread-safe.

  Some files in the source distribution may be interesting in learning
more about how this module operates. The include file `Include/pyfpe.h'
discusses the implementation of this module at some length.
`Modules/fpetestmodule.c' gives several examples of use. Many
additional examples can be found in `Objects/floatobject.c'.

5.27.17 `distutils' -- Building and installing Python modules
-------------------------------------------------------------

The distutils package provides support for building and
installing additional modules into a Python installation.  The new
modules may be either 100%-pure Python, or may be extension modules
written in C, or may be collections of Python packages which include
modules coded in both Python and C.

  This package is discussed in two separate chapters:

Distributing Python Modules.
     The manual for developers and packagers of Python modules.  This
     describes how to prepare distutils.-based packages so
     that they may be easily installed into an existing Python
     installation.

See Installing Python Modules.
     An "administrators" manual which includes information on
     installing modules into an existing Python installation. You do
     not need to be a Python programmer to read this manual.

5.28 Custom Python Interpreters
===============================

The modules described in this chapter allow writing interfaces similar
to Python's interactive interpreter.  If you want a Python interpreter
that supports some special feature in addition to the Python language,
you should look at the code module.  (The codeop.
module is lower-level, used to support compiling a possibly-incomplete
chunk of Python code.)

  The full list of modules described in this chapter is:

5.28.1 `code' -- Interpreter base classes
-----------------------------------------

The `code' module provides facilities to implement read-eval-print
loops in Python.  Two classes and convenience functions are included
which can be used to build applications which provide an interactive
interpreter prompt.

 -- Class: code.InteractiveInterpreter ([locals])
     This class deals with parsing and interpreter state (the user's
     namespace); it does not deal with input buffering or prompting or
     input file naming (the filename is always passed in explicitly).
     The optional _locals_ argument specifies the dictionary in which
     code will be executed; it defaults to a newly created dictionary
     with key `'__name__'' set to `'__console__'' and key `'__doc__''
     set to `None'.

 -- Class: code.InteractiveConsole ([locals[, filename]])
     Closely emulate the behavior of the interactive Python
     interpreter. This class builds on InteractiveInterpreter
     and adds prompting using the familiar `sys.ps1' and
     `sys.ps2', and input buffering.

 -- Function: code.interact ([banner[, readfunc[, local]]])
     Convenience function to run a read-eval-print loop.  This creates
     a new instance of InteractiveConsole and sets
     _readfunc_ to be used as the raw_input() method, if
     provided.  If _local_ is provided, it is passed to the
     InteractiveConsole constructor for use as the default
     namespace for the interpreter loop.  The interact().
     method of the instance is then run with _banner_ passed as the
     banner to use, if provided.  The console object is discarded after
     use.

 -- Function: code.compile_command (source[, filename[, symbol]])
     This function is useful for programs that want to emulate Python's
     interpreter main loop (a.k.a. the read-eval-print loop).  The
     tricky part is to determine when the user has entered an
     incomplete command that can be completed by entering more text (as
     opposed to a complete command or a syntax error).  This function
     _almost_ always makes the same decision as the real interpreter
     main loop.

     _source_ is the source string; _filename_ is the optional filename
     from which source was read, defaulting to `'<input>''; and
     _symbol_ is the optional grammar start symbol, which should be
     either `'single'' (the default) or `'eval''.

     Returns a code object (the same as `compile(source, filename,
     symbol)') if the command is complete and valid; `None' if the
     command is incomplete; raises SyntaxError if the
     command is complete and contains a syntax error, or raises
     OverflowError or ValueError if the command
     contains an invalid literal.

5.28.1.1 Interactive Interpreter Objects
........................................

 -- Method: InteractiveInterpreter.runsource (source[, filename[,
          symbol]])
     Compile and run some source in the interpreter. Arguments are the
     same as for compile_command().; the default for
     _filename_ is `'<input>'', and for _symbol_ is `'single''.  One
     several things can happen:

        * The input is incorrect; see compile_command() raised
          an exception (see SyntaxError or OverflowError).
          A syntax traceback will be printed by calling the
          showsyntaxerror() method.  See runsource().
          returns `False'.

        * The input is incomplete, and more input is required; see
          compile_command().  returned `None'. runsource()
          returns `True'.

        * The input is complete; see compile_command().
          returned a code object.  The code is executed by calling the
          runcode(). (which also handles run-time
          exceptions, except for SystemExit).
          runsource() returns `False'.

     The return value can be used to decide whether to use `sys.ps1' or
     `sys.ps2' to prompt the next line.

 -- Method: InteractiveInterpreter.runcode (code)
     Execute a code object. When an exception occurs, see
     showtraceback() is called to display a traceback.  All
     exceptions are caught except SystemExit, which is
     allowed to propagate.

     A note about KeyboardInterrupt.: this exception may
     occur elsewhere in this code, and may not always be caught.  The
     caller should be prepared to deal with it.

 -- Method: InteractiveInterpreter.showsyntaxerror ([filename])
     Display the syntax error that just occurred.  This does not
     display a stack trace because there isn't one for syntax errors.
     If _filename_ is given, it is stuffed into the exception instead
     of the default filename provided by Python's parser, because it
     always uses `'<string>'' when reading from a string. The output is
     written by the write() method.

 -- Method: InteractiveInterpreter.showtraceback ()
     Display the exception that just occurred.  We remove the first
     stack item because it is within the interpreter object
     implementation. The output is written by the write().
     method.

 -- Method: InteractiveInterpreter.write (data)
     Write a string to the standard error stream (`sys.stderr').
     Derived classes should override this to provide the appropriate
     output handling as needed.

5.28.1.2 Interactive Console Objects
....................................

The InteractiveConsole class is a subclass of
InteractiveInterpreter, and so offers all the methods of the
interpreter objects as well as the following additions.

 -- Method: InteractiveConsole.interact ([banner])
     Closely emulate the interactive Python console. The optional
     banner argument specify the banner to print before the first
     interaction; by default it prints a banner similar to the one
     printed by the standard Python interpreter, followed by the class
     name of the console object in parentheses (so as not to confuse
     this with the real interpreter - since it's so close!).

 -- Method: InteractiveConsole.push (line)
     Push a line of source text to the interpreter. The line should not
     have a trailing newline; it may have internal newlines.  The line
     is appended to a buffer and the interpreter's `runsource()' method
     is called with the concatenated contents of the buffer as source.
     If this indicates that the command was executed or invalid, the
     buffer is reset; otherwise, the command is incomplete, and the
     buffer is left as it was after the line was appended.  The return
     value is `True' if more input is required, `False' if the line was
     dealt with in some way (this is the same as `runsource()').

 -- Method: InteractiveConsole.resetbuffer ()
     Remove any unhandled source text from the input buffer.

 -- Method: InteractiveConsole.raw_input ([prompt])
     Write a prompt and read a line.  The returned line does not
     include the trailing newline.  When the user enters the EOF key
     sequence, see EOFError is raised.  The base implementation
     uses the built-in function raw_input().; a subclass may
     replace this with a different implementation.

5.28.2 `codeop' -- Compile Python code
--------------------------------------

The codeop module provides utilities upon which the Python
read-eval-print loop can be emulated, as is done in the code.
module.  As a result, you probably don't want to use the module
directly; if you want to include such a loop in your program you
probably want to use the code.  module instead.

  There are two parts to this job:

  1. Being able to tell if a line of input completes a Python
     statement: in short, telling whether to print '`>>>'' or '`...''
     next.

  2. Remembering which future statements the user has entered, so
     subsequent input can be compiled with these in effect.

  The codeop module provides a way of doing each of these
things, and a way of doing them both.

  To do just the former:

 -- Function: codeop.compile_command (source[, filename[, symbol]])
     Tries to compile _source_, which should be a string of Python code
     and return a code object if _source_ is valid Python code. In that
     case, the filename attribute of the code object will be
     _filename_, which defaults to `'<input>''. Returns `None' if
     _source_ is _not_ valid Python code, but is a prefix of valid
     Python code.

     If there is a problem with _source_, an exception will be raised.
     SyntaxError is raised if there is invalid Python
     syntax, and OverflowError or ValueError if
     there is an invalid literal.

     The _symbol_ argument determines whether _source_ is compiled as a
     statement (`'single'', the default) or as an expression.
     (`'eval'').  Any other value will cause ValueError
     to  be raised.

          Note: It is possible (but not likely) that the parser stops
          parsing with a successful outcome before reaching the end of
          the source; in this case, trailing symbols may be ignored
          instead of causing an error.  For example, a backslash
          followed by two newlines may be followed by arbitrary garbage.
          This will be fixed once the API for the parser is better.

 -- Class: codeop.Compile
     Instances of this class have __call__() methods
     identical in signature to the built-in function compile(),
     but with the difference that if the instance compiles program
     text containing a __future__ statement, the instance
     'remembers' and compiles all subsequent program texts with the
     statement in force.

 -- Class: codeop.CommandCompiler
     Instances of this class have __call__() methods
     identical in signature to compile_command().; the
     difference is that if the instance compiles program text
     containing a `__future__' statement, the instance 'remembers' and
     compiles all subsequent program texts with the statement in force.

  A note on version compatibility: the Compile and
CommandCompiler are new in Python 2.2.  If you want to enable the
future-tracking features of 2.2 but also retain compatibility with 2.1
and earlier versions of Python you can either write

    try:
        from codeop import CommandCompiler
        compile_command = CommandCompiler()
        del CommandCompiler
    except ImportError:
        from codeop import compile_command

which is a low-impact change, but introduces possibly unwanted global
state into your program, or you can write:

    try:
        from codeop import CommandCompiler
    except ImportError:
        def CommandCompiler():
            from codeop import compile_command
            return compile_command

and then call `CommandCompiler' every time you need a fresh compiler
object.

5.29 Restricted Execution
=========================

     Warning: In Python 2.3 these modules have been disabled due to
     various known and not readily fixable security holes.  The modules
     are still documented here to help in reading old code that uses
     the rexec and Bastion modules.

_Restricted execution_ is the basic framework in Python that allows for
the segregation of trusted and untrusted code.  The framework is based
on the notion that trusted Python code (a _supervisor_) can create a
"padded cell' (or environment) with limited permissions, and run the
untrusted code within this cell.  The untrusted code cannot break out
of its cell, and can only interact with sensitive system resources
through interfaces defined and managed by the trusted code.  The term
"restricted execution" is favored over "safe-Python" since true safety
is hard to define, and is determined by the way the restricted
environment is created.  Note that the restricted environments can be
nested, with inner cells creating subcells of lesser, but never
greater, privilege.

  An interesting aspect of Python's restricted execution model is that
the interfaces presented to untrusted code usually have the same names
as those presented to trusted code.  Therefore no special interfaces
need to be learned to write code designed to run in a restricted
environment.  And because the exact nature of the padded cell is
determined by the supervisor, different restrictions can be imposed,
depending on the application.  For example, it might be deemed "safe"
for untrusted code to read any file within a specified directory, but
never to write a file.  In this case, the supervisor may redefine the
built-in open() function so that it raises an exception
whenever the _mode_ parameter is `'w''.  It might also perform a
`chroot()'-like operation on the _filename_ parameter, such that root
is always relative to some safe "sandbox" area of the filesystem.  In
this case, the untrusted code would still see an built-in open()
function in its environment, with the same calling interface.  The
semantics would be identical too, with IOError.s being
raised when the supervisor determined that an unallowable parameter is
being used.

  The Python run-time determines whether a particular code block is
executing in restricted execution mode based on the identity of the
`__builtins__' object in its global variables: if this is (the
dictionary of) the standard __builtin__ module, the code is
deemed to be unrestricted, else it is deemed to be restricted.

  Python code executing in restricted mode faces a number of
limitations that are designed to prevent it from escaping from the
padded cell. For instance, the function object attribute `func_globals'
and the class and instance object attribute `__dict__' are unavailable.

  Two modules provide the framework for setting up restricted execution
environments:

5.29.1 `rexec' -- Restricted execution framework
------------------------------------------------

Deprecated since version 2.6: The rexec module has been
removed in Python 3.0.

  Changed in version 2.3: Disabled module.

     Warning: The documentation has been left in place to help in
     reading old code that uses the module.

  This module contains the RExec class, which supports
`r_eval()', `r_execfile()', `r_exec()', and `r_import()' methods, which
are restricted versions of the standard Python functions eval(),
see execfile() and the exec and import
statements. Code executed in this restricted environment will only
have access to modules and functions that are deemed safe; you can
subclass RExec to add or remove capabilities as desired.

     Warning: While the rexec module is designed to perform
     as described below, it does have a few known vulnerabilities which
     could be exploited by carefully written code.  Thus it should not
     be relied upon in situations requiring "production ready"
     security.  In such situations, execution via sub-processes or very
     careful "cleansing" of both code and data to be processed may be
     necessary.  Alternatively, help in patching known rexec
     vulnerabilities would be welcomed.

     Note: The RExec class can prevent code from
     performing unsafe operations like reading or writing disk files,
     or using TCP/IP sockets.  However, it does not protect against
     code using extremely large amounts of memory or processor time.

 -- Class: rexec.RExec ([hooks[, verbose]])
     Returns an instance of the RExec class.

     _hooks_ is an instance of the `RHooks' class or a subclass of it.
     If it is omitted or `None', the default `RHooks' class is
     instantiated.  Whenever the rexec module searches for
     a module (even a built-in one) or reads a module's code, it
     doesn't actually go out to the file system itself.  Rather, it
     calls methods of an `RHooks' instance that was passed to or
     created by its constructor.  (Actually, the RExec.
     object doesn't make these calls -- they are made by a module
     loader object that's part of the RExec object.  This
     allows another level of flexibility, which can be useful when
     changing the mechanics of import within the restricted
     environment.)

     By providing an alternate `RHooks' object, we can control the file
     system accesses made to import a module, without changing the
     actual algorithm that controls the order in which those accesses
     are made.  For instance, we could substitute an `RHooks' object
     that passes all filesystem requests to a file server elsewhere,
     via some RPC mechanism such as ILU.  Grail's applet loader uses
     this to support importing applets from a URL for a directory.

     If _verbose_ is true, additional debugging output may be sent to
     standard output.

  It is important to be aware that code running in a restricted
environment can still call the sys.exit() function.  To
disallow restricted code from exiting the interpreter, always protect
calls that cause restricted code to run with a try./see
except statement that catches the SystemExit.
exception.  Removing the sys.exit() function from the
restricted environment is not sufficient -- the restricted code could
still use `raise SystemExit'.  Removing SystemExit is not a
reasonable option; some library code makes use of this and would break
were it not available.

Grail Home Page (http://grail.sourceforge.net/)
     Grail is a Web browser written entirely in Python.  It uses the
     rexec.  module as a foundation for supporting Python
     applets, and can be used as an example usage of this module.

5.29.1.1 RExec Objects
......................

RExec instances support the following methods:

 -- Method: RExec.r_eval (code)
     _code_ must either be a string containing a Python expression, or
     a compiled code object, which will be evaluated in the restricted
     environment's __main__ module.  The value of the
     expression or code object will be returned.

 -- Method: RExec.r_exec (code)
     _code_ must either be a string containing one or more lines of
     Python code, or a compiled code object, which will be executed in
     the restricted environment's __main__ module.

 -- Method: RExec.r_execfile (filename)
     Execute the Python code contained in the file _filename_ in the
     restricted environment's __main__ module.

  Methods whose names begin with `s_' are similar to the functions
beginning with `r_', but the code will be granted access to restricted
versions of the standard I/O streams `sys.stdin', `sys.stderr', and
`sys.stdout'.

 -- Method: RExec.s_eval (code)
     _code_ must be a string containing a Python expression, which will
     be evaluated in the restricted environment.

 -- Method: RExec.s_exec (code)
     _code_ must be a string containing one or more lines of Python
     code, which will be executed in the restricted environment.

 -- Method: RExec.s_execfile (code)
     Execute the Python code contained in the file _filename_ in the
     restricted environment.

  RExec objects must also support various methods which
will be implicitly called by code executing in the restricted
environment. Overriding these methods in a subclass is used to change
the policies enforced by a restricted environment.

 -- Method: RExec.r_import (modulename[, globals[, locals[, fromlist]]])
     Import the module _modulename_, raising an ImportError.
     exception if the module is considered unsafe.

 -- Method: RExec.r_open (filename[, mode[, bufsize]])
     Method called when open() is called in the restricted
     environment.  The arguments are identical to those of
     open(), and a file object (or a class instance compatible
     with file objects) should be returned.  See RExec.'s
     default behaviour is allow opening any file for reading, but
     forbidding any attempt to write a file.  See the example below for
     an implementation of a less restrictive r_open().

 -- Method: RExec.r_reload (module)
     Reload the module object _module_, re-parsing and re-initializing
     it.

 -- Method: RExec.r_unload (module)
     Unload the module object _module_ (remove it from the restricted
     environment's `sys.modules' dictionary).

  And their equivalents with access to restricted standard I/O streams:

 -- Method: RExec.s_import (modulename[, globals[, locals[, fromlist]]])
     Import the module _modulename_, raising an ImportError.
     exception if the module is considered unsafe.

 -- Method: RExec.s_reload (module)
     Reload the module object _module_, re-parsing and re-initializing
     it.

 -- Method: RExec.s_unload (module)
     Unload the module object _module_.

5.29.1.2 Defining restricted environments
.........................................

The RExec class has the following class attributes, which
are used by the __init__() method.  Changing them on an
existing instance won't have any effect; instead, create a subclass of
RExec and assign them new values in the class definition.
Instances of the new class will then use those new values.  All these
attributes are tuples of strings.

 -- Attribute: RExec.nok_builtin_names
     Contains the names of built-in functions which will _not_ be
     available to programs running in the restricted environment.  The
     value for RExec is `('open', 'reload',
     '__import__')'. (This gives the exceptions, because by far the
     majority of built-in functions are harmless.  A subclass that
     wants to override this variable should probably start with the
     value from the base class and concatenate additional forbidden
     functions -- when new dangerous built-in functions are added to
     Python, they will also be added to this module.)

 -- Attribute: RExec.ok_builtin_modules
     Contains the names of built-in modules which can be safely
     imported. The value for RExec is `('audioop',
     'array', 'binascii', 'cmath', 'errno', 'imageop', 'marshal',
     'math', 'md5', 'operator', 'parser', 'regex', 'select', 'sha',
     '_sre', 'strop', 'struct', 'time')'.  A similar remark about
     overriding this variable applies -- use the value from the base
     class as a starting point.

 -- Attribute: RExec.ok_path
     Contains the directories which will be searched when an
     import is performed in the restricted environment.   The
     value for RExec is the same as `sys.path' (at the
     time the module is loaded) for unrestricted code.

 -- Attribute: RExec.ok_posix_names
     Contains the names of the functions in the os module
     which will be available to programs running in the restricted
     environment.  The value for RExec is `('error',
     'fstat', 'listdir', 'lstat', 'readlink', 'stat', 'times', 'uname',
     'getpid', 'getppid', 'getcwd', 'getuid', 'getgid', 'geteuid',
     'getegid')'.

 -- Attribute: RExec.ok_sys_names
     Contains the names of the functions and variables in the
     sys module which will be available to programs running in
     the restricted environment.  The value for RExec is
     `('ps1', 'ps2', 'copyright', 'version', 'platform', 'exit',
     'maxint')'.

 -- Attribute: RExec.ok_file_types
     Contains the file types from which modules are allowed to be
     loaded. Each file type is an integer constant defined in the
     imp module. The meaningful values are `PY_SOURCE',
     `PY_COMPILED', and `C_EXTENSION'.  The value for RExec
     is `(C_EXTENSION, PY_SOURCE)'.  Adding `PY_COMPILED' in
     subclasses is not recommended; an attacker could exit the
     restricted execution mode by putting a forged byte-compiled file
     (`.pyc') anywhere in your file system, for example by writing it to
     `/tmp' or uploading it to the `/incoming' directory of your public
     FTP server.

5.29.1.3 An example
...................

Let us say that we want a slightly more relaxed policy than the standard
RExec class.  For example, if we're willing to allow files
in `/tmp' to be written, we can subclass the RExec class:

    class TmpWriterRExec(rexec.RExec):
        def r_open(self, file, mode='r', buf=-1):
            if mode in ('r', 'rb'):
                pass
            elif mode in ('w', 'wb', 'a', 'ab'):
                # check filename : must begin with /tmp/
                if file[:5]!='/tmp/':
                    raise IOError("can't write outside /tmp")
                elif (string.find(file, '/../') >= 0 or
                     file[:3] == '../' or file[-3:] == '/..'):
                    raise IOError("'..' in filename forbidden")
            else: raise IOError("Illegal open() mode")
            return open(file, mode, buf)

Notice that the above code will occasionally forbid a perfectly valid
filename; for example, code in the restricted environment won't be able
to open a file called `/tmp/foo/../bar'.  To fix this, the `r_open()'
method would have to simplify the filename to `/tmp/bar', which would
require splitting apart the filename and performing various operations
on it.  In cases where security is at stake, it may be preferable to
write simple code which is sometimes overly restrictive, instead of
more general code that is also more complex and may harbor a subtle
security hole.

5.29.2 `Bastion' -- Restricting access to objects
-------------------------------------------------

Deprecated since version 2.6: The Bastion module has been
removed in Python 3.0.

  Changed in version 2.3: Disabled module.

     Note: The documentation has been left in place to help in reading
     old code that uses the module.

  According to the dictionary, a bastion is "a fortified area or
position", or "something that is considered a stronghold."  It's a
suitable name for this module, which provides a way to forbid access to
certain attributes of an object.  It must always be used with the
rexec module, in order to allow restricted-mode programs access
to certain safe attributes of an object, while denying access to other,
unsafe attributes.

 -- Function: Bastion.Bastion (object[, filter[, name[, class]]])
     Protect the object _object_, returning a bastion for the object.
     Any attempt to access one of the object's attributes will have to
     be approved by the _filter_ function; if the access is denied an
     AttributeError exception will be raised.

     If present, _filter_ must be a function that accepts a string
     containing an attribute name, and returns true if access to that
     attribute will be permitted; if _filter_ returns false, the access
     is denied.  The default filter denies access to any function
     beginning with an underscore (`'_'').  The bastion's string
     representation will be `<Bastion for name>' if a value for _name_
     is provided; otherwise, `repr(object)' will be used.

     _class_, if present, should be a subclass of BastionClass.;
     see the code in `bastion.py' for the details.  Overriding
     the default BastionClass will rarely be required.

 -- Class: Bastion.BastionClass (getfunc, name)
     Class which actually implements bastion objects.  This is the
     default class used by Bastion().  The _getfunc_
     parameter is a function which returns the value of an attribute
     which should be exposed to the restricted execution environment
     when called with the name of the attribute as the only parameter.
     _name_ is used to construct the repr() of the
     BastionClass.  instance.

Grail Home Page (http://grail.sourceforge.net/)
     Grail, an Internet browser written in Python, uses these modules
     to support Python applets.  More information on the use of
     Python's restricted execution mode in Grail is available on the
     Web site.

5.30 Importing Modules
======================

The modules described in this chapter provide new ways to import other
Python modules and hooks for customizing the import process.

  The full list of modules described in this chapter is:

5.30.1 `imp' -- Access the `import' internals
---------------------------------------------

This module provides an interface to the mechanisms used to implement
the import statement.  It defines the following constants
and functions:

 -- Function: imp.get_magic ()
     Return the magic string value used to recognize byte-compiled code
     files (`.pyc' files).  (This value may be different for each
     Python version.)

 -- Function: imp.get_suffixes ()
     Return a list of 3-element tuples, each describing a particular
     type of module. Each triple has the form `(suffix, mode, type)',
     where _suffix_ is a string to be appended to the module name to
     form the filename to search for, _mode_ is the mode string to pass
     to the built-in open() function to open the file (this
     can be `'r'' for text files or `'rb'' for binary files), and
     _type_ is the file type, which has one of the values
     PY_SOURCE, PY_COMPILED, or C_EXTENSION,
     described below.

 -- Function: imp.find_module (name[, path])
     Try to find the module _name_.  If _path_ is omitted or `None',
     the list of directory names given by `sys.path' is searched, but
     first a few special places are searched: the function tries to
     find a built-in module with the given name (see C_BUILTIN),
     then a frozen module (see PY_FROZEN), and on some
     systems some other places are looked in as well (on Windows, it
     looks in the registry which may point to a specific file).

     Otherwise, _path_ must be a list of directory names; each
     directory is searched for files with any of the suffixes returned
     by get_suffixes().  above.  Invalid names in the list
     are silently ignored (but all list items must be strings).

     If search is successful, the return value is a 3-element tuple
     `(file, pathname, description)':

     _file_ is an open file object positioned at the beginning,
     _pathname_ is the pathname of the file found, and _description_ is
     a 3-element tuple as contained in the list returned by
     get_suffixes() describing the kind of module found.

     If the module does not live in a file, the returned _file_ is
     `None', _pathname_ is the empty string, and the _description_
     tuple contains empty strings for its suffix and mode; the module
     type is indicated as given in parentheses above.  If the search is
     unsuccessful, see ImportError is raised.  Other exceptions
     indicate problems with the arguments or environment.

     If the module is a package, _file_ is `None', _pathname_ is the
     package path and the last item in the _description_ tuple is
     PKG_DIRECTORY.

     This function does not handle hierarchical module names (names
     containing dots).  In order to find _P_.*M*, that is, submodule
     _M_ of package _P_, use find_module() and
     load_module() to find and load package _P_, and then use
     find_module() with the _path_ argument set to
     `P.__path__'.  When _P_ itself has a dotted name, apply this
     recipe recursively.

 -- Function: imp.load_module (name, file, pathname, description)
     Load a module that was previously found by find_module().
     (or by an otherwise conducted search yielding compatible
     results).  This function does more than importing the module: if
     the module was already imported, it is equivalent to a
     reload().!  The _name_ argument indicates the full module
     name (including the package name, if this is a submodule of a
     package).  The _file_ argument is an open file, and _pathname_ is
     the corresponding file name; these can be `None' and `''',
     respectively, when the module is a package or not being loaded
     from a file.  The _description_ argument is a tuple, as would be
     returned by get_suffixes(), describing what kind of
     module must be loaded.

     If the load is successful, the return value is the module object;
     otherwise, an exception (usually ImportError) is
     raised.

     *Important:* the caller is responsible for closing the _file_
     argument, if it was not `None', even when an exception is raised.
     This is best done using a try... See finally.
     statement.

 -- Function: imp.new_module (name)
     Return a new empty module object called _name_.  This object is
     _not_ inserted in `sys.modules'.

 -- Function: imp.lock_held ()
     Return `True' if the import lock is currently held, else `False'.
     On platforms without threads, always return `False'.

     On platforms with threads, a thread executing an import holds an
     internal lock until the import is complete. This lock blocks other
     threads from doing an import until the original import completes,
     which in turn prevents other threads from seeing incomplete module
     objects constructed by the original thread while in the process of
     completing its import (and the imports, if any, triggered by that).

 -- Function: imp.acquire_lock ()
     Acquire the interpreter's import lock for the current thread.
     This lock should be used by import hooks to ensure thread-safety
     when importing modules.

     Once a thread has acquired the import lock, the same thread may
     acquire it again without blocking; the thread must release it once
     for each time it has acquired it.

     On platforms without threads, this function does nothing.

     New in version 2.3.

 -- Function: imp.release_lock ()
     Release the interpreter's import lock. On platforms without
     threads, this function does nothing.

     New in version 2.3.

  The following constants with integer values, defined in this module,
are used to indicate the search result of find_module().

 -- Data: imp.PY_SOURCE
     The module was found as a source file.

 -- Data: imp.PY_COMPILED
     The module was found as a compiled code object file.

 -- Data: imp.C_EXTENSION
     The module was found as dynamically loadable shared library.

 -- Data: imp.PKG_DIRECTORY
     The module was found as a package directory.

 -- Data: imp.C_BUILTIN
     The module was found as a built-in module.

 -- Data: imp.PY_FROZEN
     The module was found as a frozen module (see init_frozen()).

  The following constant and functions are obsolete; their
functionality is available through find_module() or
load_module(). They are kept around for backward compatibility:

 -- Data: imp.SEARCH_ERROR
     Unused.

 -- Function: imp.init_builtin (name)
     Initialize the built-in module called _name_ and return its module
     object along with storing it in `sys.modules'.  If the module was
     already initialized, it will be initialized _again_.
     Re-initialization involves the copying of the built-in module's
     `__dict__' from the cached module over the module's entry in
     `sys.modules'.  If there is no built-in module called _name_,
     `None' is returned.

 -- Function: imp.init_frozen (name)
     Initialize the frozen module called _name_ and return its module
     object.  If the module was already initialized, it will be
     initialized _again_.  If there is no frozen module called _name_,
     `None' is returned.  (Frozen modules are modules written in Python
     whose compiled byte-code object is incorporated into a
     custom-built Python interpreter by Python's *freeze* utility. See
     `Tools/freeze/' for now.)

 -- Function: imp.is_builtin (name)
     Return `1' if there is a built-in module called _name_ which can be
     initialized again.  Return `-1' if there is a built-in module
     called _name_ which cannot be initialized again (see
     init_builtin()).  Return `0' if there is no built-in module
     called _name_.

 -- Function: imp.is_frozen (name)
     Return `True' if there is a frozen module (see
     init_frozen()) called _name_, or `False' if there is no
     such module.

 -- Function: imp.load_compiled (name, pathname[, file])
     Load and initialize a module implemented as a byte-compiled code
     file and return its module object.  If the module was already
     initialized, it will be initialized _again_.  The _name_ argument
     is used to create or access a module object.  The _pathname_
     argument points to the byte-compiled code file.  The _file_
     argument is the byte-compiled code file, open for reading in
     binary mode, from the beginning. It must currently be a real file
     object, not a user-defined class emulating a file.

 -- Function: imp.load_dynamic (name, pathname[, file])
     Load and initialize a module implemented as a dynamically loadable
     shared library and return its module object.  If the module was
     already initialized, it will be initialized _again_.
     Re-initialization involves copying the `__dict__' attribute of the
     cached instance of the module over the value used in the module
     cached in `sys.modules'.  The _pathname_ argument must point to
     the shared library.  The _name_ argument is used to construct the
     name of the initialization function: an external C function called
     `initname()' in the shared library is called.  The optional _file_
     argument is ignored.  (Note: using shared libraries is highly
     system dependent, and not all systems support it.)

 -- Function: imp.load_source (name, pathname[, file])
     Load and initialize a module implemented as a Python source file
     and return its module object.  If the module was already
     initialized, it will be initialized _again_.  The _name_ argument
     is used to create or access a module object.  The _pathname_
     argument points to the source file.  The _file_ argument is the
     source file, open for reading as text, from the beginning. It must
     currently be a real file object, not a user-defined class
     emulating a file.  Note that if a properly matching byte-compiled
     file (with suffix `.pyc' or `.pyo') exists, it will be used
     instead of parsing the given source file.

 -- Class: imp.NullImporter (path_string)
     The NullImporter type is a PEP 302
     (http://www.python.org/dev/peps/pep-0302) import hook that handles
     non-directory path strings by failing to find any modules.
     Calling this type with an existing directory or empty string
     raises ImportError.  Otherwise, a NullImporter
     instance is returned.

     Python adds instances of this type to `sys.path_importer_cache'
     for any path entries that are not directories and are not handled
     by any other path hooks on `sys.path_hooks'.  Instances have only
     one method:

      -- Method: find_module (fullname[, path])
          This method always returns `None', indicating that the
          requested module could not be found.

     New in version 2.5.

5.30.1.1 Examples
.................

The following function emulates what was the standard import statement
up to Python 1.4 (no hierarchical module names).  (This
_implementation_ wouldn't work in that version, since
find_module() has been extended and load_module().
has been added in 1.4.)

    import imp
    import sys

    def __import__(name, globals=None, locals=None, fromlist=None):
        # Fast path: see if the module has already been imported.
        try:
            return sys.modules[name]
        except KeyError:
            pass

        # If any of the following calls raises an exception,
        # there's a problem we can't handle -- let the caller handle it.

        fp, pathname, description = imp.find_module(name)

        try:
            return imp.load_module(name, fp, pathname, description)
        finally:
            # Since we may exit via an exception, close fp explicitly.
            if fp:
                fp.close()

A more complete example that implements hierarchical module names and
includes a reload() function can be found in the module
`knee'.  The `knee' module can be found in `Demo/imputil/' in the
Python source distribution.

5.30.2 `importlib' - Convenience wrappers for `__import__()'
------------------------------------------------------------

New in version 2.7.

  This module is a minor subset of what is available in the more
full-featured package of the same name from Python 3.1 that provides a
complete implementation of import. What is here has been
provided to help ease in transitioning from 2.7 to 3.1.

 -- Function: importlib.import_module (name, package=None)
     Import a module. The _name_ argument specifies what module to
     import in absolute or relative terms (e.g. either `pkg.mod' or
     `..mod'). If the name is specified in relative terms, then the
     _package_ argument must be specified to the package which is to
     act as the anchor for resolving the package name (e.g.
     `import_module('..mod', 'pkg.subpkg')' will import `pkg.mod').
     The specified module will be inserted into sys.modules.
     and returned.

5.30.3 `imputil' -- Import utilities
------------------------------------

Deprecated since version 2.6: The imputil module has been
removed in Python 3.0.

  This module provides a very handy and useful mechanism for custom
import hooks. Compared to the older `ihooks' module, see
imputil takes a dramatically simpler and more straight-forward
approach to custom import functions.

 -- Class: imputil.ImportManager ([fs_imp])
     Manage the import process.

      -- Method: install ([namespace])
          Install this ImportManager into the specified namespace.

      -- Method: uninstall ()
          Restore the previous import mechanism.

      -- Method: add_suffix (suffix, importFunc)
          Undocumented.

 -- Class: imputil.Importer
     Base class for replacing standard import functions.

      -- Method: import_top (name)
          Import a top-level module.

      -- Method: get_code (parent, modname, fqname)
          Find and retrieve the code for the given module.

          _parent_ specifies a parent module to define a context for
          importing.  It may be `None', indicating no particular
          context for the search.

          _modname_ specifies a single module (not dotted) within the
          parent.

          _fqname_ specifies the fully-qualified module name. This is a
          (potentially) dotted name from the "root" of the module
          namespace down to the modname.

          If there is no parent, then modname==fqname.

          This method should return `None', or a 3-tuple.

                  * If the module was not found, then `None' should be
                    returned.

                  * The first item of the 2- or 3-tuple should be the
                    integer 0 or 1, specifying whether the module that
                    was found is a package or not.

                  * The second item is the code object for the module
                    (it will be executed within the new module's
                    namespace). This item can also be a fully-loaded
                    module object (e.g. loaded from a shared lib).

                  * The third item is a dictionary of name/value pairs
                    that will be inserted into new module before the
                    code object is executed. This is provided in case
                    the module's code expects certain values (such as
                    where the module was found). When the second item
                    is a module object, then these names/values will be
                    inserted _after_ the module has been
                    loaded/initialized.

 -- Class: imputil.BuiltinImporter
     Emulate the import mechanism for built-in and frozen modules.
     This is a sub-class of the Importer class.

      -- Method: get_code (parent, modname, fqname)
          Undocumented.

 -- Function: imputil.py_suffix_importer (filename, finfo, fqname)
     Undocumented.

 -- Class: imputil.DynLoadSuffixImporter ([desc])
     Undocumented.

      -- Method: import_file (filename, finfo, fqname)
          Undocumented.

5.30.3.1 Examples
.................

This is a re-implementation of hierarchical module import.

  This code is intended to be read, not executed.  However, it does work
- all you need to do to enable it is "import knee".

  (The name is a pun on the clunkier predecessor of this module, "ni".)

    import sys, imp, __builtin__

    # Replacement for __import__()
    def import_hook(name, globals=None, locals=None, fromlist=None):
        parent = determine_parent(globals)
        q, tail = find_head_package(parent, name)
        m = load_tail(q, tail)
        if not fromlist:
            return q
        if hasattr(m, "__path__"):
            ensure_fromlist(m, fromlist)
        return m

    def determine_parent(globals):
        if not globals or  not globals.has_key("__name__"):
            return None
        pname = globals['__name__']
        if globals.has_key("__path__"):
            parent = sys.modules[pname]
            assert globals is parent.__dict__
            return parent
        if '.' in pname:
            i = pname.rfind('.')
            pname = pname[:i]
            parent = sys.modules[pname]
            assert parent.__name__ == pname
            return parent
        return None

    def find_head_package(parent, name):
        if '.' in name:
            i = name.find('.')
            head = name[:i]
            tail = name[i+1:]
        else:
            head = name
            tail = ""
        if parent:
            qname = "%s.%s" % (parent.__name__, head)
        else:
            qname = head
        q = import_module(head, qname, parent)
        if q: return q, tail
        if parent:
            qname = head
            parent = None
            q = import_module(head, qname, parent)
            if q: return q, tail
        raise ImportError("No module named " + qname)

    def load_tail(q, tail):
        m = q
        while tail:
            i = tail.find('.')
            if i < 0: i = len(tail)
            head, tail = tail[:i], tail[i+1:]
            mname = "%s.%s" % (m.__name__, head)
            m = import_module(head, mname, m)
            if not m:
                raise ImportError("No module named " + mname)
        return m

    def ensure_fromlist(m, fromlist, recursive=0):
        for sub in fromlist:
            if sub == "*":
                if not recursive:
                    try:
                        all = m.__all__
                    except AttributeError:
                        pass
                    else:
                        ensure_fromlist(m, all, 1)
                continue
            if sub != "*" and not hasattr(m, sub):
                subname = "%s.%s" % (m.__name__, sub)
                submod = import_module(sub, subname, m)
                if not submod:
                    raise ImportError("No module named " + subname)

    def import_module(partname, fqname, parent):
        try:
            return sys.modules[fqname]
        except KeyError:
            pass
        try:
            fp, pathname, stuff = imp.find_module(partname,
                                                  parent and parent.__path__)
        except ImportError:
            return None
        try:
            m = imp.load_module(fqname, fp, pathname, stuff)
        finally:
            if fp: fp.close()
        if parent:
            setattr(parent, partname, m)
        return m

    # Replacement for reload()
    def reload_hook(module):
        name = module.__name__
        if '.' not in name:
            return import_module(name, name, None)
        i = name.rfind('.')
        pname = name[:i]
        parent = sys.modules[pname]
        return import_module(name[i+1:], name, parent)

    # Save the original hooks
    original_import = __builtin__.__import__
    original_reload = __builtin__.reload

    # Now install our hooks
    __builtin__.__import__ = import_hook
    __builtin__.reload = reload_hook

Also see the `importers' module (which can be found in `Demo/imputil/'
in the Python source distribution) for additional examples.

5.30.4 `zipimport' -- Import modules from Zip archives
------------------------------------------------------

New in version 2.3.

  This module adds the ability to import Python modules (`*.py',
`*.py[co]') and packages from ZIP-format archives. It is usually not
needed to use the zipimport module explicitly; it is
automatically used by the built-in import mechanism for
`sys.path' items that are paths to ZIP archives.

  Typically, `sys.path' is a list of directory names as strings.  This
module also allows an item of `sys.path' to be a string naming a ZIP
file archive.  The ZIP archive can contain a subdirectory structure to
support package imports, and a path within the archive can be specified
to only import from a subdirectory.  For example, the path
`/tmp/example.zip/lib/' would only import from the `lib/' subdirectory
within the archive.

  Any files may be present in the ZIP archive, but only files `.py' and
`.py[co]' are available for import.  ZIP import of dynamic modules
(`.pyd', `.so') is disallowed. Note that if an archive only contains
`.py' files, Python will not attempt to modify the archive by adding the
corresponding `.pyc' or `.pyo' file, meaning that if a ZIP archive
doesn't contain `.pyc' files, importing may be rather slow.

  Using the built-in reload() function will fail if called
on a module loaded from a ZIP archive; it is unlikely that
reload() would be needed, since this would imply that the ZIP has
been altered during runtime.

  ZIP archives with an archive comment are currently not supported.

PKZIP Application Note (http://www.pkware.com/documents/casestudies/APPNOTE.TXT)
     Documentation on the ZIP file format by Phil Katz, the creator of
     the format and algorithms used.

PEP 273 (http://www.python.org/dev/peps/pep-0273) - Import Modules from Zip Archives
     Written by James C. Ahlstrom, who also provided an implementation.
     Python 2.3 follows the specification in PEP 273, but uses an
     implementation written by Just van Rossum that uses the import
     hooks described in PEP 302.

PEP 302 (http://www.python.org/dev/peps/pep-0302) - New Import Hooks
     The PEP to add the import hooks that help this module work.

  This module defines an exception:

 -- Exception: zipimport.ZipImportError
     Exception raised by zipimporter objects. It's a subclass of
     ImportError, so it can be caught as ImportError,
     too.

5.30.4.1 zipimporter Objects
............................

zipimporter is the class for importing ZIP files.

 -- Class: zipimport.zipimporter (archivepath)
     Create a new zipimporter instance. _archivepath_ must be a path to
     a ZIP file, or to a specific path within a ZIP file.  For example,
     an _archivepath_ of `foo/bar.zip/lib' will look for modules in the
     `lib' directory inside the ZIP file `foo/bar.zip' (provided that
     it exists).

     ZipImportError is raised if _archivepath_ doesn't
     point to a valid ZIP archive.

      -- Method: find_module (fullname[, path])
          Search for a module specified by _fullname_. _fullname_ must
          be the fully qualified (dotted) module name. It returns the
          zipimporter instance itself if the module was found, or
          None if it wasn't. The optional _path_ argument is
          ignored--it's there for compatibility with the importer
          protocol.

      -- Method: get_code (fullname)
          Return the code object for the specified module. Raise
          ZipImportError if the module couldn't be found.

      -- Method: get_data (pathname)
          Return the data associated with _pathname_. Raise
          IOError if the file wasn't found.

      -- Method: get_filename (fullname)
          Return the value `__file__' would be set to if the specified
          module was imported. Raise ZipImportError if the
          module couldn't be found.

     New in version 2.7.

      -- Method: get_source (fullname)
          Return the source code for the specified module. Raise
          ZipImportError if the module couldn't be found, return
          None if the archive does contain the module, but
          has no source for it.

      -- Method: is_package (fullname)
          Return True if the module specified by _fullname_ is a
          package. Raise ZipImportError if the module
          couldn't be found.

      -- Method: load_module (fullname)
          Load the module specified by _fullname_. _fullname_ must be
          the fully qualified (dotted) module name. It returns the
          imported module, or raises ZipImportError if it
          wasn't found.

      -- Attribute: archive
          The file name of the importer's associated ZIP file, without
          a possible subpath.

      -- Attribute: prefix
          The subpath within the ZIP file where modules are searched.
          This is the empty string for zipimporter objects which point
          to the root of the ZIP file.

     The archive and prefix attributes, when
     combined with a slash, equal the original _archivepath_ argument
     given to the zipimporter constructor.

5.30.4.2 Examples
.................

Here is an example that imports a module from a ZIP archive - note that
the zipimport module is not explicitly used.

    $ unzip -l /tmp/example.zip
    Archive:  /tmp/example.zip
      Length     Date   Time    Name
     --------    ----   ----    ----
         8467  11-26-02 22:30   jwzthreading.py
     --------                   -------
         8467                   1 file
    $ ./python
    Python 2.3 (#1, Aug 1 2003, 19:54:32)
    >>> import sys
    >>> sys.path.insert(0, '/tmp/example.zip')  # Add .zip file to front of path
    >>> import jwzthreading
    >>> jwzthreading.__file__
    '/tmp/example.zip/jwzthreading.py'

5.30.5 `pkgutil' -- Package extension utility
---------------------------------------------

This module provides utilities for the import system, in particular
package support.

  New in version 2.3.

 -- Function: pkgutil.extend_path (path, name)
     Extend the search path for the modules which comprise a package.
     Intended use is to place the following code in a package's
     `__init__.py':

         from pkgutil import extend_path
         __path__ = extend_path(__path__, __name__)

     This will add to the package's `__path__' all subdirectories of
     directories on `sys.path' named after the package.  This is useful
     if one wants to distribute different parts of a single logical
     package as multiple directories.

     It also looks for `*.pkg' files beginning where `*' matches the
     _name_ argument.  This feature is similar to `*.pth' files (see the
     site module for more information), except that it
     doesn't special-case lines starting with `import'.  A `*.pkg' file
     is trusted at face value: apart from checking for duplicates, all
     entries found in a `*.pkg' file are added to the path, regardless
     of whether they exist on the filesystem.  (This is a feature.)

     If the input path is not a list (as is the case for frozen
     packages) it is returned unchanged.  The input path is not
     modified; an extended copy is returned.  Items are only appended
     to the copy at the end.

     It is assumed that sys.path is a sequence.  Items of
     sys.path.  that are not (Unicode or 8-bit) strings
     referring to existing directories are ignored.  Unicode items on
     sys.path that cause errors when used as filenames may
     cause this function to raise an exception (in line with
     os.path.isdir() behavior).

 -- Class: pkgutil.ImpImporter (dirname=None)
     PEP 302 (http://www.python.org/dev/peps/pep-0302) Importer that
     wraps Python's "classic" import algorithm.

     If _dirname_ is a string, a PEP 302
     (http://www.python.org/dev/peps/pep-0302) importer is created that
     searches that directory.  If _dirname_ is `None', a PEP 302
     (http://www.python.org/dev/peps/pep-0302) importer is created that
     searches the current sys.path, plus any modules that
     are frozen or built-in.

     Note that ImpImporter does not currently support
     being used by placement on sys.meta_path.

 -- Class: pkgutil.ImpLoader (fullname, file, filename, etc)
     PEP 302 (http://www.python.org/dev/peps/pep-0302) Loader that
     wraps Python's "classic" import algorithm.

 -- Function: pkgutil.find_loader (fullname)
     Find a PEP 302 (http://www.python.org/dev/peps/pep-0302) "loader"
     object for _fullname_.

     If _fullname_ contains dots, path must be the containing package's
     `__path__'.  Returns `None' if the module cannot be found or
     imported.  This function uses iter_importers(), and is
     thus subject to the same limitations regarding platform-specific
     special import locations such as the Windows registry.

 -- Function: pkgutil.get_importer (path_item)
     Retrieve a PEP 302 (http://www.python.org/dev/peps/pep-0302)
     importer for the given _path_item_.

     The returned importer is cached in sys.path_importer_cache
     if it was newly created by a path hook.

     If there is no importer, a wrapper around the basic import
     machinery is returned.  This wrapper is never inserted into the
     importer cache (None is inserted instead).

     The cache (or part of it) can be cleared manually if a rescan of
     sys.path_hooks is necessary.

 -- Function: pkgutil.get_loader (module_or_name)
     Get a PEP 302 (http://www.python.org/dev/peps/pep-0302) "loader"
     object for _module_or_name_.

     If the module or package is accessible via the normal import
     mechanism, a wrapper around the relevant part of that machinery is
     returned.  Returns `None' if the module cannot be found or
     imported.  If the named module is not already imported, its
     containing package (if any) is imported, in order to establish the
     package `__path__'.

     This function uses iter_importers(), and is thus
     subject to the same limitations regarding platform-specific
     special import locations such as the Windows registry.

 -- Function: pkgutil.iter_importers (fullname=")
     Yield PEP 302 (http://www.python.org/dev/peps/pep-0302) importers
     for the given module name.

     If fullname contains a '.', the importers will be for the package
     containing fullname, otherwise they will be importers for
     sys.meta_path, sys.path, and Python's "classic"
     import machinery, in that order.  If the named module is in a
     package, that package is imported as a side effect of invoking
     this function.

     Non- PEP 302 (http://www.python.org/dev/peps/pep-0302) mechanisms
     (e.g. the Windows registry) used by the standard import machinery
     to find files in alternative locations are partially supported,
     but are searched _after_ See sys.path.  Normally, these
     locations are searched _before_ See sys.path, preventing
     sys.path.  entries from shadowing them.

     For this to cause a visible difference in behaviour, there must be
     a module or package name that is accessible via both
     sys.path and one of the non- PEP 302
     (http://www.python.org/dev/peps/pep-0302) file system mechanisms.
     In this case, the emulation will find the former version, while
     the builtin import mechanism will find the latter.

     Items of the following types can be affected by this discrepancy:
     `imp.C_EXTENSION', `imp.PY_SOURCE', `imp.PY_COMPILED',
     `imp.PKG_DIRECTORY'.

 -- Function: pkgutil.iter_modules (path=None, prefix=")
     Yields `(module_loader, name, ispkg)' for all submodules on
     _path_, or, if path is `None', all top-level modules on `sys.path'.

     _path_ should be either `None' or a list of paths to look for
     modules in.

     _prefix_ is a string to output on the front of every module name
     on output.

 -- Function: pkgutil.walk_packages (path=None, prefix=", onerror=None)
     Yields `(module_loader, name, ispkg)' for all modules recursively
     on _path_, or, if path is `None', all accessible modules.

     _path_ should be either `None' or a list of paths to look for
     modules in.

     _prefix_ is a string to output on the front of every module name
     on output.

     Note that this function must import all _packages_ (_not_ all
     modules!) on the given _path_, in order to access the `__path__'
     attribute to find submodules.

     _onerror_ is a function which gets called with one argument (the
     name of the package which was being imported) if any exception
     occurs while trying to import a package.  If no _onerror_ function
     is supplied, see ImportError.s are caught and ignored,
     while all other exceptions are propagated, terminating the search.

     Examples:

         # list all modules python can access
         walk_packages()

         # list all submodules of ctypes
         walk_packages(ctypes.__path__, ctypes.__name__ + '.')

 -- Function: pkgutil.get_data (package, resource)
     Get a resource from a package.

     This is a wrapper for the PEP 302
     (http://www.python.org/dev/peps/pep-0302) loader get_data().
     API.  The _package_ argument should be the name of a
     package, in standard module format (`foo.bar').  The _resource_
     argument should be in the form of a relative filename, using `/'
     as the path separator.  The parent directory name `..' is not
     allowed, and nor is a rooted name (starting with a `/').

     The function returns a binary string that is the contents of the
     specified resource.

     For packages located in the filesystem, which have already been
     imported, this is the rough equivalent of:

         d = os.path.dirname(sys.modules[package].__file__)
         data = open(os.path.join(d, resource), 'rb').read()

     If the package cannot be located or loaded, or it uses a PEP 302
     (http://www.python.org/dev/peps/pep-0302) loader which does not
     support get_data(), then `None' is returned.

5.30.6 `modulefinder' -- Find modules used by a script
------------------------------------------------------

New in version 2.3.

  This module provides a ModuleFinder class that can be
used to determine the set of modules imported by a script.
`modulefinder.py' can also be run as a script, giving the filename of a
Python script as its argument, after which a report of the imported
modules will be printed.

 -- Function: modulefinder.AddPackagePath (pkg_name, path)
     Record that the package named _pkg_name_ can be found in the
     specified _path_.

 -- Function: modulefinder.ReplacePackage (oldname, newname)
     Allows specifying that the module named _oldname_ is in fact the
     package named _newname_.  The most common usage would be  to
     handle how the `_xmlplus' package replaces the `xml' package.

 -- Class: modulefinder.ModuleFinder ([path=None, debug=0, excludes=[],
          replace_paths=[]])
     This class provides run_script() and report()
     methods to determine the set of modules imported by a
     script. _path_ can be a list of directories to search for modules;
     if not specified, `sys.path' is used.  _debug_ sets the debugging
     level; higher values make the class print  debugging messages about
     what it's doing. _excludes_ is a list of module names to exclude
     from the analysis. _replace_paths_ is a list of `(oldpath,
     newpath)' tuples that will be replaced in module paths.

      -- Method: report ()
          Print a report to standard output that lists the modules
          imported by the script and their paths, as well as modules
          that are missing or seem to be missing.

      -- Method: run_script (pathname)
          Analyze the contents of the _pathname_ file, which must
          contain Python code.

      -- Attribute: modules
          A dictionary mapping module names to modules. See
          Example usage of ModuleFinder.

5.30.6.1 Example usage of `ModuleFinder'
........................................

The script that is going to get analyzed later on (bacon.py):

    import re, itertools

    try:
        import baconhameggs
    except ImportError:
        pass

    try:
        import guido.python.ham
    except ImportError:
        pass

The script that will output the report of bacon.py:

    from modulefinder import ModuleFinder

    finder = ModuleFinder()
    finder.run_script('bacon.py')

    print 'Loaded modules:'
    for name, mod in finder.modules.iteritems():
        print '%s: ' % name,
        print ','.join(mod.globalnames.keys()[:3])

    print '-'*50
    print 'Modules not imported:'
    print '\n'.join(finder.badmodules.iterkeys())

Sample output (may vary depending on the architecture):

    Loaded modules:
    _types:
    copy_reg:  _inverted_registry,_slotnames,__all__
    sre_compile:  isstring,_sre,_optimize_unicode
    _sre:
    sre_constants:  REPEAT_ONE,makedict,AT_END_LINE
    sys:
    re:  __module__,finditer,_expand
    itertools:
    __main__:  re,itertools,baconhameggs
    sre_parse:  __getslice__,_PATTERNENDERS,SRE_FLAG_UNICODE
    array:
    types:  __module__,IntType,TypeType
    ---------------------------------------------------
    Modules not imported:
    guido.python.ham
    baconhameggs

5.30.7 `runpy' -- Locating and executing Python modules
-------------------------------------------------------

New in version 2.5.

  The runpy module is used to locate and run Python modules
without importing them first. Its main use is to implement the
-m command line switch that allows scripts to be located using
the Python module namespace rather than the filesystem.

  The runpy module provides two functions:

 -- Function: runpy.run_module (mod_name, init_globals=None,
          run_name=None, alter_sys=False)
     Execute the code of the specified module and return the resulting
     module globals dictionary. The module's code is first located
     using the standard import mechanism (refer to PEP 302
     (http://www.python.org/dev/peps/pep-0302) for details) and then
     executed in a fresh module namespace.

     If the supplied module name refers to a package rather than a
     normal module, then that package is imported and the `__main__'
     submodule within that package is then executed and the resulting
     module globals dictionary returned.

     The optional dictionary argument _init_globals_ may be used to
     pre-populate the module's globals dictionary before the code is
     executed. The supplied dictionary will not be modified. If any of
     the special global variables below are defined in the supplied
     dictionary, those definitions are overridden by
     run_module().

     The special global variables `__name__', `__file__', `__loader__'
     and `__package__' are set in the globals dictionary before the
     module code is executed (Note that this is a minimal set of
     variables - other variables may be set implicitly as an
     interpreter implementation detail).

     `__name__' is set to _run_name_ if this optional argument is not
     None, to `mod_name + '.__main__'' if the named module
     is a package and to the _mod_name_ argument otherwise.

     `__file__' is set to the name provided by the module loader. If the
     loader does not make filename information available, this variable
     is set to None.

     `__loader__' is set to the PEP 302
     (http://www.python.org/dev/peps/pep-0302) module loader used to
     retrieve the code for the module (This loader may be a wrapper
     around the standard import mechanism).

     `__package__' is set to _mod_name_ if the named module is a
     package and to `mod_name.rpartition('.')[0]' otherwise.

     If the argument _alter_sys_ is supplied and evaluates to
     True, then `sys.argv[0]' is updated with the value of
     `__file__' and `sys.modules[__name__]' is updated with a temporary
     module object for the module being executed. Both `sys.argv[0]'
     and `sys.modules[__name__]' are restored to their original values
     before the function returns.

     Note that this manipulation of sys is not thread-safe.
     Other threads may see the partially initialised module, as well as
     the altered list of arguments. It is recommended that the
     sys module be left alone when invoking this function from
     threaded code.

     Changed in version 2.7: Added ability to execute packages by
     looking for a `__main__' submodule

 -- Function: runpy.run_path (file_path, init_globals=None,
          run_name=None)
     Execute the code at the named filesystem location and return the
     resulting module globals dictionary. As with a script name
     supplied to the CPython command line, the supplied path may refer
     to a Python source file, a compiled bytecode file or a valid
     sys.path entry containing a `__main__' module (e.g. a zipfile
     containing a top-level `__main__.py' file).

     For a simple script, the specified code is simply executed in a
     fresh module namespace. For a valid sys.path entry (typically a
     zipfile or directory), the entry is first added to the beginning
     of `sys.path'. The function then looks for and executes a
     __main__ module using the updated path. Note that there is no
     special protection against invoking an existing __main__.
     entry located elsewhere on `sys.path' if there is no such module
     at the specified location.

     The optional dictionary argument _init_globals_ may be used to
     pre-populate the module's globals dictionary before the code is
     executed. The supplied dictionary will not be modified. If any of
     the special global variables below are defined in the supplied
     dictionary, those definitions are overridden by run_path().

     The special global variables `__name__', `__file__', `__loader__'
     and `__package__' are set in the globals dictionary before the
     module code is executed (Note that this is a minimal set of
     variables - other variables may be set implicitly as an
     interpreter implementation detail).

     `__name__' is set to _run_name_ if this optional argument is not
     None and to `'<run_path>'' otherwise.

     `__file__' is set to the name provided by the module loader. If the
     loader does not make filename information available, this variable
     is set to None. For a simple script, this will be set
     to `file_path'.

     `__loader__' is set to the PEP 302
     (http://www.python.org/dev/peps/pep-0302) module loader used to
     retrieve the code for the module (This loader may be a wrapper
     around the standard import mechanism). For a simple script, this
     will be set to None.

     `__package__' is set to `__name__.rpartition('.')[0]'.

     A number of alterations are also made to the sys.
     module. Firstly, `sys.path' may be altered as described above.
     `sys.argv[0]' is updated with the value of `file_path' and
     `sys.modules[__name__]' is updated with a temporary module object
     for the module being executed. All modifications to items in
     sys are reverted before the function returns.

     Note that, unlike run_module(), the alterations made to
     sys.  are not optional in this function as these
     adjustments are essential to allowing the execution of sys.path
     entries. As the thread-safety limitations still apply, use of this
     function in threaded code should be either serialised with the
     import lock or delegated to a separate process.

     New in version 2.7.

PEP 338 (http://www.python.org/dev/peps/pep-0338) - Executing modules as scripts
     PEP written and implemented by Nick Coghlan.

PEP 366 (http://www.python.org/dev/peps/pep-0366) - Main module explicit relative imports
     PEP written and implemented by Nick Coghlan.

  See Command line and environment. - CPython command line
details

5.31 Python Language Services
=============================

Python provides a number of modules to assist in working with the Python
language.  These modules support tokenizing, parsing, syntax analysis,
bytecode disassembly, and various other facilities.

  These modules include:

5.31.1 `parser' -- Access Python parse trees
--------------------------------------------

The parser module provides an interface to Python's
internal parser and byte-code compiler.  The primary purpose for this
interface is to allow Python code to edit the parse tree of a Python
expression and create executable code from this.  This is better than
trying to parse and modify an arbitrary Python code fragment as a
string because parsing is performed in a manner identical to the code
forming the application.  It is also faster.

     Note: From Python 2.5 onward, it's much more convenient to cut in
     at the Abstract Syntax Tree (AST) generation and compilation
     stage, using the ast.  module.

     The parser module exports the names documented here
     also with "st" replaced by "ast"; this is a legacy from the time
     when there was no other AST and has nothing to do with the AST
     found in Python 2.5.  This is also the reason for the functions'
     keyword arguments being called _ast_, not _st_.  The "ast"
     functions will be removed in Python 3.0.

  There are a few things to note about this module which are important
to making use of the data structures created.  This is not a tutorial
on editing the parse trees for Python code, but some examples of using
the parser module are presented.

  Most importantly, a good understanding of the Python grammar
processed by the internal parser is required.  For full information on
the language syntax, refer to The Python Language Reference.
The parser itself is created from a grammar specification defined in
the file `Grammar/Grammar' in the standard Python distribution.  The
parse trees stored in the ST objects created by this module are the
actual output from the internal parser when created by the
expr() or suite() functions, described below.  The
ST objects created by sequence2st() faithfully simulate
those structures.  Be aware that the values of the sequences which are
considered "correct" will vary from one version of Python to another as
the formal grammar for the language is revised.  However, transporting
code from one Python version to another as source text will always
allow correct parse trees to be created in the target version, with the
only restriction being that migrating to an older version of the
interpreter will not support more recent language constructs.  The
parse trees are not typically compatible from one version to another,
whereas source code has always been forward-compatible.

  Each element of the sequences returned by st2list() or
st2tuple().  has a simple form.  Sequences representing
non-terminal elements in the grammar always have a length greater than
one.  The first element is an integer which identifies a production in
the grammar.  These integers are given symbolic names in the C header
file `Include/graminit.h' and the Python module symbol.
Each additional element of the sequence represents a component of the
production as recognized in the input string: these are always sequences
which have the same form as the parent.  An important aspect of this
structure which should be noted is that keywords used to identify the
parent node type, such as the keyword if in an `if_stmt',
are included in the node tree without any special treatment.  For
example, the if keyword is represented by the tuple `(1,
'if')', where `1' is the numeric value associated with all `NAME'
tokens, including variable and function names defined by the user.  In
an alternate form returned when line number information is requested,
the same token might be represented as `(1, 'if', 12)', where the `12'
represents the line number at which the terminal symbol was found.

  Terminal elements are represented in much the same way, but without
any child elements and the addition of the source text which was
identified.  The example of the if keyword above is
representative.  The various types of terminal symbols are defined in
the C header file `Include/token.h' and the Python module token.

  The ST objects are not required to support the functionality of this
module, but are provided for three purposes: to allow an application to
amortize the cost of processing complex parse trees, to provide a parse
tree representation which conserves memory space when compared to the
Python list or tuple representation, and to ease the creation of
additional modules in C which manipulate parse trees.  A simple
"wrapper" class may be created in Python to hide the use of ST objects.

  The parser module defines functions for a few distinct
purposes.  The most important purposes are to create ST objects and to
convert ST objects to other representations such as parse trees and
compiled code objects, but there are also functions which serve to
query the type of parse tree represented by an ST object.

Module symbol.
     Useful constants representing internal nodes of the parse tree.

Module token.
     Useful constants representing leaf nodes of the parse tree and
     functions for testing node values.

5.31.1.1 Creating ST Objects
............................

ST objects may be created from source code or from a parse tree. When
creating an ST object from source, different functions are used to
create the `'eval'' and `'exec'' forms.

 -- Function: parser.expr (source)
     The expr() function parses the parameter _source_ as
     if it were an input to `compile(source, 'file.py', 'eval')'.  If
     the parse succeeds, an ST object is created to hold the internal
     parse tree representation, otherwise an appropriate exception is
     raised.

 -- Function: parser.suite (source)
     The suite() function parses the parameter _source_ as
     if it were an input to `compile(source, 'file.py', 'exec')'.  If
     the parse succeeds, an ST object is created to hold the internal
     parse tree representation, otherwise an appropriate exception is
     raised.

 -- Function: parser.sequence2st (sequence)
     This function accepts a parse tree represented as a sequence and
     builds an internal representation if possible.  If it can validate
     that the tree conforms to the Python grammar and all nodes are
     valid node types in the host version of Python, an ST object is
     created from the internal representation and returned to the
     called.  If there is a problem creating the internal
     representation, or if the tree cannot be validated, a
     ParserError exception is raised.  An ST object created this
     way should not be assumed to compile correctly; normal exceptions
     raised by compilation may still be initiated when the ST object is
     passed to compilest().  This may indicate problems not
     related to syntax (such as a MemoryError exception),
     but may also be due to constructs such as the result of parsing
     `del f(0)', which escapes the Python parser but is checked by the
     bytecode compiler.

     Sequences representing terminal tokens may be represented as
     either two-element lists of the form `(1, 'name')' or as
     three-element lists of the form `(1, 'name', 56)'.  If the third
     element is present, it is assumed to be a valid line number.  The
     line number may be specified for any subset of the terminal
     symbols in the input tree.

 -- Function: parser.tuple2st (sequence)
     This is the same function as sequence2st().  This
     entry point is maintained for backward compatibility.

5.31.1.2 Converting ST Objects
..............................

ST objects, regardless of the input used to create them, may be
converted to parse trees represented as list- or tuple- trees, or may
be compiled into executable code objects.  Parse trees may be extracted
with or without line numbering information.

 -- Function: parser.st2list (ast[, line_info])
     This function accepts an ST object from the caller in _ast_ and
     returns a Python list representing the equivalent parse tree.  The
     resulting list representation can be used for inspection or the
     creation of a new parse tree in list form.  This function does not
     fail so long as memory is available to build the list
     representation.  If the parse tree will only be used for
     inspection, see st2tuple() should be used instead to
     reduce memory consumption and fragmentation.  When the list
     representation is required, this function is significantly faster
     than retrieving a tuple representation and converting that to
     nested lists.

     If _line_info_ is true, line number information will be included
     for all terminal tokens as a third element of the list
     representing the token.  Note that the line number provided
     specifies the line on which the token _ends_.  This information is
     omitted if the flag is false or omitted.

 -- Function: parser.st2tuple (ast[, line_info])
     This function accepts an ST object from the caller in _ast_ and
     returns a Python tuple representing the equivalent parse tree.
     Other than returning a tuple instead of a list, this function is
     identical to st2list().

     If _line_info_ is true, line number information will be included
     for all terminal tokens as a third element of the list
     representing the token.  This information is omitted if the flag
     is false or omitted.

 -- Function: parser.compilest (ast[, filename='<syntax-tree>'])
     The Python byte compiler can be invoked on an ST object to produce
     code objects which can be used as part of an exec.
     statement or a call to the built-in eval() function.
     This function provides the interface to the compiler, passing the
     internal parse tree from _ast_ to the parser, using the source
     file name specified by the _filename_ parameter. The default value
     supplied for _filename_ indicates that the source was an ST object.

     Compiling an ST object may result in exceptions related to
     compilation; an example would be a SyntaxError caused
     by the parse tree for `del f(0)': this statement is considered
     legal within the formal grammar for Python but is not a legal
     language construct.  The SyntaxError raised for this
     condition is actually generated by the Python byte-compiler
     normally, which is why it can be raised at this point by the
     parser module.  Most causes of compilation failure can be
     diagnosed programmatically by inspection of the parse tree.

5.31.1.3 Queries on ST Objects
..............................

Two functions are provided which allow an application to determine if
an ST was created as an expression or a suite.  Neither of these
functions can be used to determine if an ST was created from source
code via expr() or suite() or from a parse
tree via sequence2st().

 -- Function: parser.isexpr (ast)
     When _ast_ represents an `'eval'' form, this function returns
     true, otherwise it returns false.  This is useful, since code
     objects normally cannot be queried for this information using
     existing built-in functions.  Note that the code objects created
     by compilest() cannot be queried like this either, and
     are identical to those created by the built-in compile()
     function.

 -- Function: parser.issuite (ast)
     This function mirrors isexpr() in that it reports
     whether an ST object represents an `'exec'' form, commonly known
     as a "suite."  It is not safe to assume that this function is
     equivalent to `not isexpr(ast)', as additional syntactic fragments
     may be supported in the future.

5.31.1.4 Exceptions and Error Handling
......................................

The parser module defines a single exception, but may also pass other
built-in exceptions from other portions of the Python runtime
environment.  See each function for information about the exceptions it
can raise.

 -- Exception: parser.ParserError
     Exception raised when a failure occurs within the parser module.
     This is generally produced for validation failures rather than the
     built-in SyntaxError raised during normal parsing. The
     exception argument is either a string describing the reason of the
     failure or a tuple containing a sequence causing the failure from
     a parse tree passed to sequence2st().  and an
     explanatory string.  Calls to sequence2st() need to
     be able to handle either type of exception, while calls to other
     functions in the module will only need to be aware of the simple
     string values.

  Note that the functions compilest(), expr(),
and suite() may raise exceptions which are normally thrown
by the parsing and compilation process.  These include the built in
exceptions MemoryError, OverflowError,
SyntaxError, and SystemError.  In these cases, these
exceptions carry all the meaning normally associated with them.  Refer
to the descriptions of each function for detailed information.

5.31.1.5 ST Objects
...................

Ordered and equality comparisons are supported between ST objects.
Pickling of ST objects (using the pickle module) is also
supported.

 -- Data: parser.STType
     The type of the objects returned by expr(),
     suite() and sequence2st().

  ST objects have the following methods:

 -- Method: ST.compile ([filename])
     Same as `compilest(st, filename)'.

 -- Method: ST.isexpr ()
     Same as `isexpr(st)'.

 -- Method: ST.issuite ()
     Same as `issuite(st)'.

 -- Method: ST.tolist ([line_info])
     Same as `st2list(st, line_info)'.

 -- Method: ST.totuple ([line_info])
     Same as `st2tuple(st, line_info)'.

5.31.1.6 Example: Emulation of `compile()'
..........................................

While many useful operations may take place between parsing and bytecode
generation, the simplest operation is to do nothing.  For this purpose,
using the parser module to produce an intermediate data
structure is equivalent to the code

    >>> code = compile('a + 5', 'file.py', 'eval')
    >>> a = 5
    >>> eval(code)
    10

The equivalent operation using the parser module is
somewhat longer, and allows the intermediate internal parse tree to be
retained as an ST object:

    >>> import parser
    >>> st = parser.expr('a + 5')
    >>> code = st.compile('file.py')
    >>> a = 5
    >>> eval(code)
    10

An application which needs both ST and code objects can package this
code into readily available functions:

    import parser

    def load_suite(source_string):
        st = parser.suite(source_string)
        return st, st.compile()

    def load_expression(source_string):
        st = parser.expr(source_string)
        return st, st.compile()

5.31.2 Abstract Syntax Trees
----------------------------

New in version 2.5: The low-level `_ast' module containing only the
node classes.

  New in version 2.6: The high-level `ast' module containing all
helpers.

  The ast module helps Python applications to process trees
of the Python abstract syntax grammar.  The abstract syntax itself
might change with each Python release; this module helps to find out
programmatically what the current grammar looks like.

  An abstract syntax tree can be generated by passing
`ast.PyCF_ONLY_AST' as a flag to the compile() built-in
function, or using the parse().  helper provided in this
module.  The result will be a tree of objects whose classes all inherit
from ast.AST.  An abstract syntax tree can be compiled into
a Python code object using the built-in compile() function.

  Latest version of the ast module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/ast.py?view=markup)

5.31.2.1 Node classes
.....................

 -- Class: ast.AST
     This is the base of all AST node classes.  The actual node classes
     are derived from the `Parser/Python.asdl' file, which is reproduced
     below.  They are defined in the `_ast' C module and
     re-exported in ast.

     There is one class defined for each left-hand side symbol in the
     abstract grammar (for example, `ast.stmt' or `ast.expr').  In
     addition, there is one class defined for each constructor on the
     right-hand side; these classes inherit from the classes for the
     left-hand side trees.  For example, `ast.BinOp' inherits from
     `ast.expr'.  For production rules with alternatives (aka "sums"),
     the left-hand side class is abstract: only instances of specific
     constructor nodes are ever created.

      -- Attribute: _fields
          Each concrete class has an attribute _fields.
          which gives the names of all child nodes.

          Each instance of a concrete class has one attribute for each
          child node, of the type as defined in the grammar.  For
          example, `ast.BinOp' instances have an attribute `left' of
          type `ast.expr'.

          If these attributes are marked as optional in the grammar
          (using a question mark), the value might be `None'.  If the
          attributes can have zero-or-more values (marked with an
          asterisk), the values are represented as Python lists.  All
          possible attributes must be present and have valid values
          when compiling an AST with compile().

      -- Attribute: lineno
      -- Attribute: col_offset
          Instances of `ast.expr' and `ast.stmt' subclasses have
          lineno and col_offset attributes.  The
          lineno is the line number of source text
          (1-indexed so the first line is line 1) and the
          col_offset is the UTF-8 byte offset of the first token
          that generated the node.  The UTF-8 offset is recorded
          because the parser uses UTF-8 internally.

     The constructor of a class `ast.T' parses its arguments as follows:

        * If there are positional arguments, there must be as many as
          there are items in `T._fields'; they will be assigned as
          attributes of these names.

        * If there are keyword arguments, they will set the attributes
          of the same names to the given values.

     For example, to create and populate an `ast.UnaryOp' node, you
     could use

         node = ast.UnaryOp()
         node.op = ast.USub()
         node.operand = ast.Num()
         node.operand.n = 5
         node.operand.lineno = 0
         node.operand.col_offset = 0
         node.lineno = 0
         node.col_offset = 0

     or the more compact

         node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
                            lineno=0, col_offset=0)

     New in version 2.6: The constructor as explained above was added.
     In Python 2.5 nodes had to be created by calling the class
     constructor without arguments and setting the attributes
     afterwards.

5.31.2.2 Abstract Grammar
.........................

The module defines a string constant `__version__' which is the decimal
Subversion revision number of the file shown below.

  The abstract grammar is currently defined as follows:

    -- ASDL's five builtin types are identifier, int, string, object, bool

    module Python version "$Revision: 82160 $"
    {
    	mod = Module(stmt* body)
    	    | Interactive(stmt* body)
    	    | Expression(expr body)

    	    -- not really an actual node but useful in Jython's typesystem.
    	    | Suite(stmt* body)

    	stmt = FunctionDef(identifier name, arguments args,
                                stmt* body, expr* decorator_list)
    	      | ClassDef(identifier name, expr* bases, stmt* body, expr* decorator_list)
    	      | Return(expr? value)

    	      | Delete(expr* targets)
    	      | Assign(expr* targets, expr value)
    	      | AugAssign(expr target, operator op, expr value)

    	      -- not sure if bool is allowed, can always use int
     	      | Print(expr? dest, expr* values, bool nl)

    	      -- use 'orelse' because else is a keyword in target languages
    	      | For(expr target, expr iter, stmt* body, stmt* orelse)
    	      | While(expr test, stmt* body, stmt* orelse)
    	      | If(expr test, stmt* body, stmt* orelse)
    	      | With(expr context_expr, expr? optional_vars, stmt* body)

    	      -- 'type' is a bad name
    	      | Raise(expr? type, expr? inst, expr? tback)
    	      | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse)
    	      | TryFinally(stmt* body, stmt* finalbody)
    	      | Assert(expr test, expr? msg)

    	      | Import(alias* names)
    	      | ImportFrom(identifier? module, alias* names, int? level)

    	      -- Doesn't capture requirement that locals must be
    	      -- defined if globals is
    	      -- still supports use as a function!
    	      | Exec(expr body, expr? globals, expr? locals)

    	      | Global(identifier* names)
    	      | Expr(expr value)
    	      | Pass | Break | Continue

    	      -- XXX Jython will be different
    	      -- col_offset is the byte offset in the utf8 string the parser uses
    	      attributes (int lineno, int col_offset)

    	      -- BoolOp() can use left & right?
    	expr = BoolOp(boolop op, expr* values)
    	     | BinOp(expr left, operator op, expr right)
    	     | UnaryOp(unaryop op, expr operand)
    	     | Lambda(arguments args, expr body)
    	     | IfExp(expr test, expr body, expr orelse)
    	     | Dict(expr* keys, expr* values)
    	     | Set(expr* elts)
    	     | ListComp(expr elt, comprehension* generators)
    	     | SetComp(expr elt, comprehension* generators)
    	     | DictComp(expr key, expr value, comprehension* generators)
    	     | GeneratorExp(expr elt, comprehension* generators)
    	     -- the grammar constrains where yield expressions can occur
    	     | Yield(expr? value)
    	     -- need sequences for compare to distinguish between
    	     -- x < 4 < 3 and (x < 4) < 3
    	     | Compare(expr left, cmpop* ops, expr* comparators)
    	     | Call(expr func, expr* args, keyword* keywords,
    			 expr? starargs, expr? kwargs)
    	     | Repr(expr value)
    	     | Num(object n) -- a number as a PyObject.
    	     | Str(string s) -- need to specify raw, unicode, etc?
    	     -- other literals? bools?

    	     -- the following expression can appear in assignment context
    	     | Attribute(expr value, identifier attr, expr_context ctx)
    	     | Subscript(expr value, slice slice, expr_context ctx)
    	     | Name(identifier id, expr_context ctx)
    	     | List(expr* elts, expr_context ctx)
    	     | Tuple(expr* elts, expr_context ctx)

    	      -- col_offset is the byte offset in the utf8 string the parser uses
    	      attributes (int lineno, int col_offset)

    	expr_context = Load | Store | Del | AugLoad | AugStore | Param

    	slice = Ellipsis | Slice(expr? lower, expr? upper, expr? step)
    	      | ExtSlice(slice* dims)
    	      | Index(expr value)

    	boolop = And | Or

    	operator = Add | Sub | Mult | Div | Mod | Pow | LShift
                     | RShift | BitOr | BitXor | BitAnd | FloorDiv

    	unaryop = Invert | Not | UAdd | USub

    	cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn

    	comprehension = (expr target, expr iter, expr* ifs)

    	-- not sure what to call the first argument for raise and except
    	excepthandler = ExceptHandler(expr? type, expr? name, stmt* body)
    	                attributes (int lineno, int col_offset)

    	arguments = (expr* args, identifier? vararg,
    		     identifier? kwarg, expr* defaults)

            -- keyword arguments supplied to call
            keyword = (identifier arg, expr value)

            -- import name with optional 'as' alias.
            alias = (identifier name, identifier? asname)
    }

5.31.2.3 `ast' Helpers
......................

New in version 2.6.

  Apart from the node classes, see ast module defines these
utility functions and classes for traversing abstract syntax trees:

 -- Function: ast.parse (expr, filename='<unknown>', mode='exec')
     Parse an expression into an AST node.  Equivalent to `compile(expr,
     filename, mode, ast.PyCF_ONLY_AST)'.

 -- Function: ast.literal_eval (node_or_string)
     Safely evaluate an expression node or a string containing a Python
     expression.  The string or node provided may only consist of the
     following Python literal structures: strings, numbers, tuples,
     lists, dicts, booleans, and `None'.

     This can be used for safely evaluating strings containing Python
     expressions from untrusted sources without the need to parse the
     values oneself.

 -- Function: ast.get_docstring (node, clean=True)
     Return the docstring of the given _node_ (which must be a
     `FunctionDef', `ClassDef' or `Module' node), or `None' if it has
     no docstring.  If _clean_ is true, clean up the docstring's
     indentation with inspect.cleandoc().

 -- Function: ast.fix_missing_locations (node)
     When you compile a node tree with compile(), the
     compiler expects `lineno' and `col_offset' attributes for every
     node that supports them.  This is rather tedious to fill in for
     generated nodes, so this helper adds these attributes recursively
     where not already set, by setting them to the values of the parent
     node.  It works recursively starting at _node_.

 -- Function: ast.increment_lineno (node, n=1)
     Increment the line number of each node in the tree starting at
     _node_ by _n_.  This is useful to "move code" to a different
     location in a file.

 -- Function: ast.copy_location (new_node, old_node)
     Copy source location (`lineno' and `col_offset') from _old_node_
     to _new_node_ if possible, and return _new_node_.

 -- Function: ast.iter_fields (node)
     Yield a tuple of `(fieldname, value)' for each field in
     `node._fields' that is present on _node_.

 -- Function: ast.iter_child_nodes (node)
     Yield all direct child nodes of _node_, that is, all fields that
     are nodes and all items of fields that are lists of nodes.

 -- Function: ast.walk (node)
     Recursively yield all child nodes of _node_, in no specified
     order.  This is useful if you only want to modify nodes in place
     and don't care about the context.

 -- Class: ast.NodeVisitor
     A node visitor base class that walks the abstract syntax tree and
     calls a visitor function for every node found.  This function may
     return a value which is forwarded by the visit().
     method.

     This class is meant to be subclassed, with the subclass adding
     visitor methods.

      -- Method: visit (node)
          Visit a node.  The default implementation calls the method
          called `self.visit__classname_' where _classname_ is the name
          of the node class, or generic_visit() if that
          method doesn't exist.

      -- Method: generic_visit (node)
          This visitor calls visit() on all children of
          the node.

          Note that child nodes of nodes that have a custom visitor
          method won't be visited unless the visitor calls
          generic_visit() or visits them itself.

     Don't use the NodeVisitor if you want to apply
     changes to nodes during traversal.  For this a special visitor
     exists (see NodeTransformer) that allows modifications.

 -- Class: ast.NodeTransformer
     A NodeVisitor subclass that walks the abstract syntax
     tree and allows modification of nodes.

     The NodeTransformer will walk the AST and use the
     return value of the visitor methods to replace or remove the old
     node.  If the return value of the visitor method is `None', the
     node will be removed from its location, otherwise it is replaced
     with the return value.  The return value may be the original node
     in which case no replacement takes place.

     Here is an example transformer that rewrites all occurrences of
     name lookups (`foo') to `data['foo']':

         class RewriteName(NodeTransformer):

             def visit_Name(self, node):
                 return copy_location(Subscript(
                     value=Name(id='data', ctx=Load()),
                     slice=Index(value=Str(s=node.id)),
                     ctx=node.ctx
                 ), node)

     Keep in mind that if the node you're operating on has child nodes
     you must either transform the child nodes yourself or call the
     `generic_visit()' method for the node first.

     For nodes that were part of a collection of statements (that
     applies to all statement nodes), the visitor may also return a
     list of nodes rather than just a single node.

     Usually you use the transformer like this:

         node = YourTransformer().visit(node)

 -- Function: ast.dump (node, annotate_fields=True,
          include_attributes=False)
     Return a formatted dump of the tree in _node_.  This is mainly
     useful for debugging purposes.  The returned string will show the
     names and the values for fields.  This makes the code impossible
     to evaluate, so if evaluation is wanted _annotate_fields_ must be
     set to False.  Attributes such as line numbers and column offsets
     are not dumped by default.  If this is wanted,
     _include_attributes_ can be set to `True'.

5.31.3 `symtable' -- Access to the compiler's symbol tables
-----------------------------------------------------------

Symbol tables are generated by the compiler from AST just before
bytecode is generated.  The symbol table is responsible for calculating
the scope of every identifier in the code.  See symtable.
provides an interface to examine these tables.

5.31.3.1 Generating Symbol Tables
.................................

 -- Function: symtable.symtable (code, filename, compile_type)
     Return the toplevel SymbolTable for the Python source
     _code_.  _filename_ is the name of the file containing the code.
     _compile_type_ is like the _mode_ argument to compile().

5.31.3.2 Examining Symbol Tables
................................

 -- Class: symtable.SymbolTable
     A namespace table for a block.  The constructor is not public.

      -- Method: get_type ()
          Return the type of the symbol table.  Possible values are
          `'class'', `'module'', and `'function''.

      -- Method: get_id ()
          Return the table's identifier.

      -- Method: get_name ()
          Return the table's name.  This is the name of the class if
          the table is for a class, the name of the function if the
          table is for a function, or `'top'' if the table is global
          (see get_type() returns `'module'').

      -- Method: get_lineno ()
          Return the number of the first line in the block this table
          represents.

      -- Method: is_optimized ()
          Return `True' if the locals in this table can be optimized.

      -- Method: is_nested ()
          Return `True' if the block is a nested class or function.

      -- Method: has_children ()
          Return `True' if the block has nested namespaces within it.
          These can be obtained with get_children().

      -- Method: has_exec ()
          Return `True' if the block uses `exec'.

      -- Method: has_import_star ()
          Return `True' if the block uses a starred from-import.

      -- Method: get_identifiers ()
          Return a list of names of symbols in this table.

      -- Method: lookup (name)
          Lookup _name_ in the table and return a Symbol.
          instance.

      -- Method: get_symbols ()
          Return a list of Symbol instances for names in
          the table.

      -- Method: get_children ()
          Return a list of the nested symbol tables.

 -- Class: symtable.Function
     A namespace for a function or method.  This class inherits
     SymbolTable.

      -- Method: get_parameters ()
          Return a tuple containing names of parameters to this
          function.

      -- Method: get_locals ()
          Return a tuple containing names of locals in this function.

      -- Method: get_globals ()
          Return a tuple containing names of globals in this function.

      -- Method: get_frees ()
          Return a tuple containing names of free variables in this
          function.

 -- Class: symtable.Class
     A namespace of a class.  This class inherits SymbolTable.

      -- Method: get_methods ()
          Return a tuple containing the names of methods declared in
          the class.

 -- Class: symtable.Symbol
     An entry in a SymbolTable corresponding to an
     identifier in the source.  The constructor is not public.

      -- Method: get_name ()
          Return the symbol's name.

      -- Method: is_referenced ()
          Return `True' if the symbol is used in its block.

      -- Method: is_imported ()
          Return `True' if the symbol is created from an import
          statement.

      -- Method: is_parameter ()
          Return `True' if the symbol is a parameter.

      -- Method: is_global ()
          Return `True' if the symbol is global.

      -- Method: is_declared_global ()
          Return `True' if the symbol is declared global with a global
          statement.

      -- Method: is_local ()
          Return `True' if the symbol is local to its block.

      -- Method: is_free ()
          Return `True' if the symbol is referenced in its block, but
          not assigned to.

      -- Method: is_assigned ()
          Return `True' if the symbol is assigned to in its block.

      -- Method: is_namespace ()
          Return `True' if name binding introduces new namespace.

          If the name is used as the target of a function or class
          statement, this will be true.

          For example:

              >>> table = symtable.symtable("def some_func(): pass", "string", "exec")
              >>> table.lookup("some_func").is_namespace()
              True

          Note that a single name can be bound to multiple objects.  If
          the result is `True', the name may also be bound to other
          objects, like an int or list, that does not introduce a new
          namespace.

      -- Method: get_namespaces ()
          Return a list of namespaces bound to this name.

      -- Method: get_namespace ()
          Return the namespace bound to this name.  If more than one
          namespace is bound, a ValueError is raised.

5.31.4 `symbol' -- Constants used with Python parse trees
---------------------------------------------------------

This module provides constants which represent the numeric values of
internal nodes of the parse tree.  Unlike most Python constants, these
use lower-case names.  Refer to the file `Grammar/Grammar' in the
Python distribution for the definitions of the names in the context of
the language grammar.  The specific numeric values which the names map
to may change between Python versions.

  This module also provides one additional data object:

 -- Data: symbol.sym_name
     Dictionary mapping the numeric values of the constants defined in
     this module back to name strings, allowing more human-readable
     representation of parse trees to be generated.

Module parser.
     The second example for the parser module shows how to
     use the symbol module.

5.31.5 `token' -- Constants used with Python parse trees
--------------------------------------------------------

This module provides constants which represent the numeric values of
leaf nodes of the parse tree (terminal tokens).  Refer to the file
`Grammar/Grammar' in the Python distribution for the definitions of the
names in the context of the language grammar.  The specific numeric
values which the names map to may change between Python versions.

  The module also provides a mapping from numeric codes to names and
some functions.  The functions mirror definitions in the Python C
header files.

 -- Data: token.tok_name
     Dictionary mapping the numeric values of the constants defined in
     this module back to name strings, allowing more human-readable
     representation of parse trees to be generated.

 -- Function: token.ISTERMINAL (x)
     Return true for terminal token values.

 -- Function: token.ISNONTERMINAL (x)
     Return true for non-terminal token values.

 -- Function: token.ISEOF (x)
     Return true if _x_ is the marker indicating the end of input.

  The token constants are:

 -- Data: token.ENDMARKER
 -- Data: token.NAME
 -- Data: token.NUMBER
 -- Data: token.STRING
 -- Data: token.NEWLINE
 -- Data: token.INDENT
 -- Data: token.DEDENT
 -- Data: token.LPAR
 -- Data: token.RPAR
 -- Data: token.LSQB
 -- Data: token.RSQB
 -- Data: token.COLON
 -- Data: token.COMMA
 -- Data: token.SEMI
 -- Data: token.PLUS
 -- Data: token.MINUS
 -- Data: token.STAR
 -- Data: token.SLASH
 -- Data: token.VBAR
 -- Data: token.AMPER
 -- Data: token.LESS
 -- Data: token.GREATER
 -- Data: token.EQUAL
 -- Data: token.DOT
 -- Data: token.PERCENT
 -- Data: token.BACKQUOTE
 -- Data: token.LBRACE
 -- Data: token.RBRACE
 -- Data: token.EQEQUAL
 -- Data: token.NOTEQUAL
 -- Data: token.LESSEQUAL
 -- Data: token.GREATEREQUAL
 -- Data: token.TILDE
 -- Data: token.CIRCUMFLEX
 -- Data: token.LEFTSHIFT
 -- Data: token.RIGHTSHIFT
 -- Data: token.DOUBLESTAR
 -- Data: token.PLUSEQUAL
 -- Data: token.MINEQUAL
 -- Data: token.STAREQUAL
 -- Data: token.SLASHEQUAL
 -- Data: token.PERCENTEQUAL
 -- Data: token.AMPEREQUAL
 -- Data: token.VBAREQUAL
 -- Data: token.CIRCUMFLEXEQUAL
 -- Data: token.LEFTSHIFTEQUAL
 -- Data: token.RIGHTSHIFTEQUAL
 -- Data: token.DOUBLESTAREQUAL
 -- Data: token.DOUBLESLASH
 -- Data: token.DOUBLESLASHEQUAL
 -- Data: token.AT
 -- Data: token.OP
 -- Data: token.ERRORTOKEN
 -- Data: token.N_TOKENS
 -- Data: token.NT_OFFSET

Module parser.
     The second example for the parser module shows how to
     use the symbol module.

5.31.6 `keyword' -- Testing for Python keywords
-----------------------------------------------

This module allows a Python program to determine if a string is a
keyword.

 -- Function: keyword.iskeyword (s)
     Return true if _s_ is a Python keyword.

 -- Data: keyword.kwlist
     Sequence containing all the keywords defined for the interpreter.
     If any keywords are defined to only be active when particular
     __future__.  statements are in effect, these will be
     included as well.

  Latest version of the keyword module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/keyword.py?view=markup)

5.31.7 `tokenize' -- Tokenizer for Python source
------------------------------------------------

The tokenize module provides a lexical scanner for Python
source code, implemented in Python.  The scanner in this module returns
comments as tokens as well, making it useful for implementing
"pretty-printers," including colorizers for on-screen displays.

  Latest version of the tokenize module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/tokenize.py?view=markup)The
primary entry point is a generator.:

 -- Function: tokenize.generate_tokens (readline)
     The generate_tokens() generator requires one
     argument, _readline_, which must be a callable object which
     provides the same interface as the readline() method
     of built-in file objects (see section File Objects).
     Each call to the function should return one line of input as a
     string.

     The generator produces 5-tuples with these members: the token
     type; the token string; a 2-tuple `(srow, scol)' of ints
     specifying the row and column where the token begins in the
     source; a 2-tuple `(erow, ecol)' of ints specifying the row and
     column where the token ends in the source; and the line on which
     the token was found.  The line passed (the last tuple item) is the
     _logical_ line; continuation lines are included.

     New in version 2.2.

  An older entry point is retained for backward compatibility:

 -- Function: tokenize.tokenize (readline[, tokeneater])
     The tokenize() function accepts two parameters: one
     representing the input stream, and one providing an output
     mechanism for tokenize().

     The first parameter, _readline_, must be a callable object which
     provides the same interface as the readline() method
     of built-in file objects (see section File Objects).
     Each call to the function should return one line of input as a
     string. Alternately, _readline_ may be a callable object that
     signals completion by raising StopIteration.

     Changed in version 2.5: Added StopIteration support.

     The second parameter, _tokeneater_, must also be a callable
     object.  It is called once for each token, with five arguments,
     corresponding to the tuples generated by generate_tokens().

  All constants from the token module are also exported from
tokenize, as are two additional token type values that might
be passed to the _tokeneater_ function by tokenize().:

 -- Data: tokenize.COMMENT
     Token value used to indicate a comment.

 -- Data: tokenize.NL
     Token value used to indicate a non-terminating newline.  The
     NEWLINE token indicates the end of a logical line of Python code;
     NL tokens are generated when a logical line of code is continued
     over multiple physical lines.

  Another function is provided to reverse the tokenization process.
This is useful for creating tools that tokenize a script, modify the
token stream, and write back the modified script.

 -- Function: tokenize.untokenize (iterable)
     Converts tokens back into Python source code.  The _iterable_ must
     return sequences with at least two elements, the token type and
     the token string.  Any additional sequence elements are ignored.

     The reconstructed script is returned as a single string.  The
     result is guaranteed to tokenize back to match the input so that
     the conversion is lossless and round-trips are assured.  The
     guarantee applies only to the token type and token string as the
     spacing between tokens (column positions) may change.

     New in version 2.5.

  Example of a script re-writer that transforms float literals into
Decimal objects:

    def decistmt(s):
        """Substitute Decimals for floats in a string of statements.

        >>> from decimal import Decimal
        >>> s = 'print +21.3e-5*-.1234/81.7'
        >>> decistmt(s)
        "print +Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')"

        >>> exec(s)
        -3.21716034272e-007
        >>> exec(decistmt(s))
        -3.217160342717258261933904529E-7

        """
        result = []
        g = generate_tokens(StringIO(s).readline)   # tokenize the string
        for toknum, tokval, _, _, _  in g:
            if toknum == NUMBER and '.' in tokval:  # replace NUMBER tokens
                result.extend([
                    (NAME, 'Decimal'),
                    (OP, '('),
                    (STRING, repr(tokval)),
                    (OP, ')')
                ])
            else:
                result.append((toknum, tokval))
        return untokenize(result)

5.31.8 `tabnanny' -- Detection of ambiguous indentation
-------------------------------------------------------

For the time being this module is intended to be called as a script.
However it is possible to import it into an IDE and use the function
check().  described below.

     Note: The API provided by this module is likely to change in
     future releases; such changes may not be backward compatible.

 -- Function: tabnanny.check (file_or_dir)
     If _file_or_dir_ is a directory and not a symbolic link, then
     recursively descend the directory tree named by _file_or_dir_,
     checking all `.py' files along the way.  If _file_or_dir_ is an
     ordinary Python source file, it is checked for whitespace related
     problems.  The diagnostic messages are written to standard output
     using the print statement.

 -- Data: tabnanny.verbose
     Flag indicating whether to print verbose messages. This is
     incremented by the `-v' option if called as a script.

 -- Data: tabnanny.filename_only
     Flag indicating whether to print only the filenames of files
     containing whitespace related problems.  This is set to true by
     the `-q' option if called as a script.

 -- Exception: tabnanny.NannyNag
     Raised by tokeneater() if detecting an ambiguous
     indent. Captured and handled in check().

 -- Function: tabnanny.tokeneater (type, token, start, end, line)
     This function is used by check() as a callback
     parameter to the function tokenize.tokenize().

Module tokenize.
     Lexical scanner for Python source code.

5.31.9 `pyclbr' -- Python class browser support
-----------------------------------------------

The pyclbr module can be used to determine some limited
information about the classes, methods and top-level functions defined
in a module.  The information provided is sufficient to implement a
traditional three-pane class browser.  The information is extracted
from the source code rather than by importing the module, so this
module is safe to use with untrusted code.  This restriction makes it
impossible to use this module with modules not implemented in Python,
including all standard and optional extension modules.

 -- Function: pyclbr.readmodule (module[, path=None])
     Read a module and return a dictionary mapping class names to class
     descriptor objects.  The parameter _module_ should be the name of a
     module as a string; it may be the name of a module within a
     package.  The _path_ parameter should be a sequence, and is used
     to augment the value of `sys.path', which is used to locate module
     source code.

 -- Function: pyclbr.readmodule_ex (module[, path=None])
     Like readmodule(), but the returned dictionary, in
     addition to mapping class names to class descriptor objects, also
     maps top-level function names to function descriptor objects.
     Moreover, if the module being read is a package, the key
     `'__path__'' in the returned dictionary has as its value a list
     which contains the package search path.

5.31.9.1 Class Objects
......................

The `Class' objects used as values in the dictionary returned by
readmodule() and readmodule_ex() provide the
following data members:

 -- Attribute: Class.module
     The name of the module defining the class described by the class
     descriptor.

 -- Attribute: Class.name
     The name of the class.

 -- Attribute: Class.super
     A list of `Class' objects which describe the immediate base
     classes of the class being described.  Classes which are named as
     superclasses but which are not discoverable by readmodule()
     are listed as a string with the class name instead of as
     `Class' objects.

 -- Attribute: Class.methods
     A dictionary mapping method names to line numbers.

 -- Attribute: Class.file
     Name of the file containing the `class' statement defining the
     class.

 -- Attribute: Class.lineno
     The line number of the `class' statement within the file named by
     file.

5.31.9.2 Function Objects
.........................

The `Function' objects used as values in the dictionary returned by
readmodule_ex() provide the following data members:

 -- Attribute: Function.module
     The name of the module defining the function described by the
     function descriptor.

 -- Attribute: Function.name
     The name of the function.

 -- Attribute: Function.file
     Name of the file containing the `def' statement defining the
     function.

 -- Attribute: Function.lineno
     The line number of the `def' statement within the file named by
     file.

5.31.10 `py_compile' -- Compile Python source files
---------------------------------------------------

The py_compile module provides a function to generate a
byte-code file from a source file, and another function used when the
module source file is invoked as a script.

  Though not often needed, this function can be useful when installing
modules for shared use, especially if some of the users may not have
permission to write the byte-code cache files in the directory
containing the source code.

 -- Exception: py_compile.PyCompileError
     Exception raised when an error occurs while attempting to compile
     the file.

 -- Function: py_compile.compile (file[, cfile[, dfile[, doraise]]])
     Compile a source file to byte-code and write out the byte-code
     cache  file.  The source code is loaded from the file name _file_.
     The  byte-code is written to _cfile_, which defaults to _file_ `+'
     `'c'' (`'o'' if optimization is enabled in the current
     interpreter).  If _dfile_ is specified, it is used as the name of
     the source file in error messages instead of _file_.  If _doraise_
     is true, a PyCompileError is raised when an error is
     encountered while compiling _file_. If _doraise_ is false (the
     default), an error string is written to `sys.stderr', but no
     exception is raised.

 -- Function: py_compile.main ([args])
     Compile several source files.  The files named in _args_ (or on
     the command line, if _args_ is not specified) are compiled and the
     resulting bytecode is cached in the normal manner.  This function
     does not search a directory structure to locate source files; it
     only compiles files named explicitly.

  When this module is run as a script, the main() is used
to compile all the files named on the command line.  The exit status is
nonzero if one of the files could not be compiled.

  Changed in version 2.6: Added the nonzero exit status when module is
run as a script.

Module compileall.
     Utilities to compile all Python source files in a directory tree.

5.31.11 `compileall' -- Byte-compile Python libraries
-----------------------------------------------------

This module provides some utility functions to support installing Python
libraries.  These functions compile Python source files in a directory
tree, allowing users without permission to write to the libraries to
take advantage of cached byte-code files.

  This module may also be used as a script (using the -m.
Python flag) to compile Python sources.  Directories to recursively
traverse (passing `-l' stops the recursive behavior) for sources are
listed on the command line.  If no arguments are given, the invocation
is equivalent to `-l sys.path'.  Printing lists of the files compiled
can be disabled with the `-q' flag.  In addition, the -x.
option takes a regular expression argument.  All files that match the
expression will be skipped.

 -- Function: compileall.compile_dir (dir[, maxlevels[, ddir[, force[,
          rx[, quiet]]]]])
     Recursively descend the directory tree named by _dir_, compiling
     all `.py' files along the way.  The _maxlevels_ parameter is used
     to limit the depth of the recursion; it defaults to `10'.  If
     _ddir_ is given, it is used as the base path from  which the
     filenames used in error messages will be generated.  If _force_ is
     true, modules are re-compiled even if the timestamps are up to
     date.

     If _rx_ is given, it specifies a regular expression of file names
     to exclude from the search; that expression is searched for in the
     full path.

     If _quiet_ is true, nothing is printed to the standard output in
     normal operation.

 -- Function: compileall.compile_path ([skip_curdir[, maxlevels[,
          force]]])
     Byte-compile all the `.py' files found along `sys.path'. If
     _skip_curdir_ is true (the default), the current directory is not
     included in the search.  The _maxlevels_ and _force_ parameters
     default to `0' and are passed to the compile_dir().
     function.

  To force a recompile of all the `.py' files in the `Lib/'
subdirectory and all its subdirectories:

    import compileall

    compileall.compile_dir('Lib/', force=True)

    # Perform same compilation, excluding files in .svn directories.
    import re
    compileall.compile_dir('Lib/', rx=re.compile('/[.]svn'), force=True)

Module py_compile.
     Byte-compile a single source file.

5.31.12 `dis' -- Disassembler for Python bytecode
-------------------------------------------------

The dis module supports the analysis of CPython
bytecode by disassembling it. The CPython bytecode which this
module takes as an input is defined in the file `Include/opcode.h' and
used by the compiler and the interpreter.

  Latest version of the dis module Python source code
(http://svn.python.org/view/python/branches/release27-maint/Lib/dis.py?view=markup)*CPython
implementation detail:* Bytecode is an implementation detail of the
CPython interpreter!  No guarantees are made that bytecode will not be
added, removed, or changed between versions of Python.  Use of this
module should not be considered to work across Python VMs or Python
releases.

  Example: Given the function `myfunc()':

    def myfunc(alist):
        return len(alist)

the following command can be used to get the disassembly of `myfunc()':

    >>> dis.dis(myfunc)
      2           0 LOAD_GLOBAL              0 (len)
                  3 LOAD_FAST                0 (alist)
                  6 CALL_FUNCTION            1
                  9 RETURN_VALUE

(The "2" is a line number).

  The dis module defines the following functions and
constants:

 -- Function: dis.dis ([bytesource])
     Disassemble the _bytesource_ object. _bytesource_ can denote
     either a module, a class, a method, a function, or a code object.
     For a module, it disassembles all functions.  For a class, it
     disassembles all methods.  For a single code sequence, it prints
     one line per bytecode instruction.  If no object is provided, it
     disassembles the last traceback.

 -- Function: dis.distb ([tb])
     Disassembles the top-of-stack function of a traceback, using the
     last traceback if none was passed.  The instruction causing the
     exception is indicated.

 -- Function: dis.disassemble (code[, lasti])
     Disassembles a code object, indicating the last instruction if
     _lasti_ was provided.  The output is divided in the following
     columns:

       1. the line number, for the first instruction of each line

       2. the current instruction, indicated as `-->',

       3. a labelled instruction, indicated with `>>',

       4. the address of the instruction,

       5. the operation code name,

       6. operation parameters, and

       7. interpretation of the parameters in parentheses.

     The parameter interpretation recognizes local and global variable
     names, constant values, branch targets, and compare operators.

 -- Function: dis.disco (code[, lasti])
     A synonym for disassemble().  It is more convenient to
     type, and kept for compatibility with earlier Python releases.

 -- Function: dis.findlinestarts (code)
     This generator function uses the `co_firstlineno' and `co_lnotab'
     attributes of the code object _code_ to find the offsets which are
     starts of lines in the source code.  They are generated as
     `(offset, lineno)' pairs.

 -- Function: dis.findlabels (code)
     Detect all offsets in the code object _code_ which are jump
     targets, and return a list of these offsets.

 -- Data: dis.opname
     Sequence of operation names, indexable using the bytecode.

 -- Data: dis.opmap
     Dictionary mapping operation names to bytecodes.

 -- Data: dis.cmp_op
     Sequence of all compare operation names.

 -- Data: dis.hasconst
     Sequence of bytecodes that have a constant parameter.

 -- Data: dis.hasfree
     Sequence of bytecodes that access a free variable.

 -- Data: dis.hasname
     Sequence of bytecodes that access an attribute by name.

 -- Data: dis.hasjrel
     Sequence of bytecodes that have a relative jump target.

 -- Data: dis.hasjabs
     Sequence of bytecodes that have an absolute jump target.

 -- Data: dis.haslocal
     Sequence of bytecodes that access a local variable.

 -- Data: dis.hascompare
     Sequence of bytecodes of Boolean operations.

5.31.12.1 Python Bytecode Instructions
......................................

The Python compiler currently generates the following bytecode
instructions.

 -- Opcode: STOP_CODE ()
     Indicates end-of-code to the compiler, not used by the interpreter.

 -- Opcode: NOP ()
     Do nothing code.  Used as a placeholder by the bytecode optimizer.

 -- Opcode: POP_TOP ()
     Removes the top-of-stack (TOS) item.

 -- Opcode: ROT_TWO ()
     Swaps the two top-most stack items.

 -- Opcode: ROT_THREE ()
     Lifts second and third stack item one position up, moves top down
     to position three.

 -- Opcode: ROT_FOUR ()
     Lifts second, third and forth stack item one position up, moves
     top down to position four.

 -- Opcode: DUP_TOP ()
     Duplicates the reference on top of the stack.

  Unary Operations take the top of the stack, apply the operation, and
push the result back on the stack.

 -- Opcode: UNARY_POSITIVE ()
     Implements `TOS = +TOS'.

 -- Opcode: UNARY_NEGATIVE ()
     Implements `TOS = -TOS'.

 -- Opcode: UNARY_NOT ()
     Implements `TOS = not TOS'.

 -- Opcode: UNARY_CONVERT ()
     Implements `TOS = `TOS`'.

 -- Opcode: UNARY_INVERT ()
     Implements `TOS = ~TOS'.

 -- Opcode: GET_ITER ()
     Implements `TOS = iter(TOS)'.

  Binary operations remove the top of the stack (TOS) and the second
top-most stack item (TOS1) from the stack.  They perform the operation,
and put the result back on the stack.

 -- Opcode: BINARY_POWER ()
     Implements `TOS = TOS1 ** TOS'.

 -- Opcode: BINARY_MULTIPLY ()
     Implements `TOS = TOS1 * TOS'.

 -- Opcode: BINARY_DIVIDE ()
     Implements `TOS = TOS1 / TOS' when `from __future__ import
     division' is not in effect.

 -- Opcode: BINARY_FLOOR_DIVIDE ()
     Implements `TOS = TOS1 // TOS'.

 -- Opcode: BINARY_TRUE_DIVIDE ()
     Implements `TOS = TOS1 / TOS' when `from __future__ import
     division' is in effect.

 -- Opcode: BINARY_MODULO ()
     Implements `TOS = TOS1 % TOS'.

 -- Opcode: BINARY_ADD ()
     Implements `TOS = TOS1 + TOS'.

 -- Opcode: BINARY_SUBTRACT ()
     Implements `TOS = TOS1 - TOS'.

 -- Opcode: BINARY_SUBSCR ()
     Implements `TOS = TOS1[TOS]'.

 -- Opcode: BINARY_LSHIFT ()
     Implements `TOS = TOS1 << TOS'.

 -- Opcode: BINARY_RSHIFT ()
     Implements `TOS = TOS1 >> TOS'.

 -- Opcode: BINARY_AND ()
     Implements `TOS = TOS1 & TOS'.

 -- Opcode: BINARY_XOR ()
     Implements `TOS = TOS1 ^ TOS'.

 -- Opcode: BINARY_OR ()
     Implements `TOS = TOS1 | TOS'.

  In-place operations are like binary operations, in that they remove
TOS and TOS1, and push the result back on the stack, but the operation
is done in-place when TOS1 supports it, and the resulting TOS may be
(but does not have to be) the original TOS1.

 -- Opcode: INPLACE_POWER ()
     Implements in-place `TOS = TOS1 ** TOS'.

 -- Opcode: INPLACE_MULTIPLY ()
     Implements in-place `TOS = TOS1 * TOS'.

 -- Opcode: INPLACE_DIVIDE ()
     Implements in-place `TOS = TOS1 / TOS' when `from __future__ import
     division' is not in effect.

 -- Opcode: INPLACE_FLOOR_DIVIDE ()
     Implements in-place `TOS = TOS1 // TOS'.

 -- Opcode: INPLACE_TRUE_DIVIDE ()
     Implements in-place `TOS = TOS1 / TOS' when `from __future__ import
     division' is in effect.

 -- Opcode: INPLACE_MODULO ()
     Implements in-place `TOS = TOS1 % TOS'.

 -- Opcode: INPLACE_ADD ()
     Implements in-place `TOS = TOS1 + TOS'.

 -- Opcode: INPLACE_SUBTRACT ()
     Implements in-place `TOS = TOS1 - TOS'.

 -- Opcode: INPLACE_LSHIFT ()
     Implements in-place `TOS = TOS1 << TOS'.

 -- Opcode: INPLACE_RSHIFT ()
     Implements in-place `TOS = TOS1 >> TOS'.

 -- Opcode: INPLACE_AND ()
     Implements in-place `TOS = TOS1 & TOS'.

 -- Opcode: INPLACE_XOR ()
     Implements in-place `TOS = TOS1 ^ TOS'.

 -- Opcode: INPLACE_OR ()
     Implements in-place `TOS = TOS1 | TOS'.

  The slice opcodes take up to three parameters.

 -- Opcode: SLICE+0 ()
     Implements `TOS = TOS[:]'.

 -- Opcode: SLICE+1 ()
     Implements `TOS = TOS1[TOS:]'.

 -- Opcode: SLICE+2 ()
     Implements `TOS = TOS1[:TOS]'.

 -- Opcode: SLICE+3 ()
     Implements `TOS = TOS2[TOS1:TOS]'.

  Slice assignment needs even an additional parameter.  As any
statement, they put nothing on the stack.

 -- Opcode: STORE_SLICE+0 ()
     Implements `TOS[:] = TOS1'.

 -- Opcode: STORE_SLICE+1 ()
     Implements `TOS1[TOS:] = TOS2'.

 -- Opcode: STORE_SLICE+2 ()
     Implements `TOS1[:TOS] = TOS2'.

 -- Opcode: STORE_SLICE+3 ()
     Implements `TOS2[TOS1:TOS] = TOS3'.

 -- Opcode: DELETE_SLICE+0 ()
     Implements `del TOS[:]'.

 -- Opcode: DELETE_SLICE+1 ()
     Implements `del TOS1[TOS:]'.

 -- Opcode: DELETE_SLICE+2 ()
     Implements `del TOS1[:TOS]'.

 -- Opcode: DELETE_SLICE+3 ()
     Implements `del TOS2[TOS1:TOS]'.

 -- Opcode: STORE_SUBSCR ()
     Implements `TOS1[TOS] = TOS2'.

 -- Opcode: DELETE_SUBSCR ()
     Implements `del TOS1[TOS]'.

  Miscellaneous opcodes.

 -- Opcode: PRINT_EXPR ()
     Implements the expression statement for the interactive mode.  TOS
     is removed from the stack and printed.  In non-interactive mode,
     an expression statement is terminated with `POP_STACK'.

 -- Opcode: PRINT_ITEM ()
     Prints TOS to the file-like object bound to `sys.stdout'.  There
     is one such instruction for each item in the print.
     statement.

 -- Opcode: PRINT_ITEM_TO ()
     Like `PRINT_ITEM', but prints the item second from TOS to the
     file-like object at TOS.  This is used by the extended print
     statement.

 -- Opcode: PRINT_NEWLINE ()
     Prints a new line on `sys.stdout'.  This is generated as the last
     operation of a print statement, unless the statement
     ends with a comma.

 -- Opcode: PRINT_NEWLINE_TO ()
     Like `PRINT_NEWLINE', but prints the new line on the file-like
     object on the TOS.  This is used by the extended print statement.

 -- Opcode: BREAK_LOOP ()
     Terminates a loop due to a break statement.

 -- Opcode: CONTINUE_LOOP (target)
     Continues a loop due to a continue statement.
     _target_ is the address to jump to (which should be a `FOR_ITER'
     instruction).

 -- Opcode: LIST_APPEND (i)
     Calls `list.append(TOS[-i], TOS)'.  Used to implement list
     comprehensions.  While the appended value is popped off, the list
     object remains on the stack so that it is available for further
     iterations of the loop.

 -- Opcode: LOAD_LOCALS ()
     Pushes a reference to the locals of the current scope on the
     stack. This is used in the code for a class definition: After the
     class body is evaluated, the locals are passed to the class
     definition.

 -- Opcode: RETURN_VALUE ()
     Returns with TOS to the caller of the function.

 -- Opcode: YIELD_VALUE ()
     Pops `TOS' and yields it from a generator.

 -- Opcode: IMPORT_STAR ()
     Loads all symbols not starting with `'_'' directly from the module
     TOS to the local namespace. The module is popped after loading all
     names. This opcode implements `from module import *'.

 -- Opcode: EXEC_STMT ()
     Implements `exec TOS2,TOS1,TOS'.  The compiler fills missing
     optional parameters with `None'.

 -- Opcode: POP_BLOCK ()
     Removes one block from the block stack.  Per frame, there is a
     stack of blocks, denoting nested loops, try statements, and such.

 -- Opcode: END_FINALLY ()
     Terminates a finally clause.  The interpreter recalls
     whether the exception has to be re-raised, or whether the function
     returns, and continues with the outer-next block.

 -- Opcode: BUILD_CLASS ()
     Creates a new class object.  TOS is the methods dictionary, TOS1
     the tuple of the names of the base classes, and TOS2 the class
     name.

 -- Opcode: SETUP_WITH (delta)
     This opcode performs several operations before a with block
     starts.  First, it loads __exit__() from the context
     manager and pushes it onto the stack for later use by
     WITH_CLEANUP.  Then, see __enter__() is called, and
     a finally block pointing to _delta_ is pushed.  Finally, the
     result of calling the enter method is pushed onto the stack.  The
     next opcode will either ignore it (see POP_TOP), or store
     it in (a) variable(s) (see STORE_FAST, STORE_NAME,
     or UNPACK_SEQUENCE).

 -- Opcode: WITH_CLEANUP ()
     Cleans up the stack when a with statement block exits.
     On top of the stack are 1-3 values indicating how/why the finally
     clause was entered:

        * TOP = `None'

        * (TOP, SECOND) = (`WHY_{RETURN,CONTINUE}'), retval

        * TOP = `WHY_*'; no retval below it

        * (TOP, SECOND, THIRD) = exc_info()

     Under them is EXIT, the context manager's __exit__().
     bound method.

     In the last case, `EXIT(TOP, SECOND, THIRD)' is called, otherwise
     `EXIT(None, None, None)'.

     EXIT is removed from the stack, leaving the values above it in the
     same order. In addition, if the stack represents an exception,
     _and_ the function call returns a 'true' value, this information
     is "zapped", to prevent `END_FINALLY' from re-raising the
     exception.  (But non-local gotos should still be resumed.)

  All of the following opcodes expect arguments.  An argument is two
bytes, with the more significant byte last.

 -- Opcode: STORE_NAME (namei)
     Implements `name = TOS'. _namei_ is the index of _name_ in the
     attribute `co_names' of the code object. The compiler tries to use
     `STORE_FAST' or `STORE_GLOBAL' if possible.

 -- Opcode: DELETE_NAME (namei)
     Implements `del name', where _namei_ is the index into `co_names'
     attribute of the code object.

 -- Opcode: UNPACK_SEQUENCE (count)
     Unpacks TOS into _count_ individual values, which are put onto the
     stack right-to-left.

 -- Opcode: DUP_TOPX (count)
     Duplicate _count_ items, keeping them in the same order. Due to
     implementation limits, _count_ should be between 1 and 5 inclusive.

 -- Opcode: STORE_ATTR (namei)
     Implements `TOS.name = TOS1', where _namei_ is the index of name in
     `co_names'.

 -- Opcode: DELETE_ATTR (namei)
     Implements `del TOS.name', using _namei_ as index into `co_names'.

 -- Opcode: STORE_GLOBAL (namei)
     Works as `STORE_NAME', but stores the name as a global.

 -- Opcode: DELETE_GLOBAL (namei)
     Works as `DELETE_NAME', but deletes a global name.

 -- Opcode: LOAD_CONST (consti)
     Pushes `co_consts[consti]' onto the stack.

 -- Opcode: LOAD_NAME (namei)
     Pushes the value associated with `co_names[namei]' onto the stack.

 -- Opcode: BUILD_TUPLE (count)
     Creates a tuple consuming _count_ items from the stack, and pushes
     the resulting tuple onto the stack.

 -- Opcode: BUILD_LIST (count)
     Works as `BUILD_TUPLE', but creates a list.

 -- Opcode: BUILD_MAP (count)
     Pushes a new dictionary object onto the stack.  The dictionary is
     pre-sized to hold _count_ entries.

 -- Opcode: LOAD_ATTR (namei)
     Replaces TOS with `getattr(TOS, co_names[namei])'.

 -- Opcode: COMPARE_OP (opname)
     Performs a Boolean operation.  The operation name can be found in
     `cmp_op[opname]'.

 -- Opcode: IMPORT_NAME (namei)
     Imports the module `co_names[namei]'.  TOS and TOS1 are popped and
     provide the _fromlist_ and _level_ arguments of
     __import__().  The module object is pushed onto the stack.
     The current namespace is not affected: for a proper import
     statement, a subsequent `STORE_FAST' instruction modifies the
     namespace.

 -- Opcode: IMPORT_FROM (namei)
     Loads the attribute `co_names[namei]' from the module found in
     TOS. The resulting object is pushed onto the stack, to be
     subsequently stored by a `STORE_FAST' instruction.

 -- Opcode: JUMP_FORWARD (delta)
     Increments bytecode counter by _delta_.

 -- Opcode: POP_JUMP_IF_TRUE (target)
     If TOS is true, sets the bytecode counter to _target_.  TOS is
     popped.

 -- Opcode: POP_JUMP_IF_FALSE (target)
     If TOS is false, sets the bytecode counter to _target_.  TOS is
     popped.

 -- Opcode: JUMP_IF_TRUE_OR_POP (target)
     If TOS is true, sets the bytecode counter to _target_ and leaves
     TOS on the stack.  Otherwise (TOS is false), TOS is popped.

 -- Opcode: JUMP_IF_FALSE_OR_POP (target)
     If TOS is false, sets the bytecode counter to _target_ and leaves
     TOS on the stack.  Otherwise (TOS is true), TOS is popped.

 -- Opcode: JUMP_ABSOLUTE (target)
     Set bytecode counter to _target_.

 -- Opcode: FOR_ITER (delta)
     `TOS' is an iterator.  Call its `next()' method.  If
     this yields a new value, push it on the stack (leaving the
     iterator below it).  If the iterator indicates it is exhausted
     `TOS' is popped, and the bytecode counter is incremented by
     _delta_.

 -- Opcode: LOAD_GLOBAL (namei)
     Loads the global named `co_names[namei]' onto the stack.

 -- Opcode: SETUP_LOOP (delta)
     Pushes a block for a loop onto the block stack.  The block spans
     from the current instruction with a size of _delta_ bytes.

 -- Opcode: SETUP_EXCEPT (delta)
     Pushes a try block from a try-except clause onto the block stack.
     _delta_ points to the first except block.

 -- Opcode: SETUP_FINALLY (delta)
     Pushes a try block from a try-except clause onto the block stack.
     _delta_ points to the finally block.

 -- Opcode: STORE_MAP ()
     Store a key and value pair in a dictionary.  Pops the key and
     value while leaving the dictionary on the stack.

 -- Opcode: LOAD_FAST (var_num)
     Pushes a reference to the local `co_varnames[var_num]' onto the
     stack.

 -- Opcode: STORE_FAST (var_num)
     Stores TOS into the local `co_varnames[var_num]'.

 -- Opcode: DELETE_FAST (var_num)
     Deletes local `co_varnames[var_num]'.

 -- Opcode: LOAD_CLOSURE (i)
     Pushes a reference to the cell contained in slot _i_ of the cell
     and free variable storage.  The name of the variable is
     `co_cellvars[i]' if _i_ is less than the length of _co_cellvars_.
     Otherwise it is  `co_freevars[i - len(co_cellvars)]'.

 -- Opcode: LOAD_DEREF (i)
     Loads the cell contained in slot _i_ of the cell and free variable
     storage.  Pushes a reference to the object the cell contains on
     the stack.

 -- Opcode: STORE_DEREF (i)
     Stores TOS into the cell contained in slot _i_ of the cell and
     free variable storage.

 -- Opcode: SET_LINENO (lineno)
     This opcode is obsolete.

 -- Opcode: RAISE_VARARGS (argc)
     Raises an exception. _argc_ indicates the number of parameters to
     the raise statement, ranging from 0 to 3.  The handler will find
     the traceback as TOS2, the parameter as TOS1, and the exception as
     TOS.

 -- Opcode: CALL_FUNCTION (argc)
     Calls a function.  The low byte of _argc_ indicates the number of
     positional parameters, the high byte the number of keyword
     parameters. On the stack, the opcode finds the keyword parameters
     first.  For each keyword argument, the value is on top of the key.
     Below the keyword parameters, the positional parameters are on the
     stack, with the right-most parameter on top.  Below the parameters,
     the function object to call is on the stack.  Pops all function
     arguments, and the function itself off the stack, and pushes the
     return value.

 -- Opcode: MAKE_FUNCTION (argc)
     Pushes a new function object on the stack.  TOS is the code
     associated with the function.  The function object is defined to
     have _argc_ default parameters, which are found below TOS.

 -- Opcode: MAKE_CLOSURE (argc)
     Creates a new function object, sets its _func_closure_ slot, and
     pushes it on the stack.  TOS is the code associated with the
     function, TOS1 the tuple containing cells for the closure's free
     variables.  The function also has _argc_ default parameters, which
     are found below the cells.

 -- Opcode: BUILD_SLICE (argc)
     Pushes a slice object on the stack.  _argc_ must be 2 or 3.  If it
     is 2, `slice(TOS1, TOS)' is pushed; if it is 3, `slice(TOS2, TOS1,
     TOS)' is pushed. See the slice() built-in function for
     more information.

 -- Opcode: EXTENDED_ARG (ext)
     Prefixes any opcode which has an argument too big to fit into the
     default two bytes.  _ext_ holds two additional bytes which, taken
     together with the subsequent opcode's argument, comprise a
     four-byte argument, _ext_ being the two most-significant bytes.

 -- Opcode: CALL_FUNCTION_VAR (argc)
     Calls a function. _argc_ is interpreted as in `CALL_FUNCTION'. The
     top element on the stack contains the variable argument list,
     followed by keyword and positional arguments.

 -- Opcode: CALL_FUNCTION_KW (argc)
     Calls a function. _argc_ is interpreted as in `CALL_FUNCTION'. The
     top element on the stack contains the keyword arguments
     dictionary,  followed by explicit keyword and positional arguments.

 -- Opcode: CALL_FUNCTION_VAR_KW (argc)
     Calls a function. _argc_ is interpreted as in `CALL_FUNCTION'.
     The top element on the stack contains the keyword arguments
     dictionary, followed by the variable-arguments tuple, followed by
     explicit keyword and positional arguments.

 -- Opcode: HAVE_ARGUMENT ()
     This is not really an opcode.  It identifies the dividing line
     between opcodes which don't take arguments `< HAVE_ARGUMENT' and
     those which do `>= HAVE_ARGUMENT'.

5.31.13 `pickletools' -- Tools for pickle developers
----------------------------------------------------

New in version 2.3.

  This module contains various constants relating to the intimate
details of the pickle module, some lengthy comments about
the implementation, and a few useful functions for analyzing pickled
data.  The contents of this module are useful for Python core
developers who are working on the pickle and cPickle
implementations; ordinary users of the pickle module
probably won't find the pickletools module relevant.

 -- Function: pickletools.dis (pickle[, out=None, memo=None,
          indentlevel=4])
     Outputs a symbolic disassembly of the pickle to the file-like
     object _out_, defaulting to `sys.stdout'.  _pickle_ can be a
     string or a file-like object.  _memo_ can be a Python dictionary
     that will be used as the pickle's memo; it can be used to perform
     disassemblies across multiple pickles created by the same pickler.
     Successive levels, indicated by `MARK' opcodes in the stream, are
     indented by _indentlevel_ spaces.

 -- Function: pickletools.genops (pickle)
     Provides an iterator over all of the opcodes in a
     pickle, returning a sequence of `(opcode, arg, pos)' triples.
     _opcode_ is an instance of an `OpcodeInfo' class; _arg_ is the
     decoded value, as a Python object, of the opcode's argument; _pos_
     is the position at which this opcode is located.  _pickle_ can be
     a string or a file-like object.

 -- Function: pickletools.optimize (picklestring)
     Returns a new equivalent pickle string after eliminating unused
     `PUT' opcodes. The optimized pickle is shorter, takes less
     transmission time, requires less storage space, and unpickles more
     efficiently.

     New in version 2.6.

5.32 Python compiler package
============================

Deprecated since version 2.6: The compiler package has been
removed in Python 3.0.

  The Python compiler package is a tool for analyzing Python source
code and generating Python bytecode.  The compiler contains libraries
to generate an abstract syntax tree from Python source code and to
generate Python bytecode from the tree.

  The compiler package is a Python source to bytecode
translator written in Python.  It uses the built-in parser and standard
parser module to generate a concrete syntax tree.  This
tree is used to generate an abstract syntax tree (AST) and then Python
bytecode.

  The full functionality of the package duplicates the built-in
compiler provided with the Python interpreter.  It is intended to match
its behavior almost exactly.  Why implement another compiler that does
the same thing?  The package is useful for a variety of purposes.  It
can be modified more easily than the built-in compiler.  The AST it
generates is useful for analyzing Python source code.

  This chapter explains how the various components of the
compiler package work.  It blends reference material with a
tutorial.

5.32.1 The basic interface
--------------------------

The top-level of the package defines four functions.  If you import
compiler, you will get these functions and a collection of
modules contained in the package.

 -- Function: compiler.parse (buf)
     Returns an abstract syntax tree for the Python source code in
     _buf_. The function raises SyntaxError if there is an
     error in the source code.  The return value is a
     `compiler.ast.Module' instance that contains the tree.

 -- Function: compiler.parseFile (path)
     Return an abstract syntax tree for the Python source code in the
     file specified by _path_.  It is equivalent to
     `parse(open(path).read())'.

 -- Function: compiler.walk (ast, visitor[, verbose])
     Do a pre-order walk over the abstract syntax tree _ast_.  Call the
     appropriate method on the _visitor_ instance for each node
     encountered.

 -- Function: compiler.compile (source, filename, mode, flags=None,
          dont_inherit=None)
     Compile the string _source_, a Python module, statement or
     expression, into a code object that can be executed by the exec
     statement or eval(). This function is a replacement for
     the built-in compile() function.

     The _filename_ will be used for run-time error messages.

     The _mode_ must be 'exec' to compile a module, 'single' to compile
     a single (interactive) statement, or 'eval' to compile an
     expression.

     The _flags_ and _dont_inherit_ arguments affect future-related
     statements, but are not supported yet.

 -- Function: compiler.compileFile (source)
     Compiles the file _source_ and generates a .pyc file.

  The compiler package contains the following modules:
ast, `consts', `future', `misc', `pyassem', `pycodegen', `symbols',
`transformer', and `visitor'.

5.32.2 Limitations
------------------

There are some problems with the error checking of the compiler
package.  The interpreter detects syntax errors in two distinct phases.
One set of errors is detected by the interpreter's parser, the other
set by the compiler.  The compiler package relies on the interpreter's
parser, so it get the first phases of error checking for free.  It
implements the second phase itself, and that implementation is
incomplete.  For example, the compiler package does not raise an error
if a name appears more than once in an argument list:  `def f(x, x):
...'

  A future version of the compiler should fix these problems.

5.32.3 Python Abstract Syntax
-----------------------------

The compiler.ast module defines an abstract syntax for
Python.  In the abstract syntax tree, each node represents a syntactic
construct.  The root of the tree is `Module' object.

  The abstract syntax offers a higher level interface to parsed Python
source code.  The parser module and the compiler written in
C for the Python interpreter use a concrete syntax tree.  The concrete
syntax is tied closely to the grammar description used for the Python
parser.  Instead of a single node for a construct, there are often
several levels of nested nodes that are introduced by Python's
precedence rules.

  The abstract syntax tree is created by the `compiler.transformer'
module.  The transformer relies on the built-in Python parser to
generate a concrete syntax tree.  It generates an abstract syntax tree
from the concrete tree.

  The `transformer' module was created by Greg Stein and Bill Tutt for
an experimental Python-to-C compiler.  The current version contains a
number of modifications and improvements, but the basic form of the
abstract syntax and of the transformer are due to Stein and Tutt.

5.32.3.1 AST Nodes
..................

The compiler.ast module is generated from a text file that
describes each node type and its elements.  Each node type is
represented as a class that inherits from the abstract base class
compiler.ast.Node and defines a set of named attributes for
child nodes.

 -- Class: compiler.ast.Node
     The Node instances are created automatically by the
     parser generator.  The recommended interface for specific
     Node instances is to use the public attributes to access
     child nodes.  A public attribute may be bound to a single node or
     to a sequence of nodes, depending on the Node type.
     For example, the `bases' attribute of the `Class' node, is bound
     to a list of base class nodes, and the `doc' attribute is bound to
     a single node.

     Each Node instance has a `lineno' attribute which may
     be `None'.  XXX Not sure what the rules are for which nodes will
     have a useful lineno.

     All Node objects offer the following methods:

      -- Method: getChildren ()
          Returns a flattened list of the child nodes and objects in
          the order they occur.  Specifically, the order of the nodes
          is the order in which they appear in the Python grammar.  Not
          all of the children are Node.  instances.  The
          names of functions and classes, for example, are plain
          strings.

      -- Method: getChildNodes ()
          Returns a flattened list of the child nodes in the order they
          occur.  This method is like getChildren(), except
          that it only returns those children that are Node
          instances.

  Two examples illustrate the general structure of Node.
classes.  The while statement is defined by the following
grammar production:

    while_stmt:     "while" expression ":" suite
                   ["else" ":" suite]

The `While' node has three attributes: test, `body', and
`else_'.  (If the natural name for an attribute is also a Python
reserved word, it can't be used as an attribute name.  An underscore is
appended to the word to make it a legal identifier, hence `else_'
instead of else)

  The if statement is more complicated because it can
include several tests.

    if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]

The `If' node only defines two attributes: `tests' and `else_'.  The
`tests' attribute is a sequence of test expression, consequent body
pairs.  There is one pair for each if./see elif.
clause.  The first element of the pair is the test expression.  The
second elements is a `Stmt' node that contains the code to execute if
the test is true.

  The `getChildren()' method of `If' returns a flat list of child
nodes.  If there are three if./see elif clauses and
no else clause, then `getChildren()' will return a list of
six elements: the first test expression, the first `Stmt', the second
text expression, etc.

  The following table lists each of the Node subclasses
defined in compiler.ast and each of the public attributes
available on their instances.  The values of most of the attributes are
themselves Node.  instances or sequences of instances.
When the value is something other than an instance, the type is noted
in the comment.  The attributes are listed in the order in which they
are returned by `getChildren()' and `getChildNodes()'.

Node type                   Attribute                Value
-------------------------------------------------------------------------------------------
`Add'                       `left'                   left operand
                            `right'                  right operand
`And'                       `nodes'                  list of operands
`AssAttr'                                            _attribute as target of assignment_
                            `expr'                   expression on the left-hand side of
                                                     the dot
                            `attrname'               the attribute name, a string
                            `flags'                  XXX
`AssList'                   `nodes'                  list of list elements being assigned
                                                     to
`AssName'                   `name'                   name being assigned to
                            `flags'                  XXX
`AssTuple'                  `nodes'                  list of tuple elements being
                                                     assigned to
`Assert'                    test.         the expression to be tested
                            `fail'                   the value of the
                                                     AssertionError.
`Assign'                    `nodes'                  a list of assignment targets, one
                                                     per equal sign
                            `expr'                   the value being assigned
`AugAssign'                 `node'
                            `op'
                            `expr'
`Backquote'                 `expr'
`Bitand'                    `nodes'
`Bitor'                     `nodes'
`Bitxor'                    `nodes'
`Break'
`CallFunc'                  `node'                   expression for the callee
                            `args'                   a list of arguments
                            `star_args'              the extended *-arg value
                            `dstar_args'             the extended **-arg value
`Class'                     `name'                   the name of the class, a string
                            `bases'                  a list of base classes
                            `doc'                    doc string, a string or `None'
                            code.          the body of the class statement
`Compare'                   `expr'
                            `ops'
`Const'                     `value'
`Continue'
`Decorators'                `nodes'                  List of function decorator
                                                     expressions
`Dict'                      `items'
`Discard'                   `expr'
`Div'                       `left'
                            `right'
Ellipsis.
`Expression'                `node'
`Exec'                      `expr'
                            See locals.
                            See globals.
`FloorDiv'                  `left'
                            `right'
`For'                       `assign'
                            See list.
                            `body'
                            `else_'
`From'                      `modname'
                            `names'
`Function'                  `decorators'             `Decorators' or `None'
                            `name'                   name used in def, a string
                            `argnames'               list of argument names, as strings
                            `defaults'               list of default values
                            `flags'                  xxx
                            `doc'                    doc string, a string or `None'
                            code.          the body of the function
`GenExpr'                   code.
`GenExprFor'                `assign'
                            See iter.
                            `ifs'
`GenExprIf'                 test.
`GenExprInner'              `expr'
                            `quals'
`Getattr'                   `expr'
                            `attrname'
`Global'                    `names'
`If'                        `tests'
                            `else_'
`Import'                    `names'
`Invert'                    `expr'
`Keyword'                   `name'
                            `expr'
`Lambda'                    `argnames'
                            `defaults'
                            `flags'
                            code.
`LeftShift'                 `left'
                            `right'
`List'                      `nodes'
`ListComp'                  `expr'
                            `quals'
`ListCompFor'               `assign'
                            See list.
                            `ifs'
`ListCompIf'                test.
`Mod'                       `left'
                            `right'
`Module'                    `doc'                    doc string, a string or `None'
                            `node'                   body of the module, a `Stmt'
`Mul'                       `left'
                            `right'
`Name'                      `name'
`Not'                       `expr'
`Or'                        `nodes'
`Pass'
`Power'                     `left'
                            `right'
`Print'                     `nodes'
                            `dest'
`Printnl'                   `nodes'
                            `dest'
`Raise'                     `expr1'
                            `expr2'
                            `expr3'
`Return'                    `value'
`RightShift'                `left'
                            `right'
`Slice'                     `expr'
                            `flags'
                            `lower'
                            `upper'
`Sliceobj'                  `nodes'                  list of statements
`Stmt'                      `nodes'
`Sub'                       `left'
                            `right'
`Subscript'                 `expr'
                            `flags'
                            `subs'
`TryExcept'                 `body'
                            `handlers'
                            `else_'
`TryFinally'                `body'
                            `final'
`Tuple'                     `nodes'
`UnaryAdd'                  `expr'
`UnarySub'                  `expr'
`While'                     test.
                            `body'
                            `else_'
`With'                      `expr'
                            See vars.
                            `body'
`Yield'                     `value'

5.32.3.2 Assignment nodes
.........................

There is a collection of nodes used to represent assignments.  Each
assignment statement in the source code becomes a single `Assign' node
in the AST.  The `nodes' attribute is a list that contains a node for
each assignment target.  This is necessary because assignment can be
chained, e.g. `a = b = 2'. Each Node in the list will be
one of the following classes: `AssAttr', `AssList', `AssName', or
`AssTuple'.

  Each target assignment node will describe the kind of object being
assigned to: `AssName' for a simple name, e.g. `a = 1'. `AssAttr' for an
attribute assigned, e.g. `a.x = 1'. `AssList' and `AssTuple' for list
and tuple expansion respectively, e.g. `a, b, c = a_tuple'.

  The target assignment nodes also have a `flags' attribute that
indicates whether the node is being used for assignment or in a delete
statement.  The `AssName' is also used to represent a delete statement,
e.g. `del x'.

  When an expression contains several attribute references, an
assignment or delete statement will contain only one `AssAttr' node -
for the final attribute reference.  The other attribute references will
be represented as `Getattr' nodes in the `expr' attribute of the
`AssAttr' instance.

5.32.3.3 Examples
.................

This section shows several simple examples of ASTs for Python source
code.  The examples demonstrate how to use the `parse()' function, what
the repr of an AST looks like, and how to access attributes of an AST
node.

  The first module defines a single function.  Assume it is stored in
`/tmp/doublelib.py'.

    """This is an example module.

    This is the docstring.
    """

    def double(x):
        "Return twice the argument"
        return x * 2

In the interactive interpreter session below, I have reformatted the
long AST reprs for readability.  The AST reprs use unqualified class
names.  If you want to create an instance from a repr, you must import
the class names from the compiler.ast module.

    >>> import compiler
    >>> mod = compiler.parseFile("/tmp/doublelib.py")
    >>> mod
    Module('This is an example module.\n\nThis is the docstring.\n',
           Stmt([Function(None, 'double', ['x'], [], 0,
                          'Return twice the argument',
                          Stmt([Return(Mul((Name('x'), Const(2))))]))]))
    >>> from compiler.ast import *
    >>> Module('This is an example module.\n\nThis is the docstring.\n',
    ...    Stmt([Function(None, 'double', ['x'], [], 0,
    ...                   'Return twice the argument',
    ...                   Stmt([Return(Mul((Name('x'), Const(2))))]))]))
    Module('This is an example module.\n\nThis is the docstring.\n',
           Stmt([Function(None, 'double', ['x'], [], 0,
                          'Return twice the argument',
                          Stmt([Return(Mul((Name('x'), Const(2))))]))]))
    >>> mod.doc
    'This is an example module.\n\nThis is the docstring.\n'
    >>> for node in mod.node.nodes:
    ...     print node
    ...
    Function(None, 'double', ['x'], [], 0, 'Return twice the argument',
             Stmt([Return(Mul((Name('x'), Const(2))))]))
    >>> func = mod.node.nodes[0]
    >>> func.code
    Stmt([Return(Mul((Name('x'), Const(2))))])

5.32.4 Using Visitors to Walk ASTs
----------------------------------

The visitor pattern is ...  The compiler package uses a
variant on the visitor pattern that takes advantage of Python's
introspection features to eliminate the need for much of the visitor's
infrastructure.

  The classes being visited do not need to be programmed to accept
visitors.  The visitor need only define visit methods for classes it is
specifically interested in; a default visit method can handle the rest.

  XXX The magic `visit()' method for visitors.

 -- Function: compiler.visitor.walk (tree, visitor[, verbose])

 -- Class: compiler.visitor.ASTVisitor
     The ASTVisitor is responsible for walking over the
     tree in the correct order.  A walk begins with a call to
     preorder().  For each node, it checks the _visitor_ argument
     to preorder() for a method named 'visitNodeType,'
     where NodeType is the name of the node's class, e.g. for a `While'
     node a `visitWhile()' would be called.  If the method exists, it
     is called with the node as its first argument.

     The visitor method for a particular node type can control how
     child nodes are visited during the walk.  The ASTVisitor
     modifies the visitor argument by adding a visit method to
     the visitor; this method can be used to visit a particular child
     node.  If no visitor is found for a particular node type, the
     default() method is called.

     ASTVisitor objects have the following methods:

     XXX describe extra arguments

      -- Method: default (node[, ...])

      -- Method: dispatch (node[, ...])

      -- Method: preorder (tree, visitor)

5.32.5 Bytecode Generation
--------------------------

The code generator is a visitor that emits bytecodes.  Each visit
method can call the `emit()' method to emit a new bytecode.  The basic
code generator is specialized for modules, classes, and functions.  An
assembler converts that emitted instructions to the low-level bytecode
format.  It handles things like generation of constant lists of code
objects and calculation of jump offsets.

5.33 Miscellaneous Services
===========================

The modules described in this chapter provide miscellaneous services
that are available in all Python versions.  Here's an overview:

5.33.1 `formatter' -- Generic output formatting
-----------------------------------------------

This module supports two interface definitions, each with multiple
implementations.  The _formatter_ interface is used by the
HTMLParser.  class of the htmllib module, and the
_writer_ interface is required by the formatter interface.

  Formatter objects transform an abstract flow of formatting events
into specific output events on writer objects.  Formatters manage
several stack structures to allow various properties of a writer object
to be changed and restored; writers need not be able to handle relative
changes nor any sort of "change back" operation.  Specific writer
properties which may be controlled via formatter objects are horizontal
alignment, font, and left margin indentations.  A mechanism is provided
which supports providing arbitrary, non-exclusive style settings to a
writer as well.  Additional interfaces facilitate formatting events
which are not reversible, such as paragraph separation.

  Writer objects encapsulate device interfaces.  Abstract devices, such
as file formats, are supported as well as physical devices.  The
provided implementations all work with abstract devices.  The interface
makes available mechanisms for setting the properties which formatter
objects manage and inserting data into the output.

5.33.1.1 The Formatter Interface
................................

Interfaces to create formatters are dependent on the specific formatter
class being instantiated.  The interfaces described below are the
required interfaces which all formatters must support once initialized.

  One data element is defined at the module level:

 -- Data: formatter.AS_IS
     Value which can be used in the font specification passed to the
     `push_font()' method described below, or as the new value to any
     other `push_property()' method.  Pushing the `AS_IS' value allows
     the corresponding `pop_property()' method to be called without
     having to track whether the property was changed.

  The following attributes are defined for formatter instance objects:

 -- Attribute: formatter.writer
     The writer instance with which the formatter interacts.

 -- Method: formatter.end_paragraph (blanklines)
     Close any open paragraphs and insert at least _blanklines_ before
     the next paragraph.

 -- Method: formatter.add_line_break ()
     Add a hard line break if one does not already exist.  This does
     not break the logical paragraph.

 -- Method: formatter.add_hor_rule (*args, **kw)
     Insert a horizontal rule in the output.  A hard break is inserted
     if there is data in the current paragraph, but the logical
     paragraph is not broken.  The arguments and keywords are passed on
     to the writer's `send_line_break()' method.

 -- Method: formatter.add_flowing_data (data)
     Provide data which should be formatted with collapsed whitespace.
     Whitespace from preceding and successive calls to
     add_flowing_data() is considered as well when the
     whitespace collapse is performed.  The data which is passed to
     this method is expected to be word-wrapped by the output device.
     Note that any word-wrapping still must be performed by the writer
     object due to the need to rely on device and font information.

 -- Method: formatter.add_literal_data (data)
     Provide data which should be passed to the writer unchanged.
     Whitespace, including newline and tab characters, are considered
     legal in the value of _data_.

 -- Method: formatter.add_label_data (format, counter)
     Insert a label which should be placed to the left of the current
     left margin.  This should be used for constructing bulleted or
     numbered lists.  If the _format_ value is a string, it is
     interpreted as a format specification for _counter_, which should
     be an integer. The result of this formatting becomes the value of
     the label; if _format_ is not a string it is used as the label
     value directly. The label value is passed as the only argument to
     the writer's `send_label_data()' method.  Interpretation of
     non-string label values is dependent on the associated writer.

     Format specifications are strings which, in combination with a
     counter value, are used to compute label values.  Each character
     in the format string is copied to the label value, with some
     characters recognized to indicate a transform on the counter
     value.  Specifically, the character `'1'' represents the counter
     value formatter as an Arabic number, the characters `'A'' and `'a''
     represent alphabetic representations of the counter value in upper
     and lower case, respectively, and `'I'' and `'i'' represent the
     counter value in Roman numerals, in upper and lower case.  Note
     that the alphabetic and roman transforms require that the counter
     value be greater than zero.

 -- Method: formatter.flush_softspace ()
     Send any pending whitespace buffered from a previous call to
     add_flowing_data() to the associated writer object.  This
     should be called before any direct manipulation of the writer
     object.

 -- Method: formatter.push_alignment (align)
     Push a new alignment setting onto the alignment stack.  This may be
     AS_IS if no change is desired.  If the alignment
     value is changed from the previous setting, the writer's
     `new_alignment()' method is called with the _align_ value.

 -- Method: formatter.pop_alignment ()
     Restore the previous alignment.

 -- Method: formatter.push_font ((size, italic, bold, teletype))
     Change some or all font properties of the writer object.
     Properties which are not set to AS_IS are set to the
     values passed in while others are maintained at their current
     settings.  The writer's `new_font()' method is called with the
     fully resolved font specification.

 -- Method: formatter.pop_font ()
     Restore the previous font.

 -- Method: formatter.push_margin (margin)
     Increase the number of left margin indentations by one,
     associating the logical tag _margin_ with the new indentation.
     The initial margin level is `0'.  Changed values of the logical
     tag must be true values; false values other than AS_IS
     are not sufficient to change the margin.

 -- Method: formatter.pop_margin ()
     Restore the previous margin.

 -- Method: formatter.push_style (*styles)
     Push any number of arbitrary style specifications.  All styles are
     pushed onto the styles stack in order.  A tuple representing the
     entire stack, including AS_IS values, is passed to
     the writer's `new_styles()' method.

 -- Method: formatter.pop_style ([n=1])
     Pop the last _n_ style specifications passed to
     push_style().  A tuple representing the revised stack,
     including AS_IS values, is passed to the writer's
     `new_styles()' method.

 -- Method: formatter.set_spacing (spacing)
     Set the spacing style for the writer.

 -- Method: formatter.assert_line_data ([flag=1])
     Inform the formatter that data has been added to the current
     paragraph out-of-band.  This should be used when the writer has
     been manipulated directly.  The optional _flag_ argument can be
     set to false if the writer manipulations produced a hard line
     break at the end of the output.

5.33.1.2 Formatter Implementations
..................................

Two implementations of formatter objects are provided by this module.
Most applications may use one of these classes without modification or
subclassing.

 -- Class: formatter.NullFormatter ([writer])
     A formatter which does nothing.  If _writer_ is omitted, a
     NullWriter.  instance is created.  No methods of the writer
     are called by NullFormatter instances.
     Implementations should inherit from this class if implementing a
     writer interface but don't need to inherit any implementation.

 -- Class: formatter.AbstractFormatter (writer)
     The standard formatter.  This implementation has demonstrated wide
     applicability to many writers, and may be used directly in most
     circumstances.  It has been used to implement a full-featured
     World Wide Web browser.

5.33.1.3 The Writer Interface
.............................

Interfaces to create writers are dependent on the specific writer class
being instantiated.  The interfaces described below are the required
interfaces which all writers must support once initialized. Note that
while most applications can use the AbstractFormatter.
class as a formatter, the writer must typically be provided by the
application.

 -- Method: writer.flush ()
     Flush any buffered output or device control events.

 -- Method: writer.new_alignment (align)
     Set the alignment style.  The _align_ value can be any object, but
     by convention is a string or `None', where `None' indicates that
     the writer's "preferred" alignment should be used. Conventional
     _align_ values are `'left'', `'center'', `'right'', and
     `'justify''.

 -- Method: writer.new_font (font)
     Set the font style.  The value of _font_ will be `None',
     indicating that the device's default font should be used, or a
     tuple of the form `(size, italic, bold, teletype)'.  Size will be
     a string indicating the size of font that should be used; specific
     strings and their interpretation must be defined by the
     application.  The _italic_, _bold_, and _teletype_ values are
     Boolean values specifying which of those font attributes should be
     used.

 -- Method: writer.new_margin (margin, level)
     Set the margin level to the integer _level_ and the logical tag to
     _margin_.  Interpretation of the logical tag is at the writer's
     discretion; the only restriction on the value of the logical tag
     is that it not be a false value for non-zero values of _level_.

 -- Method: writer.new_spacing (spacing)
     Set the spacing style to _spacing_.

 -- Method: writer.new_styles (styles)
     Set additional styles.  The _styles_ value is a tuple of arbitrary
     values; the value AS_IS should be ignored.  The
     _styles_ tuple may be interpreted either as a set or as a stack
     depending on the requirements of the application and writer
     implementation.

 -- Method: writer.send_line_break ()
     Break the current line.

 -- Method: writer.send_paragraph (blankline)
     Produce a paragraph separation of at least _blankline_ blank
     lines, or the equivalent.  The _blankline_ value will be an
     integer.  Note that the implementation will receive a call to
     send_line_break() before this call if a line break is
     needed;  this method should not include ending the last line of
     the paragraph. It is only responsible for vertical spacing between
     paragraphs.

 -- Method: writer.send_hor_rule (*args, **kw)
     Display a horizontal rule on the output device.  The arguments to
     this method are entirely application- and writer-specific, and
     should be interpreted with care.  The method implementation may
     assume that a line break has already been issued via
     send_line_break().

 -- Method: writer.send_flowing_data (data)
     Output character data which may be word-wrapped and re-flowed as
     needed.  Within any sequence of calls to this method, the writer
     may assume that spans of multiple whitespace characters have been
     collapsed to single space characters.

 -- Method: writer.send_literal_data (data)
     Output character data which has already been formatted for
     display.  Generally, this should be interpreted to mean that line
     breaks indicated by newline characters should be preserved and no
     new line breaks should be introduced.  The data may contain
     embedded newline and tab characters, unlike data provided to the
     `send_formatted_data()' interface.

 -- Method: writer.send_label_data (data)
     Set _data_ to the left of the current left margin, if possible.
     The value of _data_ is not restricted; treatment of non-string
     values is entirely application- and writer-dependent.  This method
     will only be called at the beginning of a line.

5.33.1.4 Writer Implementations
...............................

Three implementations of the writer object interface are provided as
examples by this module.  Most applications will need to derive new
writer classes from the NullWriter class.

 -- Class: formatter.NullWriter
     A writer which only provides the interface definition; no actions
     are taken on any methods.  This should be the base class for all
     writers which do not need to inherit any implementation methods.

 -- Class: formatter.AbstractWriter
     A writer which can be used in debugging formatters, but not much
     else.  Each method simply announces itself by printing its name
     and arguments on standard output.

 -- Class: formatter.DumbWriter ([file[, maxcol=72]])
     Simple writer class which writes output on the file object passed
     in as _file_ or, if _file_ is omitted, on standard output.  The
     output is simply word-wrapped to the number of columns specified
     by _maxcol_.  This class is suitable for reflowing a sequence of
     paragraphs.

5.34 MS Windows Specific Services
=================================

This chapter describes modules that are only available on MS Windows
platforms.

5.34.1 `msilib' -- Read and write Microsoft Installer files
-----------------------------------------------------------

New in version 2.5.

  The msilib supports the creation of Microsoft Installer
(`.msi') files.  Because these files often contain an embedded
"cabinet" file (`.cab'), it also exposes an API to create CAB files.
Support for reading `.cab' files is currently not implemented; read
support for the `.msi' database is possible.

  This package aims to provide complete access to all tables in an
`.msi' file, therefore, it is a fairly low-level API. Two primary
applications of this package are the distutils command
`bdist_msi', and the creation of Python installer package itself
(although that currently uses a different version of `msilib').

  The package contents can be roughly split into four parts: low-level
CAB routines, low-level MSI routines, higher-level MSI routines, and
standard table structures.

 -- Function: msilib.FCICreate (cabname, files)
     Create a new CAB file named _cabname_. _files_ must be a list of
     tuples, each containing the name of the file on disk, and the name
     of the file inside the CAB file.

     The files are added to the CAB file in the order they appear in
     the list. All files are added into a single CAB file, using the
     MSZIP compression algorithm.

     Callbacks to Python for the various steps of MSI creation are
     currently not exposed.

 -- Function: msilib.UuidCreate ()
     Return the string representation of a new unique identifier. This
     wraps the Windows API functions `UuidCreate()' and
     `UuidToString()'.

 -- Function: msilib.OpenDatabase (path, persist)
     Return a new database object by calling MsiOpenDatabase.   _path_
     is the file name of the MSI file; _persist_ can be one of the
     constants `MSIDBOPEN_CREATEDIRECT', `MSIDBOPEN_CREATE',
     `MSIDBOPEN_DIRECT', `MSIDBOPEN_READONLY', or `MSIDBOPEN_TRANSACT',
     and may include the flag `MSIDBOPEN_PATCHFILE'. See the Microsoft
     documentation for the meaning of these flags; depending on the
     flags, an existing database is opened, or a new one created.

 -- Function: msilib.CreateRecord (count)
     Return a new record object by calling `MSICreateRecord()'. _count_
     is the number of fields of the record.

 -- Function: msilib.init_database (name, schema, ProductName,
          ProductCode, ProductVersion, Manufacturer)
     Create and return a new database _name_, initialize it with
     _schema_, and set the properties _ProductName_, _ProductCode_,
     _ProductVersion_, and _Manufacturer_.

     _schema_ must be a module object containing `tables' and
     `_Validation_records' attributes; typically, see msilib.schema
     should be used.

     The database will contain just the schema and the validation
     records when this function returns.

 -- Function: msilib.add_data (database, table, records)
     Add all _records_ to the table named _table_ in _database_.

     The _table_ argument must be one of the predefined tables in the
     MSI schema, e.g. `'Feature'', `'File'', `'Component'', `'Dialog'',
     `'Control'', etc.

     _records_ should be a list of tuples, each one containing all
     fields of a record according to the schema of the table.  For
     optional fields, `None' can be passed.

     Field values can be int or long numbers, strings, or instances of
     the Binary class.

 -- Class: msilib.Binary (filename)
     Represents entries in the Binary table; inserting such an object
     using add_data() reads the file named _filename_ into
     the table.

 -- Function: msilib.add_tables (database, module)
     Add all table content from _module_ to _database_. _module_ must
     contain an attribute _tables_ listing all tables for which content
     should be added, and one attribute per table that has the actual
     content.

     This is typically used to install the sequence tables.

 -- Function: msilib.add_stream (database, name, path)
     Add the file _path_ into the `_Stream' table of _database_, with
     the stream name _name_.

 -- Function: msilib.gen_uuid ()
     Return a new UUID, in the format that MSI typically requires (i.e.
     in curly braces, and with all hexdigits in upper-case).

  FCICreateFile
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/devnotes/winprog/fcicreate.asp)UuidCreate
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/rpc/rpc/uuidcreate.asp)UuidToString
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/rpc/rpc/uuidtostring.asp)

5.34.1.1 Database Objects
.........................

 -- Method: Database.OpenView (sql)
     Return a view object, by calling `MSIDatabaseOpenView()'. _sql_ is
     the SQL statement to execute.

 -- Method: Database.Commit ()
     Commit the changes pending in the current transaction, by calling
     `MSIDatabaseCommit()'.

 -- Method: Database.GetSummaryInformation (count)
     Return a new summary information object, by calling
     `MsiGetSummaryInformation()'.  _count_ is the maximum number of
     updated values.

  MSIDatabaseOpenView
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msidatabaseopenview.asp)MSIDatabaseCommit
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msidatabasecommit.asp)MSIGetSummaryInformation
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msigetsummaryinformation.asp)

5.34.1.2 View Objects
.....................

 -- Method: View.Execute (params)
     Execute the SQL query of the view, through `MSIViewExecute()'. If
     _params_ is not `None', it is a record describing actual values of
     the parameter tokens in the query.

 -- Method: View.GetColumnInfo (kind)
     Return a record describing the columns of the view, through calling
     `MsiViewGetColumnInfo()'. _kind_ can be either `MSICOLINFO_NAMES'
     or `MSICOLINFO_TYPES'.

 -- Method: View.Fetch ()
     Return a result record of the query, through calling
     `MsiViewFetch()'.

 -- Method: View.Modify (kind, data)
     Modify the view, by calling `MsiViewModify()'. _kind_ can be one of
     `MSIMODIFY_SEEK', `MSIMODIFY_REFRESH', `MSIMODIFY_INSERT',
     `MSIMODIFY_UPDATE', `MSIMODIFY_ASSIGN', `MSIMODIFY_REPLACE',
     `MSIMODIFY_MERGE', `MSIMODIFY_DELETE',
     `MSIMODIFY_INSERT_TEMPORARY', `MSIMODIFY_VALIDATE',
     `MSIMODIFY_VALIDATE_NEW', `MSIMODIFY_VALIDATE_FIELD', or
     `MSIMODIFY_VALIDATE_DELETE'.

     _data_ must be a record describing the new data.

 -- Method: View.Close ()
     Close the view, through `MsiViewClose()'.

  MsiViewExecute
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msiviewexecute.asp)MSIViewGetColumnInfo
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msiviewgetcolumninfo.asp)MsiViewFetch
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msiviewfetch.asp)MsiViewModify
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msiviewmodify.asp)MsiViewClose
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msiviewclose.asp)

5.34.1.3 Summary Information Objects
....................................

 -- Method: SummaryInformation.GetProperty (field)
     Return a property of the summary, through
     `MsiSummaryInfoGetProperty()'.  _field_ is the name of the
     property, and can be one of the constants `PID_CODEPAGE',
     `PID_TITLE', `PID_SUBJECT', `PID_AUTHOR', `PID_KEYWORDS',
     `PID_COMMENTS', `PID_TEMPLATE', `PID_LASTAUTHOR', `PID_REVNUMBER',
     `PID_LASTPRINTED', `PID_CREATE_DTM', `PID_LASTSAVE_DTM',
     `PID_PAGECOUNT', `PID_WORDCOUNT', `PID_CHARCOUNT', `PID_APPNAME',
     or `PID_SECURITY'.

 -- Method: SummaryInformation.GetPropertyCount ()
     Return the number of summary properties, through
     `MsiSummaryInfoGetPropertyCount()'.

 -- Method: SummaryInformation.SetProperty (field, value)
     Set a property through `MsiSummaryInfoSetProperty()'. _field_ can
     have the same values as in GetProperty(), _value_ is
     the new value of the property.  Possible value types are integer
     and string.

 -- Method: SummaryInformation.Persist ()
     Write the modified properties to the summary information stream,
     using `MsiSummaryInfoPersist()'.

  MsiSummaryInfoGetProperty
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msisummaryinfogetproperty.asp)MsiSummaryInfoGetPropertyCount
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msisummaryinfogetpropertycount.asp)MsiSummaryInfoSetProperty
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msisummaryinfosetproperty.asp)MsiSummaryInfoPersist
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msisummaryinfopersist.asp)

5.34.1.4 Record Objects
.......................

 -- Method: Record.GetFieldCount ()
     Return the number of fields of the record, through
     `MsiRecordGetFieldCount()'.

 -- Method: Record.GetInteger (field)
     Return the value of _field_ as an integer where possible.  _field_
     must be an integer.

 -- Method: Record.GetString (field)
     Return the value of _field_ as a string where possible.  _field_
     must be an integer.

 -- Method: Record.SetString (field, value)
     Set _field_ to _value_ through `MsiRecordSetString()'. _field_
     must be an integer; _value_ a string.

 -- Method: Record.SetStream (field, value)
     Set _field_ to the contents of the file named _value_, through
     `MsiRecordSetStream()'. _field_ must be an integer; _value_ a
     string.

 -- Method: Record.SetInteger (field, value)
     Set _field_ to _value_ through `MsiRecordSetInteger()'. Both
     _field_ and _value_ must be an integer.

 -- Method: Record.ClearData ()
     Set all fields of the record to 0, through `MsiRecordClearData()'.

  MsiRecordGetFieldCount
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msirecordgetfieldcount.asp)MsiRecordSetString
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msirecordsetstring.asp)MsiRecordSetStream
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msirecordsetstream.asp)MsiRecordSetInteger
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msirecordsetinteger.asp)MsiRecordClear
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/msirecordclear.asp)

5.34.1.5 Errors
...............

All wrappers around MSI functions raise `MsiError'; the string inside
the exception will contain more detail.

5.34.1.6 CAB Objects
....................

 -- Class: msilib.CAB (name)
     The class CAB represents a CAB file. During MSI
     construction, files will be added simultaneously to the `Files'
     table, and to a CAB file. Then, when all files have been added,
     the CAB file can be written, then added to the MSI file.

     _name_ is the name of the CAB file in the MSI file.

      -- Method: append (full, file, logical)
          Add the file with the pathname _full_ to the CAB file, under
          the name _logical_.  If there is already a file named
          _logical_, a new file name is created.

          Return the index of the file in the CAB file, and the new
          name of the file inside the CAB file.

      -- Method: commit (database)
          Generate a CAB file, add it as a stream to the MSI file, put
          it into the `Media' table, and remove the generated file from
          the disk.

5.34.1.7 Directory Objects
..........................

 -- Class: msilib.Directory (database, cab, basedir, physical, logical,
          default, component[, componentflags])
     Create a new directory in the Directory table. There is a current
     component at each point in time for the directory, which is either
     explicitly created through start_component(), or
     implicitly when files are added for the first time.  Files are
     added into the current component, and into the cab file.  To
     create a directory, a base directory object needs to be specified
     (can be `None'), the path to the physical directory, and a logical
     directory name.  _default_ specifies the DefaultDir slot in the
     directory table. _componentflags_ specifies the default flags that
     new components get.

      -- Method: start_component ([component[, feature[, flags[,
               keyfile[, uuid]]]]])
          Add an entry to the Component table, and make this component
          the current component for this directory. If no component
          name is given, the directory name is used. If no _feature_ is
          given, the current feature is used. If no _flags_ are given,
          the directory's default flags are used. If no _keyfile_ is
          given, the KeyPath is left null in the Component table.

      -- Method: add_file (file[, src[, version[, language]]])
          Add a file to the current component of the directory,
          starting a new one if there is no current component. By
          default, the file name in the source and the file table will
          be identical. If the _src_ file is specified, it is
          interpreted relative to the current directory. Optionally, a
          _version_ and a _language_ can be specified for the entry in
          the File table.

      -- Method: glob (pattern[, exclude])
          Add a list of files to the current component as specified in
          the glob pattern.  Individual files can be excluded in the
          _exclude_ list.

      -- Method: remove_pyc ()
          Remove `.pyc'/`.pyo' files on uninstall.

Directory Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/directory_table.asp)File
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/file_table.asp)Component
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/component_table.asp)FeatureComponents
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/featurecomponents_table.asp)

5.34.1.8 Features
.................

 -- Class: msilib.Feature (database, id, title, desc, display[,
          level=1[, parent[, directory[, attributes=0]]]])
     Add a new record to the `Feature' table, using the values _id_,
     _parent.id_, _title_, _desc_, _display_, _level_, _directory_, and
     _attributes_. The resulting feature object can be passed to the
     `start_component()' method of Directory.

      -- Method: set_current ()
          Make this feature the current feature of msilib.
          New components are automatically added to the default
          feature, unless a feature is explicitly specified.

Feature Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/feature_table.asp)

5.34.1.9 GUI classes
....................

msilib provides several classes that wrap the GUI tables in
an MSI database. However, no standard user interface is provided; use
`bdist_msi' to create MSI files with a user-interface for installing
Python packages.

 -- Class: msilib.Control (dlg, name)
     Base class of the dialog controls. _dlg_ is the dialog object the
     control belongs to, and _name_ is the control's name.

      -- Method: event (event, argument[, condition=1[, ordering]])
          Make an entry into the `ControlEvent' table for this control.

      -- Method: mapping (event, attribute)
          Make an entry into the `EventMapping' table for this control.

      -- Method: condition (action, condition)
          Make an entry into the `ControlCondition' table for this
          control.

 -- Class: msilib.RadioButtonGroup (dlg, name, property)
     Create a radio button control named _name_. _property_ is the
     installer property that gets set when a radio button is selected.

      -- Method: add (name, x, y, width, height, text[, value])
          Add a radio button named _name_ to the group, at the
          coordinates _x_, _y_, _width_, _height_, and with the label
          _text_. If _value_ is omitted, it defaults to _name_.

 -- Class: msilib.Dialog (db, name, x, y, w, h, attr, title, first,
          default, cancel)
     Return a new Dialog object. An entry in the `Dialog'
     table is made, with the specified coordinates, dialog attributes,
     title, name of the first, default, and cancel controls.

      -- Method: control (name, type, x, y, width, height, attributes,
               property, text, control_next, help)
          Return a new Control object. An entry in the
          `Control' table is made with the specified parameters.

          This is a generic method; for specific types, specialized
          methods are provided.

      -- Method: text (name, x, y, width, height, attributes, text)
          Add and return a `Text' control.

      -- Method: bitmap (name, x, y, width, height, text)
          Add and return a `Bitmap' control.

      -- Method: line (name, x, y, width, height)
          Add and return a `Line' control.

      -- Method: pushbutton (name, x, y, width, height, attributes,
               text, next_control)
          Add and return a `PushButton' control.

      -- Method: radiogroup (name, x, y, width, height, attributes,
               property, text, next_control)
          Add and return a `RadioButtonGroup' control.

      -- Method: checkbox (name, x, y, width, height, attributes,
               property, text, next_control)
          Add and return a `CheckBox' control.

  Dialog Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/dialog_table.asp)Control
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/control_table.asp)Control
Types
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/controls.asp)ControlCondition
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/controlcondition_table.asp)ControlEvent
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/controlevent_table.asp)EventMapping
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/eventmapping_table.asp)RadioButton
Table
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/radiobutton_table.asp)

5.34.1.10 Precomputed tables
............................

msilib provides a few subpackages that contain only schema
and table definitions. Currently, these definitions are based on MSI
version 2.0.

 -- Data: msilib.schema
     This is the standard MSI schema for MSI 2.0, with the _tables_
     variable providing a list of table definitions, and
     __Validation_records_ providing the data for MSI validation.

 -- Data: msilib.sequence
     This module contains table contents for the standard sequence
     tables: _AdminExecuteSequence_, _AdminUISequence_,
     _AdvtExecuteSequence_, _InstallExecuteSequence_, and
     _InstallUISequence_.

 -- Data: msilib.text
     This module contains definitions for the UIText and ActionText
     tables, for the standard installer actions.

5.34.2 `msvcrt' - Useful routines from the MS VC++ runtime
----------------------------------------------------------

These functions provide access to some useful capabilities on Windows
platforms.  Some higher-level modules use these functions to build the
Windows implementations of their services.  For example, the
getpass module uses this in the implementation of the
getpass() function.

  Further documentation on these functions can be found in the Platform
API documentation.

  The module implements both the normal and wide char variants of the
console I/O api. The normal API deals only with ASCII characters and is
of limited use for internationalized applications. The wide char API
should be used where ever possible

5.34.2.1 File Operations
........................

 -- Function: msvcrt.locking (fd, mode, nbytes)
     Lock part of a file based on file descriptor _fd_ from the C
     runtime.  Raises IOError on failure.  The locked
     region of the file extends from the current file position for
     _nbytes_ bytes, and may continue beyond the end of the file.
     _mode_ must be one of the `LK_*' constants listed below. Multiple
     regions in a file may be locked at the same time, but may not
     overlap.  Adjacent regions are not merged; they must be unlocked
     individually.

 -- Data: msvcrt.LK_LOCK
 -- Data: msvcrt.LK_RLCK
     Locks the specified bytes. If the bytes cannot be locked, the
     program immediately tries again after 1 second.  If, after 10
     attempts, the bytes cannot be locked, see IOError is
     raised.

 -- Data: msvcrt.LK_NBLCK
 -- Data: msvcrt.LK_NBRLCK
     Locks the specified bytes. If the bytes cannot be locked, see
     IOError is raised.

 -- Data: msvcrt.LK_UNLCK
     Unlocks the specified bytes, which must have been previously
     locked.

 -- Function: msvcrt.setmode (fd, flags)
     Set the line-end translation mode for the file descriptor _fd_. To
     set it to text mode, _flags_ should be os.O_TEXT.; for
     binary, it should be os.O_BINARY.

 -- Function: msvcrt.open_osfhandle (handle, flags)
     Create a C runtime file descriptor from the file handle _handle_.
     The _flags_ parameter should be a bitwise OR of os.O_APPEND,
     see os.O_RDONLY, and os.O_TEXT.  The
     returned file descriptor may be used as a parameter to
     os.fdopen() to create a file object.

 -- Function: msvcrt.get_osfhandle (fd)
     Return the file handle for the file descriptor _fd_.  Raises
     IOError if _fd_ is not recognized.

5.34.2.2 Console I/O
....................

 -- Function: msvcrt.kbhit ()
     Return true if a keypress is waiting to be read.

 -- Function: msvcrt.getch ()
     Read a keypress and return the resulting character.  Nothing is
     echoed to the console.  This call will block if a keypress is not
     already available, but will not wait for `Enter' to be pressed. If
     the pressed key was a special function key, this will return
     `'\000'' or `'\xe0''; the next call will return the keycode.  The
     `Control-C' keypress cannot be read with this function.

 -- Function: msvcrt.getwch ()
     Wide char variant of getch(), returning a Unicode
     value.

     New in version 2.6.

 -- Function: msvcrt.getche ()
     Similar to getch(), but the keypress will be echoed if
     it  represents a printable character.

 -- Function: msvcrt.getwche ()
     Wide char variant of getche(), returning a Unicode
     value.

     New in version 2.6.

 -- Function: msvcrt.putch (char)
     Print the character _char_ to the console without buffering.

 -- Function: msvcrt.putwch (unicode_char)
     Wide char variant of putch(), accepting a Unicode
     value.

     New in version 2.6.

 -- Function: msvcrt.ungetch (char)
     Cause the character _char_ to be "pushed back" into the console
     buffer; it will be the next character read by getch().
     or getche().

 -- Function: msvcrt.ungetwch (unicode_char)
     Wide char variant of ungetch(), accepting a Unicode
     value.

     New in version 2.6.

5.34.2.3 Other Functions
........................

 -- Function: msvcrt.heapmin ()
     Force the `malloc()' heap to clean itself up and return unused
     blocks to the operating system.  On failure, this raises
     IOError.

5.34.3 `_winreg' - Windows registry access
------------------------------------------

     Note: The _winreg module has been renamed to `winreg' in
     Python 3.0.  The 2to3 tool will automatically adapt
     imports when converting your sources to 3.0.

New in version 2.0.

  These functions expose the Windows registry API to Python.  Instead
of using an integer as the registry handle, a handle object
is used to ensure that the handles are closed correctly, even if
the programmer neglects to explicitly close them.

  This module offers the following functions:

 -- Function: _winreg.CloseKey (hkey)
     Closes a previously opened registry key.  The _hkey_ argument
     specifies a previously opened key.

          Note: If _hkey_ is not closed using this method (or via
          hkey.Close()), it is closed when the _hkey_ object is
          destroyed by Python.

 -- Function: _winreg.ConnectRegistry (computer_name, key)
     Establishes a connection to a predefined registry handle on
     another computer, and returns a handle object.

     _computer_name_ is the name of the remote computer, of the form
     `r"\\computername"'.  If `None', the local computer is used.

     _key_ is the predefined handle to connect to.

     The return value is the handle of the opened key. If the function
     fails, a WindowsError exception is raised.

 -- Function: _winreg.CreateKey (key, sub_key)
     Creates or opens the specified key, returning a handle object.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that names the key this method opens or
     creates.

     If _key_ is one of the predefined keys, _sub_key_ may be `None'.
     In that case, the handle returned is the same key handle passed in
     to the function.

     If the key already exists, this function opens the existing key.

     The return value is the handle of the opened key. If the function
     fails, a WindowsError exception is raised.

 -- Function: _winreg.CreateKeyEx (key, sub_key[, res[, sam]])
     Creates or opens the specified key, returning a handle object.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that names the key this method opens or
     creates.

     _res_ is a reserved integer, and must be zero. The default is zero.

     _sam_ is an integer that specifies an access mask that describes
     the desired security access for the key.  Default is
     KEY_ALL_ACCESS.  See Access Rights for other
     allowed values.

     If _key_ is one of the predefined keys, _sub_key_ may be `None'.
     In that case, the handle returned is the same key handle passed in
     to the function.

     If the key already exists, this function opens the existing key.

     The return value is the handle of the opened key. If the function
     fails, a WindowsError exception is raised.

  New in version 2.7.

 -- Function: _winreg.DeleteKey (key, sub_key)
     Deletes the specified key.

     _key_ is an already open key, or any one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that must be a subkey of the key identified
     by the _key_ parameter.  This value must not be `None', and the
     key may not have subkeys.

     _This method can not delete keys with subkeys._

     If the method succeeds, the entire key, including all of its
     values, is removed.  If the method fails, a WindowsError
     exception is raised.

 -- Function: _winreg.DeleteKeyEx (key, sub_key[, sam[, res]])
     Deletes the specified key.

          Note: The DeleteKeyEx() function is implemented
          with the RegDeleteKeyEx Windows API function, which is
          specific to 64-bit versions of Windows.  See the
          RegDeleteKeyEx documentation
          (http://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx).

     _key_ is an already open key, or any one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that must be a subkey of the key identified
     by the _key_ parameter. This value must not be `None', and the key
     may not have subkeys.

     _res_ is a reserved integer, and must be zero. The default is zero.

     _sam_ is an integer that specifies an access mask that describes
     the desired security access for the key.  Default is
     KEY_WOW64_64KEY.  See Access Rights for other
     allowed values.

     _This method can not delete keys with subkeys._

     If the method succeeds, the entire key, including all of its
     values, is removed. If the method fails, a WindowsError
     exception is raised.

     On unsupported Windows versions, see NotImplementedError.
     is raised.

  New in version 2.7.

 -- Function: _winreg.DeleteValue (key, value)
     Removes a named value from a registry key.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _value_ is a string that identifies the value to remove.

 -- Function: _winreg.EnumKey (key, index)
     Enumerates subkeys of an open registry key, returning a string.

     _key_ is an already open key, or any one of the predefined
     HKEY_* constants.

     _index_ is an integer that identifies the index of the key to
     retrieve.

     The function retrieves the name of one subkey each time it is
     called.  It is typically called repeatedly until a
     WindowsError exception is raised, indicating, no more values
     are available.

 -- Function: _winreg.EnumValue (key, index)
     Enumerates values of an open registry key, returning a tuple.

     _key_ is an already open key, or any one of the predefined
     HKEY_* constants.

     _index_ is an integer that identifies the index of the value to
     retrieve.

     The function retrieves the name of one subkey each time it is
     called. It is typically called repeatedly, until a
     WindowsError exception is raised, indicating no more values.

     The result is a tuple of 3 items:

     Index       Meaning
     -------------------------------------------------------------
     `0'         A string that identifies the value name
     `1'         An object that holds the value data, and whose
                 type depends on the underlying registry type
     `2'         An integer that identifies the type of the
                 value data (see table in docs for
                 SetValueEx())

 -- Function: _winreg.ExpandEnvironmentStrings (unicode)
     Expands environment variable placeholders `%NAME%' in unicode
     strings like REG_EXPAND_SZ.:

         >>> ExpandEnvironmentStrings(u"%windir%")
         u"C:\\Windows"

     New in version 2.6.

 -- Function: _winreg.FlushKey (key)
     Writes all the attributes of a key to the registry.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     It is not necessary to call FlushKey() to change a
     key. Registry changes are flushed to disk by the registry using
     its lazy flusher.  Registry changes are also flushed to disk at
     system shutdown.  Unlike CloseKey(), the
     FlushKey() method returns only when all the data has been
     written to the registry. An application should only call
     FlushKey() if it requires absolute certainty that registry
     changes are on disk.

          Note: If you don't know whether a FlushKey().
          call is required, it probably isn't.

 -- Function: _winreg.LoadKey (key, sub_key, file_name)
     Creates a subkey under the specified key and stores registration
     information from a specified file into that subkey.

     _key_ is a handle returned by ConnectRegistry() or
     one of the constants HKEY_USERS or
     HKEY_LOCAL_MACHINE.

     _sub_key_ is a string that identifies the subkey to load.

     _file_name_ is the name of the file to load registry data from.
     This file must have been created with the SaveKey().
     function. Under the file allocation table (FAT) file system, the
     filename may not have an extension.

     A call to LoadKey() fails if the calling process does
     not have the `SE_RESTORE_PRIVILEGE' privilege.  Note that
     privileges are different from permissions - see the RegLoadKey
     documentation
     (http://msdn.microsoft.com/en-us/library/ms724889%28v=VS.85%29.aspx)
     for more details.

     If _key_ is a handle returned by ConnectRegistry(),
     then the path specified in _file_name_ is relative to the remote
     computer.

 -- Function: _winreg.OpenKey (key, sub_key[, res[, sam]])
     Opens the specified key, returning a handle object.

     _key_ is an already open key, or any one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that identifies the sub_key to open.

     _res_ is a reserved integer, and must be zero.  The default is
     zero.

     _sam_ is an integer that specifies an access mask that describes
     the desired security access for the key.  Default is
     KEY_READ.  See Access Rights for other allowed
     values.

     The result is a new handle to the specified key.

     If the function fails, see WindowsError is raised.

 -- Function: _winreg.OpenKeyEx ()
     The functionality of OpenKeyEx() is provided via
     OpenKey(), by the use of default arguments.

 -- Function: _winreg.QueryInfoKey (key)
     Returns information about a key, as a tuple.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     The result is a tuple of 3 items:

     Index       Meaning
     --------------------------------------------------------------
     `0'         An integer giving the number of sub keys this
                 key has.
     `1'         An integer giving the number of values this key
                 has.
     `2'         A long integer giving when the key was last
                 modified (if available) as 100's of nanoseconds
                 since Jan 1, 1600.

 -- Function: _winreg.QueryValue (key, sub_key)
     Retrieves the unnamed value for a key, as a string.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that holds the name of the subkey with which
     the value is associated.  If this parameter is `None' or empty,
     the function retrieves the value set by the SetValue()
     method for the key identified by _key_.

     Values in the registry have name, type, and data components. This
     method retrieves the data for a key's first value that has a NULL
     name. But the underlying API call doesn't return the type, so
     always use QueryValueEx() if possible.

 -- Function: _winreg.QueryValueEx (key, value_name)
     Retrieves the type and data for a specified value name associated
     with an open registry key.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _value_name_ is a string indicating the value to query.

     The result is a tuple of 2 items:

     Index       Meaning
     ----------------------------------------------------------
     `0'         The value of the registry item.
     `1'         An integer giving the registry type for this
                 value (see table in docs for
                 SetValueEx())

 -- Function: _winreg.SaveKey (key, file_name)
     Saves the specified key, and all its subkeys to the specified file.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _file_name_ is the name of the file to save registry data to.
     This file cannot already exist. If this filename includes an
     extension, it cannot be used on file allocation table (FAT) file
     systems by the LoadKey().  method.

     If _key_ represents a key on a remote computer, the path described
     by _file_name_ is relative to the remote computer. The caller of
     this method must possess the `SeBackupPrivilege' security
     privilege.  Note that privileges are different than permissions -
     see the Conflicts Between User Rights and Permissions
     documentation
     (http://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx)
     for more details.

     This function passes NULL for _security_attributes_ to the API.

 -- Function: _winreg.SetValue (key, sub_key, type, value)
     Associates a value with a specified key.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _sub_key_ is a string that names the subkey with which the value
     is associated.

     _type_ is an integer that specifies the type of the data.
     Currently this must be REG_SZ, meaning only strings
     are supported.  Use the SetValueEx().  function for
     support for other data types.

     _value_ is a string that specifies the new value.

     If the key specified by the _sub_key_ parameter does not exist,
     the SetValue function creates it.

     Value lengths are limited by available memory. Long values (more
     than 2048 bytes) should be stored as files with the filenames
     stored in the configuration registry.  This helps the registry
     perform efficiently.

     The key identified by the _key_ parameter must have been opened
     with KEY_SET_VALUE access.

 -- Function: _winreg.SetValueEx (key, value_name, reserved, type,
          value)
     Stores data in the value field of an open registry key.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     _value_name_ is a string that names the subkey with which the
     value is associated.

     _type_ is an integer that specifies the type of the data. See
     Value Types for the available types.

     _reserved_ can be anything - zero is always passed to the API.

     _value_ is a string that specifies the new value.

     This method can also set additional value and type information for
     the specified key.  The key identified by the key parameter must
     have been opened with KEY_SET_VALUE access.

     To open the key, use the CreateKey() or
     OpenKey() methods.

     Value lengths are limited by available memory. Long values (more
     than 2048 bytes) should be stored as files with the filenames
     stored in the configuration registry.  This helps the registry
     perform efficiently.

 -- Function: _winreg.DisableReflectionKey (key)
     Disables registry reflection for 32-bit processes running on a
     64-bit operating system.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     Will generally raise NotImplemented if executed on a
     32-bit operating system.

     If the key is not on the reflection list, the function succeeds
     but has no effect. Disabling reflection for a key does not affect
     reflection of any subkeys.

 -- Function: _winreg.EnableReflectionKey (key)
     Restores registry reflection for the specified disabled key.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     Will generally raise NotImplemented if executed on a
     32-bit operating system.

     Restoring reflection for a key does not affect reflection of any
     subkeys.

 -- Function: _winreg.QueryReflectionKey (key)
     Determines the reflection state for the specified key.

     _key_ is an already open key, or one of the predefined
     HKEY_* constants.

     Returns `True' if reflection is disabled.

     Will generally raise NotImplemented if executed on a
     32-bit operating system.

5.34.3.1 Constants
..................

The following constants are defined for use in many _winreg.
functions.

5.34.3.2 HKEY_* Constants
.........................

 -- Data: _winreg.HKEY_CLASSES_ROOT
     Registry entries subordinate to this key define types (or classes)
     of documents and the properties associated with those types. Shell
     and COM applications use the information stored under this key.

 -- Data: _winreg.HKEY_CURRENT_USER
     Registry entries subordinate to this key define the preferences of
     the current user. These preferences include the settings of
     environment variables, data about program groups, colors, printers,
     network connections, and application preferences.

 -- Data: _winreg.HKEY_LOCAL_MACHINE
     Registry entries subordinate to this key define the physical state
     of the computer, including data about the bus type, system memory,
     and installed hardware and software.

 -- Data: _winreg.HKEY_USERS
     Registry entries subordinate to this key define the default user
     configuration for new users on the local computer and the user
     configuration for the current user.

 -- Data: _winreg.HKEY_PERFORMANCE_DATA
     Registry entries subordinate to this key allow you to access
     performance data. The data is not actually stored in the registry;
     the registry functions cause the system to collect the data from
     its source.

 -- Data: _winreg.HKEY_CURRENT_CONFIG
     Contains information about the current hardware profile of the
     local computer system.

 -- Data: _winreg.HKEY_DYN_DATA
     This key is not used in versions of Windows after 98.

5.34.3.3 Access Rights
......................

For more information, see Registry Key Security and Access
(http://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx).

 -- Data: _winreg.KEY_ALL_ACCESS
     Combines the STANDARD_RIGHTS_REQUIRED, see KEY_QUERY_VALUE,
     KEY_SET_VALUE, KEY_CREATE_SUB_KEY,
     KEY_ENUMERATE_SUB_KEYS, KEY_NOTIFY, and
     KEY_CREATE_LINK access rights.

 -- Data: _winreg.KEY_WRITE
     Combines the STANDARD_RIGHTS_WRITE, see KEY_SET_VALUE, and
     KEY_CREATE_SUB_KEY access rights.

 -- Data: _winreg.KEY_READ
     Combines the STANDARD_RIGHTS_READ, see KEY_QUERY_VALUE,
     KEY_ENUMERATE_SUB_KEYS, and KEY_NOTIFY.
     values.

 -- Data: _winreg.KEY_EXECUTE
     Equivalent to KEY_READ.

 -- Data: _winreg.KEY_QUERY_VALUE
     Required to query the values of a registry key.

 -- Data: _winreg.KEY_SET_VALUE
     Required to create, delete, or set a registry value.

 -- Data: _winreg.KEY_CREATE_SUB_KEY
     Required to create a subkey of a registry key.

 -- Data: _winreg.KEY_ENUMERATE_SUB_KEYS
     Required to enumerate the subkeys of a registry key.

 -- Data: _winreg.KEY_NOTIFY
     Required to request change notifications for a registry key or for
     subkeys of a registry key.

 -- Data: _winreg.KEY_CREATE_LINK
     Reserved for system use.

5.34.3.4 64-bit Specific
........................

For more information, see Accesing an Alternate Registry View
(http://msdn.microsoft.com/en-us/library/aa384129(v=VS.85).aspx).

 -- Data: _winreg.KEY_WOW64_64KEY
     Indicates that an application on 64-bit Windows should operate on
     the 64-bit registry view.

 -- Data: _winreg.KEY_WOW64_32KEY
     Indicates that an application on 64-bit Windows should operate on
     the 32-bit registry view.

5.34.3.5 Value Types
....................

For more information, see Registry Value Types
(http://msdn.microsoft.com/en-us/library/ms724884%28v=VS.85%29.aspx).

 -- Data: _winreg.REG_BINARY
     Binary data in any form.

 -- Data: _winreg.REG_DWORD
     32-bit number.

 -- Data: _winreg.REG_DWORD_LITTLE_ENDIAN
     A 32-bit number in little-endian format.

 -- Data: _winreg.REG_DWORD_BIG_ENDIAN
     A 32-bit number in big-endian format.

 -- Data: _winreg.REG_EXPAND_SZ
     Null-terminated string containing references to environment
     variables (`%PATH%').

 -- Data: _winreg.REG_LINK
     A Unicode symbolic link.

 -- Data: _winreg.REG_MULTI_SZ
     A sequence of null-terminated strings, terminated by two null
     characters.  (Python handles this termination automatically.)

 -- Data: _winreg.REG_NONE
     No defined value type.

 -- Data: _winreg.REG_RESOURCE_LIST
     A device-driver resource list.

 -- Data: _winreg.REG_FULL_RESOURCE_DESCRIPTOR
     A hardware setting.

 -- Data: _winreg.REG_RESOURCE_REQUIREMENTS_LIST
     A hardware resource list.

 -- Data: _winreg.REG_SZ
     A null-terminated string.

5.34.3.6 Registry Handle Objects
................................

This object wraps a Windows HKEY object, automatically closing it when
the object is destroyed.  To guarantee cleanup, you can call either the
Close() method on the object, or the CloseKey()
function.

  All registry functions in this module return one of these objects.

  All registry functions in this module which accept a handle object
also accept an integer, however, use of the handle object is encouraged.

  Handle objects provide semantics for __nonzero__(). - thus:

    if handle:
        print "Yes"

will print `Yes' if the handle is currently valid (has not been closed
or detached).

  The object also support comparison semantics, so handle objects will
compare true if they both reference the same underlying Windows handle
value.

  Handle objects can be converted to an integer (e.g., using the
built-in int() function), in which case the underlying
Windows handle value is returned.  You can also use the Detach()
method to return the integer handle, and also disconnect the
Windows handle from the handle object.

 -- Method: PyHKEY.Close ()
     Closes the underlying Windows handle.

     If the handle is already closed, no error is raised.

 -- Method: PyHKEY.Detach ()
     Detaches the Windows handle from the handle object.

     The result is an integer (or long on 64 bit Windows) that holds
     the value of the handle before it is detached.  If the handle is
     already detached or closed, this will return zero.

     After calling this function, the handle is effectively
     invalidated, but the handle is not closed.  You would call this
     function when you need the underlying Win32 handle to exist beyond
     the lifetime of the handle object.

 -- Method: PyHKEY.__enter__ ()
 -- Method: PyHKEY.__exit__ (*exc_info)
     The HKEY object implements __enter__() and
     __exit__() and thus supports the context protocol for the
     with statement:

         with OpenKey(HKEY_LOCAL_MACHINE, "foo") as key:
             ...  # work with key

     will automatically close _key_ when control leaves the with
     block.

     New in version 2.6.

5.34.4 `winsound' -- Sound-playing interface for Windows
--------------------------------------------------------

New in version 1.5.2.

  The winsound module provides access to the basic
sound-playing machinery provided by Windows platforms.  It includes
functions and several constants.

 -- Function: winsound.Beep (frequency, duration)
     Beep the PC's speaker. The _frequency_ parameter specifies
     frequency, in hertz, of the sound, and must be in the range 37
     through 32,767. The _duration_ parameter specifies the number of
     milliseconds the sound should last.  If the system is not able to
     beep the speaker, see RuntimeError is raised.

     New in version 1.6.

 -- Function: winsound.PlaySound (sound, flags)
     Call the underlying `PlaySound()' function from the Platform API.
     The _sound_ parameter may be a filename, audio data as a string,
     or `None'.  Its interpretation depends on the value of _flags_,
     which can be a bitwise ORed combination of the constants described
     below. If the _sound_ parameter is `None', any currently playing
     waveform sound is stopped. If the system indicates an error, see
     RuntimeError is raised.

 -- Function: winsound.MessageBeep ([type=MB_OK])
     Call the underlying `MessageBeep()' function from the Platform
     API.  This plays a sound as specified in the registry.  The _type_
     argument specifies which sound to play; possible values are `-1',
     `MB_ICONASTERISK', `MB_ICONEXCLAMATION', `MB_ICONHAND',
     `MB_ICONQUESTION', and `MB_OK', all described below.  The value
     `-1' produces a "simple beep"; this is the final fallback if a
     sound cannot be played otherwise.

     New in version 2.3.

 -- Data: winsound.SND_FILENAME
     The _sound_ parameter is the name of a WAV file. Do not use with
     SND_ALIAS.

 -- Data: winsound.SND_ALIAS
     The _sound_ parameter is a sound association name from the
     registry.  If the registry contains no such name, play the system
     default sound unless SND_NODEFAULT is also specified.
     If no default sound is registered, raise RuntimeError.
     Do not use with SND_FILENAME.

     All Win32 systems support at least the following; most systems
     support many more:

     PlaySound().       Corresponding Control Panel Sound name
     _name_
     ----------------------------------------------------------------------------
     `'SystemAsterisk''             Asterisk
     `'SystemExclamation''          Exclamation
     `'SystemExit''                 Exit Windows
     `'SystemHand''                 Critical Stop
     `'SystemQuestion''             Question

     For example:

         import winsound
         # Play Windows exit sound.
         winsound.PlaySound("SystemExit", winsound.SND_ALIAS)

         # Probably play Windows default sound, if any is registered (because
         # "*" probably isn't the registered name of any sound).
         winsound.PlaySound("*", winsound.SND_ALIAS)

 -- Data: winsound.SND_LOOP
     Play the sound repeatedly.  The SND_ASYNC flag must
     also be used to avoid blocking.  Cannot be used with
     SND_MEMORY.

 -- Data: winsound.SND_MEMORY
     The _sound_ parameter to PlaySound() is a memory
     image of a WAV file, as a string.

          Note: This module does not support playing from a memory
          image asynchronously, so a combination of this flag and
          SND_ASYNC will raise RuntimeError.

 -- Data: winsound.SND_PURGE
     Stop playing all instances of the specified sound.

          Note: This flag is not supported on modern Windows platforms.

 -- Data: winsound.SND_ASYNC
     Return immediately, allowing sounds to play asynchronously.

 -- Data: winsound.SND_NODEFAULT
     If the specified sound cannot be found, do not play the system
     default sound.

 -- Data: winsound.SND_NOSTOP
     Do not interrupt sounds currently playing.

 -- Data: winsound.SND_NOWAIT
     Return immediately if the sound driver is busy.

 -- Data: winsound.MB_ICONASTERISK
     Play the `SystemDefault' sound.

 -- Data: winsound.MB_ICONEXCLAMATION
     Play the `SystemExclamation' sound.

 -- Data: winsound.MB_ICONHAND
     Play the `SystemHand' sound.

 -- Data: winsound.MB_ICONQUESTION
     Play the `SystemQuestion' sound.

 -- Data: winsound.MB_OK
     Play the `SystemDefault' sound.

5.35 Unix Specific Services
===========================

The modules described in this chapter provide interfaces to features
that are unique to the Unix operating system, or in some cases to some
or many variants of it.  Here's an overview:

5.35.1 `posix' -- The most common POSIX system calls
----------------------------------------------------

This module provides access to operating system functionality that is
standardized by the C Standard and the POSIX standard (a thinly
disguised Unix interface).

  *Do not import this module directly.*  Instead, import the module
os, which provides a _portable_ version of this interface.
On Unix, the os.  module provides a superset of the
posix interface.  On non-Unix operating systems the posix
module is not available, but a subset is always available through
the os interface.  Once os is imported, there is
_no_ performance penalty in using it instead of posix.  In
addition, see os provides some additional functionality, such
as automatically calling `putenv()' when an entry in `os.environ' is
changed.

  Errors are reported as exceptions; the usual exceptions are given for
type errors, while errors reported by the system calls raise
OSError.

5.35.1.1 Large File Support
...........................

Several operating systems (including AIX, HP-UX, Irix and Solaris)
provide support for files that are larger than 2 GB from a C
programming model where `int' and `long' are 32-bit values. This is
typically accomplished by defining the relevant size and offset types
as 64-bit values. Such files are sometimes referred to as _large files_.

  Large file support is enabled in Python when the size of an `off_t' is
larger than a `long' and the `long long' type is available and is at
least as large as an `off_t'. Python longs are then used to represent
file sizes, offsets and other values that can exceed the range of a
Python int.  It may be necessary to configure and compile Python with
certain compiler flags to enable this mode. For example, it is enabled
by default with recent versions of Irix, but with Solaris 2.6 and 2.7
you need to do something like:

    CFLAGS="`getconf LFS_CFLAGS`" OPT="-g -O2 $CFLAGS" \
            ./configure

On large-file-capable Linux systems, this might work:

    CFLAGS='-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64' OPT="-g -O2 $CFLAGS" \
            ./configure

5.35.1.2 Notable Module Contents
................................

In addition to many functions described in the os module
documentation, see posix defines the following data item:

 -- Data: posix.environ
     A dictionary representing the string environment at the time the
     interpreter was started.  For example, `environ['HOME']' is the
     pathname of your home directory, equivalent to `getenv("HOME")' in
     C.

     Modifying this dictionary does not affect the string environment
     passed on by `execv()', `popen()' or `system()'; if you need to
     change the environment, pass `environ' to `execve()' or add
     variable assignments and export statements to the command string
     for `system()' or `popen()'.

          Note: The os module provides an alternate
          implementation of `environ' which updates the environment on
          modification.  Note also that updating `os.environ' will
          render this dictionary obsolete.  Use of the os.
          module version of this is recommended over direct access to
          the posix module.

5.35.2 `pwd' -- The password database
-------------------------------------

This module provides access to the Unix user account and password
database.  It is available on all Unix versions.

  Password database entries are reported as a tuple-like object, whose
attributes correspond to the members of the `passwd' structure
(Attribute field below, see `<pwd.h>'):

Index       Attribute           Meaning
------------------------------------------------------------------
0           `pw_name'           Login name
1           `pw_passwd'         Optional encrypted password
2           `pw_uid'            Numerical user ID
3           `pw_gid'            Numerical group ID
4           `pw_gecos'          User name or comment field
5           `pw_dir'            User home directory
6           `pw_shell'          User command interpreter

  The uid and gid items are integers, all others are strings.
KeyError is raised if the entry asked for cannot be found.

     Note: In traditional Unix the field `pw_passwd' usually contains a
     password encrypted with a DES derived algorithm (see module
     crypt).  However most modern unices  use a so-called _shadow
     password_ system.  On those unices the _pw_passwd_ field only
     contains an asterisk (`'*'') or the  letter `'x'' where the
     encrypted password is stored in a file `/etc/shadow' which is not
     world readable.  Whether the _pw_passwd_ field contains anything
     useful is system-dependent.  If available, the spwd.
     module should be used where access to the encrypted password is
     required.

  It defines the following items:

 -- Function: pwd.getpwuid (uid)
     Return the password database entry for the given numeric user ID.

 -- Function: pwd.getpwnam (name)
     Return the password database entry for the given user name.

 -- Function: pwd.getpwall ()
     Return a list of all available password database entries, in
     arbitrary order.

Module grp.
     An interface to the group database, similar to this.

Module spwd.
     An interface to the shadow password database, similar to this.

5.35.3 `spwd' -- The shadow password database
---------------------------------------------

New in version 2.5.

  This module provides access to the Unix shadow password database. It
is available on various Unix versions.

  You must have enough privileges to access the shadow password
database (this usually means you have to be root).

  Shadow password database entries are reported as a tuple-like object,
whose attributes correspond to the members of the `spwd' structure
(Attribute field below, see `<shadow.h>'):

Index       Attribute           Meaning
----------------------------------------------------------------------
0           `sp_nam'            Login name
1           `sp_pwd'            Encrypted password
2           `sp_lstchg'         Date of last change
3           `sp_min'            Minimal number of days between
                                changes
4           `sp_max'            Maximum number of days between
                                changes
5           `sp_warn'           Number of days before password
                                expires to warn user about it
6           `sp_inact'          Number of days after password
                                expires until account is blocked
7           `sp_expire'         Number of days since 1970-01-01
                                until account is disabled
8           `sp_flag'           Reserved

  The sp_nam and sp_pwd items are strings, all others are integers.
KeyError is raised if the entry asked for cannot be found.

  It defines the following items:

 -- Function: spwd.getspnam (name)
     Return the shadow password database entry for the given user name.

 -- Function: spwd.getspall ()
     Return a list of all available shadow password database entries,
     in arbitrary order.

Module grp.
     An interface to the group database, similar to this.

Module pwd.
     An interface to the normal password database, similar to this.

5.35.4 `grp' -- The group database
----------------------------------

This module provides access to the Unix group database. It is available
on all Unix versions.

  Group database entries are reported as a tuple-like object, whose
attributes correspond to the members of the `group' structure
(Attribute field below, see `<pwd.h>'):

Index       Attribute       Meaning
------------------------------------------------------------------
0           gr_name         the name of the group
1           gr_passwd       the (encrypted) group password;
                            often empty
2           gr_gid          the numerical group ID
3           gr_mem          all the group member's  user names

  The gid is an integer, name and password are strings, and the member
list is a list of strings. (Note that most users are not explicitly
listed as members of the group they are in according to the password
database.  Check both databases to get complete membership information.)

  It defines the following items:

 -- Function: grp.getgrgid (gid)
     Return the group database entry for the given numeric group ID.
     See KeyError.  is raised if the entry asked for cannot be
     found.

 -- Function: grp.getgrnam (name)
     Return the group database entry for the given group name.
     KeyError is raised if the entry asked for cannot be found.

 -- Function: grp.getgrall ()
     Return a list of all available group entries, in arbitrary order.

Module pwd.
     An interface to the user database, similar to this.

Module spwd.
     An interface to the shadow password database, similar to this.

5.35.5 `crypt' -- Function to check Unix passwords
--------------------------------------------------

This module implements an interface to the `crypt(3)' routine, which is
a one-way hash function based upon a modified DES algorithm; see the
Unix man page for further details.  Possible uses include allowing
Python scripts to accept typed passwords from the user, or attempting
to crack Unix passwords with a dictionary.

  Notice that the behavior of this module depends on the actual
implementation  of the `crypt(3)' routine in the running system.
Therefore, any extensions available on the current implementation will
also  be available on this module.

 -- Function: crypt.crypt (word, salt)
     _word_ will usually be a user's password as typed at a prompt or
     in a graphical interface.  _salt_ is usually a random
     two-character string which will be used to perturb the DES
     algorithm in one of 4096 ways.  The characters in _salt_ must be
     in the set `[./a-zA-Z0-9]'.  Returns the hashed password as a
     string, which will be composed of characters from the same
     alphabet as the salt (the first two characters represent the salt
     itself).

     Since a few `crypt(3)' extensions allow different values, with
     different sizes in the _salt_, it is recommended to use  the full
     crypted password as salt when checking for a password.

  A simple example illustrating typical use:

    import crypt, getpass, pwd

    def login():
        username = raw_input('Python login:')
        cryptedpasswd = pwd.getpwnam(username)[1]
        if cryptedpasswd:
            if cryptedpasswd == 'x' or cryptedpasswd == '*':
                raise NotImplementedError(
                    "Sorry, currently no support for shadow passwords")
            cleartext = getpass.getpass()
            return crypt.crypt(cleartext, cryptedpasswd) == cryptedpasswd
        else:
            return 1

5.35.6 `dl' -- Call C functions in shared objects
-------------------------------------------------

Deprecated since version 2.6: The dl module has been removed
in Python 3.0. Use the ctypes.  module instead.

  The dl module defines an interface to the `dlopen()'
function, which is the most common interface on Unix platforms for
handling dynamically linked libraries. It allows the program to call
arbitrary functions in such a library.

     Warning: The dl module bypasses the Python type system
     and  error handling. If used incorrectly it may cause segmentation
     faults, crashes or other incorrect behaviour.

     Note: This module will not work unless `sizeof(int) ==
     sizeof(long) == sizeof(char *)' If this is not the case, see
     SystemError will be raised on import.

  The dl module defines the following function:

 -- Function: dl.open (name[, mode=RTLD_LAZY])
     Open a shared object file, and return a handle. Mode signifies
     late binding (see RTLD_LAZY) or immediate binding (see
     RTLD_NOW). Default is RTLD_LAZY. Note that some
     systems do not support RTLD_NOW.

     Return value is a `dlobject'.

  The dl module defines the following constants:

 -- Data: dl.RTLD_LAZY
     Useful as an argument to open().

 -- Data: dl.RTLD_NOW
     Useful as an argument to open().  Note that on systems
     which do not support immediate binding, this constant will not
     appear in the module. For maximum portability, use
     hasattr() to determine if the system supports immediate
     binding.

  The dl module defines the following exception:

 -- Exception: dl.error
     Exception raised when an error has occurred inside the dynamic
     loading and linking routines.

  Example:

    >>> import dl, time
    >>> a=dl.open('/lib/libc.so.6')
    >>> a.call('time'), time.time()
    (929723914, 929723914.498)

This example was tried on a Debian GNU/Linux system, and is a good
example of the fact that using this module is usually a bad alternative.

5.35.6.1 Dl Objects
...................

Dl objects, as returned by open() above, have the
following methods:

 -- Method: dl.close ()
     Free all resources, except the memory.

 -- Method: dl.sym (name)
     Return the pointer for the function named _name_, as a number, if
     it exists in the referenced shared object, otherwise `None'. This
     is useful in code like:

         >>> if a.sym('time'):
         ...     a.call('time')
         ... else:
         ...     time.time()

     (Note that this function will return a non-zero number, as zero is
     the _NULL_ pointer)

 -- Method: dl.call (name[, arg1[, arg2...]])
     Call the function named _name_ in the referenced shared object.
     The arguments must be either Python integers, which will be
     passed as is, Python strings, to which a pointer will be passed,
     or `None', which will be passed as _NULL_.  Note that  strings
     should only be passed to functions as `const char*', as Python
     will not like its string mutated.

     There must be at most 10 arguments, and arguments not given will
     be treated as `None'. The function's return value must be a C
     `long', which is a Python integer.

5.35.7 `termios' -- POSIX style tty control
-------------------------------------------

This module provides an interface to the POSIX calls for tty I/O
control.  For a complete description of these calls, see the POSIX or
Unix manual pages.  It is only available for those Unix versions that
support POSIX _termios_ style tty I/O control (and then only if
configured at installation time).

  All functions in this module take a file descriptor _fd_ as their
first argument.  This can be an integer file descriptor, such as
returned by `sys.stdin.fileno()', or a file object, such as `sys.stdin'
itself.

  This module also defines all the constants needed to work with the
functions provided here; these have the same name as their counterparts
in C.  Please refer to your system documentation for more information
on using these terminal control interfaces.

  The module defines the following functions:

 -- Function: termios.tcgetattr (fd)
     Return a list containing the tty attributes for file descriptor
     _fd_, as follows: `[iflag, oflag, cflag, lflag, ispeed, ospeed,
     cc]' where _cc_ is a list of the tty special characters (each a
     string of length 1, except the items with indices `VMIN' and
     `VTIME', which are integers when these fields are defined).  The
     interpretation of the flags and the speeds as well as the indexing
     in the _cc_ array must be done using the symbolic constants
     defined in the termios module.

 -- Function: termios.tcsetattr (fd, when, attributes)
     Set the tty attributes for file descriptor _fd_ from the
     _attributes_, which is a list like the one returned by
     tcgetattr().  The _when_ argument determines when the
     attributes are changed: `TCSANOW' to change immediately,
     `TCSADRAIN' to change after transmitting all queued output, or
     `TCSAFLUSH' to change after transmitting all queued output and
     discarding all queued input.

 -- Function: termios.tcsendbreak (fd, duration)
     Send a break on file descriptor _fd_.  A zero _duration_ sends a
     break for 0.25 -0.5 seconds; a nonzero _duration_ has a system
     dependent meaning.

 -- Function: termios.tcdrain (fd)
     Wait until all output written to file descriptor _fd_ has been
     transmitted.

 -- Function: termios.tcflush (fd, queue)
     Discard queued data on file descriptor _fd_.  The _queue_ selector
     specifies which queue: `TCIFLUSH' for the input queue, `TCOFLUSH'
     for the output queue, or `TCIOFLUSH' for both queues.

 -- Function: termios.tcflow (fd, action)
     Suspend or resume input or output on file descriptor _fd_.  The
     _action_ argument can be `TCOOFF' to suspend output, `TCOON' to
     restart output, `TCIOFF' to suspend input, or `TCION' to restart
     input.

Module tty.
     Convenience functions for common terminal control operations.

5.35.7.1 Example
................

Here's a function that prompts for a password with echoing turned off.
Note the technique using a separate tcgetattr() call and a
try...  finally statement to ensure that the
old tty attributes are restored exactly no matter what happens:

    def getpass(prompt="Password: "):
        import termios, sys
        fd = sys.stdin.fileno()
        old = termios.tcgetattr(fd)
        new = termios.tcgetattr(fd)
        new[3] = new[3] & ~termios.ECHO          # lflags
        try:
            termios.tcsetattr(fd, termios.TCSADRAIN, new)
            passwd = raw_input(prompt)
        finally:
            termios.tcsetattr(fd, termios.TCSADRAIN, old)
        return passwd

5.35.8 `tty' -- Terminal control functions
------------------------------------------

The tty module defines functions for putting the tty into
cbreak and raw modes.

  Because it requires the termios module, it will work only
on Unix.

  The tty module defines the following functions:

 -- Function: tty.setraw (fd[, when])
     Change the mode of the file descriptor _fd_ to raw. If _when_ is
     omitted, it defaults to `termios.TCSAFLUSH', and is passed to
     termios.tcsetattr().

 -- Function: tty.setcbreak (fd[, when])
     Change the mode of file descriptor _fd_ to cbreak. If _when_ is
     omitted, it defaults to `termios.TCSAFLUSH', and is passed to
     termios.tcsetattr().

Module termios.
     Low-level terminal control interface.

5.35.9 `pty' -- Pseudo-terminal utilities
-----------------------------------------

The pty module defines operations for handling the
pseudo-terminal concept: starting another process and being able to
write to and read from its controlling terminal programmatically.

  Because pseudo-terminal handling is highly platform dependent, there
is code to do it only for Linux. (The Linux code is supposed to work on
other platforms, but hasn't been tested yet.)

  The pty module defines the following functions:

 -- Function: pty.fork ()
     Fork. Connect the child's controlling terminal to a
     pseudo-terminal. Return value is `(pid, fd)'. Note that the child
     gets _pid_ 0, and the _fd_ is _invalid_. The parent's return value
     is the _pid_ of the child, and _fd_ is a file descriptor connected
     to the child's controlling terminal (and also to the child's
     standard input and output).

 -- Function: pty.openpty ()
     Open a new pseudo-terminal pair, using os.openpty().
     if possible, or emulation code for generic Unix systems. Return a
     pair of file descriptors `(master, slave)', for the master and the
     slave end, respectively.

 -- Function: pty.spawn (argv[, master_read[, stdin_read]])
     Spawn a process, and connect its controlling terminal with the
     current process's standard io. This is often used to baffle
     programs which insist on reading from the controlling terminal.

     The functions _master_read_ and _stdin_read_ should be functions
     which read from a file descriptor. The defaults try to read 1024
     bytes each time they are called.

5.35.10 `fcntl' -- The `fcntl()' and `ioctl()' system calls
-----------------------------------------------------------

This module performs file control and I/O control on file descriptors.
It is an interface to the `fcntl()' and `ioctl()' Unix routines.

  All functions in this module take a file descriptor _fd_ as their
first argument.  This can be an integer file descriptor, such as
returned by `sys.stdin.fileno()', or a file object, such as `sys.stdin'
itself, which provides a `fileno()' which returns a genuine file
descriptor.

  The module defines the following functions:

 -- Function: fcntl.fcntl (fd, op[, arg])
     Perform the requested operation on file descriptor _fd_ (file
     objects providing a `fileno()' method are accepted as well). The
     operation is defined by _op_ and is operating system dependent.
     These codes are also found in the fcntl module. The
     argument _arg_ is optional, and defaults to the integer value `0'.
     When present, it can either be an integer value, or a string.
     With the argument missing or an integer value, the return value of
     this function is the integer return value of the C `fcntl()' call.
     When the argument is a string it represents a binary structure,
     e.g. created by struct.pack().  The binary data is
     copied to a buffer whose address is passed to the C `fcntl()'
     call.  The return value after a successful call is the contents of
     the buffer, converted to a string object.  The length of the
     returned string will be the same as the length of the _arg_
     argument.  This is limited to 1024 bytes.  If the information
     returned in the buffer by the operating system is larger than 1024
     bytes, this is most likely to result in a segmentation violation
     or a more subtle data corruption.

     If the `fcntl()' fails, an IOError is raised.

 -- Function: fcntl.ioctl (fd, op[, arg[, mutate_flag]])
     This function is identical to the fcntl() function,
     except that the operations are typically defined in the library
     module termios and the argument handling is even more
     complicated.

     The op parameter is limited to values that can fit in 32-bits.

     The parameter _arg_ can be one of an integer, absent (treated
     identically to the integer `0'), an object supporting the
     read-only buffer interface (most likely a plain Python string) or
     an object supporting the read-write buffer interface.

     In all but the last case, behaviour is as for the fcntl()
     function.

     If a mutable buffer is passed, then the behaviour is determined by
     the value of the _mutate_flag_ parameter.

     If it is false, the buffer's mutability is ignored and behaviour
     is as for a read-only buffer, except that the 1024 byte limit
     mentioned above is avoided - so long as the buffer you pass is as
     least as long as what the operating system wants to put there,
     things should work.

     If _mutate_flag_ is true, then the buffer is (in effect) passed to
     the underlying ioctl() system call, the latter's
     return code is passed back to the calling Python, and the buffer's
     new contents reflect the action of the ioctl().  This
     is a slight simplification, because if the supplied buffer is less
     than 1024 bytes long it is first copied into a static buffer 1024
     bytes long which is then passed to ioctl() and copied
     back into the supplied buffer.

     If _mutate_flag_ is not supplied, then from Python 2.5 it defaults
     to true, which is a change from versions 2.3 and 2.4. Supply the
     argument explicitly if version portability is a priority.

     An example:

         >>> import array, fcntl, struct, termios, os
         >>> os.getpgrp()
         13341
         >>> struct.unpack('h', fcntl.ioctl(0, termios.TIOCGPGRP, "  "))[0]
         13341
         >>> buf = array.array('h', [0])
         >>> fcntl.ioctl(0, termios.TIOCGPGRP, buf, 1)
         0
         >>> buf
         array('h', [13341])

 -- Function: fcntl.flock (fd, op)
     Perform the lock operation _op_ on file descriptor _fd_ (file
     objects providing a `fileno()' method are accepted as well). See
     the Unix manual `flock(2)' for details.  (On some systems, this
     function is emulated using `fcntl()'.)

 -- Function: fcntl.lockf (fd, operation[, length[, start[, whence]]])
     This is essentially a wrapper around the fcntl().
     locking calls.  _fd_ is the file descriptor of the file to lock or
     unlock, and _operation_ is one of the following values:

        * `LOCK_UN' - unlock

        * `LOCK_SH' - acquire a shared lock

        * `LOCK_EX' - acquire an exclusive lock

     When _operation_ is `LOCK_SH' or `LOCK_EX', it can also be bitwise
     ORed with `LOCK_NB' to avoid blocking on lock acquisition.  If
     `LOCK_NB' is used and the lock cannot be acquired, an
     IOError will be raised and the exception will have an _errno_
     attribute set to `EACCES' or `EAGAIN' (depending on the operating
     system; for portability, check for both values).  On at least some
     systems, `LOCK_EX' can only be used if the file descriptor refers
     to a file opened for writing.

     _length_ is the number of bytes to lock, _start_ is the byte
     offset at which the lock starts, relative to _whence_, and
     _whence_ is as with `fileobj.seek()', specifically:

        * `0' - relative to the start of the file (`SEEK_SET')

        * `1' - relative to the current buffer position (`SEEK_CUR')

        * `2' - relative to the end of the file (`SEEK_END')

     The default for _start_ is 0, which means to start at the
     beginning of the file.  The default for _length_ is 0 which means
     to lock to the end of the file.  The default for _whence_ is also
     0.

  Examples (all on a SVR4 compliant system):

    import struct, fcntl, os

    f = open(...)
    rv = fcntl.fcntl(f, fcntl.F_SETFL, os.O_NDELAY)

    lockdata = struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0)
    rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata)

Note that in the first example the return value variable _rv_ will hold
an integer value; in the second example it will hold a string value.
The structure lay-out for the _lockdata_ variable is system dependent
-- therefore using the flock() call may be better.

Module os.
     If the locking flags `O_SHLOCK' and `O_EXLOCK' are present in the
     os module (on BSD only), the os.open().
     function provides an alternative to the lockf() and
     flock() functions.

5.35.11 `pipes' -- Interface to shell pipelines
-----------------------------------------------

The pipes module defines a class to abstract the concept of
a _pipeline_ -- a sequence of converters from one file to  another.

  Because the module uses */bin/sh* command lines, a POSIX or compatible
shell for os.system() and os.popen() is
required.

  The pipes module defines the following class:

 -- Class: pipes.Template
     An abstraction of a pipeline.

  Example:

    >>> import pipes
    >>> t=pipes.Template()
    >>> t.append('tr a-z A-Z', '--')
    >>> f=t.open('/tmp/1', 'w')
    >>> f.write('hello world')
    >>> f.close()
    >>> open('/tmp/1').read()
    'HELLO WORLD'

5.35.11.1 Template Objects
..........................

Template objects following methods:

 -- Method: Template.reset ()
     Restore a pipeline template to its initial state.

 -- Method: Template.clone ()
     Return a new, equivalent, pipeline template.

 -- Method: Template.debug (flag)
     If _flag_ is true, turn debugging on. Otherwise, turn debugging
     off. When debugging is on, commands to be executed are printed,
     and the shell is given `set -x' command to be more verbose.

 -- Method: Template.append (cmd, kind)
     Append a new action at the end. The _cmd_ variable must be a valid
     bourne shell command. The _kind_ variable consists of two letters.

     The first letter can be either of `'-'' (which means the command
     reads its standard input), `'f'' (which means the commands reads a
     given file on the command line) or `'.'' (which means the commands
     reads no input, and hence must be first.)

     Similarly, the second letter can be either of `'-'' (which means
     the command writes to standard output), `'f'' (which means the
     command writes a file on the command line) or `'.'' (which means
     the command does not write anything, and hence must be last.)

 -- Method: Template.prepend (cmd, kind)
     Add a new action at the beginning. See append() for
     explanations of the arguments.

 -- Method: Template.open (file, mode)
     Return a file-like object, open to _file_, but read from or
     written to by the pipeline.  Note that only one of `'r'', `'w''
     may be given.

 -- Method: Template.copy (infile, outfile)
     Copy _infile_ to _outfile_ through the pipe.

5.35.12 `posixfile' -- File-like objects with locking support
-------------------------------------------------------------

Deprecated since version 1.5: The locking operation that this module
provides is done better and more portably by the fcntl.lockf()
call.

  This module implements some additional functionality over the
built-in file objects.  In particular, it implements file locking,
control over the file flags, and an easy interface to duplicate the
file object. The module defines a new file object, the posixfile
object.  It has all the standard file object methods and adds the
methods described below.  This module only works for certain flavors of
Unix, since it uses fcntl.fcntl() for file locking.

  To instantiate a posixfile object, use the posixfile.open()
function.  The resulting object looks and feels roughly the same
as a standard file object.

  The posixfile module defines the following constants:

 -- Data: posixfile.SEEK_SET
     Offset is calculated from the start of the file.

 -- Data: posixfile.SEEK_CUR
     Offset is calculated from the current position in the file.

 -- Data: posixfile.SEEK_END
     Offset is calculated from the end of the file.

  The posixfile module defines the following functions:

 -- Function: posixfile.open (filename[, mode[, bufsize]])
     Create a new posixfile object with the given filename and mode.
     The _filename_, _mode_ and _bufsize_ arguments are interpreted the
     same way as by the built-in open() function.

 -- Function: posixfile.fileopen (fileobject)
     Create a new posixfile object with the given standard file object.
     The resulting object has the same filename and mode as the
     original file object.

  The posixfile object defines the following additional methods:

 -- Method: posixfile.lock (fmt[, len[, start[, whence]]])
     Lock the specified section of the file that the file object is
     referring to.  The format is explained below in a table.  The
     _len_ argument specifies the length of the section that should be
     locked. The default is `0'. _start_ specifies the starting offset
     of the section, where the default is `0'.  The _whence_ argument
     specifies where the offset is relative to. It accepts one of the
     constants SEEK_SET, SEEK_CUR or
     SEEK_END.  The default is SEEK_SET.  For more
     information about the arguments refer to the `fcntl(2)' manual
     page on your system.

 -- Method: posixfile.flags ([flags])
     Set the specified flags for the file that the file object is
     referring to.  The new flags are ORed with the old flags, unless
     specified otherwise.  The format is explained below in a table.
     Without the _flags_ argument a string indicating the current flags
     is returned (this is the same as the `?' modifier).  For more
     information about the flags refer to the `fcntl(2)' manual page on
     your system.

 -- Method: posixfile.dup ()
     Duplicate the file object and the underlying file pointer and file
     descriptor.  The resulting object behaves as if it were newly
     opened.

 -- Method: posixfile.dup2 (fd)
     Duplicate the file object and the underlying file pointer and file
     descriptor.  The new object will have the given file descriptor.
     Otherwise the resulting object behaves as if it were newly opened.

 -- Method: posixfile.file ()
     Return the standard file object that the posixfile object is based
     on.  This is sometimes necessary for functions that insist on a
     standard file object.

  All methods raise IOError when the request fails.

  Format characters for the `lock()' method have the following meaning:

Format       Meaning
-----------------------------------------------------------------
`u'          unlock the specified region
`r'          request a read lock for the specified section
`w'          request a write lock for the specified section

  In addition the following modifiers can be added to the format:

Modifier       Meaning                              Notes
----------------------------------------------------------------
`|'            wait until the lock has been granted
`?'            return the first lock conflicting    (1)
               with the requested lock, or `None'
               if there is no conflict.

  Note:

  1. The lock returned is in the format `(mode, len, start, whence,
     pid)' where _mode_ is a character representing the type of lock
     ('r' or 'w').  This modifier prevents a request from being
     granted; it is for query purposes only.

  Format characters for the `flags()' method have the following
meanings:

Format       Meaning
-----------------------------------------------------------------
`a'          append only flag
`c'          close on exec flag
`n'          no delay flag (also called non-blocking flag)
`s'          synchronization flag

  In addition the following modifiers can be added to the format:

Modifier       Meaning                               Notes
-----------------------------------------------------------------
`!'            turn the specified flags 'off',       (1)
               instead of the default 'on'
`='            replace the flags, instead of the     (1)
               default 'OR' operation
`?'            return a string in which the          (2)
               characters represent the flags that
               are set.

  Notes:

  1. The `!' and `=' modifiers are mutually exclusive.

  2. This string represents the flags after they may have been altered
     by the same call.

  Examples:

    import posixfile

    file = posixfile.open('/tmp/test', 'w')
    file.lock('w|')
    ...
    file.lock('u')
    file.close()

5.35.13 `resource' -- Resource usage information
------------------------------------------------

This module provides basic mechanisms for measuring and controlling
system resources utilized by a program.

  Symbolic constants are used to specify particular system resources
and to request usage information about either the current process or
its children.

  A single exception is defined for errors:

 -- Exception: resource.error
     The functions described below may raise this error if the
     underlying system call failures unexpectedly.

5.35.13.1 Resource Limits
.........................

Resources usage can be limited using the setrlimit().
function described below. Each resource is controlled by a pair of
limits: a soft limit and a hard limit. The soft limit is the current
limit, and may be lowered or raised by a process over time. The soft
limit can never exceed the hard limit. The hard limit can be lowered to
any value greater than the soft limit, but not raised.  (Only processes
with the effective UID of the super-user can raise a hard limit.)

  The specific resources that can be limited are system dependent. They
are described in the `getrlimit(2)' man page.  The resources listed
below are supported when the underlying operating system supports them;
resources which cannot be checked or controlled by the operating system
are not defined in this module for those platforms.

 -- Function: resource.getrlimit (resource)
     Returns a tuple `(soft, hard)' with the current soft and hard
     limits of _resource_. Raises ValueError if an invalid
     resource is specified, or error if the underlying
     system call fails unexpectedly.

 -- Function: resource.setrlimit (resource, limits)
     Sets new limits of consumption of _resource_. The _limits_
     argument must be a tuple `(soft, hard)' of two integers describing
     the new limits. A value of `-1' can be used to specify the maximum
     possible upper limit.

     Raises ValueError if an invalid resource is specified,
     if the new soft limit exceeds the hard limit, or if a process
     tries to raise its hard limit (unless the process has an effective
     UID of super-user).  Can also raise error if the
     underlying system call fails.

  These symbols define resources whose consumption can be controlled
using the setrlimit() and getrlimit().
functions described below. The values of these symbols are exactly the
constants used by C programs.

  The Unix man page for `getrlimit(2)' lists the available resources.
Note that not all systems use the same symbol or same value to denote
the same resource.  This module does not attempt to mask platform
differences -- symbols not defined for a platform will not be available
from this module on that platform.

 -- Data: resource.RLIMIT_CORE
     The maximum size (in bytes) of a core file that the current
     process can create.  This may result in the creation of a partial
     core file if a larger core would be required to contain the entire
     process image.

 -- Data: resource.RLIMIT_CPU
     The maximum amount of processor time (in seconds) that a process
     can use. If this limit is exceeded, a `SIGXCPU' signal is sent to
     the process. (See the signal module documentation for
     information about how to catch this signal and do something
     useful, e.g. flush open files to disk.)

 -- Data: resource.RLIMIT_FSIZE
     The maximum size of a file which the process may create.  This
     only affects the stack of the main thread in a multi-threaded
     process.

 -- Data: resource.RLIMIT_DATA
     The maximum size (in bytes) of the process's heap.

 -- Data: resource.RLIMIT_STACK
     The maximum size (in bytes) of the call stack for the current
     process.

 -- Data: resource.RLIMIT_RSS
     The maximum resident set size that should be made available to the
     process.

 -- Data: resource.RLIMIT_NPROC
     The maximum number of processes the current process may create.

 -- Data: resource.RLIMIT_NOFILE
     The maximum number of open file descriptors for the current
     process.

 -- Data: resource.RLIMIT_OFILE
     The BSD name for RLIMIT_NOFILE.

 -- Data: resource.RLIMIT_MEMLOCK
     The maximum address space which may be locked in memory.

 -- Data: resource.RLIMIT_VMEM
     The largest area of mapped memory which the process may occupy.

 -- Data: resource.RLIMIT_AS
     The maximum area (in bytes) of address space which may be taken by
     the process.

5.35.13.2 Resource Usage
........................

These functions are used to retrieve resource usage information:

 -- Function: resource.getrusage (who)
     This function returns an object that describes the resources
     consumed by either the current process or its children, as
     specified by the _who_ parameter.  The _who_ parameter should be
     specified using one of the `RUSAGE_*' constants described below.

     The fields of the return value each describe how a particular
     system resource has been used, e.g. amount of time spent running
     is user mode or number of times the process was swapped out of
     main memory. Some values are dependent on the clock tick internal,
     e.g. the amount of memory the process is using.

     For backward compatibility, the return value is also accessible as
     a tuple of 16 elements.

     The fields `ru_utime' and `ru_stime' of the return value are
     floating point values representing the amount of time spent
     executing in user mode and the amount of time spent executing in
     system mode, respectively. The remaining values are integers.
     Consult the `getrusage(2)' man page for detailed information about
     these values. A brief summary is presented here:

     Index        Field                     Resource
     ---------------------------------------------------------------------------
     `0'          `ru_utime'                time in user mode (float)
     `1'          `ru_stime'                time in system mode (float)
     `2'          `ru_maxrss'               maximum resident set size
     `3'          `ru_ixrss'                shared memory size
     `4'          `ru_idrss'                unshared memory size
     `5'          `ru_isrss'                unshared stack size
     `6'          `ru_minflt'               page faults not requiring I/O
     `7'          `ru_majflt'               page faults requiring I/O
     `8'          `ru_nswap'                number of swap outs
     `9'          `ru_inblock'              block input operations
     `10'         `ru_oublock'              block output operations
     `11'         `ru_msgsnd'               messages sent
     `12'         `ru_msgrcv'               messages received
     `13'         `ru_nsignals'             signals received
     `14'         `ru_nvcsw'                voluntary context switches
     `15'         `ru_nivcsw'               involuntary context switches

     This function will raise a ValueError if an invalid
     _who_ parameter is specified. It may also raise error.
     exception in unusual circumstances.

     Changed in version 2.3: Added access to values as attributes of
     the returned object.

 -- Function: resource.getpagesize ()
     Returns the number of bytes in a system page. (This need not be
     the same as the hardware page size.) This function is useful for
     determining the number of bytes of memory a process is using. The
     third element of the tuple returned by getrusage().
     describes memory usage in pages; multiplying by page size produces
     number of bytes.

  The following `RUSAGE_*' symbols are passed to the getrusage().
function to specify which processes information should be
provided for.

 -- Data: resource.RUSAGE_SELF
     RUSAGE_SELF should be used to request information
     pertaining only to the process itself.

 -- Data: resource.RUSAGE_CHILDREN
     Pass to getrusage() to request resource information
     for child processes of the calling process.

 -- Data: resource.RUSAGE_BOTH
     Pass to getrusage() to request resources consumed by
     both the current process and child processes.  May not be
     available on all systems.

5.35.14 `nis' -- Interface to Sun's NIS (Yellow Pages)
------------------------------------------------------

The nis module gives a thin wrapper around the NIS library,
useful for central administration of several hosts.

  Because NIS exists only on Unix systems, this module is only
available for Unix.

  The nis module defines the following functions:

 -- Function: nis.match (key, mapname[, domain=default_domain])
     Return the match for _key_ in map _mapname_, or raise an error
     (see nis.error) if there is none. Both should be strings,
     _key_ is 8-bit clean. Return value is an arbitrary array of bytes
     (may contain `NULL' and other joys).

     Note that _mapname_ is first checked if it is an alias to another
     name.

     Changed in version 2.5: The _domain_ argument allows to override
     the NIS domain used for the lookup. If unspecified, lookup is in
     the default NIS domain.

 -- Function: nis.cat (mapname[, domain=default_domain])
     Return a dictionary mapping _key_ to _value_ such that `match(key,
     mapname)==value'. Note that both keys and values of the dictionary
     are arbitrary arrays of bytes.

     Note that _mapname_ is first checked if it is an alias to another
     name.

     Changed in version 2.5: The _domain_ argument allows to override
     the NIS domain used for the lookup. If unspecified, lookup is in
     the default NIS domain.

 -- Function: nis.maps ([domain=default_domain])
     Return a list of all valid maps.

     Changed in version 2.5: The _domain_ argument allows to override
     the NIS domain used for the lookup. If unspecified, lookup is in
     the default NIS domain.

 -- Function: nis.get_default_domain ()
     Return the system default NIS domain.

     New in version 2.5.

  The nis module defines the following exception:

 -- Exception: nis.error
     An error raised when a NIS function returns an error code.

5.35.15 `syslog' -- Unix syslog library routines
------------------------------------------------

This module provides an interface to the Unix `syslog' library routines.
Refer to the Unix manual pages for a detailed description of the
`syslog' facility.

  This module wraps the system `syslog' family of routines.  A pure
Python library that can speak to a syslog server is available in the
`logging.handlers' module as `SysLogHandler'.

  The module defines the following functions:

 -- Function: syslog.syslog ([priority], message)
     Send the string _message_ to the system logger.  A trailing
     newline is added if necessary.  Each message is tagged with a
     priority composed of a _facility_ and a _level_.  The optional
     _priority_ argument, which defaults to `LOG_INFO', determines the
     message priority.  If the facility is not encoded in _priority_
     using logical-or (`LOG_INFO | LOG_USER'), the value given in the
     openlog() call is used.

     If openlog() has not been called prior to the call to
     syslog(), `openlog()' will be called with no arguments.

 -- Function: syslog.openlog ([ident[, logopt[, facility]]])
     Logging options of subsequent syslog() calls can be
     set by calling openlog().  syslog() will
     call openlog() with no arguments if the log is not
     currently open.

     The optional _ident_ keyword argument is a string which is
     prepended to every message, and defaults to `sys.argv[0]' with
     leading path components stripped.  The optional _logopt_ keyword
     argument (default is 0) is a bit field - see below for possible
     values to combine.  The optional _facility_ keyword argument
     (default is `LOG_USER') sets the default facility for messages
     which do not have a facility explicitly encoded.

 -- Function: syslog.closelog ()
     Reset the syslog module values and call the system library
     `closelog()'.

     This causes the module to behave as it does when initially
     imported.  For example, see openlog() will be called on
     the first syslog() call (if openlog().
     hasn't already been called), and _ident_ and other
     openlog() parameters are reset to defaults.

 -- Function: syslog.setlogmask (maskpri)
     Set the priority mask to _maskpri_ and return the previous mask
     value.  Calls to syslog() with a priority level not
     set in _maskpri_ are ignored.  The default is to log all
     priorities.  The function `LOG_MASK(pri)' calculates the mask for
     the individual priority _pri_.  The function `LOG_UPTO(pri)'
     calculates the mask for all priorities up to and including _pri_.

  The module defines the following constants:

Priority levels (high to low):
     `LOG_EMERG', `LOG_ALERT', `LOG_CRIT', `LOG_ERR', `LOG_WARNING',
     `LOG_NOTICE', `LOG_INFO', `LOG_DEBUG'.

Facilities:
     `LOG_KERN', `LOG_USER', `LOG_MAIL', `LOG_DAEMON', `LOG_AUTH',
     `LOG_LPR', `LOG_NEWS', `LOG_UUCP', `LOG_CRON' and `LOG_LOCAL0' to
     `LOG_LOCAL7'.

Log options:
     `LOG_PID', `LOG_CONS', `LOG_NDELAY', `LOG_NOWAIT' and `LOG_PERROR'
     if defined in `<syslog.h>'.

5.35.15.1 Examples
..................

5.35.15.2 Simple example
........................

A simple set of examples:

    import syslog

    syslog.syslog('Processing started')
    if error:
        syslog.syslog(syslog.LOG_ERR, 'Processing started')

An example of setting some log options, these would include the process
ID in logged messages, and write the messages to the destination
facility used for mail logging:

    syslog.openlog(logopt=syslog.LOG_PID, facility=syslog.LOG_MAIL)
    syslog.syslog('E-mail processing initiated...')

5.35.16 `commands' -- Utilities for running commands
----------------------------------------------------

Deprecated since version 2.6: The commands module has been
removed in Python 3.0.  Use the subprocess module instead.

  The commands module contains wrapper functions for
os.popen() which take a system command as a string and return any
output generated by the command and, optionally, the exit status.

  The subprocess module provides more powerful facilities
for spawning new processes and retrieving their results.  Using the
subprocess module is preferable to using the
commands module.

     Note: In Python 3.x, see getstatus() and two undocumented
     functions (`mk2arg()' and `mkarg()') have been removed.  Also,
     see getstatusoutput() and getoutput() have
     been moved to the subprocess module.

  The commands module defines the following functions:

 -- Function: commands.getstatusoutput (cmd)
     Execute the string _cmd_ in a shell with os.popen().
     and return a 2-tuple `(status, output)'.  _cmd_ is actually run as
     `{ cmd ; } 2>&1', so that the returned output will contain output
     or error messages. A trailing newline is stripped from the output.
     The exit status for the command can be interpreted according to
     the rules for the C function `wait()'.

 -- Function: commands.getoutput (cmd)
     Like getstatusoutput(), except the exit status is
     ignored and the return value is a string containing the command's
     output.

 -- Function: commands.getstatus (file)
     Return the output of `ls -ld file' as a string.  This function
     uses the getoutput() function, and properly escapes
     backslashes and dollar signs in the argument.

     Deprecated since version 2.6: This function is nonobvious and
     useless.  The name is also misleading in the presence of
     getstatusoutput().

  Example:

    >>> import commands
    >>> commands.getstatusoutput('ls /bin/ls')
    (0, '/bin/ls')
    >>> commands.getstatusoutput('cat /bin/junk')
    (256, 'cat: /bin/junk: No such file or directory')
    >>> commands.getstatusoutput('/bin/junk')
    (256, 'sh: /bin/junk: not found')
    >>> commands.getoutput('ls /bin/ls')
    '/bin/ls'
    >>> commands.getstatus('/bin/ls')
    '-rwxr-xr-x  1 root        13352 Oct 14  1994 /bin/ls'

Module subprocess.
     Module for spawning and managing subprocesses.

5.36 Mac OS X specific services
===============================

This chapter describes modules that are only available on the Mac OS X
platform.

  See the chapters MacPython OSA Modules and
Undocumented Mac OS modules for more modules, and the HOWTO
Using Python on a Macintosh for a general introduction to
Mac-specific Python programming.

     Note: These modules are deprecated and have been removed in Python
     3.x.

5.36.1 `ic' -- Access to the Mac OS X Internet Config
-----------------------------------------------------

This module provides access to various internet-related preferences set
through *System Preferences* or the *Finder*.

     Note: This module has been removed in Python 3.x.

  There is a low-level companion module `icglue' which provides the
basic Internet Config access functionality.  This low-level module is
not documented, but the docstrings of the routines document the
parameters and the routine names are the same as for the Pascal or C
API to Internet Config, so the standard IC programmers' documentation
can be used if this module is needed.

  The ic module defines the error exception and
symbolic names for all error codes Internet Config can produce; see the
source for details.

 -- Exception: ic.error
     Exception raised on errors in the ic module.

  The ic module defines the following class and function:

 -- Class: ic.IC ([signature[, ic]])
     Create an Internet Config object. The signature is a 4-character
     creator code of the current application (default `'Pyth'') which
     may influence some of ICs settings. The optional _ic_ argument is
     a low-level `icglue.icinstance' created beforehand, this may be
     useful if you want to get preferences from a different config
     file, etc.

 -- Function: ic.launchurl (url[, hint])
 -- Function: ic.parseurl (data[, start[, end[, hint]]])
 -- Function: ic.mapfile (file)
 -- Function: ic.maptypecreator (type, creator[, filename])
 -- Function: ic.settypecreator (file)
     These functions are "shortcuts" to the methods of the same name,
     described below.

5.36.1.1 IC Objects
...................

IC objects have a mapping interface, hence to obtain the
mail address you simply get `ic['MailAddress']'. Assignment also works,
and changes the option in the configuration file.

  The module knows about various datatypes, and converts the internal IC
representation to a "logical" Python data structure. Running the
ic.  module standalone will run a test program that lists all keys
and values in your IC database, this will have to serve as
documentation.

  If the module does not know how to represent the data it returns an
instance of the `ICOpaqueData' type, with the raw data in its `data'
attribute.  Objects of this type are also acceptable values for
assignment.

  Besides the dictionary interface, see IC objects have the
following methods:

 -- Method: IC.launchurl (url[, hint])
     Parse the given URL, launch the correct application and pass it
     the URL. The optional _hint_ can be a scheme name such as
     `'mailto:'', in which case incomplete URLs are completed with this
     scheme.  If _hint_ is not provided, incomplete URLs are invalid.

 -- Method: IC.parseurl (data[, start[, end[, hint]]])
     Find an URL somewhere in _data_ and return start position, end
     position and the URL. The optional _start_ and _end_ can be used
     to limit the search, so for instance if a user clicks in a long
     text field you can pass the whole text field and the
     click-position in _start_ and this routine will return the whole
     URL in which the user clicked.  As above, _hint_ is an optional
     scheme used to complete incomplete URLs.

 -- Method: IC.mapfile (file)
     Return the mapping entry for the given _file_, which can be passed
     as either a filename or an `FSSpec()' result, and which need not
     exist.

     The mapping entry is returned as a tuple `(version, type, creator,
     postcreator, flags, extension, appname, postappname, mimetype,
     entryname)', where _version_ is the entry version number, _type_
     is the 4-character filetype, _creator_ is the 4-character creator
     type, _postcreator_ is the 4-character creator code of an optional
     application to post-process the file after downloading, _flags_ are
     various bits specifying whether to transfer in binary or ascii and
     such, _extension_ is the filename extension for this file type,
     _appname_ is the printable name of the application to which this
     file belongs, _postappname_ is the name of the postprocessing
     application, _mimetype_ is the MIME type of this file and
     _entryname_ is the name of this entry.

 -- Method: IC.maptypecreator (type, creator[, filename])
     Return the mapping entry for files with given 4-character _type_
     and _creator_ codes. The optional _filename_ may be specified to
     further help finding the correct entry (if the creator code is
     `'????'', for instance).

     The mapping entry is returned in the same format as for _mapfile_.

 -- Method: IC.settypecreator (file)
     Given an existing _file_, specified either as a filename or as an
     `FSSpec()' result, set its creator and type correctly based on its
     extension.  The finder is told about the change, so the finder
     icon will be updated quickly.

5.36.2 `MacOS' -- Access to Mac OS interpreter features
-------------------------------------------------------

This module provides access to MacOS specific functionality in the
Python interpreter, such as how the interpreter eventloop functions and
the like. Use with care.

     Note: This module has been removed in Python 3.x.

  Note the capitalization of the module name; this is a historical
artifact.

 -- Data: MacOS.runtimemodel
     Always `'macho'', from Python 2.4 on. In earlier versions of
     Python the value could also be `'ppc'' for the classic Mac OS 8
     runtime model or `'carbon'' for the Mac OS 9 runtime model.

 -- Data: MacOS.linkmodel
     The way the interpreter has been linked. As extension modules may
     be incompatible between linking models, packages could use this
     information to give more decent error messages. The value is one
     of `'static'' for a statically linked Python, `'framework'' for
     Python in a Mac OS X framework, `'shared'' for Python in a
     standard Unix shared library. Older Pythons could also have the
     value `'cfm'' for Mac OS 9-compatible Python.

 -- Exception: MacOS.Error
     This exception is raised on MacOS generated errors, either from
     functions in this module or from other mac-specific modules like
     the toolbox interfaces. The arguments are the integer error code
     (the `OSErr' value) and a textual description of the error code.
     Symbolic names for all known error codes are defined in the
     standard module macerrors.

 -- Function: MacOS.GetErrorString (errno)
     Return the textual description of MacOS error code _errno_.

 -- Function: MacOS.DebugStr (message[, object])
     On Mac OS X the string is simply printed to stderr (on older Mac
     OS systems more elaborate functionality was available), but it
     provides a convenient location to attach a breakpoint in a
     low-level debugger like *gdb*.

          Note: Not available in 64-bit mode.

 -- Function: MacOS.SysBeep ()
     Ring the bell.

          Note: Not available in 64-bit mode.

 -- Function: MacOS.GetTicks ()
     Get the number of clock ticks (1/60th of a second) since system
     boot.

 -- Function: MacOS.GetCreatorAndType (file)
     Return the file creator and file type as two four-character
     strings. The _file_ parameter can be a pathname or an `FSSpec' or
     `FSRef' object.

          Note: It is not possible to use an `FSSpec' in 64-bit mode.

 -- Function: MacOS.SetCreatorAndType (file, creator, type)
     Set the file creator and file type. The _file_ parameter can be a
     pathname or an `FSSpec' or  `FSRef' object. _creator_ and _type_
     must be four character strings.

          Note: It is not possible to use an `FSSpec' in 64-bit mode.

 -- Function: MacOS.openrf (name[, mode])
     Open the resource fork of a file. Arguments are the same as for
     the built-in function open(). The object returned has
     file-like semantics, but it is not a Python file object, so there
     may be subtle differences.

 -- Function: MacOS.WMAvailable ()
     Checks whether the current process has access to the window
     manager. The method will return `False' if the window manager is
     not available, for instance when running on Mac OS X Server or
     when logged in via ssh, or when the current interpreter is not
     running from a fullblown application bundle. A script runs from an
     application bundle either when it has been started with *pythonw*
     instead of *python* or when running  as an applet.

 -- Function: MacOS.splash ([resourceid])
     Opens a splash screen by resource id. Use resourceid `0' to close
     the splash screen.

          Note: Not available in 64-bit mode.

5.36.3 `macostools' -- Convenience routines for file manipulation
-----------------------------------------------------------------

This module contains some convenience routines for file-manipulation on
the Macintosh. All file parameters can be specified as pathnames,
`FSRef' or `FSSpec' objects.  This module expects a filesystem which
supports forked files, so it should not be used on UFS partitions.

     Note: This module has been removed in Python 3.0.

  The macostools module defines the following functions:

 -- Function: macostools.copy (src, dst[, createpath[, copytimes]])
     Copy file _src_ to _dst_.  If _createpath_ is non-zero the folders
     leading to _dst_ are created if necessary. The method copies data
     and resource fork and some finder information (creator, type,
     flags) and optionally the creation, modification and backup times
     (default is to copy them). Custom icons, comments and icon
     position are not copied.

          Note: This function does not work in 64-bit code because it
          uses APIs that are not available in 64-bit mode.

 -- Function: macostools.copytree (src, dst)
     Recursively copy a file tree from _src_ to _dst_, creating folders
     as needed.  _src_ and _dst_ should be specified as pathnames.

          Note: This function does not work in 64-bit code because it
          uses APIs that are not available in 64-bit mode.

 -- Function: macostools.mkalias (src, dst)
     Create a finder alias _dst_ pointing to _src_.

          Note: This function does not work in 64-bit code because it
          uses APIs that are not available in 64-bit mode.

 -- Function: macostools.touched (dst)
     Tell the finder that some bits of finder-information such as
     creator or type for file _dst_ has changed. The file can be
     specified by pathname or fsspec. This call should tell the finder
     to redraw the files icon.

     Deprecated since version 2.6: The function is a no-op on OS X.

 -- Data: macostools.BUFSIZ
     The buffer size for `copy', default 1 megabyte.

  Note that the process of creating finder aliases is not specified in
the Apple documentation. Hence, aliases created with mkalias()
could conceivably have incompatible behaviour in some cases.

5.36.4 `findertools' -- The *finder*'s Apple Events interface
-------------------------------------------------------------

This module contains routines that give Python programs access to some
functionality provided by the finder. They are implemented as wrappers
around the AppleEvent interface to the finder.

  All file and folder parameters can be specified either as full
pathnames, or as `FSRef' or `FSSpec' objects.

  The findertools module defines the following functions:

 -- Function: findertools.launch (file)
     Tell the finder to launch _file_. What launching means depends on
     the file: applications are started, folders are opened and
     documents are opened in the correct application.

 -- Function: findertools.Print (file)
     Tell the finder to print a file. The behaviour is identical to
     selecting the file and using the print command in the finder's
     file menu.

 -- Function: findertools.copy (file, destdir)
     Tell the finder to copy a file or folder _file_ to folder
     _destdir_. The function returns an `Alias' object pointing to the
     new file.

 -- Function: findertools.move (file, destdir)
     Tell the finder to move a file or folder _file_ to folder
     _destdir_. The function returns an `Alias' object pointing to the
     new file.

 -- Function: findertools.sleep ()
     Tell the finder to put the Macintosh to sleep, if your machine
     supports it.

 -- Function: findertools.restart ()
     Tell the finder to perform an orderly restart of the machine.

 -- Function: findertools.shutdown ()
     Tell the finder to perform an orderly shutdown of the machine.

5.36.5 `EasyDialogs' -- Basic Macintosh dialogs
-----------------------------------------------

The EasyDialogs module contains some simple dialogs for the
Macintosh.  The dialogs get launched in a separate application which
appears in the dock and must be clicked on for the dialogs be
displayed.  All routines take an optional resource ID parameter _id_
with which one can override the `DLOG' resource used for the dialog,
provided that the dialog items correspond (both type and item number)
to those in the default `DLOG' resource. See source code for details.

     Note: This module has been removed in Python 3.x.

  The EasyDialogs module defines the following functions:

 -- Function: EasyDialogs.Message (str[, id[, ok]])
     Displays a modal dialog with the message text _str_, which should
     be at most 255 characters long. The button text defaults to "OK",
     but is set to the string argument _ok_ if the latter is supplied.
     Control is returned when the user clicks the "OK" button.

 -- Function: EasyDialogs.AskString (prompt[, default[, id[, ok[,
          cancel]]]])
     Asks the user to input a string value via a modal dialog. _prompt_
     is the prompt message, and the optional _default_ supplies the
     initial value for the string (otherwise `""' is used). The text of
     the "OK" and "Cancel" buttons can be changed with the _ok_ and
     _cancel_ arguments. All strings can be at most 255 bytes long.
     AskString() returns the string entered or None
     in case the user cancelled.

 -- Function: EasyDialogs.AskPassword (prompt[, default[, id[, ok[,
          cancel]]]])
     Asks the user to input a string value via a modal dialog. Like
     AskString(), but with the text shown as bullets. The
     arguments have the same meaning as for AskString().

 -- Function: EasyDialogs.AskYesNoCancel (question[, default[, yes[,
          no[, cancel[, id]]]]])
     Presents a dialog with prompt _question_ and three buttons
     labelled "Yes", "No", and "Cancel". Returns `1' for "Yes", `0' for
     "No" and `-1' for "Cancel".  The value of _default_ (or `0' if
     _default_ is not supplied) is returned when the `RETURN' key is
     pressed. The text of the buttons can be changed with the _yes_,
     _no_, and _cancel_ arguments; to prevent a button from appearing,
     supply `""' for the corresponding argument.

 -- Function: EasyDialogs.ProgressBar ([title[, maxval[, label[, id]]]])
     Displays a modeless progress-bar dialog. This is the constructor
     for the ProgressBar class described below. _title_ is
     the text string displayed (default "Working..."), _maxval_ is the
     value at which progress is complete (default `0', indicating that
     an indeterminate amount of work remains to be done), and _label_
     is the text that is displayed above the progress bar itself.

 -- Function: EasyDialogs.GetArgv ([optionlist[ commandlist[,
          addoldfile[, addnewfile[, addfolder[, id]]]]]])
     Displays a dialog which aids the user in constructing a
     command-line argument list.  Returns the list in `sys.argv'
     format, suitable for passing as an argument to
     getopt.getopt().  _addoldfile_, _addnewfile_, and _addfolder_
     are boolean arguments.  When nonzero, they enable the user to
     insert into the command line paths to an existing file, a
     (possibly) not-yet-existent file, and a folder, respectively.
     (Note: Option arguments must appear in the command line before
     file and folder arguments in order to be recognized by
     getopt.getopt())  Arguments containing spaces can be
     specified by enclosing them within single or double quotes.  A
     SystemExit exception is raised if the user presses the
     "Cancel" button.

     _optionlist_ is a list that determines a popup menu from which the
     allowed options are selected.  Its items can take one of two
     forms: _optstr_ or `(optstr, descr)'.  When present, _descr_ is a
     short descriptive string that is displayed in the dialog while
     this option is selected in the popup menu.  The correspondence
     between _optstr_s and command-line arguments is:

     _optstr_ format            Command-line format
     --------------------------------------------------------------------------
     `x'                        -x. (short option)
     `x:' or `x='               -x. (short option with value)
     `xyz'                      `--xyz' (long option)
     `xyz:' or `xyz='           `--xyz' (long option with value)

     _commandlist_ is a list of items of the form _cmdstr_ or `(cmdstr,
     descr)', where _descr_ is as above.  The _cmdstr_s will appear in
     a popup menu.  When chosen, the text of _cmdstr_ will be appended
     to the command line as is, except that a trailing `':'' or `'=''
     (if present) will be trimmed off.

     New in version 2.0.

 -- Function: EasyDialogs.AskFileForOpen ( [message] [, typeList] [,
          defaultLocation] [, defaultOptionFlags] [, location] [,
          clientName] [, windowTitle] [, actionButtonLabel] [,
          cancelButtonLabel] [, preferenceKey] [, popupExtension] [,
          eventProc] [, previewProc] [, filterProc] [, wanted] )
     Post a dialog asking the user for a file to open, and return the
     file selected or None if the user cancelled. _message_
     is a text message to display, _typeList_ is a list of 4-char
     filetypes allowable, _defaultLocation_ is the pathname, `FSSpec'
     or `FSRef' of the folder to show initially, _location_ is the `(x,
     y)' position on the screen where the dialog is shown,
     _actionButtonLabel_ is a string to show instead of "Open" in the
     OK button, _cancelButtonLabel_ is a string to show instead of
     "Cancel" in the cancel button, _wanted_ is the type of value
     wanted as a return: str, unicode, `FSSpec',
     `FSRef' and subtypes thereof are acceptable.

     For a description of the other arguments please see the Apple
     Navigation Services documentation and the EasyDialogs.
     source code.

 -- Function: EasyDialogs.AskFileForSave ( [message] [, savedFileName]
          [, defaultLocation] [, defaultOptionFlags] [, location] [,
          clientName] [, windowTitle] [, actionButtonLabel] [,
          cancelButtonLabel] [, preferenceKey] [, popupExtension] [,
          fileType] [, fileCreator] [, eventProc] [, wanted] )
     Post a dialog asking the user for a file to save to, and return
     the file selected or None if the user cancelled.
     _savedFileName_ is the default for the file name to save to (the
     return value). See AskFileForOpen() for a description
     of the other arguments.

 -- Function: EasyDialogs.AskFolder ( [message] [, defaultLocation] [,
          defaultOptionFlags] [, location] [, clientName] [,
          windowTitle] [, actionButtonLabel] [, cancelButtonLabel] [,
          preferenceKey] [, popupExtension] [, eventProc] [,
          filterProc] [, wanted] )
     Post a dialog asking the user to select a folder, and return the
     folder selected or None if the user cancelled. See
     AskFileForOpen() for a description of the arguments.

Navigation Services Reference (http://developer.apple.com/documentation/Carbon/Reference/Navigation_Services_Ref/)
     Programmer's reference documentation for the Navigation Services,
     a part of the Carbon framework.

5.36.5.1 ProgressBar Objects
............................

ProgressBar objects provide support for modeless
progress-bar dialogs.  Both determinate (thermometer style) and
indeterminate (barber-pole style) progress bars are supported.  The bar
will be determinate if its maximum value is greater than zero;
otherwise it will be indeterminate.

  Changed in version 2.2: Support for indeterminate-style progress bars
was added.

  The dialog is displayed immediately after creation. If the dialog's
"Cancel" button is pressed, or if `Cmd-.' or `ESC' is typed, the dialog
window is hidden and KeyboardInterrupt is raised (but note
that this response does not occur until the progress bar is next
updated, typically via a call to `inc()' or set()).
Otherwise, the bar remains visible until the ProgressBar.
object is discarded.

  ProgressBar objects possess the following attributes and
methods:

 -- Attribute: ProgressBar.curval
     The current value (of type integer or long integer) of the
     progress bar.  The normal access methods coerce curval
     between `0' and maxval.  This attribute should
     not be altered directly.

 -- Attribute: ProgressBar.maxval
     The maximum value (of type integer or long integer) of the
     progress bar; the progress bar (thermometer style) is full when
     curval equals maxval.  If maxval
     is `0', the bar will be indeterminate (barber-pole).  This
     attribute should not be altered directly.

 -- Method: ProgressBar.title ([newstr])
     Sets the text in the title bar of the progress dialog to _newstr_.

 -- Method: ProgressBar.label ([newstr])
     Sets the text in the progress box of the progress dialog to
     _newstr_.

 -- Method: ProgressBar.set (value[, max])
     Sets the progress bar's curval to _value_, and also
     maxval to _max_ if the latter is provided.  _value_
     is first coerced between 0 and maxval.  The
     thermometer bar is updated to reflect the changes, including a
     change from indeterminate to determinate or vice versa.

 -- Method: ProgressBar.inc ([n])
     Increments the progress bar's curval by _n_, or by
     `1' if _n_ is not provided.  (Note that _n_ may be negative, in
     which case the effect is a decrement.)  The progress bar is
     updated to reflect the change.  If the bar is indeterminate, this
     causes one "spin" of the barber pole.  The resulting curval
     is coerced between 0 and maxval if incrementing
     causes it to fall outside this range.

5.36.6 `FrameWork' -- Interactive application framework
-------------------------------------------------------

The FrameWork module contains classes that together provide
a framework for an interactive Macintosh application. The programmer
builds an application by creating subclasses that override various
methods of the bases classes, thereby implementing the functionality
wanted. Overriding functionality can often be done on various different
levels, i.e. to handle clicks in a single dialog window in a
non-standard way it is not necessary to override the complete event
handling.

     Note: This module has been removed in Python 3.x.

  Work on the FrameWork has pretty much stopped, now that
`PyObjC' is available for full Cocoa access from Python, and the
documentation describes only the most important functionality, and not
in the most logical manner at that. Examine the source or the examples
for more details.  The following are some comments posted on the
MacPython newsgroup about the strengths and limitations of
FrameWork.:

     The strong point of FrameWork is that it allows you to
     break into the control-flow at many different places. See W,
     for instance, uses a different way to enable/disable menus
     and that plugs right in leaving the rest intact.  The weak points
     of FrameWork are that it has no abstract command
     interface (but that shouldn't be difficult), that its dialog
     support is minimal and that its control/toolbar support is
     non-existent.

  The FrameWork module defines the following functions:

 -- Function: FrameWork.Application ()
     An object representing the complete application. See below for a
     description of the methods. The default __init__().
     routine creates an empty window dictionary and a menu bar with an
     apple menu.

 -- Function: FrameWork.MenuBar ()
     An object representing the menubar. This object is usually not
     created by the user.

 -- Function: FrameWork.Menu (bar, title[, after])
     An object representing a menu. Upon creation you pass the
     `MenuBar' the menu appears in, the _title_ string and a position
     (1-based) _after_ where the menu should appear (default: at the
     end).

 -- Function: FrameWork.MenuItem (menu, title[, shortcut, callback])
     Create a menu item object. The arguments are the menu to create,
     the item title string and optionally the keyboard shortcut and a
     callback routine. The callback is called with the arguments
     menu-id, item number within menu (1-based), current front window
     and the event record.

     Instead of a callable object the callback can also be a string. In
     this case menu selection causes the lookup of a method in the
     topmost window and the application. The method name is the
     callback string with `'domenu_'' prepended.

     Calling the `MenuBar' `fixmenudimstate()' method sets the correct
     dimming for all menu items based on the current front window.

 -- Function: FrameWork.Separator (menu)
     Add a separator to the end of a menu.

 -- Function: FrameWork.SubMenu (menu, label)
     Create a submenu named _label_ under menu _menu_. The menu object
     is returned.

 -- Function: FrameWork.Window (parent)
     Creates a (modeless) window. _Parent_ is the application object to
     which the window belongs. The window is not displayed until later.

 -- Function: FrameWork.DialogWindow (parent)
     Creates a modeless dialog window.

 -- Function: FrameWork.windowbounds (width, height)
     Return a `(left, top, right, bottom)' tuple suitable for creation
     of a window of given width and height. The window will be
     staggered with respect to previous windows, and an attempt is made
     to keep the whole window on-screen. However, the window will
     however always be the exact size given, so parts may be offscreen.

 -- Function: FrameWork.setwatchcursor ()
     Set the mouse cursor to a watch.

 -- Function: FrameWork.setarrowcursor ()
     Set the mouse cursor to an arrow.

5.36.6.1 Application Objects
............................

Application objects have the following methods, among others:

 -- Method: Application.makeusermenus ()
     Override this method if you need menus in your application. Append
     the menus to the attribute `menubar'.

 -- Method: Application.getabouttext ()
     Override this method to return a text string describing your
     application.  Alternatively, override the `do_about()' method for
     more elaborate "about" messages.

 -- Method: Application.mainloop ([mask[, wait]])
     This routine is the main event loop, call it to set your
     application rolling.  _Mask_ is the mask of events you want to
     handle, _wait_ is the number of ticks you want to leave to other
     concurrent application (default 0, which is probably not a good
     idea). While raising _self_ to exit the mainloop is still supported
     it is not recommended: call `self._quit()' instead.

     The event loop is split into many small parts, each of which can
     be overridden.  The default methods take care of dispatching
     events to windows and dialogs, handling drags and resizes, Apple
     Events, events for non-FrameWork windows, etc.

     In general, all event handlers should return `1' if the event is
     fully handled and `0' otherwise (because the front window was not
     a FrameWork window, for instance). This is needed so that update
     events and such can be passed on to other windows like the Sioux
     console window. Calling `MacOS.HandleEvent()' is not allowed
     within _our_dispatch_ or its callees, since this may result in an
     infinite loop if the code is called through the Python inner-loop
     event handler.

 -- Method: Application.asyncevents (onoff)
     Call this method with a nonzero parameter to enable asynchronous
     event handling.  This will tell the inner interpreter loop to call
     the application event handler _async_dispatch_ whenever events are
     available. This will cause FrameWork window updates and the user
     interface to remain working during long computations, but will
     slow the interpreter down and may cause surprising results in
     non-reentrant code (such as FrameWork itself). By default
     _async_dispatch_ will immediately call _our_dispatch_ but you may
     override this to handle only certain events asynchronously. Events
     you do not handle will be passed to Sioux and such.

     The old on/off value is returned.

 -- Method: Application._quit ()
     Terminate the running mainloop() call at the next
     convenient moment.

 -- Method: Application.do_char (c, event)
     The user typed character _c_. The complete details of the event
     can be found in the _event_ structure. This method can also be
     provided in a `Window' object, which overrides the
     application-wide handler if the window is frontmost.

 -- Method: Application.do_dialogevent (event)
     Called early in the event loop to handle modeless dialog events.
     The default method simply dispatches the event to the relevant
     dialog (not through the `DialogWindow' object involved). Override
     if you need special handling of dialog events (keyboard shortcuts,
     etc).

 -- Method: Application.idle (event)
     Called by the main event loop when no events are available. The
     null-event is passed (so you can look at mouse position, etc).

5.36.6.2 Window Objects
.......................

Window objects have the following methods, among others:

 -- Method: Window.open ()
     Override this method to open a window. Store the Mac OS window-id
     in `self.wid' and call the `do_postopen()' method to register the
     window with the parent application.

 -- Method: Window.close ()
     Override this method to do any special processing on window close.
     Call the `do_postclose()' method to cleanup the parent state.

 -- Method: Window.do_postresize (width, height, macoswindowid)
     Called after the window is resized. Override if more needs to be
     done than calling `InvalRect'.

 -- Method: Window.do_contentclick (local, modifiers, event)
     The user clicked in the content part of a window. The arguments
     are the coordinates (window-relative), the key modifiers and the
     raw event.

 -- Method: Window.do_update (macoswindowid, event)
     An update event for the window was received. Redraw the window.

 -- Method: Window.do_activate (activate, event)
     The window was activated (`activate == 1') or deactivated
     (`activate == 0').  Handle things like focus highlighting, etc.

5.36.6.3 ControlsWindow Object
..............................

ControlsWindow objects have the following methods besides those of
`Window' objects:

 -- Method: ControlsWindow.do_controlhit (window, control, pcode, event)
     Part _pcode_ of control _control_ was hit by the user. Tracking
     and such has already been taken care of.

5.36.6.4 ScrolledWindow Object
..............................

ScrolledWindow objects are ControlsWindow objects with the following
extra methods:

 -- Method: ScrolledWindow.scrollbars ([wantx[, wanty]])
     Create (or destroy) horizontal and vertical scrollbars. The
     arguments specify which you want (default: both). The scrollbars
     always have minimum `0' and maximum `32767'.

 -- Method: ScrolledWindow.getscrollbarvalues ()
     You must supply this method. It should return a tuple `(x, y)'
     giving the current position of the scrollbars (between `0' and
     `32767'). You can return `None' for either to indicate the whole
     document is visible in that direction.

 -- Method: ScrolledWindow.updatescrollbars ()
     Call this method when the document has changed. It will call
     getscrollbarvalues() and update the scrollbars.

 -- Method: ScrolledWindow.scrollbar_callback (which, what, value)
     Supplied by you and called after user interaction. _which_ will be
     `'x'' or `'y'', _what_ will be `'-'', `'--'', `'set'', `'++'' or
     `'+''. For `'set'', _value_ will contain the new scrollbar
     position.

 -- Method: ScrolledWindow.scalebarvalues (absmin, absmax, curmin,
          curmax)
     Auxiliary method to help you calculate values to return from
     getscrollbarvalues(). You pass document minimum and maximum
     value and topmost (leftmost) and bottommost (rightmost) visible
     values and it returns the correct number or `None'.

 -- Method: ScrolledWindow.do_activate (onoff, event)
     Takes care of dimming/highlighting scrollbars when a window
     becomes frontmost.  If you override this method, call this one at
     the end of your method.

 -- Method: ScrolledWindow.do_postresize (width, height, window)
     Moves scrollbars to the correct position. Call this method
     initially if you override it.

 -- Method: ScrolledWindow.do_controlhit (window, control, pcode, event)
     Handles scrollbar interaction. If you override it call this method
     first, a nonzero return value indicates the hit was in the
     scrollbars and has been handled.

5.36.6.5 DialogWindow Objects
.............................

DialogWindow objects have the following methods besides those of
`Window' objects:

 -- Method: DialogWindow.open (resid)
     Create the dialog window, from the DLOG resource with id _resid_.
     The dialog object is stored in `self.wid'.

 -- Method: DialogWindow.do_itemhit (item, event)
     Item number _item_ was hit. You are responsible for redrawing
     toggle buttons, etc.

5.36.7 `autoGIL' -- Global Interpreter Lock handling in event loops
-------------------------------------------------------------------

The autoGIL module provides a function
installAutoGIL() that automatically locks and unlocks Python's
Global Interpreter Lock when running an event loop.

     Note: This module has been removed in Python 3.x.

 -- Exception: autoGIL.AutoGILError
     Raised if the observer callback cannot be installed, for example
     because the current thread does not have a run loop.

 -- Function: autoGIL.installAutoGIL ()
     Install an observer callback in the event loop (CFRunLoop) for the
     current thread, that will lock and unlock the Global Interpreter
     Lock (GIL) at appropriate times, allowing other Python threads to
     run while the event loop is idle.

     Availability: OSX 10.1 or later.

5.36.8 Mac OS Toolbox Modules
-----------------------------

There are a set of modules that provide interfaces to various Mac OS
toolboxes.  If applicable the module will define a number of Python
objects for the various structures declared by the toolbox, and
operations will be implemented as methods of the object.  Other
operations will be implemented as functions in the module.  Not all
operations possible in C will also be possible in Python (callbacks are
often a problem), and parameters will occasionally be different in
Python (input and output buffers, especially).  All methods and
functions have a `__doc__' string describing their arguments and return
values, and for additional description you are referred to Inside
Macintosh (http://developer.apple.com/documentation/macos8/mac8.html)or
similar works.

  These modules all live in a package called `Carbon'. Despite that
name they are not all part of the Carbon framework: CF is really in the
CoreFoundation framework and Qt is in the QuickTime framework. The
normal use pattern is

    from Carbon import AE

     Note: The Carbon modules have been removed in Python 3.0.

5.36.8.1 `Carbon.AE' -- Apple Events
....................................

5.36.8.2 `Carbon.AH' -- Apple Help
..................................

5.36.8.3 `Carbon.App' -- Appearance Manager
...........................................

5.36.8.4 `Carbon.Appearance' -- Appearance Manager constants
............................................................

5.36.8.5 `Carbon.CF' -- Core Foundation
.......................................

The `CFBase', `CFArray', `CFData', `CFDictionary', `CFString' and
`CFURL' objects are supported, some only partially.

5.36.8.6 `Carbon.CG' -- Core Graphics
.....................................

5.36.8.7 `Carbon.CarbonEvt' -- Carbon Event Manager
...................................................

5.36.8.8 `Carbon.CarbonEvents' -- Carbon Event Manager constants
................................................................

5.36.8.9 `Carbon.Cm' -- Component Manager
.........................................

5.36.8.10 `Carbon.Components' -- Component Manager constants
............................................................

5.36.8.11 `Carbon.ControlAccessor' -- Control Manager accssors
..............................................................

5.36.8.12 `Carbon.Controls' -- Control Manager constants
........................................................

5.36.8.13 `Carbon.CoreFounation' -- CoreFounation constants
...........................................................

5.36.8.14 `Carbon.CoreGraphics' -- CoreGraphics constants
.........................................................

5.36.8.15 `Carbon.Ctl' -- Control Manager
.........................................

5.36.8.16 `Carbon.Dialogs' -- Dialog Manager constants
......................................................

5.36.8.17 `Carbon.Dlg' -- Dialog Manager
........................................

5.36.8.18 `Carbon.Drag' -- Drag and Drop Manager
................................................

5.36.8.19 `Carbon.Dragconst' -- Drag and Drop Manager constants
...............................................................

5.36.8.20 `Carbon.Events' -- Event Manager constants
....................................................

5.36.8.21 `Carbon.Evt' -- Event Manager
.......................................

5.36.8.22 `Carbon.File' -- File Manager
.......................................

5.36.8.23 `Carbon.Files' -- File Manager constants
..................................................

5.36.8.24 `Carbon.Fm' -- Font Manager
.....................................

5.36.8.25 `Carbon.Folder' -- Folder Manager
...........................................

5.36.8.26 `Carbon.Folders' -- Folder Manager constants
......................................................

5.36.8.27 `Carbon.Fonts' -- Font Manager constants
..................................................

5.36.8.28 `Carbon.Help' -- Help Manager
.......................................

5.36.8.29 `Carbon.IBCarbon' -- Carbon InterfaceBuilder
......................................................

5.36.8.30 `Carbon.IBCarbonRuntime' -- Carbon InterfaceBuilder constants
.......................................................................

5.36.8.31 `Carbon.Icn' -- Carbon Icon Manager
.............................................

5.36.8.32 `Carbon.Icons' -- Carbon Icon Manager constants
.........................................................

5.36.8.33 `Carbon.Launch' -- Carbon Launch Services
...................................................

5.36.8.34 `Carbon.LaunchServices' -- Carbon Launch Services constants
.....................................................................

5.36.8.35 `Carbon.List' -- List Manager
.......................................

5.36.8.36 `Carbon.Lists' -- List Manager constants
..................................................

5.36.8.37 `Carbon.MacHelp' -- Help Manager constants
....................................................

5.36.8.38 `Carbon.MediaDescr' -- Parsers and generators for Quicktime Media descriptors
.......................................................................................

5.36.8.39 `Carbon.Menu' -- Menu Manager
.......................................

5.36.8.40 `Carbon.Menus' -- Menu Manager constants
..................................................

5.36.8.41 `Carbon.Mlte' -- MultiLingual Text Editor
...................................................

5.36.8.42 `Carbon.OSA' -- Carbon OSA Interface
..............................................

5.36.8.43 `Carbon.OSAconst' -- Carbon OSA Interface constants
.............................................................

5.36.8.44 `Carbon.QDOffscreen' -- QuickDraw Offscreen constants
...............................................................

5.36.8.45 `Carbon.Qd' -- QuickDraw
..................................

5.36.8.46 `Carbon.Qdoffs' -- QuickDraw Offscreen
................................................

5.36.8.47 `Carbon.Qt' -- QuickTime
..................................

5.36.8.48 `Carbon.QuickDraw' -- QuickDraw constants
...................................................

5.36.8.49 `Carbon.QuickTime' -- QuickTime constants
...................................................

5.36.8.50 `Carbon.Res' -- Resource Manager and Handles
......................................................

5.36.8.51 `Carbon.Resources' -- Resource Manager and Handles constants
......................................................................

5.36.8.52 `Carbon.Scrap' -- Scrap Manager
.........................................

This module is only fully available on Mac OS 9 and earlier under
classic PPC MacPython.  Very limited functionality is available under
Carbon MacPython.

  The Scrap Manager supports the simplest form of cut & paste
operations on the Macintosh.  It can be use for both inter- and
intra-application clipboard operations.

  The `Scrap' module provides low-level access to the functions of the
Scrap Manager.  It contains the following functions:

 -- Function: Carbon.Scrap.InfoScrap ()
     Return current information about the scrap.  The information is
     encoded as a tuple containing the fields `(size, handle, count,
     state, path)'.

     Field          Meaning
     -----------------------------------------------------------------
     _size_         Size of the scrap in bytes.
     _handle_       Resource object representing the scrap.
     _count_        Serial number of the scrap contents.
     _state_        Integer; positive if in memory, `0' if on disk,
                    negative if uninitialized.
     _path_         Filename of the scrap when stored on disk.

Scrap Manager (http://developer.apple.com/documentation/mac/MoreToolbox/MoreToolbox-109.html)
     Apple's documentation for the Scrap Manager gives a lot of useful
     information about using the Scrap Manager in applications.

5.36.8.53 `Carbon.Snd' -- Sound Manager
.......................................

5.36.8.54 `Carbon.Sound' -- Sound Manager constants
...................................................

5.36.8.55 `Carbon.TE' -- TextEdit
.................................

5.36.8.56 `Carbon.TextEdit' -- TextEdit constants
.................................................

5.36.8.57 `Carbon.Win' -- Window Manager
........................................

5.36.8.58 `Carbon.Windows' -- Window Manager constants
......................................................

5.36.9 `ColorPicker' -- Color selection dialog
----------------------------------------------

The ColorPicker module provides access to the standard color
picker dialog.

     Note: This module has been removed in Python 3.x.

 -- Function: ColorPicker.GetColor (prompt, rgb)
     Show a standard color selection dialog and allow the user to
     select a color.  The user is given instruction by the _prompt_
     string, and the default color is set to _rgb_.  _rgb_ must be a
     tuple giving the red, green, and blue components of the color.
     GetColor() returns a tuple giving the user's selected
     color and a flag indicating whether they accepted the selection of
     cancelled.

5.37 MacPython OSA Modules
==========================

This chapter describes the current implementation of the Open Scripting
Architecture (OSA, also commonly referred to as AppleScript) for Python,
allowing you to control scriptable applications from your Python
program, and with a fairly pythonic interface. Development on this set
of modules has stopped. For more up-to-date implementation of
AppleScript support for Python, see the third-party py-appscript
project: <<http://pypi.python.org/pypi/appscript/>>.

  For a description of the various components of AppleScript and OSA,
and to get an understanding of the architecture and terminology, you
should read Apple's documentation. The "Applescript Language Guide"
explains the conceptual model and the terminology, and documents the
standard suite. The "Open Scripting Architecture" document explains how
to use OSA from an application programmers point of view. In the Apple
Help Viewer these books are located in the Developer Documentation,
Core Technologies section.

  As an example of scripting an application, the following piece of
AppleScript will get the name of the frontmost *Finder* window and
print it:

    tell application "Finder"
        get name of window 1
    end tell

In Python, the following code fragment will do the same:

    import Finder

    f = Finder.Finder()
    print f.get(f.window(1).name)

As distributed the Python library includes packages that implement the
standard suites, plus packages that interface to a small number of
common applications.

  To send AppleEvents to an application you must first create the
Python package interfacing to the terminology of the application (what
*Script Editor* calls the "Dictionary"). This can be done from within
the *PythonIDE* or by running the `gensuitemodule.py' module as a
standalone program from the command line.

  The generated output is a package with a number of modules, one for
every suite used in the program plus an `__init__' module to glue it
all together. The Python inheritance graph follows the AppleScript
inheritance graph, so if a program's dictionary specifies that it
includes support for the Standard Suite, but extends one or two verbs
with extra arguments then the output suite will contain a module
`Standard_Suite' that imports and re-exports everything from
`StdSuites.Standard_Suite' but overrides the methods that have extra
functionality. The output of gensuitemodule is pretty
readable, and contains the documentation that was in the original
AppleScript dictionary in Python docstrings, so reading it is a good
source of documentation.

  The output package implements a main class with the same name as the
package which contains all the AppleScript verbs as methods, with the
direct object as the first argument and all optional parameters as
keyword arguments. AppleScript classes are also implemented as Python
classes, as are comparisons and all the other thingies.

  The main Python class implementing the verbs also allows access to the
properties and elements declared in the AppleScript class
"application". In the current release that is as far as the object
orientation goes, so in the example above we need to use
`f.get(f.window(1).name)' instead of the more Pythonic
`f.window(1).name.get()'.

  If an AppleScript identifier is not a Python identifier the name is
mangled according to a small number of rules:

   * spaces are replaced with underscores

   * other non-alphanumeric characters are replaced with `_xx_' where
     `xx' is the hexadecimal character value

   * any Python reserved word gets an underscore appended

  Python also has support for creating scriptable applications in
Python, but The following modules are relevant to MacPython AppleScript
support:

5.37.1 `gensuitemodule' -- Generate OSA stub packages
-----------------------------------------------------

The gensuitemodule module creates a Python package
implementing stub code for the AppleScript suites that are implemented
by a specific application, according to its AppleScript dictionary.

  It is usually invoked by the user through the *PythonIDE*, but it can
also be run as a script from the command line (pass -help.
for help on the options) or imported from Python code. For an example
of its use see `Mac/scripts/genallsuites.py' in a source distribution,
which generates the stub packages that are included in the standard
library.

  It defines the following public functions:

 -- Function: gensuitemodule.is_scriptable (application)
     Returns true if `application', which should be passed as a
     pathname, appears to be scriptable. Take the return value with a
     grain of salt: *Internet Explorer* appears not to be scriptable
     but definitely is.

 -- Function: gensuitemodule.processfile (application[, output,
          basepkgname, edit_modnames, creatorsignature, dump, verbose])
     Create a stub package for `application', which should be passed as
     a full pathname. For a `.app' bundle this is the pathname to the
     bundle, not to the executable inside the bundle; for an unbundled
     CFM application you pass the filename of the application binary.

     This function asks the application for its OSA terminology
     resources, decodes these resources and uses the resultant data to
     create the Python code for the package implementing the client
     stubs.

     `output' is the pathname where the resulting package is stored, if
     not specified a standard "save file as" dialog is presented to the
     user.  `basepkgname' is the base package on which this package
     will build, and defaults to `StdSuites'. Only when generating
     `StdSuites' itself do you need to specify this. `edit_modnames' is
     a dictionary that can be used to change modulenames that are too
     ugly after name mangling. `creator_signature' can be used to
     override the 4-char creator code, which is normally obtained from
     the `PkgInfo' file in the package or from the CFM file creator
     signature.  When `dump' is given it should refer to a file object,
     and `processfile' will stop after decoding the resources and dump
     the Python representation of the terminology resources to this
     file. `verbose' should also be a file object, and specifying it
     will cause `processfile' to tell you what it is doing.

 -- Function: gensuitemodule.processfile_fromresource (application[,
          output, basepkgname, edit_modnames, creatorsignature, dump,
          verbose])
     This function does the same as `processfile', except that it uses
     a different method to get the terminology resources. It opens
     `application' as a resource file and reads all `"aete"' and
     `"aeut"' resources from this file.

5.37.2 `aetools' -- OSA client support
--------------------------------------

The aetools module contains the basic functionality on which
Python AppleScript client support is built. It also imports and
re-exports the core functionality of the aetypes and
aepack modules. The stub packages generated by
gensuitemodule import the relevant portions of aetools,
so usually you do not need to import it yourself. The exception to this
is when you cannot use a generated suite package and need lower-level
access to scripting.

  The aetools module itself uses the AppleEvent support
provided by the Carbon.AE module. This has one drawback: you
need access to the window manager, see section Running scripts with a GUI
for details. This restriction may be lifted in future
releases.

     Note: This module has been removed in Python 3.x.

  The aetools module defines the following functions:

 -- Function: aetools.packevent (ae, parameters, attributes)
     Stores parameters and attributes in a pre-created
     `Carbon.AE.AEDesc' object.  `parameters' and `attributes' are
     dictionaries mapping 4-character OSA parameter keys to Python
     objects. The objects are packed using `aepack.pack()'.

 -- Function: aetools.unpackevent (ae[, formodulename])
     Recursively unpacks a `Carbon.AE.AEDesc' event to Python objects.
     The function returns the parameter dictionary and the attribute
     dictionary. The `formodulename' argument is used by generated stub
     packages to control where AppleScript classes are looked up.

 -- Function: aetools.keysubst (arguments, keydict)
     Converts a Python keyword argument dictionary `arguments' to the
     format required by `packevent' by replacing the keys, which are
     Python identifiers, by the four-character OSA keys according to
     the mapping specified in `keydict'. Used by the generated suite
     packages.

 -- Function: aetools.enumsubst (arguments, key, edict)
     If the `arguments' dictionary contains an entry for `key' convert
     the value for that entry according to dictionary `edict'. This
     converts human-readable Python enumeration names to the OSA
     4-character codes. Used by the generated suite packages.

  The aetools module defines the following class:

 -- Class: aetools.TalkTo ([signature=None, start=0, timeout=0])
     Base class for the proxy used to talk to an application.
     `signature' overrides the class attribute `_signature' (which is
     usually set by subclasses) and is the 4-char creator code defining
     the application to talk to. `start' can be set to true to enable
     running the application on class instantiation.  `timeout' can be
     specified to change the default timeout used while waiting for an
     AppleEvent reply.

 -- Method: TalkTo._start ()
     Test whether the application is running, and attempt to start it
     if not.

 -- Method: TalkTo.send (code, subcode[, parameters, attributes])
     Create the AppleEvent `Carbon.AE.AEDesc' for the verb with the OSA
     designation `code, subcode' (which are the usual 4-character
     strings), pack the `parameters' and `attributes' into it, send it
     to the target application, wait for the reply, unpack the reply
     with `unpackevent' and return the reply appleevent, the unpacked
     return values as a dictionary and the return attributes.

5.37.3 `aepack' -- Conversion between Python variables and AppleEvent data containers
-------------------------------------------------------------------------------------

The aepack module defines functions for converting (packing)
Python variables to AppleEvent descriptors and back (unpacking). Within
Python the AppleEvent descriptor is handled by Python objects of
built-in type `AEDesc', defined in module Carbon.AE.

     Note: This module has been removed in Python 3.x.

  The aepack module defines the following functions:

 -- Function: aepack.pack (x[, forcetype])
     Returns an `AEDesc' object  containing a conversion of Python
     value x. If _forcetype_ is provided it specifies the descriptor
     type of the result.  Otherwise, a default mapping of Python types
     to Apple Event descriptor types is used, as follows:

     Python type           descriptor type
     --------------------------------------------------------------
     `FSSpec'              typeFSS
     `FSRef'               typeFSRef
     `Alias'               typeAlias
     integer               typeLong (32 bit integer)
     float                 typeFloat (64 bit floating point)
     string                typeText
     unicode               typeUnicodeText
     list                  typeAEList
     dictionary            typeAERecord
     instance              _see below_

     If _x_ is a Python instance then this function attempts to call an
     `__aepack__()' method.  This method should return an `AEDesc'
     object.

     If the conversion _x_ is not defined above, this function returns
     the Python string representation of a value (the repr() function)
     encoded as a text descriptor.

 -- Function: aepack.unpack (x[, formodulename])
     _x_ must be an object of type `AEDesc'. This function returns a
     Python object representation of the data in the Apple Event
     descriptor _x_. Simple AppleEvent data types (integer, text,
     float) are returned as their obvious Python counterparts. Apple
     Event lists are returned as Python lists, and the list elements
     are recursively unpacked.  Object references (ex. `line 3 of
     document 1') are returned as instances of
     aetypes.ObjectSpecifier, unless `formodulename' is
     specified.  AppleEvent descriptors with descriptor type typeFSS
     are returned as `FSSpec' objects.  AppleEvent record descriptors
     are returned as Python dictionaries, with 4-character string keys
     and elements recursively unpacked.

     The optional `formodulename' argument is used by the stub packages
     generated by gensuitemodule, and ensures that the OSA
     classes for object specifiers are looked up in the correct module.
     This ensures that if, say, the Finder returns an object specifier
     for a window you get an instance of `Finder.Window' and not a
     generic `aetypes.Window'. The former knows about all the
     properties and elements a window has in the Finder, while the
     latter knows no such things.

Module Carbon.AE.
     Built-in access to Apple Event Manager routines.

Module aetypes.
     Python definitions of codes for Apple Event descriptor types.

5.37.4 `aetypes' -- AppleEvent objects
--------------------------------------

The aetypes defines classes used to represent Apple Event data
descriptors and Apple Event object specifiers.

  Apple Event data is contained in descriptors, and these descriptors
are typed.  For many descriptors the Python representation is simply
the corresponding Python type: `typeText' in OSA is a Python string,
`typeFloat' is a float, etc. For OSA types that have no direct Python
counterpart this module declares classes. Packing and unpacking
instances of these classes is handled automatically by aepack.

  An object specifier is essentially an address of an object
implemented in a Apple Event server. An Apple Event specifier is used
as the direct object for an Apple Event or as the argument of an
optional parameter. The aetypes.  module contains the base
classes for OSA classes and properties, which are used by the packages
generated by gensuitemodule to populate the classes and
properties in a given suite.

  For reasons of backward compatibility, and for cases where you need
to script an application for which you have not generated the stub
package this module also contains object specifiers for a number of
common OSA classes such as `Document', `Window', `Character', etc.

     Note: This module has been removed in Python 3.x.

  The `AEObjects' module defines the following classes to represent
Apple Event descriptor data:

 -- Class: aetypes.Unknown (type, data)
     The representation of OSA descriptor data for which the
     aepack and aetypes modules have no support, i.e.
     anything that is not represented by the other classes here and
     that is not equivalent to a simple Python value.

 -- Class: aetypes.Enum (enum)
     An enumeration value with the given 4-character string value.

 -- Class: aetypes.InsertionLoc (of, pos)
     Position `pos' in object `of'.

 -- Class: aetypes.Boolean (bool)
     A boolean.

 -- Class: aetypes.StyledText (style, text)
     Text with style information (font, face, etc) included.

 -- Class: aetypes.AEText (script, style, text)
     Text with script system and style information included.

 -- Class: aetypes.IntlText (script, language, text)
     Text with script system and language information included.

 -- Class: aetypes.IntlWritingCode (script, language)
     Script system and language information.

 -- Class: aetypes.QDPoint (v, h)
     A quickdraw point.

 -- Class: aetypes.QDRectangle (v0, h0, v1, h1)
     A quickdraw rectangle.

 -- Class: aetypes.RGBColor (r, g, b)
     A color.

 -- Class: aetypes.Type (type)
     An OSA type value with the given 4-character name.

 -- Class: aetypes.Keyword (name)
     An OSA keyword with the given 4-character name.

 -- Class: aetypes.Range (start, stop)
     A range.

 -- Class: aetypes.Ordinal (abso)
     Non-numeric absolute positions, such as `"firs"', first, or
     `"midd"', middle.

 -- Class: aetypes.Logical (logc, term)
     The logical expression of applying operator `logc' to `term'.

 -- Class: aetypes.Comparison (obj1, relo, obj2)
     The comparison `relo' of `obj1' to `obj2'.

  The following classes are used as base classes by the generated stub
packages to represent AppleScript classes and properties in Python:

 -- Class: aetypes.ComponentItem (which[, fr])
     Abstract baseclass for an OSA class. The subclass should set the
     class attribute `want' to the 4-character OSA class code.
     Instances of subclasses of this class are equivalent to
     AppleScript Object Specifiers. Upon instantiation you should pass
     a selector in `which', and optionally a parent object in `fr'.

 -- Class: aetypes.NProperty (fr)
     Abstract baseclass for an OSA property. The subclass should set
     the class attributes `want' and `which' to designate which
     property we are talking about. Instances of subclasses of this
     class are Object Specifiers.

 -- Class: aetypes.ObjectSpecifier (want, form, seld[, fr])
     Base class of `ComponentItem' and `NProperty', a general OSA Object
     Specifier. See the Apple Open Scripting Architecture documentation
     for the parameters. Note that this class is not abstract.

5.37.5 `MiniAEFrame' -- Open Scripting Architecture server support
------------------------------------------------------------------

The module MiniAEFrame provides a framework for an
application that can function as an Open Scripting Architecture  (OSA)
server, i.e. receive and process AppleEvents. It can be used in
conjunction with FrameWork or standalone. As an example, it
is used in *PythonCGISlave*.

  The MiniAEFrame module defines the following classes:

 -- Class: MiniAEFrame.AEServer
     A class that handles AppleEvent dispatch. Your application should
     subclass this class together with either MiniApplication
     or FrameWork.Application. Your __init__()
     method should call the __init__() method for both
     classes.

 -- Class: MiniAEFrame.MiniApplication
     A class that is more or less compatible with
     FrameWork.Application but with less functionality. Its
     event loop supports the apple menu, command-dot and AppleEvents;
     other events are passed on to the Python interpreter and/or Sioux.
     Useful if your application wants to use AEServer but
     does not provide its own windows, etc.

5.37.5.1 AEServer Objects
.........................

 -- Method: AEServer.installaehandler (classe, type, callback)
     Installs an AppleEvent handler. _classe_ and _type_ are the
     four-character OSA Class and Type designators, `'****'' wildcards
     are allowed. When a matching AppleEvent is received the parameters
     are decoded and your callback is invoked.

 -- Method: AEServer.callback (_object, **kwargs)
     Your callback is called with the OSA Direct Object as first
     positional parameter. The other parameters are passed as keyword
     arguments, with the 4-character designator as name. Three extra
     keyword parameters are passed: `_class' and `_type' are the Class
     and Type designators and `_attributes' is a dictionary with the
     AppleEvent attributes.

     The return value of your method is packed with
     aetools.packevent() and sent as reply.

  Note that there are some serious problems with the current design.
AppleEvents which have non-identifier 4-character designators for
arguments are not implementable, and it is not possible to return an
error to the originator. This will be addressed in a future release.

  In addition, support modules have been pre-generated for `Finder',
`Terminal', `Explorer', `Netscape', `CodeWarrior', `SystemEvents' and
`StdSuites'.

5.38 SGI IRIX Specific Services
===============================

The modules described in this chapter provide interfaces to features
that are unique to SGI's IRIX operating system (versions 4 and 5).

5.38.1 `al' -- Audio functions on the SGI
-----------------------------------------

Deprecated since version 2.6: The al module has been
deprecated for removal in Python 3.0.

  This module provides access to the audio facilities of the SGI Indy
and Indigo workstations.  See section 3A of the IRIX man pages for
details.  You'll need to read those man pages to understand what these
functions do!  Some of the functions are not available in IRIX releases
before 4.0.5.  Again, see the manual to check whether a specific
function is available on your platform.

  All functions and methods defined in this module are equivalent to
the C functions with `AL' prefixed to their name.

  Symbolic constants from the C header file `<audio.h>' are defined in
the standard module AL, see below.

     Warning: The current version of the audio library may dump core
     when bad argument values are passed rather than returning an error
     status.  Unfortunately, since the precise circumstances under
     which this may happen are undocumented and hard to check, the
     Python interface can provide no protection against this kind of
     problems. (One example is specifying an excessive queue size --
     there is no documented upper limit.)

  The module defines the following functions:

 -- Function: al.openport (name, direction[, config])
     The name and direction arguments are strings.  The optional
     _config_ argument is a configuration object as returned by
     newconfig().  The return value is an _audio port object_;
     methods of audio port objects are described below.

 -- Function: al.newconfig ()
     The return value is a new _audio configuration object_; methods of
     audio configuration objects are described below.

 -- Function: al.queryparams (device)
     The device argument is an integer.  The return value is a list of
     integers containing the data returned by `ALqueryparams()'.

 -- Function: al.getparams (device, list)
     The _device_ argument is an integer.  The list argument is a list
     such as returned by queryparams().; it is modified in
     place (!).

 -- Function: al.setparams (device, list)
     The _device_ argument is an integer.  The _list_ argument is a
     list such as returned by queryparams().

5.38.1.1 Configuration Objects
..............................

Configuration objects returned by newconfig() have the
following methods:

 -- Method: audio configuration.getqueuesize()
     Return the queue size.

 -- Method: audio configuration.setqueuesize(size)
     Set the queue size.

 -- Method: audio configuration.getwidth()
     Get the sample width.

 -- Method: audio configuration.setwidth(width)
     Set the sample width.

 -- Method: audio configuration.getchannels()
     Get the channel count.

 -- Method: audio configuration.setchannels(nchannels)
     Set the channel count.

 -- Method: audio configuration.getsampfmt()
     Get the sample format.

 -- Method: audio configuration.setsampfmt(sampfmt)
     Set the sample format.

 -- Method: audio configuration.getfloatmax()
     Get the maximum value for floating sample formats.

 -- Method: audio configuration.setfloatmax(floatmax)
     Set the maximum value for floating sample formats.

5.38.1.2 Port Objects
.....................

Port objects, as returned by openport(), have the following
methods:

 -- Method: audio port.closeport()
     Close the port.

 -- Method: audio port.getfd()
     Return the file descriptor as an int.

 -- Method: audio port.getfilled()
     Return the number of filled samples.

 -- Method: audio port.getfillable()
     Return the number of fillable samples.

 -- Method: audio port.readsamps(nsamples)
     Read a number of samples from the queue, blocking if necessary.
     Return the data as a string containing the raw data, (e.g., 2
     bytes per sample in big-endian byte order (high byte, low byte) if
     you have set the sample width to 2 bytes).

 -- Method: audio port.writesamps(samples)
     Write samples into the queue, blocking if necessary.  The samples
     are encoded as described for the `readsamps()' return value.

 -- Method: audio port.getfillpoint()
     Return the 'fill point'.

 -- Method: audio port.setfillpoint(fillpoint)
     Set the 'fill point'.

 -- Method: audio port.getconfig()
     Return a configuration object containing the current configuration
     of the port.

 -- Method: audio port.setconfig(config)
     Set the configuration from the argument, a configuration object.

 -- Method: audio port.getstatus(list)
     Get status information on last error.

5.38.2 `AL' -- Constants used with the `al' module
--------------------------------------------------

Deprecated since version 2.6: The AL module has been
deprecated for removal in Python 3.0.

  This module defines symbolic constants needed to use the built-in
module al. (see above); they are equivalent to those defined
in the C header file `<audio.h>' except that the name prefix `AL_' is
omitted.  Read the module source for a complete list of the defined
names.  Suggested use:

    import al
    from AL import *

5.38.3 `cd' -- CD-ROM access on SGI systems
-------------------------------------------

Deprecated since version 2.6: The cd module has been
deprecated for removal in Python 3.0.

  This module provides an interface to the Silicon Graphics CD library.
It is available only on Silicon Graphics systems.

  The way the library works is as follows.  A program opens the CD-ROM
device with open() and creates a parser to parse the data
from the CD with createparser().  The object returned by
open() can be used to read data from the CD, but also to
get status information for the CD-ROM device, and to get information
about the CD, such as the table of contents.  Data from the CD is
passed to the parser, which parses the frames, and calls any callback
functions that have previously been added.

  An audio CD is divided into _tracks_ or _programs_ (the terms are used
interchangeably).  Tracks can be subdivided into _indices_.  An audio CD
contains a _table of contents_ which gives the starts of the tracks on
the CD.  Index 0 is usually the pause before the start of a track.  The
start of the track as given by the table of contents is normally the
start of index 1.

  Positions on a CD can be represented in two ways.  Either a frame
number or a tuple of three values, minutes, seconds and frames.  Most
functions use the latter representation.  Positions can be both
relative to the beginning of the CD, and to the beginning of the track.

  Module cd defines the following functions and constants:

 -- Function: cd.createparser ()
     Create and return an opaque parser object.  The methods of the
     parser object are described below.

 -- Function: cd.msftoframe (minutes, seconds, frames)
     Converts a `(minutes, seconds, frames)' triple representing time
     in absolute time code into the corresponding CD frame number.

 -- Function: cd.open ([device[, mode]])
     Open the CD-ROM device.  The return value is an opaque player
     object; methods of the player object are described below.  The
     device is the name of the SCSI device file, e.g.
     `'/dev/scsi/sc0d4l0'', or `None'.  If omitted or `None', the
     hardware inventory is consulted to locate a CD-ROM drive.  The
     _mode_, if not omitted, should be the string `'r''.

  The module defines the following variables:

 -- Exception: cd.error
     Exception raised on various errors.

 -- Data: cd.DATASIZE
     The size of one frame's worth of audio data.  This is the size of
     the audio data as passed to the callback of type `audio'.

 -- Data: cd.BLOCKSIZE
     The size of one uninterpreted frame of audio data.

  The following variables are states as returned by `getstatus()':

 -- Data: cd.READY
     The drive is ready for operation loaded with an audio CD.

 -- Data: cd.NODISC
     The drive does not have a CD loaded.

 -- Data: cd.CDROM
     The drive is loaded with a CD-ROM.  Subsequent play or read
     operations will return I/O errors.

 -- Data: cd.ERROR
     An error occurred while trying to read the disc or its table of
     contents.

 -- Data: cd.PLAYING
     The drive is in CD player mode playing an audio CD through its
     audio jacks.

 -- Data: cd.PAUSED
     The drive is in CD layer mode with play paused.

 -- Data: cd.STILL
     The equivalent of PAUSED on older (non 3301) model
     Toshiba CD-ROM drives.  Such drives have never been shipped by SGI.

 -- Data: cd.audio
 -- Data: cd.pnum
 -- Data: cd.index
 -- Data: cd.ptime
 -- Data: cd.atime
 -- Data: cd.catalog
 -- Data: cd.ident
 -- Data: cd.control
     Integer constants describing the various types of parser callbacks
     that can be set by the `addcallback()' method of CD parser objects
     (see below).

5.38.3.1 Player Objects
.......................

Player objects (returned by open()) have the following
methods:

 -- Method: CD player.allowremoval()
     Unlocks the eject button on the CD-ROM drive permitting the user
     to eject the caddy if desired.

 -- Method: CD player.bestreadsize()
     Returns the best value to use for the _num_frames_ parameter of the
     `readda()' method.  Best is defined as the value that permits a
     continuous flow of data from the CD-ROM drive.

 -- Method: CD player.close()
     Frees the resources associated with the player object.  After
     calling `close()', the methods of the object should no longer be
     used.

 -- Method: CD player.eject()
     Ejects the caddy from the CD-ROM drive.

 -- Method: CD player.getstatus()
     Returns information pertaining to the current state of the CD-ROM
     drive.  The returned information is a tuple with the following
     values: _state_, _track_, _rtime_, _atime_, _ttime_, _first_,
     _last_, _scsi_audio_, _cur_block_. _rtime_ is the time relative to
     the start of the current track; _atime_ is the time relative to
     the beginning of the disc; _ttime_ is the total time on the disc.
     For more information on the meaning of the values, see the man page
     `CDgetstatus(3dm)'. The value of _state_ is one of the following:
     ERROR, NODISC, READY,
     PLAYING, PAUSED, STILL, or
     CDROM.

 -- Method: CD player.gettrackinfo(track)
     Returns information about the specified track.  The returned
     information is a tuple consisting of two elements, the start time
     of the track and the duration of the track.

 -- Method: CD player.msftoblock(min, sec, frame)
     Converts a minutes, seconds, frames triple representing a time in
     absolute time code into the corresponding logical block number for
     the given CD-ROM drive.  You should use msftoframe().
     rather than `msftoblock()' for comparing times.  The logical block
     number differs from the frame number by an offset required by
     certain CD-ROM drives.

 -- Method: CD player.play(start, play)
     Starts playback of an audio CD in the CD-ROM drive at the
     specified track.  The audio output appears on the CD-ROM drive's
     headphone and audio jacks (if fitted).  Play stops at the end of
     the disc. _start_ is the number of the track at which to start
     playing the CD; if _play_ is 0, the CD will be set to an initial
     paused state.  The method `togglepause()' can then be used to
     commence play.

 -- Method: CD player.playabs(minutes, seconds, frames, play)
     Like `play()', except that the start is given in minutes, seconds,
     and frames instead of a track number.

 -- Method: CD player.playtrack(start, play)
     Like `play()', except that playing stops at the end of the track.

 -- Method: CD player.playtrackabs(track, minutes, seconds, frames,
          play)
     Like `play()', except that playing begins at the specified
     absolute time and ends at the end of the specified track.

 -- Method: CD player.preventremoval()
     Locks the eject button on the CD-ROM drive thus preventing the
     user from arbitrarily ejecting the caddy.

 -- Method: CD player.readda(num_frames)
     Reads the specified number of frames from an audio CD mounted in
     the CD-ROM drive.  The return value is a string representing the
     audio frames.  This string can be passed unaltered to the
     `parseframe()' method of the parser object.

 -- Method: CD player.seek(minutes, seconds, frames)
     Sets the pointer that indicates the starting point of the next
     read of digital audio data from a CD-ROM.  The pointer is set to
     an absolute time code location specified in _minutes_, _seconds_,
     and _frames_.  The return value is the logical block number to
     which the pointer has been set.

 -- Method: CD player.seekblock(block)
     Sets the pointer that indicates the starting point of the next
     read of digital audio data from a CD-ROM.  The pointer is set to
     the specified logical block number.  The return value is the
     logical block number to which the pointer has been set.

 -- Method: CD player.seektrack(track)
     Sets the pointer that indicates the starting point of the next
     read of digital audio data from a CD-ROM.  The pointer is set to
     the specified track.  The return value is the logical block number
     to which the pointer has been set.

 -- Method: CD player.stop()
     Stops the current playing operation.

 -- Method: CD player.togglepause()
     Pauses the CD if it is playing, and makes it play if it is paused.

5.38.3.2 Parser Objects
.......................

Parser objects (returned by createparser()) have the
following methods:

 -- Method: CD parser.addcallback(type, func, arg)
     Adds a callback for the parser.  The parser has callbacks for
     eight different types of data in the digital audio data stream.
     Constants for these types are defined at the cd module
     level (see above). The callback is called as follows: `func(arg,
     type, data)', where _arg_ is the user supplied argument, _type_ is
     the particular type of callback, and _data_ is the data returned
     for this _type_ of callback.  The type of the data depends on the
     _type_ of callback as follows:

     Type              Value
     --------------------------------------------------------------------
     `audio'           String which can be passed unmodified to
                       `al.writesamps()'.
     `pnum'            Integer giving the program (track) number.
     `index'           Integer giving the index number.
     `ptime'           Tuple consisting of the program time in minutes,
                       seconds, and frames.
     `atime'           Tuple consisting of the absolute time in
                       minutes, seconds, and frames.
     `catalog'         String of 13 characters, giving the catalog
                       number of the CD.
     `ident'           String of 12 characters, giving the ISRC
                       identification number of the recording.  The
                       string consists of two characters country code,
                       three characters owner code, two characters
                       giving the year, and five characters giving a
                       serial number.
     `control'         Integer giving the control bits from the CD
                       subcode data

 -- Method: CD parser.deleteparser()
     Deletes the parser and frees the memory it was using.  The object
     should not be used after this call.  This call is done
     automatically when the last reference to the object is removed.

 -- Method: CD parser.parseframe(frame)
     Parses one or more frames of digital audio data from a CD such as
     returned by `readda()'.  It determines which subcodes are present
     in the data.  If these subcodes have changed since the last frame,
     then `parseframe()' executes a callback of the appropriate type
     passing to it the subcode data found in the frame. Unlike the C
     function, more than one frame of digital audio data can be passed
     to this method.

 -- Method: CD parser.removecallback(type)
     Removes the callback for the given _type_.

 -- Method: CD parser.resetparser()
     Resets the fields of the parser used for tracking subcodes to an
     initial state.  `resetparser()' should be called after the disc
     has been changed.

5.38.4 `fl' -- FORMS library for graphical user interfaces
----------------------------------------------------------

Deprecated since version 2.6: The fl module has been
deprecated for removal in Python 3.0.

  This module provides an interface to the FORMS Library by Mark
Overmars.  The source for the library can be retrieved by anonymous ftp
from host `ftp.cs.ruu.nl', directory `SGI/FORMS'.  It was last tested
with version 2.0b.

  Most functions are literal translations of their C equivalents,
dropping the initial `fl_' from their name.  Constants used by the
library are defined in module FL described below.

  The creation of objects is a little different in Python than in C:
instead of the 'current form' maintained by the library to which new
FORMS objects are added, all functions that add a FORMS object to a
form are methods of the Python object representing the form.
Consequently, there are no Python equivalents for the C functions
`fl_addto_form()' and `fl_end_form()', and the equivalent of
`fl_bgn_form()' is called fl.make_form().

  Watch out for the somewhat confusing terminology: FORMS uses the word
_object_ for the buttons, sliders etc. that you can place in a form. In
Python, 'object' means any value.  The Python interface to FORMS
introduces two new Python object types: form objects (representing an
entire form) and FORMS objects (representing one button, slider etc.).
Hopefully this isn't too confusing.

  There are no 'free objects' in the Python interface to FORMS, nor is
there an easy way to add object classes written in Python.  The FORMS
interface to GL event handling is available, though, so you can mix
FORMS with pure GL windows.

  *Please note:* importing fl implies a call to the GL
function `foreground()' and to the FORMS routine `fl_init()'.

5.38.4.1 Functions Defined in Module `fl'
.........................................

Module fl defines the following functions.  For more
information about what they do, see the description of the equivalent C
function in the FORMS documentation:

 -- Function: fl.make_form (type, width, height)
     Create a form with given type, width and height.  This returns a
     _form_ object, whose methods are described below.

 -- Function: fl.do_forms ()
     The standard FORMS main loop.  Returns a Python object
     representing the FORMS object needing interaction, or the special
     value `FL.EVENT'.

 -- Function: fl.check_forms ()
     Check for FORMS events.  Returns what do_forms().
     above returns, or `None' if there is no event that immediately
     needs interaction.

 -- Function: fl.set_event_call_back (function)
     Set the event callback function.

 -- Function: fl.set_graphics_mode (rgbmode, doublebuffering)
     Set the graphics modes.

 -- Function: fl.get_rgbmode ()
     Return the current rgb mode.  This is the value of the C global
     variable `fl_rgbmode'.

 -- Function: fl.show_message (str1, str2, str3)
     Show a dialog box with a three-line message and an OK button.

 -- Function: fl.show_question (str1, str2, str3)
     Show a dialog box with a three-line message and YES and NO
     buttons. It returns `1' if the user pressed YES, `0' if NO.

 -- Function: fl.show_choice (str1, str2, str3, but1[, but2[, but3]])
     Show a dialog box with a three-line message and up to three
     buttons. It returns the number of the button clicked by the user
     (`1', `2' or `3').

 -- Function: fl.show_input (prompt, default)
     Show a dialog box with a one-line prompt message and text field in
     which the user can enter a string.  The second argument is the
     default input string.  It returns the string value as edited by
     the user.

 -- Function: fl.show_file_selector (message, directory, pattern,
          default)
     Show a dialog box in which the user can select a file.  It returns
     the absolute filename selected by the user, or `None' if the user
     presses Cancel.

 -- Function: fl.get_directory ()
 -- Function: fl.get_pattern ()
 -- Function: fl.get_filename ()
     These functions return the directory, pattern and filename (the
     tail part only) selected by the user in the last
     show_file_selector() call.

 -- Function: fl.qdevice (dev)
 -- Function: fl.unqdevice (dev)
 -- Function: fl.isqueued (dev)
 -- Function: fl.qtest ()
 -- Function: fl.qread ()
 -- Function: fl.qreset ()
 -- Function: fl.qenter (dev, val)
 -- Function: fl.get_mouse ()
 -- Function: fl.tie (button, valuator1, valuator2)
     These functions are the FORMS interfaces to the corresponding GL
     functions.  Use these if you want to handle some GL events
     yourself when using `fl.do_events()'.  When a GL event is detected
     that FORMS cannot handle, see fl.do_forms() returns the
     special value `FL.EVENT' and you should call fl.qread()
     to read the event from the queue.  Don't use the equivalent
     GL functions!

 -- Function: fl.color ()
 -- Function: fl.mapcolor ()
 -- Function: fl.getmcolor ()
     See the description in the FORMS documentation of `fl_color()',
     `fl_mapcolor()' and `fl_getmcolor()'.

5.38.4.2 Form Objects
.....................

Form objects (returned by make_form() above) have the
following methods.  Each method corresponds to a C function whose name
is prefixed with `fl_'; and whose first argument is a form pointer;
please refer to the official FORMS documentation for descriptions.

  All the `add_*()' methods return a Python object representing the
FORMS object.  Methods of FORMS objects are described below.  Most
kinds of FORMS object also have some methods specific to that kind;
these methods are listed here.

 -- Method: form.show_form (placement, bordertype, name)
     Show the form.

 -- Method: form.hide_form ()
     Hide the form.

 -- Method: form.redraw_form ()
     Redraw the form.

 -- Method: form.set_form_position (x, y)
     Set the form's position.

 -- Method: form.freeze_form ()
     Freeze the form.

 -- Method: form.unfreeze_form ()
     Unfreeze the form.

 -- Method: form.activate_form ()
     Activate the form.

 -- Method: form.deactivate_form ()
     Deactivate the form.

 -- Method: form.bgn_group ()
     Begin a new group of objects; return a group object.

 -- Method: form.end_group ()
     End the current group of objects.

 -- Method: form.find_first ()
     Find the first object in the form.

 -- Method: form.find_last ()
     Find the last object in the form.

 -- Method: form.add_box (type, x, y, w, h, name)
     Add a box object to the form. No extra methods.

 -- Method: form.add_text (type, x, y, w, h, name)
     Add a text object to the form. No extra methods.

 -- Method: form.add_clock (type, x, y, w, h, name)
     Add a clock object to the form.  --  Method: `get_clock()'.

 -- Method: form.add_button (type, x, y, w, h, name)
     Add a button object to the form.  --  Methods: `get_button()',
     `set_button()'.

 -- Method: form.add_lightbutton (type, x, y, w, h, name)
     Add a lightbutton object to the form.  --  Methods: `get_button()',
     `set_button()'.

 -- Method: form.add_roundbutton (type, x, y, w, h, name)
     Add a roundbutton object to the form.  --  Methods: `get_button()',
     `set_button()'.

 -- Method: form.add_slider (type, x, y, w, h, name)
     Add a slider object to the form.  --  Methods:
     `set_slider_value()', `get_slider_value()', `set_slider_bounds()',
     `get_slider_bounds()', `set_slider_return()', `set_slider_size()',
     `set_slider_precision()', `set_slider_step()'.

 -- Method: form.add_valslider (type, x, y, w, h, name)
     Add a valslider object to the form.  --  Methods:
     `set_slider_value()', `get_slider_value()', `set_slider_bounds()',
     `get_slider_bounds()', `set_slider_return()', `set_slider_size()',
     `set_slider_precision()', `set_slider_step()'.

 -- Method: form.add_dial (type, x, y, w, h, name)
     Add a dial object to the form.  --  Methods: `set_dial_value()',
     `get_dial_value()', `set_dial_bounds()', `get_dial_bounds()'.

 -- Method: form.add_positioner (type, x, y, w, h, name)
     Add a positioner object to the form.  --  Methods:
     `set_positioner_xvalue()', `set_positioner_yvalue()',
     `set_positioner_xbounds()', `set_positioner_ybounds()',
     `get_positioner_xvalue()', `get_positioner_yvalue()',
     `get_positioner_xbounds()', `get_positioner_ybounds()'.

 -- Method: form.add_counter (type, x, y, w, h, name)
     Add a counter object to the form.  --  Methods:
     `set_counter_value()', `get_counter_value()',
     `set_counter_bounds()', `set_counter_step()',
     `set_counter_precision()', `set_counter_return()'.

 -- Method: form.add_input (type, x, y, w, h, name)
     Add a input object to the form.  --  Methods: `set_input()',
     `get_input()', `set_input_color()', `set_input_return()'.

 -- Method: form.add_menu (type, x, y, w, h, name)
     Add a menu object to the form.  --  Methods: `set_menu()',
     `get_menu()', `addto_menu()'.

 -- Method: form.add_choice (type, x, y, w, h, name)
     Add a choice object to the form.  --  Methods: `set_choice()',
     `get_choice()', `clear_choice()', `addto_choice()',
     `replace_choice()', `delete_choice()', `get_choice_text()',
     `set_choice_fontsize()', `set_choice_fontstyle()'.

 -- Method: form.add_browser (type, x, y, w, h, name)
     Add a browser object to the form.  --  Methods:
     `set_browser_topline()', `clear_browser()', `add_browser_line()',
     `addto_browser()', `insert_browser_line()',
     `delete_browser_line()', `replace_browser_line()',
     `get_browser_line()', `load_browser()', `get_browser_maxline()',
     `select_browser_line()', `deselect_browser_line()',
     `deselect_browser()', `isselected_browser_line()', `get_browser()',
     `set_browser_fontsize()', `set_browser_fontstyle()',
     `set_browser_specialkey()'.

 -- Method: form.add_timer (type, x, y, w, h, name)
     Add a timer object to the form.  --  Methods: `set_timer()',
     `get_timer()'.

  Form objects have the following data attributes; see the FORMS
documentation:

Name                      C Type                Meaning
-------------------------------------------------------------------------------------
`window'                  int (read-only)       GL window id
`w'                       float                 form width
`h'                       float                 form height
`x'                       float                 form x origin
`y'                       float                 form y origin
`deactivated'             int                   nonzero if form is deactivated
`visible'                 int                   nonzero if form is visible
`frozen'                  int                   nonzero if form is frozen
`doublebuf'               int                   nonzero if double buffering on

5.38.4.3 FORMS Objects
......................

Besides methods specific to particular kinds of FORMS objects, all
FORMS objects also have the following methods:

 -- Method: FORMS object.set_call_back(function, argument)
     Set the object's callback function and argument.  When the object
     needs interaction, the callback function will be called with two
     arguments: the object, and the callback argument.  (FORMS objects
     without a callback function are returned by fl.do_forms()
     or fl.check_forms() when they need
     interaction.)  Call this method without arguments to remove the
     callback function.

 -- Method: FORMS object.delete_object()
     Delete the object.

 -- Method: FORMS object.show_object()
     Show the object.

 -- Method: FORMS object.hide_object()
     Hide the object.

 -- Method: FORMS object.redraw_object()
     Redraw the object.

 -- Method: FORMS object.freeze_object()
     Freeze the object.

 -- Method: FORMS object.unfreeze_object()
     Unfreeze the object.

  FORMS objects have these data attributes; see the FORMS documentation:

Name                     C Type                Meaning
----------------------------------------------------------------------
`objclass'               int (read-only)       object class
type.         int (read-only)       object type
`boxtype'                int                   box type
`x'                      float                 x origin
`y'                      float                 y origin
`w'                      float                 width
`h'                      float                 height
`col1'                   int                   primary color
`col2'                   int                   secondary color
`align'                  int                   alignment
`lcol'                   int                   label color
`lsize'                  float                 label font size
`label'                  string                label string
`lstyle'                 int                   label style
`pushed'                 int (read-only)       (see FORMS docs)
`focus'                  int (read-only)       (see FORMS docs)
`belowmouse'             int (read-only)       (see FORMS docs)
`frozen'                 int (read-only)       (see FORMS docs)
`active'                 int (read-only)       (see FORMS docs)
input.        int (read-only)       (see FORMS docs)
`visible'                int (read-only)       (see FORMS docs)
`radio'                  int (read-only)       (see FORMS docs)
`automatic'              int (read-only)       (see FORMS docs)

5.38.5 `FL' -- Constants used with the `fl' module
--------------------------------------------------

Deprecated since version 2.6: The FL module has been
deprecated for removal in Python 3.0.

  This module defines symbolic constants needed to use the built-in
module fl. (see above); they are equivalent to those defined
in the C header file `<forms.h>' except that the name prefix `FL_' is
omitted.  Read the module source for a complete list of the defined
names.  Suggested use:

    import fl
    from FL import *

5.38.6 `flp' -- Functions for loading stored FORMS designs
----------------------------------------------------------

Deprecated since version 2.6: The flp module has been
deprecated for removal in Python 3.0.

  This module defines functions that can read form definitions created
by the 'form designer' (*fdesign*) program that comes with the FORMS
library (see module fl above).

  For now, see the file `flp.doc' in the Python library source
directory for a description.

  XXX A complete description should be inserted here!

5.38.7 `fm' -- _Font Manager_ interface
---------------------------------------

Deprecated since version 2.6: The fm module has been
deprecated for removal in Python 3.0.

  This module provides access to the IRIS _Font Manager_ library.   It
is available only on Silicon Graphics machines. See also: _4Sight
User's Guide_, section 1, chapter 5: "Using the IRIS Font Manager."

  This is not yet a full interface to the IRIS Font Manager. Among the
unsupported features are: matrix operations; cache operations;
character operations (use string operations instead); some details of
font info; individual glyph metrics; and printer matching.

  It supports the following operations:

 -- Function: fm.init ()
     Initialization function. Calls `fminit()'. It is normally not
     necessary to call this function, since it is called automatically
     the first time the fm module is imported.

 -- Function: fm.findfont (fontname)
     Return a font handle object. Calls `fmfindfont(fontname)'.

 -- Function: fm.enumerate ()
     Returns a list of available font names. This is an interface to
     `fmenumerate()'.

 -- Function: fm.prstr (string)
     Render a string using the current font (see the `setfont()' font
     handle method below). Calls `fmprstr(string)'.

 -- Function: fm.setpath (string)
     Sets the font search path. Calls `fmsetpath(string)'. (XXX Does
     not work!?!)

 -- Function: fm.fontpath ()
     Returns the current font search path.

  Font handle objects support the following operations:

 -- Method: font handle.scalefont(factor)
     Returns a handle for a scaled version of this font. Calls
     `fmscalefont(fh, factor)'.

 -- Method: font handle.setfont()
     Makes this font the current font. Note: the effect is undone
     silently when the font handle object is deleted. Calls
     `fmsetfont(fh)'.

 -- Method: font handle.getfontname()
     Returns this font's name. Calls `fmgetfontname(fh)'.

 -- Method: font handle.getcomment()
     Returns the comment string associated with this font. Raises an
     exception if there is none. Calls `fmgetcomment(fh)'.

 -- Method: font handle.getfontinfo()
     Returns a tuple giving some pertinent data about this font. This
     is an interface to `fmgetfontinfo()'. The returned tuple contains
     the following numbers: `(printermatched, fixed_width, xorig,
     yorig, xsize, ysize, height, nglyphs)'.

 -- Method: font handle.getstrwidth(string)
     Returns the width, in pixels, of _string_ when drawn in this font.
     Calls `fmgetstrwidth(fh, string)'.

5.38.8 `gl' -- _Graphics Library_ interface
-------------------------------------------

Deprecated since version 2.6: The gl module has been
deprecated for removal in Python 3.0.

  This module provides access to the Silicon Graphics _Graphics
Library_. It is available only on Silicon Graphics machines.

     Warning: Some illegal calls to the GL library cause the Python
     interpreter to dump core.  In particular, the use of most GL calls
     is unsafe before the first window is opened.

  The module is too large to document here in its entirety, but the
following should help you to get started. The parameter conventions for
the C functions are translated to Python as follows:

   * All (short, long, unsigned) int values are represented by Python
     integers.

   * All float and double values are represented by Python floating
     point numbers.  In most cases, Python integers are also allowed.

   * All arrays are represented by one-dimensional Python lists. In
     most cases, tuples are also allowed.

   * All string and character arguments are represented by Python
     strings, for instance, `winopen('Hi There!')' and `rotate(900,
     'z')'.

   * All (short, long, unsigned) integer arguments or return values
     that are only used to specify the length of an array argument are
     omitted. For example, the C call

         lmdef(deftype, index, np, props)

     is translated to Python as

         lmdef(deftype, index, props)

   * Output arguments are omitted from the argument list; they are
     transmitted as function return values instead. If more than one
     value must be returned, the return value is a tuple. If the C
     function has both a regular return value (that is not omitted
     because of the previous rule) and an output argument, the return
     value comes first in the tuple. Examples: the C call

         getmcolor(i, &red, &green, &blue)

     is translated to Python as

         red, green, blue = getmcolor(i)

  The following functions are non-standard or have special argument
conventions:

 -- Function: gl.varray (argument)
     Equivalent to but faster than a number of `v3d()' calls. The
     _argument_ is a list (or tuple) of points. Each point must be a
     tuple of coordinates `(x, y, z)' or `(x, y)'. The points may be 2-
     or 3-dimensional but must all have the same dimension. Float and
     int values may be mixed however. The points are always converted
     to 3D double precision points by assuming `z = 0.0' if necessary
     (as indicated in the man page), and for each point `v3d()' is
     called.

 -- Function: gl.nvarray ()
     Equivalent to but faster than a number of `n3f' and `v3f' calls.
     The argument is an array (list or tuple) of pairs of normals and
     points. Each pair is a tuple of a point and a normal for that
     point. Each point or normal must be a tuple of coordinates `(x, y,
     z)'. Three coordinates must be given. Float and int values may be
     mixed. For each pair, `n3f()' is called for the normal, and then
     `v3f()' is called for the point.

 -- Function: gl.vnarray ()
     Similar to  `nvarray()' but the pairs have the point first and the
     normal second.

 -- Function: gl.nurbssurface (s_k, t_k, ctl, s_ord, t_ord, type)
     Defines a nurbs surface. The dimensions of `ctl[][]' are computed
     as follows: `[len(s_k) - s_ord]', `[len(t_k) - t_ord]'.

 -- Function: gl.nurbscurve (knots, ctlpoints, order, type)
     Defines a nurbs curve. The length of ctlpoints is `len(knots) -
     order'.

 -- Function: gl.pwlcurve (points, type)
     Defines a piecewise-linear curve. _points_ is a list of points.
     _type_ must be `N_ST'.

 -- Function: gl.pick (n)
 -- Function: gl.select (n)
     The only argument to these functions specifies the desired size of
     the pick or select buffer.

 -- Function: gl.endpick ()
 -- Function: gl.endselect ()
     These functions have no arguments. They return a list of integers
     representing the used part of the pick/select buffer. No method is
     provided to detect buffer overrun.

  Here is a tiny but complete example GL program in Python:

    import gl, GL, time

    def main():
        gl.foreground()
        gl.prefposition(500, 900, 500, 900)
        w = gl.winopen('CrissCross')
        gl.ortho2(0.0, 400.0, 0.0, 400.0)
        gl.color(GL.WHITE)
        gl.clear()
        gl.color(GL.RED)
        gl.bgnline()
        gl.v2f(0.0, 0.0)
        gl.v2f(400.0, 400.0)
        gl.endline()
        gl.bgnline()
        gl.v2f(400.0, 0.0)
        gl.v2f(0.0, 400.0)
        gl.endline()
        time.sleep(5)

    main()

PyOpenGL: The Python OpenGL Binding (http://pyopengl.sourceforge.net/)
     An interface to OpenGL is also available; see information about
     the *PyOpenGL* project online at
     <http://pyopengl.sourceforge.net/>.  This may be a better option
     if support for SGI hardware from before about 1996 is not required.

5.38.9 `DEVICE' -- Constants used with the `gl' module
------------------------------------------------------

Deprecated since version 2.6: The DEVICE module has been
deprecated for removal in Python 3.0.

  This modules defines the constants used by the Silicon Graphics
_Graphics Library_ that C programmers find in the header file
`<gl/device.h>'. Read the module source file for details.

5.38.10 `GL' -- Constants used with the `gl' module
---------------------------------------------------

Deprecated since version 2.6: The GL module has been
deprecated for removal in Python 3.0.

  This module contains constants used by the Silicon Graphics _Graphics
Library_ from the C header file `<gl/gl.h>'. Read the module source
file for details.

5.38.11 `imgfile' -- Support for SGI imglib files
-------------------------------------------------

Deprecated since version 2.6: The imgfile module has been
deprecated for removal in Python 3.0.

  The imgfile module allows Python programs to access SGI
imglib image files (also known as `.rgb' files).  The module is far
from complete, but is provided anyway since the functionality that
there is enough in some cases.  Currently, colormap files are not
supported.

  The module defines the following variables and functions:

 -- Exception: imgfile.error
     This exception is raised on all errors, such as unsupported file
     type, etc.

 -- Function: imgfile.getsizes (file)
     This function returns a tuple `(x, y, z)' where _x_ and _y_ are
     the size of the image in pixels and _z_ is the number of bytes per
     pixel. Only 3 byte RGB pixels and 1 byte greyscale pixels are
     currently supported.

 -- Function: imgfile.read (file)
     This function reads and decodes the image on the specified file,
     and returns it as a Python string. The string has either 1 byte
     greyscale pixels or 4 byte RGBA pixels. The bottom left pixel is
     the first in the string. This format is suitable to pass to
     `gl.lrectwrite()', for instance.

 -- Function: imgfile.readscaled (file, x, y, filter[, blur])
     This function is identical to read but it returns an image that is
     scaled to the given _x_ and _y_ sizes. If the _filter_ and _blur_
     parameters are omitted scaling is done by simply dropping or
     duplicating pixels, so the result will be less than perfect,
     especially for computer-generated images.

     Alternatively, you can specify a filter to use to smooth the image
     after scaling. The filter forms supported are `'impulse'', `'box'',
     `'triangle'', `'quadratic'' and `'gaussian''. If a filter is
     specified _blur_ is an optional parameter specifying the
     blurriness of the filter. It defaults to `1.0'.

     readscaled() makes no attempt to keep the aspect
     ratio correct, so that is the users' responsibility.

 -- Function: imgfile.ttob (flag)
     This function sets a global flag which defines whether the scan
     lines of the image are read or written from bottom to top (flag is
     zero, compatible with SGI GL) or from top to bottom(flag is one,
     compatible with X).  The default is zero.

 -- Function: imgfile.write (file, data, x, y, z)
     This function writes the RGB or greyscale data in _data_ to image
     file _file_.  _x_ and _y_ give the size of the image, _z_ is 1 for
     1 byte greyscale images or 3 for RGB images (which are stored as 4
     byte values of which only the lower three bytes are used). These
     are the formats returned by `gl.lrectread()'.

5.38.12 `jpeg' -- Read and write JPEG files
-------------------------------------------

Deprecated since version 2.6: The jpeg module has been
deprecated for removal in Python 3.0.

  The module jpeg provides access to the jpeg compressor and
decompressor written by the Independent JPEG Group (IJG). JPEG is a
standard for compressing pictures; it is defined in ISO 10918.  For
details on JPEG or the Independent JPEG Group software refer to the
JPEG standard or the documentation provided with the software.

  A portable interface to JPEG image files is available with the Python
Imaging Library (PIL) by Fredrik Lundh.  Information on PIL is
available at <http://www.pythonware.com/products/pil/>.

  The jpeg module defines an exception and some functions.

 -- Exception: jpeg.error
     Exception raised by compress() and
     decompress() in case of errors.

 -- Function: jpeg.compress (data, w, h, b)
     Treat data as a pixmap of width _w_ and height _h_, with _b_ bytes
     per pixel.  The data is in SGI GL order, so the first pixel is in
     the lower-left corner.  This means that `gl.lrectread()' return
     data can immediately be passed to compress().
     Currently only 1 byte and 4 byte pixels are allowed, the former
     being treated as greyscale and the latter as RGB color. See
     compress().  returns a string that contains the compressed
     picture, in JFIF format.

 -- Function: jpeg.decompress (data)
     Data is a string containing a picture in JFIF format. It returns a
     tuple `(data, width, height, bytesperpixel)'.  Again, the data is
     suitable to pass to `gl.lrectwrite()'.

 -- Function: jpeg.setoption (name, value)
     Set various options.  Subsequent compress() and
     decompress() calls will use these options.  The following
     options are available:

     Option                Effect
     ------------------------------------------------------------------------
     `'forcegray''         Force output to be grayscale, even if input is
                           RGB.
     `'quality''           Set the quality of the compressed image to a
                           value between `0' and `100' (default is `75').
                           This only affects compression.
     `'optimize''          Perform Huffman table optimization.  Takes
                           longer, but results in smaller compressed image.
                           This only affects compression.
     `'smooth''            Perform inter-block smoothing on uncompressed
                           image.  Only useful for low- quality images.
                           This only affects decompression.

JPEG Still Image Data Compression Standard
     The canonical reference for the JPEG image format, by Pennebaker
     and Mitchell.

Information Technology - Digital Compression and Coding of Continuous-tone Still Images - Requirements and Guidelines (http://www.w3.org/Graphics/JPEG/itu-t81.pdf)
     The ISO standard for JPEG is also published as ITU T.81.  This is
     available online in PDF form.

5.39 SunOS Specific Services
============================

The modules described in this chapter provide interfaces to features
that are unique to SunOS 5 (also known as Solaris version 2).

5.39.1 `sunaudiodev' -- Access to Sun audio hardware
----------------------------------------------------

Deprecated since version 2.6: The sunaudiodev module has
been deprecated for removal in Python 3.0.

  This module allows you to access the Sun audio interface. The Sun
audio hardware is capable of recording and playing back audio data in
u-LAW format with a sample rate of 8K per second. A full description
can be found in the `audio(7I)' manual page.

  The module SUNAUDIODEV.  defines constants which may be
used with this module.

  This module defines the following variables and functions:

 -- Exception: sunaudiodev.error
     This exception is raised on all errors. The argument is a string
     describing what went wrong.

 -- Function: sunaudiodev.open (mode)
     This function opens the audio device and returns a Sun audio
     device object. This object can then be used to do I/O on. The
     _mode_ parameter is one of `'r'' for record-only access, `'w'' for
     play-only access, `'rw'' for both and `'control'' for access to
     the control device. Since only one process is allowed to have the
     recorder or player open at the same time it is a good idea to open
     the device only for the activity needed. See `audio(7I)' for
     details.

     As per the manpage, this module first looks in the environment
     variable `AUDIODEV' for the base audio device filename.  If not
     found, it falls back to `/dev/audio'.  The control device is
     calculated by appending "ctl" to the base audio device.

5.39.1.1 Audio Device Objects
.............................

The audio device objects are returned by open() define the
following methods (except `control' objects which only provide
`getinfo()', `setinfo()', `fileno()', and `drain()'):

 -- Method: audio device.close()
     This method explicitly closes the device. It is useful in
     situations where deleting the object does not immediately close it
     since there are other references to it. A closed device should not
     be used again.

 -- Method: audio device.fileno()
     Returns the file descriptor associated with the device.  This can
     be used to set up `SIGPOLL' notification, as described below.

 -- Method: audio device.drain()
     This method waits until all pending output is processed and then
     returns.  Calling this method is often not necessary: destroying
     the object will automatically close the audio device and this will
     do an implicit drain.

 -- Method: audio device.flush()
     This method discards all pending output. It can be used avoid the
     slow response to a user's stop request (due to buffering of up to
     one second of sound).

 -- Method: audio device.getinfo()
     This method retrieves status information like input and output
     volume, etc. and returns it in the form of an audio status object.
     This object has no methods but it contains a number of attributes
     describing the current device status. The names and meanings of
     the attributes are described in `<sun/audioio.h>' and in the
     `audio(7I)' manual page.  Member names are slightly different from
     their C counterparts: a status object is only a single structure.
     Members of the `play' substructure have `o_' prepended to their
     name and members of the `record' structure have `i_'. So, the C
     member `play.sample_rate' is accessed as `o_sample_rate',
     `record.gain' as `i_gain' and `monitor_gain' plainly as
     `monitor_gain'.

 -- Method: audio device.ibufcount()
     This method returns the number of samples that are buffered on the
     recording side, i.e. the program will not block on a `read()' call
     of so many samples.

 -- Method: audio device.obufcount()
     This method returns the number of samples buffered on the playback
     side.  Unfortunately, this number cannot be used to determine a
     number of samples that can be written without blocking since the
     kernel output queue length seems to be variable.

 -- Method: audio device.read(size)
     This method reads _size_ samples from the audio input and returns
     them as a Python string. The function blocks until enough data is
     available.

 -- Method: audio device.setinfo(status)
     This method sets the audio device status parameters. The _status_
     parameter is an device status object as returned by `getinfo()'
     and possibly modified by the program.

 -- Method: audio device.write(samples)
     Write is passed a Python string containing audio samples to be
     played. If there is enough buffer space free it will immediately
     return, otherwise it will block.

  The audio device supports asynchronous notification of various
events, through the SIGPOLL signal.  Here's an example of how you might
enable this in Python:

    def handle_sigpoll(signum, frame):
        print 'I got a SIGPOLL update'

    import fcntl, signal, STROPTS

    signal.signal(signal.SIGPOLL, handle_sigpoll)
    fcntl.ioctl(audio_obj.fileno(), STROPTS.I_SETSIG, STROPTS.S_MSG)

5.39.2 `SUNAUDIODEV' -- Constants used with `sunaudiodev'
---------------------------------------------------------

Deprecated since version 2.6: The SUNAUDIODEV module has
been deprecated for removal in Python 3.0.

  This is a companion module to sunaudiodev which defines
useful symbolic constants like `MIN_GAIN', `MAX_GAIN', `SPEAKER', etc.
The names of the constants are the same names as used in the C include
file `<sun/audioio.h>', with the leading string `AUDIO_' stripped.

5.40 Undocumented Modules
=========================

Here's a quick listing of modules that are currently undocumented, but
that should be documented.  Feel free to contribute documentation for
them!  (Send via email to <docs@python.org>.)

  The idea and original contents for this chapter were taken from a
posting by Fredrik Lundh; the specific contents of this chapter have
been substantially revised.

5.40.1 Miscellaneous useful utilities
-------------------------------------

Some of these are very old and/or not very robust; marked with "hmm."

`ihooks'
     -- Import hook support (for rexec.; may become
     obsolete).  Removed in Python 3.x.

5.40.2 Platform specific modules
--------------------------------

These modules are used to implement the os.path module, and
are not documented beyond this mention.  There's little need to
document these.

`ntpath'
     -- Implementation of os.path on Win32, Win64, WinCE,
     and OS/2 platforms.

`posixpath'
     -- Implementation of os.path on POSIX.

`bsddb185'
     -- Backwards compatibility module for systems which still use the
     Berkeley DB 1.85 module.  It is normally only available on certain
     BSD Unix-based systems.  It should never be used directly.

5.40.3 Multimedia
-----------------

`audiodev'
     -- Platform-independent API for playing audio data.  Removed in
     Python 3.x.

`linuxaudiodev'
     -- Play audio data on the Linux audio device.  Replaced in Python
     2.3 by the ossaudiodev module.  Removed in Python 3.x.

`sunaudio'
     -- Interpret Sun audio headers (may become obsolete or a
     tool/demo).  Removed in Python 3.x.

`toaiff'
     -- Convert "arbitrary" sound files to AIFF files; should probably
     become a tool or demo.  Requires the external program *sox*.
     Removed in Python 3.x.

5.40.4 Undocumented Mac OS modules
----------------------------------

5.40.4.1 `applesingle' -- AppleSingle decoder
.............................................

Deprecated since version 2.6.

5.40.4.2 `buildtools' -- Helper module for BuildApplet and Friends
..................................................................

Deprecated since version 2.4.

5.40.4.3 `cfmfile' -- Code Fragment Resource module
...................................................

cfmfile is a module that understands Code Fragments and the
accompanying "cfrg" resources. It can parse them and merge them, and is
used by BuildApplication to combine all plugin modules to a single
executable.

  Deprecated since version 2.4.

5.40.4.4 `icopen' -- Internet Config replacement for `open()'
.............................................................

Importing icopen will replace the built-in open()
with a version that uses Internet Config to set file type and
creator for new files.

  Deprecated since version 2.6.

5.40.4.5 `macerrors' -- Mac OS Errors
.....................................

macerrors contains constant definitions for many Mac OS
error codes.

  Deprecated since version 2.6.

5.40.4.6 `macresource' -- Locate script resources
.................................................

macresource helps scripts finding their resources, such as
dialogs and menus, without requiring special case code for when the
script is run under MacPython, as a MacPython applet or under OSX
Python.

  Deprecated since version 2.6.

5.40.4.7 `Nav' -- NavServices calls
...................................

A low-level interface to Navigation Services.

  Deprecated since version 2.6.

5.40.4.8 `PixMapWrapper' -- Wrapper for PixMap objects
......................................................

PixMapWrapper wraps a PixMap object with a Python object
that allows access to the fields by name. It also has methods to
convert to and from `PIL' images.

  Deprecated since version 2.6.

5.40.4.9 `videoreader' -- Read QuickTime movies
...............................................

videoreader reads and decodes QuickTime movies and passes a
stream of images to your program. It also provides some support for
audio tracks.

  Deprecated since version 2.6.

5.40.4.10 `W' -- Widgets built on `FrameWork'
.............................................

The W widgets are used extensively in the *IDE*.

  Deprecated since version 2.6.

5.40.5 Obsolete
---------------

These modules are not normally available for import; additional work
must be done to make them available.

  These extension modules written in C are not built by default. Under
Unix, these must be enabled by uncommenting the appropriate lines in
`Modules/Setup' in the build tree and either rebuilding Python if the
modules are statically linked, or building and installing the shared
object if using dynamically-loaded extensions.

`timing'
     -- Measure time intervals to high resolution (use
     time.clock().  instead).  Removed in Python 3.x.

5.40.6 SGI-specific Extension modules
-------------------------------------

The following are SGI specific, and may be out of touch with the
current version of reality.

`cl'
     -- Interface to the SGI compression library.

`sv'
     -- Interface to the "simple video" board on SGI Indigo (obsolete
     hardware).  Removed in Python 3.x.

6 Extending and Embedding the Python Interpreter
************************************************

     Release: 2.7

     Date: February 08, 2011

  This document describes how to write modules in C or C++ to extend
the Python interpreter with new modules.  Those modules can define new
functions but also new object types and their methods.  The document
also describes how to embed the Python interpreter in another
application, for use as an extension language.  Finally, it shows how
to compile and link extension modules so that they can be loaded
dynamically (at run time) into the interpreter, if the underlying
operating system supports this feature.

  This document assumes basic knowledge about Python.  For an informal
introduction to the language, see The Python Tutorial.
See The Python Language Reference.  gives a more formal
definition of the language.  See The Python Standard Library.
documents the existing object types, functions and modules (both
built-in and written in Python) that give the language its wide
application range.

  For a detailed description of the whole Python/C API, see the separate
Python/C API Reference Manual.

6.1 Extending Python with C or C++
==================================

It is quite easy to add new built-in modules to Python, if you know how
to program in C.  Such _extension modules_ can do two things that can't
be done directly in Python: they can implement new built-in object
types, and they can call C library functions and system calls.

  To support extensions, the Python API (Application Programmers
Interface) defines a set of functions, macros and variables that
provide access to most aspects of the Python run-time system.  The
Python API is incorporated in a C source file by including the header
`"Python.h"'.

  The compilation of an extension module depends on its intended use as
well as on your system setup; details are given in later chapters.

  Do note that if your use case is calling C library functions or
system calls, you should consider using the ctypes module
rather than writing custom C code. Not only does ctypes let
you write Python code to interface with C code, but it is more portable
between implementations of Python than writing and compiling an
extension module which typically ties you to CPython.

6.1.1 A Simple Example
----------------------

Let's create an extension module called `spam' (the favorite food of
Monty Python fans...) and let's say we want to create a Python
interface to the C library function `system()'. (1) This function takes
a null-terminated character string as argument and returns an integer.
We want this function to be callable from Python as follows:

    >>> import spam
    >>> status = spam.system("ls -l")

Begin by creating a file `spammodule.c'.  (Historically, if a module is
called `spam', the C file containing its implementation is called
`spammodule.c'; if the module name is very long, like `spammify', the
module name can be just `spammify.c'.)

  The first line of our file can be:

    #include <Python.h>

which pulls in the Python API (you can add a comment describing the
purpose of the module and a copyright notice if you like).

     Note: Since Python may define some pre-processor definitions which
     affect the standard headers on some systems, you _must_ include
     `Python.h' before any standard headers are included.

  All user-visible symbols defined by `Python.h' have a prefix of `Py'
or `PY', except those defined in standard header files. For
convenience, and since they are used extensively by the Python
interpreter, `"Python.h"' includes a few standard header files:
`<stdio.h>', `<string.h>', `<errno.h>', and `<stdlib.h>'.  If the
latter header file does not exist on your system, it declares the
functions `malloc()', `free()' and `realloc()' directly.

  The next thing we add to our module file is the C function that will
be called when the Python expression `spam.system(string)' is evaluated
(we'll see shortly how it ends up being called):

    static PyObject *
    spam_system(PyObject *self, PyObject *args)
    {
        const char *command;
        int sts;

        if (!PyArg_ParseTuple(args, "s", &command))
            return NULL;
        sts = system(command);
        return Py_BuildValue("i", sts);
    }

There is a straightforward translation from the argument list in Python
(for example, the single expression `"ls -l"') to the arguments passed
to the C function.  The C function always has two arguments,
conventionally named _self_ and _args_.

  The _self_ argument points to the module object for module-level
functions; for a method it would point to the object instance.

  The _args_ argument will be a pointer to a Python tuple object
containing the arguments.  Each item of the tuple corresponds to an
argument in the call's argument list.  The arguments are Python objects
-- in order to do anything with them in our C function we have to
convert them to C values.  The function PyArg_ParseTuple().
in the Python API checks the argument types and converts them to C
values.  It uses a template string to determine the required types of
the arguments as well as the types of the C variables into which to
store the converted values.  More about this later.

  PyArg_ParseTuple() returns true (nonzero) if all
arguments have the right type and its components have been stored in
the variables whose addresses are passed.  It returns false (zero) if
an invalid argument list was passed.  In the latter case it also raises
an appropriate exception so the calling function can return _NULL_
immediately (as we saw in the example).

6.1.2 Intermezzo: Errors and Exceptions
---------------------------------------

An important convention throughout the Python interpreter is the
following: when a function fails, it should set an exception condition
and return an error value (usually a _NULL_ pointer).  Exceptions are
stored in a static global variable inside the interpreter; if this
variable is _NULL_ no exception has occurred.  A second global variable
stores the "associated value" of the exception (the second argument to
raise).  A third variable contains the stack traceback in
case the error originated in Python code.  These three variables are
the C equivalents of the Python variables `sys.exc_type',
`sys.exc_value' and `sys.exc_traceback' (see the section on module
sys in the Python Library Reference).  It is important to
know about them to understand how errors are passed around.

  The Python API defines a number of functions to set various types of
exceptions.

  The most common one is PyErr_SetString().  Its arguments
are an exception object and a C string.  The exception object is
usually a predefined object like `PyExc_ZeroDivisionError'.  The C
string indicates the cause of the error and is converted to a Python
string object and stored as the "associated value" of the exception.

  Another useful function is PyErr_SetFromErrno(), which
only takes an exception argument and constructs the associated value by
inspection of the global variable `errno'.  The most general function is
PyErr_SetObject(), which takes two object arguments, the
exception and its associated value.  You don't need to
Py_INCREF() the objects passed to any of these functions.

  You can test non-destructively whether an exception has been set with
PyErr_Occurred().  This returns the current exception
object, or _NULL_ if no exception has occurred.  You normally don't
need to call PyErr_Occurred() to see whether an error
occurred in a function call, since you should be able to tell from the
return value.

  When a function _f_ that calls another function _g_ detects that the
latter fails, _f_ should itself return an error value (usually _NULL_
or `-1').  It should _not_ call one of the `PyErr_*()' functions -- one
has already been called by _g_. _f_'s caller is then supposed to also
return an error indication to _its_ caller, again _without_ calling
`PyErr_*()', and so on -- the most detailed cause of the error was
already reported by the function that first detected it.  Once the
error reaches the Python interpreter's main loop, this aborts the
currently executing Python code and tries to find an exception handler
specified by the Python programmer.

  (There are situations where a module can actually give a more
detailed error message by calling another `PyErr_*()' function, and in
such cases it is fine to do so.  As a general rule, however, this is
not necessary, and can cause information about the cause of the error
to be lost: most operations can fail for a variety of reasons.)

  To ignore an exception set by a function call that failed, the
exception condition must be cleared explicitly by calling
PyErr_Clear().  The only time C code should call
PyErr_Clear() is if it doesn't want to pass the error on to the
interpreter but wants to handle it completely by itself (possibly by
trying something else, or pretending nothing went wrong).

  Every failing `malloc()' call must be turned into an exception -- the
direct caller of `malloc()' (or `realloc()') must call
PyErr_NoMemory() and return a failure indicator itself.  All the
object-creating functions (for example, see PyInt_FromLong())
already do this, so this note is only relevant to those who call
`malloc()' directly.

  Also note that, with the important exception of
PyArg_ParseTuple() and friends, functions that return an integer
status usually return a positive value or zero for success and `-1' for
failure, like Unix system calls.

  Finally, be careful to clean up garbage (by making
Py_XDECREF() or Py_DECREF() calls for objects you
have already created) when you return an error indicator!

  The choice of which exception to raise is entirely yours.  There are
predeclared C objects corresponding to all built-in Python exceptions,
such as `PyExc_ZeroDivisionError', which you can use directly. Of
course, you should choose exceptions wisely -- don't use
`PyExc_TypeError' to mean that a file couldn't be opened (that should
probably be `PyExc_IOError').  If something's wrong with the argument
list, the PyArg_ParseTuple().  function usually raises
`PyExc_TypeError'.  If you have an argument whose value must be in a
particular range or must satisfy other conditions, `PyExc_ValueError'
is appropriate.

  You can also define a new exception that is unique to your module.
For this, you usually declare a static object variable at the beginning
of your file:

    static PyObject *SpamError;

and initialize it in your module's initialization function
(`initspam()') with an exception object (leaving out the error checking
for now):

    PyMODINIT_FUNC
    initspam(void)
    {
        PyObject *m;

        m = Py_InitModule("spam", SpamMethods);
        if (m == NULL)
            return;

        SpamError = PyErr_NewException("spam.error", NULL, NULL);
        Py_INCREF(SpamError);
        PyModule_AddObject(m, "error", SpamError);
    }

Note that the Python name for the exception object is `spam.error'.  The
PyErr_NewException() function may create a class with the
base class being Exception. (unless another class is passed
in instead of _NULL_), described in Built-in Exceptions.

  Note also that the `SpamError' variable retains a reference to the
newly created exception class; this is intentional!  Since the
exception could be removed from the module by external code, an owned
reference to the class is needed to ensure that it will not be
discarded, causing `SpamError' to become a dangling pointer. Should it
become a dangling pointer, C code which raises the exception could
cause a core dump or other unintended side effects.

  We discuss the use of `PyMODINIT_FUNC' as a function return type
later in this sample.

  The `spam.error' exception can be raised in your extension module
using a call to PyErr_SetString() as shown below:

    static PyObject *
    spam_system(PyObject *self, PyObject *args)
    {
        const char *command;
        int sts;

        if (!PyArg_ParseTuple(args, "s", &command))
            return NULL;
        sts = system(command);
        if (sts < 0) {
            PyErr_SetString(SpamError, "System command failed");
            return NULL;
        }
        return PyLong_FromLong(sts);
    }

6.1.3 Back to the Example
-------------------------

Going back to our example function, you should now be able to
understand this statement:

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;

It returns _NULL_ (the error indicator for functions returning object
pointers) if an error is detected in the argument list, relying on the
exception set by PyArg_ParseTuple().  Otherwise the string
value of the argument has been copied to the local variable `command'.
This is a pointer assignment and you are not supposed to modify the
string to which it points (so in Standard C, the variable `command'
should properly be declared as `const char *command').

  The next statement is a call to the Unix function `system()', passing
it the string we just got from PyArg_ParseTuple().:

    sts = system(command);

Our `spam.system()' function must return the value of `sts' as a Python
object.  This is done using the function Py_BuildValue(),
which is something like the inverse of PyArg_ParseTuple().:
it takes a format string and an arbitrary number of C values, and
returns a new Python object.  More info on Py_BuildValue().
is given later.

    return Py_BuildValue("i", sts);

In this case, it will return an integer object.  (Yes, even integers
are objects on the heap in Python!)

  If you have a C function that returns no useful argument (a function
returning `void'), the corresponding Python function must return
`None'.   You need this idiom to do so (which is implemented by the
Py_RETURN_NONE.  macro):

    Py_INCREF(Py_None);
    return Py_None;

Py_None is the C name for the special Python object
`None'.  It is a genuine Python object rather than a _NULL_ pointer,
which means "error" in most contexts, as we have seen.

6.1.4 The Module's Method Table and Initialization Function
-----------------------------------------------------------

I promised to show how `spam_system()' is called from Python programs.
First, we need to list its name and address in a "method table":

    static PyMethodDef SpamMethods[] = {
        ...
        {"system",  spam_system, METH_VARARGS,
         "Execute a shell command."},
        ...
        {NULL, NULL, 0, NULL}        /* Sentinel */
    };

Note the third entry (`METH_VARARGS').  This is a flag telling the
interpreter the calling convention to be used for the C function.  It
should normally always be `METH_VARARGS' or `METH_VARARGS |
METH_KEYWORDS'; a value of `0' means that an obsolete variant of
PyArg_ParseTuple() is used.

  When using only `METH_VARARGS', the function should expect the
Python-level parameters to be passed in as a tuple acceptable for
parsing via PyArg_ParseTuple().; more information on this
function is provided below.

  The METH_KEYWORDS bit may be set in the third field if
keyword arguments should be passed to the function.  In this case, the
C function should accept a third `PyObject *' parameter which will be a
dictionary of keywords.  Use PyArg_ParseTupleAndKeywords().
to parse the arguments to such a function.

  The method table must be passed to the interpreter in the module's
initialization function.  The initialization function must be named
`initname()', where _name_ is the name of the module, and should be the
only non-`static' item defined in the module file:

    PyMODINIT_FUNC
    initspam(void)
    {
        (void) Py_InitModule("spam", SpamMethods);
    }

Note that PyMODINIT_FUNC declares the function as `void' return type,
declares any special linkage declarations required by the platform, and
for  C++ declares the function as `extern "C"'.

  When the Python program imports module `spam' for the first time,
`initspam()' is called. (See below for comments about embedding Python.)
It calls Py_InitModule(), which creates a "module object"
(which is inserted in the dictionary `sys.modules' under the key
`"spam"'), and inserts built-in function objects into the newly created
module based upon the table (an array of PyMethodDef.
structures) that was passed as its second argument.
Py_InitModule() returns a pointer to the module object that it
creates (which is unused here).  It may abort with a fatal error for
certain errors, or return _NULL_ if the module could not be initialized
satisfactorily.

  When embedding Python, the `initspam()' function is not called
automatically unless there's an entry in the `_PyImport_Inittab' table.
The easiest way to handle this is to statically initialize your
statically-linked modules by directly calling `initspam()' after the
call to Py_Initialize().:

    int
    main(int argc, char *argv[])
    {
        /* Pass argv[0] to the Python interpreter */
        Py_SetProgramName(argv[0]);

        /* Initialize the Python interpreter.  Required. */
        Py_Initialize();

        /* Add a static module */
        initspam();

An example may be found in the file `Demo/embed/demo.c' in the Python
source distribution.

     Note: Removing entries from `sys.modules' or importing compiled
     modules into multiple interpreters within a process (or following
     a `fork()' without an intervening `exec()') can create problems
     for some extension modules.  Extension module authors should
     exercise caution when initializing internal data structures. Note
     also that the reload() function can be used with
     extension modules, and will call the module initialization function
     (`initspam()' in the example), but will not load the module again
     if it was loaded from a dynamically loadable object file (`.so' on
     Unix, `.dll' on Windows).

  A more substantial example module is included in the Python source
distribution as `Modules/xxmodule.c'.  This file may be used as a
template or simply read as an example.

6.1.5 Compilation and Linkage
-----------------------------

There are two more things to do before you can use your new extension:
compiling and linking it with the Python system.  If you use dynamic
loading, the details may depend on the style of dynamic loading your
system uses; see the chapters about building extension modules (chapter
Building C and C++ Extensions with distutils) and
additional information that pertains only to building on Windows
(chapter Building C and C++ Extensions on Windows) for more
information about this.

  If you can't use dynamic loading, or if you want to make your module
a permanent part of the Python interpreter, you will have to change the
configuration setup and rebuild the interpreter.  Luckily, this is very
simple on Unix: just place your file (`spammodule.c' for example) in
the `Modules/' directory of an unpacked source distribution, add a line
to the file `Modules/Setup.local' describing your file:

    spam spammodule.o

and rebuild the interpreter by running *make* in the toplevel
directory.  You can also run *make* in the `Modules/' subdirectory, but
then you must first rebuild `Makefile' there by running '*make*
Makefile'.  (This is necessary each time you change the `Setup' file.)

  If your module requires additional libraries to link with, these can
be listed on the line in the configuration file as well, for instance:

    spam spammodule.o -lX11

6.1.6 Calling Python Functions from C
-------------------------------------

So far we have concentrated on making C functions callable from Python.
The reverse is also useful: calling Python functions from C. This is
especially the case for libraries that support so-called "callback"
functions.  If a C interface makes use of callbacks, the equivalent
Python often needs to provide a callback mechanism to the Python
programmer; the implementation will require calling the Python callback
functions from a C callback.  Other uses are also imaginable.

  Fortunately, the Python interpreter is easily called recursively, and
there is a standard interface to call a Python function.  (I won't
dwell on how to call the Python parser with a particular string as
input -- if you're interested, have a look at the implementation of the
-c command line option in `Modules/main.c' from the Python
source code.)

  Calling a Python function is easy.  First, the Python program must
somehow pass you the Python function object.  You should provide a
function (or some other interface) to do this.  When this function is
called, save a pointer to the Python function object (be careful to
Py_INCREF() it!) in a global variable -- or wherever you
see fit. For example, the following function might be part of a module
definition:

    static PyObject *my_callback = NULL;

    static PyObject *
    my_set_callback(PyObject *dummy, PyObject *args)
    {
        PyObject *result = NULL;
        PyObject *temp;

        if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
            if (!PyCallable_Check(temp)) {
                PyErr_SetString(PyExc_TypeError, "parameter must be callable");
                return NULL;
            }
            Py_XINCREF(temp);         /* Add a reference to new callback */
            Py_XDECREF(my_callback);  /* Dispose of previous callback */
            my_callback = temp;       /* Remember new callback */
            /* Boilerplate to return "None" */
            Py_INCREF(Py_None);
            result = Py_None;
        }
        return result;
    }

This function must be registered with the interpreter using the
METH_VARARGS flag; this is described in section The Module's Method Table and Initialization Function.
The
PyArg_ParseTuple() function and its arguments are documented in
section Extracting Parameters in Extension Functions.

  The macros Py_XINCREF() and Py_XDECREF().
increment/decrement the reference count of an object and are safe in
the presence of _NULL_ pointers (but note that _temp_ will not be
_NULL_ in this context).  More info on them in section Reference Counts.

  Later, when it is time to call the function, you call the C function
PyObject_CallObject().  This function has two arguments,
both pointers to arbitrary Python objects: the Python function, and the
argument list.  The argument list must always be a tuple object, whose
length is the number of arguments.  To call the Python function with no
arguments, pass in NULL, or an empty tuple; to call it with one
argument, pass a singleton tuple.  Py_BuildValue() returns
a tuple when its format string consists of zero or more format codes
between parentheses.  For example:

    int arg;
    PyObject *arglist;
    PyObject *result;
    ...
    arg = 123;
    ...
    /* Time to call the callback */
    arglist = Py_BuildValue("(i)", arg);
    result = PyObject_CallObject(my_callback, arglist);
    Py_DECREF(arglist);

PyObject_CallObject() returns a Python object pointer:
this is the return value of the Python function.
PyObject_CallObject() is "reference-count-neutral" with respect
to its arguments.  In the example a new tuple was created to serve as
the argument list, which is Py_DECREF().-ed immediately
after the call.

  The return value of PyObject_CallObject() is "new":
either it is a brand new object, or it is an existing object whose
reference count has been incremented.  So, unless you want to save it
in a global variable, you should somehow Py_DECREF() the
result, even (especially!) if you are not interested in its value.

  Before you do this, however, it is important to check that the return
value isn't _NULL_.  If it is, the Python function terminated by
raising an exception.  If the C code that called
PyObject_CallObject() is called from Python, it should now
return an error indication to its Python caller, so the interpreter can
print a stack trace, or the calling Python code can handle the
exception.  If this is not possible or desirable, the exception should
be cleared by calling PyErr_Clear().  For example:

    if (result == NULL)
        return NULL; /* Pass error back */
    ...use result...
    Py_DECREF(result);

Depending on the desired interface to the Python callback function, you
may also have to provide an argument list to
PyObject_CallObject().  In some cases the argument list is also
provided by the Python program, through the same interface that
specified the callback function.  It can then be saved and used in the
same manner as the function object.  In other cases, you may have to
construct a new tuple to pass as the argument list.  The simplest way
to do this is to call Py_BuildValue().  For example, if you
want to pass an integral event code, you might use the following code:

    PyObject *arglist;
    ...
    arglist = Py_BuildValue("(l)", eventcode);
    result = PyObject_CallObject(my_callback, arglist);
    Py_DECREF(arglist);
    if (result == NULL)
        return NULL; /* Pass error back */
    /* Here maybe use the result */
    Py_DECREF(result);

Note the placement of `Py_DECREF(arglist)' immediately after the call,
before the error check!  Also note that strictly speaking this code is
not complete: Py_BuildValue() may run out of memory, and
this should be checked.

  You may also call a function with keyword arguments by using
PyObject_Call(), which supports arguments and keyword arguments.
As in the above example, we use Py_BuildValue() to
construct the dictionary.

    PyObject *dict;
    ...
    dict = Py_BuildValue("{s:i}", "name", val);
    result = PyObject_Call(my_callback, NULL, dict);
    Py_DECREF(dict);
    if (result == NULL)
        return NULL; /* Pass error back */
    /* Here maybe use the result */
    Py_DECREF(result);

6.1.7 Extracting Parameters in Extension Functions
--------------------------------------------------

The PyArg_ParseTuple() function is declared as follows:

    int PyArg_ParseTuple(PyObject *arg, char *format, ...);

The _arg_ argument must be a tuple object containing an argument list
passed from Python to a C function.  The _format_ argument must be a
format string, whose syntax is explained in Parsing arguments and building values
in the Python/C API Reference Manual.  The
remaining arguments must be addresses of variables whose type is
determined by the format string.

  Note that while PyArg_ParseTuple() checks that the Python
arguments have the required types, it cannot check the validity of the
addresses of C variables passed to the call: if you make mistakes
there, your code will probably crash or at least overwrite random bits
in memory.  So be careful!

  Note that any Python object references which are provided to the
caller are _borrowed_ references; do not decrement their reference
count!

  Some example calls:

    int ok;
    int i, j;
    long k, l;
    const char *s;
    int size;

    ok = PyArg_ParseTuple(args, ""); /* No arguments */
        /* Python call: f() */

    ok = PyArg_ParseTuple(args, "s", &s); /* A string */
        /* Possible Python call: f('whoops!') */

    ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
        /* Possible Python call: f(1, 2, 'three') */

    ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
        /* A pair of ints and a string, whose size is also returned */
        /* Possible Python call: f((1, 2), 'three') */

    {
        const char *file;
        const char *mode = "r";
        int bufsize = 0;
        ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
        /* A string, and optionally another string and an integer */
        /* Possible Python calls:
           f('spam')
           f('spam', 'w')
           f('spam', 'wb', 100000) */
    }

    {
        int left, top, right, bottom, h, v;
        ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
                 &left, &top, &right, &bottom, &h, &v);
        /* A rectangle and a point */
        /* Possible Python call:
           f(((0, 0), (400, 300)), (10, 10)) */
    }

    {
        Py_complex c;
        ok = PyArg_ParseTuple(args, "D:myfunction", &c);
        /* a complex, also providing a function name for errors */
        /* Possible Python call: myfunction(1+2j) */
    }

6.1.8 Keyword Parameters for Extension Functions
------------------------------------------------

The PyArg_ParseTupleAndKeywords() function is declared as
follows:

    int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
                                    char *format, char *kwlist[], ...);

The _arg_ and _format_ parameters are identical to those of the
PyArg_ParseTuple() function.  The _kwdict_ parameter is the
dictionary of keywords received as the third parameter from the Python
runtime.  The _kwlist_ parameter is a _NULL_-terminated list of strings
which identify the parameters; the names are matched with the type
information from _format_ from left to right.  On success, see
PyArg_ParseTupleAndKeywords() returns true, otherwise it returns
false and raises an appropriate exception.

     Note: Nested tuples cannot be parsed when using keyword arguments!
     Keyword parameters passed in which are not present in the _kwlist_
     will cause TypeError to be raised.

  Here is an example module which uses keywords, based on an example by
Geoff Philbrick (<philbrick@hks.com>):

    #include "Python.h"

    static PyObject *
    keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
    {
        int voltage;
        char *state = "a stiff";
        char *action = "voom";
        char *type = "Norwegian Blue";

        static char *kwlist[] = {"voltage", "state", "action", "type", NULL};

        if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
                                         &voltage, &state, &action, &type))
            return NULL;

        printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
               action, voltage);
        printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);

        Py_INCREF(Py_None);

        return Py_None;
    }

    static PyMethodDef keywdarg_methods[] = {
        /* The cast of the function is necessary since PyCFunction values
         * only take two PyObject* parameters, and keywdarg_parrot() takes
         * three.
         */
        {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
         "Print a lovely skit to standard output."},
        {NULL, NULL, 0, NULL}   /* sentinel */
    };

    void
    initkeywdarg(void)
    {
      /* Create the module and add the functions */
      Py_InitModule("keywdarg", keywdarg_methods);
    }

6.1.9 Building Arbitrary Values
-------------------------------

This function is the counterpart to PyArg_ParseTuple().  It
is declared as follows:

    PyObject *Py_BuildValue(char *format, ...);

It recognizes a set of format units similar to the ones recognized by
PyArg_ParseTuple(), but the arguments (which are input to
the function, not output) must not be pointers, just values.  It
returns a new Python object, suitable for returning from a C function
called from Python.

  One difference with PyArg_ParseTuple().: while the latter
requires its first argument to be a tuple (since Python argument lists
are always represented as tuples internally), see Py_BuildValue()
does not always build a tuple.  It builds a tuple only if its
format string contains two or more format units. If the format string
is empty, it returns `None'; if it contains exactly one format unit, it
returns whatever object is described by that format unit.  To force it
to return a tuple of size 0 or one, parenthesize the format string.

  Examples (to the left the call, to the right the resulting Python
value):

    Py_BuildValue("")                        None
    Py_BuildValue("i", 123)                  123
    Py_BuildValue("iii", 123, 456, 789)      (123, 456, 789)
    Py_BuildValue("s", "hello")              'hello'
    Py_BuildValue("ss", "hello", "world")    ('hello', 'world')
    Py_BuildValue("s#", "hello", 4)          'hell'
    Py_BuildValue("()")                      ()
    Py_BuildValue("(i)", 123)                (123,)
    Py_BuildValue("(ii)", 123, 456)          (123, 456)
    Py_BuildValue("(i,i)", 123, 456)         (123, 456)
    Py_BuildValue("[i,i]", 123, 456)         [123, 456]
    Py_BuildValue("{s:i,s:i}",
                  "abc", 123, "def", 456)    {'abc': 123, 'def': 456}
    Py_BuildValue("((ii)(ii)) (ii)",
                  1, 2, 3, 4, 5, 6)          (((1, 2), (3, 4)), (5, 6))

6.1.10 Reference Counts
-----------------------

In languages like C or C++, the programmer is responsible for dynamic
allocation and deallocation of memory on the heap.  In C, this is done
using the functions `malloc()' and `free()'.  In C++, the operators
`new' and `delete' are used with essentially the same meaning and we'll
restrict the following discussion to the C case.

  Every block of memory allocated with `malloc()' should eventually be
returned to the pool of available memory by exactly one call to
`free()'.  It is important to call `free()' at the right time.  If a
block's address is forgotten but `free()' is not called for it, the
memory it occupies cannot be reused until the program terminates.  This
is called a _memory leak_.  On the other hand, if a program calls
`free()' for a block and then continues to use the block, it creates a
conflict with re-use of the block through another `malloc()' call.
This is called _using freed memory_.  It has the same bad consequences
as referencing uninitialized data -- core dumps, wrong results,
mysterious crashes.

  Common causes of memory leaks are unusual paths through the code.
For instance, a function may allocate a block of memory, do some
calculation, and then free the block again.  Now a change in the
requirements for the function may add a test to the calculation that
detects an error condition and can return prematurely from the
function.  It's easy to forget to free the allocated memory block when
taking this premature exit, especially when it is added later to the
code.  Such leaks, once introduced, often go undetected for a long
time: the error exit is taken only in a small fraction of all calls,
and most modern machines have plenty of virtual memory, so the leak
only becomes apparent in a long-running process that uses the leaking
function frequently.  Therefore, it's important to prevent leaks from
happening by having a coding convention or strategy that minimizes this
kind of errors.

  Since Python makes heavy use of `malloc()' and `free()', it needs a
strategy to avoid memory leaks as well as the use of freed memory.  The
chosen method is called _reference counting_.  The principle is simple:
every object contains a counter, which is incremented when a reference
to the object is stored somewhere, and which is decremented when a
reference to it is deleted.  When the counter reaches zero, the last
reference to the object has been deleted and the object is freed.

  An alternative strategy is called _automatic garbage collection_.
(Sometimes, reference counting is also referred to as a garbage
collection strategy, hence my use of "automatic" to distinguish the
two.)  The big advantage of automatic garbage collection is that the
user doesn't need to call `free()' explicitly.  (Another claimed
advantage is an improvement in speed or memory usage -- this is no hard
fact however.)  The disadvantage is that for C, there is no truly
portable automatic garbage collector, while reference counting can be
implemented portably (as long as the functions `malloc()' and `free()'
are available -- which the C Standard guarantees). Maybe some day a
sufficiently portable automatic garbage collector will be available for
C.  Until then, we'll have to live with reference counts.

  While Python uses the traditional reference counting implementation,
it also offers a cycle detector that works to detect reference cycles.
This allows applications to not worry about creating direct or indirect
circular references; these are the weakness of garbage collection
implemented using only reference counting.  Reference cycles consist of
objects which contain (possibly indirect) references to themselves, so
that each object in the cycle has a reference count which is non-zero.
Typical reference counting implementations are not able to reclaim the
memory belonging to any objects in a reference cycle, or referenced
from the objects in the cycle, even though there are no further
references to the cycle itself.

  The cycle detector is able to detect garbage cycles and can reclaim
them so long as there are no finalizers implemented in Python (see
__del__() methods).  When there are such finalizers, the detector
exposes the cycles through the gc module (specifically, the
`garbage' variable in that module).  The gc module also
exposes a way to run the detector (the `collect()' function), as well
as configuration interfaces and the ability to disable the detector at
runtime.  The cycle detector is considered an optional component;
though it is included by default, it can be disabled at build time
using the `--without-cycle-gc' option to the *configure* script on Unix
platforms (including Mac OS X) or by removing the definition of
`WITH_CYCLE_GC' in the `pyconfig.h' header on other platforms.  If the
cycle detector is disabled in this way, the gc.  module will
not be available.

6.1.10.1 Reference Counting in Python
.....................................

There are two macros, `Py_INCREF(x)' and `Py_DECREF(x)', which handle
the incrementing and decrementing of the reference count.
Py_DECREF() also frees the object when the count reaches zero.
For flexibility, it doesn't call `free()' directly -- rather, it makes
a call through a function pointer in the object's _type object_.  For
this purpose (and others), every object also contains a pointer to its
type object.

  The big question now remains: when to use `Py_INCREF(x)' and
`Py_DECREF(x)'?  Let's first introduce some terms.  Nobody "owns" an
object; however, you can _own a reference_ to an object.  An object's
reference count is now defined as the number of owned references to it.
The owner of a reference is responsible for calling Py_DECREF()
when the reference is no longer needed.  Ownership of a reference
can be transferred.  There are three ways to dispose of an owned
reference: pass it on, store it, or call Py_DECREF().
Forgetting to dispose of an owned reference creates a memory leak.

  It is also possible to _borrow_ (2) a reference to an object.  The
borrower of a reference should not call Py_DECREF().  The
borrower must not hold on to the object longer than the owner from
which it was borrowed.  Using a borrowed reference after the owner has
disposed of it risks using freed memory and should be avoided
completely. (3)

  The advantage of borrowing over owning a reference is that you don't
need to take care of disposing of the reference on all possible paths
through the code -- in other words, with a borrowed reference you don't
run the risk of leaking when a premature exit is taken.  The
disadvantage of borrowing over owning is that there are some subtle
situations where in seemingly correct code a borrowed reference can be
used after the owner from which it was borrowed has in fact disposed of
it.

  A borrowed reference can be changed into an owned reference by calling
Py_INCREF().  This does not affect the status of the owner
from which the reference was borrowed -- it creates a new owned
reference, and gives full owner responsibilities (the new owner must
dispose of the reference properly, as well as the previous owner).

6.1.10.2 Ownership Rules
........................

Whenever an object reference is passed into or out of a function, it is
part of the function's interface specification whether ownership is
transferred with the reference or not.

  Most functions that return a reference to an object pass on ownership
with the reference.  In particular, all functions whose function it is
to create a new object, such as PyInt_FromLong() and
Py_BuildValue(), pass ownership to the receiver.  Even if the
object is not actually new, you still receive ownership of a new
reference to that object.  For instance, see PyInt_FromLong().
maintains a cache of popular values and can return a reference to a
cached item.

  Many functions that extract objects from other objects also transfer
ownership with the reference, for instance
PyObject_GetAttrString().  The picture is less clear, here,
however, since a few common routines are exceptions:
PyTuple_GetItem(), PyList_GetItem(),
PyDict_GetItem(), and PyDict_GetItemString() all
return references that you borrow from the tuple, list or dictionary.

  The function PyImport_AddModule() also returns a
borrowed reference, even though it may actually create the object it
returns: this is possible because an owned reference to the object is
stored in `sys.modules'.

  When you pass an object reference into another function, in general,
the function borrows the reference from you -- if it needs to store it,
it will use Py_INCREF() to become an independent owner.
There are exactly two important exceptions to this rule:
PyTuple_SetItem() and PyList_SetItem().  These
functions take over ownership of the item passed to them -- even if
they fail!  (Note that PyDict_SetItem() and friends don't
take over ownership -- they are "normal.")

  When a C function is called from Python, it borrows references to its
arguments from the caller.  The caller owns a reference to the object,
so the borrowed reference's lifetime is guaranteed until the function
returns.  Only when such a borrowed reference must be stored or passed
on, it must be turned into an owned reference by calling
Py_INCREF().

  The object reference returned from a C function that is called from
Python must be an owned reference -- ownership is transferred from the
function to its caller.

6.1.10.3 Thin Ice
.................

There are a few situations where seemingly harmless use of a borrowed
reference can lead to problems.  These all have to do with implicit
invocations of the interpreter, which can cause the owner of a
reference to dispose of it.

  The first and most important case to know about is using
Py_DECREF() on an unrelated object while borrowing a reference
to a list item.  For instance:

    void
    bug(PyObject *list)
    {
        PyObject *item = PyList_GetItem(list, 0);

        PyList_SetItem(list, 1, PyInt_FromLong(0L));
        PyObject_Print(item, stdout, 0); /* BUG! */
    }

This function first borrows a reference to `list[0]', then replaces
`list[1]' with the value `0', and finally prints the borrowed reference.
Looks harmless, right?  But it's not!

  Let's follow the control flow into PyList_SetItem().  The
list owns references to all its items, so when item 1 is replaced, it
has to dispose of the original item 1.  Now let's suppose the original
item 1 was an instance of a user-defined class, and let's further
suppose that the class defined a __del__() method.  If this
class instance has a reference count of 1, disposing of it will call
its __del__() method.

  Since it is written in Python, the __del__() method can
execute arbitrary Python code.  Could it perhaps do something to
invalidate the reference to `item' in `bug()'?  You bet!  Assuming that
the list passed into `bug()' is accessible to the __del__().
method, it could execute a statement to the effect of `del list[0]',
and assuming this was the last reference to that object, it would free
the memory associated with it, thereby invalidating `item'.

  The solution, once you know the source of the problem, is easy:
temporarily increment the reference count.  The correct version of the
function reads:

    void
    no_bug(PyObject *list)
    {
        PyObject *item = PyList_GetItem(list, 0);

        Py_INCREF(item);
        PyList_SetItem(list, 1, PyInt_FromLong(0L));
        PyObject_Print(item, stdout, 0);
        Py_DECREF(item);
    }

This is a true story.  An older version of Python contained variants of
this bug and someone spent a considerable amount of time in a C
debugger to figure out why his __del__() methods would
fail...

  The second case of problems with a borrowed reference is a variant
involving threads.  Normally, multiple threads in the Python
interpreter can't get in each other's way, because there is a global
lock protecting Python's entire object space.  However, it is possible
to temporarily release this lock using the macro
Py_BEGIN_ALLOW_THREADS, and to re-acquire it using
Py_END_ALLOW_THREADS.  This is common around blocking I/O calls,
to let other threads use the processor while waiting for the I/O to
complete.  Obviously, the following function has the same problem as
the previous one:

    void
    bug(PyObject *list)
    {
        PyObject *item = PyList_GetItem(list, 0);
        Py_BEGIN_ALLOW_THREADS
        ...some blocking I/O call...
        Py_END_ALLOW_THREADS
        PyObject_Print(item, stdout, 0); /* BUG! */
    }

6.1.10.4 NULL Pointers
......................

In general, functions that take object references as arguments do not
expect you to pass them _NULL_ pointers, and will dump core (or cause
later core dumps) if you do so.  Functions that return object
references generally return _NULL_ only to indicate that an exception
occurred.  The reason for not testing for _NULL_ arguments is that
functions often pass the objects they receive on to other function --
if each function were to test for _NULL_, there would be a lot of
redundant tests and the code would run more slowly.

  It is better to test for _NULL_ only at the "source:" when a pointer
that may be _NULL_ is received, for example, from `malloc()' or from a
function that may raise an exception.

  The macros Py_INCREF() and Py_DECREF() do
not check for _NULL_ pointers -- however, their variants
Py_XINCREF() and Py_XDECREF().  do.

  The macros for checking for a particular object type
(`Pytype_Check()') don't check for _NULL_ pointers -- again, there is
much code that calls several of these in a row to test an object
against various different expected types, and this would generate
redundant tests.  There are no variants with _NULL_ checking.

  The C function calling mechanism guarantees that the argument list
passed to C functions (`args' in the examples) is never _NULL_ -- in
fact it guarantees that it is always a tuple. (4)

  It is a severe error to ever let a _NULL_ pointer "escape" to the
Python user.

6.1.11 Writing Extensions in C++
--------------------------------

It is possible to write extension modules in C++.  Some restrictions
apply.  If the main program (the Python interpreter) is compiled and
linked by the C compiler, global or static objects with constructors
cannot be used.  This is not a problem if the main program is linked by
the C++ compiler.  Functions that will be called by the Python
interpreter (in particular, module initialization functions) have to be
declared using `extern "C"'. It is unnecessary to enclose the Python
header files in `extern "C" {...}' -- they use this form already if the
symbol `__cplusplus' is defined (all recent C++ compilers define this
symbol).

6.1.12 Providing a C API for an Extension Module
------------------------------------------------

Many extension modules just provide new functions and types to be used
from Python, but sometimes the code in an extension module can be
useful for other extension modules. For example, an extension module
could implement a type "collection" which works like lists without
order. Just like the standard Python list type has a C API which
permits extension modules to create and manipulate lists, this new
collection type should have a set of C functions for direct
manipulation from other extension modules.

  At first sight this seems easy: just write the functions (without
declaring them `static', of course), provide an appropriate header
file, and document the C API. And in fact this would work if all
extension modules were always linked statically with the Python
interpreter. When modules are used as shared libraries, however, the
symbols defined in one module may not be visible to another module. The
details of visibility depend on the operating system; some systems use
one global namespace for the Python interpreter and all extension
modules (Windows, for example), whereas others require an explicit list
of imported symbols at module link time (AIX is one example), or offer
a choice of different strategies (most Unices). And even if symbols are
globally visible, the module whose functions one wishes to call might
not have been loaded yet!

  Portability therefore requires not to make any assumptions about
symbol visibility. This means that all symbols in extension modules
should be declared `static', except for the module's initialization
function, in order to avoid name clashes with other extension modules
(as discussed in section The Module's Method Table and Initialization Function).
And it means that symbols that
_should_ be accessible from other extension modules must be exported in
a different way.

  Python provides a special mechanism to pass C-level information
(pointers) from one extension module to another one: Capsules. A
Capsule is a Python data type which stores a pointer (`void *').
Capsules can only be created and accessed via their C API, but they can
be passed around like any other Python object. In particular,  they can
be assigned to a name in an extension module's namespace. Other
extension modules can then import this module, retrieve the value of
this name, and then retrieve the pointer from the Capsule.

  There are many ways in which Capsules can be used to export the C API
of an extension module. Each function could get its own Capsule, or all
C API pointers could be stored in an array whose address is published
in a Capsule. And the various tasks of storing and retrieving the
pointers can be distributed in different ways between the module
providing the code and the client modules.

  Whichever method you choose, it's important to name your Capsules
properly.  The function PyCapsule_New() takes a name
parameter (`const char *'); you're permitted to pass in a _NULL_ name,
but we strongly encourage you to specify a name.  Properly named
Capsules provide a degree of runtime type-safety; there is no feasible
way to tell one unnamed Capsule from another.

  In particular, Capsules used to expose C APIs should be given a name
following this convention:

    modulename.attributename

The convenience function PyCapsule_Import() makes it easy
to load a C API provided via a Capsule, but only if the Capsule's name
matches this convention.  This behavior gives C API users a high degree
of certainty that the Capsule they load contains the correct C API.

  The following example demonstrates an approach that puts most of the
burden on the writer of the exporting module, which is appropriate for
commonly used library modules. It stores all C API pointers (just one
in the example!) in an array of `void' pointers which becomes the value
of a Capsule. The header file corresponding to the module provides a
macro that takes care of importing the module and retrieving its C API
pointers; client modules only have to call this macro before accessing
the C API.

  The exporting module is a modification of the `spam' module from
section A Simple Example. The function `spam.system()' does
not call the C library function `system()' directly, but a function
`PySpam_System()', which would of course do something more complicated
in reality (such as adding "spam" to every command). This function
`PySpam_System()' is also exported to other extension modules.

  The function `PySpam_System()' is a plain C function, declared
`static' like everything else:

    static int
    PySpam_System(const char *command)
    {
        return system(command);
    }

The function `spam_system()' is modified in a trivial way:

    static PyObject *
    spam_system(PyObject *self, PyObject *args)
    {
        const char *command;
        int sts;

        if (!PyArg_ParseTuple(args, "s", &command))
            return NULL;
        sts = PySpam_System(command);
        return Py_BuildValue("i", sts);
    }

In the beginning of the module, right after the line

    #include "Python.h"

two more lines must be added:

    #define SPAM_MODULE
    #include "spammodule.h"

The `#define' is used to tell the header file that it is being included
in the exporting module, not a client module. Finally, the module's
initialization function must take care of initializing the C API
pointer array:

    PyMODINIT_FUNC
    initspam(void)
    {
        PyObject *m;
        static void *PySpam_API[PySpam_API_pointers];
        PyObject *c_api_object;

        m = Py_InitModule("spam", SpamMethods);
        if (m == NULL)
            return;

        /* Initialize the C API pointer array */
        PySpam_API[PySpam_System_NUM] = (void *)PySpam_System;

        /* Create a Capsule containing the API pointer array's address */
        c_api_object = PyCapsule_New((void *)PySpam_API, "spam._C_API", NULL);

        if (c_api_object != NULL)
            PyModule_AddObject(m, "_C_API", c_api_object);
    }

Note that `PySpam_API' is declared `static'; otherwise the pointer
array would disappear when `initspam()' terminates!

  The bulk of the work is in the header file `spammodule.h', which looks
like this:

    #ifndef Py_SPAMMODULE_H
    #define Py_SPAMMODULE_H
    #ifdef __cplusplus
    extern "C" {
    #endif

    /* Header file for spammodule */

    /* C API functions */
    #define PySpam_System_NUM 0
    #define PySpam_System_RETURN int
    #define PySpam_System_PROTO (const char *command)

    /* Total number of C API pointers */
    #define PySpam_API_pointers 1

    #ifdef SPAM_MODULE
    /* This section is used when compiling spammodule.c */

    static PySpam_System_RETURN PySpam_System PySpam_System_PROTO;

    #else
    /* This section is used in modules that use spammodule's API */

    static void **PySpam_API;

    #define PySpam_System \
     (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM])

    /* Return -1 on error, 0 on success.
     * PyCapsule_Import will set an exception if there's an error.
     */
    static int
    import_spam(void)
    {
        PySpam_API = (void **)PyCapsule_Import("spam._C_API", 0);
        return (PySpam_API != NULL) ? 0 : -1;
    }

    #endif

    #ifdef __cplusplus
    }
    #endif

    #endif /* !defined(Py_SPAMMODULE_H) */

All that a client module must do in order to have access to the function
`PySpam_System()' is to call the function (or rather macro)
`import_spam()' in its initialization function:

    PyMODINIT_FUNC
    initclient(void)
    {
        PyObject *m;

        m = Py_InitModule("client", ClientMethods);
        if (m == NULL)
            return;
        if (import_spam() < 0)
            return;
        /* additional initialization can happen here */
    }

The main disadvantage of this approach is that the file `spammodule.h'
is rather complicated. However, the basic structure is the same for
each function that is exported, so it has to be learned only once.

  Finally it should be mentioned that Capsules offer additional
functionality, which is especially useful for memory allocation and
deallocation of the pointer stored in a Capsule. The details are
described in the Python/C API Reference Manual in the section
Capsules and in the implementation of Capsules (files
`Include/pycapsule.h' and `Objects/pycapsule.c' in the Python source
code distribution).

Footnotes
.........

     (1) An interface for this function already exists in the standard
     module os.  -- it was chosen as a simple and
     straightforward example.

     (2) The metaphor of "borrowing" a reference is not completely
     correct: the owner still has a copy of the reference.

     (3) Checking that the reference count is at least 1 *does not
     work* -- the reference count itself could be in freed memory and
     may thus be reused for another object!

     (4) These guarantees don't hold when you use the "old" style
     calling convention -- this is still found in much existing code.

6.2 Defining New Types
======================

As mentioned in the last chapter, Python allows the writer of an
extension module to define new types that can be manipulated from
Python code, much like strings and lists in core Python.

  This is not hard; the code for all extension types follows a pattern,
but there are some details that you need to understand before you can
get started.

     Note: The way new types are defined changed dramatically (and for
     the better) in Python 2.2.  This document documents how to define
     new types for Python 2.2 and later.  If you need to support older
     versions of Python, you will need to refer to older versions of
     this documentation (http://www.python.org/doc/versions/).

6.2.1 The Basics
----------------

The Python runtime sees all Python objects as variables of type
PyObject*.  A PyObject is not a very magnificent
object - it just contains the refcount and a pointer to the object's
"type object".  This is where the action is; the type object determines
which (C) functions get called when, for instance, an attribute gets
looked up on an object or it is multiplied by another object.  These C
functions are called "type methods" to distinguish them from things
like `[].append' (which we call "object methods").

  So, if you want to define a new object type, you need to create a new
type object.

  This sort of thing can only be explained by example, so here's a
minimal, but complete, module that defines a new type:

    #include <Python.h>

    typedef struct {
        PyObject_HEAD
        /* Type-specific fields go here. */
    } noddy_NoddyObject;

    static PyTypeObject noddy_NoddyType = {
        PyObject_HEAD_INIT(NULL)
        0,                         /*ob_size*/
        "noddy.Noddy",             /*tp_name*/
        sizeof(noddy_NoddyObject), /*tp_basicsize*/
        0,                         /*tp_itemsize*/
        0,                         /*tp_dealloc*/
        0,                         /*tp_print*/
        0,                         /*tp_getattr*/
        0,                         /*tp_setattr*/
        0,                         /*tp_compare*/
        0,                         /*tp_repr*/
        0,                         /*tp_as_number*/
        0,                         /*tp_as_sequence*/
        0,                         /*tp_as_mapping*/
        0,                         /*tp_hash */
        0,                         /*tp_call*/
        0,                         /*tp_str*/
        0,                         /*tp_getattro*/
        0,                         /*tp_setattro*/
        0,                         /*tp_as_buffer*/
        Py_TPFLAGS_DEFAULT,        /*tp_flags*/
        "Noddy objects",           /* tp_doc */
    };

    static PyMethodDef noddy_methods[] = {
        {NULL}  /* Sentinel */
    };

    #ifndef PyMODINIT_FUNC	/* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif
    PyMODINIT_FUNC
    initnoddy(void)
    {
        PyObject* m;

        noddy_NoddyType.tp_new = PyType_GenericNew;
        if (PyType_Ready(&noddy_NoddyType) < 0)
            return;

        m = Py_InitModule3("noddy", noddy_methods,
                           "Example module that creates an extension type.");

        Py_INCREF(&noddy_NoddyType);
        PyModule_AddObject(m, "Noddy", (PyObject *)&noddy_NoddyType);
    }

Now that's quite a bit to take in at once, but hopefully bits will seem
familiar from the last chapter.

  The first bit that will be new is:

    typedef struct {
        PyObject_HEAD
    } noddy_NoddyObject;

This is what a Noddy object will contain--in this case, nothing more
than every Python object contains, namely a refcount and a pointer to a
type object.  These are the fields the `PyObject_HEAD' macro brings in.
The reason for the macro is to standardize the layout and to enable
special debugging fields in debug builds.  Note that there is no
semicolon after the `PyObject_HEAD' macro; one is included in the macro
definition.  Be wary of adding one by accident; it's easy to do from
habit, and your compiler might not complain, but someone else's
probably will!  (On Windows, MSVC is known to call this an error and
refuse to compile the code.)

  For contrast, let's take a look at the corresponding definition for
standard Python integers:

    typedef struct {
        PyObject_HEAD
        long ob_ival;
    } PyIntObject;

Moving on, we come to the crunch -- the type object.

    static PyTypeObject noddy_NoddyType = {
        PyObject_HEAD_INIT(NULL)
        0,                         /*ob_size*/
        "noddy.Noddy",             /*tp_name*/
        sizeof(noddy_NoddyObject), /*tp_basicsize*/
        0,                         /*tp_itemsize*/
        0,                         /*tp_dealloc*/
        0,                         /*tp_print*/
        0,                         /*tp_getattr*/
        0,                         /*tp_setattr*/
        0,                         /*tp_compare*/
        0,                         /*tp_repr*/
        0,                         /*tp_as_number*/
        0,                         /*tp_as_sequence*/
        0,                         /*tp_as_mapping*/
        0,                         /*tp_hash */
        0,                         /*tp_call*/
        0,                         /*tp_str*/
        0,                         /*tp_getattro*/
        0,                         /*tp_setattro*/
        0,                         /*tp_as_buffer*/
        Py_TPFLAGS_DEFAULT,        /*tp_flags*/
        "Noddy objects",           /* tp_doc */
    };

Now if you go and look up the definition of PyTypeObject in
`object.h' you'll see that it has many more fields that the definition
above.  The remaining fields will be filled with zeros by the C
compiler, and it's common practice to not specify them explicitly
unless you need them.

  This is so important that we're going to pick the top of it apart
still further:

    PyObject_HEAD_INIT(NULL)

This line is a bit of a wart; what we'd like to write is:

    PyObject_HEAD_INIT(&PyType_Type)

as the type of a type object is "type", but this isn't strictly
conforming C and some compilers complain.  Fortunately, this member
will be filled in for us by PyType_Ready().

    0,                          /* ob_size */

The `ob_size' field of the header is not used; its presence in the type
structure is a historical artifact that is maintained for binary
compatibility with extension modules compiled for older versions of
Python.  Always set this field to zero.

    "noddy.Noddy",              /* tp_name */

The name of our type.  This will appear in the default textual
representation of our objects and in some error messages, for example:

    >>> "" + noddy.new_noddy()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: cannot add type "noddy.Noddy" to string

Note that the name is a dotted name that includes both the module name
and the name of the type within the module. The module in this case is
`noddy' and the type is `Noddy', so we set the type name to
`noddy.Noddy'.

    sizeof(noddy_NoddyObject),  /* tp_basicsize */

This is so that Python knows how much memory to allocate when you call
PyObject_New().

     Note: If you want your type to be subclassable from Python, and
     your type has the same `tp_basicsize' as its base type, you may
     have problems with multiple inheritance.  A Python subclass of
     your type will have to list your type first in its `__bases__', or
     else it will not be able to call your type's __new__().
     method without getting an error.  You can avoid this problem by
     ensuring that your type has a larger value for `tp_basicsize' than
     its base type does.  Most of the time, this will be true anyway,
     because either your base type will be object, or else
     you will be adding data members to your base type, and therefore
     increasing its size.

    0,                          /* tp_itemsize */

This has to do with variable length objects like lists and strings.
Ignore this for now.

  Skipping a number of type methods that we don't provide, we set the
class flags to Py_TPFLAGS_DEFAULT.

    Py_TPFLAGS_DEFAULT,        /*tp_flags*/

All types should include this constant in their flags.  It enables all
of the members defined by the current version of Python.

  We provide a doc string for the type in `tp_doc'.

    "Noddy objects",           /* tp_doc */

Now we get into the type methods, the things that make your objects
different from the others.  We aren't going to implement any of these
in this version of the module.  We'll expand this example later to have
more interesting behavior.

  For now, all we want to be able to do is to create new `Noddy'
objects.  To enable object creation, we have to provide a `tp_new'
implementation.  In this case, we can just use the default
implementation provided by the API function PyType_GenericNew().
We'd like to just assign this to the `tp_new' slot, but we
can't, for portability sake, On some platforms or compilers, we can't
statically initialize a structure member with a function defined in
another C module, so, instead, we'll assign the `tp_new' slot in the
module initialization function just before calling
PyType_Ready().:

    noddy_NoddyType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&noddy_NoddyType) < 0)
        return;

All the other type methods are _NULL_, so we'll go over them later --
that's for a later section!

  Everything else in the file should be familiar, except for some code
in `initnoddy()':

    if (PyType_Ready(&noddy_NoddyType) < 0)
        return;

This initializes the `Noddy' type, filing in a number of members,
including `ob_type' that we initially set to _NULL_.

    PyModule_AddObject(m, "Noddy", (PyObject *)&noddy_NoddyType);

This adds the type to the module dictionary.  This allows us to create
`Noddy' instances by calling the `Noddy' class:

    >>> import noddy
    >>> mynoddy = noddy.Noddy()

That's it!  All that remains is to build it; put the above code in a
file called `noddy.c' and

    from distutils.core import setup, Extension
    setup(name="noddy", version="1.0",
          ext_modules=[Extension("noddy", ["noddy.c"])])

in a file called `setup.py'; then typing

    $ python setup.py build

at a shell should produce a file `noddy.so' in a subdirectory; move to
that directory and fire up Python -- you should be able to `import
noddy' and play around with Noddy objects.

  That wasn't so hard, was it?

  Of course, the current Noddy type is pretty uninteresting. It has no
data and doesn't do anything. It can't even be subclassed.

6.2.1.1 Adding data and methods to the Basic example
....................................................

Let's expend the basic example to add some data and methods.  Let's
also make the type usable as a base class. We'll create a new module,
`noddy2' that adds these capabilities:

    #include <Python.h>
    #include "structmember.h"

    typedef struct {
        PyObject_HEAD
        PyObject *first; /* first name */
        PyObject *last;  /* last name */
        int number;
    } Noddy;

    static void
    Noddy_dealloc(Noddy* self)
    {
        Py_XDECREF(self->first);
        Py_XDECREF(self->last);
        self->ob_type->tp_free((PyObject*)self);
    }

    static PyObject *
    Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
    {
        Noddy *self;

        self = (Noddy *)type->tp_alloc(type, 0);
        if (self != NULL) {
            self->first = PyString_FromString("");
            if (self->first == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->last = PyString_FromString("");
            if (self->last == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->number = 0;
        }

        return (PyObject *)self;
    }

    static int
    Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
    {
        PyObject *first=NULL, *last=NULL, *tmp;

        static char *kwlist[] = {"first", "last", "number", NULL};

        if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
                                          &first, &last,
                                          &self->number))
            return -1;

        if (first) {
            tmp = self->first;
            Py_INCREF(first);
            self->first = first;
            Py_XDECREF(tmp);
        }

        if (last) {
            tmp = self->last;
            Py_INCREF(last);
            self->last = last;
            Py_XDECREF(tmp);
        }

        return 0;
    }

    static PyMemberDef Noddy_members[] = {
        {"first", T_OBJECT_EX, offsetof(Noddy, first), 0,
         "first name"},
        {"last", T_OBJECT_EX, offsetof(Noddy, last), 0,
         "last name"},
        {"number", T_INT, offsetof(Noddy, number), 0,
         "noddy number"},
        {NULL}  /* Sentinel */
    };

    static PyObject *
    Noddy_name(Noddy* self)
    {
        static PyObject *format = NULL;
        PyObject *args, *result;

        if (format == NULL) {
            format = PyString_FromString("%s %s");
            if (format == NULL)
                return NULL;
        }

        if (self->first == NULL) {
            PyErr_SetString(PyExc_AttributeError, "first");
            return NULL;
        }

        if (self->last == NULL) {
            PyErr_SetString(PyExc_AttributeError, "last");
            return NULL;
        }

        args = Py_BuildValue("OO", self->first, self->last);
        if (args == NULL)
            return NULL;

        result = PyString_Format(format, args);
        Py_DECREF(args);

        return result;
    }

    static PyMethodDef Noddy_methods[] = {
        {"name", (PyCFunction)Noddy_name, METH_NOARGS,
         "Return the name, combining the first and last name"
        },
        {NULL}  /* Sentinel */
    };

    static PyTypeObject NoddyType = {
        PyObject_HEAD_INIT(NULL)
        0,                         /*ob_size*/
        "noddy.Noddy",             /*tp_name*/
        sizeof(Noddy),             /*tp_basicsize*/
        0,                         /*tp_itemsize*/
        (destructor)Noddy_dealloc, /*tp_dealloc*/
        0,                         /*tp_print*/
        0,                         /*tp_getattr*/
        0,                         /*tp_setattr*/
        0,                         /*tp_compare*/
        0,                         /*tp_repr*/
        0,                         /*tp_as_number*/
        0,                         /*tp_as_sequence*/
        0,                         /*tp_as_mapping*/
        0,                         /*tp_hash */
        0,                         /*tp_call*/
        0,                         /*tp_str*/
        0,                         /*tp_getattro*/
        0,                         /*tp_setattro*/
        0,                         /*tp_as_buffer*/
        Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
        "Noddy objects",           /* tp_doc */
        0,		               /* tp_traverse */
        0,		               /* tp_clear */
        0,		               /* tp_richcompare */
        0,		               /* tp_weaklistoffset */
        0,		               /* tp_iter */
        0,		               /* tp_iternext */
        Noddy_methods,             /* tp_methods */
        Noddy_members,             /* tp_members */
        0,                         /* tp_getset */
        0,                         /* tp_base */
        0,                         /* tp_dict */
        0,                         /* tp_descr_get */
        0,                         /* tp_descr_set */
        0,                         /* tp_dictoffset */
        (initproc)Noddy_init,      /* tp_init */
        0,                         /* tp_alloc */
        Noddy_new,                 /* tp_new */
    };

    static PyMethodDef module_methods[] = {
        {NULL}  /* Sentinel */
    };

    #ifndef PyMODINIT_FUNC	/* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif
    PyMODINIT_FUNC
    initnoddy2(void)
    {
        PyObject* m;

        if (PyType_Ready(&NoddyType) < 0)
            return;

        m = Py_InitModule3("noddy2", module_methods,
                           "Example module that creates an extension type.");

        if (m == NULL)
          return;

        Py_INCREF(&NoddyType);
        PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
    }

This version of the module has a number of changes.

  We've added an extra include:

    #include <structmember.h>

This include provides declarations that we use to handle attributes, as
described a bit later.

  The name of the `Noddy' object structure has been shortened to
`Noddy'.  The type object name has been shortened to `NoddyType'.

  The  `Noddy' type now has three data attributes, _first_, _last_, and
_number_.  The _first_ and _last_ variables are Python strings
containing first and last names. The _number_ attribute is an integer.

  The object structure is updated accordingly:

    typedef struct {
        PyObject_HEAD
        PyObject *first;
        PyObject *last;
        int number;
    } Noddy;

Because we now have data to manage, we have to be more careful about
object allocation and deallocation.  At a minimum, we need a
deallocation method:

    static void
    Noddy_dealloc(Noddy* self)
    {
        Py_XDECREF(self->first);
        Py_XDECREF(self->last);
        self->ob_type->tp_free((PyObject*)self);
    }

which is assigned to the `tp_dealloc' member:

    (destructor)Noddy_dealloc, /*tp_dealloc*/

This method decrements the reference counts of the two Python
attributes. We use Py_XDECREF() here because the `first'
and `last' members could be _NULL_.  It then calls the `tp_free' member
of the object's type to free the object's memory.  Note that the
object's type might not be `NoddyType', because the object may be an
instance of a subclass.

  We want to make sure that the first and last names are initialized to
empty strings, so we provide a new method:

    static PyObject *
    Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
    {
        Noddy *self;

        self = (Noddy *)type->tp_alloc(type, 0);
        if (self != NULL) {
            self->first = PyString_FromString("");
            if (self->first == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->last = PyString_FromString("");
            if (self->last == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->number = 0;
        }

        return (PyObject *)self;
    }

and install it in the `tp_new' member:

    Noddy_new,                 /* tp_new */

The new member is responsible for creating (as opposed to initializing)
objects of the type.  It is exposed in Python as the __new__()
method.  See the paper titled "Unifying types and classes in
Python" for a detailed discussion of the __new__() method.
One reason to implement a new method is to assure the initial values of
instance variables.  In this case, we use the new method to make sure
that the initial values of the members `first' and `last' are not
_NULL_. If we didn't care whether the initial values were _NULL_, we
could have used PyType_GenericNew() as our new method, as
we did before.  PyType_GenericNew() initializes all of the
instance variable members to _NULL_.

  The new method is a static method that is passed the type being
instantiated and any arguments passed when the type was called, and
that returns the new object created. New methods always accept
positional and keyword arguments, but they often ignore the arguments,
leaving the argument handling to initializer methods. Note that if the
type supports subclassing, the type passed may not be the type being
defined.  The new method calls the tp_alloc slot to allocate memory. We
don't fill the `tp_alloc' slot ourselves. Rather PyType_Ready()
fills it for us by inheriting it from our base class, which is
object by default.  Most types use the default allocation.

     Note: If you are creating a co-operative `tp_new' (one that calls
     a base type's `tp_new' or __new__()), you must _not_
     try to determine what method to call using method resolution order
     at runtime.  Always statically determine what type you are going
     to call, and call its `tp_new' directly, or via
     `type->tp_base->tp_new'.  If you do not do this, Python subclasses
     of your type that also inherit from other Python-defined classes
     may not work correctly.  (Specifically, you may not be able to
     create instances of such subclasses without getting a
     TypeError)

  We provide an initialization function:

    static int
    Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
    {
        PyObject *first=NULL, *last=NULL, *tmp;

        static char *kwlist[] = {"first", "last", "number", NULL};

        if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
                                          &first, &last,
                                          &self->number))
            return -1;

        if (first) {
            tmp = self->first;
            Py_INCREF(first);
            self->first = first;
            Py_XDECREF(tmp);
        }

        if (last) {
            tmp = self->last;
            Py_INCREF(last);
            self->last = last;
            Py_XDECREF(tmp);
        }

        return 0;
    }

by filling the `tp_init' slot.

    (initproc)Noddy_init,         /* tp_init */

The `tp_init' slot is exposed in Python as the __init__().
method. It is used to initialize an object after it's created. Unlike
the new method, we can't guarantee that the initializer is called.  The
initializer isn't called when unpickling objects and it can be
overridden.  Our initializer accepts arguments to provide initial
values for our instance. Initializers always accept positional and
keyword arguments.

  Initializers can be called multiple times.  Anyone can call the
__init__().  method on our objects.  For this reason, we have to
be extra careful when assigning the new values.  We might be tempted,
for example to assign the `first' member like this:

    if (first) {
        Py_XDECREF(self->first);
        Py_INCREF(first);
        self->first = first;
    }

But this would be risky.  Our type doesn't restrict the type of the
`first' member, so it could be any kind of object.  It could have a
destructor that causes code to be executed that tries to access the
`first' member.  To be paranoid and protect ourselves against this
possibility, we almost always reassign members before decrementing their
reference counts.  When don't we have to do this?

   * when we absolutely know that the reference count is greater than 1

   * when we know that deallocation of the object (1) will not cause
     any calls back into our type's code

   * when decrementing a reference count in a `tp_dealloc' handler when
     garbage-collections is not supported (2)

  We want to expose our instance variables as attributes. There are a
number of ways to do that. The simplest way is to define member
definitions:

    static PyMemberDef Noddy_members[] = {
        {"first", T_OBJECT_EX, offsetof(Noddy, first), 0,
         "first name"},
        {"last", T_OBJECT_EX, offsetof(Noddy, last), 0,
         "last name"},
        {"number", T_INT, offsetof(Noddy, number), 0,
         "noddy number"},
        {NULL}  /* Sentinel */
    };

and put the definitions in the `tp_members' slot:

    Noddy_members,             /* tp_members */

Each member definition has a member name, type, offset, access flags and
documentation string. See the Generic Attribute Management.
section below for details.

  A disadvantage of this approach is that it doesn't provide a way to
restrict the types of objects that can be assigned to the Python
attributes.  We expect the first and last names to be strings, but any
Python objects can be assigned.  Further, the attributes can be
deleted, setting the C pointers to _NULL_.  Even though we can make
sure the members are initialized to non-_NULL_ values, the members can
be set to _NULL_ if the attributes are deleted.

  We define a single method, `name()', that outputs the objects name as
the concatenation of the first and last names.

    static PyObject *
    Noddy_name(Noddy* self)
    {
        static PyObject *format = NULL;
        PyObject *args, *result;

        if (format == NULL) {
            format = PyString_FromString("%s %s");
            if (format == NULL)
                return NULL;
        }

        if (self->first == NULL) {
            PyErr_SetString(PyExc_AttributeError, "first");
            return NULL;
        }

        if (self->last == NULL) {
            PyErr_SetString(PyExc_AttributeError, "last");
            return NULL;
        }

        args = Py_BuildValue("OO", self->first, self->last);
        if (args == NULL)
            return NULL;

        result = PyString_Format(format, args);
        Py_DECREF(args);

        return result;
    }

The method is implemented as a C function that takes a `Noddy' (or
`Noddy' subclass) instance as the first argument.  Methods always take
an instance as the first argument. Methods often take positional and
keyword arguments as well, but in this cased we don't take any and
don't need to accept a positional argument tuple or keyword argument
dictionary. This method is equivalent to the Python method:

    def name(self):
       return "%s %s" % (self.first, self.last)

Note that we have to check for the possibility that our `first' and
`last' members are _NULL_.  This is because they can be deleted, in
which case they are set to _NULL_.  It would be better to prevent
deletion of these attributes and to restrict the attribute values to be
strings.  We'll see how to do that in the next section.

  Now that we've defined the method, we need to create an array of
method definitions:

    static PyMethodDef Noddy_methods[] = {
        {"name", (PyCFunction)Noddy_name, METH_NOARGS,
         "Return the name, combining the first and last name"
        },
        {NULL}  /* Sentinel */
    };

and assign them to the `tp_methods' slot:

    Noddy_methods,             /* tp_methods */

Note that we used the METH_NOARGS flag to indicate that the
method is passed no arguments.

  Finally, we'll make our type usable as a base class.  We've written
our methods carefully so far so that they don't make any assumptions
about the type of the object being created or used, so all we need to
do is to add the Py_TPFLAGS_BASETYPE to our class flag
definition:

    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/

We rename `initnoddy()' to `initnoddy2()' and update the module name
passed to Py_InitModule3().

  Finally, we update our `setup.py' file to build the new module:

    from distutils.core import setup, Extension
    setup(name="noddy", version="1.0",
          ext_modules=[
             Extension("noddy", ["noddy.c"]),
             Extension("noddy2", ["noddy2.c"]),
             ])

6.2.1.2 Providing finer control over data attributes
....................................................

In this section, we'll provide finer control over how the `first' and
`last' attributes are set in the `Noddy' example. In the previous
version of our module, the instance variables `first' and `last' could
be set to non-string values or even deleted. We want to make sure that
these attributes always contain strings.

    #include <Python.h>
    #include "structmember.h"

    typedef struct {
        PyObject_HEAD
        PyObject *first;
        PyObject *last;
        int number;
    } Noddy;

    static void
    Noddy_dealloc(Noddy* self)
    {
        Py_XDECREF(self->first);
        Py_XDECREF(self->last);
        self->ob_type->tp_free((PyObject*)self);
    }

    static PyObject *
    Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
    {
        Noddy *self;

        self = (Noddy *)type->tp_alloc(type, 0);
        if (self != NULL) {
            self->first = PyString_FromString("");
            if (self->first == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->last = PyString_FromString("");
            if (self->last == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->number = 0;
        }

        return (PyObject *)self;
    }

    static int
    Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
    {
        PyObject *first=NULL, *last=NULL, *tmp;

        static char *kwlist[] = {"first", "last", "number", NULL};

        if (! PyArg_ParseTupleAndKeywords(args, kwds, "|SSi", kwlist,
                                          &first, &last,
                                          &self->number))
            return -1;

        if (first) {
            tmp = self->first;
            Py_INCREF(first);
            self->first = first;
            Py_DECREF(tmp);
        }

        if (last) {
            tmp = self->last;
            Py_INCREF(last);
            self->last = last;
            Py_DECREF(tmp);
        }

        return 0;
    }

    static PyMemberDef Noddy_members[] = {
        {"number", T_INT, offsetof(Noddy, number), 0,
         "noddy number"},
        {NULL}  /* Sentinel */
    };

    static PyObject *
    Noddy_getfirst(Noddy *self, void *closure)
    {
        Py_INCREF(self->first);
        return self->first;
    }

    static int
    Noddy_setfirst(Noddy *self, PyObject *value, void *closure)
    {
      if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
        return -1;
      }

      if (! PyString_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The first attribute value must be a string");
        return -1;
      }

      Py_DECREF(self->first);
      Py_INCREF(value);
      self->first = value;

      return 0;
    }

    static PyObject *
    Noddy_getlast(Noddy *self, void *closure)
    {
        Py_INCREF(self->last);
        return self->last;
    }

    static int
    Noddy_setlast(Noddy *self, PyObject *value, void *closure)
    {
      if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");
        return -1;
      }

      if (! PyString_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The last attribute value must be a string");
        return -1;
      }

      Py_DECREF(self->last);
      Py_INCREF(value);
      self->last = value;

      return 0;
    }

    static PyGetSetDef Noddy_getseters[] = {
        {"first",
         (getter)Noddy_getfirst, (setter)Noddy_setfirst,
         "first name",
         NULL},
        {"last",
         (getter)Noddy_getlast, (setter)Noddy_setlast,
         "last name",
         NULL},
        {NULL}  /* Sentinel */
    };

    static PyObject *
    Noddy_name(Noddy* self)
    {
        static PyObject *format = NULL;
        PyObject *args, *result;

        if (format == NULL) {
            format = PyString_FromString("%s %s");
            if (format == NULL)
                return NULL;
        }

        args = Py_BuildValue("OO", self->first, self->last);
        if (args == NULL)
            return NULL;

        result = PyString_Format(format, args);
        Py_DECREF(args);

        return result;
    }

    static PyMethodDef Noddy_methods[] = {
        {"name", (PyCFunction)Noddy_name, METH_NOARGS,
         "Return the name, combining the first and last name"
        },
        {NULL}  /* Sentinel */
    };

    static PyTypeObject NoddyType = {
        PyObject_HEAD_INIT(NULL)
        0,                         /*ob_size*/
        "noddy.Noddy",             /*tp_name*/
        sizeof(Noddy),             /*tp_basicsize*/
        0,                         /*tp_itemsize*/
        (destructor)Noddy_dealloc, /*tp_dealloc*/
        0,                         /*tp_print*/
        0,                         /*tp_getattr*/
        0,                         /*tp_setattr*/
        0,                         /*tp_compare*/
        0,                         /*tp_repr*/
        0,                         /*tp_as_number*/
        0,                         /*tp_as_sequence*/
        0,                         /*tp_as_mapping*/
        0,                         /*tp_hash */
        0,                         /*tp_call*/
        0,                         /*tp_str*/
        0,                         /*tp_getattro*/
        0,                         /*tp_setattro*/
        0,                         /*tp_as_buffer*/
        Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
        "Noddy objects",           /* tp_doc */
        0,		               /* tp_traverse */
        0,		               /* tp_clear */
        0,		               /* tp_richcompare */
        0,		               /* tp_weaklistoffset */
        0,		               /* tp_iter */
        0,		               /* tp_iternext */
        Noddy_methods,             /* tp_methods */
        Noddy_members,             /* tp_members */
        Noddy_getseters,           /* tp_getset */
        0,                         /* tp_base */
        0,                         /* tp_dict */
        0,                         /* tp_descr_get */
        0,                         /* tp_descr_set */
        0,                         /* tp_dictoffset */
        (initproc)Noddy_init,      /* tp_init */
        0,                         /* tp_alloc */
        Noddy_new,                 /* tp_new */
    };

    static PyMethodDef module_methods[] = {
        {NULL}  /* Sentinel */
    };

    #ifndef PyMODINIT_FUNC	/* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif
    PyMODINIT_FUNC
    initnoddy3(void)
    {
        PyObject* m;

        if (PyType_Ready(&NoddyType) < 0)
            return;

        m = Py_InitModule3("noddy3", module_methods,
                           "Example module that creates an extension type.");

        if (m == NULL)
          return;

        Py_INCREF(&NoddyType);
        PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
    }

To provide greater control, over the `first' and `last' attributes,
we'll use custom getter and setter functions.  Here are the functions
for getting and setting the `first' attribute:

    Noddy_getfirst(Noddy *self, void *closure)
    {
        Py_INCREF(self->first);
        return self->first;
    }

    static int
    Noddy_setfirst(Noddy *self, PyObject *value, void *closure)
    {
      if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
        return -1;
      }

      if (! PyString_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The first attribute value must be a string");
        return -1;
      }

      Py_DECREF(self->first);
      Py_INCREF(value);
      self->first = value;

      return 0;
    }

The getter function is passed a `Noddy' object and a "closure", which is
void pointer. In this case, the closure is ignored. (The closure
supports an advanced usage in which definition data is passed to the
getter and setter. This could, for example, be used to allow a single
set of getter and setter functions that decide the attribute to get or
set based on data in the closure.)

  The setter function is passed the `Noddy' object, the new value, and
the closure. The new value may be _NULL_, in which case the attribute
is being deleted.  In our setter, we raise an error if the attribute is
deleted or if the attribute value is not a string.

  We create an array of `PyGetSetDef' structures:

    static PyGetSetDef Noddy_getseters[] = {
        {"first",
         (getter)Noddy_getfirst, (setter)Noddy_setfirst,
         "first name",
         NULL},
        {"last",
         (getter)Noddy_getlast, (setter)Noddy_setlast,
         "last name",
         NULL},
        {NULL}  /* Sentinel */
    };

and register it in the `tp_getset' slot:

    Noddy_getseters,           /* tp_getset */

to register our attribute getters and setters.

  The last item in a `PyGetSetDef' structure is the closure mentioned
above. In this case, we aren't using the closure, so we just pass
_NULL_.

  We also remove the member definitions for these attributes:

    static PyMemberDef Noddy_members[] = {
        {"number", T_INT, offsetof(Noddy, number), 0,
         "noddy number"},
        {NULL}  /* Sentinel */
    };

We also need to update the `tp_init' handler to only allow strings (3)
to be passed:

    static int
    Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
    {
        PyObject *first=NULL, *last=NULL, *tmp;

        static char *kwlist[] = {"first", "last", "number", NULL};

        if (! PyArg_ParseTupleAndKeywords(args, kwds, "|SSi", kwlist,
                                          &first, &last,
                                          &self->number))
            return -1;

        if (first) {
            tmp = self->first;
            Py_INCREF(first);
            self->first = first;
            Py_DECREF(tmp);
        }

        if (last) {
            tmp = self->last;
            Py_INCREF(last);
            self->last = last;
            Py_DECREF(tmp);
        }

        return 0;
    }

With these changes, we can assure that the `first' and `last' members
are never _NULL_ so we can remove checks for _NULL_ values in almost all
cases. This means that most of the Py_XDECREF() calls can
be converted to Py_DECREF() calls. The only place we can't
change these calls is in the deallocator, where there is the
possibility that the initialization of these members failed in the
constructor.

  We also rename the module initialization function and module name in
the initialization function, as we did before, and we add an extra
definition to the `setup.py' file.

6.2.1.3 Supporting cyclic garbage collection
............................................

Python has a cyclic-garbage collector that can identify unneeded
objects even when their reference counts are not zero. This can happen
when objects are involved in cycles.  For example, consider:

    >>> l = []
    >>> l.append(l)
    >>> del l

In this example, we create a list that contains itself. When we delete
it, it still has a reference from itself. Its reference count doesn't
drop to zero.  Fortunately, Python's cyclic-garbage collector will
eventually figure out that the list is garbage and free it.

  In the second version of the `Noddy' example, we allowed any kind of
object to be stored in the `first' or `last' attributes. (4) This means
that `Noddy' objects can participate in cycles:

    >>> import noddy2
    >>> n = noddy2.Noddy()
    >>> l = [n]
    >>> n.first = l

This is pretty silly, but it gives us an excuse to add support for the
cyclic-garbage collector to the `Noddy' example.  To support cyclic
garbage collection, types need to fill two slots and set a class flag
that enables these slots:

    #include <Python.h>
    #include "structmember.h"

    typedef struct {
        PyObject_HEAD
        PyObject *first;
        PyObject *last;
        int number;
    } Noddy;

    static int
    Noddy_traverse(Noddy *self, visitproc visit, void *arg)
    {
        int vret;

        if (self->first) {
            vret = visit(self->first, arg);
            if (vret != 0)
                return vret;
        }
        if (self->last) {
            vret = visit(self->last, arg);
            if (vret != 0)
                return vret;
        }

        return 0;
    }

    static int
    Noddy_clear(Noddy *self)
    {
        PyObject *tmp;

        tmp = self->first;
        self->first = NULL;
        Py_XDECREF(tmp);

        tmp = self->last;
        self->last = NULL;
        Py_XDECREF(tmp);

        return 0;
    }

    static void
    Noddy_dealloc(Noddy* self)
    {
        Noddy_clear(self);
        self->ob_type->tp_free((PyObject*)self);
    }

    static PyObject *
    Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
    {
        Noddy *self;

        self = (Noddy *)type->tp_alloc(type, 0);
        if (self != NULL) {
            self->first = PyString_FromString("");
            if (self->first == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->last = PyString_FromString("");
            if (self->last == NULL)
              {
                Py_DECREF(self);
                return NULL;
              }

            self->number = 0;
        }

        return (PyObject *)self;
    }

    static int
    Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
    {
        PyObject *first=NULL, *last=NULL, *tmp;

        static char *kwlist[] = {"first", "last", "number", NULL};

        if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
                                          &first, &last,
                                          &self->number))
            return -1;

        if (first) {
            tmp = self->first;
            Py_INCREF(first);
            self->first = first;
            Py_XDECREF(tmp);
        }

        if (last) {
            tmp = self->last;
            Py_INCREF(last);
            self->last = last;
            Py_XDECREF(tmp);
        }

        return 0;
    }

    static PyMemberDef Noddy_members[] = {
        {"first", T_OBJECT_EX, offsetof(Noddy, first), 0,
         "first name"},
        {"last", T_OBJECT_EX, offsetof(Noddy, last), 0,
         "last name"},
        {"number", T_INT, offsetof(Noddy, number), 0,
         "noddy number"},
        {NULL}  /* Sentinel */
    };

    static PyObject *
    Noddy_name(Noddy* self)
    {
        static PyObject *format = NULL;
        PyObject *args, *result;

        if (format == NULL) {
            format = PyString_FromString("%s %s");
            if (format == NULL)
                return NULL;
        }

        if (self->first == NULL) {
            PyErr_SetString(PyExc_AttributeError, "first");
            return NULL;
        }

        if (self->last == NULL) {
            PyErr_SetString(PyExc_AttributeError, "last");
            return NULL;
        }

        args = Py_BuildValue("OO", self->first, self->last);
        if (args == NULL)
            return NULL;

        result = PyString_Format(format, args);
        Py_DECREF(args);

        return result;
    }

    static PyMethodDef Noddy_methods[] = {
        {"name", (PyCFunction)Noddy_name, METH_NOARGS,
         "Return the name, combining the first and last name"
        },
        {NULL}  /* Sentinel */
    };

    static PyTypeObject NoddyType = {
        PyObject_HEAD_INIT(NULL)
        0,                         /*ob_size*/
        "noddy.Noddy",             /*tp_name*/
        sizeof(Noddy),             /*tp_basicsize*/
        0,                         /*tp_itemsize*/
        (destructor)Noddy_dealloc, /*tp_dealloc*/
        0,                         /*tp_print*/
        0,                         /*tp_getattr*/
        0,                         /*tp_setattr*/
        0,                         /*tp_compare*/
        0,                         /*tp_repr*/
        0,                         /*tp_as_number*/
        0,                         /*tp_as_sequence*/
        0,                         /*tp_as_mapping*/
        0,                         /*tp_hash */
        0,                         /*tp_call*/
        0,                         /*tp_str*/
        0,                         /*tp_getattro*/
        0,                         /*tp_setattro*/
        0,                         /*tp_as_buffer*/
        Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
        "Noddy objects",           /* tp_doc */
        (traverseproc)Noddy_traverse,   /* tp_traverse */
        (inquiry)Noddy_clear,           /* tp_clear */
        0,		               /* tp_richcompare */
        0,		               /* tp_weaklistoffset */
        0,		               /* tp_iter */
        0,		               /* tp_iternext */
        Noddy_methods,             /* tp_methods */
        Noddy_members,             /* tp_members */
        0,                         /* tp_getset */
        0,                         /* tp_base */
        0,                         /* tp_dict */
        0,                         /* tp_descr_get */
        0,                         /* tp_descr_set */
        0,                         /* tp_dictoffset */
        (initproc)Noddy_init,      /* tp_init */
        0,                         /* tp_alloc */
        Noddy_new,                 /* tp_new */
    };

    static PyMethodDef module_methods[] = {
        {NULL}  /* Sentinel */
    };

    #ifndef PyMODINIT_FUNC	/* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif
    PyMODINIT_FUNC
    initnoddy4(void)
    {
        PyObject* m;

        if (PyType_Ready(&NoddyType) < 0)
            return;

        m = Py_InitModule3("noddy4", module_methods,
                           "Example module that creates an extension type.");

        if (m == NULL)
          return;

        Py_INCREF(&NoddyType);
        PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
    }

The traversal method provides access to subobjects that could
participate in cycles:

    static int
    Noddy_traverse(Noddy *self, visitproc visit, void *arg)
    {
        int vret;

        if (self->first) {
            vret = visit(self->first, arg);
            if (vret != 0)
                return vret;
        }
        if (self->last) {
            vret = visit(self->last, arg);
            if (vret != 0)
                return vret;
        }

        return 0;
    }

For each subobject that can participate in cycles, we need to call the
`visit()' function, which is passed to the traversal method. The
`visit()' function takes as arguments the subobject and the extra
argument _arg_ passed to the traversal method.  It returns an integer
value that must be returned if it is non-zero.

  Python 2.4 and higher provide a Py_VISIT() macro that
automates calling visit functions.  With Py_VISIT(),
`Noddy_traverse()' can be simplified:

    static int
    Noddy_traverse(Noddy *self, visitproc visit, void *arg)
    {
        Py_VISIT(self->first);
        Py_VISIT(self->last);
        return 0;
    }

     Note: Note that the `tp_traverse' implementation must name its
     arguments exactly _visit_ and _arg_ in order to use
     Py_VISIT().  This is to encourage uniformity across these
     boring implementations.

  We also need to provide a method for clearing any subobjects that can
participate in cycles.  We implement the method and reimplement the
deallocator to use it:

    static int
    Noddy_clear(Noddy *self)
    {
        PyObject *tmp;

        tmp = self->first;
        self->first = NULL;
        Py_XDECREF(tmp);

        tmp = self->last;
        self->last = NULL;
        Py_XDECREF(tmp);

        return 0;
    }

    static void
    Noddy_dealloc(Noddy* self)
    {
        Noddy_clear(self);
        self->ob_type->tp_free((PyObject*)self);
    }

Notice the use of a temporary variable in `Noddy_clear()'. We use the
temporary variable so that we can set each member to _NULL_ before
decrementing its reference count.  We do this because, as was discussed
earlier, if the reference count drops to zero, we might cause code to
run that calls back into the object.  In addition, because we now
support garbage collection, we also have to worry about code being run
that triggers garbage collection.  If garbage collection is run, our
`tp_traverse' handler could get called. We can't take a chance of
having `Noddy_traverse()' called when a member's reference count has
dropped to zero and its value hasn't been set to _NULL_.

  Python 2.4 and higher provide a Py_CLEAR() that
automates the careful decrementing of reference counts.  With
Py_CLEAR(), the `Noddy_clear()' function can be simplified:

    static int
    Noddy_clear(Noddy *self)
    {
        Py_CLEAR(self->first);
        Py_CLEAR(self->last);
        return 0;
    }

Finally, we add the Py_TPFLAGS_HAVE_GC flag to the class
flags:

    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /*tp_flags*/

That's pretty much it.  If we had written custom `tp_alloc' or
`tp_free' slots, we'd need to modify them for cyclic-garbage collection.
Most extensions will use the versions automatically provided.

6.2.1.4 Subclassing other types
...............................

It is possible to create new extension types that are derived from
existing types. It is easiest to inherit from the built in types, since
an extension can easily use the `PyTypeObject' it needs. It can be
difficult to share these `PyTypeObject' structures between extension
modules.

  In this example we will create a `Shoddy' type that inherits from the
built-in list type. The new type will be completely
compatible with regular lists, but will have an additional
`increment()' method that increases an internal counter.

    >>> import shoddy
    >>> s = shoddy.Shoddy(range(3))
    >>> s.extend(s)
    >>> print len(s)
    6
    >>> print s.increment()
    1
    >>> print s.increment()
    2

    #include <Python.h>

    typedef struct {
        PyListObject list;
        int state;
    } Shoddy;

    static PyObject *
    Shoddy_increment(Shoddy *self, PyObject *unused)
    {
        self->state++;
        return PyInt_FromLong(self->state);
    }

    static PyMethodDef Shoddy_methods[] = {
        {"increment", (PyCFunction)Shoddy_increment, METH_NOARGS,
         PyDoc_STR("increment state counter")},
        {NULL,	NULL},
    };

    static int
    Shoddy_init(Shoddy *self, PyObject *args, PyObject *kwds)
    {
        if (PyList_Type.tp_init((PyObject *)self, args, kwds) < 0)
            return -1;
        self->state = 0;
        return 0;
    }

    static PyTypeObject ShoddyType = {
        PyObject_HEAD_INIT(NULL)
        0,                       /* ob_size */
        "shoddy.Shoddy",         /* tp_name */
        sizeof(Shoddy),          /* tp_basicsize */
        0,                       /* tp_itemsize */
        0,                       /* tp_dealloc */
        0,                       /* tp_print */
        0,                       /* tp_getattr */
        0,                       /* tp_setattr */
        0,                       /* tp_compare */
        0,                       /* tp_repr */
        0,                       /* tp_as_number */
        0,                       /* tp_as_sequence */
        0,                       /* tp_as_mapping */
        0,                       /* tp_hash */
        0,                       /* tp_call */
        0,                       /* tp_str */
        0,                       /* tp_getattro */
        0,                       /* tp_setattro */
        0,                       /* tp_as_buffer */
        Py_TPFLAGS_DEFAULT |
          Py_TPFLAGS_BASETYPE,   /* tp_flags */
        0,                       /* tp_doc */
        0,                       /* tp_traverse */
        0,                       /* tp_clear */
        0,                       /* tp_richcompare */
        0,                       /* tp_weaklistoffset */
        0,                       /* tp_iter */
        0,                       /* tp_iternext */
        Shoddy_methods,          /* tp_methods */
        0,                       /* tp_members */
        0,                       /* tp_getset */
        0,                       /* tp_base */
        0,                       /* tp_dict */
        0,                       /* tp_descr_get */
        0,                       /* tp_descr_set */
        0,                       /* tp_dictoffset */
        (initproc)Shoddy_init,   /* tp_init */
        0,                       /* tp_alloc */
        0,                       /* tp_new */
    };

    PyMODINIT_FUNC
    initshoddy(void)
    {
        PyObject *m;

        ShoddyType.tp_base = &PyList_Type;
        if (PyType_Ready(&ShoddyType) < 0)
            return;

        m = Py_InitModule3("shoddy", NULL, "Shoddy module");
        if (m == NULL)
            return;

        Py_INCREF(&ShoddyType);
        PyModule_AddObject(m, "Shoddy", (PyObject *) &ShoddyType);
    }

As you can see, the source code closely resembles the `Noddy' examples
in previous sections. We will break down the main differences between
them.

    typedef struct {
        PyListObject list;
        int state;
    } Shoddy;

The primary difference for derived type objects is that the base type's
object structure must be the first value. The base type will already
include the PyObject_HEAD() at the beginning of its
structure.

  When a Python object is a `Shoddy' instance, its _PyObject*_ pointer
can be safely cast to both _PyListObject*_ and _Shoddy*_.

    static int
    Shoddy_init(Shoddy *self, PyObject *args, PyObject *kwds)
    {
        if (PyList_Type.tp_init((PyObject *)self, args, kwds) < 0)
           return -1;
        self->state = 0;
        return 0;
    }

In the `__init__' method for our type, we can see how to call through to
the `__init__' method of the base type.

  This pattern is important when writing a type with custom new
and `dealloc' methods. The new method should not
actually create the memory for the object with `tp_alloc', that will be
handled by the base class when calling its `tp_new'.

  When filling out the PyTypeObject() for the `Shoddy'
type, you see a slot for `tp_base()'. Due to cross platform compiler
issues, you can't fill that field directly with the
PyList_Type().; it can be done later in the module's `init()'
function.

    PyMODINIT_FUNC
    initshoddy(void)
    {
        PyObject *m;

        ShoddyType.tp_base = &PyList_Type;
        if (PyType_Ready(&ShoddyType) < 0)
            return;

        m = Py_InitModule3("shoddy", NULL, "Shoddy module");
        if (m == NULL)
            return;

        Py_INCREF(&ShoddyType);
        PyModule_AddObject(m, "Shoddy", (PyObject *) &ShoddyType);
    }

Before calling PyType_Ready(), the type structure must have
the `tp_base' slot filled in. When we are deriving a new type, it is not
necessary to fill out the `tp_alloc' slot with
PyType_GenericNew().  - the allocate function from the base type
will be inherited.

  After that, calling PyType_Ready() and adding the type
object to the module is the same as with the basic `Noddy' examples.

6.2.2 Type Methods
------------------

This section aims to give a quick fly-by on the various type methods
you can implement and what they do.

  Here is the definition of PyTypeObject, with some fields
only used in debug builds omitted:

    typedef struct _typeobject {
        PyObject_VAR_HEAD
        char *tp_name; /* For printing, in format "<module>.<name>" */
        int tp_basicsize, tp_itemsize; /* For allocation */

        /* Methods to implement standard operations */

        destructor tp_dealloc;
        printfunc tp_print;
        getattrfunc tp_getattr;
        setattrfunc tp_setattr;
        cmpfunc tp_compare;
        reprfunc tp_repr;

        /* Method suites for standard classes */

        PyNumberMethods *tp_as_number;
        PySequenceMethods *tp_as_sequence;
        PyMappingMethods *tp_as_mapping;

        /* More standard operations (here for binary compatibility) */

        hashfunc tp_hash;
        ternaryfunc tp_call;
        reprfunc tp_str;
        getattrofunc tp_getattro;
        setattrofunc tp_setattro;

        /* Functions to access object as input/output buffer */
        PyBufferProcs *tp_as_buffer;

        /* Flags to define presence of optional/expanded features */
        long tp_flags;

        char *tp_doc; /* Documentation string */

        /* Assigned meaning in release 2.0 */
        /* call function for all accessible objects */
        traverseproc tp_traverse;

        /* delete references to contained objects */
        inquiry tp_clear;

        /* Assigned meaning in release 2.1 */
        /* rich comparisons */
        richcmpfunc tp_richcompare;

        /* weak reference enabler */
        long tp_weaklistoffset;

        /* Added in release 2.2 */
        /* Iterators */
        getiterfunc tp_iter;
        iternextfunc tp_iternext;

        /* Attribute descriptor and subclassing stuff */
        struct PyMethodDef *tp_methods;
        struct PyMemberDef *tp_members;
        struct PyGetSetDef *tp_getset;
        struct _typeobject *tp_base;
        PyObject *tp_dict;
        descrgetfunc tp_descr_get;
        descrsetfunc tp_descr_set;
        long tp_dictoffset;
        initproc tp_init;
        allocfunc tp_alloc;
        newfunc tp_new;
        freefunc tp_free; /* Low-level free-memory routine */
        inquiry tp_is_gc; /* For PyObject_IS_GC */
        PyObject *tp_bases;
        PyObject *tp_mro; /* method resolution order */
        PyObject *tp_cache;
        PyObject *tp_subclasses;
        PyObject *tp_weaklist;

    } PyTypeObject;

Now that's a _lot_ of methods.  Don't worry too much though - if you
have a type you want to define, the chances are very good that you will
only implement a handful of these.

  As you probably expect by now, we're going to go over this and give
more information about the various handlers.  We won't go in the order
they are defined in the structure, because there is a lot of historical
baggage that impacts the ordering of the fields; be sure your type
initialization keeps the fields in the right order!  It's often easiest
to find an example that includes all the fields you need (even if
they're initialized to `0') and then change the values to suit your new
type.

    char *tp_name; /* For printing */

The name of the type - as mentioned in the last section, this will
appear in various places, almost entirely for diagnostic purposes. Try
to choose something that will be helpful in such a situation!

    int tp_basicsize, tp_itemsize; /* For allocation */

These fields tell the runtime how much memory to allocate when new
objects of this type are created.  Python has some built-in support for
variable length structures (think: strings, lists) which is where the
`tp_itemsize' field comes in.  This will be dealt with later.

    char *tp_doc;

Here you can put a string (or its address) that you want returned when
the Python script references `obj.__doc__' to retrieve the doc string.

  Now we come to the basic type methods--the ones most extension types
will implement.

6.2.2.1 Finalization and De-allocation
......................................

    destructor tp_dealloc;

This function is called when the reference count of the instance of
your type is reduced to zero and the Python interpreter wants to
reclaim it.  If your type has memory to free or other clean-up to
perform, put it here.  The object itself needs to be freed here as
well.  Here is an example of this function:

    static void
    newdatatype_dealloc(newdatatypeobject * obj)
    {
        free(obj->obj_UnderlyingDatatypePtr);
        obj->ob_type->tp_free(obj);
    }

One important requirement of the deallocator function is that it leaves
any pending exceptions alone.  This is important since deallocators are
frequently called as the interpreter unwinds the Python stack; when the
stack is unwound due to an exception (rather than normal returns),
nothing is done to protect the deallocators from seeing that an
exception has already been set.  Any actions which a deallocator
performs which may cause additional Python code to be executed may
detect that an exception has been set.  This can lead to misleading
errors from the interpreter.  The proper way to protect against this is
to save a pending exception before performing the unsafe action, and
restoring it when done.  This can be done using the
PyErr_Fetch() and PyErr_Restore() functions:

    static void
    my_dealloc(PyObject *obj)
    {
        MyObject *self = (MyObject *) obj;
        PyObject *cbresult;

        if (self->my_callback != NULL) {
            PyObject *err_type, *err_value, *err_traceback;
            int have_error = PyErr_Occurred() ? 1 : 0;

            if (have_error)
                PyErr_Fetch(&err_type, &err_value, &err_traceback);

            cbresult = PyObject_CallObject(self->my_callback, NULL);
            if (cbresult == NULL)
                PyErr_WriteUnraisable(self->my_callback);
            else
                Py_DECREF(cbresult);

            if (have_error)
                PyErr_Restore(err_type, err_value, err_traceback);

            Py_DECREF(self->my_callback);
        }
        obj->ob_type->tp_free((PyObject*)self);
    }

6.2.2.2 Object Presentation
...........................

In Python, there are three ways to generate a textual representation of
an object: the repr() function (or equivalent back-tick
syntax), the str() function, and the print.
statement.  For most objects, the print statement is
equivalent to the str() function, but it is possible to
special-case printing to a `FILE*' if necessary; this should only be
done if efficiency is identified as a problem and profiling suggests
that creating a temporary string object to be written to a file is too
expensive.

  These handlers are all optional, and most types at most need to
implement the `tp_str' and `tp_repr' handlers.

    reprfunc tp_repr;
    reprfunc tp_str;
    printfunc tp_print;

The `tp_repr' handler should return a string object containing a
representation of the instance for which it is called.  Here is a simple
example:

    static PyObject *
    newdatatype_repr(newdatatypeobject * obj)
    {
        return PyString_FromFormat("Repr-ified_newdatatype{{size:\%d}}",
                                   obj->obj_UnderlyingDatatypePtr->size);
    }

If no `tp_repr' handler is specified, the interpreter will supply a
representation that uses the type's `tp_name' and a uniquely-identifying
value for the object.

  The `tp_str' handler is to str() what the `tp_repr'
handler described above is to repr().; that is, it is called
when Python code calls str() on an instance of your object.
Its implementation is very similar to the `tp_repr' function, but the
resulting string is intended for human consumption.  If `tp_str' is not
specified, the `tp_repr' handler is used instead.

  Here is a simple example:

    static PyObject *
    newdatatype_str(newdatatypeobject * obj)
    {
        return PyString_FromFormat("Stringified_newdatatype{{size:\%d}}",
                                   obj->obj_UnderlyingDatatypePtr->size);
    }

The print function will be called whenever Python needs to "print" an
instance of the type.  For example, if 'node' is an instance of type
TreeNode, then the print function is called when Python code calls:

    print node

There is a flags argument and one flag, `Py_PRINT_RAW', and it suggests
that you print without string quotes and possibly without interpreting
escape sequences.

  The print function receives a file object as an argument. You will
likely want to write to that file object.

  Here is a sample print function:

    static int
    newdatatype_print(newdatatypeobject *obj, FILE *fp, int flags)
    {
        if (flags & Py_PRINT_RAW) {
            fprintf(fp, "<{newdatatype object--size: %d}>",
                    obj->obj_UnderlyingDatatypePtr->size);
        }
        else {
            fprintf(fp, "\"<{newdatatype object--size: %d}>\"",
                    obj->obj_UnderlyingDatatypePtr->size);
        }
        return 0;
    }

6.2.2.3 Attribute Management
............................

For every object which can support attributes, the corresponding type
must provide the functions that control how the attributes are
resolved.  There needs to be a function which can retrieve attributes
(if any are defined), and another to set attributes (if setting
attributes is allowed).  Removing an attribute is a special case, for
which the new value passed to the handler is _NULL_.

  Python supports two pairs of attribute handlers; a type that supports
attributes only needs to implement the functions for one pair.  The
difference is that one pair takes the name of the attribute as a
`char*', while the other accepts a PyObject*.  Each type can
use whichever pair makes more sense for the implementation's
convenience.

    getattrfunc  tp_getattr;        /* char * version */
    setattrfunc  tp_setattr;
    /* ... */
    getattrofunc tp_getattrofunc;   /* PyObject * version */
    setattrofunc tp_setattrofunc;

If accessing attributes of an object is always a simple operation (this
will be explained shortly), there are generic implementations which can
be used to provide the PyObject* version of the attribute
management functions.  The actual need for type-specific attribute
handlers almost completely disappeared starting with Python 2.2, though
there are many examples which have not been updated to use some of the
new generic mechanism that is available.

6.2.2.4 Generic Attribute Management
....................................

New in version 2.2.

  Most extension types only use _simple_ attributes.  So, what makes the
attributes simple?  There are only a couple of conditions that must be
met:

  1. The name of the attributes must be known when
     PyType_Ready() is called.

  2. No special processing is needed to record that an attribute was
     looked up or set, nor do actions need to be taken based on the
     value.

  Note that this list does not place any restrictions on the values of
the attributes, when the values are computed, or how relevant data is
stored.

  When PyType_Ready() is called, it uses three tables
referenced by the type object to create descriptor.s which
are placed in the dictionary of the type object.  Each descriptor
controls access to one attribute of the instance object.  Each of the
tables is optional; if all three are _NULL_, instances of the type will
only have attributes that are inherited from their base type, and
should leave the `tp_getattro' and `tp_setattro' fields _NULL_ as well,
allowing the base type to handle attributes.

  The tables are declared as three fields of the type object:

    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;

If `tp_methods' is not _NULL_, it must refer to an array of
PyMethodDef structures.  Each entry in the table is an instance
of this structure:

    typedef struct PyMethodDef {
        char        *ml_name;       /* method name */
        PyCFunction  ml_meth;       /* implementation function */
        int          ml_flags;      /* flags */
        char        *ml_doc;        /* docstring */
    } PyMethodDef;

One entry should be defined for each method provided by the type; no
entries are needed for methods inherited from a base type.  One
additional entry is needed at the end; it is a sentinel that marks the
end of the array.  The `ml_name' field of the sentinel must be _NULL_.

  XXX Need to refer to some unified discussion of the structure fields,
shared with the next section.

  The second table is used to define attributes which map directly to
data stored in the instance.  A variety of primitive C types are
supported, and access may be read-only or read-write.  The structures
in the table are defined as:

    typedef struct PyMemberDef {
        char *name;
        int   type;
        int   offset;
        int   flags;
        char *doc;
    } PyMemberDef;

For each entry in the table, a descriptor will be
constructed and added to the type which will be able to extract a value
from the instance structure.  The type field should contain
one of the type codes defined in the `structmember.h' header; the value
will be used to determine how to convert Python values to and from C
values.  The `flags' field is used to store flags which control how the
attribute can be accessed.

  XXX Need to move some of this to a shared section!

  The following flag constants are defined in `structmember.h'; they
may be combined using bitwise-OR.

Constant                        Meaning
-----------------------------------------------------------------------------------
`READONLY'                      Never writable.
`RO'                            Shorthand for `READONLY'.
`READ_RESTRICTED'               Not readable in restricted mode.
`WRITE_RESTRICTED'              Not writable in restricted mode.
`RESTRICTED'                    Not readable or writable in restricted mode.

  An interesting advantage of using the `tp_members' table to build
descriptors that are used at runtime is that any attribute defined this
way can have an associated doc string simply by providing the text in
the table.  An application can use the introspection API to retrieve
the descriptor from the class object, and get the doc string using its
`__doc__' attribute.

  As with the `tp_methods' table, a sentinel entry with a `name' value
of _NULL_ is required.

6.2.2.5 Type-specific Attribute Management
..........................................

For simplicity, only the `char*' version will be demonstrated here; the
type of the name parameter is the only difference between the `char*'
and PyObject* flavors of the interface. This example
effectively does the same thing as the generic example above, but does
not use the generic support added in Python 2.2.  The value in showing
this is two-fold: it demonstrates how basic attribute management can be
done in a way that is portable to older versions of Python, and
explains how the handler functions are called, so that if you do need
to extend their functionality, you'll understand what needs to be done.

  The `tp_getattr' handler is called when the object requires an
attribute look-up.  It is called in the same situations where the
__getattr__().  method of a class would be called.

  A likely way to handle this is (1) to implement a set of functions
(such as `newdatatype_getSize()' and `newdatatype_setSize()' in the
example below), (2) provide a method table listing these functions, and
(3) provide a getattr function that returns the result of a lookup in
that table.  The method table uses the same structure as the
`tp_methods' field of the type object.

  Here is an example:

    static PyMethodDef newdatatype_methods[] = {
        {"getSize", (PyCFunction)newdatatype_getSize, METH_VARARGS,
         "Return the current size."},
        {"setSize", (PyCFunction)newdatatype_setSize, METH_VARARGS,
         "Set the size."},
        {NULL, NULL, 0, NULL}           /* sentinel */
    };

    static PyObject *
    newdatatype_getattr(newdatatypeobject *obj, char *name)
    {
        return Py_FindMethod(newdatatype_methods, (PyObject *)obj, name);
    }

The `tp_setattr' handler is called when the __setattr__() or
__delattr__() method of a class instance would be called.
When an attribute should be deleted, the third parameter will be
_NULL_.  Here is an example that simply raises an exception; if this
were really all you wanted, the `tp_setattr' handler should be set to
_NULL_.

    static int
    newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
    {
        (void)PyErr_Format(PyExc_RuntimeError, "Read-only attribute: \%s", name);
        return -1;
    }

6.2.2.6 Object Comparison
.........................

    cmpfunc tp_compare;

The `tp_compare' handler is called when comparisons are needed and the
object does not implement the specific rich comparison method which
matches the requested comparison.  (It is always used if defined and the
PyObject_Compare() or PyObject_Cmp().
functions are used, or if cmp() is used from Python.) It is
analogous to the __cmp__() method.  This function should
return `-1' if _obj1_ is less than _obj2_, `0' if they are equal, and
`1' if _obj1_ is greater than _obj2_. (It was previously allowed to
return arbitrary negative or positive integers for less than and
greater than, respectively; as of Python 2.2, this is no longer
allowed.  In the future, other return values may be assigned a
different meaning.)

  A `tp_compare' handler may raise an exception.  In this case it should
return a negative value.  The caller has to test for the exception using
PyErr_Occurred().

  Here is a sample implementation:

    static int
    newdatatype_compare(newdatatypeobject * obj1, newdatatypeobject * obj2)
    {
        long result;

        if (obj1->obj_UnderlyingDatatypePtr->size <
            obj2->obj_UnderlyingDatatypePtr->size) {
            result = -1;
        }
        else if (obj1->obj_UnderlyingDatatypePtr->size >
                 obj2->obj_UnderlyingDatatypePtr->size) {
            result = 1;
        }
        else {
            result = 0;
        }
        return result;
    }

6.2.2.7 Abstract Protocol Support
.................................

Python supports a variety of _abstract_ 'protocols;' the specific
interfaces provided to use these interfaces are documented in
Abstract Objects Layer.

  A number of these abstract interfaces were defined early in the
development of the Python implementation.  In particular, the number,
mapping, and sequence protocols have been part of Python since the
beginning.  Other protocols have been added over time.  For protocols
which depend on several handler routines from the type implementation,
the older protocols have been defined as optional blocks of handlers
referenced by the type object.  For newer protocols there are
additional slots in the main type object, with a flag bit being set to
indicate that the slots are present and should be checked by the
interpreter.  (The flag bit does not indicate that the slot values are
non-_NULL_. The flag may be set to indicate the presence of a slot, but
a slot may still be unfilled.)

    PyNumberMethods   tp_as_number;
    PySequenceMethods tp_as_sequence;
    PyMappingMethods  tp_as_mapping;

If you wish your object to be able to act like a number, a sequence, or
a mapping object, then you place the address of a structure that
implements the C type PyNumberMethods,
PySequenceMethods, or PyMappingMethods, respectively.
It is up to you to fill in this structure with appropriate values. You
can find examples of the use of each of these in the `Objects'
directory of the Python source distribution.

    hashfunc tp_hash;

This function, if you choose to provide it, should return a hash number
for an instance of your data type. Here is a moderately pointless
example:

    static long
    newdatatype_hash(newdatatypeobject *obj)
    {
        long result;
        result = obj->obj_UnderlyingDatatypePtr->size;
        result = result * 3;
        return result;
    }

    ternaryfunc tp_call;

This function is called when an instance of your data type is "called",
for example, if `obj1' is an instance of your data type and the Python
script contains `obj1('hello')', the `tp_call' handler is invoked.

  This function takes three arguments:

  1. _arg1_ is the instance of the data type which is the subject of
     the call. If the call is `obj1('hello')', then _arg1_ is `obj1'.

  2. _arg2_ is a tuple containing the arguments to the call.  You can
     use PyArg_ParseTuple() to extract the arguments.

  3. _arg3_ is a dictionary of keyword arguments that were passed. If
     this is non-_NULL_ and you support keyword arguments, use
     PyArg_ParseTupleAndKeywords() to extract the arguments.  If
     you do not want to support keyword arguments and this is
     non-_NULL_, raise a TypeError with a message saying
     that keyword arguments are not supported.

  Here is a desultory example of the implementation of the call
function.

    /* Implement the call function.
     *    obj1 is the instance receiving the call.
     *    obj2 is a tuple containing the arguments to the call, in this
     *         case 3 strings.
     */
    static PyObject *
    newdatatype_call(newdatatypeobject *obj, PyObject *args, PyObject *other)
    {
        PyObject *result;
        char *arg1;
        char *arg2;
        char *arg3;

        if (!PyArg_ParseTuple(args, "sss:call", &arg1, &arg2, &arg3)) {
            return NULL;
        }
        result = PyString_FromFormat(
            "Returning -- value: [\%d] arg1: [\%s] arg2: [\%s] arg3: [\%s]\n",
            obj->obj_UnderlyingDatatypePtr->size,
            arg1, arg2, arg3);
        printf("\%s", PyString_AS_STRING(result));
        return result;
    }

XXX some fields need to be added here...

    /* Added in release 2.2 */
    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

These functions provide support for the iterator protocol.  Any object
which wishes to support iteration over its contents (which may be
generated during iteration) must implement the `tp_iter' handler.
Objects which are returned by a `tp_iter' handler must implement both
the `tp_iter' and `tp_iternext' handlers. Both handlers take exactly
one parameter, the instance for which they are being called, and return
a new reference.  In the case of an error, they should set an exception
and return _NULL_.

  For an object which represents an iterable collection, the `tp_iter'
handler must return an iterator object.  The iterator object is
responsible for maintaining the state of the iteration.  For
collections which can support multiple iterators which do not interfere
with each other (as lists and tuples do), a new iterator should be
created and returned.  Objects which can only be iterated over once
(usually due to side effects of iteration) should implement this
handler by returning a new reference to themselves, and should also
implement the `tp_iternext' handler.  File objects are an example of
such an iterator.

  Iterator objects should implement both handlers.  The `tp_iter'
handler should return a new reference to the iterator (this is the same
as the `tp_iter' handler for objects which can only be iterated over
destructively).  The `tp_iternext' handler should return a new
reference to the next object in the iteration if there is one.  If the
iteration has reached the end, it may return _NULL_ without setting an
exception or it may set StopIteration.; avoiding the
exception can yield slightly better performance.  If an actual error
occurs, it should set an exception and return _NULL_.

6.2.2.8 Weak Reference Support
..............................

One of the goals of Python's weak-reference implementation is to allow
any type to participate in the weak reference mechanism without
incurring the overhead on those objects which do not benefit by weak
referencing (such as numbers).

  For an object to be weakly referencable, the extension must include a
PyObject* field in the instance structure for the use of
the weak reference mechanism; it must be initialized to _NULL_ by the
object's constructor.  It must also set the `tp_weaklistoffset' field
of the corresponding type object to the offset of the field. For
example, the instance type is defined with the following structure:

    typedef struct {
        PyObject_HEAD
        PyClassObject *in_class;       /* The class object */
        PyObject      *in_dict;        /* A dictionary */
        PyObject      *in_weakreflist; /* List of weak references */
    } PyInstanceObject;

The statically-declared type object for instances is defined this way:

    PyTypeObject PyInstance_Type = {
        PyObject_HEAD_INIT(&PyType_Type)
        0,
        "module.instance",

        /* Lots of stuff omitted for brevity... */

        Py_TPFLAGS_DEFAULT,                         /* tp_flags */
        0,                                          /* tp_doc */
        0,                                          /* tp_traverse */
        0,                                          /* tp_clear */
        0,                                          /* tp_richcompare */
        offsetof(PyInstanceObject, in_weakreflist), /* tp_weaklistoffset */
    };

The type constructor is responsible for initializing the weak reference
list to _NULL_:

    static PyObject *
    instance_new() {
        /* Other initialization stuff omitted for brevity */

        self->in_weakreflist = NULL;

        return (PyObject *) self;
    }

The only further addition is that the destructor needs to call the weak
reference manager to clear any weak references.  This should be done
before any other parts of the destruction have occurred, but is only
required if the weak reference list is non-_NULL_:

    static void
    instance_dealloc(PyInstanceObject *inst)
    {
        /* Allocate temporaries if needed, but do not begin
           destruction just yet.
         */

        if (inst->in_weakreflist != NULL)
            PyObject_ClearWeakRefs((PyObject *) inst);

        /* Proceed with object destruction normally. */
    }

6.2.2.9 More Suggestions
........................

Remember that you can omit most of these functions, in which case you
provide `0' as a value.  There are type definitions for each of the
functions you must provide.  They are in `object.h' in the Python
include directory that comes with the source distribution of Python.

  In order to learn how to implement any specific method for your new
data type, do the following: Download and unpack the Python source
distribution.  Go the `Objects' directory, then search the C source
files for `tp_' plus the function you want (for example, `tp_print' or
`tp_compare').  You will find examples of the function you want to
implement.

  When you need to verify that an object is an instance of the type you
are implementing, use the PyObject_TypeCheck() function. A
sample of its use might be something like the following:

    if (! PyObject_TypeCheck(some_object, &MyType)) {
        PyErr_SetString(PyExc_TypeError, "arg #1 not a mything");
        return NULL;
    }

Footnotes
.........

     (1) This is true when we know that the object is a basic type,
     like a string or a float.

     (2) We relied on this in the `tp_dealloc' handler in this example,
     because our type doesn't support garbage collection. Even if a
     type supports garbage collection, there are calls that can be made
     to "untrack" the object from garbage collection, however, these
     calls are advanced and not covered here.

     (3) We now know that the first and last members are strings, so
     perhaps we could be less careful about decrementing their
     reference counts, however, we accept instances of string
     subclasses. Even though deallocating normal strings won't call
     back into our objects, we can't guarantee that deallocating an
     instance of a string subclass won't call back into our objects.

     (4) Even in the third version, we aren't guaranteed to avoid
     cycles.  Instances of string subclasses are allowed and string
     subclasses could allow cycles even if normal strings don't.

6.3 Building C and C++ Extensions with distutils
================================================

Starting in Python 1.4, Python provides, on Unix, a special make file
for building make files for building dynamically-linked extensions and
custom interpreters.  Starting with Python 2.0, this mechanism (known
as related to Makefile.pre.in, and Setup files) is no longer supported.
Building custom interpreters was rarely used, and extension modules can
be built using distutils.

  Building an extension module using distutils requires that distutils
is installed on the build machine, which is included in Python 2.x and
available separately for Python 1.5. Since distutils also supports
creation of binary packages, users don't necessarily need a compiler
and distutils to install the extension.

  A distutils package contains a driver script, `setup.py'. This is a
plain Python file, which, in the most simple case, could look like this:

    from distutils.core import setup, Extension

    module1 = Extension('demo',
                        sources = ['demo.c'])

    setup (name = 'PackageName',
           version = '1.0',
           description = 'This is a demo package',
           ext_modules = [module1])

With this `setup.py', and a file `demo.c', running

    python setup.py build

will compile `demo.c', and produce an extension module named `demo' in
the `build' directory. Depending on the system, the module file will end
up in a subdirectory `build/lib.system', and may have a name like
`demo.so' or `demo.pyd'.

  In the `setup.py', all execution is performed by calling the `setup'
function. This takes a variable number of keyword arguments, of which
the example above uses only a subset. Specifically, the example
specifies meta-information to build packages, and it specifies the
contents of the package.  Normally, a package will contain of addition
modules, like Python source modules, documentation, subpackages, etc.
Please refer to the distutils documentation in Distributing Python Modules
to learn more about the features of distutils;
this section explains building extension modules only.

  It is common to pre-compute arguments to `setup()', to better
structure the driver script. In the example above, the`ext_modules'
argument to `setup()' is a list of extension modules, each of which is
an instance of the `Extension'. In the example, the instance defines an
extension named `demo' which is build by compiling a single source
file, `demo.c'.

  In many cases, building an extension is more complex, since additional
preprocessor defines and libraries may be needed. This is demonstrated
in the example below.

    from distutils.core import setup, Extension

    module1 = Extension('demo',
                        define_macros = [('MAJOR_VERSION', '1'),
                                         ('MINOR_VERSION', '0')],
                        include_dirs = ['/usr/local/include'],
                        libraries = ['tcl83'],
                        library_dirs = ['/usr/local/lib'],
                        sources = ['demo.c'])

    setup (name = 'PackageName',
           version = '1.0',
           description = 'This is a demo package',
           author = 'Martin v. Loewis',
           author_email = 'martin@v.loewis.de',
           url = 'http://docs.python.org/extending/building',
           long_description = '''
    This is really just a demo package.
    ''',
           ext_modules = [module1])

In this example, `setup()' is called with additional meta-information,
which is recommended when distribution packages have to be built. For
the extension itself, it specifies preprocessor defines, include
directories, library directories, and libraries. Depending on the
compiler, distutils passes this information in different ways to the
compiler. For example, on Unix, this may result in the compilation
commands

    gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -DMAJOR_VERSION=1 -DMINOR_VERSION=0 -I/usr/local/include -I/usr/local/include/python2.2 -c demo.c -o build/temp.linux-i686-2.2/demo.o

    gcc -shared build/temp.linux-i686-2.2/demo.o -L/usr/local/lib -ltcl83 -o build/lib.linux-i686-2.2/demo.so

These lines are for demonstration purposes only; distutils users should
trust that distutils gets the invocations right.

6.3.1 Distributing your extension modules
-----------------------------------------

When an extension has been successfully build, there are three ways to
use it.

  End-users will typically want to install the module, they do so by
running

    python setup.py install

Module maintainers should produce source packages; to do so, they run

    python setup.py sdist

In some cases, additional files need to be included in a source
distribution; this is done through a `MANIFEST.in' file; see the
distutils documentation for details.

  If the source distribution has been build successfully, maintainers
can also create binary distributions. Depending on the platform, one of
the following commands can be used to do so.

    python setup.py bdist_wininst
    python setup.py bdist_rpm
    python setup.py bdist_dumb

6.4 Building C and C++ Extensions on Windows
============================================

This chapter briefly explains how to create a Windows extension module
for Python using Microsoft Visual C++, and follows with more detailed
background information on how it works.  The explanatory material is
useful for both the Windows programmer learning to build Python
extensions and the Unix programmer interested in producing software
which can be successfully built on both Unix and Windows.

  Module authors are encouraged to use the distutils approach for
building extension modules, instead of the one described in this
section. You will still need the C compiler that was used to build
Python; typically Microsoft Visual C++.

     Note: This chapter mentions a number of filenames that include an
     encoded Python version number.  These filenames are represented
     with the version number shown as `XY'; in practice, `'X'' will be
     the major version number and `'Y'' will be the minor version
     number of the Python release you're working with.  For example, if
     you are using Python 2.2.1, `XY' will actually be `22'.

6.4.1 A Cookbook Approach
-------------------------

There are two approaches to building extension modules on Windows, just
as there are on Unix: use the distutils package to control
the build process, or do things manually.  The distutils approach works
well for most extensions; documentation on using distutils.
to build and package extension modules is available in
Distributing Python Modules.  This section describes the manual
approach to building Python extensions written in C or C++.

  To build extensions using these instructions, you need to have a copy
of the Python sources of the same version as your installed Python. You
will need Microsoft Visual C++ "Developer Studio"; project files are
supplied for VC++ version 7.1, but you can use older versions of VC++.
Notice that you should use the same version of VC++that was used to
build Python itself. The example files described here are distributed
with the Python sources in the `PC\example_nt\' directory.

  1. *Copy the example files* --  The `example_nt' directory is a
     subdirectory of the `PC' directory, in order to keep all the
     PC-specific files under the same directory in the source
     distribution.  However, the `example_nt' directory can't actually
     be used from this location.  You first need to copy or move it up
     one level, so that `example_nt' is a sibling of the `PC' and
     `Include' directories.  Do all your work from within this new
     location.

  2. *Open the project* --  From VC++, use the _File ‣ Open Solution_
     dialog (not _File ‣ Open_!).  Navigate to and select the file
     `example.sln', in the _copy_ of the `example_nt' directory you
     made above.  Click Open.

  3. *Build the example DLL* --  In order to check that everything is
     set up right, try building:

  4. Select a configuration.  This step is optional.  Choose _Build ‣
     Configuration Manager ‣ Active Solution Configuration_ and select
     either _Release_  or _Debug_.  If you skip this step, VC++ will
     use the Debug configuration by default.

  5. Build the DLL.  Choose _Build ‣ Build Solution_.  This creates all
     intermediate and result files in a subdirectory called either
     `Debug' or `Release', depending on which configuration you selected
     in the preceding step.

  6. *Testing the debug-mode DLL* --  Once the Debug build has
     succeeded, bring up a DOS box, and change to the
     `example_nt\Debug' directory.  You should now be able to repeat
     the following session (`C>' is the DOS prompt, `>>>' is the Python
     prompt; note that build information and various debug output from
     Python may not match this screen dump exactly):

         C>..\..\PCbuild\python_d
         Adding parser accelerators ...
         Done.
         Python 2.2 (#28, Dec 19 2001, 23:26:37) [MSC 32 bit (Intel)] on win32
         Type "copyright", "credits" or "license" for more information.
         >>> import example
         [4897 refs]
         >>> example.foo()
         Hello, world
         [4903 refs]
         >>>

     Congratulations!  You've successfully built your first Python
     extension module.

  7. *Creating your own project* --  Choose a name and create a
     directory for it.  Copy your C sources into it.  Note that the
     module source file name does not necessarily have to match the
     module name, but the name of the initialization function should
     match the module name -- you can only import a module `spam' if
     its initialization function is called `initspam()', and it should
     call Py_InitModule() with the string `"spam"' as its
     first argument (use the minimal `example.c' in this directory as a
     guide).  By convention, it lives in a file called `spam.c' or
     `spammodule.c'.  The output file should be called `spam.pyd' (in
     Release mode) or `spam_d.pyd' (in Debug mode). The extension
     `.pyd' was chosen to avoid confusion with a system library
     `spam.dll' to which your module could be a Python interface.

     Changed in version 2.5: Previously, file names like `spam.dll' (in
     release mode) or `spam_d.dll' (in debug mode) were also recognized.

     Now your options are:

  8.
    Copy `example.sln' and `example.vcproj', rename them to
          `spam.*', and edit them by hand, or

  9. Create a brand new project; instructions are below.

     In either case, copy `example_nt\example.def' to `spam\spam.def',
     and edit the new `spam.def' so its second line contains the string
     '`initspam''.  If you created a new project yourself, add the file
     `spam.def' to the project now.  (This is an annoying little file
     with only two lines.  An alternative approach is to forget about
     the `.def' file, and add the option `/export:initspam' somewhere
     to the Link settings, by manually editing the setting in Project
     Properties dialog).

 10. *Creating a brand new project* --  Use the _File ‣ New ‣ Project_
     dialog to create a new Project Workspace.  Select _Visual C++
     Projects/Win32/ Win32 Project_, enter the name (`spam'), and make
     sure the Location is set to parent of the `spam' directory you
     have created (which should be a direct subdirectory of the Python
     build tree, a sibling of `Include' and `PC').  Select Win32 as the
     platform (in my version, this is the only choice).  Make sure the
     Create new workspace radio button is selected.  Click OK.

     You should now create the file `spam.def' as instructed in the
     previous section. Add the source files to the project, using
     _Project ‣ Add Existing Item_. Set the pattern to `*.*' and select
     both `spam.c' and `spam.def' and click OK.  (Inserting them one by
     one is fine too.)

     Now open the _Project ‣ spam properties_ dialog. You only need to
     change a few settings.  Make sure _All Configurations_ is selected
     from the _Settings for:_ dropdown list.  Select the C/C++ tab.
     Choose the General category in the popup menu at the top.  Type
     the following text in the entry box labeled _Additional Include
     Directories_:

         ..\Include,..\PC

     Then, choose the General category in the Linker tab, and enter

         ..\PCbuild

     in the text box labelled _Additional library Directories_.

     Now you need to add some mode-specific settings:

     Select _Release_ in the _Configuration_ dropdown list.  Choose the
     _Link_ tab, choose the _Input_ category, and append `pythonXY.lib'
     to the list in the _Additional Dependencies_ box.

     Select _Debug_ in the _Configuration_ dropdown list, and append
     `pythonXY_d.lib' to the list in the _Additional Dependencies_ box.
     Then click the C/C++ tab, select _Code Generation_, and select
     _Multi-threaded Debug DLL_ from the _Runtime library_ dropdown
     list.

     Select _Release_ again from the _Configuration_ dropdown list.
     Select _Multi-threaded DLL_ from the _Runtime library_ dropdown
     list.

  If your module creates a new type, you may have trouble with this
line:

    PyObject_HEAD_INIT(&PyType_Type)

Static type object initializers in extension modules may cause compiles
to fail with an error message like "initializer not a constant".  This
shows up when building DLL under MSVC.  Change it to:

    PyObject_HEAD_INIT(NULL)

and add the following to the module initialization function:

    MyObject_Type.ob_type = &PyType_Type;

6.4.2 Differences Between Unix and Windows
------------------------------------------

Unix and Windows use completely different paradigms for run-time
loading of code.  Before you try to build a module that can be
dynamically loaded, be aware of how your system works.

  In Unix, a shared object (`.so') file contains code to be used by the
program, and also the names of functions and data that it expects to
find in the program.  When the file is joined to the program, all
references to those functions and data in the file's code are changed
to point to the actual locations in the program where the functions and
data are placed in memory.  This is basically a link operation.

  In Windows, a dynamic-link library (`.dll') file has no dangling
references.  Instead, an access to functions or data goes through a
lookup table.  So the DLL code does not have to be fixed up at runtime
to refer to the program's memory; instead, the code already uses the
DLL's lookup table, and the lookup table is modified at runtime to
point to the functions and data.

  In Unix, there is only one type of library file (`.a') which contains
code from several object files (`.o').  During the link step to create
a shared object file (`.so'), the linker may find that it doesn't know
where an identifier is defined.  The linker will look for it in the
object files in the libraries; if it finds it, it will include all the
code from that object file.

  In Windows, there are two types of library, a static library and an
import library (both called `.lib').  A static library is like a Unix
`.a' file; it contains code to be included as necessary. An import
library is basically used only to reassure the linker that a certain
identifier is legal, and will be present in the program when the DLL is
loaded.  So the linker uses the information from the import library to
build the lookup table for using identifiers that are not included in
the DLL.  When an application or a DLL is linked, an import library may
be generated, which will need to be used for all future DLLs that
depend on the symbols in the application or DLL.

  Suppose you are building two dynamic-load modules, B and C, which
should share another block of code A.  On Unix, you would _not_ pass
`A.a' to the linker for `B.so' and `C.so'; that would cause it to be
included twice, so that B and C would each have their own copy.  In
Windows, building `A.dll' will also build `A.lib'.  You _do_ pass
`A.lib' to the linker for B and C.  `A.lib' does not contain code; it
just contains information which will be used at runtime to access A's
code.

  In Windows, using an import library is sort of like using `import
spam'; it gives you access to spam's names, but does not create a
separate copy.  On Unix, linking with a library is more like `from spam
import *'; it does create a separate copy.

6.4.3 Using DLLs in Practice
----------------------------

Windows Python is built in Microsoft Visual C++; using other compilers
may or may not work (though Borland seems to).  The rest of this
section is MSVC++ specific.

  When creating DLLs in Windows, you must pass `pythonXY.lib' to the
linker.  To build two DLLs, spam and ni (which uses C functions found
in spam), you could use these commands:

    cl /LD /I/python/include spam.c ../libs/pythonXY.lib
    cl /LD /I/python/include ni.c spam.lib ../libs/pythonXY.lib

The first command created three files: `spam.obj', `spam.dll' and
`spam.lib'.  `Spam.dll' does not contain any Python functions (such as
PyArg_ParseTuple()), but it does know how to find the
Python code thanks to `pythonXY.lib'.

  The second command created `ni.dll' (and `.obj' and `.lib'), which
knows how to find the necessary functions from spam, and also from the
Python executable.

  Not every identifier is exported to the lookup table.  If you want
any other modules (including Python) to be able to see your
identifiers, you have to say `_declspec(dllexport)', as in `void
_declspec(dllexport) initspam(void)' or `PyObject _declspec(dllexport)
*NiGetSpamData(void)'.

  Developer Studio will throw in a lot of import libraries that you do
not really need, adding about 100K to your executable.  To get rid of
them, use the Project Settings dialog, Link tab, to specify _ignore
default libraries_.  Add the correct `msvcrtxx.lib' to the list of
libraries.

6.5 Embedding Python in Another Application
===========================================

The previous chapters discussed how to extend Python, that is, how to
extend the functionality of Python by attaching a library of C
functions to it.  It is also possible to do it the other way around:
enrich your C/C++ application by embedding Python in it.  Embedding
provides your application with the ability to implement some of the
functionality of your application in Python rather than C or C++. This
can be used for many purposes; one example would be to allow users to
tailor the application to their needs by writing some scripts in
Python.  You can also use it yourself if some of the functionality can
be written in Python more easily.

  Embedding Python is similar to extending it, but not quite.  The
difference is that when you extend Python, the main program of the
application is still the Python interpreter, while if you embed Python,
the main program may have nothing to do with Python -- instead, some
parts of the application occasionally call the Python interpreter to
run some Python code.

  So if you are embedding Python, you are providing your own main
program.  One of the things this main program has to do is initialize
the Python interpreter.  At the very least, you have to call the
function Py_Initialize().  There are optional calls to pass
command line arguments to Python.  Then later you can call the
interpreter from any part of the application.

  There are several different ways to call the interpreter: you can
pass a string containing Python statements to
PyRun_SimpleString(), or you can pass a stdio file pointer and a
file name (for identification in error messages only) to
PyRun_SimpleFile().  You can also call the lower-level operations
described in the previous chapters to construct and use Python objects.

  A simple demo of embedding Python can be found in the directory
`Demo/embed/' of the source distribution.

See Python/C API Reference Manual.
     The details of Python's C interface are given in this manual. A
     great deal of necessary information can be found here.

6.5.1 Very High Level Embedding
-------------------------------

The simplest form of embedding Python is the use of the very high level
interface. This interface is intended to execute a Python script
without needing to interact with the application directly. This can for
example be used to perform some operation on a file.

    #include <Python.h>

    int
    main(int argc, char *argv[])
    {
      Py_Initialize();
      PyRun_SimpleString("from time import time,ctime\n"
                         "print 'Today is',ctime(time())\n");
      Py_Finalize();
      return 0;
    }

The above code first initializes the Python interpreter with
Py_Initialize(), followed by the execution of a hard-coded Python
script that print the date and time.  Afterwards, the
Py_Finalize() call shuts the interpreter down, followed by the
end of the program.  In a real program, you may want to get the Python
script from another source, perhaps a text-editor routine, a file, or a
database.  Getting the Python code from a file can better be done by
using the PyRun_SimpleFile() function, which saves you the
trouble of allocating memory space and loading the file contents.

6.5.2 Beyond Very High Level Embedding: An overview
---------------------------------------------------

The high level interface gives you the ability to execute arbitrary
pieces of Python code from your application, but exchanging data values
is quite cumbersome to say the least. If you want that, you should use
lower level calls.  At the cost of having to write more C code, you can
achieve almost anything.

  It should be noted that extending Python and embedding Python is
quite the same activity, despite the different intent. Most topics
discussed in the previous chapters are still valid. To show this,
consider what the extension code from Python to C really does:

  1. Convert data values from Python to C,

  2. Perform a function call to a C routine using the converted values,
     and

  3. Convert the data values from the call from C to Python.

  When embedding Python, the interface code does:

  1. Convert data values from C to Python,

  2. Perform a function call to a Python interface routine using the
     converted values, and

  3. Convert the data values from the call from Python to C.

  As you can see, the data conversion steps are simply swapped to
accommodate the different direction of the cross-language transfer. The
only difference is the routine that you call between both data
conversions. When extending, you call a C routine, when embedding, you
call a Python routine.

  This chapter will not discuss how to convert data from Python to C
and vice versa.  Also, proper use of references and dealing with errors
is assumed to be understood.  Since these aspects do not differ from
extending the interpreter, you can refer to earlier chapters for the
required information.

6.5.3 Pure Embedding
--------------------

The first program aims to execute a function in a Python script. Like
in the section about the very high level interface, the Python
interpreter does not directly interact with the application (but that
will change in the next section).

  The code to run a function defined in a Python script is:

    #include <Python.h>

    int
    main(int argc, char *argv[])
    {
        PyObject *pName, *pModule, *pDict, *pFunc;
        PyObject *pArgs, *pValue;
        int i;

        if (argc < 3) {
            fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
            return 1;
        }

        Py_Initialize();
        pName = PyString_FromString(argv[1]);
        /* Error checking of pName left out */

        pModule = PyImport_Import(pName);
        Py_DECREF(pName);

        if (pModule != NULL) {
            pFunc = PyObject_GetAttrString(pModule, argv[2]);
            /* pFunc is a new reference */

            if (pFunc && PyCallable_Check(pFunc)) {
                pArgs = PyTuple_New(argc - 3);
                for (i = 0; i < argc - 3; ++i) {
                    pValue = PyInt_FromLong(atoi(argv[i + 3]));
                    if (!pValue) {
                        Py_DECREF(pArgs);
                        Py_DECREF(pModule);
                        fprintf(stderr, "Cannot convert argument\n");
                        return 1;
                    }
                    /* pValue reference stolen here: */
                    PyTuple_SetItem(pArgs, i, pValue);
                }
                pValue = PyObject_CallObject(pFunc, pArgs);
                Py_DECREF(pArgs);
                if (pValue != NULL) {
                    printf("Result of call: %ld\n", PyInt_AsLong(pValue));
                    Py_DECREF(pValue);
                }
                else {
                    Py_DECREF(pFunc);
                    Py_DECREF(pModule);
                    PyErr_Print();
                    fprintf(stderr,"Call failed\n");
                    return 1;
                }
            }
            else {
                if (PyErr_Occurred())
                    PyErr_Print();
                fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
            }
            Py_XDECREF(pFunc);
            Py_DECREF(pModule);
        }
        else {
            PyErr_Print();
            fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
            return 1;
        }
        Py_Finalize();
        return 0;
    }

This code loads a Python script using `argv[1]', and calls the function
named in `argv[2]'.  Its integer arguments are the other values of the
`argv' array.  If you compile and link this program (let's call the
finished executable *call*), and use it to execute a Python script,
such as:

    def multiply(a,b):
        print "Will compute", a, "times", b
        c = 0
        for i in range(0, a):
            c = c + b
        return c

then the result should be:

    $ call multiply multiply 3 2
    Will compute 3 times 2
    Result of call: 6

Although the program is quite large for its functionality, most of the
code is for data conversion between Python and C, and for error
reporting.  The interesting part with respect to embedding Python
starts with

    Py_Initialize();
    pName = PyString_FromString(argv[1]);
    /* Error checking of pName left out */
    pModule = PyImport_Import(pName);

After initializing the interpreter, the script is loaded using
PyImport_Import().  This routine needs a Python string as its
argument, which is constructed using the PyString_FromString()
data conversion routine.

    pFunc = PyObject_GetAttrString(pModule, argv[2]);
    /* pFunc is a new reference */

    if (pFunc && PyCallable_Check(pFunc)) {
        ...
    }
    Py_XDECREF(pFunc);

Once the script is loaded, the name we're looking for is retrieved using
PyObject_GetAttrString().  If the name exists, and the
object returned is callable, you can safely assume that it is a
function.  The program then proceeds by constructing a tuple of
arguments as normal.  The call to the Python function is then made with:

    pValue = PyObject_CallObject(pFunc, pArgs);

Upon return of the function, `pValue' is either _NULL_ or it contains a
reference to the return value of the function.  Be sure to release the
reference after examining the value.

6.5.4 Extending Embedded Python
-------------------------------

Until now, the embedded Python interpreter had no access to
functionality from the application itself.  The Python API allows this
by extending the embedded interpreter.  That is, the embedded
interpreter gets extended with routines provided by the application.
While it sounds complex, it is not so bad.  Simply forget for a while
that the application starts the Python interpreter.  Instead, consider
the application to be a set of subroutines, and write some glue code
that gives Python access to those routines, just like you would write a
normal Python extension.  For example:

    static int numargs=0;

    /* Return the number of arguments of the application command line */
    static PyObject*
    emb_numargs(PyObject *self, PyObject *args)
    {
        if(!PyArg_ParseTuple(args, ":numargs"))
            return NULL;
        return Py_BuildValue("i", numargs);
    }

    static PyMethodDef EmbMethods[] = {
        {"numargs", emb_numargs, METH_VARARGS,
         "Return the number of arguments received by the process."},
        {NULL, NULL, 0, NULL}
    };

Insert the above code just above the `main()' function. Also, insert the
following two statements directly after Py_Initialize().:

    numargs = argc;
    Py_InitModule("emb", EmbMethods);

These two lines initialize the `numargs' variable, and make the
`emb.numargs()' function accessible to the embedded Python interpreter.
With these extensions, the Python script can do things like

    import emb
    print "Number of arguments", emb.numargs()

In a real application, the methods will expose an API of the
application to Python.

6.5.5 Embedding Python in C++
-----------------------------

It is also possible to embed Python in a C++ program; precisely how
this is done will depend on the details of the C++ system used; in
general you will need to write the main program in C++, and use the C++
compiler to compile and link your program.  There is no need to
recompile Python itself using C++.

6.5.6 Linking Requirements
--------------------------

While the *configure* script shipped with the Python sources will
correctly build Python to export the symbols needed by dynamically
linked extensions, this is not automatically inherited by applications
which embed the Python library statically, at least on Unix.  This is
an issue when the application is linked to the static runtime library
(`libpython.a') and needs to load dynamic extensions (implemented as
`.so' files).

  The problem is that some entry points are defined by the Python
runtime solely for extension modules to use.  If the embedding
application does not use any of these entry points, some linkers will
not include those entries in the symbol table of the finished
executable.  Some additional options are needed to inform the linker
not to remove these symbols.

  Determining the right options to use for any given platform can be
quite difficult, but fortunately the Python configuration already has
those values.  To retrieve them from an installed Python interpreter,
start an interactive interpreter and have a short session like this:

    >>> import distutils.sysconfig
    >>> distutils.sysconfig.get_config_var('LINKFORSHARED')
    '-Xlinker -export-dynamic'

The contents of the string presented will be the options that should be
used.  If the string is empty, there's no need to add any additional
options.  The `LINKFORSHARED' definition corresponds to the variable of
the same name in Python's top-level `Makefile'.

7 Python/C API Reference Manual
*******************************

     Release: 2.7

     Date: February 08, 2011

  This manual documents the API used by C and C++ programmers who want
to write extension modules or embed Python.  It is a companion to
Extending and Embedding the Python Interpreter, which describes
the general principles of extension writing but does not document the
API functions in detail.

7.1 Introduction
================

The Application Programmer's Interface to Python gives C and C++
programmers access to the Python interpreter at a variety of levels.
The API is equally usable from C++, but for brevity it is generally
referred to as the Python/C API.  There are two fundamentally different
reasons for using the Python/C API.  The first reason is to write
_extension modules_ for specific purposes; these are C modules that
extend the Python interpreter.  This is probably the most common use.
The second reason is to use Python as a component in a larger
application; this technique is generally referred to as _embedding_
Python in an application.

  Writing an extension module is a relatively well-understood process,
where a "cookbook" approach works well.  There are several tools  that
automate the process to some extent.  While people have embedded
Python in other applications since its early existence, the process of
embedding Python is less straightforward than writing an extension.

  Many API functions are useful independent of whether you're embedding
or extending Python; moreover, most applications that embed Python
will need to provide a custom extension as well, so it's probably a
good idea to become familiar with writing an extension before
attempting to embed Python in a real application.

7.1.1 Include Files
-------------------

All function, type and macro definitions needed to use the Python/C API
are included in your code by the following line:

    #include "Python.h"

This implies inclusion of the following standard headers: `<stdio.h>',
`<string.h>', `<errno.h>', `<limits.h>', `<assert.h>' and `<stdlib.h>'
(if available).

     Note: Since Python may define some pre-processor definitions which
     affect the standard headers on some systems, you _must_ include
     `Python.h' before any standard headers are included.

  All user visible names defined by Python.h (except those defined by
the included standard headers) have one of the prefixes `Py' or `_Py'.
Names beginning with `_Py' are for internal use by the Python
implementation and should not be used by extension writers. Structure
member names do not have a reserved prefix.

  *Important:* user code should never define names that begin with `Py'
or `_Py'.  This confuses the reader, and jeopardizes the portability of
the user code to future Python versions, which may define additional
names beginning with one of these prefixes.

  The header files are typically installed with Python.  On Unix, these
are located in the directories `_prefix_/include/pythonversion/' and
`_exec_prefix_/include/pythonversion/', where `prefix' and `exec_prefix'
are defined by the corresponding parameters to Python's *configure*
script and _version_ is `sys.version[:3]'.  On Windows, the headers are
installed in `_prefix_/include', where `prefix' is the installation
directory specified to the installer.

  To include the headers, place both directories (if different) on your
compiler's search path for includes.  Do _not_ place the parent
directories on the search path and then use `#include
<pythonX.Y/Python.h>'; this will break on multi-platform builds since
the platform independent headers under `prefix' include the platform
specific headers from `exec_prefix'.

  C++ users should note that though the API is defined entirely using
C, the header files do properly declare the entry points to be `extern
"C"', so there is no need to do anything special to use the API from
C++.

7.1.2 Objects, Types and Reference Counts
-----------------------------------------

Most Python/C API functions have one or more arguments as well as a
return value of type PyObject*.  This type is a pointer to
an opaque data type representing an arbitrary Python object.  Since all
Python object types are treated the same way by the Python language in
most situations (e.g., assignments, scope rules, and argument passing),
it is only fitting that they should be represented by a single C type.
Almost all Python objects live on the heap: you never declare an
automatic or static variable of type PyObject, only pointer
variables of type PyObject* can  be declared.  The sole
exception are the type objects; since these must never be deallocated,
they are typically static PyTypeObject objects.

  All Python objects (even Python integers) have a _type_ and a
_reference count_.  An object's type determines what kind of object it
is (e.g., an integer, a list, or a user-defined function; there are
many more as explained in The standard type hierarchy).
For each of the well-known types there is a macro to check whether an
object is of that type; for instance, `PyList_Check(a)' is true if (and
only if) the object pointed to by _a_ is a Python list.

7.1.2.1 Reference Counts
........................

The reference count is important because today's computers have a
finite (and often severely limited) memory size; it counts how many
different places there are that have a reference to an object.  Such a
place could be another object, or a global (or static) C variable, or
a local variable in some C function.  When an object's reference count
becomes zero, the object is deallocated.  If it contains references to
other objects, their reference count is decremented.  Those other
objects may be deallocated in turn, if this decrement makes their
reference count become zero, and so on.  (There's an obvious problem
with objects that reference each other here; for now, the solution is
"don't do that.")

  Reference counts are always manipulated explicitly.  The normal way
is  to use the macro Py_INCREF() to increment an object's
reference count by one, and Py_DECREF() to decrement it by
 one.  The Py_DECREF() macro is considerably more complex
than the incref one, since it must check whether the reference count
becomes zero and then cause the object's deallocator to be called. The
deallocator is a function pointer contained in the object's type
structure.  The type-specific deallocator takes care of decrementing the
reference counts for other objects contained in the object if this is a
compound object type, such as a list, as well as performing any
additional finalization that's needed.  There's no chance that the
reference count can overflow; at least as many bits are used to hold
the reference count as there are distinct memory locations in virtual
memory (assuming `sizeof(Py_ssize_t) >= sizeof(void*)').  Thus, the
reference count increment is a simple operation.

  It is not necessary to increment an object's reference count for
every  local variable that contains a pointer to an object.  In theory,
the  object's reference count goes up by one when the variable is made
to  point to it and it goes down by one when the variable goes out of
scope.  However, these two cancel each other out, so at the end the
reference count hasn't changed.  The only real reason to use the
reference count is to prevent the object from being deallocated as
long as our variable is pointing to it.  If we know that there is at
least one other reference to the object that lives at least as long as
our variable, there is no need to increment the reference count
temporarily.  An important situation where this arises is in objects
that are passed as arguments to C functions in an extension module
that are called from Python; the call mechanism guarantees to hold a
reference to every argument for the duration of the call.

  However, a common pitfall is to extract an object from a list and
hold on to it for a while without incrementing its reference count.
Some other operation might conceivably remove the object from the list,
decrementing its reference count and possible deallocating it. The real
danger is that innocent-looking operations may invoke arbitrary Python
code which could do this; there is a code path which allows control to
flow back to the user from a Py_DECREF(), so almost any
operation is potentially dangerous.

  A safe approach is to always use the generic operations (functions
whose name begins with `PyObject_', `PyNumber_', `PySequence_' or
`PyMapping_').  These operations always increment the reference count
of the object they return.  This leaves the caller with the
responsibility to call Py_DECREF() when they are done with
the result; this soon becomes second nature.

7.1.2.2 Reference Count Details
...............................

The reference count behavior of functions in the Python/C API is best
explained in terms of _ownership of references_.  Ownership pertains to
references, never to objects (objects are not owned: they are always
shared).  "Owning a reference" means being responsible for calling
Py_DECREF on it when the reference is no longer needed.  Ownership can
also be transferred, meaning that the code that receives ownership of
the reference then becomes responsible for eventually decref'ing it by
calling Py_DECREF() or Py_XDECREF().  when
it's no longer needed--or passing on this responsibility (usually to its
caller). When a function passes ownership of a reference on to its
caller, the caller is said to receive a _new_ reference.  When no
ownership is transferred, the caller is said to _borrow_ the reference.
Nothing needs to be done for a borrowed reference.

  Conversely, when a calling function passes in a reference to an
object, there are two possibilities: the function _steals_ a  reference
to the object, or it does not.  _Stealing a reference_ means that when
you pass a reference to a function, that function assumes that it now
owns that reference, and you are not responsible for it any longer.

  Few functions steal references; the two notable exceptions are
PyList_SetItem() and PyTuple_SetItem(), which  steal
a reference to the item (but not to the tuple or list into which the
item is put!).  These functions were designed to steal a reference
because of a common idiom for populating a tuple or list with newly
created objects; for example, the code to create the tuple `(1, 2,
"three")' could look like this (forgetting about error handling for the
moment; a better way to code this is shown below):

    PyObject *t;

    t = PyTuple_New(3);
    PyTuple_SetItem(t, 0, PyInt_FromLong(1L));
    PyTuple_SetItem(t, 1, PyInt_FromLong(2L));
    PyTuple_SetItem(t, 2, PyString_FromString("three"));

Here, see PyInt_FromLong() returns a new reference which is
immediately stolen by PyTuple_SetItem().  When you want to
keep using an object although the reference to it will be stolen, use
Py_INCREF() to grab another reference before calling the
reference-stealing function.

  Incidentally, see PyTuple_SetItem() is the _only_ way to set
tuple items; see PySequence_SetItem() and
PyObject_SetItem() refuse to do this since tuples are an
immutable data type.  You should only use PyTuple_SetItem()
for tuples that you are creating yourself.

  Equivalent code for populating a list can be written using
PyList_New().  and PyList_SetItem().

  However, in practice, you will rarely use these ways of creating and
populating a tuple or list.  There's a generic function, see
Py_BuildValue(), that can create most common objects from C
values, directed by a _format string_.  For example, the above two
blocks of code could be replaced by the following (which also takes
care of the error checking):

    PyObject *tuple, *list;

    tuple = Py_BuildValue("(iis)", 1, 2, "three");
    list = Py_BuildValue("[iis]", 1, 2, "three");

It is much more common to use PyObject_SetItem() and
friends with items whose references you are only borrowing, like
arguments that were passed in to the function you are writing.  In that
case, their behaviour regarding reference counts is much saner, since
you don't have to increment a reference count so you can give a
reference away ("have it be stolen").  For example, this function sets
all items of a list (actually, any mutable sequence) to a given item:

    int
    set_all(PyObject *target, PyObject *item)
    {
        int i, n;

        n = PyObject_Length(target);
        if (n < 0)
            return -1;
        for (i = 0; i < n; i++) {
            PyObject *index = PyInt_FromLong(i);
            if (!index)
                return -1;
            if (PyObject_SetItem(target, index, item) < 0)
                return -1;
            Py_DECREF(index);
        }
        return 0;
    }

The situation is slightly different for function return values.   While
passing a reference to most functions does not change your  ownership
responsibilities for that reference, many functions that  return a
reference to an object give you ownership of the reference. The reason
is simple: in many cases, the returned object is created  on the fly,
and the reference you get is the only reference to the  object.
Therefore, the generic functions that return object references, like
PyObject_GetItem() and  PySequence_GetItem(),
always return a new reference (the caller becomes the owner of the
reference).

  It is important to realize that whether you own a reference returned
by a function depends on which function you call only -- _the plumage_
(the type of the object passed as an argument to the function) _doesn't
enter into it!_ Thus, if you  extract an item from a list using
PyList_GetItem(), you don't own the reference -- but if you
obtain the same item from the same list using
PySequence_GetItem(). (which happens to take exactly the same
arguments), you do own a reference to the returned object.

  Here is an example of how you could write a function that computes
the sum of the items in a list of integers; once using
PyList_GetItem(), and once using PySequence_GetItem().

    long
    sum_list(PyObject *list)
    {
        int i, n;
        long total = 0;
        PyObject *item;

        n = PyList_Size(list);
        if (n < 0)
            return -1; /* Not a list */
        for (i = 0; i < n; i++) {
            item = PyList_GetItem(list, i); /* Can't fail */
            if (!PyInt_Check(item)) continue; /* Skip non-integers */
            total += PyInt_AsLong(item);
        }
        return total;
    }

    long
    sum_sequence(PyObject *sequence)
    {
        int i, n;
        long total = 0;
        PyObject *item;
        n = PySequence_Length(sequence);
        if (n < 0)
            return -1; /* Has no length */
        for (i = 0; i < n; i++) {
            item = PySequence_GetItem(sequence, i);
            if (item == NULL)
                return -1; /* Not a sequence, or other failure */
            if (PyInt_Check(item))
                total += PyInt_AsLong(item);
            Py_DECREF(item); /* Discard reference ownership */
        }
        return total;
    }

7.1.2.3 Types
.............

There are few other data types that play a significant role in  the
Python/C API; most are simple C types such as `int',  `long', `double'
and `char*'.  A few structure types  are used to describe static tables
used to list the functions exported  by a module or the data attributes
of a new object type, and another is used to describe the value of a
complex number.  These will  be discussed together with the functions
that use them.

7.1.3 Exceptions
----------------

The Python programmer only needs to deal with exceptions if specific
error handling is required; unhandled exceptions are automatically
propagated to the caller, then to the caller's caller, and so on, until
they reach the top-level interpreter, where they are reported to the
user accompanied by a stack traceback.

  For C programmers, however, error checking always has to be explicit.
All functions in the Python/C API can raise exceptions, unless an
explicit claim is made otherwise in a function's documentation.  In
general, when a function encounters an error, it sets an exception,
discards any object references that it owns, and returns an error
indicator.  If not documented otherwise, this indicator is either
_NULL_ or `-1', depending on the function's return type.  A few
functions return a Boolean true/false result, with false indicating an
error.  Very few functions return no explicit error indicator or have an
ambiguous return value, and require explicit testing for errors with
PyErr_Occurred().  These exceptions are always explicitly
documented.

  Exception state is maintained in per-thread storage (this is
equivalent to using global storage in an unthreaded application).  A
thread can be in one of two states: an exception has occurred, or not.
The function PyErr_Occurred() can be used to check for
this: it returns a borrowed reference to the exception type object when
an exception has occurred, and _NULL_ otherwise.  There are a number of
functions to set the exception state: PyErr_SetString() is
the most common (though not the most general) function to set the
exception state, and PyErr_Clear() clears the exception
state.

  The full exception state consists of three objects (all of which can
be _NULL_): the exception type, the corresponding exception  value, and
the traceback.  These have the same meanings as the Python   objects
`sys.exc_type', `sys.exc_value', and `sys.exc_traceback'; however, they
are not the same: the Python objects represent the last exception being
handled by a Python  try... except statement,
while the C level exception state only exists while an exception is
being passed on between C functions until it reaches the Python
bytecode interpreter's  main loop, which takes care of transferring it
to `sys.exc_type' and friends.

  Note that starting with Python 1.5, the preferred, thread-safe way to
access the exception state from Python code is to call the function
sys.exc_info(), which returns the per-thread exception state
for Python code.  Also, the semantics of both ways to access the
exception state have changed so that a function which catches an
exception will save and restore its thread's exception state so as to
preserve the exception state of its caller.  This prevents common bugs
in exception handling code caused by an innocent-looking function
overwriting the exception being handled; it also reduces the often
unwanted lifetime extension for objects that are referenced by the
stack frames in the traceback.

  As a general principle, a function that calls another function to
perform some task should check whether the called function raised an
exception, and if so, pass the exception state on to its caller.  It
should discard any object references that it owns, and return an  error
indicator, but it should _not_ set another exception -- that would
overwrite the exception that was just raised, and lose important
information about the exact cause of the error.

  A simple example of detecting exceptions and passing them on is shown
in the `sum_sequence()' example above.  It so happens that that example
doesn't need to clean up any owned references when it detects an error.
The following example function shows some error cleanup.  First, to
remind you why you like Python, we show the equivalent Python code:

    def incr_item(dict, key):
        try:
            item = dict[key]
        except KeyError:
            item = 0
        dict[key] = item + 1

Here is the corresponding C code, in all its glory:

    int
    incr_item(PyObject *dict, PyObject *key)
    {
        /* Objects all initialized to NULL for Py_XDECREF */
        PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
        int rv = -1; /* Return value initialized to -1 (failure) */

        item = PyObject_GetItem(dict, key);
        if (item == NULL) {
            /* Handle KeyError only: */
            if (!PyErr_ExceptionMatches(PyExc_KeyError))
                goto error;

            /* Clear the error and use zero: */
            PyErr_Clear();
            item = PyInt_FromLong(0L);
            if (item == NULL)
                goto error;
        }
        const_one = PyInt_FromLong(1L);
        if (const_one == NULL)
            goto error;

        incremented_item = PyNumber_Add(item, const_one);
        if (incremented_item == NULL)
            goto error;

        if (PyObject_SetItem(dict, key, incremented_item) < 0)
            goto error;
        rv = 0; /* Success */
        /* Continue with cleanup code */

     error:
        /* Cleanup code, shared by success and failure path */

        /* Use Py_XDECREF() to ignore NULL references */
        Py_XDECREF(item);
        Py_XDECREF(const_one);
        Py_XDECREF(incremented_item);

        return rv; /* -1 for error, 0 for success */
    }

This example represents an endorsed use of the `goto' statement  in C!
It illustrates the use of PyErr_ExceptionMatches() and
PyErr_Clear() to handle specific exceptions, and the use of
Py_XDECREF() to dispose of owned references that may be
_NULL_ (note the `'X'' in the name; see Py_DECREF() would
crash when confronted with a _NULL_ reference).  It is important that
the variables used to hold owned references are initialized to _NULL_
for this to work; likewise, the proposed return value is initialized to
`-1' (failure) and only set to success after the final call made is
successful.

7.1.4 Embedding Python
----------------------

The one important task that only embedders (as opposed to extension
writers) of the Python interpreter have to worry about is the
initialization, and possibly the finalization, of the Python
interpreter.  Most functionality of the interpreter can only be used
after the interpreter has been initialized.

  The basic initialization function is Py_Initialize().
This initializes the table of loaded modules, and creates the
fundamental modules __builtin__, __main__, sys,
and exceptions.  It also initializes the module search
path (`sys.path').

  Py_Initialize() does not set the "script argument list"
(`sys.argv').  If this variable is needed by Python code that will be
executed later, it must be set explicitly with a call to
`PySys_SetArgvEx(argc, argv, updatepath)' after the call to
Py_Initialize().

  On most systems (in particular, on Unix and Windows, although the
details are slightly different), see Py_Initialize().
calculates the module search path based upon its best guess for the
location of the standard Python interpreter executable, assuming that
the Python library is found in a fixed location relative to the Python
interpreter executable.  In particular, it looks for a directory named
`lib/python_X.Y_' relative to the parent directory where the executable
named `python' is found on the shell command search path (the
environment variable `PATH').

  For instance, if the Python executable is found in
`/usr/local/bin/python', it will assume that the libraries are in
`/usr/local/lib/python_X.Y_'.  (In fact, this particular path is also
the "fallback" location, used when no executable file named `python' is
found along `PATH'.)  The user can override this behavior by setting the
environment variable PYTHONHOME, or insert additional
directories in front of the standard path by setting PYTHONPATH.

  The embedding application can steer the search by calling
`Py_SetProgramName(file)' _before_ calling  Py_Initialize().
Note that PYTHONHOME still overrides this and
PYTHONPATH is still inserted in front of the standard path.  An
application that requires total control has to provide its own
implementation of Py_GetPath(), Py_GetPrefix(),
Py_GetExecPrefix(), and Py_GetProgramFullPath().
(all defined in `Modules/getpath.c').

  Sometimes, it is desirable to "uninitialize" Python.  For instance,
the application may want to start over (make another call to
Py_Initialize()) or the application is simply done with its  use
of Python and wants to free memory allocated by Python.  This can be
accomplished by calling Py_Finalize().  The function
Py_IsInitialized() returns true if Python is currently in the
initialized state.  More information about these functions is given in
a later chapter. Notice that Py_Finalize().  does _not_ free
all memory allocated by the Python interpreter, e.g. memory allocated
by extension modules currently cannot be released.

7.1.5 Debugging Builds
----------------------

Python can be built with several macros to enable extra checks of the
interpreter and extension modules.  These checks tend to add a large
amount of overhead to the runtime so they are not enabled by default.

  A full list of the various types of debugging builds is in the file
`Misc/SpecialBuilds.txt' in the Python source distribution. Builds are
available that support tracing of reference counts, debugging the memory
allocator, or low-level profiling of the main interpreter loop.  Only
the most frequently-used builds will be described in the remainder of
this section.

  Compiling the interpreter with the `Py_DEBUG' macro defined produces
what is generally meant by "a debug build" of Python. `Py_DEBUG' is
enabled in the Unix build by adding `--with-pydebug' to the `configure'
command.  It is also implied by the presence of the not-Python-specific
`_DEBUG' macro.  When `Py_DEBUG' is enabled in the Unix build, compiler
optimization is disabled.

  In addition to the reference count debugging described below, the
following extra checks are performed:

   * Extra checks are added to the object allocator.

   * Extra checks are added to the parser and compiler.

   * Downcasts from wide types to narrow types are checked for loss of
     information.

   * A number of assertions are added to the dictionary and set
     implementations.  In addition, the set object acquires a
     `test_c_api()' method.

   * Sanity checks of the input arguments are added to frame creation.

   * The storage for long ints is initialized with a known invalid
     pattern to catch reference to uninitialized digits.

   * Low-level tracing and extra exception checking are added to the
     runtime virtual machine.

   * Extra checks are added to the memory arena implementation.

   * Extra debugging is added to the thread module.

  There may be additional checks not mentioned here.

  Defining `Py_TRACE_REFS' enables reference tracing.  When defined, a
circular doubly linked list of active objects is maintained by adding
two extra fields to every PyObject.  Total allocations are
tracked as well.  Upon exit, all existing references are printed.  (In
interactive mode this happens after every statement run by the
interpreter.)  Implied by `Py_DEBUG'.

  Please refer to `Misc/SpecialBuilds.txt' in the Python source
distribution for more detailed information.

7.2 The Very High Level Layer
=============================

The functions in this chapter will let you execute Python source code
given in a file or a buffer, but they will not let you interact in a
more detailed way with the interpreter.

  Several of these functions accept a start symbol from the grammar as a
parameter.  The available start symbols are `Py_eval_input',
`Py_file_input', and `Py_single_input'.  These are described following
the functions which accept them as parameters.

  Note also that several of these functions take `FILE*' parameters.
One particular issue which needs to be handled carefully is that the
`FILE' structure for different C libraries can be different and
incompatible.  Under Windows (at least), it is possible for dynamically
linked extensions to actually use different libraries, so care should
be taken that `FILE*' parameters are only passed to these functions if
it is certain that they were created by the same library that the
Python runtime is using.

 -- Function: int Py_Main (int argc, char **argv)
     The main program for the standard interpreter.  This is made
     available for programs which embed Python.  The _argc_ and _argv_
     parameters should be prepared exactly as those which are passed to
     a C program's `main()' function.  It is important to note that the
     argument list may be modified (but the contents of the strings
     pointed to by the argument list are not). The return value will be
     the integer passed to the sys.exit() function, `1' if
     the interpreter exits due to an exception, or `2' if the parameter
     list does not represent a valid Python command line.

     Note that if an otherwise unhandled SystemError is
     raised, this function will not return `1', but exit the process,
     as long as `Py_InspectFlag' is not set.

 -- Function: int PyRun_AnyFile (FILE *fp, const char *filename)
     This is a simplified interface to PyRun_AnyFileExFlags()
     below, leaving _closeit_ set to `0' and _flags_ set to
     _NULL_.

 -- Function: int PyRun_AnyFileFlags (FILE *fp, const char *filename,
          PyCompilerFlags *flags)
     This is a simplified interface to PyRun_AnyFileExFlags()
     below, leaving the _closeit_ argument set to `0'.

 -- Function: int PyRun_AnyFileEx (FILE *fp, const char *filename,
          int closeit)
     This is a simplified interface to PyRun_AnyFileExFlags()
     below, leaving the _flags_ argument set to _NULL_.

 -- Function: int PyRun_AnyFileExFlags (FILE *fp, const char *filename,
          int closeit, PyCompilerFlags *flags)
     If _fp_ refers to a file associated with an interactive device
     (console or terminal input or Unix pseudo-terminal), return the
     value of PyRun_InteractiveLoop(), otherwise return the
     result of PyRun_SimpleFile().  If _filename_ is
     _NULL_, this function uses `"???"' as the filename.

 -- Function: int PyRun_SimpleString (const char *command)
     This is a simplified interface to PyRun_SimpleStringFlags()
     below, leaving the _PyCompilerFlags*_ argument set to NULL.

 -- Function: int PyRun_SimpleStringFlags (const char *command,
          PyCompilerFlags *flags)
     Executes the Python source code from _command_ in the
     __main__ module according to the _flags_ argument. If
     __main__ does not already exist, it is created.  Returns `0'
     on success or `-1' if an exception was raised.  If there was an
     error, there is no way to get the exception information. For the
     meaning of _flags_, see below.

     Note that if an otherwise unhandled SystemError is
     raised, this function will not return `-1', but exit the process,
     as long as `Py_InspectFlag' is not set.

 -- Function: int PyRun_SimpleFile (FILE *fp, const char *filename)
     This is a simplified interface to PyRun_SimpleFileExFlags()
     below, leaving _closeit_ set to `0' and _flags_ set to
     _NULL_.

 -- Function: int PyRun_SimpleFileFlags (FILE *fp, const
          char *filename, PyCompilerFlags *flags)
     This is a simplified interface to PyRun_SimpleFileExFlags()
     below, leaving _closeit_ set to `0'.

 -- Function: int PyRun_SimpleFileEx (FILE *fp, const char *filename,
          int closeit)
     This is a simplified interface to PyRun_SimpleFileExFlags()
     below, leaving _flags_ set to _NULL_.

 -- Function: int PyRun_SimpleFileExFlags (FILE *fp, const
          char *filename, int closeit, PyCompilerFlags *flags)
     Similar to PyRun_SimpleStringFlags(), but the Python
     source code is read from _fp_ instead of an in-memory string.
     _filename_ should be the name of the file.  If _closeit_ is true,
     the file is closed before PyRun_SimpleFileExFlags returns.

 -- Function: int PyRun_InteractiveOne (FILE *fp, const char *filename)
     This is a simplified interface to
     PyRun_InteractiveOneFlags() below, leaving _flags_ set to
     _NULL_.

 -- Function: int PyRun_InteractiveOneFlags (FILE *fp, const
          char *filename, PyCompilerFlags *flags)
     Read and execute a single statement from a file associated with an
     interactive device according to the _flags_ argument.  The user
     will be prompted using `sys.ps1' and `sys.ps2'.  Returns `0' when
     the input was executed successfully, `-1' if there was an
     exception, or an error code from the `errcode.h' include file
     distributed as part of Python if there was a parse error.  (Note
     that `errcode.h' is not included by `Python.h', so must be
     included specifically if needed.)

 -- Function: int PyRun_InteractiveLoop (FILE *fp, const char *filename)
     This is a simplified interface to
     PyRun_InteractiveLoopFlags() below, leaving _flags_ set to
     _NULL_.

 -- Function: int PyRun_InteractiveLoopFlags (FILE *fp, const
          char *filename, PyCompilerFlags *flags)
     Read and execute statements from a file associated with an
     interactive device until EOF is reached.  The user will be
     prompted using `sys.ps1' and `sys.ps2'.  Returns `0' at EOF.

 -- Function: struct _node* PyParser_SimpleParseString (const
          char *str, int start)
     This is a simplified interface to
     PyParser_SimpleParseStringFlagsFilename() below, leaving
     _filename_ set to _NULL_ and _flags_ set to `0'.

 -- Function: struct _node* PyParser_SimpleParseStringFlags (const
          char *str, int start, int flags)
     This is a simplified interface to
     PyParser_SimpleParseStringFlagsFilename() below, leaving
     _filename_ set to _NULL_.

 -- Function: struct _node* PyParser_SimpleParseStringFlagsFilename
          (const char *str, const char *filename, int start, int flags)
     Parse Python source code from _str_ using the start token _start_
     according to the _flags_ argument.  The result can be used to
     create a code object which can be evaluated efficiently. This is
     useful if a code fragment must be evaluated many times.

 -- Function: struct _node* PyParser_SimpleParseFile (FILE *fp, const
          char *filename, int start)
     This is a simplified interface to
     PyParser_SimpleParseFileFlags() below, leaving _flags_ set
     to `0'

 -- Function: struct _node* PyParser_SimpleParseFileFlags (FILE *fp,
          const char *filename, int start, int flags)
     Similar to PyParser_SimpleParseStringFlagsFilename(),
     but the Python source code is read from _fp_ instead of an
     in-memory string.

 -- Function: PyObject* PyRun_String (const char *str, int start,
          PyObject *globals, PyObject *locals)
     Return value: New reference.

     This is a simplified interface to PyRun_StringFlags().
     below, leaving _flags_ set to _NULL_.

 -- Function: PyObject* PyRun_StringFlags (const char *str, int start,
          PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
     Return value: New reference.

     Execute Python source code from _str_ in the context specified by
     the dictionaries _globals_ and _locals_ with the compiler flags
     specified by _flags_.  The parameter _start_ specifies the start
     token that should be used to parse the source code.

     Returns the result of executing the code as a Python object, or
     _NULL_ if an exception was raised.

 -- Function: PyObject* PyRun_File (FILE *fp, const char *filename,
          int start, PyObject *globals, PyObject *locals)
     Return value: New reference.

     This is a simplified interface to PyRun_FileExFlags().
     below, leaving _closeit_ set to `0' and _flags_ set to _NULL_.

 -- Function: PyObject* PyRun_FileEx (FILE *fp, const char *filename,
          int start, PyObject *globals, PyObject *locals, int closeit)
     Return value: New reference.

     This is a simplified interface to PyRun_FileExFlags().
     below, leaving _flags_ set to _NULL_.

 -- Function: PyObject* PyRun_FileFlags (FILE *fp, const
          char *filename, int start, PyObject *globals,
          PyObject *locals, PyCompilerFlags *flags)
     Return value: New reference.

     This is a simplified interface to PyRun_FileExFlags().
     below, leaving _closeit_ set to `0'.

 -- Function: PyObject* PyRun_FileExFlags (FILE *fp, const
          char *filename, int start, PyObject *globals,
          PyObject *locals, int closeit, PyCompilerFlags *flags)
     Return value: New reference.

     Similar to PyRun_StringFlags(), but the Python source
     code is read from _fp_ instead of an in-memory string. _filename_
     should be the name of the file.  If _closeit_ is true, the file is
     closed before PyRun_FileExFlags().  returns.

 -- Function: PyObject* Py_CompileString (const char *str, const
          char *filename, int start)
     Return value: New reference.

     This is a simplified interface to Py_CompileStringFlags()
     below, leaving _flags_ set to _NULL_.

 -- Function: PyObject* Py_CompileStringFlags (const char *str, const
          char *filename, int start, PyCompilerFlags *flags)
     Return value: New reference.

     Parse and compile the Python source code in _str_, returning the
     resulting code object.  The start token is given by _start_; this
     can be used to constrain the code which can be compiled and should
     be `Py_eval_input', `Py_file_input', or `Py_single_input'.  The
     filename specified by _filename_ is used to construct the code
     object and may appear in tracebacks or SyntaxError.
     exception messages.  This returns _NULL_ if the code cannot be
     parsed or compiled.

 -- Function: PyObject* PyEval_EvalCode (PyCodeObject *co,
          PyObject *globals, PyObject *locals)
     Return value: New reference.

     This is a simplified interface to PyEval_EvalCodeEx(),
     with just the code object, and the dictionaries of global and
     local variables.  The other arguments are set to _NULL_.

 -- Function: PyObject* PyEval_EvalCodeEx (PyCodeObject *co,
          PyObject *globals, PyObject *locals, PyObject **args,
          int argcount, PyObject **kws, int kwcount, PyObject **defs,
          int defcount, PyObject *closure)
     Evaluate a precompiled code object, given a particular environment
     for its evaluation.  This environment consists of dictionaries of
     global and local variables, arrays of arguments, keywords and
     defaults, and a closure tuple of cells.

 -- Function: PyObject* PyEval_EvalFrame (PyFrameObject *f)
     Evaluate an execution frame.  This is a simplified interface to
     PyEval_EvalFrameEx, for backward compatibility.

 -- Function: PyObject* PyEval_EvalFrameEx (PyFrameObject *f,
          int throwflag)
     This is the main, unvarnished function of Python interpretation.
     It is literally 2000 lines long.  The code object associated with
     the execution frame _f_ is executed, interpreting bytecode and
     executing calls as needed.  The additional _throwflag_ parameter
     can mostly be ignored - if true, then it causes an exception to
     immediately be thrown; this is used for the `throw()' methods of
     generator objects.

 -- Function: int PyEval_MergeCompilerFlags (PyCompilerFlags *cf)
     This function changes the flags of the current evaluation frame,
     and returns true on success, false on failure.

 -- var: int Py_eval_input
     The start symbol from the Python grammar for isolated expressions;
     for use with Py_CompileString().

 -- var: int Py_file_input
     The start symbol from the Python grammar for sequences of
     statements as read from a file or other source; for use with
     Py_CompileString().  This is the symbol to use when
     compiling arbitrarily long Python source code.

 -- var: int Py_single_input
     The start symbol from the Python grammar for a single statement;
     for use with Py_CompileString(). This is the symbol
     used for the interactive interpreter loop.

 -- type: struct PyCompilerFlags
     This is the structure used to hold compiler flags.  In cases where
     code is only being compiled, it is passed as `int flags', and in
     cases where code is being executed, it is passed as
     `PyCompilerFlags *flags'.  In this case, `from __future__ import'
     can modify _flags_.

     Whenever `PyCompilerFlags *flags' is _NULL_, `cf_flags' is treated
     as equal to `0', and any modification due to `from __future__
     import' is discarded.

         struct PyCompilerFlags {
             int cf_flags;
         }

 -- var: int CO_FUTURE_DIVISION
     This bit can be set in _flags_ to cause division operator `/' to be
     interpreted as "true division" according to PEP 238
     (http://www.python.org/dev/peps/pep-0238).

7.3 Reference Counting
======================

The macros in this section are used for managing reference counts of
Python objects.

 -- Function: void Py_INCREF (PyObject *o)
     Increment the reference count for object _o_.  The object must not
     be _NULL_; if you aren't sure that it isn't _NULL_, use
     Py_XINCREF().

 -- Function: void Py_XINCREF (PyObject *o)
     Increment the reference count for object _o_.  The object may be
     _NULL_, in which case the macro has no effect.

 -- Function: void Py_DECREF (PyObject *o)
     Decrement the reference count for object _o_.  The object must not
     be _NULL_; if you aren't sure that it isn't _NULL_, use
     Py_XDECREF().  If the reference count reaches zero, the
     object's type's deallocation function (which must not be _NULL_)
     is invoked.

          Warning: The deallocation function can cause arbitrary Python
          code to be invoked (e.g.  when a class instance with a
          __del__() method is deallocated).  While exceptions in
          such code are not propagated, the executed code has free
          access to all Python global variables.  This means that any
          object that is reachable from a global variable should be in
          a consistent state before Py_DECREF() is
          invoked.  For example, code to delete an object from a list
          should copy a reference to the deleted object in a temporary
          variable, update the list data structure, and then call
          Py_DECREF() for the temporary variable.

 -- Function: void Py_XDECREF (PyObject *o)
     Decrement the reference count for object _o_.  The object may be
     _NULL_, in which case the macro has no effect; otherwise the
     effect is the same as for Py_DECREF(), and the same
     warning applies.

 -- Function: void Py_CLEAR (PyObject *o)
     Decrement the reference count for object _o_.  The object may be
     _NULL_, in which case the macro has no effect; otherwise the
     effect is the same as for Py_DECREF(), except that the
     argument is also set to _NULL_.  The warning for
     Py_DECREF() does not apply with respect to the object
     passed because the macro carefully uses a temporary variable and
     sets the argument to _NULL_ before decrementing its reference
     count.

     It is a good idea to use this macro whenever decrementing the
     value of a variable that might be traversed during garbage
     collection.

     New in version 2.4.

  The following functions are for runtime dynamic embedding of Python:
`Py_IncRef(PyObject *o)', `Py_DecRef(PyObject *o)'. They are simply
exported function versions of Py_XINCREF() and
Py_XDECREF(), respectively.

  The following functions or macros are only for use within the
interpreter core: `_Py_Dealloc()', `_Py_ForgetReference()',
`_Py_NewReference()', as well as the global variable `_Py_RefTotal'.

7.4 Exception Handling
======================

The functions described in this chapter will let you handle and raise
Python exceptions.  It is important to understand some of the basics of
Python exception handling.  It works somewhat like the Unix `errno'
variable: there is a global indicator (per thread) of the last error
that occurred.  Most functions don't clear this on success, but will
set it to indicate the cause of the error on failure.  Most functions
also return an error indicator, usually _NULL_ if they are supposed to
return a pointer, or `-1' if they return an integer (exception: the
`PyArg_*()' functions return `1' for success and `0' for failure).

  When a function must fail because some function it called failed, it
generally doesn't set the error indicator; the function it called
already set it.  It is responsible for either handling the error and
clearing the exception or returning after cleaning up any resources it
holds (such as object references or memory allocations); it should
_not_ continue normally if it is not prepared to handle the error.  If
returning due to an error, it is important to indicate to the caller
that an error has been set.  If the error is not handled or carefully
propagated, additional calls into the Python/C API may not behave as
intended and may fail in mysterious ways.

  The error indicator consists of three Python objects corresponding to
 the Python variables `sys.exc_type', `sys.exc_value' and
`sys.exc_traceback'.  API functions exist to interact with the error
indicator in various ways.  There is a separate error indicator for
each thread.

 -- Function: void PyErr_PrintEx (int set_sys_last_vars)
     Print a standard traceback to `sys.stderr' and clear the error
     indicator.  Call this function only when the error indicator is
     set.  (Otherwise it will cause a fatal error!)

     If _set_sys_last_vars_ is nonzero, the variables
     sys.last_type, sys.last_value and
     sys.last_traceback will be set to the type, value and
     traceback of the printed exception, respectively.

 -- Function: void PyErr_Print ()
     Alias for `PyErr_PrintEx(1)'.

 -- Function: PyObject* PyErr_Occurred ()
     Return value: Borrowed reference.

     Test whether the error indicator is set.  If set, return the
     exception _type_ (the first argument to the last call to one of
     the `PyErr_Set*()' functions or to PyErr_Restore()).
     If not set, return _NULL_.  You do not own a reference to the
     return value, so you do not need to Py_DECREF().  it.

          Note: Do not compare the return value to a specific
          exception; use PyErr_ExceptionMatches() instead,
          shown below.  (The comparison could easily fail since the
          exception may be an instance instead of a class, in the case
          of a class exception, or it may the a subclass of the
          expected exception.)

 -- Function: int PyErr_ExceptionMatches (PyObject *exc)
     Equivalent to `PyErr_GivenExceptionMatches(PyErr_Occurred(),
     exc)'.  This should only be called when an exception is actually
     set; a memory access violation will occur if no exception has been
     raised.

 -- Function: int PyErr_GivenExceptionMatches (PyObject *given,
          PyObject *exc)
     Return true if the _given_ exception matches the exception in
     _exc_.  If _exc_ is a class object, this also returns true when
     _given_ is an instance of a subclass.  If _exc_ is a tuple, all
     exceptions in the tuple (and recursively in subtuples) are
     searched for a match.

 -- Function: void PyErr_NormalizeException (PyObject**exc,
          PyObject**val, PyObject**tb)
     Under certain circumstances, the values returned by
     PyErr_Fetch() below can be "unnormalized", meaning that
     `*exc' is a class object but `*val' is not an instance of the
     same class.  This function can be used to instantiate the class in
     that case.  If the values are already normalized, nothing happens.
     The delayed normalization is implemented to improve performance.

 -- Function: void PyErr_Clear ()
     Clear the error indicator.  If the error indicator is not set,
     there is no effect.

 -- Function: void PyErr_Fetch (PyObject **ptype, PyObject **pvalue,
          PyObject **ptraceback)
     Retrieve the error indicator into three variables whose addresses
     are passed.  If the error indicator is not set, set all three
     variables to _NULL_.  If it is set, it will be cleared and you own
     a reference to each object retrieved.  The value and traceback
     object may be _NULL_ even when the type object is not.

          Note: This function is normally only used by code that needs
          to handle exceptions or by code that needs to save and
          restore the error indicator temporarily.

 -- Function: void PyErr_Restore (PyObject *type, PyObject *value,
          PyObject *traceback)
     Set  the error indicator from the three objects.  If the error
     indicator is already set, it is cleared first.  If the objects are
     _NULL_, the error indicator is cleared.  Do not pass a _NULL_ type
     and non-_NULL_ value or traceback.  The exception type should be a
     class.  Do not pass an invalid exception type or value. (Violating
     these rules will cause subtle problems later.)  This call takes
     away a reference to each object: you must own a reference to each
     object before the call and after the call you no longer own these
     references.  (If you don't understand this, don't use this
     function.  I warned you.)

          Note: This function is normally only used by code that needs
          to save and restore the error indicator temporarily; use
          PyErr_Fetch() to save the current exception
          state.

 -- Function: void PyErr_SetString (PyObject *type, const char *message)
     This is the most common way to set the error indicator.  The first
     argument specifies the exception type; it is normally one of the
     standard exceptions, e.g. `PyExc_RuntimeError'.  You need not
     increment its reference count.  The second argument is an error
     message; it is converted to a string object.

 -- Function: void PyErr_SetObject (PyObject *type, PyObject *value)
     This function is similar to PyErr_SetString() but
     lets you specify an arbitrary Python object for the "value" of the
     exception.

 -- Function: PyObject* PyErr_Format (PyObject *exception, const
          char *format, ...)
     Return value: Always NULL.

     This function sets the error indicator and returns _NULL_.
     _exception_ should be a Python exception (class, not an instance).
     _format_ should be a string, containing format codes, similar to
     `printf()'. The `width.precision' before a format code is parsed,
     but the width part is ignored.

     Format Characters       Type                Comment
     ---------------------------------------------------------------------------------
     `%%'                    _n/a_               The literal % character.
     `%c'                    int                 A single character, represented as
                                                 an C int.
     `%d'                    int                 Exactly equivalent to
                                                 `printf("%d")'.
     `%u'                    unsigned int        Exactly equivalent to
                                                 `printf("%u")'.
     `%ld'                   long                Exactly equivalent to
                                                 `printf("%ld")'.
     `%lu'                   unsigned long       Exactly equivalent to
                                                 `printf("%lu")'.
     `%lld'                  long long           Exactly equivalent to
                                                 `printf("%lld")'.
     `%llu'                  unsigned long long  Exactly equivalent to
                                                 `printf("%llu")'.
     `%zd'                   Py_ssize_t          Exactly equivalent to
                                                 `printf("%zd")'.
     `%zu'                   size_t              Exactly equivalent to
                                                 `printf("%zu")'.
     `%i'                    int                 Exactly equivalent to
                                                 `printf("%i")'.
     `%x'                    int                 Exactly equivalent to
                                                 `printf("%x")'.
     `%s'                    char*               A null-terminated C character array.
     `%p'                    void*               The hex representation of a C
                                                 pointer. Mostly equivalent to
                                                 `printf("%p")' except that it is
                                                 guaranteed to start with the
                                                 literal `0x' regardless of what the
                                                 platform's `printf' yields.

     An unrecognized format character causes all the rest of the format
     string to be copied as-is to the result string, and any extra
     arguments discarded.

          Note: The `"%lld"' and `"%llu"' format specifiers are only
          available when `HAVE_LONG_LONG' is defined.

     Changed in version 2.7: Support for `"%lld"' and `"%llu"' added.

 -- Function: void PyErr_SetNone (PyObject *type)
     This is a shorthand for `PyErr_SetObject(type, Py_None)'.

 -- Function: int PyErr_BadArgument ()
     This is a shorthand for `PyErr_SetString(PyExc_TypeError,
     message)', where _message_ indicates that a built-in operation was
     invoked with an illegal argument.  It is mostly for internal use.

 -- Function: PyObject* PyErr_NoMemory ()
     Return value: Always NULL.

     This is a shorthand for `PyErr_SetNone(PyExc_MemoryError)'; it
     returns _NULL_ so an object allocation function can write `return
     PyErr_NoMemory();' when it runs out of memory.

 -- Function: PyObject* PyErr_SetFromErrno (PyObject *type)
     Return value: Always NULL.

     This is a convenience function to raise an exception when a C
     library function has returned an error and set the C variable
     `errno'.  It constructs a tuple object whose first item is the
     integer `errno' value and whose second item is the corresponding
     error message (gotten from `strerror()'), and then calls
     `PyErr_SetObject(type, object)'.  On Unix, when the `errno' value
     is `EINTR', indicating an interrupted system call, this calls
     PyErr_CheckSignals(), and if that set the error
     indicator, leaves it set to that.  The function always returns
     _NULL_, so a wrapper function around a system call can write
     `return PyErr_SetFromErrno(type);' when the system call returns an
     error.

 -- Function: PyObject* PyErr_SetFromErrnoWithFilename (PyObject *type,
          const char *filename)
     Return value: Always NULL.

     Similar to PyErr_SetFromErrno(), with the additional
     behavior that if _filename_ is not _NULL_, it is passed to the
     constructor of _type_ as a third parameter.  In the case of
     exceptions such as IOError and OSError,
     this is used to define the `filename' attribute of the exception
     instance.

 -- Function: PyObject* PyErr_SetFromWindowsErr (int ierr)
     Return value: Always NULL.

     This is a convenience function to raise WindowsError.
     If called with _ierr_ of `0', the error code returned by a call to
     `GetLastError()' is used instead.  It calls the Win32 function
     `FormatMessage()' to retrieve the Windows description of error
     code given by _ierr_ or `GetLastError()', then it constructs a
     tuple object whose first item is the _ierr_ value and whose second
     item is the corresponding error message (gotten from
     `FormatMessage()'), and then calls
     `PyErr_SetObject(PyExc_WindowsError, object)'. This function
     always returns _NULL_. Availability: Windows.

 -- Function: PyObject* PyErr_SetExcFromWindowsErr (PyObject *type,
          int ierr)
     Return value: Always NULL.

     Similar to PyErr_SetFromWindowsErr(), with an
     additional parameter specifying the exception type to be raised.
     Availability: Windows.

     New in version 2.3.

 -- Function: PyObject* PyErr_SetFromWindowsErrWithFilename (int ierr,
          const char *filename)
     Return value: Always NULL.

     Similar to PyErr_SetFromWindowsErr(), with the
     additional behavior that if _filename_ is not _NULL_, it is passed
     to the constructor of WindowsError as a third
     parameter. Availability: Windows.

 -- Function: PyObject* PyErr_SetExcFromWindowsErrWithFilename
          (PyObject *type, int ierr, char *filename)
     Return value: Always NULL.

     Similar to PyErr_SetFromWindowsErrWithFilename(), with
     an additional parameter specifying the exception type to be
     raised. Availability: Windows.

     New in version 2.3.

 -- Function: void PyErr_BadInternalCall ()
     This is a shorthand for `PyErr_SetString(PyExc_SystemError,
     message)', where _message_ indicates that an internal operation
     (e.g. a Python/C API function) was invoked with an illegal
     argument.  It is mostly for internal use.

 -- Function: int PyErr_WarnEx (PyObject *category, char *message,
          int stacklevel)
     Issue a warning message.  The _category_ argument is a warning
     category (see below) or _NULL_; the _message_ argument is a
     message string.  _stacklevel_ is a positive number giving a number
     of stack frames; the warning will be issued from the  currently
     executing line of code in that stack frame.  A _stacklevel_ of 1
     is the function calling PyErr_WarnEx(), 2 is  the
     function above that, and so forth.

     This function normally prints a warning message to _sys.stderr_;
     however, it is also possible that the user has specified that
     warnings are to be turned into errors, and in that case this will
     raise an exception.  It is also possible that the function raises
     an exception because of a problem with the warning machinery (the
     implementation imports the warnings module to do the
     heavy lifting).  The return value is `0' if no exception is
     raised, or `-1' if an exception is raised.  (It is not possible to
     determine whether a warning message is actually printed, nor what
     the reason is for the exception; this is intentional.)  If an
     exception is raised, the caller should do its normal exception
     handling (for example, see Py_DECREF() owned references
     and return an error value).

     Warning categories must be subclasses of `Warning'; the default
     warning category is `RuntimeWarning'.  The standard Python warning
     categories are available as global variables whose names are
     `PyExc_' followed by the Python exception name. These have the
     type PyObject*.; they are all class objects. Their
     names are `PyExc_Warning', `PyExc_UserWarning',
     `PyExc_UnicodeWarning', `PyExc_DeprecationWarning',
     `PyExc_SyntaxWarning', `PyExc_RuntimeWarning', and
     `PyExc_FutureWarning'.  `PyExc_Warning' is a subclass of
     `PyExc_Exception'; the other warning categories are subclasses of
     `PyExc_Warning'.

     For information about warning control, see the documentation for
     the warnings module and the -W option in
     the command line documentation.  There is no C API for warning
     control.

 -- Function: int PyErr_Warn (PyObject *category, char *message)
     Issue a warning message.  The _category_ argument is a warning
     category (see below) or _NULL_; the _message_ argument is a
     message string.  The warning will appear to be issued from the
     function calling PyErr_Warn(), equivalent to calling
     PyErr_WarnEx() with a _stacklevel_ of 1.

     Deprecated; use PyErr_WarnEx() instead.

 -- Function: int PyErr_WarnExplicit (PyObject *category, const
          char *message, const char *filename, int lineno, const
          char *module, PyObject *registry)
     Issue a warning message with explicit control over all warning
     attributes.  This is a straightforward wrapper around the Python
     function warnings.warn_explicit(), see there for more
     information.  The _module_ and _registry_ arguments may be set to
     _NULL_ to get the default effect described there.

 -- Function: int PyErr_WarnPy3k (char *message, int stacklevel)
     Issue a DeprecationWarning with the given _message_
     and _stacklevel_ if the `Py_Py3kWarningFlag' flag is enabled.

     New in version 2.6.

 -- Function: int PyErr_CheckSignals ()
     This function interacts with Python's signal handling.  It checks
     whether a signal has been sent to the processes and if so, invokes
     the corresponding signal handler.  If the signal.
     module is supported, this can invoke a signal handler written in
     Python.  In all cases, the default effect for `SIGINT' is to raise
     the  KeyboardInterrupt exception.  If an exception is
     raised the error indicator is set and the function returns `-1';
     otherwise the function returns `0'.  The error indicator may or
     may not be cleared if it was previously set.

 -- Function: void PyErr_SetInterrupt ()
     This function simulates the effect of a `SIGINT' signal arriving
     -- the next time PyErr_CheckSignals() is called,
     see KeyboardInterrupt will be raised.  It may be called
     without holding the interpreter lock.

 -- Function: int PySignal_SetWakeupFd (int fd)
     This utility function specifies a file descriptor to which a
     `'\0'' byte will be written whenever a signal is received.  It
     returns the previous such file descriptor.  The value `-1'
     disables the feature; this is the initial state.  This is
     equivalent to signal.set_wakeup_fd() in Python, but
     without any error checking.  _fd_ should be a valid file
     descriptor.  The function should only be called from the main
     thread.

 -- Function: PyObject* PyErr_NewException (char *name, PyObject *base,
          PyObject *dict)
     Return value: New reference.

     This utility function creates and returns a new exception object.
     The _name_ argument must be the name of the new exception, a C
     string of the form `module.class'.  The _base_ and _dict_
     arguments are normally _NULL_.  This creates a class object
     derived from Exception. (accessible in C as
     `PyExc_Exception').

     The `__module__' attribute of the new class is set to the first
     part (up to the last dot) of the _name_ argument, and the class
     name is set to the last part (after the last dot).  The _base_
     argument can be used to specify alternate base classes; it can
     either be only one class or a tuple of classes. The _dict_
     argument can be used to specify a dictionary of class variables
     and methods.

 -- Function: PyObject* PyErr_NewExceptionWithDoc (char *name,
          char *doc, PyObject *base, PyObject *dict)
     Return value: New reference.

     Same as PyErr_NewException(), except that the new
     exception class can easily be given a docstring: If _doc_ is
     non-_NULL_, it will be used as the docstring for the exception
     class.

     New in version 2.7.

 -- Function: void PyErr_WriteUnraisable (PyObject *obj)
     This utility function prints a warning message to `sys.stderr'
     when an exception has been set but it is impossible for the
     interpreter to actually raise the exception.  It is used, for
     example, when an exception occurs in an __del__().
     method.

     The function is called with a single argument _obj_ that
     identifies the context in which the unraisable exception occurred.
     The repr of _obj_ will be printed in the warning message.

7.4.1 Unicode Exception Objects
-------------------------------

The following functions are used to create and modify Unicode
exceptions from C.

 -- Function: PyObject* PyUnicodeDecodeError_Create (const
          char *encoding, const char *object, Py_ssize_t length,
          Py_ssize_t start, Py_ssize_t end, const char *reason)
     Create a `UnicodeDecodeError' object with the attributes
     _encoding_, _object_, _length_, _start_, _end_ and _reason_.

 -- Function: PyObject* PyUnicodeEncodeError_Create (const
          char *encoding, const Py_UNICODE *object, Py_ssize_t length,
          Py_ssize_t start, Py_ssize_t end, const char *reason)
     Create a `UnicodeEncodeError' object with the attributes
     _encoding_, _object_, _length_, _start_, _end_ and _reason_.

 -- Function: PyObject* PyUnicodeTranslateError_Create (const
          Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start,
          Py_ssize_t end, const char *reason)
     Create a `UnicodeTranslateError' object with the attributes
     _object_, _length_, _start_, _end_ and _reason_.

 -- Function: PyObject* PyUnicodeDecodeError_GetEncoding (PyObject *exc)
 -- Function: PyObject* PyUnicodeEncodeError_GetEncoding (PyObject *exc)
     Return the _encoding_ attribute of the given exception object.

 -- Function: PyObject* PyUnicodeDecodeError_GetObject (PyObject *exc)
 -- Function: PyObject* PyUnicodeEncodeError_GetObject (PyObject *exc)
 -- Function: PyObject* PyUnicodeTranslateError_GetObject
          (PyObject *exc)
     Return the _object_ attribute of the given exception object.

 -- Function: int PyUnicodeDecodeError_GetStart (PyObject *exc,
          Py_ssize_t *start)
 -- Function: int PyUnicodeEncodeError_GetStart (PyObject *exc,
          Py_ssize_t *start)
 -- Function: int PyUnicodeTranslateError_GetStart (PyObject *exc,
          Py_ssize_t *start)
     Get the _start_ attribute of the given exception object and place
     it into _*start_.  _start_ must not be _NULL_.  Return `0' on
     success, `-1' on failure.

 -- Function: int PyUnicodeDecodeError_SetStart (PyObject *exc,
          Py_ssize_t start)
 -- Function: int PyUnicodeEncodeError_SetStart (PyObject *exc,
          Py_ssize_t start)
 -- Function: int PyUnicodeTranslateError_SetStart (PyObject *exc,
          Py_ssize_t start)
     Set the _start_ attribute of the given exception object to
     _start_.  Return `0' on success, `-1' on failure.

 -- Function: int PyUnicodeDecodeError_GetEnd (PyObject *exc,
          Py_ssize_t *end)
 -- Function: int PyUnicodeEncodeError_GetEnd (PyObject *exc,
          Py_ssize_t *end)
 -- Function: int PyUnicodeTranslateError_GetEnd (PyObject *exc,
          Py_ssize_t *end)
     Get the _end_ attribute of the given exception object and place it
     into _*end_.  _end_ must not be _NULL_.  Return `0' on success,
     `-1' on failure.

 -- Function: int PyUnicodeDecodeError_SetEnd (PyObject *exc,
          Py_ssize_t end)
 -- Function: int PyUnicodeEncodeError_SetEnd (PyObject *exc,
          Py_ssize_t end)
 -- Function: int PyUnicodeTranslateError_SetEnd (PyObject *exc,
          Py_ssize_t end)
     Set the _end_ attribute of the given exception object to _end_.
     Return `0' on success, `-1' on failure.

 -- Function: PyObject* PyUnicodeDecodeError_GetReason (PyObject *exc)
 -- Function: PyObject* PyUnicodeEncodeError_GetReason (PyObject *exc)
 -- Function: PyObject* PyUnicodeTranslateError_GetReason
          (PyObject *exc)
     Return the _reason_ attribute of the given exception object.

 -- Function: int PyUnicodeDecodeError_SetReason (PyObject *exc, const
          char *reason)
 -- Function: int PyUnicodeEncodeError_SetReason (PyObject *exc, const
          char *reason)
 -- Function: int PyUnicodeTranslateError_SetReason (PyObject *exc,
          const char *reason)
     Set the _reason_ attribute of the given exception object to
     _reason_.  Return `0' on success, `-1' on failure.

7.4.2 Recursion Control
-----------------------

These two functions provide a way to perform safe recursive calls at
the C level, both in the core and in extension modules.  They are
needed if the recursive code does not necessarily invoke Python code
(which tracks its recursion depth automatically).

 -- Function: int Py_EnterRecursiveCall (char *where)
     Marks a point where a recursive C-level call is about to be
     performed.

     If `USE_STACKCHECK' is defined, this function checks if the the OS
     stack overflowed using PyOS_CheckStack().  In this is
     the case, it sets a MemoryError and returns a nonzero
     value.

     The function then checks if the recursion limit is reached.  If
     this is the case, a RuntimeError is set and a nonzero
     value is returned.  Otherwise, zero is returned.

     _where_ should be a string such as `" in instance check"' to be
     concatenated to the RuntimeError message caused by the
     recursion depth limit.

 -- Function: void Py_LeaveRecursiveCall ()
     Ends a Py_EnterRecursiveCall().  Must be called once
     for each _successful_ invocation of Py_EnterRecursiveCall().

7.4.3 Standard Exceptions
-------------------------

All standard Python exceptions are available as global variables whose
names are `PyExc_' followed by the Python exception name.  These have
the type PyObject*.; they are all class objects.  For
completeness, here are all the variables:

C Name                                   Python Name                      Notes
-----------------------------------------------------------------------------------------
`PyExc_BaseException'                    BaseException.        (1), (4)
`PyExc_Exception'                        Exception.            (1)
`PyExc_StandardError'                    StandardError.        (1)
`PyExc_ArithmeticError'                  ArithmeticError.      (1)
`PyExc_LookupError'                      LookupError.          (1)
`PyExc_AssertionError'                   AssertionError.
`PyExc_AttributeError'                   AttributeError.
`PyExc_EOFError'                         EOFError.
`PyExc_EnvironmentError'                 EnvironmentError.     (1)
`PyExc_FloatingPointError'               FloatingPointError.
`PyExc_IOError'                          IOError.
`PyExc_ImportError'                      ImportError.
`PyExc_IndexError'                       IndexError.
`PyExc_KeyError'                         KeyError.
`PyExc_KeyboardInterrupt'                KeyboardInterrupt.
`PyExc_MemoryError'                      MemoryError.
`PyExc_NameError'                        NameError.
`PyExc_NotImplementedError'              NotImplementedError.
`PyExc_OSError'                          OSError.
`PyExc_OverflowError'                    OverflowError.
`PyExc_ReferenceError'                   ReferenceError.       (2)
`PyExc_RuntimeError'                     RuntimeError.
`PyExc_SyntaxError'                      SyntaxError.
`PyExc_SystemError'                      SystemError.
`PyExc_SystemExit'                       SystemExit.
`PyExc_TypeError'                        TypeError.
`PyExc_ValueError'                       ValueError.
`PyExc_WindowsError'                     WindowsError.         (3)
`PyExc_ZeroDivisionError'                ZeroDivisionError.

  Notes:

  1. This is a base class for other standard exceptions.

  2. This is the same as weakref.ReferenceError.

  3. Only defined on Windows; protect code that uses this by testing
     that the preprocessor macro `MS_WINDOWS' is defined.

  4. New in version 2.5.

7.4.4 Deprecation of String Exceptions
--------------------------------------

All exceptions built into Python or provided in the standard library
are derived from BaseException.

  String exceptions are still supported in the interpreter to allow
existing code to run unmodified, but this will also change in a future
release.

7.5 Utilities
=============

The functions in this chapter perform various utility tasks, ranging
from helping C code be more portable across platforms, using Python
modules from C, and parsing function arguments and constructing Python
values from C values.

7.5.1 Operating System Utilities
--------------------------------

 -- Function: int Py_FdIsInteractive (FILE *fp, const char *filename)
     Return true (nonzero) if the standard I/O file _fp_ with name
     _filename_ is deemed interactive.  This is the case for files for
     which `isatty(fileno(fp))' is true.  If the global flag
     `Py_InteractiveFlag' is true, this function also returns true if
     the _filename_ pointer is _NULL_ or if the name is equal to one of
     the strings `'<stdin>'' or `'???''.

 -- Function: void PyOS_AfterFork ()
     Function to update some internal state after a process fork; this
     should be called in the new process if the Python interpreter will
     continue to be used.  If a new executable is loaded into the new
     process, this function does not need to be called.

 -- Function: int PyOS_CheckStack ()
     Return true when the interpreter runs out of stack space.  This is
     a reliable check, but is only available when `USE_STACKCHECK' is
     defined (currently on Windows using the Microsoft Visual C++
     compiler).  `USE_STACKCHECK' will be defined automatically; you
     should never change the definition in your own code.

 -- Function: PyOS_sighandler_t PyOS_getsig (int i)
     Return the current signal handler for signal _i_.  This is a thin
     wrapper around either `sigaction()' or `signal()'.  Do not call
     those functions directly! `PyOS_sighandler_t' is a typedef alias
     for `void (*)(int)'.

 -- Function: PyOS_sighandler_t PyOS_setsig (int i, PyOS_sighandler_t h)
     Set the signal handler for signal _i_ to be _h_; return the old
     signal handler.  This is a thin wrapper around either
     `sigaction()' or `signal()'.  Do not call those functions
     directly!  `PyOS_sighandler_t' is a typedef alias for `void
     (*)(int)'.

7.5.2 System Functions
----------------------

These are utility functions that make functionality from the sys
module accessible to C code.  They all work with the current
interpreter thread's sys module's dict, which is contained
in the internal thread state structure.

 -- Function: PyObject *PySys_GetObject (char *name)
     Return value: Borrowed reference.

     Return the object _name_ from the sys module or _NULL_
     if it does not exist, without setting an exception.

 -- Function: FILE *PySys_GetFile (char *name, FILE *def)
     Return the `FILE*' associated with the object _name_ in the
     sys module, or _def_ if _name_ is not in the module or is
     not associated with a `FILE*'.

 -- Function: int PySys_SetObject (char *name, PyObject *v)
     Set _name_ in the sys module to _v_ unless _v_ is
     _NULL_, in which case _name_ is deleted from the sys module.
     Returns `0' on success, `-1' on error.

 -- Function: void PySys_ResetWarnOptions ()
     Reset sys.warnoptions to an empty list.

 -- Function: void PySys_AddWarnOption (char *s)
     Append _s_ to sys.warnoptions.

 -- Function: void PySys_SetPath (char *path)
     Set sys.path to a list object of paths found in _path_
     which should be a list of paths separated with the platform's
     search path delimiter (`:' on Unix, `;' on Windows).

 -- Function: void PySys_WriteStdout (const char *format, ...)
     Write the output string described by _format_ to sys.stdout.
     No exceptions are raised, even if truncation occurs (see
     below).

     _format_ should limit the total size of the formatted output
     string to 1000 bytes or less - after 1000 bytes, the output string
     is truncated.  In particular, this means that no unrestricted "%s"
     formats should occur; these should be limited using "%.<N>s" where
     <N> is a decimal number calculated so that <N> plus the maximum
     size of other formatted text does not exceed 1000 bytes.  Also
     watch out for "%f", which can print hundreds of digits for very
     large numbers.

     If a problem occurs, or sys.stdout is unset, the
     formatted message is written to the real (C level) _stdout_.

 -- Function: void PySys_WriteStderr (const char *format, ...)
     As above, but write to sys.stderr or _stderr_ instead.

7.5.3 Process Control
---------------------

 -- Function: void Py_FatalError (const char *message)
     Print a fatal error message and kill the process.  No cleanup is
     performed.  This function should only be invoked when a condition
     is detected that would make it dangerous to continue using the
     Python interpreter; e.g., when the object administration appears
     to be corrupted.  On Unix, the standard C library function
     `abort()' is called which will attempt to produce a `core' file.

 -- Function: void Py_Exit (int status)
     Exit the current process.  This calls Py_Finalize().
     and then calls the standard C library function `exit(status)'.

 -- Function: int Py_AtExit (void (*func) ())
     Register a cleanup function to be called by Py_Finalize().
     The cleanup function will be called with no arguments and
     should return no value.  At most 32 cleanup functions can be
     registered.  When the registration is successful, see
     Py_AtExit() returns `0'; on failure, it returns `-1'.  The
     cleanup function registered last is called first. Each cleanup
     function will be called at most once.  Since Python's internal
     finalization will have completed before the cleanup function, no
     Python APIs should be called by _func_.

7.5.4 Importing Modules
-----------------------

 -- Function: PyObject* PyImport_ImportModule (const char *name)
     Return value: New reference.

     This is a simplified interface to PyImport_ImportModuleEx()
     below, leaving the _globals_ and _locals_ arguments set to
     _NULL_ and _level_ set to 0.  When the _name_ argument contains a
     dot (when it specifies a submodule of a package), the _fromlist_
     argument is set to the list `['*']' so that the return value is the
     named module rather than the top-level package containing it as
     would otherwise be the case.  (Unfortunately, this has an
     additional side effect when _name_ in fact specifies a subpackage
     instead of a submodule: the submodules specified in the package's
     `__all__' variable are  loaded.)  Return a new reference to the
     imported module, or _NULL_ with an exception set on failure.
     Before Python 2.4, the module may still be created in the failure
     case -- examine `sys.modules' to find out.  Starting with Python
     2.4, a failing import of a module no longer leaves the module in
     `sys.modules'.

     Changed in version 2.4: Failing imports remove incomplete module
     objects.

     Changed in version 2.6: Always uses absolute imports.

 -- Function: PyObject* PyImport_ImportModuleNoBlock (const char *name)
     This version of PyImport_ImportModule() does not
     block. It's intended to be used in C functions that import other
     modules to execute a function.  The import may block if another
     thread holds the import lock. The function
     PyImport_ImportModuleNoBlock() never blocks. It first tries
     to fetch the module from sys.modules and falls back to
     PyImport_ImportModule().  unless the lock is held, in which
     case the function will raise an ImportError.

     New in version 2.6.

 -- Function: PyObject* PyImport_ImportModuleEx (char *name,
          PyObject *globals, PyObject *locals, PyObject *fromlist)
     Return value: New reference.

     Import a module.  This is best described by referring to the
     built-in Python function __import__(), as the standard
     __import__() function calls this function directly.

     The return value is a new reference to the imported module or
     top-level package, or _NULL_ with an exception set on failure
     (before Python 2.4, the module may still be created in this case).
     Like for __import__(), the return value when a
     submodule of a package was requested is normally the top-level
     package, unless a non-empty _fromlist_ was given.

     Changed in version 2.4: Failing imports remove incomplete module
     objects.

     Changed in version 2.6: The function is an alias for
     PyImport_ImportModuleLevel() with -1 as level, meaning
     relative import.

 -- Function: PyObject* PyImport_ImportModuleLevel (char *name,
          PyObject *globals, PyObject *locals, PyObject *fromlist,
          int level)
     Return value: New reference.

     Import a module.  This is best described by referring to the
     built-in Python function __import__(), as the standard
     __import__() function calls this function directly.

     The return value is a new reference to the imported module or
     top-level package, or _NULL_ with an exception set on failure.
     Like for __import__(), the return value when a
     submodule of a package was requested is normally the top-level
     package, unless a non-empty _fromlist_ was given.

     New in version 2.5.

 -- Function: PyObject* PyImport_Import (PyObject *name)
     Return value: New reference.

     This is a higher-level interface that calls the current "import
     hook function".  It invokes the __import__() function
     from the `__builtins__' of the current globals.  This means that
     the import is done using whatever import hooks are installed in
     the current environment, e.g. by rexec or `ihooks'.

     Changed in version 2.6: Always uses absolute imports.

 -- Function: PyObject* PyImport_ReloadModule (PyObject *m)
     Return value: New reference.

     Reload a module.  This is best described by referring to the
     built-in Python function reload(), as the standard
     reload() function calls this function directly.
     Return a new reference to the reloaded module, or _NULL_ with an
     exception set on failure (the module still exists in this case).

 -- Function: PyObject* PyImport_AddModule (const char *name)
     Return value: Borrowed reference.

     Return the module object corresponding to a module name.  The
     _name_ argument may be of the form `package.module'. First check
     the modules dictionary if there's one there, and if not, create a
     new one and insert it in the modules dictionary. Return _NULL_
     with an exception set on failure.

          Note: This function does not load or import the module; if
          the module wasn't already loaded, you will get an empty
          module object. Use PyImport_ImportModule().  or
          one of its variants to import a module.  Package structures
          implied by a dotted name for _name_ are not created if not
          already present.

 -- Function: PyObject* PyImport_ExecCodeModule (char *name,
          PyObject *co)
     Return value: New reference.

     Given a module name (possibly of the form `package.module') and a
     code object read from a Python bytecode file or obtained from the
     built-in function compile(), load the module.  Return a
     new reference to the module object, or _NULL_ with an exception
     set if an error occurred.  Before Python 2.4, the module could
     still be created in error cases.  Starting with Python 2.4, _name_
     is removed from sys.modules in error cases, and even
     if _name_ was already in sys.modules on entry to
     PyImport_ExecCodeModule().  Leaving incompletely initialized
     modules in sys.modules is dangerous, as imports of
     such modules have no way to know that the module object is an
     unknown (and probably damaged with respect to the module author's
     intents) state.

     The module's `__file__' attribute will be set to the code object's
     `co_filename'.

     This function will reload the module if it was already imported.
     See PyImport_ReloadModule() for the intended way to
     reload a module.

     If _name_ points to a dotted name of the form `package.module',
     any package structures not already created will still not be
     created.

     Changed in version 2.4: _name_ is removed from sys.modules
     in error cases.

 -- Function: PyObject* PyImport_ExecCodeModuleEx (char *name,
          PyObject *co, char *pathname)
     Return value: New reference.

     Like PyImport_ExecCodeModule(), but the `__file__'
     attribute of the module object is set to _pathname_ if it is
     non-`NULL'.

 -- Function: long PyImport_GetMagicNumber ()
     Return the magic number for Python bytecode files (a.k.a. `.pyc'
     and `.pyo' files).  The magic number should be present in the
     first four bytes of the bytecode file, in little-endian byte order.

 -- Function: PyObject* PyImport_GetModuleDict ()
     Return value: Borrowed reference.

     Return the dictionary used for the module administration (a.k.a.
     `sys.modules').  Note that this is a per-interpreter variable.

 -- Function: PyObject* PyImport_GetImporter (PyObject *path)
     Return an importer object for a sys.path.
     /`pkg.__path__' item _path_, possibly by fetching it from the
     sys.path_importer_cache.  dict.  If it wasn't yet
     cached, traverse sys.path_hooks until a hook is found
     that can handle the path item.  Return `None' if no hook could;
     this tells our caller it should fall back to the built-in import
     mechanism.  Cache the result in sys.path_importer_cache.
     Return a new reference to the importer object.

     New in version 2.6.

 -- Function: void _PyImport_Init ()
     Initialize the import mechanism.  For internal use only.

 -- Function: void PyImport_Cleanup ()
     Empty the module table.  For internal use only.

 -- Function: void _PyImport_Fini ()
     Finalize the import mechanism.  For internal use only.

 -- Function: PyObject* _PyImport_FindExtension (char *, char *)
     For internal use only.

 -- Function: PyObject* _PyImport_FixupExtension (char *, char *)
     For internal use only.

 -- Function: int PyImport_ImportFrozenModule (char *name)
     Load a frozen module named _name_.  Return `1' for success, `0' if
     the module is not found, and `-1' with an exception set if the
     initialization failed.  To access the imported module on a
     successful load, use PyImport_ImportModule().  (Note
     the misnomer -- this function would reload the module if it was
     already imported.)

 -- type: struct _frozen
     This is the structure type definition for frozen module
     descriptors, as generated by the *freeze* utility (see
     `Tools/freeze/' in the Python source distribution).  Its
     definition, found in `Include/import.h', is:

         struct _frozen {
             char *name;
             unsigned char *code;
             int size;
         };

 -- var: struct _frozen* PyImport_FrozenModules
     This pointer is initialized to point to an array of `struct
     _frozen' records, terminated by one whose members are all _NULL_
     or zero.  When a frozen module is imported, it is searched in this
     table.  Third-party code could play tricks with this to provide a
     dynamically created collection of frozen modules.

 -- Function: int PyImport_AppendInittab (const char *name,
          void (*initfunc)(void))
     Add a single module to the existing table of built-in modules.
     This is a convenience wrapper around
     PyImport_ExtendInittab(), returning `-1' if the table could
     not be extended.  The new module can be imported by the name
     _name_, and uses the function _initfunc_ as the initialization
     function called on the first attempted import.  This should be
     called before Py_Initialize().

 -- type: struct _inittab
     Structure describing a single entry in the list of built-in
     modules.  Each of these structures gives the name and
     initialization function for a module built into the interpreter.
     Programs which embed Python may use an array of these structures
     in conjunction with PyImport_ExtendInittab() to
     provide additional built-in modules.  The structure is defined in
     `Include/import.h' as:

         struct _inittab {
             char *name;
             void (*initfunc)(void);
         };

 -- Function: int PyImport_ExtendInittab (struct _inittab *newtab)
     Add a collection of modules to the table of built-in modules.  The
     _newtab_ array must end with a sentinel entry which contains
     _NULL_ for the `name' field; failure to provide the sentinel value
     can result in a memory fault.  Returns `0' on success or `-1' if
     insufficient memory could be allocated to extend the internal
     table.  In the event of failure, no modules are added to the
     internal table.  This should be called before
     Py_Initialize().

7.5.5 Data marshalling support
------------------------------

These routines allow C code to work with serialized objects using the
same data format as the marshal module.  There are
functions to write data into the serialization format, and additional
functions that can be used to read the data back.  Files used to store
marshalled data must be opened in binary mode.

  Numeric values are stored with the least significant byte first.

  The module supports two versions of the data format: version 0 is the
historical version, version 1 (new in Python 2.4) shares interned
strings in the file, and upon unmarshalling.  Version 2 (new in Python
2.5) uses a binary format for floating point numbers.
_Py_MARSHAL_VERSION_ indicates the current file format (currently 2).

 -- Function: void PyMarshal_WriteLongToFile (long value, FILE *file,
          int version)
     Marshal a `long' integer, _value_, to _file_.  This will only write
     the least-significant 32 bits of _value_; regardless of the size
     of the native `long' type.

     Changed in version 2.4: _version_ indicates the file format.

 -- Function: void PyMarshal_WriteObjectToFile (PyObject *value,
          FILE *file, int version)
     Marshal a Python object, _value_, to _file_.

     Changed in version 2.4: _version_ indicates the file format.

 -- Function: PyObject* PyMarshal_WriteObjectToString (PyObject *value,
          int version)
     Return value: New reference.

     Return a string object containing the marshalled representation of
     _value_.

     Changed in version 2.4: _version_ indicates the file format.

  The following functions allow marshalled values to be read back in.

  XXX What about error detection?  It appears that reading past the end
of the file will always result in a negative numeric value (where
that's relevant), but it's not clear that negative values won't be
handled properly when there's no error.  What's the right way to tell?
Should only non-negative values be written using these routines?

 -- Function: long PyMarshal_ReadLongFromFile (FILE *file)
     Return a C `long' from the data stream in a `FILE*' opened for
     reading.  Only a 32-bit value can be read in using this function,
     regardless of the native size of `long'.

 -- Function: int PyMarshal_ReadShortFromFile (FILE *file)
     Return a C `short' from the data stream in a `FILE*' opened for
     reading.  Only a 16-bit value can be read in using this function,
     regardless of the native size of `short'.

 -- Function: PyObject* PyMarshal_ReadObjectFromFile (FILE *file)
     Return value: New reference.

     Return a Python object from the data stream in a `FILE*' opened for
     reading.  On error, sets the appropriate exception (see
     EOFError or TypeError) and returns _NULL_.

 -- Function: PyObject* PyMarshal_ReadLastObjectFromFile (FILE *file)
     Return value: New reference.

     Return a Python object from the data stream in a `FILE*' opened for
     reading.  Unlike PyMarshal_ReadObjectFromFile(), this
     function assumes that no further objects will be read from the
     file, allowing it to aggressively load file data into memory so
     that the de-serialization can operate from data in memory rather
     than reading a byte at a time from the file.  Only use these
     variant if you are certain that you won't be reading anything else
     from the file.  On error, sets the appropriate exception (see
     EOFError or TypeError) and returns _NULL_.

 -- Function: PyObject* PyMarshal_ReadObjectFromString (char *string,
          Py_ssize_t len)
     Return value: New reference.

     Return a Python object from the data stream in a character buffer
     containing _len_ bytes pointed to by _string_.  On error, sets the
     appropriate exception (see EOFError or TypeError)
     and returns _NULL_.

     Changed in version 2.5: This function used an `int' type for
     _len_. This might require changes in your code for properly
     supporting 64-bit systems.

7.5.6 Parsing arguments and building values
-------------------------------------------

These functions are useful when creating your own extensions functions
and methods.  Additional information and examples are available in
Extending and Embedding the Python Interpreter.

  The first three of these functions described, see
PyArg_ParseTuple(), PyArg_ParseTupleAndKeywords(), and
PyArg_Parse(), all use _format strings_ which are used to
tell the function about the expected arguments.  The format strings use
the same syntax for each of these functions.

  A format string consists of zero or more "format units."  A format
unit describes one Python object; it is usually a single character or a
parenthesized sequence of format units.  With a few exceptions, a
format unit that is not a parenthesized sequence normally corresponds
to a single address argument to these functions.  In the following
description, the quoted form is the format unit; the entry in (round)
parentheses is the Python object type that matches the format unit; and
the entry in [square] brackets is the type of the C variable(s) whose
address should be passed.

`s' (string or Unicode) [const char *]
     Convert a Python string or Unicode object to a C pointer to a
     character string.  You must not provide storage for the string
     itself; a pointer to an existing string is stored into the
     character pointer variable whose address you pass.  The C string
     is NUL-terminated.  The Python string must not contain embedded
     NUL bytes; if it does, a TypeError exception is
     raised. Unicode objects are converted to C strings using the
     default encoding.  If this conversion fails, a UnicodeError
     is raised.

`s#' (string, Unicode or any read buffer compatible object) [const char *, int (or `Py_ssize_t', see below)]
     This variant on `s' stores into two C variables, the first one a
     pointer to a character string, the second one its length.  In this
     case the Python string may contain embedded null bytes.  Unicode
     objects pass back a pointer to the default encoded string version
     of the object if such a conversion is possible.  All other
     read-buffer compatible objects pass back a reference to the raw
     internal data representation.

     Starting with Python 2.5 the type of the length argument can be
     controlled by defining the macro `PY_SSIZE_T_CLEAN' before
     including `Python.h'.  If the macro is defined, length is a
     `Py_ssize_t' rather than an int.

`s*' (string, Unicode, or any buffer compatible object) [Py_buffer]
     Similar to `s#', this code fills a Py_buffer structure provided by
     the caller.  The buffer gets locked, so that the caller can
     subsequently use the buffer even inside a `Py_BEGIN_ALLOW_THREADS'
     block; the caller is responsible for calling `PyBuffer_Release'
     with the structure after it has processed the data.

     New in version 2.6.

`z' (string, Unicode  or `None') [const char *]
     Like `s', but the Python object may also be `None', in which case
     the C pointer is set to _NULL_.

`z#' (string, Unicode, `None' or any read buffer compatible object) [const char *, int]
     This is to `s#' as `z' is to `s'.

`z*' (string, Unicode, `None' or any buffer compatible object) [Py_buffer]
     This is to `s*' as `z' is to `s'.

     New in version 2.6.

`u' (Unicode) [Py_UNICODE *]
     Convert a Python Unicode object to a C pointer to a NUL-terminated
     buffer of 16-bit Unicode (UTF-16) data.  As with `s', there is no
     need to provide storage for the Unicode data buffer; a pointer to
     the existing Unicode data is stored into the Py_UNICODE
     pointer variable whose address you pass.

`u#' (Unicode) [Py_UNICODE *, int]
     This variant on `u' stores into two C variables, the first one a
     pointer to a Unicode data buffer, the second one its length.
     Non-Unicode objects are handled by interpreting their read-buffer
     pointer as pointer to a Py_UNICODE array.

`es' (string, Unicode or character buffer compatible object) [const char *encoding, char **buffer]
     This variant on `s' is used for encoding Unicode and objects
     convertible to Unicode into a character buffer. It only works for
     encoded data without embedded NUL bytes.

     This format requires two arguments.  The first is only used as
     input, and must be a `const char*' which points to the name of an
     encoding as a NUL-terminated string, or _NULL_, in which case the
     default encoding is used.  An exception is raised if the named
     encoding is not known to Python.  The second argument must be a
     `char**'; the value of the pointer it references will be set to a
     buffer with the contents of the argument text.  The text will be
     encoded in the encoding specified by the first argument.

     PyArg_ParseTuple() will allocate a buffer of the
     needed size, copy the encoded data into this buffer and adjust
     _*buffer_ to reference the newly allocated storage.  The caller is
     responsible for calling PyMem_Free() to free the
     allocated buffer after use.

`et' (string, Unicode or character buffer compatible object) [const char *encoding, char **buffer]
     Same as `es' except that 8-bit string objects are passed through
     without recoding them.  Instead, the implementation assumes that
     the string object uses the encoding passed in as parameter.

`es#' (string, Unicode or character buffer compatible object) [const char *encoding, char **buffer, int *buffer_length]
     This variant on `s#' is used for encoding Unicode and objects
     convertible to Unicode into a character buffer.  Unlike the `es'
     format, this variant allows input data which contains NUL
     characters.

     It requires three arguments.  The first is only used as input, and
     must be a `const char*' which points to the name of an encoding as
     a NUL-terminated string, or _NULL_, in which case the default
     encoding is used.  An exception is raised if the named encoding is
     not known to Python.  The second argument must be a `char**'; the
     value of the pointer it references will be set to a buffer with
     the contents of the argument text.  The text will be encoded in
     the encoding specified by the first argument.  The third argument
     must be a pointer to an integer; the referenced integer will be
     set to the number of bytes in the output buffer.

     There are two modes of operation:

     If _*buffer_ points a _NULL_ pointer, the function will allocate a
     buffer of the needed size, copy the encoded data into this buffer
     and set _*buffer_ to reference the newly allocated storage.  The
     caller is responsible for calling PyMem_Free() to free
     the allocated buffer after usage.

     If _*buffer_ points to a non-_NULL_ pointer (an already allocated
     buffer), see PyArg_ParseTuple() will use this location as
     the buffer and interpret the initial value of _*buffer_length_ as
     the buffer size.  It will then copy the encoded data into the
     buffer and NUL-terminate it.  If the buffer is not large enough, a
     ValueError will be set.

     In both cases, _*buffer_length_ is set to the length of the
     encoded data without the trailing NUL byte.

`et#' (string, Unicode or character buffer compatible object) [const char *encoding, char **buffer, int *buffer_length]
     Same as `es#' except that string objects are passed through without
     recoding them. Instead, the implementation assumes that the string
     object uses the encoding passed in as parameter.

`b' (integer) [unsigned char]
     Convert a nonnegative Python integer to an unsigned tiny int,
     stored in a C `unsigned char'.

`B' (integer) [unsigned char]
     Convert a Python integer to a tiny int without overflow checking,
     stored in a C `unsigned char'.

     New in version 2.3.

`h' (integer) [short int]
     Convert a Python integer to a C `short int'.

`H' (integer) [unsigned short int]
     Convert a Python integer to a C `unsigned short int', without
     overflow checking.

     New in version 2.3.

`i' (integer) [int]
     Convert a Python integer to a plain C `int'.

`I' (integer) [unsigned int]
     Convert a Python integer to a C `unsigned int', without overflow
     checking.

     New in version 2.3.

`l' (integer) [long int]
     Convert a Python integer to a C `long int'.

`k' (integer) [unsigned long]
     Convert a Python integer or long integer to a C `unsigned long'
     without overflow checking.

     New in version 2.3.

`L' (integer) [PY_LONG_LONG]
     Convert a Python integer to a C `long long'.  This format is only
     available on platforms that support `long long' (or `_int64' on
     Windows).

`K' (integer) [unsigned PY_LONG_LONG]
     Convert a Python integer or long integer to a C `unsigned long
     long' without overflow checking.  This format is only available on
     platforms that support `unsigned long long' (or `unsigned _int64'
     on Windows).

     New in version 2.3.

`n' (integer) [Py_ssize_t]
     Convert a Python integer or long integer to a C `Py_ssize_t'.

     New in version 2.5.

`c' (string of length 1) [char]
     Convert a Python character, represented as a string of length 1,
     to a C `char'.

`f' (float) [float]
     Convert a Python floating point number to a C `float'.

`d' (float) [double]
     Convert a Python floating point number to a C `double'.

`D' (complex) [Py_complex]
     Convert a Python complex number to a C Py_complex.
     structure.

`O' (object) [PyObject *]
     Store a Python object (without any conversion) in a C object
     pointer.  The C program thus receives the actual object that was
     passed.  The object's reference count is not increased.  The
     pointer stored is not _NULL_.

`O!' (object) [_typeobject_, PyObject *]
     Store a Python object in a C object pointer.  This is similar to
     `O', but takes two C arguments: the first is the address of a
     Python type object, the second is the address of the C variable
     (of type PyObject*)  into which the object pointer is
     stored.  If the Python object does not have the required type,
     see TypeError is raised.

`O&' (object) [_converter_, _anything_]
     Convert a Python object to a C variable through a _converter_
     function.  This takes two arguments: the first is a function, the
     second is the address of a C variable (of arbitrary type),
     converted to `void *'.  The _converter_ function in turn is called
     as follows:

         status = converter(object, address);

     where _object_ is the Python object to be converted and _address_
     is the `void*' argument that was passed to the
     PyArg_Parse*().  function.  The returned _status_ should be
     `1' for a successful conversion and `0' if the conversion has
     failed.  When the conversion fails, the _converter_ function
     should raise an exception and leave the content of _address_
     unmodified.

`S' (string) [PyStringObject *]
     Like `O' but requires that the Python object is a string object.
     Raises TypeError if the object is not a string object.
     The C variable may also be declared as PyObject*.

`U' (Unicode string) [PyUnicodeObject *]
     Like `O' but requires that the Python object is a Unicode object.
     Raises TypeError if the object is not a Unicode
     object.  The C variable may also be declared as PyObject*.

`t#' (read-only character buffer) [char *, int]
     Like `s#', but accepts any object which implements the read-only
     buffer interface.  The `char*' variable is set to point to the
     first byte of the buffer, and the `int' is set to the length of
     the buffer.  Only single-segment buffer objects are accepted;
     see TypeError is raised for all others.

`w' (read-write character buffer) [char *]
     Similar to `s', but accepts any object which implements the
     read-write buffer interface.  The caller must determine the length
     of the buffer by other means, or use `w#' instead.  Only
     single-segment buffer objects are accepted; see TypeError.
     is raised for all others.

`w#' (read-write character buffer) [char *, Py_ssize_t]
     Like `s#', but accepts any object which implements the read-write
     buffer interface.  The `char *' variable is set to point to the
     first byte of the buffer, and the `Py_ssize_t' is set to the
     length of the buffer.  Only single-segment buffer objects are
     accepted; see TypeError.  is raised for all others.

`w*' (read-write byte-oriented buffer) [Py_buffer]
     This is to `w' what `s*' is to `s'.

     New in version 2.6.

`(items)' (tuple) [_matching-items_]
     The object must be a Python sequence whose length is the number of
     format units in _items_.  The C arguments must correspond to the
     individual format units in _items_.  Format units for sequences
     may be nested.

          Note: Prior to Python version 1.5.2, this format specifier
          only accepted a tuple containing the individual parameters,
          not an arbitrary sequence.  Code which previously caused
          TypeError to be raised here may now proceed
          without an exception.  This is not expected to be a problem
          for existing code.

  It is possible to pass Python long integers where integers are
requested; however no proper range checking is done -- the most
significant bits are silently truncated when the receiving field is too
small to receive the value (actually, the semantics are inherited from
downcasts in C -- your mileage may vary).

  A few other characters have a meaning in a format string.  These may
not occur inside nested parentheses.  They are:

`|'
     Indicates that the remaining arguments in the Python argument list
     are optional.  The C variables corresponding to optional arguments
     should be initialized to their default value -- when an optional
     argument is not specified, see PyArg_ParseTuple() does not
     touch the contents of the corresponding C variable(s).

`:'
     The list of format units ends here; the string after the colon is
     used as the function name in error messages (the "associated
     value" of the exception that PyArg_ParseTuple().
     raises).

`;'
     The list of format units ends here; the string after the semicolon
     is used as the error message _instead_ of the default error
     message.  `:' and `;' mutually exclude each other.

  Note that any Python object references which are provided to the
caller are _borrowed_ references; do not decrement their reference
count!

  Additional arguments passed to these functions must be addresses of
variables whose type is determined by the format string; these are used
to store values from the input tuple.  There are a few cases, as
described in the list of format units above, where these parameters are
used as input values; they should match what is specified for the
corresponding format unit in that case.

  For the conversion to succeed, the _arg_ object must match the format
and the format must be exhausted.  On success, the
PyArg_Parse*() functions return true, otherwise they return
false and raise an appropriate exception.  When the
PyArg_Parse*() functions fail due to conversion failure in one
of the format units, the variables at the addresses corresponding to
that and the following format units are left untouched.

 -- Function: int PyArg_ParseTuple (PyObject *args, const char *format,
          ...)
     Parse the parameters of a function that takes only positional
     parameters into local variables.  Returns true on success; on
     failure, it returns false and raises the appropriate exception.

 -- Function: int PyArg_VaParse (PyObject *args, const char *format,
          va_list vargs)
     Identical to PyArg_ParseTuple(), except that it accepts
     a va_list rather than a variable number of arguments.

 -- Function: int PyArg_ParseTupleAndKeywords (PyObject *args,
          PyObject *kw, const char *format, char *keywords[], ...)
     Parse the parameters of a function that takes both positional and
     keyword parameters into local variables.  Returns true on success;
     on failure, it returns false and raises the appropriate exception.

 -- Function: int PyArg_VaParseTupleAndKeywords (PyObject *args,
          PyObject *kw, const char *format, char *keywords[],
          va_list vargs)
     Identical to PyArg_ParseTupleAndKeywords(), except that
     it accepts a va_list rather than a variable number of arguments.

 -- Function: int PyArg_Parse (PyObject *args, const char *format, ...)
     Function used to deconstruct the argument lists of "old-style"
     functions -- these are functions which use the METH_OLDARGS
     parameter parsing method.  This is not recommended for use in
     parameter parsing in new code, and most code in the standard
     interpreter has been modified to no longer use this for that
     purpose.  It does remain a convenient way to decompose other
     tuples, however, and may continue to be used for that purpose.

 -- Function: int PyArg_UnpackTuple (PyObject *args, const char *name,
          Py_ssize_t min, Py_ssize_t max, ...)
     A simpler form of parameter retrieval which does not use a format
     string to specify the types of the arguments.  Functions which use
     this method to retrieve their parameters should be declared as
     METH_VARARGS in function or method tables.  The tuple
     containing the actual parameters should be passed as _args_; it
     must actually be a tuple.  The length of the tuple must be at
     least _min_ and no more than _max_; _min_ and _max_ may be equal.
     Additional arguments must be passed to the function, each of which
     should be a pointer to a PyObject* variable; these
     will be filled in with the values from _args_; they will contain
     borrowed references.  The variables which correspond to optional
     parameters not given by _args_ will not be filled in; these should
     be initialized by the caller. This function returns true on
     success and false if _args_ is not a tuple or contains the wrong
     number of elements; an exception will be set if there was a
     failure.

     This is an example of the use of this function, taken from the
     sources for the `_weakref' helper module for weak references:

         static PyObject *
         weakref_ref(PyObject *self, PyObject *args)
         {
             PyObject *object;
             PyObject *callback = NULL;
             PyObject *result = NULL;

             if (PyArg_UnpackTuple(args, "ref", 1, 2, &object, &callback)) {
                 result = PyWeakref_NewRef(object, callback);
             }
             return result;
         }

     The call to PyArg_UnpackTuple() in this example is
     entirely equivalent to this call to PyArg_ParseTuple().:

         PyArg_ParseTuple(args, "O|O:ref", &object, &callback)

     New in version 2.2.

     Changed in version 2.5: This function used an `int' type for _min_
     and _max_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* Py_BuildValue (const char *format, ...)
     Return value: New reference.

     Create a new value based on a format string similar to those
     accepted by the PyArg_Parse*() family of functions
     and a sequence of values.  Returns the value or _NULL_ in the case
     of an error; an exception will be raised if _NULL_ is returned.

     Py_BuildValue() does not always build a tuple.  It
     builds a tuple only if its format string contains two or more
     format units.  If the format string is empty, it returns `None';
     if it contains exactly one format unit, it returns whatever object
     is described by that format unit.  To force it to return a tuple
     of size 0 or one, parenthesize the format string.

     When memory buffers are passed as parameters to supply data to
     build objects, as for the `s' and `s#' formats, the required data
     is copied.  Buffers provided by the caller are never referenced by
     the objects created by Py_BuildValue().  In other
     words, if your code invokes `malloc()' and passes the allocated
     memory to Py_BuildValue(), your code is responsible for
     calling `free()' for that memory once Py_BuildValue().
     returns.

     In the following description, the quoted form is the format unit;
     the entry in (round) parentheses is the Python object type that
     the format unit will return; and the entry in [square] brackets is
     the type of the C value(s) to be passed.

     The characters space, tab, colon and comma are ignored in format
     strings (but not within format units such as `s#').  This can be
     used to make long format strings a tad more readable.

    `s' (string) [char *]
          Convert a null-terminated C string to a Python object.  If
          the C string pointer is _NULL_, `None' is used.

    `s#' (string) [char *, int]
          Convert a C string and its length to a Python object.  If the
          C string pointer is _NULL_, the length is ignored and `None'
          is returned.

    `z' (string or `None') [char *]
          Same as `s'.

    `z#' (string or `None') [char *, int]
          Same as `s#'.

    `u' (Unicode string) [Py_UNICODE *]
          Convert a null-terminated buffer of Unicode (UCS-2 or UCS-4)
          data to a Python Unicode object.  If the Unicode buffer
          pointer is _NULL_, `None' is returned.

    `u#' (Unicode string) [Py_UNICODE *, int]
          Convert a Unicode (UCS-2 or UCS-4) data buffer and its length
          to a Python Unicode object.   If the Unicode buffer pointer
          is _NULL_, the length is ignored and `None' is returned.

    `i' (integer) [int]
          Convert a plain C `int' to a Python integer object.

    `b' (integer) [char]
          Convert a plain C `char' to a Python integer object.

    `h' (integer) [short int]
          Convert a plain C `short int' to a Python integer object.

    `l' (integer) [long int]
          Convert a C `long int' to a Python integer object.

    `B' (integer) [unsigned char]
          Convert a C `unsigned char' to a Python integer object.

    `H' (integer) [unsigned short int]
          Convert a C `unsigned short int' to a Python integer object.

    `I' (integer/long) [unsigned int]
          Convert a C `unsigned int' to a Python integer object or a
          Python long integer object, if it is larger than `sys.maxint'.

    `k' (integer/long) [unsigned long]
          Convert a C `unsigned long' to a Python integer object or a
          Python long integer object, if it is larger than `sys.maxint'.

    `L' (long) [PY_LONG_LONG]
          Convert a C `long long' to a Python long integer object. Only
          available on platforms that support `long long'.

    `K' (long) [unsigned PY_LONG_LONG]
          Convert a C `unsigned long long' to a Python long integer
          object.  Only available on platforms that support `unsigned
          long long'.

    `n' (int) [Py_ssize_t]
          Convert a C `Py_ssize_t' to a Python integer or long integer.

          New in version 2.5.

    `c' (string of length 1) [char]
          Convert a C `int' representing a character to a Python string
          of length 1.

    `d' (float) [double]
          Convert a C `double' to a Python floating point number.

    `f' (float) [float]
          Same as `d'.

    `D' (complex) [Py_complex *]
          Convert a C Py_complex structure to a Python
          complex number.

    `O' (object) [PyObject *]
          Pass a Python object untouched (except for its reference
          count, which is incremented by one).  If the object passed in
          is a _NULL_ pointer, it is assumed that this was caused
          because the call producing the argument found an error and
          set an exception. Therefore, see Py_BuildValue().
          will return _NULL_ but won't raise an exception.  If no
          exception has been raised yet, see SystemError is set.

    `S' (object) [PyObject *]
          Same as `O'.

    `N' (object) [PyObject *]
          Same as `O', except it doesn't increment the reference count
          on the object.  Useful when the object is created by a call
          to an object constructor in the argument list.

    `O&' (object) [_converter_, _anything_]
          Convert _anything_ to a Python object through a _converter_
          function.  The function is called with _anything_ (which
          should be compatible with `void *') as its argument and
          should return a "new" Python object, or _NULL_ if an error
          occurred.

    `(items)' (tuple) [_matching-items_]
          Convert a sequence of C values to a Python tuple with the
          same number of items.

    `[items]' (list) [_matching-items_]
          Convert a sequence of C values to a Python list with the same
          number of items.

    `{items}' (dictionary) [_matching-items_]
          Convert a sequence of C values to a Python dictionary.  Each
          pair of consecutive C values adds one item to the dictionary,
          serving as key and value, respectively.

     If there is an error in the format string, the SystemError
     exception is set and _NULL_ returned.

 -- Function: PyObject* Py_VaBuildValue (const char *format,
          va_list vargs)
     Identical to Py_BuildValue(), except that it accepts a
     va_list rather than a variable number of arguments.

7.5.7 String conversion and formatting
--------------------------------------

Functions for number conversion and formatted string output.

 -- Function: int PyOS_snprintf (char *str, size_t size, const
          char *format, ...)
     Output not more than _size_ bytes to _str_ according to the format
     string _format_ and the extra arguments. See the Unix man page
     `snprintf(2)'.

 -- Function: int PyOS_vsnprintf (char *str, size_t size, const
          char *format, va_list va)
     Output not more than _size_ bytes to _str_ according to the format
     string _format_ and the variable argument list _va_. Unix man page
     `vsnprintf(2)'.

  PyOS_snprintf() and PyOS_vsnprintf() wrap the
Standard C library functions `snprintf()' and `vsnprintf()'. Their
purpose is to guarantee consistent behavior in corner cases, which the
Standard C functions do not.

  The wrappers ensure that _str*[*size_-1] is always `'\0'' upon
return. They never write more than _size_ bytes (including the trailing
`'\0'' into str.  Both functions require that `str != NULL', `size > 0'
and `format != NULL'.

  If the platform doesn't have `vsnprintf()' and the buffer size needed
to avoid truncation exceeds _size_ by more than 512 bytes, Python
aborts with a _Py_FatalError_.

  The return value (_rv_) for these functions should be interpreted as
follows:

   * When `0 <= rv < size', the output conversion was successful and
     _rv_ characters were written to _str_ (excluding the trailing
     `'\0'' byte at _str*[*rv_]).

   * When `rv >= size', the output conversion was truncated and a
     buffer with `rv + 1' bytes would have been needed to succeed.
     _str*[*size_-1] is `'\0'' in this case.

   * When `rv < 0', "something bad happened." _str*[*size_-1] is `'\0''
     in this case too, but the rest of _str_ is undefined. The exact
     cause of the error depends on the underlying platform.

  The following functions provide locale-independent string to number
conversions.

 -- Function: double PyOS_string_to_double (const char *s,
          char **endptr, PyObject *overflow_exception)
     Convert a string `s' to a `double', raising a Python exception on
     failure.  The set of accepted strings corresponds to the set of
     strings accepted by Python's float() constructor,
     except that `s' must not have leading or trailing whitespace.  The
     conversion is independent of the current locale.

     If `endptr' is `NULL', convert the whole string.  Raise ValueError
     and return `-1.0' if the string is not a valid representation of a
     floating-point number.

     If endptr is not `NULL', convert as much of the string as possible
     and set `*endptr' to point to the first unconverted character.  If
     no initial segment of the string is the valid representation of a
     floating-point number, set `*endptr' to point to the beginning of
     the string, raise ValueError, and return `-1.0'.

     If `s' represents a value that is too large to store in a float
     (for example, `"1e500"' is such a string on many platforms) then
     if `overflow_exception' is `NULL' return `Py_HUGE_VAL' (with an
     appropriate sign) and don't set any exception.  Otherwise,
     `overflow_exception' must point to a Python exception object;
     raise that exception and return `-1.0'.  In both cases, set
     `*endptr' to point to the first character after the converted
     value.

     If any other error occurs during the conversion (for example an
     out-of-memory error), set the appropriate Python exception and
     return `-1.0'.

     New in version 2.7.

 -- Function: double PyOS_ascii_strtod (const char *nptr, char **endptr)
     Convert a string to a `double'. This function behaves like the
     Standard C function `strtod()' does in the C locale. It does this
     without changing the current locale, since that would not be
     thread-safe.

     PyOS_ascii_strtod() should typically be used for
     reading configuration files or other non-user input that should be
     locale independent.

     See the Unix man page `strtod(2)' for details.

     New in version 2.4.

     Deprecated since version 2.7: Use PyOS_string_to_double()
     instead.

 -- Function: char* PyOS_ascii_formatd (char *buffer, size_t buf_len,
          const char *format, double d)
     Convert a `double' to a string using the `'.'' as the decimal
     separator. _format_ is a `printf()'-style format string specifying
     the number format. Allowed conversion characters are `'e'', `'E'',
     `'f'', `'F'', `'g'' and `'G''.

     The return value is a pointer to _buffer_ with the converted
     string or NULL if the conversion failed.

     New in version 2.4.

     Deprecated since version 2.7: This function is removed in Python
     2.7 and 3.1.  Use `PyOS_double_to_string()' instead.

 -- Function: char* PyOS_double_to_string (double val,
          char format_code, int precision, int flags, int *ptype)
     Convert a `double' _val_ to a string using supplied _format_code_,
     _precision_, and _flags_.

     _format_code_ must be one of `'e'', `'E'', `'f'', `'F'', `'g'',
     `'G'' or `'r''.  For `'r'', the supplied _precision_ must be 0 and
     is ignored.  The `'r'' format code specifies the standard
     repr() format.

     _flags_ can be zero or more of the values _Py_DTSF_SIGN_,
     _Py_DTSF_ADD_DOT_0_, or _Py_DTSF_ALT_, or-ed together:

        * _Py_DTSF_SIGN_ means to always precede the returned string
          with a sign character, even if _val_ is non-negative.

        * _Py_DTSF_ADD_DOT_0_ means to ensure that the returned string
          will not look like an integer.

        * _Py_DTSF_ALT_ means to apply "alternate" formatting rules.
          See the documentation for the PyOS_snprintf().
          `'#'' specifier for details.

     If _ptype_ is non-NULL, then the value it points to will be set to
     one of _Py_DTST_FINITE_, _Py_DTST_INFINITE_, or _Py_DTST_NAN_,
     signifying that _val_ is a finite number, an infinite number, or
     not a number, respectively.

     The return value is a pointer to _buffer_ with the converted
     string or _NULL_ if the conversion failed. The caller is
     responsible for freeing the returned string by calling
     PyMem_Free().

     New in version 2.7.

 -- Function: double PyOS_ascii_atof (const char *nptr)
     Convert a string to a `double' in a locale-independent way.

     See the Unix man page `atof(2)' for details.

     New in version 2.4.

     Deprecated since version 3.1: Use PyOS_string_to_double()
     instead.

 -- Function: char* PyOS_stricmp (char *s1, char *s2)
     Case insensitive comparison of strings. The function works almost
     identically to `strcmp()' except that it ignores the case.

     New in version 2.6.

 -- Function: char* PyOS_strnicmp (char *s1, char *s2, Py_ssize_t  size)
     Case insensitive comparison of strings. The function works almost
     identically to `strncmp()' except that it ignores the case.

     New in version 2.6.

7.5.8 Reflection
----------------

 -- Function: PyObject* PyEval_GetBuiltins ()
     Return value: Borrowed reference.

     Return a dictionary of the builtins in the current execution frame,
     or the interpreter of the thread state if no frame is currently
     executing.

 -- Function: PyObject* PyEval_GetLocals ()
     Return value: Borrowed reference.

     Return a dictionary of the local variables in the current
     execution frame, or _NULL_ if no frame is currently executing.

 -- Function: PyObject* PyEval_GetGlobals ()
     Return value: Borrowed reference.

     Return a dictionary of the global variables in the current
     execution frame, or _NULL_ if no frame is currently executing.

 -- Function: PyFrameObject* PyEval_GetFrame ()
     Return value: Borrowed reference.

     Return the current thread state's frame, which is _NULL_ if no
     frame is currently executing.

 -- Function: int PyFrame_GetLineNumber (PyFrameObject *frame)
     Return the line number that _frame_ is currently executing.

 -- Function: int PyEval_GetRestricted ()
     If there is a current frame and it is executing in restricted
     mode, return true, otherwise false.

 -- Function: const char* PyEval_GetFuncName (PyObject *func)
     Return the name of _func_ if it is a function, class or instance
     object, else the name of _func_s type.

 -- Function: const char* PyEval_GetFuncDesc (PyObject *func)
     Return a description string, depending on the type of _func_.
     Return values include "()" for functions and methods, "
     constructor", " instance", and " object".  Concatenated with the
     result of PyEval_GetFuncName(), the result will be a
     description of _func_.

7.5.9 Codec registry and support functions
------------------------------------------

 -- Function: int PyCodec_Register (PyObject *search_function)
     Register a new codec search function.

     As side effect, this tries to load the `encodings' package, if not
     yet done, to make sure that it is always first in the list of
     search functions.

 -- Function: int PyCodec_KnownEncoding (const char *encoding)
     Return `1' or `0' depending on whether there is a registered codec
     for the given _encoding_.

 -- Function: PyObject* PyCodec_Encode (PyObject *object, const
          char *encoding, const char *errors)
     Generic codec based encoding API.

     _object_ is passed through the encoder function found for the given
     _encoding_ using the error handling method defined by _errors_.
     _errors_ may be _NULL_ to use the default method defined for the
     codec.  Raises a LookupError if no encoder can be
     found.

 -- Function: PyObject* PyCodec_Decode (PyObject *object, const
          char *encoding, const char *errors)
     Generic codec based decoding API.

     _object_ is passed through the decoder function found for the given
     _encoding_ using the error handling method defined by _errors_.
     _errors_ may be _NULL_ to use the default method defined for the
     codec.  Raises a LookupError if no encoder can be
     found.

7.5.9.1 Codec lookup API
........................

In the following functions, the _encoding_ string is looked up
converted to all lower-case characters, which makes encodings looked up
through this mechanism effectively case-insensitive.  If no codec is
found, a KeyError is set and _NULL_ returned.

 -- Function: PyObject* PyCodec_Encoder (const char *encoding)
     Get an encoder function for the given _encoding_.

 -- Function: PyObject* PyCodec_Decoder (const char *encoding)
     Get a decoder function for the given _encoding_.

 -- Function: PyObject* PyCodec_IncrementalEncoder (const
          char *encoding, const char *errors)
     Get an `IncrementalEncoder' object for the given _encoding_.

 -- Function: PyObject* PyCodec_IncrementalDecoder (const
          char *encoding, const char *errors)
     Get an `IncrementalDecoder' object for the given _encoding_.

 -- Function: PyObject* PyCodec_StreamReader (const char *encoding,
          PyObject *stream, const char *errors)
     Get a `StreamReader' factory function for the given _encoding_.

 -- Function: PyObject* PyCodec_StreamWriter (const char *encoding,
          PyObject *stream, const char *errors)
     Get a `StreamWriter' factory function for the given _encoding_.

7.5.9.2 Registry API for Unicode encoding error handlers
........................................................

 -- Function: int PyCodec_RegisterError (const char *name,
          PyObject *error)
     Register the error handling callback function _error_ under the
     given _name_.  This callback function will be called by a codec
     when it encounters unencodable characters/undecodable bytes and
     _name_ is specified as the error parameter in the call to the
     encode/decode function.

     The callback gets a single argument, an instance of
     UnicodeEncodeError, UnicodeDecodeError or
     UnicodeTranslateError that holds information about the
     problematic sequence of characters or bytes and their offset in
     the original string (see Unicode Exception Objects.
     for functions to extract this information).  The callback must
     either raise the given exception, or return a two-item tuple
     containing the replacement for the problematic sequence, and an
     integer giving the offset in the original string at which
     encoding/decoding should be resumed.

     Return `0' on success, `-1' on error.

 -- Function: PyObject* PyCodec_LookupError (const char *name)
     Lookup the error handling callback function registered under
     _name_.  As a special case _NULL_ can be passed, in which case the
     error handling callback for "strict" will be returned.

 -- Function: PyObject* PyCodec_StrictErrors (PyObject *exc)
     Raise _exc_ as an exception.

 -- Function: PyObject* PyCodec_IgnoreErrors (PyObject *exc)
     Ignore the unicode error, skipping the faulty input.

 -- Function: PyObject* PyCodec_ReplaceErrors (PyObject *exc)
     Replace the unicode encode error with `?' or `U+FFFD'.

 -- Function: PyObject* PyCodec_XMLCharRefReplaceErrors (PyObject *exc)
     Replace the unicode encode error with XML character references.

 -- Function: PyObject* PyCodec_BackslashReplaceErrors (PyObject *exc)
     Replace the unicode encode error with backslash escapes (`\x',
     `\u' and `\U').

7.6 Abstract Objects Layer
==========================

The functions in this chapter interact with Python objects regardless
of their type, or with wide classes of object types (e.g. all numerical
types, or all sequence types).  When used on object types for which
they do not apply, they will raise a Python exception.

  It is not possible to use these functions on objects that are not
properly initialized, such as a list object that has been created by
PyList_New(), but whose items have not been set to some
non-`NULL' value yet.

7.6.1 Object Protocol
---------------------

 -- Function: int PyObject_Print (PyObject *o, FILE *fp, int flags)
     Print an object _o_, on file _fp_.  Returns `-1' on error.  The
     flags argument is used to enable certain printing options.  The
     only option currently supported is `Py_PRINT_RAW'; if given, the
     str() of the object is written instead of the
     repr().

 -- Function: int PyObject_HasAttr (PyObject *o, PyObject *attr_name)
     Returns `1' if _o_ has the attribute _attr_name_, and `0'
     otherwise.  This is equivalent to the Python expression
     `hasattr(o, attr_name)'.  This function always succeeds.

 -- Function: int PyObject_HasAttrString (PyObject *o, const
          char *attr_name)
     Returns `1' if _o_ has the attribute _attr_name_, and `0'
     otherwise.  This is equivalent to the Python expression
     `hasattr(o, attr_name)'.  This function always succeeds.

 -- Function: PyObject* PyObject_GetAttr (PyObject *o,
          PyObject *attr_name)
     Return value: New reference.

     Retrieve an attribute named _attr_name_ from object _o_. Returns
     the attribute value on success, or _NULL_ on failure.  This is the
     equivalent of the Python expression `o.attr_name'.

 -- Function: PyObject* PyObject_GetAttrString (PyObject *o, const
          char *attr_name)
     Return value: New reference.

     Retrieve an attribute named _attr_name_ from object _o_. Returns
     the attribute value on success, or _NULL_ on failure. This is the
     equivalent of the Python expression `o.attr_name'.

 -- Function: PyObject* PyObject_GenericGetAttr (PyObject *o,
          PyObject *name)
     Generic attribute getter function that is meant to be put into a
     type object's `tp_getattro' slot.  It looks for a descriptor in
     the dictionary of classes in the object's MRO as well as an
     attribute in the object's `__dict__' (if present).  As outlined in
     Implementing Descriptors, data descriptors take
     preference over instance attributes, while non-data descriptors
     don't.  Otherwise, an AttributeError is raised.

 -- Function: int PyObject_SetAttr (PyObject *o, PyObject *attr_name,
          PyObject *v)
     Set the value of the attribute named _attr_name_, for object _o_,
     to the value _v_. Returns `-1' on failure.  This is the equivalent
     of the Python statement `o.attr_name = v'.

 -- Function: int PyObject_SetAttrString (PyObject *o, const
          char *attr_name, PyObject *v)
     Set the value of the attribute named _attr_name_, for object _o_,
     to the value _v_. Returns `-1' on failure.  This is the equivalent
     of the Python statement `o.attr_name = v'.

 -- Function: int PyObject_GenericSetAttr (PyObject *o, PyObject *name,
          PyObject *value)
     Generic attribute setter function that is meant to be put into a
     type object's `tp_setattro' slot.  It looks for a data descriptor
     in the dictionary of classes in the object's MRO, and if found it
     takes preference over setting the attribute in the instance
     dictionary. Otherwise, the attribute is set in the object's
     `__dict__' (if present).  Otherwise, an AttributeError.
     is raised and `-1' is returned.

 -- Function: int PyObject_DelAttr (PyObject *o, PyObject *attr_name)
     Delete attribute named _attr_name_, for object _o_. Returns `-1'
     on failure.  This is the equivalent of the Python statement `del
     o.attr_name'.

 -- Function: int PyObject_DelAttrString (PyObject *o, const
          char *attr_name)
     Delete attribute named _attr_name_, for object _o_. Returns `-1'
     on failure.  This is the equivalent of the Python statement `del
     o.attr_name'.

 -- Function: PyObject* PyObject_RichCompare (PyObject *o1,
          PyObject *o2, int opid)
     Return value: New reference.

     Compare the values of _o1_ and _o2_ using the operation specified
     by _opid_, which must be one of `Py_LT', `Py_LE', `Py_EQ',
     `Py_NE', `Py_GT', or `Py_GE', corresponding to `<', `<=', `==',
     `!=', `>', or `>=' respectively. This is the equivalent of the
     Python expression `o1 op o2', where `op' is the operator
     corresponding to _opid_. Returns the value of the comparison on
     success, or _NULL_ on failure.

 -- Function: int PyObject_RichCompareBool (PyObject *o1, PyObject *o2,
          int opid)
     Compare the values of _o1_ and _o2_ using the operation specified
     by _opid_, which must be one of `Py_LT', `Py_LE', `Py_EQ',
     `Py_NE', `Py_GT', or `Py_GE', corresponding to `<', `<=', `==',
     `!=', `>', or `>=' respectively. Returns `-1' on error, `0' if the
     result is false, `1' otherwise. This is the equivalent of the
     Python expression `o1 op o2', where `op' is the operator
     corresponding to _opid_.

 -- Function: int PyObject_Cmp (PyObject *o1, PyObject *o2, int *result)
     Compare the values of _o1_ and _o2_ using a routine provided by
     _o1_, if one exists, otherwise with a routine provided by _o2_.
     The result of the comparison is returned in _result_.  Returns
     `-1' on failure.  This is the equivalent of the Python statement
     `result = cmp(o1, o2)'.

 -- Function: int PyObject_Compare (PyObject *o1, PyObject *o2)
     Compare the values of _o1_ and _o2_ using a routine provided by
     _o1_, if one exists, otherwise with a routine provided by _o2_.
     Returns the result of the comparison on success.  On error, the
     value returned is undefined; use PyErr_Occurred() to
     detect an error.  This is equivalent to the Python expression
     `cmp(o1, o2)'.

 -- Function: PyObject* PyObject_Repr (PyObject *o)
     Return value: New reference.

     Compute a string representation of object _o_.  Returns the string
     representation on success, _NULL_ on failure.  This is the
     equivalent of the Python expression `repr(o)'.  Called by the
     repr() built-in function and by reverse quotes.

 -- Function: PyObject* PyObject_Str (PyObject *o)
     Return value: New reference.

     Compute a string representation of object _o_.  Returns the string
     representation on success, _NULL_ on failure.  This is the
     equivalent of the Python expression `str(o)'.  Called by the
     str() built-in function and by the print.
     statement.

 -- Function: PyObject* PyObject_Bytes (PyObject *o)
     Compute a bytes representation of object _o_.  In 2.x, this is
     just a alias for PyObject_Str().

 -- Function: PyObject* PyObject_Unicode (PyObject *o)
     Return value: New reference.

     Compute a Unicode string representation of object _o_.  Returns
     the Unicode string representation on success, _NULL_ on failure.
     This is the equivalent of the Python expression `unicode(o)'.
     Called by the unicode() built-in function.

 -- Function: int PyObject_IsInstance (PyObject *inst, PyObject *cls)
     Returns `1' if _inst_ is an instance of the class _cls_ or a
     subclass of _cls_, or `0' if not.  On error, returns `-1' and sets
     an exception.  If _cls_ is a type object rather than a class
     object, see PyObject_IsInstance().  returns `1' if _inst_
     is of type _cls_.  If _cls_ is a tuple, the check will be done
     against every entry in _cls_. The result will be `1' when at least
     one of the checks returns `1', otherwise it will be `0'. If _inst_
     is not a class instance and _cls_ is neither a type object, nor a
     class object, nor a tuple, _inst_ must have a `__class__'
     attribute -- the class relationship of the value of that attribute
     with _cls_ will be used to determine the result of this function.

     New in version 2.1.

     Changed in version 2.2: Support for a tuple as the second argument
     added.

  Subclass determination is done in a fairly straightforward way, but
includes a wrinkle that implementors of extensions to the class system
may want to be aware of.  If `A' and `B' are class objects, `B' is a
subclass of `A' if it inherits from `A' either directly or indirectly.
If either is not a class object, a more general mechanism is used to
determine the class relationship of the two objects.  When testing if
_B_ is a subclass of _A_, if _A_ is _B_, see PyObject_IsSubclass()
returns true.  If _A_ and _B_ are different objects, _B_'s
`__bases__' attribute is searched in a depth-first fashion for _A_ --
the presence of the `__bases__' attribute is considered sufficient for
this determination.

 -- Function: int PyObject_IsSubclass (PyObject *derived, PyObject *cls)
     Returns `1' if the class _derived_ is identical to or derived from
     the class _cls_, otherwise returns `0'.  In case of an error,
     returns `-1'. If _cls_ is a tuple, the check will be done against
     every entry in _cls_. The result will be `1' when at least one of
     the checks returns `1', otherwise it will be `0'. If either
     _derived_ or _cls_ is not an actual class object (or tuple), this
     function uses the generic algorithm described above.

     New in version 2.1.

     Changed in version 2.3: Older versions of Python did not support a
     tuple as the second argument.

 -- Function: int PyCallable_Check (PyObject *o)
     Determine if the object _o_ is callable.  Return `1' if the object
     is callable and `0' otherwise.  This function always succeeds.

 -- Function: PyObject* PyObject_Call (PyObject *callable_object,
          PyObject *args, PyObject *kw)
     Return value: New reference.

     Call a callable Python object _callable_object_, with arguments
     given by the tuple _args_, and named arguments given by the
     dictionary _kw_. If no named arguments are needed, _kw_ may be
     _NULL_. _args_ must not be _NULL_, use an empty tuple if no
     arguments are needed. Returns the result of the call on success,
     or _NULL_ on failure.  This is the equivalent of the Python
     expression `apply(callable_object, args, kw)' or
     `callable_object(*args, **kw)'.

     New in version 2.2.

 -- Function: PyObject* PyObject_CallObject (PyObject *callable_object,
          PyObject *args)
     Return value: New reference.

     Call a callable Python object _callable_object_, with arguments
     given by the tuple _args_.  If no arguments are needed, then
     _args_ may be _NULL_.  Returns the result of the call on success,
     or _NULL_ on failure.  This is the equivalent of the Python
     expression `apply(callable_object, args)' or
     `callable_object(*args)'.

 -- Function: PyObject* PyObject_CallFunction (PyObject *callable,
          char *format, ...)
     Return value: New reference.

     Call a callable Python object _callable_, with a variable number
     of C arguments.  The C arguments are described using a
     Py_BuildValue() style format string.  The format may be
     _NULL_, indicating that no arguments are provided.  Returns the
     result of the call on success, or _NULL_ on failure.  This is the
     equivalent of the Python expression `apply(callable, args)' or
     `callable(*args)'. Note that if you only pass PyObject *
     args, see PyObject_CallFunctionObjArgs() is a faster
     alternative.

 -- Function: PyObject* PyObject_CallMethod (PyObject *o, char *method,
          char *format, ...)
     Return value: New reference.

     Call the method named _method_ of object _o_ with a variable
     number of C arguments.  The C arguments are described by a
     Py_BuildValue() format string that should  produce a tuple.
     The format may be _NULL_, indicating that no arguments are
     provided. Returns the result of the call on success, or _NULL_ on
     failure.  This is the equivalent of the Python expression
     `o.method(args)'.  Note that if you only pass PyObject *
     args, see PyObject_CallMethodObjArgs() is a faster
     alternative.

 -- Function: PyObject* PyObject_CallFunctionObjArgs
          (PyObject *callable, ..., NULL)
     Return value: New reference.

     Call a callable Python object _callable_, with a variable number of
     PyObject* arguments.  The arguments are provided as a
     variable number of parameters followed by _NULL_. Returns the
     result of the call on success, or _NULL_ on failure.

     New in version 2.2.

 -- Function: PyObject* PyObject_CallMethodObjArgs (PyObject *o,
          PyObject *name, ..., NULL)
     Return value: New reference.

     Calls a method of the object _o_, where the name of the method is
     given as a Python string object in _name_.  It is called with a
     variable number of PyObject* arguments.  The arguments
     are provided as a variable number of parameters followed by
     _NULL_. Returns the result of the call on success, or _NULL_ on
     failure.

     New in version 2.2.

 -- Function: long PyObject_Hash (PyObject *o)
     Compute and return the hash value of an object _o_.  On failure,
     return `-1'.  This is the equivalent of the Python expression
     `hash(o)'.

 -- Function: long PyObject_HashNotImplemented (PyObject *o)
     Set a TypeError indicating that `type(o)' is not
     hashable and return `-1'.  This function receives special
     treatment when stored in a `tp_hash' slot, allowing a type to
     explicitly indicate to the interpreter that it is not hashable.

     New in version 2.6.

 -- Function: int PyObject_IsTrue (PyObject *o)
     Returns `1' if the object _o_ is considered to be true, and `0'
     otherwise.  This is equivalent to the Python expression `not not
     o'.  On failure, return `-1'.

 -- Function: int PyObject_Not (PyObject *o)
     Returns `0' if the object _o_ is considered to be true, and `1'
     otherwise.  This is equivalent to the Python expression `not o'.
     On failure, return `-1'.

 -- Function: PyObject* PyObject_Type (PyObject *o)
     Return value: New reference.

     When _o_ is non-_NULL_, returns a type object corresponding to the
     object type of object _o_. On failure, raises SystemError
     and returns _NULL_.  This is equivalent to the Python
     expression `type(o)'. This function increments the reference count
     of the return value. There's really no reason to use this function
     instead of the common expression `o->ob_type', which returns a
     pointer of type PyTypeObject*, except when the
     incremented reference count is needed.

 -- Function: int PyObject_TypeCheck (PyObject *o, PyTypeObject *type)
     Return true if the object _o_ is of type _type_ or a subtype of
     _type_.  Both parameters must be non-_NULL_.

     New in version 2.2.

 -- Function: Py_ssize_t PyObject_Length (PyObject *o)
 -- Function: Py_ssize_t PyObject_Size (PyObject *o)
     Return the length of object _o_.  If the object _o_ provides
     either the sequence and mapping protocols, the sequence length is
     returned.  On error, `-1' is returned.  This is the equivalent to
     the Python expression `len(o)'.

     Changed in version 2.5: These functions returned an `int' type.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: PyObject* PyObject_GetItem (PyObject *o, PyObject *key)
     Return value: New reference.

     Return element of _o_ corresponding to the object _key_ or _NULL_
     on failure.  This is the equivalent of the Python expression
     `o[key]'.

 -- Function: int PyObject_SetItem (PyObject *o, PyObject *key,
          PyObject *v)
     Map the object _key_ to the value _v_.  Returns `-1' on failure.
     This is the equivalent of the Python statement `o[key] = v'.

 -- Function: int PyObject_DelItem (PyObject *o, PyObject *key)
     Delete the mapping for _key_ from _o_.  Returns `-1' on failure.
     This is the equivalent of the Python statement `del o[key]'.

 -- Function: int PyObject_AsFileDescriptor (PyObject *o)
     Derives a file descriptor from a Python object.  If the object is
     an integer or long integer, its value is returned.  If not, the
     object's `fileno()' method is called if it exists; the method must
     return an integer or long integer, which is returned as the file
     descriptor value.  Returns `-1' on failure.

 -- Function: PyObject* PyObject_Dir (PyObject *o)
     Return value: New reference.

     This is equivalent to the Python expression `dir(o)', returning a
     (possibly empty) list of strings appropriate for the object
     argument, or _NULL_ if there was an error.  If the argument is
     _NULL_, this is like the Python `dir()', returning the names of
     the current locals; in this case, if no execution frame is active
     then _NULL_ is returned but PyErr_Occurred() will
     return false.

 -- Function: PyObject* PyObject_GetIter (PyObject *o)
     Return value: New reference.

     This is equivalent to the Python expression `iter(o)'. It returns
     a new iterator for the object argument, or the object  itself if
     the object is already an iterator.  Raises TypeError.
     and returns _NULL_ if the object cannot be iterated.

7.6.2 Number Protocol
---------------------

 -- Function: int PyNumber_Check (PyObject *o)
     Returns `1' if the object _o_ provides numeric protocols, and
     false otherwise.  This function always succeeds.

 -- Function: PyObject* PyNumber_Add (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of adding _o1_ and _o2_, or _NULL_ on failure.
     This is the equivalent of the Python expression `o1 + o2'.

 -- Function: PyObject* PyNumber_Subtract (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of subtracting _o2_ from _o1_, or _NULL_ on
     failure.  This is the equivalent of the Python expression `o1 -
     o2'.

 -- Function: PyObject* PyNumber_Multiply (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of multiplying _o1_ and _o2_, or _NULL_ on
     failure.  This is the equivalent of the Python expression `o1 *
     o2'.

 -- Function: PyObject* PyNumber_Divide (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of dividing _o1_ by _o2_, or _NULL_ on failure.
     This is the equivalent of the Python expression `o1 / o2'.

 -- Function: PyObject* PyNumber_FloorDivide (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Return the floor of _o1_ divided by _o2_, or _NULL_ on failure.
     This is equivalent to the "classic" division of integers.

     New in version 2.2.

 -- Function: PyObject* PyNumber_TrueDivide (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Return a reasonable approximation for the mathematical value of
     _o1_ divided by _o2_, or _NULL_ on failure.  The return value is
     "approximate" because binary floating point numbers are
     approximate; it is not possible to represent all real numbers in
     base two.  This function can return a floating point value when
     passed two integers.

     New in version 2.2.

 -- Function: PyObject* PyNumber_Remainder (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the remainder of dividing _o1_ by _o2_, or _NULL_ on
     failure.  This is the equivalent of the Python expression `o1 %
     o2'.

 -- Function: PyObject* PyNumber_Divmod (PyObject *o1, PyObject *o2)
     Return value: New reference.

     See the built-in function divmod(). Returns _NULL_ on
     failure.  This is the equivalent of the Python expression
     `divmod(o1, o2)'.

 -- Function: PyObject* PyNumber_Power (PyObject *o1, PyObject *o2,
          PyObject *o3)
     Return value: New reference.

     See the built-in function pow(). Returns _NULL_ on
     failure.  This is the equivalent of the Python expression `pow(o1,
     o2, o3)', where _o3_ is optional.  If _o3_ is to be ignored, pass
     Py_None in its place (passing _NULL_ for _o3_ would
     cause an illegal memory access).

 -- Function: PyObject* PyNumber_Negative (PyObject *o)
     Return value: New reference.

     Returns the negation of _o_ on success, or _NULL_ on failure. This
     is the equivalent of the Python expression `-o'.

 -- Function: PyObject* PyNumber_Positive (PyObject *o)
     Return value: New reference.

     Returns _o_ on success, or _NULL_ on failure.  This is the
     equivalent of the Python expression `+o'.

 -- Function: PyObject* PyNumber_Absolute (PyObject *o)
     Return value: New reference.

     Returns the absolute value of _o_, or _NULL_ on failure.  This is
     the equivalent of the Python expression `abs(o)'.

 -- Function: PyObject* PyNumber_Invert (PyObject *o)
     Return value: New reference.

     Returns the bitwise negation of _o_ on success, or _NULL_ on
     failure.  This is the equivalent of the Python expression `~o'.

 -- Function: PyObject* PyNumber_Lshift (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of left shifting _o1_ by _o2_ on success, or
     _NULL_ on failure.  This is the equivalent of the Python
     expression `o1 << o2'.

 -- Function: PyObject* PyNumber_Rshift (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of right shifting _o1_ by _o2_ on success, or
     _NULL_ on failure.  This is the equivalent of the Python
     expression `o1 >> o2'.

 -- Function: PyObject* PyNumber_And (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the "bitwise and" of _o1_ and _o2_ on success and _NULL_
     on failure.  This is the equivalent of the Python expression `o1 &
     o2'.

 -- Function: PyObject* PyNumber_Xor (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the "bitwise exclusive or" of _o1_ by _o2_ on success, or
     _NULL_ on failure.  This is the equivalent of the Python
     expression `o1 ^ o2'.

 -- Function: PyObject* PyNumber_Or (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the "bitwise or" of _o1_ and _o2_ on success, or _NULL_ on
     failure.  This is the equivalent of the Python expression `o1 |
     o2'.

 -- Function: PyObject* PyNumber_InPlaceAdd (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the result of adding _o1_ and _o2_, or _NULL_ on failure.
     The operation is done _in-place_ when _o1_ supports it.  This is
     the equivalent of the Python statement `o1 += o2'.

 -- Function: PyObject* PyNumber_InPlaceSubtract (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the result of subtracting _o2_ from _o1_, or _NULL_ on
     failure.  The operation is done _in-place_ when _o1_ supports it.
     This is the equivalent of the Python statement `o1 -= o2'.

 -- Function: PyObject* PyNumber_InPlaceMultiply (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the result of multiplying _o1_ and _o2_, or _NULL_ on
     failure.  The operation is done _in-place_ when _o1_ supports it.
     This is the equivalent of the Python statement `o1 *= o2'.

 -- Function: PyObject* PyNumber_InPlaceDivide (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the result of dividing _o1_ by _o2_, or _NULL_ on failure.
     The operation is done _in-place_ when _o1_ supports it. This is
     the equivalent of the Python statement `o1 /= o2'.

 -- Function: PyObject* PyNumber_InPlaceFloorDivide (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the mathematical floor of dividing _o1_ by _o2_, or _NULL_
     on failure.  The operation is done _in-place_ when _o1_ supports
     it.  This is the equivalent of the Python statement `o1 //= o2'.

     New in version 2.2.

 -- Function: PyObject* PyNumber_InPlaceTrueDivide (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Return a reasonable approximation for the mathematical value of
     _o1_ divided by _o2_, or _NULL_ on failure.  The return value is
     "approximate" because binary floating point numbers are
     approximate; it is not possible to represent all real numbers in
     base two.  This function can return a floating point value when
     passed two integers.  The operation is done _in-place_ when _o1_
     supports it.

     New in version 2.2.

 -- Function: PyObject* PyNumber_InPlaceRemainder (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the remainder of dividing _o1_ by _o2_, or _NULL_ on
     failure.  The operation is done _in-place_ when _o1_ supports it.
     This is the equivalent of the Python statement `o1 %= o2'.

 -- Function: PyObject* PyNumber_InPlacePower (PyObject *o1,
          PyObject *o2, PyObject *o3)
     Return value: New reference.

     See the built-in function pow(). Returns _NULL_ on
     failure.  The operation is done _in-place_ when _o1_ supports it.
     This is the equivalent of the Python statement `o1 **= o2' when o3
     is Py_None, or an in-place variant of `pow(o1, o2,
     o3)' otherwise. If _o3_ is to be ignored, pass Py_None.
     in its place (passing _NULL_ for _o3_ would cause an illegal
     memory access).

 -- Function: PyObject* PyNumber_InPlaceLshift (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the result of left shifting _o1_ by _o2_ on success, or
     _NULL_ on failure.  The operation is done _in-place_ when _o1_
     supports it.  This is the equivalent of the Python statement `o1
     <<= o2'.

 -- Function: PyObject* PyNumber_InPlaceRshift (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Returns the result of right shifting _o1_ by _o2_ on success, or
     _NULL_ on failure.  The operation is done _in-place_ when _o1_
     supports it.  This is the equivalent of the Python statement `o1
     >>= o2'.

 -- Function: PyObject* PyNumber_InPlaceAnd (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the "bitwise and" of _o1_ and _o2_ on success and _NULL_
     on failure. The operation is done _in-place_ when _o1_ supports
     it.  This is the equivalent of the Python statement `o1 &= o2'.

 -- Function: PyObject* PyNumber_InPlaceXor (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the "bitwise exclusive or" of _o1_ by _o2_ on success, or
     _NULL_ on failure.  The operation is done _in-place_ when _o1_
     supports it.  This is the equivalent of the Python statement `o1
     ^= o2'.

 -- Function: PyObject* PyNumber_InPlaceOr (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Returns the "bitwise or" of _o1_ and _o2_ on success, or _NULL_ on
     failure.  The operation is done _in-place_ when _o1_ supports it.
     This is the equivalent of the Python statement `o1 |= o2'.

 -- Function: int PyNumber_Coerce (PyObject **p1, PyObject **p2)
     This function takes the addresses of two variables of type
     PyObject*.  If the objects pointed to by `*p1' and `*p2' have
     the same type, increment their reference count and return `0'
     (success). If the objects can be converted to a common numeric
     type, replace `*p1' and `*p2' by their converted value (with 'new'
     reference counts), and return `0'. If no conversion is possible,
     or if some other error occurs, return `-1' (failure) and don't
     increment the reference counts.  The call `PyNumber_Coerce(&o1,
     &o2)' is equivalent to the Python statement `o1, o2 = coerce(o1,
     o2)'.

 -- Function: int PyNumber_CoerceEx (PyObject **p1, PyObject **p2)
     This function is similar to PyNumber_Coerce(), except
     that it returns `1' when the conversion is not possible and when
     no error is raised.  Reference counts are still not increased in
     this case.

 -- Function: PyObject* PyNumber_Int (PyObject *o)
     Return value: New reference.

     Returns the _o_ converted to an integer object on success, or
     _NULL_ on failure.  If the argument is outside the integer range a
     long object will be returned instead. This is the equivalent of
     the Python expression `int(o)'.

 -- Function: PyObject* PyNumber_Long (PyObject *o)
     Return value: New reference.

     Returns the _o_ converted to a long integer object on success, or
     _NULL_ on failure.  This is the equivalent of the Python
     expression `long(o)'.

 -- Function: PyObject* PyNumber_Float (PyObject *o)
     Return value: New reference.

     Returns the _o_ converted to a float object on success, or _NULL_
     on failure.  This is the equivalent of the Python expression
     `float(o)'.

 -- Function: PyObject* PyNumber_Index (PyObject *o)
     Returns the _o_ converted to a Python int or long on success or
     _NULL_ with a TypeError exception raised on failure.

     New in version 2.5.

 -- Function: PyObject* PyNumber_ToBase (PyObject *n, int base)
     Returns the integer _n_ converted to _base_ as a string with a base
     marker of `'0b'', `'0o'', or `'0x'' if applicable.  When _base_ is
     not 2, 8, 10, or 16, the format is `'x#num'' where x is the base.
     If _n_ is not an int object, it is converted with
     PyNumber_Index() first.

     New in version 2.6.

 -- Function: Py_ssize_t PyNumber_AsSsize_t (PyObject *o, PyObject *exc)
     Returns _o_ converted to a Py_ssize_t value if _o_ can be
     interpreted as an integer. If _o_ can be converted to a Python int
     or long but the attempt to convert to a Py_ssize_t value would
     raise an OverflowError, then the _exc_ argument is the
     type of exception that will be raised (usually IndexError
     or OverflowError).  If _exc_ is _NULL_, then the
     exception is cleared and the value is clipped to _PY_SSIZE_T_MIN_
     for a negative integer or _PY_SSIZE_T_MAX_ for a positive integer.

     New in version 2.5.

 -- Function: int PyIndex_Check (PyObject *o)
     Returns True if _o_ is an index integer (has the nb_index slot of
     the tp_as_number structure filled in).

     New in version 2.5.

7.6.3 Sequence Protocol
-----------------------

 -- Function: int PySequence_Check (PyObject *o)
     Return `1' if the object provides sequence protocol, and `0'
     otherwise.  This function always succeeds.

 -- Function: Py_ssize_t PySequence_Size (PyObject *o)
 -- Function: Py_ssize_t PySequence_Length (PyObject *o)
     Returns the number of objects in sequence _o_ on success, and `-1'
     on failure.  For objects that do not provide sequence protocol,
     this is equivalent to the Python expression `len(o)'.

     Changed in version 2.5: These functions returned an `int' type.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: PyObject* PySequence_Concat (PyObject *o1, PyObject *o2)
     Return value: New reference.

     Return the concatenation of _o1_ and _o2_ on success, and _NULL_
     on failure.  This is the equivalent of the Python expression `o1 +
     o2'.

 -- Function: PyObject* PySequence_Repeat (PyObject *o,
          Py_ssize_t count)
     Return value: New reference.

     Return the result of repeating sequence object _o_ _count_ times,
     or _NULL_ on failure.  This is the equivalent of the Python
     expression `o * count'.

     Changed in version 2.5: This function used an `int' type for
     _count_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PySequence_InPlaceConcat (PyObject *o1,
          PyObject *o2)
     Return value: New reference.

     Return the concatenation of _o1_ and _o2_ on success, and _NULL_
     on failure.  The operation is done _in-place_ when _o1_ supports
     it.  This is the equivalent of the Python expression `o1 += o2'.

 -- Function: PyObject* PySequence_InPlaceRepeat (PyObject *o,
          Py_ssize_t count)
     Return value: New reference.

     Return the result of repeating sequence object _o_ _count_ times,
     or _NULL_ on failure.  The operation is done _in-place_ when _o_
     supports it.  This is the equivalent of the Python expression `o
     *= count'.

     Changed in version 2.5: This function used an `int' type for
     _count_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PySequence_GetItem (PyObject *o, Py_ssize_t i)
     Return value: New reference.

     Return the _i_th element of _o_, or _NULL_ on failure. This is the
     equivalent of the Python expression `o[i]'.

     Changed in version 2.5: This function used an `int' type for _i_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: PyObject* PySequence_GetSlice (PyObject *o,
          Py_ssize_t i1, Py_ssize_t i2)
     Return value: New reference.

     Return the slice of sequence object _o_ between _i1_ and _i2_, or
     _NULL_ on failure. This is the equivalent of the Python expression
     `o[i1:i2]'.

     Changed in version 2.5: This function used an `int' type for _i1_
     and _i2_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PySequence_SetItem (PyObject *o, Py_ssize_t i,
          PyObject *v)
     Assign object _v_ to the _i_th element of _o_.  Returns `-1' on
     failure.  This is the equivalent of the Python statement `o[i] =
     v'.  This function _does not_ steal a reference to _v_.

     Changed in version 2.5: This function used an `int' type for _i_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: int PySequence_DelItem (PyObject *o, Py_ssize_t i)
     Delete the _i_th element of object _o_.  Returns `-1' on failure.
     This is the equivalent of the Python statement `del o[i]'.

     Changed in version 2.5: This function used an `int' type for _i_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: int PySequence_SetSlice (PyObject *o, Py_ssize_t i1,
          Py_ssize_t i2, PyObject *v)
     Assign the sequence object _v_ to the slice in sequence object _o_
     from _i1_ to _i2_.  This is the equivalent of the Python statement
     `o[i1:i2] = v'.

     Changed in version 2.5: This function used an `int' type for _i1_
     and _i2_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PySequence_DelSlice (PyObject *o, Py_ssize_t i1,
          Py_ssize_t i2)
     Delete the slice in sequence object _o_ from _i1_ to _i2_.
     Returns `-1' on failure.  This is the equivalent of the Python
     statement `del o[i1:i2]'.

     Changed in version 2.5: This function used an `int' type for _i1_
     and _i2_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: Py_ssize_t PySequence_Count (PyObject *o, PyObject *value)
     Return the number of occurrences of _value_ in _o_, that is,
     return the number of keys for which `o[key] == value'.  On
     failure, return `-1'.  This is equivalent to the Python expression
     `o.count(value)'.

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: int PySequence_Contains (PyObject *o, PyObject *value)
     Determine if _o_ contains _value_.  If an item in _o_ is equal to
     _value_, return `1', otherwise return `0'. On error, return `-1'.
     This is equivalent to the Python expression `value in o'.

 -- Function: Py_ssize_t PySequence_Index (PyObject *o, PyObject *value)
     Return the first index _i_ for which `o[i] == value'.  On error,
     return `-1'.    This is equivalent to the Python expression
     `o.index(value)'.

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: PyObject* PySequence_List (PyObject *o)
     Return value: New reference.

     Return a list object with the same contents as the arbitrary
     sequence _o_.  The returned list is guaranteed to be new.

 -- Function: PyObject* PySequence_Tuple (PyObject *o)
     Return value: New reference.

     Return a tuple object with the same contents as the arbitrary
     sequence _o_ or _NULL_ on failure.  If _o_ is a tuple, a new
     reference will be returned, otherwise a tuple will be constructed
     with the appropriate contents.  This is equivalent to the Python
     expression `tuple(o)'.

 -- Function: PyObject* PySequence_Fast (PyObject *o, const char *m)
     Return value: New reference.

     Returns the sequence _o_ as a tuple, unless it is already a tuple
     or list, in which case _o_ is returned.  Use
     PySequence_Fast_GET_ITEM() to access the members of the
     result.  Returns _NULL_ on failure.  If the object is not a
     sequence, raises TypeError with _m_ as the message
     text.

 -- Function: PyObject* PySequence_Fast_GET_ITEM (PyObject *o,
          Py_ssize_t i)
     Return value: Borrowed reference.

     Return the _i_th element of _o_, assuming that _o_ was returned by
     PySequence_Fast(), _o_ is not _NULL_, and that _i_ is
     within bounds.

     Changed in version 2.5: This function used an `int' type for _i_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: PyObject** PySequence_Fast_ITEMS (PyObject *o)
     Return the underlying array of PyObject pointers.  Assumes that
     _o_ was returned by PySequence_Fast() and _o_ is not
     _NULL_.

     Note, if a list gets resized, the reallocation may relocate the
     items array.  So, only use the underlying array pointer in
     contexts where the sequence cannot change.

     New in version 2.4.

 -- Function: PyObject* PySequence_ITEM (PyObject *o, Py_ssize_t i)
     Return value: New reference.

     Return the _i_th element of _o_ or _NULL_ on failure. Macro form of
     PySequence_GetItem() but without checking that
     `PySequence_Check(o)()' is true and without adjustment for negative
     indices.

     New in version 2.3.

     Changed in version 2.5: This function used an `int' type for _i_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: Py_ssize_t PySequence_Fast_GET_SIZE (PyObject *o)
     Returns the length of _o_, assuming that _o_ was returned by
     PySequence_Fast() and that _o_ is not _NULL_.  The size can
     also be gotten by calling PySequence_Size() on _o_,
     but PySequence_Fast_GET_SIZE() is faster because it
     can assume _o_ is a list or tuple.

7.6.4 Mapping Protocol
----------------------

 -- Function: int PyMapping_Check (PyObject *o)
     Return `1' if the object provides mapping protocol, and `0'
     otherwise.  This function always succeeds.

 -- Function: Py_ssize_t PyMapping_Size (PyObject *o)
 -- Function: Py_ssize_t PyMapping_Length (PyObject *o)
     Returns the number of keys in object _o_ on success, and `-1' on
     failure.  For objects that do not provide mapping protocol, this
     is equivalent to the Python expression `len(o)'.

     Changed in version 2.5: These functions returned an `int' type.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: int PyMapping_DelItemString (PyObject *o, char *key)
     Remove the mapping for object _key_ from the object _o_. Return
     `-1' on failure.  This is equivalent to the Python statement `del
     o[key]'.

 -- Function: int PyMapping_DelItem (PyObject *o, PyObject *key)
     Remove the mapping for object _key_ from the object _o_. Return
     `-1' on failure.  This is equivalent to the Python statement `del
     o[key]'.

 -- Function: int PyMapping_HasKeyString (PyObject *o, char *key)
     On success, return `1' if the mapping object has the key _key_ and
     `0' otherwise.  This is equivalent to `o[key]', returning `True'
     on success and `False' on an exception.  This function always
     succeeds.

 -- Function: int PyMapping_HasKey (PyObject *o, PyObject *key)
     Return `1' if the mapping object has the key _key_ and `0'
     otherwise.  This is equivalent to `o[key]', returning `True' on
     success and `False' on an exception.  This function always
     succeeds.

 -- Function: PyObject* PyMapping_Keys (PyObject *o)
     Return value: New reference.

     On success, return a list of the keys in object _o_.  On failure,
     return _NULL_.  This is equivalent to the Python expression
     `o.keys()'.

 -- Function: PyObject* PyMapping_Values (PyObject *o)
     Return value: New reference.

     On success, return a list of the values in object _o_.  On
     failure, return _NULL_. This is equivalent to the Python
     expression `o.values()'.

 -- Function: PyObject* PyMapping_Items (PyObject *o)
     Return value: New reference.

     On success, return a list of the items in object _o_, where each
     item is a tuple containing a key-value pair.  On failure, return
     _NULL_. This is equivalent to the Python expression `o.items()'.

 -- Function: PyObject* PyMapping_GetItemString (PyObject *o, char *key)
     Return value: New reference.

     Return element of _o_ corresponding to the object _key_ or _NULL_
     on failure.  This is the equivalent of the Python expression
     `o[key]'.

 -- Function: int PyMapping_SetItemString (PyObject *o, char *key,
          PyObject *v)
     Map the object _key_ to the value _v_ in object _o_. Returns `-1'
     on failure.  This is the equivalent of the Python statement
     `o[key] = v'.

7.6.5 Iterator Protocol
-----------------------

New in version 2.2.

  There are only a couple of functions specifically for working with
iterators.

 -- Function: int PyIter_Check (PyObject *o)
     Return true if the object _o_ supports the iterator protocol.

 -- Function: PyObject* PyIter_Next (PyObject *o)
     Return value: New reference.

     Return the next value from the iteration _o_.  If the object is an
     iterator, this retrieves the next value from the iteration, and
     returns _NULL_ with no exception set if there are no remaining
     items.  If the object is not an iterator, see TypeError is
     raised, or if there is an error in retrieving the item, returns
     _NULL_ and passes along the exception.

  To write a loop which iterates over an iterator, the C code should
look something like this:

    PyObject *iterator = PyObject_GetIter(obj);
    PyObject *item;

    if (iterator == NULL) {
        /* propagate error */
    }

    while (item = PyIter_Next(iterator)) {
        /* do something with item */
        ...
        /* release reference when done */
        Py_DECREF(item);
    }

    Py_DECREF(iterator);

    if (PyErr_Occurred()) {
        /* propagate error */
    }
    else {
        /* continue doing useful work */
    }

7.6.6 Old Buffer Protocol
-------------------------

This section describes the legacy buffer protocol, which has been
introduced in Python 1.6. It is still supported but deprecated in the
Python 2.x series.  Python 3.0 introduces a new buffer protocol which
fixes weaknesses and shortcomings of the protocol, and has been
backported to Python 2.6.  See Buffers and Memoryview Objects
for more information.

 -- Function: int PyObject_AsCharBuffer (PyObject *obj, const
          char **buffer, Py_ssize_t *buffer_len)
     Returns a pointer to a read-only memory location usable as
     character-based input.  The _obj_ argument must support the
     single-segment character buffer interface.  On success, returns
     `0', sets _buffer_ to the memory location and _buffer_len_ to the
     buffer length.  Returns `-1' and sets a TypeError on
     error.

     New in version 1.6.

     Changed in version 2.5: This function used an `int *' type for
     _buffer_len_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyObject_AsReadBuffer (PyObject *obj, const
          void **buffer, Py_ssize_t *buffer_len)
     Returns a pointer to a read-only memory location containing
     arbitrary data.  The _obj_ argument must support the
     single-segment readable buffer interface.  On success, returns
     `0', sets _buffer_ to the memory location and _buffer_len_ to the
     buffer length.  Returns `-1' and sets a TypeError on
     error.

     New in version 1.6.

     Changed in version 2.5: This function used an `int *' type for
     _buffer_len_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyObject_CheckReadBuffer (PyObject *o)
     Returns `1' if _o_ supports the single-segment readable buffer
     interface.  Otherwise returns `0'.

     New in version 2.2.

 -- Function: int PyObject_AsWriteBuffer (PyObject *obj, void **buffer,
          Py_ssize_t *buffer_len)
     Returns a pointer to a writeable memory location.  The _obj_
     argument must support the single-segment, character buffer
     interface.  On success, returns `0', sets _buffer_ to the memory
     location and _buffer_len_ to the buffer length.  Returns `-1' and
     sets a TypeError on error.

     New in version 1.6.

     Changed in version 2.5: This function used an `int *' type for
     _buffer_len_. This might require changes in your code for properly
     supporting 64-bit systems.

7.7 Concrete Objects Layer
==========================

The functions in this chapter are specific to certain Python object
types.  Passing them an object of the wrong type is not a good idea; if
you receive an object from a Python program and you are not sure that
it has the right type, you must perform a type check first; for
example, to check that an object is a dictionary, use
PyDict_Check().  The chapter is structured like the "family tree"
of Python object types.

     Warning: While the functions described in this chapter carefully
     check the type of the objects which are passed in, many of them do
     not check for _NULL_ being passed instead of a valid object.
     Allowing _NULL_ to be passed in can cause memory access violations
     and immediate termination of the interpreter.

7.7.1 Fundamental Objects
-------------------------

This section describes Python type objects and the singleton object
`None'.

7.7.1.1 Type Objects
....................

 -- type: PyTypeObject
     The C structure of the objects used to describe built-in types.

 -- var: PyObject* PyType_Type
     This is the type object for type objects; it is the same object as
     `type' and `types.TypeType' in the Python layer.

 -- Function: int PyType_Check (PyObject *o)
     Return true if the object _o_ is a type object, including
     instances of types derived from the standard type object.  Return
     false in all other cases.

 -- Function: int PyType_CheckExact (PyObject *o)
     Return true if the object _o_ is a type object, but not a subtype
     of the standard type object.  Return false in all other cases.

     New in version 2.2.

 -- Function: unsigned int PyType_ClearCache ()
     Clear the internal lookup cache. Return the current version tag.

     New in version 2.6.

 -- Function: void PyType_Modified (PyTypeObject *type)
     Invalidate the internal lookup cache for the type and all of its
     subtypes.  This function must be called after any manual
     modification of the attributes or base classes of the type.

     New in version 2.6.

 -- Function: int PyType_HasFeature (PyObject *o, int feature)
     Return true if the type object _o_ sets the feature _feature_.
     Type features are denoted by single bit flags.

 -- Function: int PyType_IS_GC (PyObject *o)
     Return true if the type object includes support for the cycle
     detector; this tests the type flag Py_TPFLAGS_HAVE_GC.

     New in version 2.0.

 -- Function: int PyType_IsSubtype (PyTypeObject *a, PyTypeObject *b)
     Return true if _a_ is a subtype of _b_.

     New in version 2.2.

 -- Function: PyObject* PyType_GenericAlloc (PyTypeObject *type,
          Py_ssize_t nitems)
     Return value: New reference.

     New in version 2.2.

     Changed in version 2.5: This function used an `int' type for
     _nitems_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyType_GenericNew (PyTypeObject *type,
          PyObject *args, PyObject *kwds)
     Return value: New reference.

     New in version 2.2.

 -- Function: int PyType_Ready (PyTypeObject *type)
     Finalize a type object.  This should be called on all type objects
     to finish their initialization.  This function is responsible for
     adding inherited slots from a type's base class.  Return `0' on
     success, or return `-1' and sets an exception on error.

     New in version 2.2.

7.7.1.2 The None Object
.......................

Note that the PyTypeObject for `None' is not directly
exposed in the Python/C API.  Since `None' is a singleton, testing for
object identity (using `==' in C) is sufficient. There is no
`PyNone_Check()' function for the same reason.

 -- var: PyObject* Py_None
     The Python `None' object, denoting lack of value.  This object has
     no methods.  It needs to be treated just like any other object
     with respect to reference counts.

 -- macro: Py_RETURN_NONE
     Properly handle returning Py_None from within a C
     function.

     New in version 2.4.

7.7.2 Numeric Objects
---------------------

7.7.2.1 Plain Integer Objects
.............................

 -- type: PyIntObject
     This subtype of PyObject represents a Python integer
     object.

 -- var: PyTypeObject PyInt_Type
     This instance of PyTypeObject represents the Python
     plain integer type.  This is the same object as `int' and
     `types.IntType'.

 -- Function: int PyInt_Check (PyObject *o)
     Return true if _o_ is of type PyInt_Type or a subtype
     of PyInt_Type.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyInt_CheckExact (PyObject *o)
     Return true if _o_ is of type PyInt_Type, but not a
     subtype of PyInt_Type.

     New in version 2.2.

 -- Function: PyObject* PyInt_FromString (char *str, char **pend,
          int base)
     Return value: New reference.

     Return a new PyIntObject or PyLongObject.
     based on the string value in _str_, which is interpreted according
     to the radix in _base_.  If _pend_ is non-_NULL_, `*pend' will
     point to the first character in _str_ which follows the
     representation of the number.  If _base_ is `0', the radix will be
     determined based on the leading characters of _str_: if _str_
     starts with `'0x'' or `'0X'', radix 16 will be used; if _str_
     starts with `'0'', radix 8 will be used; otherwise radix 10 will
     be used.  If _base_ is not `0', it must be between `2' and `36',
     inclusive.  Leading spaces are ignored.  If there are no digits,
     see ValueError will be raised.  If the string represents a
     number too large to be contained within the machine's `long int'
     type and overflow warnings are being suppressed, a
     PyLongObject will be returned.  If overflow warnings are
     not being suppressed, _NULL_ will be returned in this case.

 -- Function: PyObject* PyInt_FromLong (long ival)
     Return value: New reference.

     Create a new integer object with a value of _ival_.

     The current implementation keeps an array of integer objects for
     all integers between `-5' and `256', when you create an int in
     that range you actually just get back a reference to the existing
     object. So it should be possible to change the value of `1'.  I
     suspect the behaviour of Python in this case is undefined. :-)

 -- Function: PyObject* PyInt_FromSsize_t (Py_ssize_t ival)
     Return value: New reference.

     Create a new integer object with a value of _ival_. If the value
     is larger than `LONG_MAX' or smaller than `LONG_MIN', a long
     integer object is returned.

     New in version 2.5.

 -- Function: PyObject* PyInt_FromSize_t (size_t ival)
     Create a new integer object with a value of _ival_. If the value
     exceeds `LONG_MAX', a long integer object is returned.

     New in version 2.5.

 -- Function: long PyInt_AsLong (PyObject *io)
     Will first attempt to cast the object to a PyIntObject,
     if it is not already one, and then return its value. If
     there is an error, `-1' is returned, and the caller should check
     `PyErr_Occurred()' to find out whether there was an error, or
     whether the value just happened to be -1.

 -- Function: long PyInt_AS_LONG (PyObject *io)
     Return the value of the object _io_.  No error checking is
     performed.

 -- Function: unsigned long PyInt_AsUnsignedLongMask (PyObject *io)
     Will first attempt to cast the object to a PyIntObject
     or PyLongObject, if it is not already one, and
     then return its value as unsigned long.  This function does not
     check for overflow.

     New in version 2.3.

 -- Function: unsigned PY_LONG_LONG PyInt_AsUnsignedLongLongMask
          (PyObject *io)
     Will first attempt to cast the object to a PyIntObject
     or PyLongObject, if it is not already one, and
     then return its value as unsigned long long, without checking for
     overflow.

     New in version 2.3.

 -- Function: Py_ssize_t PyInt_AsSsize_t (PyObject *io)
     Will first attempt to cast the object to a PyIntObject
     or PyLongObject, if it is not already one, and
     then return its value as `Py_ssize_t'.

     New in version 2.5.

 -- Function: long PyInt_GetMax ()
     Return the system's idea of the largest integer it can handle
     (`LONG_MAX', as defined in the system header files).

 -- Function: int PyInt_ClearFreeList ()
     Clear the integer free list. Return the number of items that could
     not be freed.

     New in version 2.6.

7.7.2.2 Boolean Objects
.......................

Booleans in Python are implemented as a subclass of integers.  There
are only two booleans, `Py_False' and `Py_True'.  As such, the normal
creation and deletion functions don't apply to booleans.  The following
macros are available, however.

 -- Function: int PyBool_Check (PyObject *o)
     Return true if _o_ is of type `PyBool_Type'.

     New in version 2.3.

 -- var: PyObject* Py_False
     The Python `False' object.  This object has no methods.  It needs
     to be treated just like any other object with respect to reference
     counts.

 -- var: PyObject* Py_True
     The Python `True' object.  This object has no methods.  It needs
     to be treated just like any other object with respect to reference
     counts.

 -- macro: Py_RETURN_FALSE
     Return `Py_False' from a function, properly incrementing its
     reference count.

     New in version 2.4.

 -- macro: Py_RETURN_TRUE
     Return `Py_True' from a function, properly incrementing its
     reference count.

     New in version 2.4.

 -- Function: PyObject* PyBool_FromLong (long v)
     Return value: New reference.

     Return a new reference to `Py_True' or `Py_False' depending on the
     truth value of _v_.

     New in version 2.3.

7.7.2.3 Long Integer Objects
............................

 -- type: PyLongObject
     This subtype of PyObject represents a Python long
     integer object.

 -- var: PyTypeObject PyLong_Type
     This instance of PyTypeObject represents the Python
     long integer type.  This is the same object as `long' and
     `types.LongType'.

 -- Function: int PyLong_Check (PyObject *p)
     Return true if its argument is a PyLongObject or a
     subtype of PyLongObject.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyLong_CheckExact (PyObject *p)
     Return true if its argument is a PyLongObject, but not
     a subtype of PyLongObject.

     New in version 2.2.

 -- Function: PyObject* PyLong_FromLong (long v)
     Return value: New reference.

     Return a new PyLongObject object from _v_, or _NULL_
     on failure.

 -- Function: PyObject* PyLong_FromUnsignedLong (unsigned long v)
     Return value: New reference.

     Return a new PyLongObject object from a C `unsigned
     long', or _NULL_ on failure.

 -- Function: PyObject* PyLong_FromSsize_t (Py_ssize_t v)
     Return value: New reference.

     Return a new PyLongObject object from a C
     `Py_ssize_t', or _NULL_ on failure.

     New in version 2.6.

 -- Function: PyObject* PyLong_FromSize_t (size_t v)
     Return value: New reference.

     Return a new PyLongObject object from a C `size_t', or
     _NULL_ on failure.

     New in version 2.6.

 -- Function: PyObject* PyLong_FromSsize_t (Py_ssize_t v)
     Return a new PyLongObject object with a value of _v_,
     or _NULL_ on failure.

     New in version 2.6.

 -- Function: PyObject* PyLong_FromSize_t (size_t v)
     Return a new PyLongObject object with a value of _v_,
     or _NULL_ on failure.

     New in version 2.6.

 -- Function: PyObject* PyLong_FromLongLong (PY_LONG_LONG v)
     Return value: New reference.

     Return a new PyLongObject object from a C `long
     long', or _NULL_ on failure.

 -- Function: PyObject* PyLong_FromUnsignedLongLong (unsigned
          PY_LONG_LONG v)
     Return value: New reference.

     Return a new PyLongObject object from a C `unsigned
     long long', or _NULL_ on failure.

 -- Function: PyObject* PyLong_FromDouble (double v)
     Return value: New reference.

     Return a new PyLongObject object from the integer
     part of _v_, or _NULL_ on failure.

 -- Function: PyObject* PyLong_FromString (char *str, char **pend,
          int base)
     Return value: New reference.

     Return a new PyLongObject based on the string value
     in _str_, which is interpreted according to the radix in _base_.
     If _pend_ is non-_NULL_, _*pend_ will point to the first character
     in _str_ which follows the representation of the number.  If
     _base_ is `0', the radix will be determined based on the leading
     characters of _str_: if _str_ starts with `'0x'' or `'0X'', radix
     16 will be used; if _str_ starts with `'0'', radix 8 will be used;
     otherwise radix 10 will be used.  If _base_ is not `0', it must be
     between `2' and `36', inclusive.  Leading spaces are ignored.  If
     there are no digits, see ValueError will be raised.

 -- Function: PyObject* PyLong_FromUnicode (Py_UNICODE *u,
          Py_ssize_t length, int base)
     Return value: New reference.

     Convert a sequence of Unicode digits to a Python long integer
     value.  The first parameter, _u_, points to the first character of
     the Unicode string, _length_ gives the number of characters, and
     _base_ is the radix for the conversion.  The radix must be in the
     range [2, 36]; if it is out of range, see ValueError.  will
     be raised.

     New in version 1.6.

     Changed in version 2.5: This function used an `int' for _length_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: PyObject* PyLong_FromVoidPtr (void *p)
     Return value: New reference.

     Create a Python integer or long integer from the pointer _p_. The
     pointer value can be retrieved from the resulting value using
     PyLong_AsVoidPtr().

     New in version 1.5.2.

     Changed in version 2.5: If the integer is larger than LONG_MAX, a
     positive long integer is returned.

 -- Function: long PyLong_AsLong (PyObject *pylong)
     Return a C `long' representation of the contents of _pylong_.  If
     _pylong_ is greater than `LONG_MAX', an OverflowError.
     is raised and `-1' will be returned.

 -- Function: long PyLong_AsLongAndOverflow (PyObject *pylong,
          int *overflow)
     Return a C `long' representation of the contents of _pylong_.  If
     _pylong_ is greater than `LONG_MAX' or less than `LONG_MIN', set
     _*overflow_ to `1' or `-1', respectively, and return `-1';
     otherwise, set _*overflow_ to `0'.  If any other exception occurs
     (for example a TypeError or MemoryError), then `-1' will be
     returned and _*overflow_ will be `0'.

     New in version 2.7.

 -- Function: PY_LONG_LONG PyLong_AsLongLongAndOverflow
          (PyObject *pylong, int *overflow)
     Return a C `long long' representation of the contents of _pylong_.
     If _pylong_ is greater than `PY_LLONG_MAX' or less than
     `PY_LLONG_MIN', set _*overflow_ to `1' or `-1', respectively, and
     return `-1'; otherwise, set _*overflow_ to `0'.  If any other
     exception occurs (for example a TypeError or MemoryError), then
     `-1' will be returned and _*overflow_ will be `0'.

     New in version 2.7.

 -- Function: Py_ssize_t PyLong_AsSsize_t (PyObject *pylong)
     Return a C `Py_ssize_t' representation of the contents of
     _pylong_.  If _pylong_ is greater than `PY_SSIZE_T_MAX', an
     OverflowError is raised and `-1' will be returned.

     New in version 2.6.

 -- Function: unsigned long PyLong_AsUnsignedLong (PyObject *pylong)
     Return a C `unsigned long' representation of the contents of
     _pylong_.  If _pylong_ is greater than `ULONG_MAX', an
     OverflowError is raised.

 -- Function: Py_ssize_t PyLong_AsSsize_t (PyObject *pylong)
     Return a `Py_ssize_t' representation of the contents of _pylong_.
     If _pylong_ is greater than `PY_SSIZE_T_MAX', an
     OverflowError is raised.

     New in version 2.6.

 -- Function: PY_LONG_LONG PyLong_AsLongLong (PyObject *pylong)
     Return a C `long long' from a Python long integer.  If _pylong_
     cannot be represented as a `long long', an OverflowError
     is raised and `-1' is returned.

     New in version 2.2.

 -- Function: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLong
          (PyObject *pylong)
     Return a C `unsigned long long' from a Python long integer. If
     _pylong_ cannot be represented as an `unsigned long long', an
     OverflowError is raised and `(unsigned long long)-1' is
     returned.

     New in version 2.2.

     Changed in version 2.7: A negative _pylong_ now raises
     OverflowError, not TypeError.

 -- Function: unsigned long PyLong_AsUnsignedLongMask (PyObject *io)
     Return a C `unsigned long' from a Python long integer, without
     checking for overflow.

     New in version 2.3.

 -- Function: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask
          (PyObject *io)
     Return a C `unsigned long long' from a Python long integer, without
     checking for overflow.

     New in version 2.3.

 -- Function: double PyLong_AsDouble (PyObject *pylong)
     Return a C `double' representation of the contents of _pylong_.  If
     _pylong_ cannot be approximately represented as a `double', an
     OverflowError exception is raised and `-1.0' will be
     returned.

 -- Function: void* PyLong_AsVoidPtr (PyObject *pylong)
     Convert a Python integer or long integer _pylong_ to a C `void'
     pointer.  If _pylong_ cannot be converted, an OverflowError
     will be raised.  This is only assured to produce a usable
     `void' pointer for values created with PyLong_FromVoidPtr().

     New in version 1.5.2.

     Changed in version 2.5: For values outside 0..LONG_MAX, both
     signed and unsigned integers are accepted.

7.7.2.4 Floating Point Objects
..............................

 -- type: PyFloatObject
     This subtype of PyObject represents a Python floating
     point object.

 -- var: PyTypeObject PyFloat_Type
     This instance of PyTypeObject represents the Python
     floating point type.  This is the same object as `float' and
     `types.FloatType'.

 -- Function: int PyFloat_Check (PyObject *p)
     Return true if its argument is a PyFloatObject or a
     subtype of PyFloatObject.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyFloat_CheckExact (PyObject *p)
     Return true if its argument is a PyFloatObject, but
     not a subtype of PyFloatObject.

     New in version 2.2.

 -- Function: PyObject* PyFloat_FromString (PyObject *str, char **pend)
     Return value: New reference.

     Create a PyFloatObject object based on the string
     value in _str_, or _NULL_ on failure.  The _pend_ argument is
     ignored.  It remains only for backward compatibility.

 -- Function: PyObject* PyFloat_FromDouble (double v)
     Return value: New reference.

     Create a PyFloatObject object from _v_, or _NULL_ on
     failure.

 -- Function: double PyFloat_AsDouble (PyObject *pyfloat)
     Return a C `double' representation of the contents of _pyfloat_.
     If _pyfloat_ is not a Python floating point object but has a
     __float__().  method, this method will first be called to
     convert _pyfloat_ into a float.

 -- Function: double PyFloat_AS_DOUBLE (PyObject *pyfloat)
     Return a C `double' representation of the contents of _pyfloat_,
     but without error checking.

 -- Function: PyObject* PyFloat_GetInfo (void)
     Return a structseq instance which contains information about the
     precision, minimum and maximum values of a float. It's a thin
     wrapper around the header file `float.h'.

     New in version 2.6.

 -- Function: double PyFloat_GetMax ()
     Return the maximum representable finite float _DBL_MAX_ as C
     `double'.

     New in version 2.6.

 -- Function: double PyFloat_GetMin ()
     Return the minimum normalized positive float _DBL_MIN_ as C
     `double'.

     New in version 2.6.

 -- Function: int PyFloat_ClearFreeList ()
     Clear the float free list. Return the number of items that could
     not be freed.

     New in version 2.6.

 -- Function: void PyFloat_AsString (char *buf, PyFloatObject *v)
     Convert the argument _v_ to a string, using the same rules as
     str(). The length of _buf_ should be at least 100.

     This function is unsafe to call because it writes to a buffer whose
     length it does not know.

     Deprecated since version 2.7: Use `PyObject_Str()' or
     `PyOS_double_to_string()' instead.

 -- Function: void PyFloat_AsReprString (char *buf, PyFloatObject *v)
     Same as PyFloat_AsString, except uses the same rules as
     repr().  The length of _buf_ should be at least 100.

     This function is unsafe to call because it writes to a buffer whose
     length it does not know.

     Deprecated since version 2.7: Use `PyObject_Repr()' or
     `PyOS_double_to_string()' instead.

7.7.2.5 Complex Number Objects
..............................

Python's complex number objects are implemented as two distinct types
when viewed from the C API:  one is the Python object exposed to Python
programs, and the other is a C structure which represents the actual
complex number value.  The API provides functions for working with both.

7.7.2.6 Complex Numbers as C Structures
.......................................

Note that the functions which accept these structures as parameters and
return them as results do so _by value_ rather than dereferencing them
through pointers.  This is consistent throughout the API.

 -- type: Py_complex
     The C structure which corresponds to the value portion of a Python
     complex number object.  Most of the functions for dealing with
     complex number objects use structures of this type as input or
     output values, as appropriate.  It is defined as:

         typedef struct {
            double real;
            double imag;
         } Py_complex;

 -- Function: Py_complex _Py_c_sum (Py_complex left, Py_complex right)
     Return the sum of two complex numbers, using the C
     Py_complex.  representation.

 -- Function: Py_complex _Py_c_diff (Py_complex left, Py_complex right)
     Return the difference between two complex numbers, using the C
     Py_complex representation.

 -- Function: Py_complex _Py_c_neg (Py_complex complex)
     Return the negation of the complex number _complex_, using the C
     Py_complex representation.

 -- Function: Py_complex _Py_c_prod (Py_complex left, Py_complex right)
     Return the product of two complex numbers, using the C
     Py_complex.  representation.

 -- Function: Py_complex _Py_c_quot (Py_complex dividend,
          Py_complex divisor)
     Return the quotient of two complex numbers, using the C
     Py_complex.  representation.

 -- Function: Py_complex _Py_c_pow (Py_complex num, Py_complex exp)
     Return the exponentiation of _num_ by _exp_, using the C
     Py_complex.  representation.

7.7.2.7 Complex Numbers as Python Objects
.........................................

 -- type: PyComplexObject
     This subtype of PyObject represents a Python complex
     number object.

 -- var: PyTypeObject PyComplex_Type
     This instance of PyTypeObject represents the Python
     complex number type. It is the same object as `complex' and
     `types.ComplexType'.

 -- Function: int PyComplex_Check (PyObject *p)
     Return true if its argument is a PyComplexObject or a
     subtype of PyComplexObject.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyComplex_CheckExact (PyObject *p)
     Return true if its argument is a PyComplexObject, but
     not a subtype of PyComplexObject.

     New in version 2.2.

 -- Function: PyObject* PyComplex_FromCComplex (Py_complex v)
     Return value: New reference.

     Create a new Python complex number object from a C
     Py_complex value.

 -- Function: PyObject* PyComplex_FromDoubles (double real, double imag)
     Return value: New reference.

     Return a new PyComplexObject object from _real_ and
     _imag_.

 -- Function: double PyComplex_RealAsDouble (PyObject *op)
     Return the real part of _op_ as a C `double'.

 -- Function: double PyComplex_ImagAsDouble (PyObject *op)
     Return the imaginary part of _op_ as a C `double'.

 -- Function: Py_complex PyComplex_AsCComplex (PyObject *op)
     Return the Py_complex value of the complex number
     _op_.

     Changed in version 2.6: If _op_ is not a Python complex number
     object but has a __complex__().  method, this method
     will first be called to convert _op_ to a Python complex number
     object.

7.7.3 Sequence Objects
----------------------

Generic operations on sequence objects were discussed in the previous
chapter; this section deals with the specific kinds of sequence objects
that are intrinsic to the Python language.

7.7.3.1 Byte Array Objects
..........................

New in version 2.6.

 -- type: PyByteArrayObject
     This subtype of PyObject represents a Python bytearray
     object.

 -- var: PyTypeObject PyByteArray_Type
     This instance of PyTypeObject represents the Python
     bytearray type; it is the same object as `bytearray' in the Python
     layer.

7.7.3.2 Type check macros
.........................

 -- Function: int PyByteArray_Check (PyObject *o)
     Return true if the object _o_ is a bytearray object or an instance
     of a subtype of the bytearray type.

 -- Function: int PyByteArray_CheckExact (PyObject *o)
     Return true if the object _o_ is a bytearray object, but not an
     instance of a subtype of the bytearray type.

7.7.3.3 Direct API functions
............................

 -- Function: PyObject* PyByteArray_FromObject (PyObject *o)
     Return a new bytearray object from any object, _o_, that
     implements the buffer protocol.

 -- Function: PyObject* PyByteArray_FromStringAndSize (const
          char *string, Py_ssize_t len)
     Create a new bytearray object from _string_ and its length, _len_.
     On failure, _NULL_ is returned.

 -- Function: PyObject* PyByteArray_Concat (PyObject *a, PyObject *b)
     Concat bytearrays _a_ and _b_ and return a new bytearray with the
     result.

 -- Function: Py_ssize_t PyByteArray_Size (PyObject *bytearray)
     Return the size of _bytearray_ after checking for a _NULL_ pointer.

 -- Function: char* PyByteArray_AsString (PyObject *bytearray)
     Return the contents of _bytearray_ as a char array after checking
     for a _NULL_ pointer.

 -- Function: int PyByteArray_Resize (PyObject *bytearray,
          Py_ssize_t len)
     Resize the internal buffer of _bytearray_ to _len_.

7.7.3.4 Macros
..............

These macros trade safety for speed and they don't check pointers.

 -- Function: char* PyByteArray_AS_STRING (PyObject *bytearray)
     Macro version of PyByteArray_AsString().

 -- Function: Py_ssize_t PyByteArray_GET_SIZE (PyObject *bytearray)
     Macro version of PyByteArray_Size().

7.7.3.5 String/Bytes Objects
............................

These functions raise TypeError when expecting a string
parameter and are called with a non-string parameter.

     Note: These functions have been renamed to PyBytes_* in Python
     3.x. Unless otherwise noted, the PyBytes functions available in
     3.x are aliased to their PyString_* equivalents to help porting.

 -- type: PyStringObject
     This subtype of PyObject represents a Python string
     object.

 -- var: PyTypeObject PyString_Type
     This instance of PyTypeObject represents the Python
     string type; it is the same object as `str' and `types.StringType'
     in the Python layer. .

 -- Function: int PyString_Check (PyObject *o)
     Return true if the object _o_ is a string object or an instance of
     a subtype of the string type.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyString_CheckExact (PyObject *o)
     Return true if the object _o_ is a string object, but not an
     instance of a subtype of the string type.

     New in version 2.2.

 -- Function: PyObject* PyString_FromString (const char *v)
     Return value: New reference.

     Return a new string object with a copy of the string _v_ as value
     on success, and _NULL_ on failure.  The parameter _v_ must not be
     _NULL_; it will not be checked.

 -- Function: PyObject* PyString_FromStringAndSize (const char *v,
          Py_ssize_t len)
     Return value: New reference.

     Return a new string object with a copy of the string _v_ as value
     and length _len_ on success, and _NULL_ on failure.  If _v_ is
     _NULL_, the contents of the string are uninitialized.

     Changed in version 2.5: This function used an `int' type for
     _len_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyString_FromFormat (const char *format, ...)
     Return value: New reference.

     Take a C `printf()'-style _format_ string and a variable number of
     arguments, calculate the size of the resulting Python string and
     return a string with the values formatted into it.  The variable
     arguments must be C types and must correspond exactly to the
     format characters in the _format_ string.  The following format
     characters are allowed:

     Format Characters       Type                Comment
     ---------------------------------------------------------------------------------
     `%%'                    _n/a_               The literal % character.
     `%c'                    int                 A single character, represented as
                                                 an C int.
     `%d'                    int                 Exactly equivalent to
                                                 `printf("%d")'.
     `%u'                    unsigned int        Exactly equivalent to
                                                 `printf("%u")'.
     `%ld'                   long                Exactly equivalent to
                                                 `printf("%ld")'.
     `%lu'                   unsigned long       Exactly equivalent to
                                                 `printf("%lu")'.
     `%lld'                  long long           Exactly equivalent to
                                                 `printf("%lld")'.
     `%llu'                  unsigned long long  Exactly equivalent to
                                                 `printf("%llu")'.
     `%zd'                   Py_ssize_t          Exactly equivalent to
                                                 `printf("%zd")'.
     `%zu'                   size_t              Exactly equivalent to
                                                 `printf("%zu")'.
     `%i'                    int                 Exactly equivalent to
                                                 `printf("%i")'.
     `%x'                    int                 Exactly equivalent to
                                                 `printf("%x")'.
     `%s'                    char*               A null-terminated C character array.
     `%p'                    void*               The hex representation of a C
                                                 pointer. Mostly equivalent to
                                                 `printf("%p")' except that it is
                                                 guaranteed to start with the
                                                 literal `0x' regardless of what the
                                                 platform's `printf' yields.

     An unrecognized format character causes all the rest of the format
     string to be copied as-is to the result string, and any extra
     arguments discarded.

          Note: The `"%lld"' and `"%llu"' format specifiers are only
          available when `HAVE_LONG_LONG' is defined.

     Changed in version 2.7: Support for `"%lld"' and `"%llu"' added.

 -- Function: PyObject* PyString_FromFormatV (const char *format,
          va_list vargs)
     Return value: New reference.

     Identical to PyString_FromFormat() except that it
     takes exactly two arguments.

 -- Function: Py_ssize_t PyString_Size (PyObject *string)
     Return the length of the string in string object _string_.

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: Py_ssize_t PyString_GET_SIZE (PyObject *string)
     Macro form of PyString_Size() but without error
     checking.

     Changed in version 2.5: This macro returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: char* PyString_AsString (PyObject *string)
     Return a NUL-terminated representation of the contents of
     _string_.  The pointer refers to the internal buffer of _string_,
     not a copy.  The data must not be modified in any way, unless the
     string was just created using `PyString_FromStringAndSize(NULL,
     size)'. It must not be deallocated.  If _string_ is a Unicode
     object, this function computes the default encoding of _string_
     and operates on that.  If _string_ is not a string object at all,
     see PyString_AsString() returns _NULL_ and raises
     TypeError.

 -- Function: char* PyString_AS_STRING (PyObject *string)
     Macro form of PyString_AsString() but without error
     checking.  Only string objects are supported; no Unicode objects
     should be passed.

 -- Function: int PyString_AsStringAndSize (PyObject *obj,
          char **buffer, Py_ssize_t *length)
     Return a NUL-terminated representation of the contents of the
     object _obj_ through the output variables _buffer_ and _length_.

     The function accepts both string and Unicode objects as input. For
     Unicode objects it returns the default encoded version of the
     object.  If _length_ is _NULL_, the resulting buffer may not
     contain NUL characters; if it does, the function returns `-1' and
     a TypeError is raised.

     The buffer refers to an internal string buffer of _obj_, not a
     copy. The data must not be modified in any way, unless the string
     was just created using `PyString_FromStringAndSize(NULL, size)'.
     It must not be deallocated.  If _string_ is a Unicode object, this
     function computes the default encoding of _string_ and operates on
     that.  If _string_ is not a string object at all, see
     PyString_AsStringAndSize() returns `-1' and raises
     TypeError.

     Changed in version 2.5: This function used an `int *' type for
     _length_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: void PyString_Concat (PyObject **string,
          PyObject *newpart)
     Create a new string object in _*string_ containing the contents of
     _newpart_ appended to _string_; the caller will own the new
     reference.  The reference to the old value of _string_ will be
     stolen.  If the new string cannot be created, the old reference to
     _string_ will still be discarded and the value of _*string_ will
     be set to _NULL_; the appropriate exception will be set.

 -- Function: void PyString_ConcatAndDel (PyObject **string,
          PyObject *newpart)
     Create a new string object in _*string_ containing the contents of
     _newpart_ appended to _string_.  This version decrements the
     reference count of _newpart_.

 -- Function: int _PyString_Resize (PyObject **string,
          Py_ssize_t newsize)
     A way to resize a string object even though it is "immutable".
     Only use this to build up a brand new string object; don't use
     this if the string may already be known in other parts of the
     code.  It is an error to call this function if the refcount on the
     input string object is not one. Pass the address of an existing
     string object as an lvalue (it may be written into), and the new
     size desired.  On success, _*string_ holds the resized string
     object and `0' is returned; the address in _*string_ may differ
     from its input value.  If the reallocation fails, the original
     string object at _*string_ is deallocated, _*string_ is set to
     _NULL_, a memory exception is set, and `-1' is returned.

     Changed in version 2.5: This function used an `int' type for
     _newsize_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyString_Format (PyObject *format,
          PyObject *args)
     Return value: New reference.

     Return a new string object from _format_ and _args_. Analogous to
     `format % args'.  The _args_ argument must be a tuple.

 -- Function: void PyString_InternInPlace (PyObject **string)
     Intern the argument _*string_ in place.  The argument must be the
     address of a pointer variable pointing to a Python string object.
     If there is an existing interned string that is the same as
     _*string_, it sets _*string_ to it (decrementing the reference
     count of the old string object and incrementing the reference
     count of the interned string object), otherwise it leaves _*string_
     alone and interns it (incrementing its reference count).
     (Clarification: even though there is a lot of talk about reference
     counts, think of this function as reference-count-neutral; you own
     the object after the call if and only if you owned it before the
     call.)

          Note: This function is not available in 3.x and does not have
          a PyBytes alias.

 -- Function: PyObject* PyString_InternFromString (const char *v)
     Return value: New reference.

     A combination of PyString_FromString() and
     PyString_InternInPlace(), returning either a new string
     object that has been interned, or a new ("owned") reference to an
     earlier interned string object with the same value.

          Note: This function is not available in 3.x and does not have
          a PyBytes alias.

 -- Function: PyObject* PyString_Decode (const char *s,
          Py_ssize_t size, const char *encoding, const char *errors)
     Return value: New reference.

     Create an object by decoding _size_ bytes of the encoded buffer
     _s_ using the codec registered for _encoding_.  _encoding_ and
     _errors_ have the same meaning as the parameters of the same name
     in the unicode() built-in function.  The codec to be
     used is looked up using the Python codec registry.  Return _NULL_
     if an exception was raised by the codec.

          Note: This function is not available in 3.x and does not have
          a PyBytes alias.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyString_AsDecodedObject (PyObject *str, const
          char *encoding, const char *errors)
     Return value: New reference.

     Decode a string object by passing it to the codec registered for
     _encoding_ and return the result as Python object. _encoding_ and
     _errors_ have the same meaning as the parameters of the same name
     in the string `encode()' method.  The codec to be used is looked
     up using the Python codec registry. Return _NULL_ if an exception
     was raised by the codec.

          Note: This function is not available in 3.x and does not have
          a PyBytes alias.

 -- Function: PyObject* PyString_Encode (const char *s,
          Py_ssize_t size, const char *encoding, const char *errors)
     Return value: New reference.

     Encode the `char' buffer of the given size by passing it to the
     codec registered for _encoding_ and return a Python object.
     _encoding_ and _errors_ have the same meaning as the parameters of
     the same name in the string `encode()' method. The codec to be
     used is looked up using the Python codec registry.  Return _NULL_
     if an exception was raised by the codec.

          Note: This function is not available in 3.x and does not have
          a PyBytes alias.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyString_AsEncodedObject (PyObject *str, const
          char *encoding, const char *errors)
     Return value: New reference.

     Encode a string object using the codec registered for _encoding_
     and return the result as Python object. _encoding_ and _errors_
     have the same meaning as the parameters of the same name in the
     string `encode()' method. The codec to be used is looked up using
     the Python codec registry. Return _NULL_ if an exception was
     raised by the codec.

          Note: This function is not available in 3.x and does not have
          a PyBytes alias.

7.7.3.6 Unicode Objects and Codecs
..................................

7.7.3.7 Unicode Objects
.......................

7.7.3.8 Unicode Type
....................

These are the basic Unicode object types used for the Unicode
implementation in Python:

 -- type: Py_UNICODE
     This type represents the storage type which is used by Python
     internally as basis for holding Unicode ordinals.  Python's
     default builds use a 16-bit type for Py_UNICODE and
     store Unicode values internally as UCS2. It is also possible to
     build a UCS4 version of Python (most recent Linux distributions
     come with UCS4 builds of Python). These builds then use a 32-bit
     type for Py_UNICODE and store Unicode data internally
     as UCS4. On platforms where `wchar_t' is available and compatible
     with the chosen Python Unicode build variant, see Py_UNICODE
     is a typedef alias for `wchar_t' to enhance native platform
     compatibility. On all other platforms, see Py_UNICODE is a
     typedef alias for either `unsigned short' (UCS2) or `unsigned
     long' (UCS4).

  Note that UCS2 and UCS4 Python builds are not binary compatible.
Please keep this in mind when writing extensions or interfaces.

 -- type: PyUnicodeObject
     This subtype of PyObject represents a Python Unicode
     object.

 -- var: PyTypeObject PyUnicode_Type
     This instance of PyTypeObject represents the Python
     Unicode type.  It is exposed to Python code as `unicode' and
     `types.UnicodeType'.

  The following APIs are really C macros and can be used to do fast
checks and to access internal read-only data of Unicode objects:

 -- Function: int PyUnicode_Check (PyObject *o)
     Return true if the object _o_ is a Unicode object or an instance
     of a Unicode subtype.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyUnicode_CheckExact (PyObject *o)
     Return true if the object _o_ is a Unicode object, but not an
     instance of a subtype.

     New in version 2.2.

 -- Function: Py_ssize_t PyUnicode_GET_SIZE (PyObject *o)
     Return the size of the object.  _o_ has to be a
     PyUnicodeObject. (not checked).

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: Py_ssize_t PyUnicode_GET_DATA_SIZE (PyObject *o)
     Return the size of the object's internal buffer in bytes.  _o_ has
     to be a PyUnicodeObject. (not checked).

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: Py_UNICODE* PyUnicode_AS_UNICODE (PyObject *o)
     Return a pointer to the internal Py_UNICODE buffer of
     the object.  _o_ has to be a PyUnicodeObject. (not
     checked).

 -- Function: const char* PyUnicode_AS_DATA (PyObject *o)
     Return a pointer to the internal buffer of the object. _o_ has to
     be a PyUnicodeObject. (not checked).

 -- Function: int PyUnicode_ClearFreeList ()
     Clear the free list. Return the total number of freed items.

     New in version 2.6.

7.7.3.9 Unicode Character Properties
....................................

Unicode provides many different character properties. The most often
needed ones are available through these macros which are mapped to C
functions depending on the Python configuration.

 -- Function: int Py_UNICODE_ISSPACE (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a whitespace character.

 -- Function: int Py_UNICODE_ISLOWER (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a lowercase character.

 -- Function: int Py_UNICODE_ISUPPER (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is an uppercase character.

 -- Function: int Py_UNICODE_ISTITLE (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a titlecase character.

 -- Function: int Py_UNICODE_ISLINEBREAK (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a linebreak character.

 -- Function: int Py_UNICODE_ISDECIMAL (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a decimal character.

 -- Function: int Py_UNICODE_ISDIGIT (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a digit character.

 -- Function: int Py_UNICODE_ISNUMERIC (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is a numeric character.

 -- Function: int Py_UNICODE_ISALPHA (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is an alphabetic character.

 -- Function: int Py_UNICODE_ISALNUM (Py_UNICODE ch)
     Return 1 or 0 depending on whether _ch_ is an alphanumeric
     character.

  These APIs can be used for fast direct character conversions:

 -- Function: Py_UNICODE Py_UNICODE_TOLOWER (Py_UNICODE ch)
     Return the character _ch_ converted to lower case.

 -- Function: Py_UNICODE Py_UNICODE_TOUPPER (Py_UNICODE ch)
     Return the character _ch_ converted to upper case.

 -- Function: Py_UNICODE Py_UNICODE_TOTITLE (Py_UNICODE ch)
     Return the character _ch_ converted to title case.

 -- Function: int Py_UNICODE_TODECIMAL (Py_UNICODE ch)
     Return the character _ch_ converted to a decimal positive integer.
     Return `-1' if this is not possible.  This macro does not raise
     exceptions.

 -- Function: int Py_UNICODE_TODIGIT (Py_UNICODE ch)
     Return the character _ch_ converted to a single digit integer.
     Return `-1' if this is not possible.  This macro does not raise
     exceptions.

 -- Function: double Py_UNICODE_TONUMERIC (Py_UNICODE ch)
     Return the character _ch_ converted to a double. Return `-1.0' if
     this is not possible.  This macro does not raise exceptions.

7.7.3.10 Plain Py_UNICODE
.........................

To create Unicode objects and access their basic sequence properties,
use these APIs:

 -- Function: PyObject* PyUnicode_FromUnicode (const Py_UNICODE *u,
          Py_ssize_t size)
     Return value: New reference.

     Create a Unicode object from the Py_UNICODE buffer _u_ of the
     given size. _u_ may be _NULL_ which causes the contents to be
     undefined. It is the user's responsibility to fill in the needed
     data.  The buffer is copied into the new object. If the buffer is
     not _NULL_, the return value might be a shared object.  Therefore,
     modification of the resulting Unicode object is only allowed when
     _u_ is _NULL_.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_FromStringAndSize (const char *u,
          Py_ssize_t size)
     Return value: New reference.

     Create a Unicode object from the char buffer _u_.  The bytes will
     be interpreted as being UTF-8 encoded.  _u_ may also be _NULL_
     which causes the contents to be undefined. It is the user's
     responsibility to fill in the needed data.  The buffer is copied
     into the new object. If the buffer is not _NULL_, the return value
     might be a shared object. Therefore, modification of the resulting
     Unicode object is only allowed when _u_ is _NULL_.

     New in version 2.6.

 -- Function: PyObject *PyUnicode_FromString (const char *u)
     Return value: New reference.

     Create a Unicode object from an UTF-8 encoded null-terminated char
     buffer _u_.

     New in version 2.6.

 -- Function: PyObject* PyUnicode_FromFormat (const char *format, ...)
     Return value: New reference.

     Take a C `printf()'-style _format_ string and a variable number of
     arguments, calculate the size of the resulting Python unicode
     string and return a string with the values formatted into it.  The
     variable arguments must be C types and must correspond exactly to
     the format characters in the _format_ string.  The following
     format characters are allowed:

     Format Characters       Type                      Comment
     ---------------------------------------------------------------------------------------
     `%%'                    _n/a_                     The literal % character.
     `%c'                    int                       A single character, represented as
                                                       an C int.
     `%d'                    int                       Exactly equivalent to
                                                       `printf("%d")'.
     `%u'                    unsigned int              Exactly equivalent to
                                                       `printf("%u")'.
     `%ld'                   long                      Exactly equivalent to
                                                       `printf("%ld")'.
     `%lu'                   unsigned long             Exactly equivalent to
                                                       `printf("%lu")'.
     `%zd'                   Py_ssize_t                Exactly equivalent to
                                                       `printf("%zd")'.
     `%zu'                   size_t                    Exactly equivalent to
                                                       `printf("%zu")'.
     `%i'                    int                       Exactly equivalent to
                                                       `printf("%i")'.
     `%x'                    int                       Exactly equivalent to
                                                       `printf("%x")'.
     `%s'                    char*                     A null-terminated C character array.
     `%p'                    void*                     The hex representation of a C
                                                       pointer. Mostly equivalent to
                                                       `printf("%p")' except that it is
                                                       guaranteed to start with the
                                                       literal `0x' regardless of what the
                                                       platform's `printf' yields.
     `%U'                    PyObject*                 A unicode object.
     `%V'                    PyObject*, char *         A unicode object (which may be
                                                       _NULL_) and a null-terminated C
                                                       character array as a second
                                                       parameter (which will be used, if
                                                       the first parameter is _NULL_).
     `%S'                    PyObject*                 The result of calling
                                                       `PyObject_Unicode()'.
     `%R'                    PyObject*                 The result of calling
                                                       `PyObject_Repr()'.

     An unrecognized format character causes all the rest of the format
     string to be copied as-is to the result string, and any extra
     arguments discarded.

     New in version 2.6.

 -- Function: PyObject* PyUnicode_FromFormatV (const char *format,
          va_list vargs)
     Return value: New reference.

     Identical to `PyUnicode_FromFormat()' except that it takes exactly
     two arguments.

     New in version 2.6.

 -- Function: Py_UNICODE* PyUnicode_AsUnicode (PyObject *unicode)
     Return a read-only pointer to the Unicode object's internal
     Py_UNICODE.  buffer, _NULL_ if _unicode_ is not a Unicode
     object.

 -- Function: Py_ssize_t PyUnicode_GetSize (PyObject *unicode)
     Return the length of the Unicode object.

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: PyObject* PyUnicode_FromEncodedObject (PyObject *obj,
          const char *encoding, const char *errors)
     Return value: New reference.

     Coerce an encoded object _obj_ to an Unicode object and return a
     reference with incremented refcount.

     String and other char buffer compatible objects are decoded
     according to the given encoding and using the error handling
     defined by errors.  Both can be _NULL_ to have the interface use
     the default values (see the next section for details).

     All other objects, including Unicode objects, cause a
     TypeError to be set.

     The API returns _NULL_ if there was an error.  The caller is
     responsible for decref'ing the returned objects.

 -- Function: PyObject* PyUnicode_FromObject (PyObject *obj)
     Return value: New reference.

     Shortcut for `PyUnicode_FromEncodedObject(obj, NULL, "strict")'
     which is used throughout the interpreter whenever coercion to
     Unicode is needed.

  If the platform supports `wchar_t' and provides a header file wchar.h,
Python can interface directly to this type using the following
functions.  Support is optimized if Python's own Py_UNICODE.
type is identical to the system's `wchar_t'.

7.7.3.11 wchar_t Support
........................

wchar_t support for platforms which support it:

 -- Function: PyObject* PyUnicode_FromWideChar (const wchar_t *w,
          Py_ssize_t size)
     Return value: New reference.

     Create a Unicode object from the `wchar_t' buffer _w_ of the given
     size.  Return _NULL_ on failure.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: Py_ssize_t PyUnicode_AsWideChar
          (PyUnicodeObject *unicode, wchar_t *w, Py_ssize_t size)
     Copy the Unicode object contents into the `wchar_t' buffer _w_.
     At most _size_ `wchar_t' characters are copied (excluding a
     possibly trailing 0-termination character).  Return the number of
     `wchar_t' characters copied or -1 in case of an error.  Note that
     the resulting `wchar_t' string may or may not be 0-terminated.  It
     is the responsibility of the caller to make sure that the
     `wchar_t' string is 0-terminated in case this is required by the
     application.

     Changed in version 2.5: This function returned an `int' type and
     used an `int' type for _size_. This might require changes in your
     code for properly supporting 64-bit systems.

7.7.3.12 Built-in Codecs
........................

Python provides a set of built-in codecs which are written in C for
speed. All of these codecs are directly usable via the following
functions.

  Many of the following APIs take two arguments encoding and errors.
These parameters encoding and errors have the same semantics as the
ones of the built-in unicode(). Unicode object constructor.

  Setting encoding to _NULL_ causes the default encoding to be used
which is ASCII.  The file system calls should use
`Py_FileSystemDefaultEncoding' as the encoding for file names. This
variable should be treated as read-only: On some systems, it will be a
pointer to a static string, on others, it will change at run-time (such
as when the application invokes setlocale).

  Error handling is set by errors which may also be set to _NULL_
meaning to use the default handling defined for the codec.  Default
error handling for all built-in codecs is "strict" (see ValueError
is raised).

  The codecs all use a similar interface.  Only deviation from the
following generic ones are documented for simplicity.

7.7.3.13 Generic Codecs
.......................

These are the generic codec APIs:

 -- Function: PyObject* PyUnicode_Decode (const char *s,
          Py_ssize_t size, const char *encoding, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the encoded
     string _s_.  _encoding_ and _errors_ have the same meaning as the
     parameters of the same name in the unicode() built-in
     function.  The codec to be used is looked up using the Python
     codec registry.  Return _NULL_ if an exception was raised by the
     codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_Encode (const Py_UNICODE *s,
          Py_ssize_t size, const char *encoding, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size and
     return a Python string object.  _encoding_ and _errors_ have the
     same meaning as the parameters of the same name in the Unicode
     `encode()' method.  The codec to be used is looked up using the
     Python codec registry.  Return _NULL_ if an exception was raised
     by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsEncodedString (PyObject *unicode,
          const char *encoding, const char *errors)
     Return value: New reference.

     Encode a Unicode object and return the result as Python string
     object.  _encoding_ and _errors_ have the same meaning as the
     parameters of the same name in the Unicode `encode()' method. The
     codec to be used is looked up using the Python codec registry.
     Return _NULL_ if an exception was raised by the codec.

7.7.3.14 UTF-8 Codecs
.....................

These are the UTF-8 codec APIs:

 -- Function: PyObject* PyUnicode_DecodeUTF8 (const char *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the UTF-8
     encoded string _s_. Return _NULL_ if an exception was raised by
     the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_DecodeUTF8Stateful (const char *s,
          Py_ssize_t size, const char *errors, Py_ssize_t *consumed)
     Return value: New reference.

     If _consumed_ is _NULL_, behave like PyUnicode_DecodeUTF8().
     If _consumed_ is not _NULL_, trailing incomplete UTF-8 byte
     sequences will not be treated as an error. Those bytes will not be
     decoded and the number of bytes that have been decoded will be
     stored in _consumed_.

     New in version 2.4.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeUTF8 (const Py_UNICODE *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     UTF-8 and return a Python string object.  Return _NULL_ if an
     exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsUTF8String (PyObject *unicode)
     Return value: New reference.

     Encode a Unicode object using UTF-8 and return the result as
     Python string object.  Error handling is "strict".  Return _NULL_
     if an exception was raised by the codec.

7.7.3.15 UTF-32 Codecs
......................

These are the UTF-32 codec APIs:

 -- Function: PyObject* PyUnicode_DecodeUTF32 (const char *s,
          Py_ssize_t size, const char *errors, int *byteorder)
     Decode _length_ bytes from a UTF-32 encoded buffer string and
     return the corresponding Unicode object.  _errors_ (if non-_NULL_)
     defines the error handling. It defaults to "strict".

     If _byteorder_ is non-_NULL_, the decoder starts decoding using
     the given byte order:

         *byteorder == -1: little endian
         *byteorder == 0:  native order
         *byteorder == 1:  big endian

     If `*byteorder' is zero, and the first four bytes of the input
     data are a byte order mark (BOM), the decoder switches to this
     byte order and the BOM is not copied into the resulting Unicode
     string.  If `*byteorder' is `-1' or `1', any byte order mark is
     copied to the output.

     After completion, _*byteorder_ is set to the current byte order at
     the end of input data.

     In a narrow build codepoints outside the BMP will be decoded as
     surrogate pairs.

     If _byteorder_ is _NULL_, the codec starts in native order mode.

     Return _NULL_ if an exception was raised by the codec.

     New in version 2.6.

 -- Function: PyObject* PyUnicode_DecodeUTF32Stateful (const char *s,
          Py_ssize_t size, const char *errors, int *byteorder,
          Py_ssize_t *consumed)
     If _consumed_ is _NULL_, behave like
     PyUnicode_DecodeUTF32(). If _consumed_ is not _NULL_, see
     PyUnicode_DecodeUTF32Stateful() will not treat trailing
     incomplete UTF-32 byte sequences (such as a number of bytes not
     divisible by four) as an error. Those bytes will not be decoded
     and the number of bytes that have been decoded will be stored in
     _consumed_.

     New in version 2.6.

 -- Function: PyObject* PyUnicode_EncodeUTF32 (const Py_UNICODE *s,
          Py_ssize_t size, const char *errors, int byteorder)
     Return a Python bytes object holding the UTF-32 encoded value of
     the Unicode data in _s_.  Output is written according to the
     following byte order:

         byteorder == -1: little endian
         byteorder == 0:  native byte order (writes a BOM mark)
         byteorder == 1:  big endian

     If byteorder is `0', the output string will always start with the
     Unicode BOM mark (U+FEFF). In the other two modes, no BOM mark is
     prepended.

     If _Py_UNICODE_WIDE_ is not defined, surrogate pairs will be output
     as a single codepoint.

     Return _NULL_ if an exception was raised by the codec.

     New in version 2.6.

 -- Function: PyObject* PyUnicode_AsUTF32String (PyObject *unicode)
     Return a Python string using the UTF-32 encoding in native byte
     order. The string always starts with a BOM mark.  Error handling
     is "strict".  Return _NULL_ if an exception was raised by the
     codec.

     New in version 2.6.

7.7.3.16 UTF-16 Codecs
......................

These are the UTF-16 codec APIs:

 -- Function: PyObject* PyUnicode_DecodeUTF16 (const char *s,
          Py_ssize_t size, const char *errors, int *byteorder)
     Return value: New reference.

     Decode _length_ bytes from a UTF-16 encoded buffer string and
     return the corresponding Unicode object.  _errors_ (if non-_NULL_)
     defines the error handling. It defaults to "strict".

     If _byteorder_ is non-_NULL_, the decoder starts decoding using
     the given byte order:

         *byteorder == -1: little endian
         *byteorder == 0:  native order
         *byteorder == 1:  big endian

     If `*byteorder' is zero, and the first two bytes of the input data
     are a byte order mark (BOM), the decoder switches to this byte
     order and the BOM is not copied into the resulting Unicode string.
     If `*byteorder' is `-1' or `1', any byte order mark is copied to
     the output (where it will result in either a `\ufeff' or a
     `\ufffe' character).

     After completion, _*byteorder_ is set to the current byte order at
     the end of input data.

     If _byteorder_ is _NULL_, the codec starts in native order mode.

     Return _NULL_ if an exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_DecodeUTF16Stateful (const char *s,
          Py_ssize_t size, const char *errors, int *byteorder,
          Py_ssize_t *consumed)
     Return value: New reference.

     If _consumed_ is _NULL_, behave like
     PyUnicode_DecodeUTF16(). If _consumed_ is not _NULL_, see
     PyUnicode_DecodeUTF16Stateful() will not treat trailing
     incomplete UTF-16 byte sequences (such as an odd number of bytes
     or a split surrogate pair) as an error. Those bytes will not be
     decoded and the number of bytes that have been decoded will be
     stored in _consumed_.

     New in version 2.4.

     Changed in version 2.5: This function used an `int' type for
     _size_ and an `int *' type for _consumed_. This might require
     changes in your code for properly supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeUTF16 (const Py_UNICODE *s,
          Py_ssize_t size, const char *errors, int byteorder)
     Return value: New reference.

     Return a Python string object holding the UTF-16 encoded value of
     the Unicode data in _s_.  Output is written according to the
     following byte order:

         byteorder == -1: little endian
         byteorder == 0:  native byte order (writes a BOM mark)
         byteorder == 1:  big endian

     If byteorder is `0', the output string will always start with the
     Unicode BOM mark (U+FEFF). In the other two modes, no BOM mark is
     prepended.

     If _Py_UNICODE_WIDE_ is defined, a single Py_UNICODE.
     value may get represented as a surrogate pair. If it is not
     defined, each Py_UNICODE.  values is interpreted as an
     UCS-2 character.

     Return _NULL_ if an exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsUTF16String (PyObject *unicode)
     Return value: New reference.

     Return a Python string using the UTF-16 encoding in native byte
     order. The string always starts with a BOM mark.  Error handling
     is "strict".  Return _NULL_ if an exception was raised by the
     codec.

7.7.3.17 UTF-7 Codecs
.....................

These are the UTF-7 codec APIs:

 -- Function: PyObject* PyUnicode_DecodeUTF7 (const char *s,
          Py_ssize_t size, const char *errors)
     Create a Unicode object by decoding _size_ bytes of the UTF-7
     encoded string _s_.  Return _NULL_ if an exception was raised by
     the codec.

 -- Function: PyObject* PyUnicode_DecodeUTF7Stateful (const char *s,
          Py_ssize_t size, const char *errors, Py_ssize_t *consumed)
     If _consumed_ is _NULL_, behave like PyUnicode_DecodeUTF7().
     If _consumed_ is not _NULL_, trailing incomplete UTF-7
     base-64 sections will not be treated as an error.  Those bytes
     will not be decoded and the number of bytes that have been decoded
     will be stored in _consumed_.

 -- Function: PyObject* PyUnicode_EncodeUTF7 (const Py_UNICODE *s,
          Py_ssize_t size, int base64SetO, int base64WhiteSpace, const
          char *errors)
     Encode the Py_UNICODE buffer of the given size using
     UTF-7 and return a Python bytes object.  Return _NULL_ if an
     exception was raised by the codec.

     If _base64SetO_ is nonzero, "Set O" (punctuation that has no
     otherwise special meaning) will be encoded in base-64.  If
     _base64WhiteSpace_ is nonzero, whitespace will be encoded in
     base-64.  Both are set to zero for the Python "utf-7" codec.

7.7.3.18 Unicode-Escape Codecs
..............................

These are the "Unicode Escape" codec APIs:

 -- Function: PyObject* PyUnicode_DecodeUnicodeEscape (const char *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the
     Unicode-Escape encoded string _s_.  Return _NULL_ if an exception
     was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeUnicodeEscape (const
          Py_UNICODE *s, Py_ssize_t size)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     Unicode-Escape and return a Python string object.  Return _NULL_
     if an exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsUnicodeEscapeString
          (PyObject *unicode)
     Return value: New reference.

     Encode a Unicode object using Unicode-Escape and return the result
     as Python string object.  Error handling is "strict". Return
     _NULL_ if an exception was raised by the codec.

7.7.3.19 Raw-Unicode-Escape Codecs
..................................

These are the "Raw Unicode Escape" codec APIs:

 -- Function: PyObject* PyUnicode_DecodeRawUnicodeEscape (const
          char *s, Py_ssize_t size, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the
     Raw-Unicode-Escape encoded string _s_.  Return _NULL_ if an
     exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeRawUnicodeEscape (const
          Py_UNICODE *s, Py_ssize_t size, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     Raw-Unicode-Escape and return a Python string object.  Return
     _NULL_ if an exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsRawUnicodeEscapeString
          (PyObject *unicode)
     Return value: New reference.

     Encode a Unicode object using Raw-Unicode-Escape and return the
     result as Python string object. Error handling is "strict". Return
     _NULL_ if an exception was raised by the codec.

7.7.3.20 Latin-1 Codecs
.......................

These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256
Unicode ordinals and only these are accepted by the codecs during
encoding.

 -- Function: PyObject* PyUnicode_DecodeLatin1 (const char *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the Latin-1
     encoded string _s_.  Return _NULL_ if an exception was raised by
     the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeLatin1 (const Py_UNICODE *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     Latin-1 and return a Python string object.  Return _NULL_ if an
     exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsLatin1String (PyObject *unicode)
     Return value: New reference.

     Encode a Unicode object using Latin-1 and return the result as
     Python string object.  Error handling is "strict".  Return _NULL_
     if an exception was raised by the codec.

7.7.3.21 ASCII Codecs
.....................

These are the ASCII codec APIs.  Only 7-bit ASCII data is accepted. All
other codes generate errors.

 -- Function: PyObject* PyUnicode_DecodeASCII (const char *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the ASCII
     encoded string _s_.  Return _NULL_ if an exception was raised by
     the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeASCII (const Py_UNICODE *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     ASCII and return a Python string object.  Return _NULL_ if an
     exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsASCIIString (PyObject *unicode)
     Return value: New reference.

     Encode a Unicode object using ASCII and return the result as
     Python string object.  Error handling is "strict".  Return _NULL_
     if an exception was raised by the codec.

7.7.3.22 Character Map Codecs
.............................

These are the mapping codec APIs:

  This codec is special in that it can be used to implement many
different codecs (and this is in fact what was done to obtain most of
the standard codecs included in the `encodings' package). The codec
uses mapping to encode and decode characters.

  Decoding mappings must map single string characters to single Unicode
characters, integers (which are then interpreted as Unicode ordinals)
or None (meaning "undefined mapping" and causing an error).

  Encoding mappings must map single Unicode characters to single string
characters, integers (which are then interpreted as Latin-1 ordinals)
or None (meaning "undefined mapping" and causing an error).

  The mapping objects provided must only support the __getitem__ mapping
interface.

  If a character lookup fails with a LookupError, the character is
copied as-is meaning that its ordinal value will be interpreted as
Unicode or Latin-1 ordinal resp. Because of this, mappings only need to
contain those mappings which map characters to different code points.

 -- Function: PyObject* PyUnicode_DecodeCharmap (const char *s,
          Py_ssize_t size, PyObject *mapping, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the encoded
     string _s_ using the given _mapping_ object.  Return _NULL_ if an
     exception was raised by the codec. If _mapping_ is _NULL_ latin-1
     decoding will be done. Else it can be a dictionary mapping byte or
     a unicode string, which is treated as a lookup table.  Byte values
     greater that the length of the string and U+FFFE "characters" are
     treated as "undefined mapping".

     Changed in version 2.4: Allowed unicode string as mapping argument.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_EncodeCharmap (const Py_UNICODE *s,
          Py_ssize_t size, PyObject *mapping, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     the given _mapping_ object and return a Python string object.
     Return _NULL_ if an exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsCharmapString (PyObject *unicode,
          PyObject *mapping)
     Return value: New reference.

     Encode a Unicode object using the given _mapping_ object and
     return the result as Python string object.  Error handling is
     "strict".  Return _NULL_ if an exception was raised by the codec.

  The following codec API is special in that maps Unicode to Unicode.

 -- Function: PyObject* PyUnicode_TranslateCharmap (const
          Py_UNICODE *s, Py_ssize_t size, PyObject *table, const
          char *errors)
     Return value: New reference.

     Translate a Py_UNICODE buffer of the given length by
     applying a character mapping _table_ to it and return the
     resulting Unicode object.  Return _NULL_ when an exception was
     raised by the codec.

     The _mapping_ table must map Unicode ordinal integers to Unicode
     ordinal integers or None (causing deletion of the character).

     Mapping tables need only provide the __getitem__().
     interface; dictionaries and sequences work well.  Unmapped
     character ordinals (ones which cause a LookupError)
     are left untouched and are copied as-is.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

  These are the MBCS codec APIs. They are currently only available on
Windows and use the Win32 MBCS converters to implement the conversions.
Note that MBCS (or DBCS) is a class of encodings, not just one.  The
target encoding is defined by the user settings on the machine running
the codec.

7.7.3.23 MBCS codecs for Windows
................................

 -- Function: PyObject* PyUnicode_DecodeMBCS (const char *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Create a Unicode object by decoding _size_ bytes of the MBCS
     encoded string _s_.  Return _NULL_ if an exception was raised by
     the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_DecodeMBCSStateful (const char *s,
          int size, const char *errors, int *consumed)
     If _consumed_ is _NULL_, behave like PyUnicode_DecodeMBCS().
     If _consumed_ is not _NULL_, see
     PyUnicode_DecodeMBCSStateful() will not decode trailing
     lead byte and the number of bytes that have been decoded will be
     stored in _consumed_.

     New in version 2.5.

 -- Function: PyObject* PyUnicode_EncodeMBCS (const Py_UNICODE *s,
          Py_ssize_t size, const char *errors)
     Return value: New reference.

     Encode the Py_UNICODE buffer of the given size using
     MBCS and return a Python string object.  Return _NULL_ if an
     exception was raised by the codec.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_AsMBCSString (PyObject *unicode)
     Return value: New reference.

     Encode a Unicode object using MBCS and return the result as Python
     string object.  Error handling is "strict".  Return _NULL_ if an
     exception was raised by the codec.

7.7.3.24 Methods & Slots
........................

7.7.3.25 Methods and Slot Functions
...................................

The following APIs are capable of handling Unicode objects and strings
on input (we refer to them as strings in the descriptions) and return
Unicode objects or integers as appropriate.

  They all return _NULL_ or `-1' if an exception occurs.

 -- Function: PyObject* PyUnicode_Concat (PyObject *left,
          PyObject *right)
     Return value: New reference.

     Concat two strings giving a new Unicode string.

 -- Function: PyObject* PyUnicode_Split (PyObject *s, PyObject *sep,
          Py_ssize_t maxsplit)
     Return value: New reference.

     Split a string giving a list of Unicode strings.  If sep is
     _NULL_, splitting will be done at all whitespace substrings.
     Otherwise, splits occur at the given separator.  At most
     _maxsplit_ splits will be done.  If negative, no limit is set.
     Separators are not included in the resulting list.

     Changed in version 2.5: This function used an `int' type for
     _maxsplit_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_Splitlines (PyObject *s, int keepend)
     Return value: New reference.

     Split a Unicode string at line breaks, returning a list of Unicode
     strings.  CRLF is considered to be one line break.  If _keepend_
     is 0, the Line break characters are not included in the resulting
     strings.

 -- Function: PyObject* PyUnicode_Translate (PyObject *str,
          PyObject *table, const char *errors)
     Return value: New reference.

     Translate a string by applying a character mapping table to it and
     return the resulting Unicode object.

     The mapping table must map Unicode ordinal integers to Unicode
     ordinal integers or None (causing deletion of the character).

     Mapping tables need only provide the __getitem__().
     interface; dictionaries and sequences work well.  Unmapped
     character ordinals (ones which cause a LookupError)
     are left untouched and are copied as-is.

     _errors_ has the usual meaning for codecs. It may be _NULL_ which
     indicates to use the default error handling.

 -- Function: PyObject* PyUnicode_Join (PyObject *separator,
          PyObject *seq)
     Return value: New reference.

     Join a sequence of strings using the given separator and return
     the resulting Unicode string.

 -- Function: int PyUnicode_Tailmatch (PyObject *str, PyObject *substr,
          Py_ssize_t start, Py_ssize_t end, int direction)
     Return 1 if _substr_ matches _str*[*start_:_end_] at the given
     tail end (_direction_ == -1 means to do a prefix match,
     _direction_ == 1 a suffix match), 0 otherwise. Return `-1' if an
     error occurred.

     Changed in version 2.5: This function used an `int' type for
     _start_ and _end_. This might require changes in your code for
     properly supporting 64-bit systems.

 -- Function: Py_ssize_t PyUnicode_Find (PyObject *str,
          PyObject *substr, Py_ssize_t start, Py_ssize_t end,
          int direction)
     Return the first position of _substr_ in _str*[*start_:_end_]
     using the given _direction_ (_direction_ == 1 means to do a
     forward search, _direction_ == -1 a backward search).  The return
     value is the index of the first match; a value of `-1' indicates
     that no match was found, and `-2' indicates that an error occurred
     and an exception has been set.

     Changed in version 2.5: This function used an `int' type for
     _start_ and _end_. This might require changes in your code for
     properly supporting 64-bit systems.

 -- Function: Py_ssize_t PyUnicode_Count (PyObject *str,
          PyObject *substr, Py_ssize_t start, Py_ssize_t end)
     Return the number of non-overlapping occurrences of _substr_ in
     `str[start:end]'.  Return `-1' if an error occurred.

     Changed in version 2.5: This function returned an `int' type and
     used an `int' type for _start_ and _end_. This might require
     changes in your code for properly supporting 64-bit systems.

 -- Function: PyObject* PyUnicode_Replace (PyObject *str,
          PyObject *substr, PyObject *replstr, Py_ssize_t maxcount)
     Return value: New reference.

     Replace at most _maxcount_ occurrences of _substr_ in _str_ with
     _replstr_ and return the resulting Unicode object. _maxcount_ ==
     -1 means replace all occurrences.

     Changed in version 2.5: This function used an `int' type for
     _maxcount_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyUnicode_Compare (PyObject *left, PyObject *right)
     Compare two strings and return -1, 0, 1 for less than, equal, and
     greater than, respectively.

 -- Function: int PyUnicode_RichCompare (PyObject *left,
          PyObject *right, int op)
     Rich compare two unicode strings and return one of the following:

        * `NULL' in case an exception was raised

        * `Py_True' or `Py_False' for successful comparisons

        * `Py_NotImplemented' in case the type combination is unknown

     Note that `Py_EQ' and `Py_NE' comparisons can cause a
     UnicodeWarning in case the conversion of the arguments to
     Unicode fails with a UnicodeDecodeError.

     Possible values for _op_ are `Py_GT', `Py_GE', `Py_EQ', `Py_NE',
     `Py_LT', and `Py_LE'.

 -- Function: PyObject* PyUnicode_Format (PyObject *format,
          PyObject *args)
     Return value: New reference.

     Return a new string object from _format_ and _args_; this is
     analogous to `format % args'.  The _args_ argument must be a tuple.

 -- Function: int PyUnicode_Contains (PyObject *container,
          PyObject *element)
     Check whether _element_ is contained in _container_ and return
     true or false accordingly.

     _element_ has to coerce to a one element Unicode string. `-1' is
     returned if there was an error.

7.7.3.26 Buffers and Memoryview Objects
.......................................

Python objects implemented in C can export a group of functions called
the "buffer interface."  These functions can be used by an object to
expose its data in a raw, byte-oriented format. Clients of the object
can use the buffer interface to access the object data directly,
without needing to copy it first.

  Two examples of objects that support the buffer interface are strings
and arrays. The string object exposes the character contents in the
buffer interface's byte-oriented form. An array can also expose its
contents, but it should be noted that array elements may be multi-byte
values.

  An example user of the buffer interface is the file object's `write()'
method. Any object that can export a series of bytes through the buffer
interface can be written to a file. There are a number of format codes
to PyArg_ParseTuple() that operate against an object's
buffer interface, returning data from the target object.

  Starting from version 1.6, Python has been providing Python-level
buffer objects and a C-level buffer API so that any built-in or
used-defined type can expose its characteristics. Both, however, have
been deprecated because of various shortcomings, and have been
officially removed in Python 3.0 in favour of a new C-level buffer API
and a new Python-level object named memoryview.

  The new buffer API has been backported to Python 2.6, and the
memoryview object has been backported to Python 2.7. It is
strongly advised to use them rather than the old APIs, unless you are
blocked from doing so for compatibility reasons.

7.7.3.27 The new-style Py_buffer struct
.......................................

 -- type: Py_buffer
      -- member: void *buf
          A pointer to the start of the memory for the object.

      -- member: Py_ssize_t len
          The total length of the memory in bytes.

      -- member: int readonly
          An indicator of whether the buffer is read only.

      -- member: const char *format
          A _NULL_ terminated string in struct module style
          syntax giving the contents of the elements available through
          the buffer.  If this is _NULL_, `"B"' (unsigned bytes) is
          assumed.

      -- member: int ndim
          The number of dimensions the memory represents as a
          multi-dimensional array.  If it is 0, `strides' and
          `suboffsets' must be _NULL_.

      -- member: Py_ssize_t *shape
          An array of `Py_ssize_t's the length of `ndim' giving the
          shape of the memory as a multi-dimensional array.  Note that
          `((*shape)[0] * ... * (*shape)[ndims-1])*itemsize' should be
          equal to `len'.

      -- member: Py_ssize_t *strides
          An array of `Py_ssize_t's the length of `ndim' giving the
          number of bytes to skip to get to a new element in each
          dimension.

      -- member: Py_ssize_t *suboffsets
          An array of `Py_ssize_t's the length of `ndim'.  If these
          suboffset numbers are greater than or equal to 0, then the
          value stored along the indicated dimension is a pointer and
          the suboffset value dictates how many bytes to add to the
          pointer after de-referencing. A suboffset value that it
          negative indicates that no de-referencing should occur
          (striding in a contiguous memory block).

          Here is a function that returns a pointer to the element in
          an N-D array pointed to by an N-dimesional index when there
          are both non-NULL strides and suboffsets:

              void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
                  Py_ssize_t *suboffsets, Py_ssize_t *indices) {
                  char *pointer = (char*)buf;
                  int i;
                  for (i = 0; i < ndim; i++) {
                      pointer += strides[i] * indices[i];
                      if (suboffsets[i] >=0 ) {
                          pointer = *((char**)pointer) + suboffsets[i];
                      }
                  }
                  return (void*)pointer;
               }

      -- member: Py_ssize_t itemsize
          This is a storage for the itemsize (in bytes) of each element
          of the shared memory. It is technically un-necessary as it
          can be obtained using PyBuffer_SizeFromFormat(),
          however an exporter may know this information without parsing
          the format string and it is necessary to know the itemsize
          for proper interpretation of striding. Therefore, storing it
          is more convenient and faster.

      -- member: void *internal
          This is for use internally by the exporting object. For
          example, this might be re-cast as an integer by the exporter
          and used to store flags about whether or not the shape,
          strides, and suboffsets arrays must be freed when the buffer
          is released. The consumer should never alter this value.

7.7.3.28 Buffer related functions
.................................

 -- Function: int PyObject_CheckBuffer (PyObject *obj)
     Return 1 if _obj_ supports the buffer interface otherwise 0.

 -- Function: int PyObject_GetBuffer (PyObject *obj, Py_buffer *view,
          int flags)
     Export _obj_ into a Py_buffer, _view_.  These
     arguments must never be _NULL_.  The _flags_ argument is a bit
     field indicating what kind of buffer the caller is prepared to
     deal with and therefore what kind of buffer the exporter is
     allowed to return.  The buffer interface allows for complicated
     memory sharing possibilities, but some caller may not be able to
     handle all the complexity but may want to see if the exporter will
     let them take a simpler view to its memory.

     Some exporters may not be able to share memory in every possible
     way and may need to raise errors to signal to some consumers that
     something is just not possible. These errors should be a
     `BufferError' unless there is another error that is actually
     causing the problem. The exporter can use flags information to
     simplify how much of the Py_buffer structure is
     filled in with non-default values and/or raise an error if the
     object can't support a simpler view of its memory.

     0 is returned on success and -1 on error.

     The following table gives possible values to the _flags_ arguments.

     Flag                               Description
     -------------------------------------------------------------------------------------------
     `PyBUF_SIMPLE'                     This is the default flag state.  The returned buffer
                                        may or may not have writable memory.  The format of
                                        the data will be assumed to be unsigned bytes.  This
                                        is a "stand-alone" flag constant. It never needs to be
                                        '|'d to the others. The exporter will raise an error
                                        if it cannot provide such a contiguous buffer of bytes.
     `PyBUF_WRITABLE'                   The returned buffer must be writable.  If it is not
                                        writable, then raise an error.
     `PyBUF_STRIDES'                    This implies `PyBUF_ND'. The returned buffer must
                                        provide strides information (i.e. the strides cannot
                                        be NULL). This would be used when the consumer can
                                        handle strided, discontiguous arrays.  Handling
                                        strides automatically assumes you can handle shape.
                                        The exporter can raise an error if a strided
                                        representation of the data is not possible (i.e.
                                        without the suboffsets).
     `PyBUF_ND'                         The returned buffer must provide shape information.
                                        The memory will be assumed C-style contiguous (last
                                        dimension varies the fastest). The exporter may raise
                                        an error if it cannot provide this kind of contiguous
                                        buffer. If this is not given then shape will be _NULL_.
     `PyBUF_C_CONTIGUOUS'               These flags indicate that the contiguity returned
     `PyBUF_F_CONTIGUOUS'               buffer must be respectively, C-contiguous (last
     `PyBUF_ANY_CONTIGUOUS'             dimension varies the fastest), Fortran contiguous
                                        (first dimension varies the fastest) or either one.
                                        All of these flags imply `PyBUF_STRIDES' and guarantee
                                        that the strides buffer info structure will be filled
                                        in correctly.
     `PyBUF_INDIRECT'                   This flag indicates the returned buffer must have
                                        suboffsets information (which can be NULL if no
                                        suboffsets are needed).  This can be used when the
                                        consumer can handle indirect array referencing implied
                                        by these suboffsets. This implies `PyBUF_STRIDES'.
     `PyBUF_FORMAT'                     The returned buffer must have true format information
                                        if this flag is provided. This would be used when the
                                        consumer is going to be checking for what 'kind' of
                                        data is actually stored. An exporter should always be
                                        able to provide this information if requested. If
                                        format is not explicitly requested then the format
                                        must be returned as _NULL_ (which means `'B'', or
                                        unsigned bytes)
     `PyBUF_STRIDED'                    This is equivalent to `(PyBUF_STRIDES |
                                        PyBUF_WRITABLE)'.
     `PyBUF_STRIDED_RO'                 This is equivalent to `(PyBUF_STRIDES)'.
     `PyBUF_RECORDS'                    This is equivalent to `(PyBUF_STRIDES | PyBUF_FORMAT |
                                        PyBUF_WRITABLE)'.
     `PyBUF_RECORDS_RO'                 This is equivalent to `(PyBUF_STRIDES | PyBUF_FORMAT)'.
     `PyBUF_FULL'                       This is equivalent to `(PyBUF_INDIRECT | PyBUF_FORMAT
                                        | PyBUF_WRITABLE)'.
     `PyBUF_FULL_RO'                    This is equivalent to `(PyBUF_INDIRECT |
                                        PyBUF_FORMAT)'.
     `PyBUF_CONTIG'                     This is equivalent to `(PyBUF_ND | PyBUF_WRITABLE)'.
     `PyBUF_CONTIG_RO'                  This is equivalent to `(PyBUF_ND)'.

 -- Function: void PyBuffer_Release (Py_buffer *view)
     Release the buffer _view_.  This should be called when the buffer
     is no longer being used as it may free memory from it.

 -- Function: Py_ssize_t PyBuffer_SizeFromFormat (const char *)
     Return the implied itemsize from the struct-stype
     `format'.

 -- Function: int PyObject_CopyToObject (PyObject *obj, void *buf,
          Py_ssize_t len, char fortran)
     Copy _len_ bytes of data pointed to by the contiguous chunk of
     memory pointed to by _buf_ into the buffer exported by obj.  The
     buffer must of course be writable.  Return 0 on success and return
     -1 and raise an error on failure.  If the object does not have a
     writable buffer, then an error is raised.  If _fortran_ is `'F'',
     then if the object is multi-dimensional, then the data will be
     copied into the array in Fortran-style (first dimension varies the
     fastest).  If _fortran_ is `'C'', then the data will be copied
     into the array in C-style (last dimension varies the fastest).  If
     _fortran_ is `'A'', then it does not matter and the copy will be
     made in whatever way is more efficient.

 -- Function: int PyBuffer_IsContiguous (Py_buffer *view, char fortran)
     Return 1 if the memory defined by the _view_ is C-style (_fortran_
     is `'C'') or Fortran-style (_fortran_ is `'F'') contiguous or
     either one (_fortran_ is `'A'').  Return 0 otherwise.

 -- Function: void PyBuffer_FillContiguousStrides (int ndim,
          Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize,
          char fortran)
     Fill the _strides_ array with byte-strides of a contiguous
     (C-style if _fortran_ is `'C'' or Fortran-style if _fortran_ is
     `'F'' array of the given shape with the given number of bytes per
     element.

 -- Function: int PyBuffer_FillInfo (Py_buffer *view, PyObject *obj,
          void *buf, Py_ssize_t len, int readonly, int infoflags)
     Fill in a buffer-info structure, _view_, correctly for an exporter
     that can only share a contiguous chunk of memory of "unsigned
     bytes" of the given length.  Return 0 on success and -1 (with
     raising an error) on error.

7.7.3.29 MemoryView objects
...........................

New in version 2.7.

  A memoryview object exposes the new C level buffer
interface as a Python object which can then be passed around like any
other object.

 -- Function: PyObject *PyMemoryView_FromObject (PyObject *obj)
     Create a memoryview object from an object that defines the new
     buffer interface.

 -- Function: PyObject *PyMemoryView_FromBuffer (Py_buffer *view)
     Create a memoryview object wrapping the given buffer-info
     structure _view_.  The memoryview object then owns the buffer,
     which means you shouldn't try to release it yourself: it will be
     released on deallocation of the memoryview object.

 -- Function: PyObject *PyMemoryView_GetContiguous (PyObject *obj,
          int buffertype, char order)
     Create a memoryview object to a contiguous chunk of memory (in
     either 'C' or 'F'ortran _order_) from an object that defines the
     buffer interface. If memory is contiguous, the memoryview object
     points to the original memory. Otherwise copy is made and the
     memoryview points to a new bytes object.

 -- Function: int PyMemoryView_Check (PyObject *obj)
     Return true if the object _obj_ is a memoryview object.  It is not
     currently allowed to create subclasses of memoryview.

 -- Function: Py_buffer *PyMemoryView_GET_BUFFER (PyObject *obj)
     Return a pointer to the buffer-info structure wrapped by the given
     object.  The object *must* be a memoryview instance; this macro
     doesn't check its type, you must do it yourself or you will risk
     crashes.

7.7.3.30 Old-style buffer objects
.................................

More information on the old buffer interface is provided in the section
Buffer Object Structures, under the description for
PyBufferProcs.

  A "buffer object" is defined in the `bufferobject.h' header (included
by `Python.h'). These objects look very similar to string objects at the
Python programming level: they support slicing, indexing,
concatenation, and some other standard string operations. However,
their data can come from one of two sources: from a block of memory, or
from another object which exports the buffer interface.

  Buffer objects are useful as a way to expose the data from another
object's buffer interface to the Python programmer. They can also be
used as a zero-copy slicing mechanism. Using their ability to reference
a block of memory, it is possible to expose any data to the Python
programmer quite easily. The memory could be a large, constant array in
a C extension, it could be a raw block of memory for manipulation
before passing to an operating system library, or it could be used to
pass around structured data in its native, in-memory format.

 -- type: PyBufferObject
     This subtype of PyObject represents a buffer object.

 -- var: PyTypeObject PyBuffer_Type
     The instance of PyTypeObject which represents the
     Python buffer type; it is the same object as `buffer' and
     `types.BufferType' in the Python layer. .

 -- var: int Py_END_OF_BUFFER
     This constant may be passed as the _size_ parameter to
     PyBuffer_FromObject() or
     PyBuffer_FromReadWriteObject().  It indicates that the new
     PyBufferObject should refer to _base_ object from the
     specified _offset_ to the end of its exported buffer.  Using this
     enables the caller to avoid querying the _base_ object for its
     length.

 -- Function: int PyBuffer_Check (PyObject *p)
     Return true if the argument has type PyBuffer_Type.

 -- Function: PyObject* PyBuffer_FromObject (PyObject *base,
          Py_ssize_t offset, Py_ssize_t size)
     Return value: New reference.

     Return a new read-only buffer object.  This raises
     TypeError if _base_ doesn't support the read-only buffer
     protocol or doesn't provide exactly one buffer segment, or it
     raises ValueError if _offset_ is less than zero.  The
     buffer will hold a reference to the _base_ object, and the
     buffer's contents will refer to the _base_ object's buffer
     interface, starting as position _offset_ and extending for _size_
     bytes. If _size_ is `Py_END_OF_BUFFER', then the new buffer's
     contents extend to the length of the _base_ object's exported
     buffer data.

     Changed in version 2.5: This function used an `int' type for
     _offset_ and _size_. This might require changes in your code for
     properly supporting 64-bit systems.

 -- Function: PyObject* PyBuffer_FromReadWriteObject (PyObject *base,
          Py_ssize_t offset, Py_ssize_t size)
     Return value: New reference.

     Return a new writable buffer object.  Parameters and exceptions
     are similar to those for PyBuffer_FromObject().  If
     the _base_ object does not export the writeable buffer protocol,
     then TypeError is raised.

     Changed in version 2.5: This function used an `int' type for
     _offset_ and _size_. This might require changes in your code for
     properly supporting 64-bit systems.

 -- Function: PyObject* PyBuffer_FromMemory (void *ptr, Py_ssize_t size)
     Return value: New reference.

     Return a new read-only buffer object that reads from a specified
     location in memory, with a specified size.  The caller is
     responsible for ensuring that the memory buffer, passed in as
     _ptr_, is not deallocated while the returned buffer object exists.
     Raises ValueError if _size_ is less than zero.  Note
     that `Py_END_OF_BUFFER' may _not_ be passed for the _size_
     parameter; see ValueError will be raised in that case.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyBuffer_FromReadWriteMemory (void *ptr,
          Py_ssize_t size)
     Return value: New reference.

     Similar to PyBuffer_FromMemory(), but the returned
     buffer is writable.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyBuffer_New (Py_ssize_t size)
     Return value: New reference.

     Return a new writable buffer object that maintains its own memory
     buffer of _size_ bytes.  ValueError is returned if
     _size_ is not zero or positive.  Note that the memory buffer (as
     returned by PyObject_AsWriteBuffer()) is not
     specifically aligned.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

7.7.3.31 Tuple Objects
......................

 -- type: PyTupleObject
     This subtype of PyObject represents a Python tuple
     object.

 -- var: PyTypeObject PyTuple_Type
     This instance of PyTypeObject represents the Python
     tuple type; it is the same object as `tuple' and `types.TupleType'
     in the Python layer..

 -- Function: int PyTuple_Check (PyObject *p)
     Return true if _p_ is a tuple object or an instance of a subtype
     of the tuple type.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyTuple_CheckExact (PyObject *p)
     Return true if _p_ is a tuple object, but not an instance of a
     subtype of the tuple type.

     New in version 2.2.

 -- Function: PyObject* PyTuple_New (Py_ssize_t len)
     Return value: New reference.

     Return a new tuple object of size _len_, or _NULL_ on failure.

     Changed in version 2.5: This function used an `int' type for
     _len_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyTuple_Pack (Py_ssize_t n, ...)
     Return value: New reference.

     Return a new tuple object of size _n_, or _NULL_ on failure. The
     tuple values are initialized to the subsequent _n_ C arguments
     pointing to Python objects.  `PyTuple_Pack(2, a, b)' is equivalent
     to `Py_BuildValue("(OO)", a, b)'.

     New in version 2.4.

     Changed in version 2.5: This function used an `int' type for _n_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: Py_ssize_t PyTuple_Size (PyObject *p)
     Take a pointer to a tuple object, and return the size of that
     tuple.

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: Py_ssize_t PyTuple_GET_SIZE (PyObject *p)
     Return the size of the tuple _p_, which must be non-_NULL_ and
     point to a tuple; no error checking is performed.

     Changed in version 2.5: This function returned an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: PyObject* PyTuple_GetItem (PyObject *p, Py_ssize_t pos)
     Return value: Borrowed reference.

     Return the object at position _pos_ in the tuple pointed to by
     _p_.  If _pos_ is out of bounds, return _NULL_ and sets an
     IndexError exception.

     Changed in version 2.5: This function used an `int' type for
     _pos_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyTuple_GET_ITEM (PyObject *p, Py_ssize_t pos)
     Return value: Borrowed reference.

     Like PyTuple_GetItem(), but does no checking of its
     arguments.

     Changed in version 2.5: This function used an `int' type for
     _pos_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: PyObject* PyTuple_GetSlice (PyObject *p, Py_ssize_t low,
          Py_ssize_t high)
     Return value: New reference.

     Take a slice of the tuple pointed to by _p_ from _low_ to _high_
     and return it as a new tuple.

     Changed in version 2.5: This function used an `int' type for _low_
     and _high_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyTuple_SetItem (PyObject *p, Py_ssize_t pos,
          PyObject *o)
     Insert a reference to object _o_ at position _pos_ of the tuple
     pointed to by _p_. Return `0' on success.

          Note: This function "steals" a reference to _o_.

     Changed in version 2.5: This function used an `int' type for
     _pos_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: void PyTuple_SET_ITEM (PyObject *p, Py_ssize_t pos,
          PyObject *o)
     Like PyTuple_SetItem(), but does no error checking,
     and should _only_ be used to fill in brand new tuples.

          Note: This function "steals" a reference to _o_.

     Changed in version 2.5: This function used an `int' type for
     _pos_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int _PyTuple_Resize (PyObject **p, Py_ssize_t newsize)
     Can be used to resize a tuple.  _newsize_ will be the new length
     of the tuple.  Because tuples are _supposed_ to be immutable, this
     should only be used if there is only one reference to the object.
     Do _not_ use this if the tuple may already be known to some other
     part of the code.  The tuple will always grow or shrink at the
     end.  Think of this as destroying the old tuple and creating a new
     one, only more efficiently.  Returns `0' on success. Client code
     should never assume that the resulting value of `*p' will be the
     same as before calling this function. If the object referenced by
     `*p' is replaced, the original `*p' is destroyed.  On failure,
     returns `-1' and sets `*p' to _NULL_, and raises
     MemoryError or SystemError.

     Changed in version 2.2: Removed unused third parameter,
     _last_is_sticky_.

     Changed in version 2.5: This function used an `int' type for
     _newsize_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyTuple_ClearFreeList ()
     Clear the free list. Return the total number of freed items.

     New in version 2.6.

7.7.3.32 List Objects
.....................

 -- type: PyListObject
     This subtype of PyObject represents a Python list
     object.

 -- var: PyTypeObject PyList_Type
     This instance of PyTypeObject represents the Python
     list type.  This is the same object as `list' in the Python layer.

 -- Function: int PyList_Check (PyObject *p)
     Return true if _p_ is a list object or an instance of a subtype of
     the list type.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyList_CheckExact (PyObject *p)
     Return true if _p_ is a list object, but not an instance of a
     subtype of the list type.

     New in version 2.2.

 -- Function: PyObject* PyList_New (Py_ssize_t len)
     Return value: New reference.

     Return a new list of length _len_ on success, or _NULL_ on failure.

          Note: If _length_ is greater than zero, the returned list
          object's items are set to `NULL'.  Thus you cannot use
          abstract API functions such as PySequence_SetItem().
          or expose the object to Python code before setting all
          items to a real object with PyList_SetItem().

     Changed in version 2.5: This function used an `int' for _size_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: Py_ssize_t PyList_Size (PyObject *list)
     Return the length of the list object in _list_; this is equivalent
     to `len(list)' on a list object.

     Changed in version 2.5: This function returned an `int'. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: Py_ssize_t PyList_GET_SIZE (PyObject *list)
     Macro form of PyList_Size() without error checking.

     Changed in version 2.5: This macro returned an `int'. This might
     require changes in your code for properly supporting 64-bit
     systems.

 -- Function: PyObject* PyList_GetItem (PyObject *list,
          Py_ssize_t index)
     Return value: Borrowed reference.

     Return the object at position _pos_ in the list pointed to by _p_.
     The position must be positive, indexing from the end of the list
     is not supported.  If _pos_ is out of bounds, return _NULL_ and
     set an IndexError exception.

     Changed in version 2.5: This function used an `int' for _index_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: PyObject* PyList_GET_ITEM (PyObject *list, Py_ssize_t i)
     Return value: Borrowed reference.

     Macro form of PyList_GetItem() without error checking.

     Changed in version 2.5: This macro used an `int' for _i_. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: int PyList_SetItem (PyObject *list, Py_ssize_t index,
          PyObject *item)
     Set the item at index _index_ in list to _item_.  Return `0' on
     success or `-1' on failure.

          Note: This function "steals" a reference to _item_ and
          discards a reference to an item already in the list at the
          affected position.

     Changed in version 2.5: This function used an `int' for _index_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: void PyList_SET_ITEM (PyObject *list, Py_ssize_t i,
          PyObject *o)
     Macro form of PyList_SetItem() without error
     checking. This is normally only used to fill in new lists where
     there is no previous content.

          Note: This macro "steals" a reference to _item_, and, unlike
          PyList_SetItem(), does _not_ discard a reference
          to any item that it being replaced; any reference in _list_
          at position _i_ will be leaked.

     Changed in version 2.5: This macro used an `int' for _i_. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: int PyList_Insert (PyObject *list, Py_ssize_t index,
          PyObject *item)
     Insert the item _item_ into list _list_ in front of index _index_.
     Return `0' if successful; return `-1' and set an exception if
     unsuccessful.  Analogous to `list.insert(index, item)'.

     Changed in version 2.5: This function used an `int' for _index_.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: int PyList_Append (PyObject *list, PyObject *item)
     Append the object _item_ at the end of list _list_. Return `0' if
     successful; return `-1' and set an exception if unsuccessful.
     Analogous to `list.append(item)'.

 -- Function: PyObject* PyList_GetSlice (PyObject *list,
          Py_ssize_t low, Py_ssize_t high)
     Return value: New reference.

     Return a list of the objects in _list_ containing the objects
     _between_ _low_ and _high_.  Return _NULL_ and set an exception if
     unsuccessful.  Analogous to `list[low:high]'.  Negative indices,
     as when slicing from Python, are not supported.

     Changed in version 2.5: This function used an `int' for _low_ and
     _high_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyList_SetSlice (PyObject *list, Py_ssize_t low,
          Py_ssize_t high, PyObject *itemlist)
     Set the slice of _list_ between _low_ and _high_ to the contents of
     _itemlist_.  Analogous to `list[low:high] = itemlist'. The
     _itemlist_ may be _NULL_, indicating the assignment of an empty
     list (slice deletion).  Return `0' on success, `-1' on failure.
     Negative indices, as when slicing from Python, are not supported.

     Changed in version 2.5: This function used an `int' for _low_ and
     _high_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyList_Sort (PyObject *list)
     Sort the items of _list_ in place.  Return `0' on success, `-1' on
     failure.  This is equivalent to `list.sort()'.

 -- Function: int PyList_Reverse (PyObject *list)
     Reverse the items of _list_ in place.  Return `0' on success, `-1'
     on failure.  This is the equivalent of `list.reverse()'.

 -- Function: PyObject* PyList_AsTuple (PyObject *list)
     Return value: New reference.

     Return a new tuple object containing the contents of _list_;
     equivalent to `tuple(list)'.

7.7.4 Mapping Objects
---------------------

7.7.4.1 Dictionary Objects
..........................

 -- type: PyDictObject
     This subtype of PyObject represents a Python
     dictionary object.

 -- var: PyTypeObject PyDict_Type
     This instance of PyTypeObject represents the Python
     dictionary type.  This is exposed to Python programs as `dict' and
     `types.DictType'.

 -- Function: int PyDict_Check (PyObject *p)
     Return true if _p_ is a dict object or an instance of a subtype of
     the dict type.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyDict_CheckExact (PyObject *p)
     Return true if _p_ is a dict object, but not an instance of a
     subtype of the dict type.

     New in version 2.4.

 -- Function: PyObject* PyDict_New ()
     Return value: New reference.

     Return a new empty dictionary, or _NULL_ on failure.

 -- Function: PyObject* PyDictProxy_New (PyObject *dict)
     Return value: New reference.

     Return a proxy object for a mapping which enforces read-only
     behavior.  This is normally used to create a proxy to prevent
     modification of the dictionary for non-dynamic class types.

     New in version 2.2.

 -- Function: void PyDict_Clear (PyObject *p)
     Empty an existing dictionary of all key-value pairs.

 -- Function: int PyDict_Contains (PyObject *p, PyObject *key)
     Determine if dictionary _p_ contains _key_.  If an item in _p_ is
     matches _key_, return `1', otherwise return `0'.  On error, return
     `-1'.  This is equivalent to the Python expression `key in p'.

     New in version 2.4.

 -- Function: PyObject* PyDict_Copy (PyObject *p)
     Return value: New reference.

     Return a new dictionary that contains the same key-value pairs as
     _p_.

     New in version 1.6.

 -- Function: int PyDict_SetItem (PyObject *p, PyObject *key,
          PyObject *val)
     Insert _value_ into the dictionary _p_ with a key of _key_.  _key_
     must be hashable.; if it isn't, see TypeError.
     will be raised. Return `0' on success or `-1' on failure.

 -- Function: int PyDict_SetItemString (PyObject *p, const char *key,
          PyObject *val)
     Insert _value_ into the dictionary _p_ using _key_ as a key. _key_
     should be a `char*'.  The key object is created using
     `PyString_FromString(key)'.  Return `0' on success or `-1' on
     failure.

 -- Function: int PyDict_DelItem (PyObject *p, PyObject *key)
     Remove the entry in dictionary _p_ with key _key_. _key_ must be
     hashable; if it isn't, see TypeError is raised.  Return
     `0' on success or `-1' on failure.

 -- Function: int PyDict_DelItemString (PyObject *p, char *key)
     Remove the entry in dictionary _p_ which has a key specified by
     the string _key_.  Return `0' on success or `-1' on failure.

 -- Function: PyObject* PyDict_GetItem (PyObject *p, PyObject *key)
     Return value: Borrowed reference.

     Return the object from dictionary _p_ which has a key _key_.
     Return _NULL_ if the key _key_ is not present, but _without_
     setting an exception.

 -- Function: PyObject* PyDict_GetItemString (PyObject *p, const
          char *key)
     Return value: Borrowed reference.

     This is the same as PyDict_GetItem(), but _key_ is
     specified as a `char*', rather than a PyObject*.

 -- Function: PyObject* PyDict_Items (PyObject *p)
     Return value: New reference.

     Return a PyListObject containing all the items from
     the dictionary, as in the dictionary method dict.items().

 -- Function: PyObject* PyDict_Keys (PyObject *p)
     Return value: New reference.

     Return a PyListObject containing all the keys from
     the dictionary, as in the dictionary method dict.keys().

 -- Function: PyObject* PyDict_Values (PyObject *p)
     Return value: New reference.

     Return a PyListObject containing all the values from
     the dictionary _p_, as in the dictionary method
     dict.values().

 -- Function: Py_ssize_t PyDict_Size (PyObject *p)
     Return the number of items in the dictionary.  This is equivalent
     to `len(p)' on a dictionary.

     Changed in version 2.5: This function returned an `int' type.
     This might require changes in your code for properly supporting
     64-bit systems.

 -- Function: int PyDict_Next (PyObject *p, Py_ssize_t *ppos,
          PyObject **pkey, PyObject **pvalue)
     Iterate over all key-value pairs in the dictionary _p_.  The
     `Py_ssize_t' referred to by _ppos_ must be initialized to `0'
     prior to the first call to this function to start the iteration;
     the function returns true for each pair in the dictionary, and
     false once all pairs have been reported.  The parameters _pkey_
     and _pvalue_ should either point to PyObject*.
     variables that will be filled in with each key and value,
     respectively, or may be _NULL_.  Any references returned through
     them are borrowed.  _ppos_ should not be altered during iteration.
     Its value represents offsets within the internal dictionary
     structure, and since the structure is sparse, the offsets are not
     consecutive.

     For example:

         PyObject *key, *value;
         Py_ssize_t pos = 0;

         while (PyDict_Next(self->dict, &pos, &key, &value)) {
             /* do something interesting with the values... */
             ...
         }

     The dictionary _p_ should not be mutated during iteration.  It is
     safe (since Python 2.1) to modify the values of the keys as you
     iterate over the dictionary, but only so long as the set of keys
     does not change.  For example:

         PyObject *key, *value;
         Py_ssize_t pos = 0;

         while (PyDict_Next(self->dict, &pos, &key, &value)) {
             int i = PyInt_AS_LONG(value) + 1;
             PyObject *o = PyInt_FromLong(i);
             if (o == NULL)
                 return -1;
             if (PyDict_SetItem(self->dict, key, o) < 0) {
                 Py_DECREF(o);
                 return -1;
             }
             Py_DECREF(o);
         }

     Changed in version 2.5: This function used an `int *' type for
     _ppos_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: int PyDict_Merge (PyObject *a, PyObject *b, int override)
     Iterate over mapping object _b_ adding key-value pairs to
     dictionary _a_.  _b_ may be a dictionary, or any object supporting
     PyMapping_Keys().  and PyObject_GetItem().
     If _override_ is true, existing pairs in _a_ will be replaced if a
     matching key is found in _b_, otherwise pairs will only be added
     if there is not a matching key in _a_. Return `0' on success or
     `-1' if an exception was raised.

     New in version 2.2.

 -- Function: int PyDict_Update (PyObject *a, PyObject *b)
     This is the same as `PyDict_Merge(a, b, 1)' in C, or `a.update(b)'
     in Python.  Return `0' on success or `-1' if an exception was
     raised.

     New in version 2.2.

 -- Function: int PyDict_MergeFromSeq2 (PyObject *a, PyObject *seq2,
          int override)
     Update or merge into dictionary _a_, from the key-value pairs in
     _seq2_.  _seq2_ must be an iterable object producing iterable
     objects of length 2, viewed as key-value pairs.  In case of
     duplicate keys, the last wins if _override_ is true, else the
     first wins. Return `0' on success or `-1' if an exception was
     raised. Equivalent Python (except for the return value):

         def PyDict_MergeFromSeq2(a, seq2, override):
             for key, value in seq2:
                 if override or key not in a:
                     a[key] = value

     New in version 2.2.

7.7.5 Other Objects
-------------------

7.7.5.1 Class and Instance Objects
..................................

Note that the class objects described here represent old-style classes,
which will go away in Python 3. When creating new types for extension
modules, you will want to work with type objects (section Type Objects).

 -- type: PyClassObject
     The C structure of the objects used to describe built-in classes.

 -- var: PyObject* PyClass_Type
     This is the type object for class objects; it is the same object as
     `types.ClassType' in the Python layer.

 -- Function: int PyClass_Check (PyObject *o)
     Return true if the object _o_ is a class object, including
     instances of types derived from the standard class object.  Return
     false in all other cases.

 -- Function: int PyClass_IsSubclass (PyObject *klass, PyObject *base)
     Return true if _klass_ is a subclass of _base_. Return false in
     all other cases.

  There are very few functions specific to instance objects.

 -- var: PyTypeObject PyInstance_Type
     Type object for class instances.

 -- Function: int PyInstance_Check (PyObject *obj)
     Return true if _obj_ is an instance.

 -- Function: PyObject* PyInstance_New (PyObject *class, PyObject *arg,
          PyObject *kw)
     Return value: New reference.

     Create a new instance of a specific class.  The parameters _arg_
     and _kw_ are used as the positional and keyword parameters to the
     object's constructor.

 -- Function: PyObject* PyInstance_NewRaw (PyObject *class,
          PyObject *dict)
     Return value: New reference.

     Create a new instance of a specific class without calling its
     constructor.  _class_ is the class of new object.  The _dict_
     parameter will be used as the object's `__dict__'; if _NULL_, a
     new dictionary will be created for the instance.

7.7.5.2 Function Objects
........................

There are a few functions specific to Python functions.

 -- type: PyFunctionObject
     The C structure used for functions.

 -- var: PyTypeObject PyFunction_Type
     This is an instance of PyTypeObject and represents
     the Python function type.  It is exposed to Python programmers as
     `types.FunctionType'.

 -- Function: int PyFunction_Check (PyObject *o)
     Return true if _o_ is a function object (has type
     PyFunction_Type).  The parameter must not be _NULL_.

 -- Function: PyObject* PyFunction_New (PyObject *code,
          PyObject *globals)
     Return value: New reference.

     Return a new function object associated with the code object
     _code_. _globals_ must be a dictionary with the global variables
     accessible to the function.

     The function's docstring, name and ___module___ are retrieved from
     the code object, the argument defaults and closure are set to
     _NULL_.

 -- Function: PyObject* PyFunction_GetCode (PyObject *op)
     Return value: Borrowed reference.

     Return the code object associated with the function object _op_.

 -- Function: PyObject* PyFunction_GetGlobals (PyObject *op)
     Return value: Borrowed reference.

     Return the globals dictionary associated with the function object
     _op_.

 -- Function: PyObject* PyFunction_GetModule (PyObject *op)
     Return value: Borrowed reference.

     Return the ___module___ attribute of the function object _op_.
     This is normally a string containing the module name, but can be
     set to any other object by Python code.

 -- Function: PyObject* PyFunction_GetDefaults (PyObject *op)
     Return value: Borrowed reference.

     Return the argument default values of the function object _op_.
     This can be a tuple of arguments or _NULL_.

 -- Function: int PyFunction_SetDefaults (PyObject *op,
          PyObject *defaults)
     Set the argument default values for the function object _op_.
     _defaults_ must be _Py_None_ or a tuple.

     Raises SystemError and returns `-1' on failure.

 -- Function: PyObject* PyFunction_GetClosure (PyObject *op)
     Return value: Borrowed reference.

     Return the closure associated with the function object _op_. This
     can be _NULL_ or a tuple of cell objects.

 -- Function: int PyFunction_SetClosure (PyObject *op,
          PyObject *closure)
     Set the closure associated with the function object _op_.
     _closure_ must be _Py_None_ or a tuple of cell objects.

     Raises SystemError and returns `-1' on failure.

7.7.5.3 Method Objects
......................

There are some useful functions that are useful for working with method
objects.

 -- var: PyTypeObject PyMethod_Type
     This instance of PyTypeObject represents the Python
     method type.  This is exposed to Python programs as
     `types.MethodType'.

 -- Function: int PyMethod_Check (PyObject *o)
     Return true if _o_ is a method object (has type
     PyMethod_Type).  The parameter must not be _NULL_.

 -- Function: PyObject* PyMethod_New (PyObject *func, PyObject *self,
          PyObject *class)
     Return value: New reference.

     Return a new method object, with _func_ being any callable object;
     this is the function that will be called when the method is
     called.  If this method should be bound to an instance, _self_
     should be the instance and _class_ should be the class of _self_,
     otherwise _self_ should be _NULL_ and _class_ should be the class
     which provides the unbound method..

 -- Function: PyObject* PyMethod_Class (PyObject *meth)
     Return value: Borrowed reference.

     Return the class object from which the method _meth_ was created;
     if this was created from an instance, it will be the class of the
     instance.

 -- Function: PyObject* PyMethod_GET_CLASS (PyObject *meth)
     Return value: Borrowed reference.

     Macro version of PyMethod_Class() which avoids error
     checking.

 -- Function: PyObject* PyMethod_Function (PyObject *meth)
     Return value: Borrowed reference.

     Return the function object associated with the method _meth_.

 -- Function: PyObject* PyMethod_GET_FUNCTION (PyObject *meth)
     Return value: Borrowed reference.

     Macro version of PyMethod_Function() which avoids
     error checking.

 -- Function: PyObject* PyMethod_Self (PyObject *meth)
     Return value: Borrowed reference.

     Return the instance associated with the method _meth_ if it is
     bound, otherwise return _NULL_.

 -- Function: PyObject* PyMethod_GET_SELF (PyObject *meth)
     Return value: Borrowed reference.

     Macro version of PyMethod_Self() which avoids error
     checking.

 -- Function: int PyMethod_ClearFreeList ()
     Clear the free list. Return the total number of freed items.

     New in version 2.6.

7.7.5.4 File Objects
....................

Python's built-in file objects are implemented entirely on the `FILE*'
support from the C standard library.  This is an implementation detail
and may change in future releases of Python.

 -- type: PyFileObject
     This subtype of PyObject represents a Python file
     object.

 -- var: PyTypeObject PyFile_Type
     This instance of PyTypeObject represents the Python
     file type.  This is exposed to Python programs as `file' and
     `types.FileType'.

 -- Function: int PyFile_Check (PyObject *p)
     Return true if its argument is a PyFileObject or a
     subtype of PyFileObject.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyFile_CheckExact (PyObject *p)
     Return true if its argument is a PyFileObject, but not
     a subtype of PyFileObject.

     New in version 2.2.

 -- Function: PyObject* PyFile_FromString (char *filename, char *mode)
     Return value: New reference.

     On success, return a new file object that is opened on the file
     given by _filename_, with a file mode given by _mode_, where
     _mode_ has the same semantics as the standard C routine `fopen()'.
     On failure, return _NULL_.

 -- Function: PyObject* PyFile_FromFile (FILE *fp, char *name,
          char *mode, int (*close)(FILE*))
     Return value: New reference.

     Create a new PyFileObject from the already-open
     standard C file pointer, _fp_.  The function _close_ will be
     called when the file should be closed.  Return _NULL_ on failure.

 -- Function: FILE* PyFile_AsFile (PyObject *p)
     Return the file object associated with _p_ as a `FILE*'.

     If the caller will ever use the returned `FILE*' object while the
     GIL is released it must also call the
     PyFile_IncUseCount() and PyFile_DecUseCount().
     functions described below as appropriate.

 -- Function: void PyFile_IncUseCount (PyFileObject *p)
     Increments the PyFileObject's internal use count to indicate that
     the underlying `FILE*' is being used.  This prevents Python from
     calling f_close() on it from another thread.  Callers of this must
     call PyFile_DecUseCount() when they are finished with
     the `FILE*'.  Otherwise the file object will never be closed by
     Python.

     The GIL must be held while calling this function.

     The suggested use is to call this after PyFile_AsFile()
     and before you release the GIL:

         FILE *fp = PyFile_AsFile(p);
         PyFile_IncUseCount(p);
         /* ... */
         Py_BEGIN_ALLOW_THREADS
         do_something(fp);
         Py_END_ALLOW_THREADS
         /* ... */
         PyFile_DecUseCount(p);

     New in version 2.6.

 -- Function: void PyFile_DecUseCount (PyFileObject *p)
     Decrements the PyFileObject's internal unlocked_count member to
     indicate that the caller is done with its own use of the `FILE*'.
     This may only be called to undo a prior call to
     PyFile_IncUseCount().

     The GIL must be held while calling this function (see
     the example above).

     New in version 2.6.

 -- Function: PyObject* PyFile_GetLine (PyObject *p, int n)
     Return value: New reference.

     Equivalent to `p.readline([n])', this function reads one line from
     the object _p_.  _p_ may be a file object or any object with a
     readline().  method.  If _n_ is `0', exactly one line
     is read, regardless of the length of the line.  If _n_ is greater
     than `0', no more than _n_ bytes will be read from the file; a
     partial line can be returned.  In both cases, an empty string is
     returned if the end of the file is reached immediately.  If _n_ is
     less than `0', however, one line is read regardless of length, but
     EOFError is raised if the end of the file is reached
     immediately.

 -- Function: PyObject* PyFile_Name (PyObject *p)
     Return value: Borrowed reference.

     Return the name of the file specified by _p_ as a string object.

 -- Function: void PyFile_SetBufSize (PyFileObject *p, int n)
     Available on systems with `setvbuf()' only.  This should only be
     called immediately after file object creation.

 -- Function: int PyFile_SetEncoding (PyFileObject *p, const char *enc)
     Set the file's encoding for Unicode output to _enc_. Return 1 on
     success and 0 on failure.

     New in version 2.3.

 -- Function: int PyFile_SetEncodingAndErrors (PyFileObject *p, const
          char *enc, *errors)
     Set the file's encoding for Unicode output to _enc_, and its error
     mode to _err_. Return 1 on success and 0 on failure.

     New in version 2.6.

 -- Function: int PyFile_SoftSpace (PyObject *p, int newflag)
     This function exists for internal use by the interpreter.  Set the
     `softspace' attribute of _p_ to _newflag_ and return the previous
     value.  _p_ does not have to be a file object for this function to
     work properly; any object is supported (thought its only
     interesting if the `softspace' attribute can be set).  This
     function clears any errors, and will return `0' as the previous
     value if the attribute either does not exist or if there were
     errors in retrieving it.  There is no way to detect errors from
     this function, but doing so should not be needed.

 -- Function: int PyFile_WriteObject (PyObject *obj, PyObject *p,
          int flags)
     Write object _obj_ to file object _p_.  The only supported flag
     for _flags_ is `Py_PRINT_RAW'; if given, the str() of
     the object is written instead of the repr().  Return
     `0' on success or `-1' on failure; the appropriate exception will
     be set.

 -- Function: int PyFile_WriteString (const char *s, PyObject *p)
     Write string _s_ to file object _p_.  Return `0' on success or
     `-1' on failure; the appropriate exception will be set.

7.7.5.5 Module Objects
......................

There are only a few functions special to module objects.

 -- var: PyTypeObject PyModule_Type
     This instance of PyTypeObject represents the Python
     module type.  This is exposed to Python programs as
     `types.ModuleType'.

 -- Function: int PyModule_Check (PyObject *p)
     Return true if _p_ is a module object, or a subtype of a module
     object.

     Changed in version 2.2: Allowed subtypes to be accepted.

 -- Function: int PyModule_CheckExact (PyObject *p)
     Return true if _p_ is a module object, but not a subtype of
     PyModule_Type.

     New in version 2.2.

 -- Function: PyObject* PyModule_New (const char *name)
     Return value: New reference.

     Return a new module object with the `__name__' attribute set to
     _name_.  Only the module's `__doc__' and `__name__' attributes are
     filled in; the caller is responsible for providing a `__file__'
     attribute.

 -- Function: PyObject* PyModule_GetDict (PyObject *module)
     Return value: Borrowed reference.

     Return the dictionary object that implements _module_'s namespace;
     this object is the same as the `__dict__' attribute of the module
     object.  This function never fails.  It is recommended extensions
     use other `PyModule_*()' and `PyObject_*()' functions rather than
     directly manipulate a module's `__dict__'.

 -- Function: char* PyModule_GetName (PyObject *module)
     Return _module_'s `__name__' value.  If the module does not
     provide one, or if it is not a string, see SystemError is
     raised and _NULL_ is returned.

 -- Function: char* PyModule_GetFilename (PyObject *module)
     Return the name of the file from which _module_ was loaded using
     _module_'s `__file__' attribute.  If this is not defined, or if it
     is not a string, raise SystemError and return _NULL_.

 -- Function: int PyModule_AddObject (PyObject *module, const
          char *name, PyObject *value)
     Add an object to _module_ as _name_.  This is a convenience
     function which can be used from the module's initialization
     function.  This steals a reference to _value_.  Return `-1' on
     error, `0' on success.

     New in version 2.0.

 -- Function: int PyModule_AddIntConstant (PyObject *module, const
          char *name, long value)
     Add an integer constant to _module_ as _name_.  This convenience
     function can be used from the module's initialization function.
     Return `-1' on error, `0' on success.

     New in version 2.0.

 -- Function: int PyModule_AddStringConstant (PyObject *module, const
          char *name, const char *value)
     Add a string constant to _module_ as _name_.  This convenience
     function can be used from the module's initialization function.
     The string _value_ must be null-terminated.  Return `-1' on error,
     `0' on success.

     New in version 2.0.

 -- Function: int PyModule_AddIntMacro (PyObject *module, macro)
     Add an int constant to _module_. The name and the value are taken
     from _macro_. For example `PyModule_AddConstant(module, AF_INET)'
     adds the int constant _AF_INET_ with the value of _AF_INET_ to
     _module_.  Return `-1' on error, `0' on success.

     New in version 2.6.

 -- Function: int PyModule_AddStringMacro (PyObject *module, macro)
          Add a string constant to _module_.

     New in version 2.6.

7.7.5.6 Iterator Objects
........................

Python provides two general-purpose iterator objects.  The first, a
sequence iterator, works with an arbitrary sequence supporting the
__getitem__().  method.  The second works with a callable
object and a sentinel value, calling the callable for each item in the
sequence, and ending the iteration when the sentinel value is returned.

 -- var: PyTypeObject PySeqIter_Type
     Type object for iterator objects returned by
     PySeqIter_New() and the one-argument form of the
     iter() built-in function for built-in sequence types.

     New in version 2.2.

 -- Function: int PySeqIter_Check (op)
     Return true if the type of _op_ is PySeqIter_Type.

     New in version 2.2.

 -- Function: PyObject* PySeqIter_New (PyObject *seq)
     Return value: New reference.

     Return an iterator that works with a general sequence object,
     _seq_.  The iteration ends when the sequence raises
     IndexError for the subscripting operation.

     New in version 2.2.

 -- var: PyTypeObject PyCallIter_Type
     Type object for iterator objects returned by
     PyCallIter_New() and the two-argument form of the
     iter() built-in function.

     New in version 2.2.

 -- Function: int PyCallIter_Check (op)
     Return true if the type of _op_ is PyCallIter_Type.

     New in version 2.2.

 -- Function: PyObject* PyCallIter_New (PyObject *callable,
          PyObject *sentinel)
     Return value: New reference.

     Return a new iterator.  The first parameter, _callable_, can be
     any Python callable object that can be called with no parameters;
     each call to it should return the next item in the iteration.
     When _callable_ returns a value equal to _sentinel_, the iteration
     will be terminated.

     New in version 2.2.

7.7.5.7 Descriptor Objects
..........................

"Descriptors" are objects that describe some attribute of an object.
They are found in the dictionary of type objects.

 -- var: PyTypeObject PyProperty_Type
     The type object for the built-in descriptor types.

     New in version 2.2.

 -- Function: PyObject* PyDescr_NewGetSet (PyTypeObject *type, struct
          PyGetSetDef *getset)
     Return value: New reference.

     New in version 2.2.

 -- Function: PyObject* PyDescr_NewMember (PyTypeObject *type, struct
          PyMemberDef *meth)
     Return value: New reference.

     New in version 2.2.

 -- Function: PyObject* PyDescr_NewMethod (PyTypeObject *type, struct
          PyMethodDef *meth)
     Return value: New reference.

     New in version 2.2.

 -- Function: PyObject* PyDescr_NewWrapper (PyTypeObject *type, struct
          wrapperbase *wrapper, void *wrapped)
     Return value: New reference.

     New in version 2.2.

 -- Function: PyObject* PyDescr_NewClassMethod (PyTypeObject *type,
          PyMethodDef *method)
     Return value: New reference.

     New in version 2.3.

 -- Function: int PyDescr_IsData (PyObject *descr)
     Return true if the descriptor objects _descr_ describes a data
     attribute, or false if it describes a method.  _descr_ must be a
     descriptor object; there is no error checking.

     New in version 2.2.

 -- Function: PyObject* PyWrapper_New (PyObject *, PyObject *)
     Return value: New reference.

     New in version 2.2.

7.7.5.8 Slice Objects
.....................

 -- var: PyTypeObject PySlice_Type
     The type object for slice objects.  This is the same as `slice' and
     `types.SliceType'.

 -- Function: int PySlice_Check (PyObject *ob)
     Return true if _ob_ is a slice object; _ob_ must not be _NULL_.

 -- Function: PyObject* PySlice_New (PyObject *start, PyObject *stop,
          PyObject *step)
     Return value: New reference.

     Return a new slice object with the given values.  The _start_,
     _stop_, and _step_ parameters are used as the values of the slice
     object attributes of the same names.  Any of the values may be
     _NULL_, in which case the `None' will be used for the
     corresponding attribute.  Return _NULL_ if the new object could
     not be allocated.

 -- Function: int PySlice_GetIndices (PySliceObject *slice,
          Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop,
          Py_ssize_t *step)
     Retrieve the start, stop and step indices from the slice object
     _slice_, assuming a sequence of length _length_. Treats indices
     greater than _length_ as errors.

     Returns 0 on success and -1 on error with no exception set (unless
     one of the indices was not None and failed to be
     converted to an integer, in which case -1 is returned with an
     exception set).

     You probably do not want to use this function.  If you want to use
     slice objects in versions of Python prior to 2.3, you would
     probably do well to incorporate the source of
     PySlice_GetIndicesEx(), suitably renamed, in the source of
     your extension.

     Changed in version 2.5: This function used an `int' type for
     _length_ and an `int *' type for _start_, _stop_, and _step_. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: int PySlice_GetIndicesEx (PySliceObject *slice,
          Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop,
          Py_ssize_t *step, Py_ssize_t *slicelength)
     Usable replacement for PySlice_GetIndices().  Retrieve
     the start, stop, and step indices from the slice object _slice_
     assuming a sequence of length _length_, and store the length of
     the slice in _slicelength_.  Out of bounds indices are clipped in
     a manner consistent with the handling of normal slices.

     Returns 0 on success and -1 on error with exception set.

     New in version 2.3.

     Changed in version 2.5: This function used an `int' type for
     _length_ and an `int *' type for _start_, _stop_, _step_, and
     _slicelength_. This might require changes in your code for
     properly supporting 64-bit systems.

7.7.5.9 Weak Reference Objects
..............................

Python supports _weak references_ as first-class objects.  There are two
specific object types which directly implement weak references.  The
first is a simple reference object, and the second acts as a proxy for
the original object as much as it can.

 -- Function: int PyWeakref_Check (ob)
     Return true if _ob_ is either a reference or proxy object.

     New in version 2.2.

 -- Function: int PyWeakref_CheckRef (ob)
     Return true if _ob_ is a reference object.

     New in version 2.2.

 -- Function: int PyWeakref_CheckProxy (ob)
     Return true if _ob_ is a proxy object.

     New in version 2.2.

 -- Function: PyObject* PyWeakref_NewRef (PyObject *ob,
          PyObject *callback)
     Return value: New reference.

     Return a weak reference object for the object _ob_.  This will
     always return a new reference, but is not guaranteed to create a
     new object; an existing reference object may be returned.  The
     second parameter, _callback_, can be a callable object that
     receives notification when _ob_ is garbage collected; it should
     accept a single parameter, which will be the weak reference object
     itself. _callback_ may also be `None' or _NULL_.  If _ob_ is not a
     weakly-referencable object, or if _callback_ is not callable,
     `None', or _NULL_, this will return _NULL_ and raise
     TypeError.

     New in version 2.2.

 -- Function: PyObject* PyWeakref_NewProxy (PyObject *ob,
          PyObject *callback)
     Return value: New reference.

     Return a weak reference proxy object for the object _ob_.  This
     will always return a new reference, but is not guaranteed to
     create a new object; an existing proxy object may be returned.
     The second parameter, _callback_, can be a callable object that
     receives notification when _ob_ is garbage collected; it should
     accept a single parameter, which will be the weak reference object
     itself. _callback_ may also be `None' or _NULL_.  If _ob_ is not a
     weakly-referencable object, or if _callback_ is not callable,
     `None', or _NULL_, this will return _NULL_ and raise
     TypeError.

     New in version 2.2.

 -- Function: PyObject* PyWeakref_GetObject (PyObject *ref)
     Return value: Borrowed reference.

     Return the referenced object from a weak reference, _ref_.  If the
     referent is no longer live, returns `Py_None'.

     New in version 2.2.

          Warning: This function returns a *borrowed reference* to the
          referenced object.  This means that you should always call
          Py_INCREF() on the object except if you know
          that it cannot be destroyed while you are still using it.

 -- Function: PyObject* PyWeakref_GET_OBJECT (PyObject *ref)
     Return value: Borrowed reference.

     Similar to PyWeakref_GetObject(), but implemented as a
     macro that does no error checking.

     New in version 2.2.

7.7.5.10 Capsules
.................

Refer to Providing a C API for an Extension Module for more
information on using these objects.

 -- type: PyCapsule
     This subtype of PyObject represents an opaque value,
     useful for C extension modules who need to pass an opaque value
     (as a `void*' pointer) through Python code to other C code.  It is
     often used to make a C function pointer defined in one module
     available to other modules, so the regular import mechanism can be
     used to access C APIs defined in dynamically loaded modules.

 -- type: PyCapsule_Destructor
     The type of a destructor callback for a capsule.  Defined as:

         typedef void (*PyCapsule_Destructor)(PyObject *);

     See PyCapsule_New() for the semantics of
     PyCapsule_Destructor callbacks.

 -- Function: int PyCapsule_CheckExact (PyObject *p)
     Return true if its argument is a PyCapsule.

 -- Function: PyObject* PyCapsule_New (void *pointer, const char *name,
          PyCapsule_Destructor destructor)
     Return value: New reference.

     Create a PyCapsule encapsulating the _pointer_.  The
     _pointer_ argument may not be _NULL_.

     On failure, set an exception and return _NULL_.

     The _name_ string may either be _NULL_ or a pointer to a valid C
     string.  If non-_NULL_, this string must outlive the capsule.
     (Though it is permitted to free it inside the _destructor_.)

     If the _destructor_ argument is not _NULL_, it will be called with
     the capsule as its argument when it is destroyed.

     If this capsule will be stored as an attribute of a module, the
     _name_ should be specified as `modulename.attributename'.  This
     will enable other modules to import the capsule using
     PyCapsule_Import().

 -- Function: void* PyCapsule_GetPointer (PyObject *capsule, const
          char *name)
     Retrieve the _pointer_ stored in the capsule.  On failure, set an
     exception and return _NULL_.

     The _name_ parameter must compare exactly to the name stored in
     the capsule.  If the name stored in the capsule is _NULL_, the
     _name_ passed in must also be _NULL_.  Python uses the C function
     `strcmp()' to compare capsule names.

 -- Function: PyCapsule_Destructor PyCapsule_GetDestructor
          (PyObject *capsule)
     Return the current destructor stored in the capsule.  On failure,
     set an exception and return _NULL_.

     It is legal for a capsule to have a _NULL_ destructor.  This makes
     a _NULL_ return code somewhat ambiguous; use
     PyCapsule_IsValid() or PyErr_Occurred() to
     disambiguate.

 -- Function: void* PyCapsule_GetContext (PyObject *capsule)
     Return the current context stored in the capsule.  On failure, set
     an exception and return _NULL_.

     It is legal for a capsule to have a _NULL_ context.  This makes a
     _NULL_ return code somewhat ambiguous; use
     PyCapsule_IsValid() or PyErr_Occurred() to
     disambiguate.

 -- Function: const char* PyCapsule_GetName (PyObject *capsule)
     Return the current name stored in the capsule.  On failure, set an
     exception and return _NULL_.

     It is legal for a capsule to have a _NULL_ name.  This makes a
     _NULL_ return code somewhat ambiguous; use
     PyCapsule_IsValid() or PyErr_Occurred() to
     disambiguate.

 -- Function: void* PyCapsule_Import (const char *name, int no_block)
     Import a pointer to a C object from a capsule attribute in a
     module.  The _name_ parameter should specify the full name to the
     attribute, as in `module.attribute'.  The _name_ stored in the
     capsule must match this string exactly.  If _no_block_ is true,
     import the module without blocking (using
     PyImport_ImportModuleNoBlock()).  If _no_block_ is false,
     import the module conventionally (using
     PyImport_ImportModule()).

     Return the capsule's internal _pointer_ on success.  On failure,
     set an exception and return _NULL_.  However, if
     PyCapsule_Import() failed to import the module, and
     _no_block_ was true, no exception is set.

 -- Function: int PyCapsule_IsValid (PyObject *capsule, const
          char *name)
     Determines whether or not _capsule_ is a valid capsule.  A valid
     capsule is non-_NULL_, passes PyCapsule_CheckExact(),
     has a non-_NULL_ pointer stored in it, and its internal name
     matches the _name_ parameter.  (See PyCapsule_GetPointer()
     for information on how capsule names are compared.)

     In other words, if PyCapsule_IsValid() returns a true
     value, calls to any of the accessors (any function starting with
     `PyCapsule_Get()') are guaranteed to succeed.

     Return a nonzero value if the object is valid and matches the name
     passed in.  Return 0 otherwise.  This function will not fail.

 -- Function: int PyCapsule_SetContext (PyObject *capsule,
          void *context)
     Set the context pointer inside _capsule_ to _context_.

     Return 0 on success.  Return nonzero and set an exception on
     failure.

 -- Function: int PyCapsule_SetDestructor (PyObject *capsule,
          PyCapsule_Destructor destructor)
     Set the destructor inside _capsule_ to _destructor_.

     Return 0 on success.  Return nonzero and set an exception on
     failure.

 -- Function: int PyCapsule_SetName (PyObject *capsule, const
          char *name)
     Set the name inside _capsule_ to _name_.  If non-_NULL_, the name
     must outlive the capsule.  If the previous _name_ stored in the
     capsule was not _NULL_, no attempt is made to free it.

     Return 0 on success.  Return nonzero and set an exception on
     failure.

 -- Function: int PyCapsule_SetPointer (PyObject *capsule,
          void *pointer)
     Set the void pointer inside _capsule_ to _pointer_.  The pointer
     may not be _NULL_.

     Return 0 on success.  Return nonzero and set an exception on
     failure.

7.7.5.11 CObjects
.................

     Warning: The CObject API is deprecated as of Python 2.7.  Please
     switch to the new Capsules. API.

 -- type: PyCObject
     This subtype of PyObject represents an opaque value,
     useful for C extension modules who need to pass an opaque value
     (as a `void*' pointer) through Python code to other C code.  It is
     often used to make a C function pointer defined in one module
     available to other modules, so the regular import mechanism can be
     used to access C APIs defined in dynamically loaded modules.

 -- Function: int PyCObject_Check (PyObject *p)
     Return true if its argument is a PyCObject.

 -- Function: PyObject* PyCObject_FromVoidPtr (void* cobj, void
          (*destr)(void *))
     Return value: New reference.

     Create a PyCObject from the `void *' _cobj_.  The
     _destr_ function will be called when the object is reclaimed,
     unless it is _NULL_.

 -- Function: PyObject* PyCObject_FromVoidPtrAndDesc (void* cobj,
          void* desc, void (*destr)(void *, void *))
     Return value: New reference.

     Create a PyCObject from the `void *' _cobj_.  The
     _destr_ function will be called when the object is reclaimed. The
     _desc_ argument can be used to pass extra callback data for the
     destructor function.

 -- Function: void* PyCObject_AsVoidPtr (PyObject* self)
     Return the object `void *' that the PyCObject. _self_
     was created with.

 -- Function: void* PyCObject_GetDesc (PyObject* self)
     Return the description `void *' that the PyCObject.
     _self_ was created with.

 -- Function: int PyCObject_SetVoidPtr (PyObject* self, void* cobj)
     Set the void pointer inside _self_ to _cobj_. The PyCObject
     must not have an associated destructor. Return true on
     success, false on failure.

7.7.5.12 Cell Objects
.....................

"Cell" objects are used to implement variables referenced by multiple
scopes.  For each such variable, a cell object is created to store the
value; the local variables of each stack frame that references the
value contains a reference to the cells from outer scopes which also
use that variable.  When the value is accessed, the value contained in
the cell is used instead of the cell object itself.  This
de-referencing of the cell object requires support from the generated
byte-code; these are not automatically de-referenced when accessed.
Cell objects are not likely to be useful elsewhere.

 -- type: PyCellObject
     The C structure used for cell objects.

 -- var: PyTypeObject PyCell_Type
     The type object corresponding to cell objects.

 -- Function: int PyCell_Check (ob)
     Return true if _ob_ is a cell object; _ob_ must not be _NULL_.

 -- Function: PyObject* PyCell_New (PyObject *ob)
     Return value: New reference.

     Create and return a new cell object containing the value _ob_. The
     parameter may be _NULL_.

 -- Function: PyObject* PyCell_Get (PyObject *cell)
     Return value: New reference.

     Return the contents of the cell _cell_.

 -- Function: PyObject* PyCell_GET (PyObject *cell)
     Return value: Borrowed reference.

     Return the contents of the cell _cell_, but without checking that
     _cell_ is non-_NULL_ and a cell object.

 -- Function: int PyCell_Set (PyObject *cell, PyObject *value)
     Set the contents of the cell object _cell_ to _value_.  This
     releases the reference to any current content of the cell. _value_
     may be _NULL_.  _cell_ must be non-_NULL_; if it is not a cell
     object, `-1' will be returned.  On success, `0' will be returned.

 -- Function: void PyCell_SET (PyObject *cell, PyObject *value)
     Sets the value of the cell object _cell_ to _value_.  No reference
     counts are adjusted, and no checks are made for safety; _cell_
     must be non-_NULL_ and must be a cell object.

7.7.5.13 Generator Objects
..........................

Generator objects are what Python uses to implement generator
iterators. They are normally created by iterating over a function that
yields values, rather than explicitly calling PyGen_New().

 -- type: PyGenObject
     The C structure used for generator objects.

 -- var: PyTypeObject PyGen_Type
     The type object corresponding to generator objects

 -- Function: int PyGen_Check (ob)
     Return true if _ob_ is a generator object; _ob_ must not be _NULL_.

 -- Function: int PyGen_CheckExact (ob)
     Return true if _ob_'s type is _PyGen_Type_ is a generator object;
     _ob_ must not be _NULL_.

 -- Function: PyObject* PyGen_New (PyFrameObject *frame)
     Return value: New reference.

     Create and return a new generator object based on the _frame_
     object. A reference to _frame_ is stolen by this function. The
     parameter must not be _NULL_.

7.7.5.14 DateTime Objects
.........................

Various date and time objects are supplied by the datetime.
module.  Before using any of these functions, the header file
`datetime.h' must be included in your source (note that this is not
included by `Python.h'), and the macro `PyDateTime_IMPORT' must be
invoked, usually as part of the module initialisation function.  The
macro puts a pointer to a C structure into a static variable,
`PyDateTimeAPI', that is used by the following macros.

  Type-check macros:

 -- Function: int PyDate_Check (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_DateType' or a subtype
     of `PyDateTime_DateType'.  _ob_ must not be _NULL_.

     New in version 2.4.

 -- Function: int PyDate_CheckExact (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_DateType'. _ob_ must
     not be _NULL_.

     New in version 2.4.

 -- Function: int PyDateTime_Check (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_DateTimeType' or a
     subtype of `PyDateTime_DateTimeType'.  _ob_ must not be _NULL_.

     New in version 2.4.

 -- Function: int PyDateTime_CheckExact (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_DateTimeType'. _ob_
     must not be _NULL_.

     New in version 2.4.

 -- Function: int PyTime_Check (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_TimeType' or a subtype
     of `PyDateTime_TimeType'.  _ob_ must not be _NULL_.

     New in version 2.4.

 -- Function: int PyTime_CheckExact (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_TimeType'. _ob_ must
     not be _NULL_.

     New in version 2.4.

 -- Function: int PyDelta_Check (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_DeltaType' or a subtype
     of `PyDateTime_DeltaType'.  _ob_ must not be _NULL_.

     New in version 2.4.

 -- Function: int PyDelta_CheckExact (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_DeltaType'. _ob_ must
     not be _NULL_.

     New in version 2.4.

 -- Function: int PyTZInfo_Check (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_TZInfoType' or a
     subtype of `PyDateTime_TZInfoType'.  _ob_ must not be _NULL_.

     New in version 2.4.

 -- Function: int PyTZInfo_CheckExact (PyObject *ob)
     Return true if _ob_ is of type `PyDateTime_TZInfoType'. _ob_ must
     not be _NULL_.

     New in version 2.4.

  Macros to create objects:

 -- Function: PyObject* PyDate_FromDate (int year, int month, int day)
     Return value: New reference.

     Return a `datetime.date' object with the specified year, month and
     day.

     New in version 2.4.

 -- Function: PyObject* PyDateTime_FromDateAndTime (int year,
          int month, int day, int hour, int minute, int second,
          int usecond)
     Return value: New reference.

     Return a `datetime.datetime' object with the specified year,
     month, day, hour, minute, second and microsecond.

     New in version 2.4.

 -- Function: PyObject* PyTime_FromTime (int hour, int minute,
          int second, int usecond)
     Return value: New reference.

     Return a `datetime.time' object with the specified hour, minute,
     second and microsecond.

     New in version 2.4.

 -- Function: PyObject* PyDelta_FromDSU (int days, int seconds,
          int useconds)
     Return value: New reference.

     Return a `datetime.timedelta' object representing the given number
     of days, seconds and microseconds.  Normalization is performed so
     that the resulting number of microseconds and seconds lie in the
     ranges documented for `datetime.timedelta' objects.

     New in version 2.4.

  Macros to extract fields from date objects.  The argument must be an
instance of `PyDateTime_Date', including subclasses (such as
`PyDateTime_DateTime').  The argument must not be _NULL_, and the type
is not checked:

 -- Function: int PyDateTime_GET_YEAR (PyDateTime_Date *o)
     Return the year, as a positive int.

     New in version 2.4.

 -- Function: int PyDateTime_GET_MONTH (PyDateTime_Date *o)
     Return the month, as an int from 1 through 12.

     New in version 2.4.

 -- Function: int PyDateTime_GET_DAY (PyDateTime_Date *o)
     Return the day, as an int from 1 through 31.

     New in version 2.4.

  Macros to extract fields from datetime objects.  The argument must be
an instance of `PyDateTime_DateTime', including subclasses. The argument
must not be _NULL_, and the type is not checked:

 -- Function: int PyDateTime_DATE_GET_HOUR (PyDateTime_DateTime *o)
     Return the hour, as an int from 0 through 23.

     New in version 2.4.

 -- Function: int PyDateTime_DATE_GET_MINUTE (PyDateTime_DateTime *o)
     Return the minute, as an int from 0 through 59.

     New in version 2.4.

 -- Function: int PyDateTime_DATE_GET_SECOND (PyDateTime_DateTime *o)
     Return the second, as an int from 0 through 59.

     New in version 2.4.

 -- Function: int PyDateTime_DATE_GET_MICROSECOND
          (PyDateTime_DateTime *o)
     Return the microsecond, as an int from 0 through 999999.

     New in version 2.4.

  Macros to extract fields from time objects.  The argument must be an
instance of `PyDateTime_Time', including subclasses. The argument must
not be _NULL_, and the type is not checked:

 -- Function: int PyDateTime_TIME_GET_HOUR (PyDateTime_Time *o)
     Return the hour, as an int from 0 through 23.

     New in version 2.4.

 -- Function: int PyDateTime_TIME_GET_MINUTE (PyDateTime_Time *o)
     Return the minute, as an int from 0 through 59.

     New in version 2.4.

 -- Function: int PyDateTime_TIME_GET_SECOND (PyDateTime_Time *o)
     Return the second, as an int from 0 through 59.

     New in version 2.4.

 -- Function: int PyDateTime_TIME_GET_MICROSECOND (PyDateTime_Time *o)
     Return the microsecond, as an int from 0 through 999999.

     New in version 2.4.

  Macros for the convenience of modules implementing the DB API:

 -- Function: PyObject* PyDateTime_FromTimestamp (PyObject *args)
     Return value: New reference.

     Create and return a new `datetime.datetime' object given an
     argument tuple suitable for passing to
     `datetime.datetime.fromtimestamp()'.

     New in version 2.4.

 -- Function: PyObject* PyDate_FromTimestamp (PyObject *args)
     Return value: New reference.

     Create and return a new `datetime.date' object given an argument
     tuple suitable for passing to `datetime.date.fromtimestamp()'.

     New in version 2.4.

7.7.5.15 Set Objects
....................

New in version 2.5.

  This section details the public API for set and
frozenset.  objects.  Any functionality not listed below is best
accessed using the either the abstract object protocol (including
PyObject_CallMethod(), PyObject_RichCompareBool(),
PyObject_Hash(), PyObject_Repr(),
PyObject_IsTrue(), PyObject_Print(), and
PyObject_GetIter()) or the abstract number protocol (including
PyNumber_And(), PyNumber_Subtract(),
PyNumber_Or(), PyNumber_Xor(),
PyNumber_InPlaceAnd(), PyNumber_InPlaceSubtract(),
PyNumber_InPlaceOr(), and PyNumber_InPlaceXor()).

 -- type: PySetObject
     This subtype of PyObject is used to hold the internal
     data for both set and frozenset objects.
     It is like a PyDictObject.  in that it is a fixed size
     for small sets (much like tuple storage) and will point to a
     separate, variable sized block of memory for medium and large sized
     sets (much like list storage). None of the fields of this
     structure should be considered public and are subject to change.
     All access should be done through the documented API rather than
     by manipulating the values in the structure.

 -- var: PyTypeObject PySet_Type
     This is an instance of PyTypeObject representing the
     Python set type.

 -- var: PyTypeObject PyFrozenSet_Type
     This is an instance of PyTypeObject representing the
     Python frozenset type.

  The following type check macros work on pointers to any Python
object. Likewise, the constructor functions work with any iterable
Python object.

 -- Function: int PySet_Check (PyObject *p)
     Return true if _p_ is a set object or an instance of a
     subtype.

     New in version 2.6.

 -- Function: int PyFrozenSet_Check (PyObject *p)
     Return true if _p_ is a frozenset object or an
     instance of a subtype.

     New in version 2.6.

 -- Function: int PyAnySet_Check (PyObject *p)
     Return true if _p_ is a set object, a frozenset
     object, or an instance of a subtype.

 -- Function: int PyAnySet_CheckExact (PyObject *p)
     Return true if _p_ is a set object or a
     frozenset object but not an instance of a subtype.

 -- Function: int PyFrozenSet_CheckExact (PyObject *p)
     Return true if _p_ is a frozenset object but not an
     instance of a subtype.

 -- Function: PyObject* PySet_New (PyObject *iterable)
     Return value: New reference.

     Return a new set containing objects returned by the
     _iterable_.  The _iterable_ may be _NULL_ to create a new empty
     set.  Return the new set on success or _NULL_ on failure.  Raise
     TypeError if _iterable_ is not actually iterable.  The
     constructor is also useful for copying a set (`c=set(s)').

 -- Function: PyObject* PyFrozenSet_New (PyObject *iterable)
     Return value: New reference.

     Return a new frozenset containing objects returned by
     the _iterable_.  The _iterable_ may be _NULL_ to create a new
     empty frozenset.  Return the new set on success or _NULL_ on
     failure.  Raise TypeError if _iterable_ is not
     actually iterable.

     Changed in version 2.6: Now guaranteed to return a brand-new
     frozenset.  Formerly, frozensets of zero-length were a
     singleton.  This got in the way of building-up new frozensets with
     `PySet_Add()'.

  The following functions and macros are available for instances of
set.  or frozenset or instances of their
subtypes.

 -- Function: Py_ssize_t PySet_Size (PyObject *anyset)
     Return the length of a set or frozenset.
     object. Equivalent to `len(anyset)'.  Raises a `PyExc_SystemError'
     if _anyset_ is not a set, frozenset, or an
     instance of a subtype.

     Changed in version 2.5: This function returned an `int'. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- Function: Py_ssize_t PySet_GET_SIZE (PyObject *anyset)
     Macro form of PySet_Size() without error checking.

 -- Function: int PySet_Contains (PyObject *anyset, PyObject *key)
     Return 1 if found, 0 if not found, and -1 if an error is
     encountered.  Unlike the Python __contains__() method,
     this function does not automatically convert unhashable sets into
     temporary frozensets.  Raise a TypeError if the _key_
     is unhashable. Raise `PyExc_SystemError' if _anyset_ is not a
     set, frozenset, or an instance of a subtype.

 -- Function: int PySet_Add (PyObject *set, PyObject *key)
     Add _key_ to a set instance.  Does not apply to
     frozenset.  instances.  Return 0 on success or -1 on failure.
     Raise a TypeError if the _key_ is unhashable. Raise a
     MemoryError if there is no room to grow.  Raise a
     SystemError if _set_ is an not an instance of
     set or its subtype.

     Changed in version 2.6: Now works with instances of
     frozenset or its subtypes.  Like PyTuple_SetItem()
     in that it can be used to fill-in the values of brand new
     frozensets before they are exposed to other code.

  The following functions are available for instances of set
or its subtypes but not for instances of frozenset or
its subtypes.

 -- Function: int PySet_Discard (PyObject *set, PyObject *key)
     Return 1 if found and removed, 0 if not found (no action taken),
     and -1 if an error is encountered.  Does not raise KeyError
     for missing keys.  Raise a TypeError if the _key_
     is unhashable.  Unlike the Python `discard()' method, this
     function does not automatically convert unhashable sets into
     temporary frozensets. Raise `PyExc_SystemError' if _set_ is an not
     an instance of set or its subtype.

 -- Function: PyObject* PySet_Pop (PyObject *set)
     Return value: New reference.

     Return a new reference to an arbitrary object in the _set_, and
     removes the object from the _set_.  Return _NULL_ on failure.
     Raise KeyError if the set is empty. Raise a
     SystemError if _set_ is an not an instance of set
     or its subtype.

 -- Function: int PySet_Clear (PyObject *set)
     Empty an existing set of all elements.

7.7.5.16 Code Objects
.....................

Code objects are a low-level detail of the CPython implementation.
Each one represents a chunk of executable code that hasn't yet been
bound into a function.

 -- type: PyCodeObject
     The C structure of the objects used to describe code objects.  The
     fields of this type are subject to change at any time.

 -- var: PyTypeObject PyCode_Type
     This is an instance of PyTypeObject representing the
     Python code type.

 -- Function: int PyCode_Check (PyObject *co)
     Return true if _co_ is a code object

 -- Function: int PyCode_GetNumFree (PyObject *co)
     Return the number of free variables in _co_.

 -- Function: PyCodeObject *PyCode_New (int argcount, int nlocals,
          int stacksize, int flags, PyObject *code, PyObject *consts,
          PyObject *names, PyObject *varnames, PyObject *freevars,
          PyObject *cellvars, PyObject *filename, PyObject *name,
          int firstlineno, PyObject *lnotab)
     Return a new code object.  If you need a dummy code object to
     create a frame, use PyCode_NewEmpty() instead.  Calling
     PyCode_New() directly can bind you to a precise Python
     version since the definition of the bytecode changes often.

 -- Function: int PyCode_NewEmpty (const char *filename, const
          char *funcname, int firstlineno)
     Return a new empty code object with the specified filename,
     function name, and first line number.  It is illegal to
     exec or eval() the resulting code object.

7.8 Initialization, Finalization, and Threads
=============================================

 -- Function: void Py_Initialize ()
     Initialize the Python interpreter.  In an application embedding
     Python, this should be called before using any other Python/C API
     functions; with the exception of Py_SetProgramName(),
     PyEval_InitThreads(), PyEval_ReleaseLock(),
     and PyEval_AcquireLock(). This initializes the
     table of loaded modules (`sys.modules'), and creates the
     fundamental modules __builtin__, __main__ and
     sys.  It also initializes the module search path
     (`sys.path'). It does not set `sys.argv'; use
     PySys_SetArgvEx() for that.  This is a no-op when called for
     a second time (without calling Py_Finalize() first).
     There is no return value; it is a fatal error if the
     initialization fails.

 -- Function: void Py_InitializeEx (int initsigs)
     This function works like Py_Initialize() if
     _initsigs_ is 1. If _initsigs_ is 0, it skips initialization
     registration of signal handlers, which might be useful when Python
     is embedded.

     New in version 2.4.

 -- Function: int Py_IsInitialized ()
     Return true (nonzero) when the Python interpreter has been
     initialized, false (zero) if not.  After Py_Finalize().
     is called, this returns false until Py_Initialize().
     is called again.

 -- Function: void Py_Finalize ()
     Undo all initializations made by Py_Initialize() and
     subsequent use of Python/C API functions, and destroy all
     sub-interpreters (see Py_NewInterpreter() below) that
     were created and not yet destroyed since the last call to
     Py_Initialize().  Ideally, this frees all memory allocated
     by the Python interpreter.  This is a no-op when called for a
     second time (without calling Py_Initialize() again
     first).  There is no return value; errors during finalization are
     ignored.

     This function is provided for a number of reasons.  An embedding
     application might want to restart Python without having to restart
     the application itself.  An application that has loaded the Python
     interpreter from a dynamically loadable library (or DLL) might
     want to free all memory allocated by Python before unloading the
     DLL. During a hunt for memory leaks in an application a developer
     might want to free all memory allocated by Python before exiting
     from the application.

     *Bugs and caveats:* The destruction of modules and objects in
     modules is done in random order; this may cause destructors (see
     __del__() methods) to fail when they depend on other objects
     (even functions) or modules.  Dynamically loaded extension modules
     loaded by Python are not unloaded.  Small amounts of memory
     allocated by the Python interpreter may not be freed (if you find
     a leak, please report it).  Memory tied up in circular references
     between objects is not freed.  Some memory allocated by extension
     modules may not be freed.  Some extensions may not work properly
     if their initialization routine is called more than once; this can
     happen if an application calls Py_Initialize() and
     Py_Finalize() more than once.

 -- Function: PyThreadState* Py_NewInterpreter ()
     Create a new sub-interpreter.  This is an (almost) totally
     separate environment for the execution of Python code.  In
     particular, the new interpreter has separate, independent versions
     of all imported modules, including the fundamental modules
     __builtin__, __main__ and sys.  The table
     of loaded modules (`sys.modules') and the module search path
     (`sys.path') are also separate.  The new environment has no
     `sys.argv' variable.  It has new standard I/O stream file objects
     `sys.stdin', `sys.stdout' and `sys.stderr' (however these refer to
     the same underlying `FILE' structures in the C library).

     The return value points to the first thread state created in the
     new sub-interpreter.  This thread state is made in the current
     thread state.  Note that no actual thread is created; see the
     discussion of thread states below.  If creation of the new
     interpreter is unsuccessful, _NULL_ is returned; no exception is
     set since the exception state is stored in the current thread
     state and there may not be a current thread state.  (Like all
     other Python/C API functions, the global interpreter lock must be
     held before calling this function and is still held when it
     returns; however, unlike most other Python/C API functions, there
     needn't be a current thread state on entry.)

     Extension modules are shared between (sub-)interpreters as
     follows: the first time a particular extension is imported, it is
     initialized normally, and a (shallow) copy of its module's
     dictionary is squirreled away.  When the same extension is
     imported by another (sub-)interpreter, a new module is initialized
     and filled with the contents of this copy; the extension's `init'
     function is not called.  Note that this is different from what
     happens when an extension is imported after the interpreter has
     been completely re-initialized by calling Py_Finalize()
     and Py_Initialize().; in that case, the
     extension's `initmodule' function _is_ called again.

     *Bugs and caveats:* Because sub-interpreters (and the main
     interpreter) are part of the same process, the insulation between
     them isn't perfect -- for example, using low-level file operations
     like  os.close() they can (accidentally or
     maliciously) affect each other's open files.  Because of the way
     extensions are shared between (sub-)interpreters, some extensions
     may not work properly; this is especially likely when the
     extension makes use of (static) global variables, or when the
     extension manipulates its module's dictionary after its
     initialization.  It is possible to insert objects created in one
     sub-interpreter into a namespace of another sub-interpreter; this
     should be done with great care to avoid sharing user-defined
     functions, methods, instances or classes between sub-interpreters,
     since import operations executed by such objects may affect the
     wrong (sub-)interpreter's dictionary of loaded modules.  (XXX This
     is a hard-to-fix bug that will be addressed in a future release.)

     Also note that the use of this functionality is incompatible with
     extension modules such as PyObjC and ctypes that use the
     `PyGILState_*()' APIs (and this is inherent in the way the
     `PyGILState_*()' functions work).  Simple things may work, but
     confusing behavior will always be near.

 -- Function: void Py_EndInterpreter (PyThreadState *tstate)
     Destroy the (sub-)interpreter represented by the given thread
     state. The given thread state must be the current thread state.
     See the discussion of thread states below.  When the call returns,
     the current thread state is _NULL_.  All thread states associated
     with this interpreter are destroyed.  (The global interpreter lock
     must be held before calling this function and is still held when
     it returns.)  Py_Finalize() will destroy all
     sub-interpreters that haven't been explicitly destroyed at that
     point.

 -- Function: void Py_SetProgramName (char *name)
     This function should be called before Py_Initialize().
     is called for the first time, if it is called at all.  It tells
     the interpreter the value of the `argv[0]' argument to the
     `main()' function of the program.  This is used by
     Py_GetPath() and some other functions below to find the
     Python run-time libraries relative to the interpreter executable.
     The default value is `'python''.  The argument should point to a
     zero-terminated character string in static storage whose contents
     will not change for the duration of the program's execution.  No
     code in the Python interpreter will change the contents of this
     storage.

 -- Function: char* Py_GetProgramName ()
     Return the program name set with Py_SetProgramName(),
     or the default.  The returned string points into static storage;
     the caller should not modify its value.

 -- Function: char* Py_GetPrefix ()
     Return the _prefix_ for installed platform-independent files. This
     is derived through a number of complicated rules from the program
     name set with Py_SetProgramName() and some
     environment variables; for example, if the program name is
     `'/usr/local/bin/python'', the prefix is `'/usr/local''. The
     returned string points into static storage; the caller should not
     modify its value.  This corresponds to the *prefix* variable in
     the top-level `Makefile' and the `--prefix' argument to the
     *configure* script at build time.  The value is available to
     Python code as `sys.prefix'.  It is only useful on Unix.  See also
     the next function.

 -- Function: char* Py_GetExecPrefix ()
     Return the _exec-prefix_ for installed platform-_dependent_ files.
     This is derived through a number of complicated rules from the
     program name set with Py_SetProgramName() and some
     environment variables; for example, if the program name is
     `'/usr/local/bin/python'', the exec-prefix is `'/usr/local''.  The
     returned string points into static storage; the caller should not
     modify its value.  This corresponds to the *exec_prefix* variable
     in the top-level `Makefile' and the `--exec-prefix' argument to
     the *configure* script at build  time.  The value is available to
     Python code as `sys.exec_prefix'.  It is only useful on Unix.

     Background: The exec-prefix differs from the prefix when platform
     dependent files (such as executables and shared libraries) are
     installed in a different directory tree.  In a typical
     installation, platform dependent files may be installed in the
     `/usr/local/plat' subtree while platform independent may be
     installed in `/usr/local'.

     Generally speaking, a platform is a combination of hardware and
     software families, e.g.  Sparc machines running the Solaris 2.x
     operating system are considered the same platform, but Intel
     machines running Solaris 2.x are another platform, and Intel
     machines running Linux are yet another platform.  Different major
     revisions of the same operating system generally also form
     different platforms.  Non-Unix operating systems are a different
     story; the installation strategies on those systems are so
     different that the prefix and exec-prefix are meaningless, and set
     to the empty string. Note that compiled Python bytecode files are
     platform independent (but not independent from the Python version
     by which they were compiled!).

     System administrators will know how to configure the *mount* or
     *automount* programs to share `/usr/local' between platforms while
     having `/usr/local/plat' be a different filesystem for each
     platform.

 -- Function: char* Py_GetProgramFullPath ()
     Return the full program name of the Python executable; this is
     computed as a side-effect of deriving the default module search
     path  from the program name (set by Py_SetProgramName()
     above). The returned string points into static storage; the
     caller should not modify its value.  The value is available to
     Python code as `sys.executable'.

 -- Function: char* Py_GetPath ()
     Return the default module search path; this is computed from the
     program name (set by Py_SetProgramName() above) and
     some environment variables.  The returned string consists of a
     series of directory names separated by a platform dependent
     delimiter character.  The delimiter character is `':'' on Unix and
     Mac OS X, `';'' on Windows.  The returned string points into
     static storage; the caller should not modify its value.  The list
     sys.path is initialized with this value on interpreter
     startup; it can be (and usually is) modified later to change the
     search path for loading modules.

 -- Function: const char* Py_GetVersion ()
     Return the version of this Python interpreter.  This is a string
     that looks something like

         "1.5 (#67, Dec 31 1997, 22:34:28) [GCC 2.7.2.2]"

     The first word (up to the first space character) is the current
     Python version; the first three characters are the major and minor
     version separated by a period.  The returned string points into
     static storage; the caller should not modify its value.  The value
     is available to Python code as `sys.version'.

 -- Function: const char* Py_GetPlatform ()
     Return the platform identifier for the current platform.  On Unix,
     this is formed from the "official" name of the operating system,
     converted to lower case, followed by the major revision number;
     e.g., for Solaris 2.x, which is also known as SunOS 5.x, the value
     is `'sunos5''.  On Mac OS X, it is `'darwin''.  On Windows, it is
     `'win''.  The returned string points into static storage; the
     caller should not modify its value.  The value is available to
     Python code as `sys.platform'.

 -- Function: const char* Py_GetCopyright ()
     Return the official copyright string for the current Python
     version, for example

     `'Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam''

     The returned string points into static storage; the caller should
     not modify its value.  The value is available to Python code as
     `sys.copyright'.

 -- Function: const char* Py_GetCompiler ()
     Return an indication of the compiler used to build the current
     Python version, in square brackets, for example:

         "[GCC 2.7.2.2]"

     The returned string points into static storage; the caller should
     not modify its value.  The value is available to Python code as
     part of the variable `sys.version'.

 -- Function: const char* Py_GetBuildInfo ()
     Return information about the sequence number and build date and
     time  of the current Python interpreter instance, for example

         "#67, Aug  1 1997, 22:34:28"

     The returned string points into static storage; the caller should
     not modify its value.  The value is available to Python code as
     part of the variable `sys.version'.

 -- Function: void PySys_SetArgvEx (int argc, char **argv,
          int updatepath)
     Set sys.argv based on _argc_ and _argv_.  These
     parameters are similar to those passed to the program's `main()'
     function with the difference that the first entry should refer to
     the script file to be executed rather than the executable hosting
     the Python interpreter.  If there isn't a script that will be run,
     the first entry in _argv_ can be an empty string.  If this
     function fails to initialize sys.argv, a fatal
     condition is signalled using Py_FatalError().

     If _updatepath_ is zero, this is all the function does.  If
     _updatepath_ is non-zero, the function also modifies
     sys.path according to the following algorithm:

        - If the name of an existing script is passed in `argv[0]', the
          absolute path of the directory where the script is located is
          prepended to sys.path.

        - Otherwise (that is, if _argc_ is 0 or `argv[0]' doesn't point
          to an existing file name), an empty string is prepended to
          sys.path, which is the same as prepending the
          current working directory (`"."').

          Note: It is recommended that applications embedding the
          Python interpreter for purposes other than executing a single
          script pass 0 as _updatepath_, and update sys.path
          themselves if desired.  See CVE-2008-5983
          (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983).

          On versions before 2.6.6, you can achieve the same effect by
          manually popping the first sys.path element after
          having called PySys_SetArgv(), for example using:

              PyRun_SimpleString("import sys; sys.path.pop(0)\n");

     New in version 2.6.6.

 -- Function: void PySys_SetArgv (int argc, char **argv)
     This function works like PySys_SetArgvEx() with
     _updatepath_ set to 1.

 -- Function: void Py_SetPythonHome (char *home)
     Set the default "home" directory, that is, the location of the
     standard Python libraries.  The libraries are searched in
     `_home_/lib/python_version_' and `_home_/lib/python_version_'.
     The argument should point to a zero-terminated character string in
     static storage whose contents will not change for the duration of
     the program's execution.  No code in the Python interpreter will
     change the contents of this storage.

 -- Function: char* Py_GetPythonHome ()
     Return the default "home", that is, the value set by a previous
     call to Py_SetPythonHome(), or the value of the
     PYTHONHOME.  environment variable if it is set.

7.8.1 Thread State and the Global Interpreter Lock
--------------------------------------------------

The Python interpreter is not fully thread-safe.  In order to support
multi-threaded Python programs, there's a global lock, called the
_global interpreter lock_ or _GIL_, that must be held by the current
thread before it can safely access Python objects. Without the lock,
even the simplest operations could cause problems in a multi-threaded
program: for example, when two threads simultaneously increment the
reference count of the same object, the reference count could end up
being incremented only once instead of twice.

  Therefore, the rule exists that only the thread that has acquired the
global interpreter lock may operate on Python objects or call Python/C
API functions.  In order to support multi-threaded Python programs, the
interpreter regularly releases and reacquires the lock -- by default,
every 100 bytecode instructions (this can be changed with
sys.setcheckinterval()).  The lock is also released and
reacquired around potentially blocking I/O operations like reading or
writing a file, so that other threads can run while the thread that
requests the I/O is waiting for the I/O operation to complete.

  The Python interpreter needs to keep some bookkeeping information
separate per thread -- for this it uses a data structure called
PyThreadState.  There's one global variable, however: the pointer
to the current PyThreadState structure.  Before the
addition of _thread-local storage_ (_TLS_) the current thread state had
to be manipulated explicitly.

  This is easy enough in most cases.  Most code manipulating the global
interpreter lock has the following simple structure:

    Save the thread state in a local variable.
    Release the global interpreter lock.
    ...Do some blocking I/O operation...
    Reacquire the global interpreter lock.
    Restore the thread state from the local variable.

This is so common that a pair of macros exists to simplify it:

    Py_BEGIN_ALLOW_THREADS
    ...Do some blocking I/O operation...
    Py_END_ALLOW_THREADS

The Py_BEGIN_ALLOW_THREADS macro opens a new block and
declares a hidden local variable; the Py_END_ALLOW_THREADS.
macro closes the block.  Another advantage of using these two macros is
that when Python is compiled without thread support, they are defined
empty, thus saving the thread state and GIL manipulations.

  When thread support is enabled, the block above expands to the
following code:

    PyThreadState *_save;

    _save = PyEval_SaveThread();
    ...Do some blocking I/O operation...
    PyEval_RestoreThread(_save);

Using even lower level primitives, we can get roughly the same effect as
follows:

    PyThreadState *_save;

    _save = PyThreadState_Swap(NULL);
    PyEval_ReleaseLock();
    ...Do some blocking I/O operation...
    PyEval_AcquireLock();
    PyThreadState_Swap(_save);

There are some subtle differences; in particular, see
PyEval_RestoreThread().  saves and restores the value of the
global variable `errno', since the lock manipulation does not guarantee
that `errno' is left alone.  Also, when thread support is disabled,
see PyEval_SaveThread() and PyEval_RestoreThread()
don't manipulate the GIL; in this case, see
PyEval_ReleaseLock() and PyEval_AcquireLock() are
not available.  This is done so that dynamically loaded extensions
compiled with thread support enabled can be loaded by an interpreter
that was compiled with disabled thread support.

  The global interpreter lock is used to protect the pointer to the
current thread state.  When releasing the lock and saving the thread
state, the current thread state pointer must be retrieved before the
lock is released (since another thread could immediately acquire the
lock and store its own thread state in the global variable).
Conversely, when acquiring the lock and restoring the thread state, the
lock must be acquired before storing the thread state pointer.

  It is important to note that when threads are created from C, they
don't have the global interpreter lock, nor is there a thread state
data structure for them.  Such threads must bootstrap themselves into
existence, by first creating a thread state data structure, then
acquiring the lock, and finally storing their thread state pointer,
before they can start using the Python/C API.  When they are done, they
should reset the thread state pointer, release the lock, and finally
free their thread state data structure.

  Beginning with version 2.3, threads can now take advantage of the
`PyGILState_*()' functions to do all of the above automatically.  The
typical idiom for calling into Python from a C thread is now:

    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    /* Perform Python actions here.  */
    result = CallSomeFunction();
    /* evaluate result */

    /* Release the thread. No Python API allowed beyond this point. */
    PyGILState_Release(gstate);

Note that the `PyGILState_*()' functions assume there is only one global
interpreter (created automatically by Py_Initialize()).
Python still supports the creation of additional interpreters (using
Py_NewInterpreter()), but mixing multiple interpreters and
the `PyGILState_*()' API is unsupported.

  Another important thing to note about threads is their behaviour in
the face of the C `fork()' call. On most systems with `fork()', after a
process forks only the thread that issued the fork will exist. That also
means any locks held by other threads will never be released. Python
solves this for os.fork() by acquiring the locks it uses
internally before the fork, and releasing them afterwards. In addition,
it resets any Lock Objects in the child. When extending or
embedding Python, there is no way to inform Python of additional
(non-Python) locks that need to be acquired before or reset after a
fork. OS facilities such as `posix_atfork()' would need to be used to
accomplish the same thing.  Additionally, when extending or embedding
Python, calling `fork()' directly rather than through os.fork().
(and returning to or calling into Python) may result in a deadlock
by one of Python's internal locks being held by a thread that is
defunct after the fork.  PyOS_AfterFork() tries to reset
the necessary locks, but is not always able to.

 -- type: PyInterpreterState
     This data structure represents the state shared by a number of
     cooperating threads.  Threads belonging to the same interpreter
     share their module administration and a few other internal items.
     There are no public members in this structure.

     Threads belonging to different interpreters initially share
     nothing, except process state like available memory, open file
     descriptors and such.  The global interpreter lock is also shared
     by all threads, regardless of to which interpreter they belong.

 -- type: PyThreadState
     This data structure represents the state of a single thread.  The
     only public data member is PyInterpreterState *.
     `interp', which points to this thread's interpreter state.

 -- Function: void PyEval_InitThreads ()
     Initialize and acquire the global interpreter lock.  It should be
     called in the main thread before creating a second thread or
     engaging in any other thread operations such as
     PyEval_ReleaseLock() or `PyEval_ReleaseThread(tstate)'. It
     is not needed before calling PyEval_SaveThread() or
     PyEval_RestoreThread().

     This is a no-op when called for a second time.  It is safe to call
     this function before calling Py_Initialize().

     When only the main thread exists, no GIL operations are needed.
     This is a common situation (most Python programs do not use
     threads), and the lock operations slow the interpreter down a bit.
     Therefore, the lock is not created initially.  This situation is
     equivalent to having acquired the lock: when there is only a
     single thread, all object accesses are safe.  Therefore, when this
     function initializes the global interpreter lock, it also acquires
     it.  Before the Python thread module creates a new
     thread, knowing that either it has the lock or the lock hasn't
     been created yet, it calls PyEval_InitThreads().  When
     this call returns, it is guaranteed that the lock has been created
     and that the calling thread has acquired it.

     It is *not* safe to call this function when it is unknown which
     thread (if any) currently has the global interpreter lock.

     This function is not available when thread support is disabled at
     compile time.

 -- Function: int PyEval_ThreadsInitialized ()
     Returns a non-zero value if PyEval_InitThreads() has
     been called.  This function can be called without holding the GIL,
     and therefore can be used to avoid calls to the locking API when
     running single-threaded.  This function is not available when
     thread support is disabled at compile time.

     New in version 2.4.

 -- Function: void PyEval_AcquireLock ()
     Acquire the global interpreter lock.  The lock must have been
     created earlier.  If this thread already has the lock, a deadlock
     ensues.  This function is not available when thread support is
     disabled at compile time.

 -- Function: void PyEval_ReleaseLock ()
     Release the global interpreter lock.  The lock must have been
     created earlier.  This function is not available when thread
     support is disabled at compile time.

 -- Function: void PyEval_AcquireThread (PyThreadState *tstate)
     Acquire the global interpreter lock and set the current thread
     state to _tstate_, which should not be _NULL_.  The lock must have
     been created earlier.  If this thread already has the lock,
     deadlock ensues.  This function is not available when thread
     support is disabled at compile time.

 -- Function: void PyEval_ReleaseThread (PyThreadState *tstate)
     Reset the current thread state to _NULL_ and release the global
     interpreter lock.  The lock must have been created earlier and
     must be held by the current thread.  The _tstate_ argument, which
     must not be _NULL_, is only used to check that it represents the
     current thread state -- if it isn't, a fatal error is reported.
     This function is not available when thread support is disabled at
     compile time.

 -- Function: PyThreadState* PyEval_SaveThread ()
     Release the global interpreter lock (if it has been created and
     thread support is enabled) and reset the thread state to _NULL_,
     returning the previous thread state (which is not _NULL_).  If the
     lock has been created, the current thread must have acquired it.
     (This function is available even when thread support is disabled
     at compile time.)

 -- Function: void PyEval_RestoreThread (PyThreadState *tstate)
     Acquire the global interpreter lock (if it has been created and
     thread support is enabled) and set the thread state to _tstate_,
     which must not be _NULL_.  If the lock has been created, the
     current thread must not have acquired it, otherwise deadlock
     ensues.  (This function is available even when thread support is
     disabled at compile time.)

 -- Function: void PyEval_ReInitThreads ()
     This function is called from PyOS_AfterFork() to
     ensure that newly created child processes don't hold locks
     referring to threads which are not running in the child process.

  The following macros are normally used without a trailing semicolon;
look for example usage in the Python source distribution.

 -- macro: Py_BEGIN_ALLOW_THREADS
     This macro expands to `{ PyThreadState *_save; _save =
     PyEval_SaveThread();'.  Note that it contains an opening brace; it
     must be matched with a following Py_END_ALLOW_THREADS.
     macro.  See above for further discussion of this macro.  It is a
     no-op when thread support is disabled at compile time.

 -- macro: Py_END_ALLOW_THREADS
     This macro expands to `PyEval_RestoreThread(_save); }'. Note that
     it contains a closing brace; it must be matched with an earlier
     Py_BEGIN_ALLOW_THREADS macro.  See above for further
     discussion of this macro.  It is a no-op when thread support is
     disabled at compile time.

 -- macro: Py_BLOCK_THREADS
     This macro expands to `PyEval_RestoreThread(_save);': it is
     equivalent to Py_END_ALLOW_THREADS without the
     closing brace.  It is a no-op when thread support is disabled at
     compile time.

 -- macro: Py_UNBLOCK_THREADS
     This macro expands to `_save = PyEval_SaveThread();': it is
     equivalent to Py_BEGIN_ALLOW_THREADS without the
     opening brace and variable declaration.  It is a no-op when thread
     support is disabled at compile time.

  All of the following functions are only available when thread support
is enabled at compile time, and must be called only when the global
interpreter lock has been created.

 -- Function: PyInterpreterState* PyInterpreterState_New ()
     Create a new interpreter state object.  The global interpreter
     lock need not be held, but may be held if it is necessary to
     serialize calls to this function.

 -- Function: void PyInterpreterState_Clear (PyInterpreterState *interp)
     Reset all information in an interpreter state object.  The global
     interpreter lock must be held.

 -- Function: void PyInterpreterState_Delete
          (PyInterpreterState *interp)
     Destroy an interpreter state object.  The global interpreter lock
     need not be held.  The interpreter state must have been reset with
     a previous call to PyInterpreterState_Clear().

 -- Function: PyThreadState* PyThreadState_New
          (PyInterpreterState *interp)
     Create a new thread state object belonging to the given
     interpreter object.  The global interpreter lock need not be held,
     but may be held if it is necessary to serialize calls to this
     function.

 -- Function: void PyThreadState_Clear (PyThreadState *tstate)
     Reset all information in a thread state object.  The global
     interpreter lock must be held.

 -- Function: void PyThreadState_Delete (PyThreadState *tstate)
     Destroy a thread state object.  The global interpreter lock need
     not be held.  The thread state must have been reset with a
     previous call to PyThreadState_Clear().

 -- Function: PyThreadState* PyThreadState_Get ()
     Return the current thread state.  The global interpreter lock must
     be held.  When the current thread state is _NULL_, this issues a
     fatal error (so that the caller needn't check for _NULL_).

 -- Function: PyThreadState* PyThreadState_Swap (PyThreadState *tstate)
     Swap the current thread state with the thread state given by the
     argument _tstate_, which may be _NULL_.  The global interpreter
     lock must be held.

 -- Function: PyObject* PyThreadState_GetDict ()
     Return value: Borrowed reference.

     Return a dictionary in which extensions can store thread-specific
     state information.  Each extension should use a unique key to use
     to store state in the dictionary.  It is okay to call this
     function when no current thread state is available. If this
     function returns _NULL_, no exception has been raised and the
     caller should assume no current thread state is available.

     Changed in version 2.3: Previously this could only be called when
     a current thread is active, and _NULL_ meant that an exception was
     raised.

 -- Function: int PyThreadState_SetAsyncExc (long id, PyObject *exc)
     Asynchronously raise an exception in a thread. The _id_ argument
     is the thread id of the target thread; _exc_ is the exception
     object to be raised. This function does not steal any references
     to _exc_. To prevent naive misuse, you must write your own C
     extension to call this.  Must be called with the GIL held.
     Returns the number of thread states modified; this is normally
     one, but will be zero if the thread id isn't found.  If _exc_ is
     `NULL', the pending exception (if any) for the thread is cleared.
     This raises no exceptions.

     New in version 2.3.

 -- Function: PyGILState_STATE PyGILState_Ensure ()
     Ensure that the current thread is ready to call the Python C API
     regardless of the current state of Python, or of the global
     interpreter lock. This may be called as many times as desired by a
     thread as long as each call is matched with a call to
     PyGILState_Release(). In general, other thread-related APIs
     may be used between PyGILState_Ensure() and
     PyGILState_Release() calls as long as the thread state is
     restored to its previous state before the Release().  For example,
     normal usage of the Py_BEGIN_ALLOW_THREADS and
     Py_END_ALLOW_THREADS macros is acceptable.

     The return value is an opaque "handle" to the thread state when
     PyGILState_Ensure() was called, and must be passed to
     PyGILState_Release() to ensure Python is left in the
     same state. Even though recursive calls are allowed, these handles
     _cannot_ be shared - each unique call to
     PyGILState_Ensure() must save the handle for its call to
     PyGILState_Release().

     When the function returns, the current thread will hold the GIL.
     Failure is a fatal error.

     New in version 2.3.

 -- Function: void PyGILState_Release (PyGILState_STATE)
     Release any resources previously acquired.  After this call,
     Python's state will be the same as it was prior to the
     corresponding PyGILState_Ensure() call (but generally
     this state will be unknown to the caller, hence the use of the
     GILState API.)

     Every call to PyGILState_Ensure() must be matched by
     a call to PyGILState_Release() on the same thread.

     New in version 2.3.

7.8.2 Asynchronous Notifications
--------------------------------

A mechanism is provided to make asynchronous notifications to the main
interpreter thread.  These notifications take the form of a function
pointer and a void argument.

  Every check interval, when the global interpreter lock is released and
reacquired, Python will also call any such provided functions.  This
can be used for example by asynchronous IO handlers.  The notification
can be scheduled from a worker thread and the actual call than made at
the earliest convenience by the main thread where it has possession of
the global interpreter lock and can perform any Python API calls.

 -- Function: void Py_AddPendingCall (int (*func)(void *, void *arg))
     Post a notification to the Python main thread.  If successful,
     _func_ will be called with the argument _arg_ at the earliest
     convenience.  _func_ will be called having the global interpreter
     lock held and can thus use the full Python API and can take any
     action such as setting object attributes to signal IO completion.
     It must return 0 on success, or -1 signalling an exception.  The
     notification function won't be interrupted to perform another
     asynchronous notification recursively, but it can still be
     interrupted to switch threads if the global interpreter lock is
     released, for example, if it calls back into Python code.

     This function returns 0 on success in which case the notification
     has been scheduled.  Otherwise, for example if the notification
     buffer is full, it returns -1 without setting any exception.

     This function can be called on any thread, be it a Python thread
     or some other system thread.  If it is a Python thread, it doesn't
     matter if it holds the global interpreter lock or not.

     New in version 2.7.

7.8.3 Profiling and Tracing
---------------------------

The Python interpreter provides some low-level support for attaching
profiling and execution tracing facilities.  These are used for
profiling, debugging, and coverage analysis tools.

  Starting with Python 2.2, the implementation of this facility was
substantially revised, and an interface from C was added.  This C
interface allows the profiling or tracing code to avoid the overhead of
calling through Python-level callable objects, making a direct C
function call instead.  The essential attributes of the facility have
not changed; the interface allows trace functions to be installed
per-thread, and the basic events reported to the trace function are the
same as had been reported to the Python-level trace functions in
previous versions.

 -- type: int (*Py_tracefunc) (PyObject *obj, PyFrameObject *frame,
          int what, PyObject *arg)
     The type of the trace function registered using
     PyEval_SetProfile() and PyEval_SetTrace(). The
     first parameter is the object passed to the registration function
     as _obj_, _frame_ is the frame object to which the event pertains,
     _what_ is one of the constants `PyTrace_CALL',
     `PyTrace_EXCEPTION', `PyTrace_LINE', `PyTrace_RETURN',
     `PyTrace_C_CALL', `PyTrace_C_EXCEPTION', or `PyTrace_C_RETURN',
     and _arg_ depends on the value of _what_:

     Value of _what_                    Meaning of _arg_
     ------------------------------------------------------------------------------
     `PyTrace_CALL'                     Always _NULL_.
     `PyTrace_EXCEPTION'                Exception information as returned by
                                        See sys.exc_info().
     `PyTrace_LINE'                     Always _NULL_.
     `PyTrace_RETURN'                   Value being returned to the caller, or
                                        _NULL_ if caused by an exception.
     `PyTrace_C_CALL'                   Function object being called.
     `PyTrace_C_EXCEPTION'              Function object being called.
     `PyTrace_C_RETURN'                 Function object being called.

 -- var: int PyTrace_CALL
     The value of the _what_ parameter to a Py_tracefunc.
     function when a new call to a function or method is being
     reported, or a new entry into a generator.  Note that the creation
     of the iterator for a generator function is not reported as there
     is no control transfer to the Python bytecode in the corresponding
     frame.

 -- var: int PyTrace_EXCEPTION
     The value of the _what_ parameter to a Py_tracefunc.
     function when an exception has been raised.  The callback function
     is called with this value for _what_ when after any bytecode is
     processed after which the exception becomes set within the frame
     being executed.  The effect of this is that as exception
     propagation causes the Python stack to unwind, the callback is
     called upon return to each frame as the exception propagates.
     Only trace functions receives these events; they are not needed by
     the profiler.

 -- var: int PyTrace_LINE
     The value passed as the _what_ parameter to a trace function (but
     not a profiling function) when a line-number event is being
     reported.

 -- var: int PyTrace_RETURN
     The value for the _what_ parameter to Py_tracefunc.
     functions when a call is returning without propagating an
     exception.

 -- var: int PyTrace_C_CALL
     The value for the _what_ parameter to Py_tracefunc.
     functions when a C function is about to be called.

 -- var: int PyTrace_C_EXCEPTION
     The value for the _what_ parameter to Py_tracefunc.
     functions when a C function has raised an exception.

 -- var: int PyTrace_C_RETURN
     The value for the _what_ parameter to Py_tracefunc.
     functions when a C function has returned.

 -- Function: void PyEval_SetProfile (Py_tracefunc func, PyObject *obj)
     Set the profiler function to _func_.  The _obj_ parameter is
     passed to the function as its first parameter, and may be any
     Python object, or _NULL_.  If the profile function needs to
     maintain state, using a different value for _obj_ for each thread
     provides a convenient and thread-safe place to store it.  The
     profile function is called for all monitored events except the
     line-number events.

 -- Function: void PyEval_SetTrace (Py_tracefunc func, PyObject *obj)
     Set the tracing function to _func_.  This is similar to
     PyEval_SetProfile(), except the tracing function does receive
     line-number events.

 -- Function: PyObject* PyEval_GetCallStats (PyObject *self)
     Return a tuple of function call counts.  There are constants
     defined for the positions within the tuple:

     Name                                Value
     ------------------------------------------------
     `PCALL_ALL'                         0
     `PCALL_FUNCTION'                    1
     `PCALL_FAST_FUNCTION'               2
     `PCALL_FASTER_FUNCTION'             3
     `PCALL_METHOD'                      4
     `PCALL_BOUND_METHOD'                5
     `PCALL_CFUNCTION'                   6
     `PCALL_TYPE'                        7
     `PCALL_GENERATOR'                   8
     `PCALL_OTHER'                       9
     `PCALL_POP'                         10

     `PCALL_FAST_FUNCTION' means no argument tuple needs to be created.
     `PCALL_FASTER_FUNCTION' means that the fast-path frame setup code
     is used.

     If there is a method call where the call can be optimized by
     changing the argument tuple and calling the function directly, it
     gets recorded twice.

     This function is only present if Python is compiled with
     `CALL_PROFILE' defined.

7.8.4 Advanced Debugger Support
-------------------------------

These functions are only intended to be used by advanced debugging
tools.

 -- Function: PyInterpreterState* PyInterpreterState_Head ()
     Return the interpreter state object at the head of the list of all
     such objects.

     New in version 2.2.

 -- Function: PyInterpreterState* PyInterpreterState_Next
          (PyInterpreterState *interp)
     Return the next interpreter state object after _interp_ from the
     list of all such objects.

     New in version 2.2.

 -- Function: PyThreadState * PyInterpreterState_ThreadHead
          (PyInterpreterState *interp)
     Return the a pointer to the first PyThreadState.
     object in the list of threads associated with the interpreter
     _interp_.

     New in version 2.2.

 -- Function: PyThreadState* PyThreadState_Next (PyThreadState *tstate)
     Return the next thread state object after _tstate_ from the list
     of all such objects belonging to the same
     PyInterpreterState object.

     New in version 2.2.

7.9 Memory Management
=====================

7.9.1 Overview
--------------

Memory management in Python involves a private heap containing all
Python objects and data structures. The management of this private heap
is ensured internally by the _Python memory manager_.  The Python
memory manager has different components which deal with various dynamic
storage management aspects, like sharing, segmentation, preallocation
or caching.

  At the lowest level, a raw memory allocator ensures that there is
enough room in the private heap for storing all Python-related data by
interacting with the memory manager of the operating system. On top of
the raw memory allocator, several object-specific allocators operate on
the same heap and implement distinct memory management policies adapted
to the peculiarities of every object type. For example, integer objects
are managed differently within the heap than strings, tuples or
dictionaries because integers imply different storage requirements and
speed/space tradeoffs. The Python memory manager thus delegates some of
the work to the object-specific allocators, but ensures that the latter
operate within the bounds of the private heap.

  It is important to understand that the management of the Python heap
is performed by the interpreter itself and that the user has no control
over it, even if she regularly manipulates object pointers to memory
blocks inside that heap.  The allocation of heap space for Python
objects and other internal buffers is performed on demand by the Python
memory manager through the Python/C API functions listed in this
document.

  To avoid memory corruption, extension writers should never try to
operate on Python objects with the functions exported by the C library:
`malloc()', `calloc()', `realloc()' and `free()'.  This will result in
mixed calls between the C allocator and the Python memory manager with
fatal consequences, because they implement different algorithms and
operate on different heaps.  However, one may safely allocate and
release memory blocks with the C library allocator for individual
purposes, as shown in the following example:

    PyObject *res;
    char *buf = (char *) malloc(BUFSIZ); /* for I/O */

    if (buf == NULL)
        return PyErr_NoMemory();
    ...Do some I/O operation involving buf...
    res = PyString_FromString(buf);
    free(buf); /* malloc'ed */
    return res;

In this example, the memory request for the I/O buffer is handled by
the C library allocator. The Python memory manager is involved only in
the allocation of the string object returned as a result.

  In most situations, however, it is recommended to allocate memory
from the Python heap specifically because the latter is under control
of the Python memory manager. For example, this is required when the
interpreter is extended with new object types written in C. Another
reason for using the Python heap is the desire to _inform_ the Python
memory manager about the memory needs of the extension module. Even
when the requested memory is used exclusively for internal,
highly-specific purposes, delegating all memory requests to the Python
memory manager causes the interpreter to have a more accurate image of
its memory footprint as a whole. Consequently, under certain
circumstances, the Python memory manager may or may not trigger
appropriate actions, like garbage collection, memory compaction or
other preventive procedures. Note that by using the C library allocator
as shown in the previous example, the allocated memory for the I/O
buffer escapes completely the Python memory manager.

7.9.2 Memory Interface
----------------------

The following function sets, modeled after the ANSI C standard, but
specifying behavior when requesting zero bytes, are available for
allocating and releasing memory from the Python heap:

 -- Function: void* PyMem_Malloc (size_t n)
     Allocates _n_ bytes and returns a pointer of type `void*' to the
     allocated memory, or _NULL_ if the request fails. Requesting zero
     bytes returns a distinct non-_NULL_ pointer if possible, as if
     `PyMem_Malloc(1)()' had been called instead. The memory will not
     have been initialized in any way.

 -- Function: void* PyMem_Realloc (void *p, size_t n)
     Resizes the memory block pointed to by _p_ to _n_ bytes. The
     contents will be unchanged to the minimum of the old and the new
     sizes. If _p_ is _NULL_, the call is equivalent to
     `PyMem_Malloc(n)()'; else if _n_ is equal to zero, the memory
     block is resized but is not freed, and the returned pointer is
     non-_NULL_.  Unless _p_ is _NULL_, it must have been returned by a
     previous call to PyMem_Malloc() or
     PyMem_Realloc(). If the request fails, see PyMem_Realloc()
     returns _NULL_ and _p_ remains a valid pointer to the
     previous memory area.

 -- Function: void PyMem_Free (void *p)
     Frees the memory block pointed to by _p_, which must have been
     returned by a previous call to PyMem_Malloc() or
     PyMem_Realloc().  Otherwise, or if `PyMem_Free(p)()' has been
     called before, undefined behavior occurs. If _p_ is _NULL_, no
     operation is performed.

  The following type-oriented macros are provided for convenience.
Note  that _TYPE_ refers to any C type.

 -- Function: TYPE* PyMem_New (TYPE, size_t n)
     Same as PyMem_Malloc(), but allocates `(n *
     sizeof(TYPE))' bytes of memory.  Returns a pointer cast to
     `TYPE*'.  The memory will not have been initialized in any way.

 -- Function: TYPE* PyMem_Resize (void *p, TYPE, size_t n)
     Same as PyMem_Realloc(), but the memory block is
     resized to `(n * sizeof(TYPE))' bytes.  Returns a pointer cast to
     `TYPE*'. On return, _p_ will be a pointer to the new memory area,
     or _NULL_ in the event of failure.  This is a C preprocessor
     macro; p is always reassigned.  Save the original value of p to
     avoid losing memory when handling errors.

 -- Function: void PyMem_Del (void *p)
     Same as PyMem_Free().

  In addition, the following macro sets are provided for calling the
Python memory allocator directly, without involving the C API functions
listed above. However, note that their use does not preserve binary
compatibility across Python versions and is therefore deprecated in
extension modules.

  `PyMem_MALLOC()', `PyMem_REALLOC()', `PyMem_FREE()'.

  `PyMem_NEW()', `PyMem_RESIZE()', `PyMem_DEL()'.

7.9.3 Examples
--------------

Here is the example from section Overview, rewritten so
that the I/O buffer is allocated from the Python heap by using the
first function set:

    PyObject *res;
    char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */

    if (buf == NULL)
        return PyErr_NoMemory();
    /* ...Do some I/O operation involving buf... */
    res = PyString_FromString(buf);
    PyMem_Free(buf); /* allocated with PyMem_Malloc */
    return res;

The same code using the type-oriented function set:

    PyObject *res;
    char *buf = PyMem_New(char, BUFSIZ); /* for I/O */

    if (buf == NULL)
        return PyErr_NoMemory();
    /* ...Do some I/O operation involving buf... */
    res = PyString_FromString(buf);
    PyMem_Del(buf); /* allocated with PyMem_New */
    return res;

Note that in the two examples above, the buffer is always manipulated
via functions belonging to the same set. Indeed, it is required to use
the same memory API family for a given memory block, so that the risk
of mixing different allocators is reduced to a minimum. The following
code sequence contains two errors, one of which is labeled as _fatal_
because it mixes two different allocators operating on different heaps.

    char *buf1 = PyMem_New(char, BUFSIZ);
    char *buf2 = (char *) malloc(BUFSIZ);
    char *buf3 = (char *) PyMem_Malloc(BUFSIZ);
    ...
    PyMem_Del(buf3);  /* Wrong -- should be PyMem_Free() */
    free(buf2);       /* Right -- allocated via malloc() */
    free(buf1);       /* Fatal -- should be PyMem_Del()  */

In addition to the functions aimed at handling raw memory blocks from
the Python heap, objects in Python are allocated and released with
PyObject_New(), PyObject_NewVar() and
PyObject_Del().

  These will be explained in the next chapter on defining and
implementing new object types in C.

7.10 Object Implementation Support
==================================

This chapter describes the functions, types, and macros used when
defining new object types.

7.10.1 Allocating Objects on the Heap
-------------------------------------

 -- Function: PyObject* _PyObject_New (PyTypeObject *type)
     Return value: New reference.

 -- Function: PyVarObject* _PyObject_NewVar (PyTypeObject *type,
          Py_ssize_t size)
     Return value: New reference.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: void _PyObject_Del (PyObject *op)

 -- Function: PyObject* PyObject_Init (PyObject *op, PyTypeObject *type)
     Return value: Borrowed reference.

     Initialize a newly-allocated object _op_ with its type and initial
     reference.  Returns the initialized object.  If _type_ indicates
     that the object participates in the cyclic garbage detector, it is
     added to the detector's set of observed objects. Other fields of
     the object are not affected.

 -- Function: PyVarObject* PyObject_InitVar (PyVarObject *op,
          PyTypeObject *type, Py_ssize_t size)
     Return value: Borrowed reference.

     This does everything PyObject_Init() does, and also
     initializes the length information for a variable-size object.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: TYPE* PyObject_New (TYPE, PyTypeObject *type)
     Return value: New reference.

     Allocate a new Python object using the C structure type _TYPE_ and
     the Python type object _type_.  Fields not defined by the Python
     object header are not initialized; the object's reference count
     will be one.  The size of the memory allocation is determined from
     the `tp_basicsize' field of the type object.

 -- Function: TYPE* PyObject_NewVar (TYPE, PyTypeObject *type,
          Py_ssize_t size)
     Return value: New reference.

     Allocate a new Python object using the C structure type _TYPE_ and
     the Python type object _type_.  Fields not defined by the Python
     object header are not initialized.  The allocated memory allows
     for the _TYPE_ structure plus _size_ fields of the size given by
     the `tp_itemsize' field of _type_.  This is useful for
     implementing objects like tuples, which are able to determine
     their size at construction time.  Embedding the array of fields
     into the same allocation decreases the number of allocations,
     improving the memory management efficiency.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: void PyObject_Del (PyObject *op)
     Releases memory allocated to an object using PyObject_New()
     or PyObject_NewVar().  This is normally called
     from the `tp_dealloc' handler specified in the object's type.  The
     fields of the object should not be accessed after this call as the
     memory is no longer a valid Python object.

 -- Function: PyObject* Py_InitModule (char *name, PyMethodDef *methods)
     Return value: Borrowed reference.

     Create a new module object based on a name and table of functions,
     returning the new module object.

     Changed in version 2.3: Older versions of Python did not support
     _NULL_ as the value for the _methods_ argument.

 -- Function: PyObject* Py_InitModule3 (char *name,
          PyMethodDef *methods, char *doc)
     Return value: Borrowed reference.

     Create a new module object based on a name and table of functions,
     returning the new module object.  If _doc_ is non-_NULL_, it will
     be used to define the docstring for the module.

     Changed in version 2.3: Older versions of Python did not support
     _NULL_ as the value for the _methods_ argument.

 -- Function: PyObject* Py_InitModule4 (char *name,
          PyMethodDef *methods, char *doc, PyObject *self, int apiver)
     Return value: Borrowed reference.

     Create a new module object based on a name and table of functions,
     returning the new module object.  If _doc_ is non-_NULL_, it will
     be used to define the docstring for the module.  If _self_ is
     non-_NULL_, it will passed to the functions of the module as their
     (otherwise _NULL_) first parameter.  (This was added as an
     experimental feature, and there are no known uses in the current
     version of Python.)  For _apiver_, the only value which should be
     passed is defined by the constant `PYTHON_API_VERSION'.

          Note: Most uses of this function should probably be using the
          Py_InitModule3() instead; only use this if you
          are sure you need it.

     Changed in version 2.3: Older versions of Python did not support
     _NULL_ as the value for the _methods_ argument.

 -- var: PyObject _Py_NoneStruct
     Object which is visible in Python as `None'.  This should only be
     accessed using the `Py_None' macro, which evaluates to a pointer
     to this object.

7.10.2 Common Object Structures
-------------------------------

There are a large number of structures which are used in the definition
of object types for Python.  This section describes these structures
and how they are used.

  All Python objects ultimately share a small number of fields at the
beginning of the object's representation in memory.  These are
represented by the PyObject and PyVarObject.
types, which are defined, in turn, by the expansions of some macros
also used, whether directly or indirectly, in the definition of all
other Python objects.

 -- type: PyObject
     All object types are extensions of this type.  This is a type which
     contains the information Python needs to treat a pointer to an
     object as an object.  In a normal "release" build, it contains
     only the object's reference count and a pointer to the
     corresponding type object.  It corresponds to the fields defined
     by the expansion of the `PyObject_HEAD' macro.

 -- type: PyVarObject
     This is an extension of PyObject that adds the
     `ob_size' field.  This is only used for objects that have some
     notion of _length_.  This type does not often appear in the
     Python/C API.  It corresponds to the fields defined by the
     expansion of the `PyObject_VAR_HEAD' macro.

  These macros are used in the definition of PyObject and
PyVarObject.:

 -- macro: PyObject_HEAD
     This is a macro which expands to the declarations of the fields of
     the PyObject type; it is used when declaring new types
     which represent objects without a varying length.  The specific
     fields it expands to depend on the definition of `Py_TRACE_REFS'.
     By default, that macro is not defined, and PyObject_HEAD
     expands to:

         Py_ssize_t ob_refcnt;
         PyTypeObject *ob_type;

     When `Py_TRACE_REFS' is defined, it expands to:

         PyObject *_ob_next, *_ob_prev;
         Py_ssize_t ob_refcnt;
         PyTypeObject *ob_type;

 -- macro: PyObject_VAR_HEAD
     This is a macro which expands to the declarations of the fields of
     the PyVarObject type; it is used when declaring new
     types which represent objects with a length that varies from
     instance to instance.  This macro always expands to:

         PyObject_HEAD
         Py_ssize_t ob_size;

     Note that PyObject_HEAD is part of the expansion, and
     that its own expansion varies depending on the definition of
     `Py_TRACE_REFS'.

 -- macro: PyObject_HEAD_INIT (type)
     This is a macro which expands to initialization values for a new
     PyObject type.  This macro expands to:

         _PyObject_EXTRA_INIT
         1, type,

 -- macro: PyVarObject_HEAD_INIT (type, size)
     This is a macro which expands to initialization values for a new
     PyVarObject type, including the `ob_size' field.
     This macro expands to:

         _PyObject_EXTRA_INIT
         1, type, size,

 -- type: PyCFunction
     Type of the functions used to implement most Python callables in C.
     Functions of this type take two PyObject* parameters
     and return one such value.  If the return value is _NULL_, an
     exception shall have been set.  If not _NULL_, the return value is
     interpreted as the return value of the function as exposed in
     Python.  The function must return a new reference.

 -- type: PyMethodDef
     Structure used to describe a method of an extension type.  This
     structure has four fields:

     Field                  C Type            Meaning
     -----------------------------------------------------------------------------
     `ml_name'              char *            name of the method
     `ml_meth'              PyCFunction       pointer to the C implementation
     `ml_flags'             int               flag bits indicating how the call
                                              should be constructed
     `ml_doc'               char *            points to the contents of the
                                              docstring

  The `ml_meth' is a C function pointer.  The functions may be of
different types, but they always return PyObject*.  If the
function is not of the PyCFunction, the compiler will
require a cast in the method table.  Even though PyCFunction
defines the first parameter as PyObject*, it is common
that the method implementation uses a the specific C type of the _self_
object.

  The `ml_flags' field is a bitfield which can include the following
flags.  The individual flags indicate either a calling convention or a
binding convention.  Of the calling convention flags, only
METH_VARARGS and METH_KEYWORDS can be combined (but
note that METH_KEYWORDS.  alone is equivalent to
`METH_VARARGS | METH_KEYWORDS'). Any of the calling convention flags
can be combined with a binding flag.

 -- Data: METH_VARARGS
     This is the typical calling convention, where the methods have the
     type PyCFunction. The function expects two
     PyObject* values.  The first one is the _self_ object for
     methods; for module functions, it is the module object.  The
     second parameter (often called _args_) is a tuple object
     representing all arguments.  This parameter is typically processed
     using PyArg_ParseTuple() or PyArg_UnpackTuple().

 -- Data: METH_KEYWORDS
     Methods with these flags must be of type `PyCFunctionWithKeywords'.
     The function expects three parameters: _self_, _args_, and a
     dictionary of all the keyword arguments.  The flag is typically
     combined with METH_VARARGS, and the parameters are
     typically processed using PyArg_ParseTupleAndKeywords().

 -- Data: METH_NOARGS
     Methods without parameters don't need to check whether arguments
     are given if they are listed with the METH_NOARGS.
     flag.  They need to be of type PyCFunction.  The first
     parameter is typically named `self' and will hold a reference to
     the module or object instance.  In all cases the second parameter
     will be _NULL_.

 -- Data: METH_O
     Methods with a single object argument can be listed with the
     METH_O.  flag, instead of invoking PyArg_ParseTuple()
     with a `"O"' argument.  They have the type PyCFunction,
     with the _self_ parameter, and a PyObject*.
     parameter representing the single argument.

 -- Data: METH_OLDARGS
     This calling convention is deprecated.  The method must be of type
     PyCFunction.  The second argument is _NULL_ if no
     arguments are given, a single object if exactly one argument is
     given, and a tuple of objects if more than one argument is given.
     There is no way for a function using this convention to
     distinguish between a call with multiple arguments and a call with
     a tuple as the only argument.

  These two constants are not used to indicate the calling convention
but the binding when use with methods of classes.  These may not be
used for functions defined for modules.  At most one of these flags may
be set for any given method.

 -- Data: METH_CLASS
     The method will be passed the type object as the first parameter
     rather than an instance of the type.  This is used to create
     _class methods_, similar to what is created when using the
     classmethod() built-in function.

     New in version 2.3.

 -- Data: METH_STATIC
     The method will be passed _NULL_ as the first parameter rather
     than an instance of the type.  This is used to create _static
     methods_, similar to what is created when using the
     staticmethod() built-in function.

     New in version 2.3.

  One other constant controls whether a method is loaded in place of
another definition with the same method name.

 -- Data: METH_COEXIST
     The method will be loaded in place of existing definitions.
     Without _METH_COEXIST_, the default is to skip repeated
     definitions.  Since slot wrappers are loaded before the method
     table, the existence of a _sq_contains_ slot, for example, would
     generate a wrapped method named __contains__() and
     preclude the loading of a corresponding PyCFunction with the same
     name.  With the flag defined, the PyCFunction will be loaded in
     place of the wrapper object and will co-exist with the slot.  This
     is helpful because calls to PyCFunctions are optimized more than
     wrapper object calls.

     New in version 2.4.

 -- type: PyMemberDef
     Structure which describes an attribute of a type which corresponds
     to a C struct member.  Its fields are:

     Field                  C Type            Meaning
     -----------------------------------------------------------------------------
     `name'                 char *            name of the member
     type.       int               the type of the member in the C
                                              struct
     `offset'               Py_ssize_t        the offset in bytes that the
                                              member is located on the type's
                                              object struct
     `flags'                int               flag bits indicating if the field
                                              should be read-only or writable
     `doc'                  char *            points to the contents of the
                                              docstring

     type can be one of many `T_' macros corresponding to
     various C types.  When the member is accessed in Python, it will
     be converted to the equivalent Python type.

     Macro name          C type
     -------------------------------------------
     T_SHORT             short
     T_INT               int
     T_LONG              long
     T_FLOAT             float
     T_DOUBLE            double
     T_STRING            char *
     T_OBJECT            PyObject *
     T_OBJECT_EX         PyObject *
     T_CHAR              char
     T_BYTE              char
     T_UBYTE             unsigned char
     T_UINT              unsigned int
     T_USHORT            unsigned short
     T_ULONG             unsigned long
     T_BOOL              char
     T_LONGLONG          long long
     T_ULONGLONG         unsigned long long
     T_PYSSIZET          Py_ssize_t

     `T_OBJECT' and `T_OBJECT_EX' differ in that `T_OBJECT' returns
     `None' if the member is _NULL_ and `T_OBJECT_EX' raises an
     AttributeError.  Try to use `T_OBJECT_EX' over `T_OBJECT'
     because `T_OBJECT_EX' handles use of the del statement
     on that attribute more correctly than `T_OBJECT'.

     `flags' can be 0 for write and read access or `READONLY' for
     read-only access.  Using `T_STRING' for type implies
     `READONLY'.  Only `T_OBJECT' and `T_OBJECT_EX' members can be
     deleted.  (They are set to _NULL_).

 -- Function: PyObject* Py_FindMethod (PyMethodDef table[],
          PyObject *ob, char *name)
     Return value: New reference.

     Return a bound method object for an extension type implemented in
     C.  This can be useful in the implementation of a `tp_getattro' or
     `tp_getattr' handler that does not use the
     PyObject_GenericGetAttr() function.

7.10.3 Type Objects
-------------------

Perhaps one of the most important structures of the Python object
system is the structure that defines a new type: the
PyTypeObject structure.  Type objects can be handled using any
of the `PyObject_*()' or `PyType_*()' functions, but do not offer much
that's interesting to most Python applications. These objects are
fundamental to how objects behave, so they are very important to the
interpreter itself and to any extension module that implements new
types.

  Type objects are fairly large compared to most of the standard types.
The reason for the size is that each type object stores a large number
of values, mostly C function pointers, each of which implements a small
part of the type's functionality.  The fields of the type object are
examined in detail in this section.  The fields will be described in
the order in which they occur in the structure.

  Typedefs: unaryfunc, binaryfunc, ternaryfunc, inquiry, coercion,
intargfunc, intintargfunc, intobjargproc, intintobjargproc,
objobjargproc, destructor, freefunc, printfunc, getattrfunc,
getattrofunc, setattrfunc, setattrofunc, cmpfunc, reprfunc, hashfunc

  The structure definition for PyTypeObject can be found in
`Include/object.h'.  For convenience of reference, this repeats the
definition found there:

    typedef struct _typeobject {
        PyObject_VAR_HEAD
        char *tp_name; /* For printing, in format "<module>.<name>" */
        int tp_basicsize, tp_itemsize; /* For allocation */

        /* Methods to implement standard operations */

        destructor tp_dealloc;
        printfunc tp_print;
        getattrfunc tp_getattr;
        setattrfunc tp_setattr;
        cmpfunc tp_compare;
        reprfunc tp_repr;

        /* Method suites for standard classes */

        PyNumberMethods *tp_as_number;
        PySequenceMethods *tp_as_sequence;
        PyMappingMethods *tp_as_mapping;

        /* More standard operations (here for binary compatibility) */

        hashfunc tp_hash;
        ternaryfunc tp_call;
        reprfunc tp_str;
        getattrofunc tp_getattro;
        setattrofunc tp_setattro;

        /* Functions to access object as input/output buffer */
        PyBufferProcs *tp_as_buffer;

        /* Flags to define presence of optional/expanded features */
        long tp_flags;

        char *tp_doc; /* Documentation string */

        /* Assigned meaning in release 2.0 */
        /* call function for all accessible objects */
        traverseproc tp_traverse;

        /* delete references to contained objects */
        inquiry tp_clear;

        /* Assigned meaning in release 2.1 */
        /* rich comparisons */
        richcmpfunc tp_richcompare;

        /* weak reference enabler */
        long tp_weaklistoffset;

        /* Added in release 2.2 */
        /* Iterators */
        getiterfunc tp_iter;
        iternextfunc tp_iternext;

        /* Attribute descriptor and subclassing stuff */
        struct PyMethodDef *tp_methods;
        struct PyMemberDef *tp_members;
        struct PyGetSetDef *tp_getset;
        struct _typeobject *tp_base;
        PyObject *tp_dict;
        descrgetfunc tp_descr_get;
        descrsetfunc tp_descr_set;
        long tp_dictoffset;
        initproc tp_init;
        allocfunc tp_alloc;
        newfunc tp_new;
        freefunc tp_free; /* Low-level free-memory routine */
        inquiry tp_is_gc; /* For PyObject_IS_GC */
        PyObject *tp_bases;
        PyObject *tp_mro; /* method resolution order */
        PyObject *tp_cache;
        PyObject *tp_subclasses;
        PyObject *tp_weaklist;

    } PyTypeObject;

The type object structure extends the PyVarObject.
structure. The `ob_size' field is used for dynamic types (created by
`type_new()', usually called from a class statement). Note that
PyType_Type. (the metatype) initializes `tp_itemsize', which
means that its instances (i.e.  type objects) _must_ have the `ob_size'
field.

 -- member: PyObject* PyObject._ob_next
 -- member: PyObject* PyObject._ob_prev
     These fields are only present when the macro `Py_TRACE_REFS' is
     defined.  Their initialization to _NULL_ is taken care of by the
     `PyObject_HEAD_INIT' macro.  For statically allocated objects,
     these fields always remain _NULL_.  For dynamically allocated
     objects, these two fields are used to link the object into a
     doubly-linked list of _all_ live objects on the heap.  This could
     be used for various debugging purposes; currently the only use is
     to print the objects that are still alive at the end of a run when
     the environment variable PYTHONDUMPREFS is set.

     These fields are not inherited by subtypes.

 -- member: Py_ssize_t PyObject.ob_refcnt
     This is the type object's reference count, initialized to `1' by
     the `PyObject_HEAD_INIT' macro.  Note that for statically
     allocated type objects, the type's instances (objects whose
     `ob_type' points back to the type) do _not_ count as references.
     But for dynamically allocated type objects, the instances _do_
     count as references.

     This field is not inherited by subtypes.

     Changed in version 2.5: This field used to be an `int' type. This
     might require changes in your code for properly supporting 64-bit
     systems.

 -- member: PyTypeObject* PyObject.ob_type
     This is the type's type, in other words its metatype.  It is
     initialized by the argument to the `PyObject_HEAD_INIT' macro, and
     its value should normally be `&PyType_Type'.  However, for
     dynamically loadable extension modules that must be usable on
     Windows (at least), the compiler complains that this is not a valid
     initializer.  Therefore, the convention is to pass _NULL_ to the
     `PyObject_HEAD_INIT' macro and to initialize this field explicitly
     at the start of the module's initialization function, before doing
     anything else.  This is typically done like this:

         Foo_Type.ob_type = &PyType_Type;

     This should be done before any instances of the type are created.
     PyType_Ready() checks if `ob_type' is _NULL_, and if
     so, initializes it: in Python 2.2, it is set to `&PyType_Type'; in
     Python 2.2.1 and later it is initialized to the `ob_type' field of
     the base class.  PyType_Ready() will not change this
     field if it is non-zero.

     In Python 2.2, this field is not inherited by subtypes.  In 2.2.1,
     and in 2.3 and beyond, it is inherited by subtypes.

 -- member: Py_ssize_t PyVarObject.ob_size
     For statically allocated type objects, this should be initialized
     to zero.  For dynamically allocated type objects, this field has a
     special internal meaning.

     This field is not inherited by subtypes.

 -- member: char* PyTypeObject.tp_name
     Pointer to a NUL-terminated string containing the name of the
     type. For types that are accessible as module globals, the string
     should be the full module name, followed by a dot, followed by the
     type name; for built-in types, it should be just the type name.
     If the module is a submodule of a package, the full package name
     is part of the full module name.  For example, a type named `T'
     defined in module `M' in subpackage `Q' in package `P' should have
     the `tp_name' initializer `"P.Q.M.T"'.

     For dynamically allocated type objects, this should just be the
     type name, and the module name explicitly stored in the type dict
     as the value for key `'__module__''.

     For statically allocated type objects, the tp_name field should
     contain a dot.  Everything before the last dot is made accessible
     as the `__module__' attribute, and everything after the last dot
     is made accessible as the `__name__' attribute.

     If no dot is present, the entire `tp_name' field is made
     accessible as the `__name__' attribute, and the `__module__'
     attribute is undefined (unless explicitly set in the dictionary,
     as explained above).  This means your type will be impossible to
     pickle.

     This field is not inherited by subtypes.

 -- member: Py_ssize_t PyTypeObject.tp_basicsize
 -- member: Py_ssize_t PyTypeObject.tp_itemsize
     These fields allow calculating the size in bytes of instances of
     the type.

     There are two kinds of types: types with fixed-length instances
     have a zero `tp_itemsize' field, types with variable-length
     instances have a non-zero `tp_itemsize' field.  For a type with
     fixed-length instances, all instances have the same size, given in
     `tp_basicsize'.

     For a type with variable-length instances, the instances must have
     an `ob_size' field, and the instance size is `tp_basicsize' plus N
     times `tp_itemsize', where N is the "length" of the object.  The
     value of N is typically stored in the instance's `ob_size' field.
     There are exceptions:  for example, long ints use a negative
     `ob_size' to indicate a negative number, and N is `abs(ob_size)'
     there.  Also, the presence of an `ob_size' field in the instance
     layout doesn't mean that the instance structure is variable-length
     (for example, the structure for the list type has fixed-length
     instances, yet those instances have a meaningful `ob_size' field).

     The basic size includes the fields in the instance declared by the
     macro PyObject_HEAD or PyObject_VAR_HEAD.
     (whichever is used to declare the instance struct) and this in
     turn includes the `_ob_prev' and `_ob_next' fields if they are
     present.  This means that the only correct way to get an
     initializer for the `tp_basicsize' is to use the `sizeof' operator
     on the struct used to declare the instance layout.  The basic size
     does not include the GC header size (this is new in Python 2.2; in
     2.1 and 2.0, the GC header size was included in `tp_basicsize').

     These fields are inherited separately by subtypes.  If the base
     type has a non-zero `tp_itemsize', it is generally not safe to set
     `tp_itemsize' to a different non-zero value in a subtype (though
     this depends on the implementation of the base type).

     A note about alignment: if the variable items require a particular
     alignment, this should be taken care of by the value of
     `tp_basicsize'.  Example: suppose a type implements an array of
     `double'. `tp_itemsize' is `sizeof(double)'. It is the
     programmer's responsibility that `tp_basicsize' is a multiple of
     `sizeof(double)' (assuming this is the alignment requirement for
     `double').

 -- member: destructor PyTypeObject.tp_dealloc
     A pointer to the instance destructor function.  This function must
     be defined unless the type guarantees that its instances will
     never be deallocated (as is the case for the singletons `None' and
     `Ellipsis').

     The destructor function is called by the Py_DECREF().
     and Py_XDECREF() macros when the new reference count
     is zero.  At this point, the instance is still in existence, but
     there are no references to it.  The destructor function should
     free all references which the instance owns, free all memory
     buffers owned by the instance (using the freeing function
     corresponding to the allocation function used to allocate the
     buffer), and finally (as its last action) call the type's
     `tp_free' function.  If the type is not subtypable (doesn't have
     the Py_TPFLAGS_BASETYPE flag bit set), it is
     permissible to call the object deallocator directly instead of via
     `tp_free'.  The object deallocator should be the one used to
     allocate the instance; this is normally PyObject_Del().
     if the instance was allocated using PyObject_New() or
     `PyObject_VarNew()', or PyObject_GC_Del() if the
     instance was allocated using PyObject_GC_New() or
     PyObject_GC_NewVar().

     This field is inherited by subtypes.

 -- member: printfunc PyTypeObject.tp_print
     An optional pointer to the instance print function.

     The print function is only called when the instance is printed to
     a _real_ file; when it is printed to a pseudo-file (like a
     StringIO instance), the instance's `tp_repr' or `tp_str'
     function is called to convert it to a string.  These are also
     called when the type's `tp_print' field is _NULL_.  A type should
     never implement `tp_print' in a way that produces different output
     than `tp_repr' or `tp_str' would.

     The print function is called with the same signature as
     PyObject_Print().: `int tp_print(PyObject *self, FILE *file,
     int flags)'.  The _self_ argument is the instance to be printed.
     The _file_ argument is the stdio file to which it is to be
     printed.  The _flags_ argument is composed of flag bits. The only
     flag bit currently defined is `Py_PRINT_RAW'. When the
     `Py_PRINT_RAW' flag bit is set, the instance should be printed the
     same way as `tp_str' would format it; when the `Py_PRINT_RAW' flag
     bit is clear, the instance should be printed the same was as
     `tp_repr' would format it. It should return `-1' and set an
     exception condition when an error occurred during the comparison.

     It is possible that the `tp_print' field will be deprecated. In
     any case, it is recommended not to define `tp_print', but instead
     to rely on `tp_repr' and `tp_str' for printing.

     This field is inherited by subtypes.

 -- member: getattrfunc PyTypeObject.tp_getattr
     An optional pointer to the get-attribute-string function.

     This field is deprecated.  When it is defined, it should point to
     a function that acts the same as the `tp_getattro' function, but
     taking a C string instead of a Python string object to give the
     attribute name.  The signature is the same as for
     PyObject_GetAttrString().

     This field is inherited by subtypes together with `tp_getattro': a
     subtype inherits both `tp_getattr' and `tp_getattro' from its base
     type when the subtype's `tp_getattr' and `tp_getattro' are both
     _NULL_.

 -- member: setattrfunc PyTypeObject.tp_setattr
     An optional pointer to the set-attribute-string function.

     This field is deprecated.  When it is defined, it should point to
     a function that acts the same as the `tp_setattro' function, but
     taking a C string instead of a Python string object to give the
     attribute name.  The signature is the same as for
     PyObject_SetAttrString().

     This field is inherited by subtypes together with `tp_setattro': a
     subtype inherits both `tp_setattr' and `tp_setattro' from its base
     type when the subtype's `tp_setattr' and `tp_setattro' are both
     _NULL_.

 -- member: cmpfunc PyTypeObject.tp_compare
     An optional pointer to the three-way comparison function.

     The signature is the same as for PyObject_Compare().
     The function should return `1' if _self_ greater than _other_, `0'
     if _self_ is equal to _other_, and `-1' if _self_ less than
     _other_.  It should return `-1' and set an exception condition
     when an error occurred during the comparison.

     This field is inherited by subtypes together with `tp_richcompare'
     and `tp_hash': a subtypes inherits all three of `tp_compare',
     `tp_richcompare', and `tp_hash' when the subtype's `tp_compare',
     `tp_richcompare', and `tp_hash' are all _NULL_.

 -- member: reprfunc PyTypeObject.tp_repr
     An optional pointer to a function that implements the built-in
     function repr().

     The signature is the same as for PyObject_Repr().; it
     must return a string or a Unicode object.  Ideally, this function
     should return a string that, when passed to eval(),
     given a suitable environment, returns an object with the same
     value.  If this is not feasible, it should return a string
     starting with `'<'' and ending with `'>'' from which both the type
     and the value of the object can be deduced.

     When this field is not set, a string of the form `<%s object at
     %p>' is returned, where `%s' is replaced by the type name, and
     `%p' by the object's memory address.

     This field is inherited by subtypes.

 -- member: PyNumberMethods* tp_as_number
     Pointer to an additional structure that contains fields relevant
     only to objects which implement the number protocol.  These fields
     are documented in Number Object Structures.

     The `tp_as_number' field is not inherited, but the contained
     fields are inherited individually.

 -- member: PySequenceMethods* tp_as_sequence
     Pointer to an additional structure that contains fields relevant
     only to objects which implement the sequence protocol.  These
     fields are documented in Sequence Object Structures.

     The `tp_as_sequence' field is not inherited, but the contained
     fields are inherited individually.

 -- member: PyMappingMethods* tp_as_mapping
     Pointer to an additional structure that contains fields relevant
     only to objects which implement the mapping protocol.  These
     fields are documented in Mapping Object Structures.

     The `tp_as_mapping' field is not inherited, but the contained
     fields are inherited individually.

 -- member: hashfunc PyTypeObject.tp_hash
     An optional pointer to a function that implements the built-in
     function hash().

     The signature is the same as for PyObject_Hash().; it
     must return a C long.  The value `-1' should not be returned as a
     normal return value; when an error occurs during the computation
     of the hash value, the function should set an exception and return
     `-1'.

     This field can be set explicitly to
     PyObject_HashNotImplemented() to block inheritance of the
     hash method from a parent type. This is interpreted as the
     equivalent of `__hash__ = None' at the Python level, causing
     `isinstance(o, collections.Hashable)' to correctly return `False'.
     Note that the converse is also true - setting `__hash__ = None' on
     a class at the Python level will result in the `tp_hash' slot
     being set to PyObject_HashNotImplemented().

     When this field is not set, two possibilities exist: if the
     `tp_compare' and `tp_richcompare' fields are both _NULL_, a
     default hash value based on the object's address is returned;
     otherwise, a TypeError is raised.

     This field is inherited by subtypes together with `tp_richcompare'
     and `tp_compare': a subtypes inherits all three of `tp_compare',
     `tp_richcompare', and `tp_hash', when the subtype's `tp_compare',
     `tp_richcompare' and `tp_hash' are all _NULL_.

 -- member: ternaryfunc PyTypeObject.tp_call
     An optional pointer to a function that implements calling the
     object.  This should be _NULL_ if the object is not callable.  The
     signature is the same as for PyObject_Call().

     This field is inherited by subtypes.

 -- member: reprfunc PyTypeObject.tp_str
     An optional pointer to a function that implements the built-in
     operation str().  (Note that str is a type
     now, and str() calls the constructor for that type.
     This constructor calls PyObject_Str() to do the
     actual work, and PyObject_Str() will call this
     handler.)

     The signature is the same as for PyObject_Str().; it
     must return a string or a Unicode object.  This function should
     return a "friendly" string representation of the object, as this
     is the representation that will be used by the print statement.

     When this field is not set, see PyObject_Repr() is called
     to return a string representation.

     This field is inherited by subtypes.

 -- member: getattrofunc PyTypeObject.tp_getattro
     An optional pointer to the get-attribute function.

     The signature is the same as for PyObject_GetAttr().
     It is usually convenient to set this field to
     PyObject_GenericGetAttr(), which implements the normal way
     of looking for object attributes.

     This field is inherited by subtypes together with `tp_getattr': a
     subtype inherits both `tp_getattr' and `tp_getattro' from its base
     type when the subtype's `tp_getattr' and `tp_getattro' are both
     _NULL_.

 -- member: setattrofunc PyTypeObject.tp_setattro
     An optional pointer to the set-attribute function.

     The signature is the same as for PyObject_SetAttr().
     It is usually convenient to set this field to
     PyObject_GenericSetAttr(), which implements the normal way
     of setting object attributes.

     This field is inherited by subtypes together with `tp_setattr': a
     subtype inherits both `tp_setattr' and `tp_setattro' from its base
     type when the subtype's `tp_setattr' and `tp_setattro' are both
     _NULL_.

 -- member: PyBufferProcs* PyTypeObject.tp_as_buffer
     Pointer to an additional structure that contains fields relevant
     only to objects which implement the buffer interface.  These
     fields are documented in Buffer Object Structures.

     The `tp_as_buffer' field is not inherited, but the contained
     fields are inherited individually.

 -- member: long PyTypeObject.tp_flags
     This field is a bit mask of various flags.  Some flags indicate
     variant semantics for certain situations; others are used to
     indicate that certain fields in the type object (or in the
     extension structures referenced via `tp_as_number',
     `tp_as_sequence', `tp_as_mapping', and `tp_as_buffer') that were
     historically not always present are valid; if such a flag bit is
     clear, the type fields it guards must not be accessed and must be
     considered to have a zero or _NULL_ value instead.

     Inheritance of this field is complicated.  Most flag bits are
     inherited individually, i.e. if the base type has a flag bit set,
     the subtype inherits this flag bit.  The flag bits that pertain to
     extension structures are strictly inherited if the extension
     structure is inherited, i.e. the base type's value of the flag bit
     is copied into the subtype together with a pointer to the extension
     structure.  The Py_TPFLAGS_HAVE_GC flag bit is
     inherited together with the `tp_traverse' and `tp_clear' fields,
     i.e. if the Py_TPFLAGS_HAVE_GC flag bit is clear in
     the subtype and the `tp_traverse' and `tp_clear' fields in the
     subtype exist (as indicated by the
     Py_TPFLAGS_HAVE_RICHCOMPARE flag bit) and have _NULL_
     values.

     The following bit masks are currently defined; these can be ORed
     together using the `|' operator to form the value of the
     `tp_flags' field.  The macro PyType_HasFeature().
     takes a type and a flags value, _tp_ and _f_, and checks whether
     `tp->tp_flags & f' is non-zero.

      -- Data: Py_TPFLAGS_HAVE_GETCHARBUFFER
          If this bit is set, the PyBufferProcs struct
          referenced by `tp_as_buffer' has the `bf_getcharbuffer' field.

      -- Data: Py_TPFLAGS_HAVE_SEQUENCE_IN
          If this bit is set, the PySequenceMethods struct
          referenced by `tp_as_sequence' has the `sq_contains' field.

      -- Data: Py_TPFLAGS_GC
          This bit is obsolete.  The bit it used to name is no longer
          in use.  The symbol is now defined as zero.

      -- Data: Py_TPFLAGS_HAVE_INPLACEOPS
          If this bit is set, the PySequenceMethods struct
          referenced by `tp_as_sequence' and the PyNumberMethods
          structure referenced by `tp_as_number' contain the
          fields for in-place operators. In particular, this means that
          the PyNumberMethods structure has the fields
          `nb_inplace_add', `nb_inplace_subtract',
          `nb_inplace_multiply', `nb_inplace_divide',
          `nb_inplace_remainder', `nb_inplace_power',
          `nb_inplace_lshift', `nb_inplace_rshift', `nb_inplace_and',
          `nb_inplace_xor', and `nb_inplace_or'; and the
          PySequenceMethods struct has the fields
          `sq_inplace_concat' and `sq_inplace_repeat'.

      -- Data: Py_TPFLAGS_CHECKTYPES
          If this bit is set, the binary and ternary operations in the
          PyNumberMethods structure referenced by
          `tp_as_number' accept arguments of arbitrary object types,
          and do their own type conversions if needed.  If this bit is
          clear, those operations require that all arguments have the
          current type as their type, and the caller is supposed to
          perform a coercion operation first.  This applies to
          `nb_add', `nb_subtract', `nb_multiply', `nb_divide',
          `nb_remainder', `nb_divmod', `nb_power', `nb_lshift',
          `nb_rshift', `nb_and', `nb_xor', and `nb_or'.

      -- Data: Py_TPFLAGS_HAVE_RICHCOMPARE
          If this bit is set, the type object has the `tp_richcompare'
          field, as well as the `tp_traverse' and the `tp_clear' fields.

      -- Data: Py_TPFLAGS_HAVE_WEAKREFS
          If this bit is set, the `tp_weaklistoffset' field is defined.
          Instances of a type are weakly referenceable if the type's
          `tp_weaklistoffset' field has a value greater than zero.

      -- Data: Py_TPFLAGS_HAVE_ITER
          If this bit is set, the type object has the `tp_iter' and
          `tp_iternext' fields.

      -- Data: Py_TPFLAGS_HAVE_CLASS
          If this bit is set, the type object has several new fields
          defined starting in Python 2.2: `tp_methods', `tp_members',
          `tp_getset', `tp_base', `tp_dict', `tp_descr_get',
          `tp_descr_set', `tp_dictoffset', `tp_init', `tp_alloc',
          `tp_new', `tp_free', `tp_is_gc', `tp_bases', `tp_mro',
          `tp_cache', `tp_subclasses', and `tp_weaklist'.

      -- Data: Py_TPFLAGS_HEAPTYPE
          This bit is set when the type object itself is allocated on
          the heap.  In this case, the `ob_type' field of its instances
          is considered a reference to the type, and the type object is
          INCREF'ed when a new instance is created, and DECREF'ed when
          an instance is destroyed (this does not apply to instances of
          subtypes; only the type referenced by the instance's ob_type
          gets INCREF'ed or DECREF'ed).

      -- Data: Py_TPFLAGS_BASETYPE
          This bit is set when the type can be used as the base type of
          another type.  If this bit is clear, the type cannot be
          subtyped (similar to a "final" class in Java).

      -- Data: Py_TPFLAGS_READY
          This bit is set when the type object has been fully
          initialized by PyType_Ready().

      -- Data: Py_TPFLAGS_READYING
          This bit is set while PyType_Ready() is in the
          process of initializing the type object.

      -- Data: Py_TPFLAGS_HAVE_GC
          This bit is set when the object supports garbage collection.
          If this bit is set, instances must be created using
          PyObject_GC_New() and destroyed using
          PyObject_GC_Del().  More information in section
          Supporting Cyclic Garbage Collection.  This bit also
          implies that the GC-related fields `tp_traverse' and
          `tp_clear' are present in the type object; but those fields
          also exist when Py_TPFLAGS_HAVE_GC is clear but
          Py_TPFLAGS_HAVE_RICHCOMPARE is set.

      -- Data: Py_TPFLAGS_DEFAULT
          This is a bitmask of all the bits that pertain to the
          existence of certain fields in the type object and its
          extension structures. Currently, it includes the following
          bits: Py_TPFLAGS_HAVE_GETCHARBUFFER,
          Py_TPFLAGS_HAVE_SEQUENCE_IN,
          Py_TPFLAGS_HAVE_INPLACEOPS,
          Py_TPFLAGS_HAVE_RICHCOMPARE,
          Py_TPFLAGS_HAVE_WEAKREFS, Py_TPFLAGS_HAVE_ITER,
          and Py_TPFLAGS_HAVE_CLASS.

 -- member: char* PyTypeObject.tp_doc
     An optional pointer to a NUL-terminated C string giving the
     docstring for this type object.  This is exposed as the `__doc__'
     attribute on the type and instances of the type.

     This field is _not_ inherited by subtypes.

  The following three fields only exist if the
Py_TPFLAGS_HAVE_RICHCOMPARE flag bit is set.

 -- member: traverseproc PyTypeObject.tp_traverse
     An optional pointer to a traversal function for the garbage
     collector.  This is only used if the Py_TPFLAGS_HAVE_GC
     flag bit is set.  More information about Python's garbage
     collection scheme can be found in section Supporting Cyclic Garbage Collection.

     The `tp_traverse' pointer is used by the garbage collector to
     detect reference cycles. A typical implementation of a
     `tp_traverse' function simply calls Py_VISIT() on
     each of the instance's members that are Python objects.  For
     example, this is function `local_traverse()' from the
     thread extension module:

         static int
         local_traverse(localobject *self, visitproc visit, void *arg)
         {
             Py_VISIT(self->args);
             Py_VISIT(self->kw);
             Py_VISIT(self->dict);
             return 0;
         }

     Note that Py_VISIT() is called only on those members
     that can participate in reference cycles.  Although there is also
     a `self->key' member, it can only be _NULL_ or a Python string and
     therefore cannot be part of a reference cycle.

     On the other hand, even if you know a member can never be part of
     a cycle, as a debugging aid you may want to visit it anyway just
     so the gc module's `get_referents()' function will
     include it.

     Note that Py_VISIT() requires the _visit_ and _arg_
     parameters to `local_traverse()' to have these specific names;
     don't name them just anything.

     This field is inherited by subtypes together with `tp_clear' and
     the Py_TPFLAGS_HAVE_GC flag bit: the flag bit,
     `tp_traverse', and `tp_clear' are all inherited from the base type
     if they are all zero in the subtype _and_ the subtype has the
     Py_TPFLAGS_HAVE_RICHCOMPARE flag bit set.

 -- member: inquiry PyTypeObject.tp_clear
     An optional pointer to a clear function for the garbage collector.
     This is only used if the Py_TPFLAGS_HAVE_GC flag bit
     is set.

     The `tp_clear' member function is used to break reference cycles
     in cyclic garbage detected by the garbage collector.  Taken
     together, all `tp_clear' functions in the system must combine to
     break all reference cycles.  This is subtle, and if in any doubt
     supply a `tp_clear' function.  For example, the tuple type does
     not implement a `tp_clear' function, because it's possible to
     prove that no reference cycle can be composed entirely of tuples.
     Therefore the `tp_clear' functions of other types must be
     sufficient to break any cycle containing a tuple.  This isn't
     immediately obvious, and there's rarely a good reason to avoid
     implementing `tp_clear'.

     Implementations of `tp_clear' should drop the instance's
     references to those of its members that may be Python objects, and
     set its pointers to those members to _NULL_, as in the following
     example:

         static int
         local_clear(localobject *self)
         {
             Py_CLEAR(self->key);
             Py_CLEAR(self->args);
             Py_CLEAR(self->kw);
             Py_CLEAR(self->dict);
             return 0;
         }

     The Py_CLEAR() macro should be used, because clearing
     references is delicate:  the reference to the contained object
     must not be decremented until after the pointer to the contained
     object is set to _NULL_.  This is because decrementing the
     reference count may cause the contained object to become trash,
     triggering a chain of reclamation activity that may include
     invoking arbitrary Python code (due to finalizers, or weakref
     callbacks, associated with the contained object). If it's possible
     for such code to reference _self_ again, it's important that the
     pointer to the contained object be _NULL_ at that time, so that
     _self_ knows the contained object can no longer be used.  The
     Py_CLEAR() macro performs the operations in a safe
     order.

     Because the goal of `tp_clear' functions is to break reference
     cycles, it's not necessary to clear contained objects like Python
     strings or Python integers, which can't participate in reference
     cycles. On the other hand, it may be convenient to clear all
     contained Python objects, and write the type's `tp_dealloc'
     function to invoke `tp_clear'.

     More information about Python's garbage collection scheme can be
     found in section Supporting Cyclic Garbage Collection.

     This field is inherited by subtypes together with `tp_traverse'
     and the Py_TPFLAGS_HAVE_GC flag bit: the flag bit,
     `tp_traverse', and `tp_clear' are all inherited from the base type
     if they are all zero in the subtype _and_ the subtype has the
     Py_TPFLAGS_HAVE_RICHCOMPARE flag bit set.

 -- member: richcmpfunc PyTypeObject.tp_richcompare
     An optional pointer to the rich comparison function, whose
     signature is `PyObject *tp_richcompare(PyObject *a, PyObject *b,
     int op)'.

     The function should return the result of the comparison (usually
     `Py_True' or `Py_False').  If the comparison is undefined, it must
     return `Py_NotImplemented', if another error occurred it must
     return `NULL' and set an exception condition.

          Note: If you want to implement a type for which only a
          limited set of comparisons makes sense (e.g. `==' and `!=',
          but not `<' and friends), directly raise TypeError
          in the rich comparison function.

     This field is inherited by subtypes together with `tp_compare' and
     `tp_hash': a subtype inherits all three of `tp_compare',
     `tp_richcompare', and `tp_hash', when the subtype's `tp_compare',
     `tp_richcompare', and `tp_hash' are all _NULL_.

     The following constants are defined to be used as the third
     argument for `tp_richcompare' and for
     PyObject_RichCompare().:

     Constant             Comparison
     --------------------------------------
     `Py_LT'              `<'
     `Py_LE'              `<='
     `Py_EQ'              `=='
     `Py_NE'              `!='
     `Py_GT'              `>'
     `Py_GE'              `>='

  The next field only exists if the Py_TPFLAGS_HAVE_WEAKREFS
flag bit is set.

 -- member: long PyTypeObject.tp_weaklistoffset
     If the instances of this type are weakly referenceable, this field
     is greater than zero and contains the offset in the instance
     structure of the weak reference list head (ignoring the GC header,
     if present); this offset is used by `PyObject_ClearWeakRefs()' and
     the `PyWeakref_*()' functions.  The instance structure needs to
     include a field of type PyObject* which is initialized
     to _NULL_.

     Do not confuse this field with `tp_weaklist'; that is the list
     head for weak references to the type object itself.

     This field is inherited by subtypes, but see the rules listed
     below. A subtype may override this offset; this means that the
     subtype uses a different weak reference list head than the base
     type.  Since the list head is always found via
     `tp_weaklistoffset', this should not be a problem.

     When a type defined by a class statement has no __slots__
     declaration, and none of its base types are weakly
     referenceable, the type is made weakly referenceable by adding a
     weak reference list head slot to the instance layout and setting
     the `tp_weaklistoffset' of that slot's offset.

     When a type's __slots__ declaration contains a slot
     named `__weakref__', that slot becomes the weak reference list
     head for instances of the type, and the slot's offset is stored in
     the type's `tp_weaklistoffset'.

     When a type's __slots__ declaration does not contain a
     slot named `__weakref__', the type inherits its
     `tp_weaklistoffset' from its base type.

  The next two fields only exist if the Py_TPFLAGS_HAVE_ITER
flag bit is set.

 -- member: getiterfunc PyTypeObject.tp_iter
     An optional pointer to a function that returns an iterator for the
     object.  Its presence normally signals that the instances of this
     type are iterable (although sequences may be iterable without this
     function, and classic instances always have this function, even if
     they don't define an __iter__() method).

     This function has the same signature as PyObject_GetIter().

     This field is inherited by subtypes.

 -- member: iternextfunc PyTypeObject.tp_iternext
     An optional pointer to a function that returns the next item in an
     iterator.  When the iterator is exhausted, it must return _NULL_;
     a StopIteration.  exception may or may not be set.
     When another error occurs, it must return _NULL_ too.  Its
     presence normally signals that the instances of this type are
     iterators (although classic instances always have this function,
     even if they don't define a next() method).

     Iterator types should also define the `tp_iter' function, and that
     function should return the iterator instance itself (not a new
     iterator instance).

     This function has the same signature as PyIter_Next().

     This field is inherited by subtypes.

  The next fields, up to and including `tp_weaklist', only exist if the
Py_TPFLAGS_HAVE_CLASS flag bit is set.

 -- member: struct PyMethodDef* PyTypeObject.tp_methods
     An optional pointer to a static _NULL_-terminated array of
     PyMethodDef.  structures, declaring regular methods of this
     type.

     For each entry in the array, an entry is added to the type's
     dictionary (see `tp_dict' below) containing a method descriptor.

     This field is not inherited by subtypes (methods are inherited
     through a different mechanism).

 -- member: struct PyMemberDef* PyTypeObject.tp_members
     An optional pointer to a static _NULL_-terminated array of
     PyMemberDef.  structures, declaring regular data members
     (fields or slots) of instances of this type.

     For each entry in the array, an entry is added to the type's
     dictionary (see `tp_dict' below) containing a member descriptor.

     This field is not inherited by subtypes (members are inherited
     through a different mechanism).

 -- member: struct PyGetSetDef* PyTypeObject.tp_getset
     An optional pointer to a static _NULL_-terminated array of
     `PyGetSetDef' structures, declaring computed attributes of
     instances of this type.

     For each entry in the array, an entry is added to the type's
     dictionary (see `tp_dict' below) containing a getset descriptor.

     This field is not inherited by subtypes (computed attributes are
     inherited through a different mechanism).

     Docs for PyGetSetDef (XXX belong elsewhere):

         typedef PyObject *(*getter)(PyObject *, void *);
         typedef int (*setter)(PyObject *, PyObject *, void *);

         typedef struct PyGetSetDef {
             char *name;    /* attribute name */
             getter get;    /* C function to get the attribute */
             setter set;    /* C function to set the attribute */
             char *doc;     /* optional doc string */
             void *closure; /* optional additional data for getter and setter */
         } PyGetSetDef;

 -- member: PyTypeObject* PyTypeObject.tp_base
     An optional pointer to a base type from which type properties are
     inherited.  At this level, only single inheritance is supported;
     multiple inheritance require dynamically creating a type object by
     calling the metatype.

     This field is not inherited by subtypes (obviously), but it
     defaults to `&PyBaseObject_Type' (which to Python programmers is
     known as the type object).

 -- member: PyObject* PyTypeObject.tp_dict
     The type's dictionary is stored here by PyType_Ready().

     This field should normally be initialized to _NULL_ before
     PyType_Ready is called; it may also be initialized to a dictionary
     containing initial attributes for the type.  Once
     PyType_Ready() has initialized the type, extra attributes
     for the type may be added to this dictionary only if they don't
     correspond to overloaded operations (like __add__()).

     This field is not inherited by subtypes (though the attributes
     defined in here are inherited through a different mechanism).

 -- member: descrgetfunc PyTypeObject.tp_descr_get
     An optional pointer to a "descriptor get" function.

     The function signature is

         PyObject * tp_descr_get(PyObject *self, PyObject *obj, PyObject *type);

     XXX explain.

     This field is inherited by subtypes.

 -- member: descrsetfunc PyTypeObject.tp_descr_set
     An optional pointer to a "descriptor set" function.

     The function signature is

         int tp_descr_set(PyObject *self, PyObject *obj, PyObject *value);

     This field is inherited by subtypes.

     XXX explain.

 -- member: long PyTypeObject.tp_dictoffset
     If the instances of this type have a dictionary containing
     instance variables, this field is non-zero and contains the offset
     in the instances of the type of the instance variable dictionary;
     this offset is used by PyObject_GenericGetAttr().

     Do not confuse this field with `tp_dict'; that is the dictionary
     for attributes of the type object itself.

     If the value of this field is greater than zero, it specifies the
     offset from the start of the instance structure.  If the value is
     less than zero, it specifies the offset from the _end_ of the
     instance structure.  A negative offset is more expensive to use,
     and should only be used when the instance structure contains a
     variable-length part.  This is used for example to add an instance
     variable dictionary to subtypes of str or tuple.
     Note that the `tp_basicsize' field should account for the
     dictionary added to the end in that case, even though the
     dictionary is not included in the basic object layout.  On a
     system with a pointer size of 4 bytes, `tp_dictoffset' should be
     set to `-4' to indicate that the dictionary is at the very end of
     the structure.

     The real dictionary offset in an instance can be computed from a
     negative `tp_dictoffset' as follows:

         dictoffset = tp_basicsize + abs(ob_size)*tp_itemsize + tp_dictoffset
         if dictoffset is not aligned on sizeof(void*):
             round up to sizeof(void*)

     where `tp_basicsize', `tp_itemsize' and `tp_dictoffset' are taken
     from the type object, and `ob_size' is taken from the instance.
     The absolute value is taken because long ints use the sign of
     `ob_size' to store the sign of the number.  (There's never a need
     to do this calculation yourself; it is done for you by
     `_PyObject_GetDictPtr()'.)

     This field is inherited by subtypes, but see the rules listed
     below. A subtype may override this offset; this means that the
     subtype instances store the dictionary at a difference offset than
     the base type.  Since the dictionary is always found via
     `tp_dictoffset', this should not be a problem.

     When a type defined by a class statement has no __slots__
     declaration, and none of its base types has an instance
     variable dictionary, a dictionary slot is added to the instance
     layout and the `tp_dictoffset' is set to that slot's offset.

     When a type defined by a class statement has a __slots__
     declaration, the type inherits its `tp_dictoffset' from its
     base type.

     (Adding a slot named `__dict__' to the __slots__.
     declaration does not have the expected effect, it just causes
     confusion.  Maybe this should be added as a feature just like
     `__weakref__' though.)

 -- member: initproc PyTypeObject.tp_init
     An optional pointer to an instance initialization function.

     This function corresponds to the __init__() method of
     classes.  Like __init__(), it is possible to create an
     instance without calling __init__(), and it is possible
     to reinitialize an instance by calling its __init__().
     method again.

     The function signature is

         int tp_init(PyObject *self, PyObject *args, PyObject *kwds)

     The self argument is the instance to be initialized; the _args_
     and _kwds_ arguments represent positional and keyword arguments of
     the call to __init__().

     The `tp_init' function, if not _NULL_, is called when an instance
     is created normally by calling its type, after the type's `tp_new'
     function has returned an instance of the type.  If the `tp_new'
     function returns an instance of some other type that is not a
     subtype of the original type, no `tp_init' function is called; if
     `tp_new' returns an instance of a subtype of the original type,
     the subtype's `tp_init' is called.  (VERSION NOTE: described here
     is what is implemented in Python 2.2.1 and later.  In Python 2.2,
     the `tp_init' of the type of the object returned by `tp_new' was
     always called, if not _NULL_.)

     This field is inherited by subtypes.

 -- member: allocfunc PyTypeObject.tp_alloc
     An optional pointer to an instance allocation function.

     The function signature is

         PyObject *tp_alloc(PyTypeObject *self, Py_ssize_t nitems)

     The purpose of this function is to separate memory allocation from
     memory initialization.  It should return a pointer to a block of
     memory of adequate length for the instance, suitably aligned, and
     initialized to zeros, but with `ob_refcnt' set to `1' and
     `ob_type' set to the type argument.  If the type's `tp_itemsize'
     is non-zero, the object's `ob_size' field should be initialized to
     _nitems_ and the length of the allocated memory block should be
     `tp_basicsize + nitems*tp_itemsize', rounded up to a multiple of
     `sizeof(void*)'; otherwise, _nitems_ is not used and the length of
     the block should be `tp_basicsize'.

     Do not use this function to do any other instance initialization,
     not even to allocate additional memory; that should be done by
     `tp_new'.

     This field is inherited by static subtypes, but not by dynamic
     subtypes (subtypes created by a class statement); in the latter,
     this field is always set to PyType_GenericAlloc(), to
     force a standard heap allocation strategy.  That is also the
     recommended value for statically defined types.

 -- member: newfunc PyTypeObject.tp_new
     An optional pointer to an instance creation function.

     If this function is _NULL_ for a particular type, that type cannot
     be called to create new instances; presumably there is some other
     way to create instances, like a factory function.

     The function signature is

         PyObject *tp_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds)

     The subtype argument is the type of the object being created; the
     _args_ and _kwds_ arguments represent positional and keyword
     arguments of the call to the type.  Note that subtype doesn't have
     to equal the type whose `tp_new' function is called; it may be a
     subtype of that type (but not an unrelated type).

     The `tp_new' function should call `subtype->tp_alloc(subtype,
     nitems)' to allocate space for the object, and then do only as
     much further initialization as is absolutely necessary.
     Initialization that can safely be ignored or repeated should be
     placed in the `tp_init' handler.  A good rule of thumb is that for
     immutable types, all initialization should take place in `tp_new',
     while for mutable types, most initialization should be deferred to
     `tp_init'.

     This field is inherited by subtypes, except it is not inherited by
     static types whose `tp_base' is _NULL_ or `&PyBaseObject_Type'.
     The latter exception is a precaution so that old extension types
     don't become callable simply by being linked with Python 2.2.

 -- member: destructor PyTypeObject.tp_free
     An optional pointer to an instance deallocation function.

     The signature of this function has changed slightly: in Python 2.2
     and 2.2.1, its signature is `destructor':

         void tp_free(PyObject *)

     In Python 2.3 and beyond, its signature is `freefunc':

         void tp_free(void *)

     The only initializer that is compatible with both versions is
     `_PyObject_Del', whose definition has suitably adapted in Python
     2.3.

     This field is inherited by static subtypes, but not by dynamic
     subtypes (subtypes created by a class statement); in the latter,
     this field is set to a deallocator suitable to match
     PyType_GenericAlloc() and the value of the
     Py_TPFLAGS_HAVE_GC flag bit.

 -- member: inquiry PyTypeObject.tp_is_gc
     An optional pointer to a function called by the garbage collector.

     The garbage collector needs to know whether a particular object is
     collectible or not.  Normally, it is sufficient to look at the
     object's type's `tp_flags' field, and check the
     Py_TPFLAGS_HAVE_GC flag bit.  But some types have a mixture
     of statically and dynamically allocated instances, and the
     statically allocated instances are not collectible.  Such types
     should define this function; it should return `1' for a
     collectible instance, and `0' for a non-collectible instance. The
     signature is

         int tp_is_gc(PyObject *self)

     (The only example of this are types themselves.  The metatype,
     see PyType_Type, defines this function to distinguish
     between statically and dynamically allocated types.)

     This field is inherited by subtypes.  (VERSION NOTE: in Python
     2.2, it was not inherited.  It is inherited in 2.2.1 and later
     versions.)

 -- member: PyObject* PyTypeObject.tp_bases
     Tuple of base types.

     This is set for types created by a class statement.  It should be
     _NULL_ for statically defined types.

     This field is not inherited.

 -- member: PyObject* PyTypeObject.tp_mro
     Tuple containing the expanded set of base types, starting with the
     type itself and ending with object, in Method
     Resolution Order.

     This field is not inherited; it is calculated fresh by
     PyType_Ready().

 -- member: PyObject* PyTypeObject.tp_cache
     Unused.  Not inherited.  Internal use only.

 -- member: PyObject* PyTypeObject.tp_subclasses
     List of weak references to subclasses.  Not inherited.  Internal
     use only.

 -- member: PyObject* PyTypeObject.tp_weaklist
     Weak reference list head, for weak references to this type object.
     Not inherited.  Internal use only.

  The remaining fields are only defined if the feature test macro
`COUNT_ALLOCS' is defined, and are for internal use only. They are
documented here for completeness.  None of these fields are inherited by
subtypes.

 -- member: Py_ssize_t PyTypeObject.tp_allocs
     Number of allocations.

 -- member: Py_ssize_t PyTypeObject.tp_frees
     Number of frees.

 -- member: Py_ssize_t PyTypeObject.tp_maxalloc
     Maximum simultaneously allocated objects.

 -- member: PyTypeObject* PyTypeObject.tp_next
     Pointer to the next type object with a non-zero `tp_allocs' field.

  Also, note that, in a garbage collected Python, tp_dealloc may be
called from any Python thread, not just the thread which created the
object (if the object becomes part of a refcount cycle, that cycle
might be collected by a garbage collection on any thread).  This is not
a problem for Python API calls, since the thread on which tp_dealloc is
called will own the Global Interpreter Lock (GIL). However, if the
object being destroyed in turn destroys objects from some other C or
C++ library, care should be taken to ensure that destroying those
objects on the thread which called tp_dealloc will not violate any
assumptions of the library.

7.10.4 Number Object Structures
-------------------------------

 -- type: PyNumberMethods
     This structure holds pointers to the functions which an object
     uses to implement the number protocol.  Almost every function
     below is used by the function of similar name documented in the
     Number Protocol section.

     Here is the structure definition:

         typedef struct {
              binaryfunc nb_add;
              binaryfunc nb_subtract;
              binaryfunc nb_multiply;
              binaryfunc nb_divide;
              binaryfunc nb_remainder;
              binaryfunc nb_divmod;
              ternaryfunc nb_power;
              unaryfunc nb_negative;
              unaryfunc nb_positive;
              unaryfunc nb_absolute;
              inquiry nb_nonzero;       /* Used by PyObject_IsTrue */
              unaryfunc nb_invert;
              binaryfunc nb_lshift;
              binaryfunc nb_rshift;
              binaryfunc nb_and;
              binaryfunc nb_xor;
              binaryfunc nb_or;
              coercion nb_coerce;       /* Used by the coerce() function */
              unaryfunc nb_int;
              unaryfunc nb_long;
              unaryfunc nb_float;
              unaryfunc nb_oct;
              unaryfunc nb_hex;

              /* Added in release 2.0 */
              binaryfunc nb_inplace_add;
              binaryfunc nb_inplace_subtract;
              binaryfunc nb_inplace_multiply;
              binaryfunc nb_inplace_divide;
              binaryfunc nb_inplace_remainder;
              ternaryfunc nb_inplace_power;
              binaryfunc nb_inplace_lshift;
              binaryfunc nb_inplace_rshift;
              binaryfunc nb_inplace_and;
              binaryfunc nb_inplace_xor;
              binaryfunc nb_inplace_or;

              /* Added in release 2.2 */
              binaryfunc nb_floor_divide;
              binaryfunc nb_true_divide;
              binaryfunc nb_inplace_floor_divide;
              binaryfunc nb_inplace_true_divide;

              /* Added in release 2.5 */
              unaryfunc nb_index;
         } PyNumberMethods;

  Binary and ternary functions may receive different kinds of
arguments, depending on the flag bit Py_TPFLAGS_CHECKTYPES.:

   - If Py_TPFLAGS_CHECKTYPES is not set, the function
     arguments are guaranteed to be of the object's type; the caller is
     responsible for calling the coercion method specified by the
     `nb_coerce' member to convert the arguments:

      -- member: coercion PyNumberMethods.nb_coerce
          This function is used by PyNumber_CoerceEx() and
          has the same signature.  The first argument is always a
          pointer to an object of the defined type.  If the conversion
          to a common "larger" type is possible, the function replaces
          the pointers with new references to the converted objects and
          returns `0'.  If the conversion is not possible, the function
          returns `1'.  If an error condition is set, it will return
          `-1'.

   - If the Py_TPFLAGS_CHECKTYPES flag is set, binary and
     ternary functions must check the type of all their operands, and
     implement the necessary conversions (at least one of the operands
     is an instance of the defined type).  This is the recommended way;
     with Python 3.0 coercion will disappear completely.

  If the operation is not defined for the given operands, binary and
ternary functions must return `Py_NotImplemented', if another error
occurred they must return `NULL' and set an exception.

7.10.5 Mapping Object Structures
--------------------------------

 -- type: PyMappingMethods
     This structure holds pointers to the functions which an object
     uses to implement the mapping protocol.  It has three members:

 -- member: lenfunc PyMappingMethods.mp_length
     This function is used by PyMapping_Length() and
     PyObject_Size(), and has the same signature.  This slot may
     be set to _NULL_ if the object has no defined length.

 -- member: binaryfunc PyMappingMethods.mp_subscript
     This function is used by PyObject_GetItem() and has
     the same signature.  This slot must be filled for the
     PyMapping_Check().  function to return `1', it can be _NULL_
     otherwise.

 -- member: objobjargproc PyMappingMethods.mp_ass_subscript
     This function is used by PyObject_SetItem() and has
     the same signature.  If this slot is _NULL_, the object does not
     support item assignment.

7.10.6 Sequence Object Structures
---------------------------------

 -- type: PySequenceMethods
     This structure holds pointers to the functions which an object
     uses to implement the sequence protocol.

 -- member: lenfunc PySequenceMethods.sq_length
     This function is used by PySequence_Size() and
     PyObject_Size(), and has the same signature.

 -- member: binaryfunc PySequenceMethods.sq_concat
     This function is used by PySequence_Concat() and has
     the same signature.  It is also used by the `+' operator, after
     trying the numeric addition via the `tp_as_number.nb_add' slot.

 -- member: ssizeargfunc PySequenceMethods.sq_repeat
     This function is used by PySequence_Repeat() and has
     the same signature.  It is also used by the `*' operator, after
     trying numeric multiplication via the `tp_as_number.nb_mul' slot.

 -- member: ssizeargfunc PySequenceMethods.sq_item
     This function is used by PySequence_GetItem() and has
     the same signature.  This slot must be filled for the
     PySequence_Check().  function to return `1', it can be
     _NULL_ otherwise.

     Negative indexes are handled as follows: if the `sq_length' slot is
     filled, it is called and the sequence length is used to compute a
     positive index which is passed to `sq_item'.  If `sq_length' is
     _NULL_, the index is passed as is to the function.

 -- member: ssizeobjargproc PySequenceMethods.sq_ass_item
     This function is used by PySequence_SetItem() and has
     the same signature.  This slot may be left to _NULL_ if the object
     does not support item assignment.

 -- member: objobjproc PySequenceMethods.sq_contains
     This function may be used by PySequence_Contains().
     and has the same signature.  This slot may be left to _NULL_, in
     this case PySequence_Contains() simply traverses the
     sequence until it finds a match.

 -- member: binaryfunc PySequenceMethods.sq_inplace_concat
     This function is used by PySequence_InPlaceConcat().
     and has the same signature.  It should modify its first operand,
     and return it.

 -- member: ssizeargfunc PySequenceMethods.sq_inplace_repeat
     This function is used by PySequence_InPlaceRepeat().
     and has the same signature.  It should modify its first operand,
     and return it.

7.10.7 Buffer Object Structures
-------------------------------

The buffer interface exports a model where an object can expose its
internal data as a set of chunks of data, where each chunk is specified
as a pointer/length pair.  These chunks are called _segments_ and are
presumed to be non-contiguous in memory.

  If an object does not export the buffer interface, then its
`tp_as_buffer' member in the PyTypeObject structure should
be _NULL_.  Otherwise, the `tp_as_buffer' will point to a
PyBufferProcs structure.

     Note: It is very important that your PyTypeObject.
     structure uses Py_TPFLAGS_DEFAULT for the value of
     the `tp_flags' member rather than `0'.  This tells the Python
     runtime that your PyBufferProcs.  structure contains
     the `bf_getcharbuffer' slot. Older versions of Python did not have
     this member, so a new Python interpreter using an old extension
     needs to be able to test for its presence before using it.

 -- type: PyBufferProcs
     Structure used to hold the function pointers which define an
     implementation of the buffer protocol.

     The first slot is `bf_getreadbuffer', of type `getreadbufferproc'.
     If this slot is _NULL_, then the object does not support reading
     from the internal data.  This is non-sensical, so implementors
     should fill this in, but callers should test that the slot
     contains a non-_NULL_ value.

     The next slot is `bf_getwritebuffer' having type
     `getwritebufferproc'.  This slot may be _NULL_ if the object does
     not allow writing into its returned buffers.

     The third slot is `bf_getsegcount', with type `getsegcountproc'.
     This slot must not be _NULL_ and is used to inform the caller how
     many segments the object contains.  Simple objects such as
     PyString_Type and PyBuffer_Type objects
     contain a single segment.

     The last slot is `bf_getcharbuffer', of type `getcharbufferproc'.
     This slot will only be present if the
     Py_TPFLAGS_HAVE_GETCHARBUFFER.  flag is present in the
     `tp_flags' field of the object's PyTypeObject. Before
     using this slot, the caller should test whether it is present by
     using the PyType_HasFeature() function.  If the flag
     is present, `bf_getcharbuffer' may be _NULL_, indicating that the
     object's contents cannot be used as _8-bit characters_. The slot
     function may also raise an error if the object's contents cannot
     be interpreted as 8-bit characters.  For example, if the object is
     an array which is configured to hold floating point values, an
     exception may be raised if a caller attempts to use
     `bf_getcharbuffer' to fetch a sequence of 8-bit characters. This
     notion of exporting the internal buffers as "text" is used to
     distinguish between objects that are binary in nature, and those
     which have character-based content.

          Note: The current policy seems to state that these characters
          may be multi-byte characters. This implies that a buffer size
          of _N_ does not mean there are _N_ characters present.

 -- Data: Py_TPFLAGS_HAVE_GETCHARBUFFER
     Flag bit set in the type structure to indicate that the
     `bf_getcharbuffer' slot is known.  This being set does not
     indicate that the object supports the buffer interface or that the
     `bf_getcharbuffer' slot is non-_NULL_.

 -- type: Py_ssize_t (*readbufferproc) (PyObject *self,
          Py_ssize_t segment, void **ptrptr)
     Return a pointer to a readable segment of the buffer in `*ptrptr'.
     This function is allowed to raise an exception, in which case it
     must return `-1'.  The _segment_ which is specified must be zero
     or positive, and strictly less than the number of segments
     returned by the `bf_getsegcount' slot function.  On success, it
     returns the length of the segment, and sets `*ptrptr' to a pointer
     to that memory.

 -- type: Py_ssize_t (*writebufferproc) (PyObject *self,
          Py_ssize_t segment, void **ptrptr)
     Return a pointer to a writable memory buffer in `*ptrptr', and the
     length of that segment as the function return value.  The memory
     buffer must correspond to buffer segment _segment_.  Must return
     `-1' and set an exception on error.  TypeError should
     be raised if the object only supports read-only buffers, and
     SystemError should be raised when _segment_ specifies a
     segment that doesn't exist.

 -- type: Py_ssize_t (*segcountproc) (PyObject *self, Py_ssize_t *lenp)
     Return the number of memory segments which comprise the buffer.
     If _lenp_ is not _NULL_, the implementation must report the sum of
     the sizes (in bytes) of all segments in `*lenp'. The function
     cannot fail.

 -- type: Py_ssize_t (*charbufferproc) (PyObject *self,
          Py_ssize_t segment, const char **ptrptr)
     Return the size of the segment _segment_ that _ptrptr_  is set to.
     `*ptrptr' is set to the memory buffer. Returns `-1' on error.

7.10.8 Supporting Cyclic Garbage Collection
-------------------------------------------

Python's support for detecting and collecting garbage which involves
circular references requires support from object types which are
"containers" for other objects which may also be containers.  Types
which do not store references to other objects, or which only store
references to atomic types (such as numbers or strings), do not need to
provide any explicit support for garbage collection.

  To create a container type, the `tp_flags' field of the type object
must include the Py_TPFLAGS_HAVE_GC and provide an
implementation of the `tp_traverse' handler.  If instances of the type
are mutable, a `tp_clear' implementation must also be provided.

 -- Data: Py_TPFLAGS_HAVE_GC
     Objects with a type with this flag set must conform with the rules
     documented here.  For convenience these objects will be referred
     to as container objects.

  Constructors for container types must conform to two rules:

  1. The memory for the object must be allocated using
     PyObject_GC_New().  or PyObject_GC_NewVar().

  2. Once all the fields which may contain references to other
     containers are initialized, it must call
     PyObject_GC_Track().

 -- Function: TYPE* PyObject_GC_New (TYPE, PyTypeObject *type)
     Analogous to PyObject_New() but for container objects
     with the Py_TPFLAGS_HAVE_GC flag set.

 -- Function: TYPE* PyObject_GC_NewVar (TYPE, PyTypeObject *type,
          Py_ssize_t size)
     Analogous to PyObject_NewVar() but for container
     objects with the Py_TPFLAGS_HAVE_GC flag set.

     Changed in version 2.5: This function used an `int' type for
     _size_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: TYPE* PyObject_GC_Resize (TYPE, PyVarObject *op,
          Py_ssize_t newsize)
     Resize an object allocated by PyObject_NewVar().
     Returns the resized object or _NULL_ on failure.

     Changed in version 2.5: This function used an `int' type for
     _newsize_. This might require changes in your code for properly
     supporting 64-bit systems.

 -- Function: void PyObject_GC_Track (PyObject *op)
     Adds the object _op_ to the set of container objects tracked by the
     collector.  The collector can run at unexpected times so objects
     must be valid while being tracked.  This should be called once all
     the fields followed by the `tp_traverse' handler become valid,
     usually near the end of the constructor.

 -- Function: void _PyObject_GC_TRACK (PyObject *op)
     A macro version of PyObject_GC_Track().  It should not
     be used for extension modules.

  Similarly, the deallocator for the object must conform to a similar
pair of rules:

  1. Before fields which refer to other containers are invalidated,
     see PyObject_GC_UnTrack() must be called.

  2. The object's memory must be deallocated using
     PyObject_GC_Del().

 -- Function: void PyObject_GC_Del (void *op)
     Releases memory allocated to an object using
     PyObject_GC_New() or PyObject_GC_NewVar().

 -- Function: void PyObject_GC_UnTrack (void *op)
     Remove the object _op_ from the set of container objects tracked
     by the collector.  Note that PyObject_GC_Track() can
     be called again on this object to add it back to the set of
     tracked objects.  The deallocator (`tp_dealloc' handler) should
     call this for the object before any of the fields used by the
     `tp_traverse' handler become invalid.

 -- Function: void _PyObject_GC_UNTRACK (PyObject *op)
     A macro version of PyObject_GC_UnTrack().  It should
     not be used for extension modules.

  The `tp_traverse' handler accepts a function parameter of this type:

 -- type: int (*visitproc) (PyObject *object, void *arg)
     Type of the visitor function passed to the `tp_traverse' handler.
     The function should be called with an object to traverse as
     _object_ and the third parameter to the `tp_traverse' handler as
     _arg_.  The Python core uses several visitor functions to
     implement cyclic garbage detection; it's not expected that users
     will need to write their own visitor functions.

  The `tp_traverse' handler must have the following type:

 -- type: int (*traverseproc) (PyObject *self, visitproc visit,
          void *arg)
     Traversal function for a container object.  Implementations must
     call the _visit_ function for each object directly contained by
     _self_, with the parameters to _visit_ being the contained object
     and the _arg_ value passed to the handler.  The _visit_ function
     must not be called with a _NULL_ object argument.  If _visit_
     returns a non-zero value that value should be returned immediately.

  To simplify writing `tp_traverse' handlers, a Py_VISIT().
macro is provided.  In order to use this macro, the `tp_traverse'
implementation must name its arguments exactly _visit_ and _arg_:

 -- Function: void Py_VISIT (PyObject *o)
     Call the _visit_ callback, with arguments _o_ and _arg_. If
     _visit_ returns a non-zero value, then return it.  Using this
     macro, `tp_traverse' handlers look like:

         static int
         my_traverse(Noddy *self, visitproc visit, void *arg)
         {
             Py_VISIT(self->foo);
             Py_VISIT(self->bar);
             return 0;
         }

     New in version 2.4.

  The `tp_clear' handler must be of the inquiry type, or
_NULL_ if the object is immutable.

 -- type: int (*inquiry) (PyObject *self)
     Drop references that may have created reference cycles.  Immutable
     objects do not have to define this method since they can never
     directly create reference cycles.  Note that the object must still
     be valid after calling this method (don't just call
     Py_DECREF() on a reference).  The collector will call this
     method if it detects that this object is involved in a reference
     cycle.

8 Distributing Python Modules
*****************************

     Authors: Greg Ward, Anthony Baxter

     Email: <distutils-sig@python.org>

     Release: 2.7

     Date: February 08, 2011

  This document describes the Python Distribution Utilities
("Distutils") from the module developer's point of view, describing how
to use the Distutils to make Python modules and extensions easily
available to a wider audience with very little overhead for
build/release/install mechanics.

8.1 An Introduction to Distutils
================================

This document covers using the Distutils to distribute your Python
modules, concentrating on the role of developer/distributor: if you're
looking for information on installing Python modules, you should refer
to the Installing Python Modules chapter.

8.1.1 Concepts & Terminology
----------------------------

Using the Distutils is quite simple, both for module developers and for
users/administrators installing third-party modules.  As a developer,
your responsibilities (apart from writing solid, well-documented and
well-tested code, of course!) are:

   * write a setup script (`setup.py' by convention)

   * (optional) write a setup configuration file

   * create a source distribution

   * (optional) create one or more built (binary) distributions

  Each of these tasks is covered in this document.

  Not all module developers have access to a multitude of platforms, so
it's not always feasible to expect them to create a multitude of built
distributions.  It is hoped that a class of intermediaries, called
_packagers_, will arise to address this need.  Packagers will take
source distributions released by module developers, build them on one
or more platforms, and release the resulting built distributions.
Thus, users on the most popular platforms will be able to install most
popular Python module distributions in the most natural way for their
platform, without having to run a single setup script or compile a line
of code.

8.1.2 A Simple Example
----------------------

The setup script is usually quite simple, although since it's written
in Python, there are no arbitrary limits to what you can do with it,
though you should be careful about putting arbitrarily expensive
operations in your setup script.  Unlike, say, Autoconf-style configure
scripts, the setup script may be run multiple times in the course of
building and installing your module distribution.

  If all you want to do is distribute a module called `foo', contained
in a file `foo.py', then your setup script can be as simple as this:

    from distutils.core import setup
    setup(name='foo',
          version='1.0',
          py_modules=['foo'],
          )

Some observations:

   * most information that you supply to the Distutils is supplied as
     keyword arguments to the `setup()' function

   * those keyword arguments fall into two categories: package metadata
     (name, version number) and information about what's in the package
     (a list of pure Python modules, in this case)

   * modules are specified by module name, not filename (the same will
     hold true for packages and extensions)

   * it's recommended that you supply a little more metadata, in
     particular your name, email address and a URL for the project (see
     section Writing the Setup Script.  for an example)

  To create a source distribution for this module, you would create a
setup script, `setup.py', containing the above code, and run:

    python setup.py sdist

which will create an archive file (e.g., tarball on Unix, ZIP file on
Windows) containing your setup script `setup.py', and your module
`foo.py'.  The archive file will be named `foo-1.0.tar.gz' (or `.zip'),
and will unpack into a directory `foo-1.0'.

  If an end-user wishes to install your `foo' module, all she has to do
is download `foo-1.0.tar.gz' (or `.zip'), unpack it, and--from the
`foo-1.0' directory--run

    python setup.py install

which will ultimately copy `foo.py' to the appropriate directory for
third-party modules in their Python installation.

  This simple example demonstrates some fundamental concepts of the
Distutils.  First, both developers and installers have the same basic
user interface, i.e.  the setup script.  The difference is which
Distutils _commands_ they use: the *sdist* command is almost
exclusively for module developers, while *install* is more often for
installers (although most developers will want to install their own
code occasionally).

  If you want to make things really easy for your users, you can create
one or more built distributions for them.  For instance, if you are
running on a Windows machine, and want to make things easy for other
Windows users, you can create an executable installer (the most
appropriate type of built distribution for this platform) with the
*bdist_wininst* command.  For example:

    python setup.py bdist_wininst

will create an executable installer, `foo-1.0.win32.exe', in the current
directory.

  Other useful built distribution formats are RPM, implemented by the
*bdist_rpm* command, Solaris *pkgtool* (*bdist_pkgtool*), and HP-UX
*swinstall* (*bdist_sdux*).  For example, the following command will
create an RPM file called `foo-1.0.noarch.rpm':

    python setup.py bdist_rpm

(The *bdist_rpm* command uses the *rpm* executable, therefore this has
to be run on an RPM-based system such as Red Hat Linux, SuSE Linux, or
Mandrake Linux.)

  You can find out what distribution formats are available at any time
by running

    python setup.py bdist --help-formats

8.1.3 General Python terminology
--------------------------------

If you're reading this document, you probably have a good idea of what
modules, extensions, and so forth are.  Nevertheless, just to be sure
that everyone is operating from a common starting point, we offer the
following glossary of common Python terms:

module
     the basic unit of code reusability in Python: a block of code
     imported by some other code.  Three types of modules concern us
     here: pure Python modules, extension modules, and packages.

pure Python module
     a module written in Python and contained in a single `.py' file
     (and possibly associated `.pyc' and/or `.pyo' files).  Sometimes
     referred to as a "pure module."

extension module
     a module written in the low-level language of the Python
     implementation: C/C++ for Python, Java for Jython. Typically
     contained in a single dynamically loadable pre-compiled file, e.g.
     a shared object (`.so') file for Python extensions on Unix, a DLL
     (given the `.pyd' extension) for Python extensions on Windows, or
     a Java class file for Jython extensions.  (Note that currently,
     the Distutils only handles C/C++ extensions for Python.)

package
     a module that contains other modules; typically contained in a
     directory in the filesystem and distinguished from other
     directories by the presence of a file `__init__.py'.

root package
     the root of the hierarchy of packages.  (This isn't really a
     package, since it doesn't have an `__init__.py' file.  But we have
     to call it something.)  The vast majority of the standard library
     is in the root package, as are many small, standalone third-party
     modules that don't belong to a larger module collection. Unlike
     regular packages, modules in the root package can be found in many
     directories: in fact, every directory listed in `sys.path'
     contributes modules to the root package.

8.1.4 Distutils-specific terminology
------------------------------------

The following terms apply more specifically to the domain of
distributing Python modules using the Distutils:

module distribution
     a collection of Python modules distributed together as a single
     downloadable resource and meant to be installed _en masse_.
     Examples of some well-known module distributions are Numeric
     Python, PyXML, PIL (the Python Imaging Library), or mxBase.  (This
     would be called a _package_, except that term is already taken in
     the Python context: a single module distribution may contain zero,
     one, or many Python packages.)

pure module distribution
     a module distribution that contains only pure Python modules and
     packages.  Sometimes referred to as a "pure distribution."

non-pure module distribution
     a module distribution that contains at least one extension module.
     Sometimes referred to as a "non-pure distribution."

distribution root
     the top-level directory of your source tree (or  source
     distribution); the directory where `setup.py' exists.  Generally
     `setup.py' will be run from this directory.

8.2 Writing the Setup Script
============================

The setup script is the centre of all activity in building,
distributing, and installing modules using the Distutils.  The main
purpose of the setup script is to describe your module distribution to
the Distutils, so that the various commands that operate on your
modules do the right thing.  As we saw in section A Simple Example
above, the setup script consists mainly of a call to
`setup()', and most information supplied to the Distutils by the module
developer is supplied as keyword arguments to `setup()'.

  Here's a slightly more involved example, which we'll follow for the
next couple of sections: the Distutils' own setup script.  (Keep in
mind that although the Distutils are included with Python 1.6 and
later, they also have an independent existence so that Python 1.5.2
users can use them to install other module distributions.  The
Distutils' own setup script, shown here, is used to install the package
into Python 1.5.2.)

    #!/usr/bin/env python

    from distutils.core import setup

    setup(name='Distutils',
          version='1.0',
          description='Python Distribution Utilities',
          author='Greg Ward',
          author_email='gward@python.net',
          url='http://www.python.org/sigs/distutils-sig/',
          packages=['distutils', 'distutils.command'],
         )

There are only two differences between this and the trivial one-file
distribution presented in section A Simple Example.: more
metadata, and the specification of pure Python modules by package,
rather than by module.  This is important since the Distutils consist
of a couple of dozen modules split into (so far) two packages; an
explicit list of every module would be tedious to generate and
difficult to maintain.  For more information on the additional
meta-data, see section Additional meta-data.

  Note that any pathnames (files or directories) supplied in the setup
script should be written using the Unix convention, i.e.
slash-separated.  The Distutils will take care of converting this
platform-neutral representation into whatever is appropriate on your
current platform before actually using the pathname.  This makes your
setup script portable across operating systems, which of course is one
of the major goals of the Distutils.  In this spirit, all pathnames in
this document are slash-separated.

  This, of course, only applies to pathnames given to Distutils
functions.  If you, for example, use standard Python functions such as
glob.glob() or os.listdir() to specify files,
you should be careful to write portable code instead of hardcoding path
separators:

    glob.glob(os.path.join('mydir', 'subdir', '*.html'))
    os.listdir(os.path.join('mydir', 'subdir'))

8.2.1 Listing whole packages
----------------------------

The `packages' option tells the Distutils to process (build, distribute,
install, etc.) all pure Python modules found in each package mentioned
in the `packages' list.  In order to do this, of course, there has to
be a correspondence between package names and directories in the
filesystem.  The default correspondence is the most obvious one, i.e.
package distutils is found in the directory `distutils'
relative to the distribution root.  Thus, when you say `packages =
['foo']' in your setup script, you are promising that the Distutils
will find a file `foo/__init__.py' (which might be spelled differently
on your system, but you get the idea) relative to the directory where
your setup script lives.  If you break this promise, the Distutils will
issue a warning but still process the broken package anyways.

  If you use a different convention to lay out your source directory,
that's no problem: you just have to supply the `package_dir' option to
tell the Distutils about your convention.  For example, say you keep
all Python source under `lib', so that modules in the "root package"
(i.e., not in any package at all) are in `lib', modules in the `foo'
package are in `lib/foo', and so forth.  Then you would put

    package_dir = {'': 'lib'}

in your setup script.  The keys to this dictionary are package names,
and an empty package name stands for the root package.  The values are
directory names relative to your distribution root.  In this case, when
you say `packages = ['foo']', you are promising that the file
`lib/foo/__init__.py' exists.

  Another possible convention is to put the `foo' package right in
`lib', the `foo.bar' package in `lib/bar', etc.  This would be written
in the setup script as

    package_dir = {'foo': 'lib'}

A `package: dir' entry in the `package_dir' dictionary implicitly
applies to all packages below _package_, so the `foo.bar' case is
automatically handled here.  In this example, having `packages = ['foo',
'foo.bar']' tells the Distutils to look for `lib/__init__.py' and
`lib/bar/__init__.py'.  (Keep in mind that although `package_dir'
applies recursively, you must explicitly list all packages in
`packages': the Distutils will _not_ recursively scan your source tree
looking for any directory with an `__init__.py' file.)

8.2.2 Listing individual modules
--------------------------------

For a small module distribution, you might prefer to list all modules
rather than listing packages--especially the case of a single module
that goes in the "root package" (i.e., no package at all).  This
simplest case was shown in section A Simple Example.; here
is a slightly more involved example:

    py_modules = ['mod1', 'pkg.mod2']

This describes two modules, one of them in the "root" package, the
other in the `pkg' package.  Again, the default package/directory
layout implies that these two modules can be found in `mod1.py' and
`pkg/mod2.py', and that `pkg/__init__.py' exists as well. And again,
you can override the package/directory correspondence using the
`package_dir' option.

8.2.3 Describing extension modules
----------------------------------

Just as writing Python extension modules is a bit more complicated than
writing pure Python modules, describing them to the Distutils is a bit
more complicated.  Unlike pure modules, it's not enough just to list
modules or packages and expect the Distutils to go out and find the
right files; you have to specify the extension name, source file(s),
and any compile/link requirements (include directories, libraries to
link with, etc.).

  All of this is done through another keyword argument to `setup()', the
`ext_modules' option.  `ext_modules' is just a list of `Extension'
instances, each of which describes a single extension module.  Suppose
your distribution includes a single extension, called `foo' and
implemented by `foo.c'.  If no additional instructions to the
compiler/linker are needed, describing this extension is quite simple:

    Extension('foo', ['foo.c'])

The `Extension' class can be imported from distutils.core.
along with `setup()'.  Thus, the setup script for a module distribution
that contains only this one extension and nothing else might be:

    from distutils.core import setup, Extension
    setup(name='foo',
          version='1.0',
          ext_modules=[Extension('foo', ['foo.c'])],
          )

The `Extension' class (actually, the underlying extension-building
machinery implemented by the *build_ext* command) supports a great deal
of flexibility in describing Python extensions, which is explained in
the following sections.

8.2.3.1 Extension names and packages
....................................

The first argument to the `Extension' constructor is always the name of
the extension, including any package names.  For example,

    Extension('foo', ['src/foo1.c', 'src/foo2.c'])

describes an extension that lives in the root package, while

    Extension('pkg.foo', ['src/foo1.c', 'src/foo2.c'])

describes the same extension in the `pkg' package.  The source files and
resulting object code are identical in both cases; the only difference
is where in the filesystem (and therefore where in Python's namespace
hierarchy) the resulting extension lives.

  If you have a number of extensions all in the same package (or all
under the same base package), use the `ext_package' keyword argument to
`setup()'.  For example,

    setup(...,
          ext_package='pkg',
          ext_modules=[Extension('foo', ['foo.c']),
                       Extension('subpkg.bar', ['bar.c'])],
         )

will compile `foo.c' to the extension `pkg.foo', and `bar.c' to
`pkg.subpkg.bar'.

8.2.3.2 Extension source files
..............................

The second argument to the `Extension' constructor is a list of source
files.  Since the Distutils currently only support C, C++, and
Objective-C extensions, these are normally C/C++/Objective-C source
files.  (Be sure to use appropriate extensions to distinguish C++source
files: `.cc' and `.cpp' seem to be recognized by both Unix and Windows
compilers.)

  However, you can also include SWIG interface (`.i') files in the
list; the *build_ext* command knows how to deal with SWIG extensions:
it will run SWIG on the interface file and compile the resulting C/C++
file into your extension.

  This warning notwithstanding, options to SWIG can be currently passed
like this:

    setup(...,
          ext_modules=[Extension('_foo', ['foo.i'],
                                 swig_opts=['-modern', '-I../include'])],
          py_modules=['foo'],
         )

Or on the commandline like this:

    > python setup.py build_ext --swig-opts="-modern -I../include"

On some platforms, you can include non-source files that are processed
by the compiler and included in your extension.  Currently, this just
means Windows message text (`.mc') files and resource definition
(`.rc') files for Visual C++. These will be compiled to binary resource
(`.res') files and linked into the executable.

8.2.3.3 Preprocessor options
............................

Three optional arguments to `Extension' will help if you need to specify
include directories to search or preprocessor macros to define/undefine:
`include_dirs', `define_macros', and `undef_macros'.

  For example, if your extension requires header files in the `include'
directory under your distribution root, use the `include_dirs' option:

    Extension('foo', ['foo.c'], include_dirs=['include'])

You can specify absolute directories there; if you know that your
extension will only be built on Unix systems with X11R6 installed to
`/usr', you can get away with

    Extension('foo', ['foo.c'], include_dirs=['/usr/include/X11'])

You should avoid this sort of non-portable usage if you plan to
distribute your code: it's probably better to write C code like

    #include <X11/Xlib.h>

If you need to include header files from some other Python extension,
you can take advantage of the fact that header files are installed in a
consistent way by the Distutils *install_header* command.  For example,
the Numerical Python header files are installed (on a standard Unix
installation) to `/usr/local/include/python1.5/Numerical'. (The exact
location will differ according to your platform and Python
installation.)  Since the Python include
directory--`/usr/local/include/python1.5' in this case--is always
included in the search path when building Python extensions, the best
approach is to write C code like

    #include <Numerical/arrayobject.h>

If you must put the `Numerical' include directory right into your header
search path, though, you can find that directory using the Distutils
distutils.sysconfig module:

    from distutils.sysconfig import get_python_inc
    incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical')
    setup(...,
          Extension(..., include_dirs=[incdir]),
          )

Even though this is quite portable--it will work on any Python
installation, regardless of platform--it's probably easier to just
write your C code in the sensible way.

  You can define and undefine pre-processor macros with the
`define_macros' and `undef_macros' options. `define_macros' takes a
list of `(name, value)' tuples, where `name' is the name of the macro
to define (a string) and `value' is its value: either a string or
`None'.  (Defining a macro `FOO' to `None' is the equivalent of a bare
`#define FOO' in your C source: with most compilers, this sets `FOO' to
the string `1'.)  `undef_macros' is just a list of macros to undefine.

  For example:

    Extension(...,
              define_macros=[('NDEBUG', '1'),
                             ('HAVE_STRFTIME', None)],
              undef_macros=['HAVE_FOO', 'HAVE_BAR'])

is the equivalent of having this at the top of every C source file:

    #define NDEBUG 1
    #define HAVE_STRFTIME
    #undef HAVE_FOO
    #undef HAVE_BAR

8.2.3.4 Library options
.......................

You can also specify the libraries to link against when building your
extension, and the directories to search for those libraries.  The
`libraries' option is a list of libraries to link against,
`library_dirs' is a list of directories to search for libraries at
link-time, and `runtime_library_dirs' is a list of directories to
search for shared (dynamically loaded) libraries at run-time.

  For example, if you need to link against libraries known to be in the
standard library search path on target systems

    Extension(...,
              libraries=['gdbm', 'readline'])

If you need to link with libraries in a non-standard location, you'll
have to include the location in `library_dirs':

    Extension(...,
              library_dirs=['/usr/X11R6/lib'],
              libraries=['X11', 'Xt'])

(Again, this sort of non-portable construct should be avoided if you
intend to distribute your code.)

8.2.3.5 Other options
.....................

There are still some other options which can be used to handle special
cases.

  The `optional' option is a boolean; if it is true, a build failure in
the extension will not abort the build process, but instead simply not
install the failing extension.

  The `extra_objects' option is a list of object files to be passed to
the linker. These files must not have extensions, as the default
extension for the compiler is used.

  `extra_compile_args' and `extra_link_args' can be used to specify
additional command line options for the respective compiler and linker
command lines.

  `export_symbols' is only useful on Windows.  It can contain a list of
symbols (functions or variables) to be exported. This option is not
needed when building compiled extensions: Distutils  will automatically
add `initmodule' to the list of exported symbols.

  The `depends' option is a list of files that the extension depends on
(for example header files). The build command will call the compiler on
the sources to rebuild extension if any on this files has been modified
since the previous build.

8.2.4 Relationships between Distributions and Packages
------------------------------------------------------

A distribution may relate to packages in three specific ways:

  1. It can require packages or modules.

  2. It can provide packages or modules.

  3. It can obsolete packages or modules.

  These relationships can be specified using keyword arguments to the
distutils.core.setup() function.

  Dependencies on other Python modules and packages can be specified by
supplying the _requires_ keyword argument to `setup()'. The value must
be a list of strings.  Each string specifies a package that is
required, and optionally what versions are sufficient.

  To specify that any version of a module or package is required, the
string should consist entirely of the module or package name. Examples
include `'mymodule'' and `'xml.parsers.expat''.

  If specific versions are required, a sequence of qualifiers can be
supplied in parentheses.  Each qualifier may consist of a comparison
operator and a version number.  The accepted comparison operators are:

    <    >    ==
    <=   >=   !=

These can be combined by using multiple qualifiers separated by commas
(and optional whitespace).  In this case, all of the qualifiers must be
matched; a logical AND is used to combine the evaluations.

  Let's look at a bunch of examples:

Requires Expression           Explanation
---------------------------------------------------------------------------------
`==1.0'                       Only version `1.0' is compatible
`>1.0, !=1.5.1, <2.0'         Any version after `1.0' and before `2.0' is
                              compatible, except `1.5.1'

  Now that we can specify dependencies, we also need to be able to
specify what we provide that other distributions can require.  This is
done using the _provides_ keyword argument to `setup()'. The value for
this keyword is a list of strings, each of which names a Python module
or package, and optionally identifies the version.  If the version is
not specified, it is assumed to match that of the distribution.

  Some examples:

Provides Expression       Explanation
-----------------------------------------------------------------------------
`mypkg'                   Provide `mypkg', using the distribution version
`mypkg (1.1)'             Provide `mypkg' version 1.1, regardless of the
                          distribution version

  A package can declare that it obsoletes other packages using the
_obsoletes_ keyword argument.  The value for this is similar to that of
the _requires_ keyword: a list of strings giving module or package
specifiers.  Each specifier consists of a module or package name
optionally followed by one or more version qualifiers.  Version
qualifiers are given in parentheses after the module or package name.

  The versions identified by the qualifiers are those that are
obsoleted by the distribution being described.  If no qualifiers are
given, all versions of the named module or package are understood to be
obsoleted.

8.2.5 Installing Scripts
------------------------

So far we have been dealing with pure and non-pure Python modules,
which are usually not run by themselves but imported by scripts.

  Scripts are files containing Python source code, intended to be
started from the command line.  Scripts don't require Distutils to do
anything very complicated.  The only clever feature is that if the
first line of the script starts with `#!' and contains the word
"python", the Distutils will adjust the first line to refer to the
current interpreter location. By default, it is replaced with the
current interpreter location.  The `--executable' (or `-e') option will
allow the interpreter path to be explicitly overridden.

  The `scripts' option simply is a list of files to be handled in this
way.  From the PyXML setup script:

    setup(...,
          scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val']
          )

Changed in version 2.7: All the scripts will also be added to the
`MANIFEST' file if no template is provided. See Specifying the files to distribute.

8.2.6 Installing Package Data
-----------------------------

Often, additional files need to be installed into a package.  These
files are often data that's closely related to the package's
implementation, or text files containing documentation that might be of
interest to programmers using the package.  These files are called
_package data_.

  Package data can be added to packages using the `package_data' keyword
argument to the `setup()' function.  The value must be a mapping from
package name to a list of relative path names that should be copied
into the package.  The paths are interpreted as relative to the
directory containing the package (information from the `package_dir'
mapping is used if appropriate); that is, the files are expected to be
part of the package in the source directories. They may contain glob
patterns as well.

  The path names may contain directory portions; any necessary
directories will be created in the installation.

  For example, if a package should contain a subdirectory with several
data files, the files can be arranged like this in the source tree:

    setup.py
    src/
        mypkg/
            __init__.py
            module.py
            data/
                tables.dat
                spoons.dat
                forks.dat

The corresponding call to `setup()' might be:

    setup(...,
          packages=['mypkg'],
          package_dir={'mypkg': 'src/mypkg'},
          package_data={'mypkg': ['data/*.dat']},
          )

New in version 2.4.

  Changed in version 2.7: All the files that match `package_data' will
be added to the `MANIFEST' file if no template is provided. See
Specifying the files to distribute.

8.2.7 Installing Additional Files
---------------------------------

The `data_files' option can be used to specify additional files needed
by the module distribution: configuration files, message catalogs, data
files, anything which doesn't fit in the previous categories.

  `data_files' specifies a sequence of (_directory_, _files_) pairs in
the following way:

    setup(...,
          data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']),
                      ('config', ['cfg/data.cfg']),
                      ('/etc/init.d', ['init-script'])]
         )

Note that you can specify the directory names where the data files will
be installed, but you cannot rename the data files themselves.

  Each (_directory_, _files_) pair in the sequence specifies the
installation directory and the files to install there.  If _directory_
is a relative path, it is interpreted relative to the installation
prefix (Python's `sys.prefix' for pure-Python packages,
`sys.exec_prefix' for packages that contain extension modules).  Each
file name in _files_ is interpreted relative to the `setup.py' script
at the top of the package source distribution.  No directory
information from _files_ is used to determine the final location of the
installed file; only the name of the file is used.

  You can specify the `data_files' options as a simple sequence of files
without specifying a target directory, but this is not recommended, and
the *install* command will print a warning in this case. To install data
files directly in the target directory, an empty string should be given
as the directory.

  Changed in version 2.7: All the files that match `data_files' will be
added to the `MANIFEST' file if no template is provided. See
Specifying the files to distribute.

8.2.8 Additional meta-data
--------------------------

The setup script may include additional meta-data beyond the name and
version.  This information includes:

Meta-Data                  Description                     Value                 Notes
----------------------------------------------------------------------------------------------
`name'                     name of the package             short string          (1)
`version'                  version of this release         short string          (1)(2)
`author'                   package author's name           short string          (3)
`author_email'             email address of the package    email address         (3)
                           author
`maintainer'               package maintainer's name       short string          (3)
`maintainer_email'         email address of the package    email address         (3)
                           maintainer
`url'                      home page for the package       URL                   (1)
`description'              short, summary description of   short string
                           the package
`long_description'         longer description of the       long string           (5)
                           package
`download_url'             location where the package may  URL                   (4)
                           be downloaded
`classifiers'              a list of classifiers           list of strings       (4)
`platforms'                a list of platforms             list of strings
`license'                  license for the package         short string          (6)

  Notes:

  1. These fields are required.

  2. It is recommended that versions take the form
     _major.minor[.patch[.sub]]_.

  3. Either the author or the maintainer must be identified.

  4. These fields should not be used if your package is to be
     compatible with Python versions prior to 2.2.3 or 2.3.  The list
     is available from the PyPI website (http://pypi.python.org/pypi).

  5. The `long_description' field is used by PyPI when you are
     registering a package, to build its home page.

  6. The `license' field is a text indicating the license covering the
     package where the license is not a selection from the "License"
     Trove classifiers. See the `Classifier' field. Notice that there's
     a `licence' distribution option which is deprecated but still acts
     as an alias for `license'.

'short string'
     A single line of text, not more than 200 characters.

'long string'
     Multiple lines of plain text in reStructuredText format (see
     <http://docutils.sf.net/>).

'list of strings'
     See below.

  None of the string values may be Unicode.

  Encoding the version information is an art in itself. Python packages
generally adhere to the version format _major.minor[.patch][sub]_. The
major number is 0 for initial, experimental releases of software. It is
incremented for releases that represent major milestones in a package.
The minor number is incremented when important new features are added
to the package. The patch number increments when bug-fix releases are
made. Additional trailing version information is sometimes used to
indicate sub-releases.  These are "a1,a2,...,aN" (for alpha releases,
where functionality and API may change), "b1,b2,...,bN" (for beta
releases, which only fix bugs) and "pr1,pr2,...,prN" (for final
pre-release release testing). Some examples:

0.1.0
     the first, experimental release of a package

1.0.1a2
     the second alpha release of the first patch version of 1.0

  `classifiers' are specified in a Python list:

    setup(...,
          classifiers=[
              'Development Status :: 4 - Beta',
              'Environment :: Console',
              'Environment :: Web Environment',
              'Intended Audience :: End Users/Desktop',
              'Intended Audience :: Developers',
              'Intended Audience :: System Administrators',
              'License :: OSI Approved :: Python Software Foundation License',
              'Operating System :: MacOS :: MacOS X',
              'Operating System :: Microsoft :: Windows',
              'Operating System :: POSIX',
              'Programming Language :: Python',
              'Topic :: Communications :: Email',
              'Topic :: Office/Business',
              'Topic :: Software Development :: Bug Tracking',
              ],
          )

If you wish to include classifiers in your `setup.py' file and also wish
to remain backwards-compatible with Python releases prior to 2.2.3,
then you can include the following code fragment in your `setup.py'
before the `setup()' call.

    # patch distutils if it can't cope with the "classifiers" or
    # "download_url" keywords
    from sys import version
    if version < '2.2.3':
        from distutils.dist import DistributionMetadata
        DistributionMetadata.classifiers = None
        DistributionMetadata.download_url = None

8.2.9 Debugging the setup script
--------------------------------

Sometimes things go wrong, and the setup script doesn't do what the
developer wants.

  Distutils catches any exceptions when running the setup script, and
print a simple error message before the script is terminated.  The
motivation for this behaviour is to not confuse administrators who
don't know much about Python and are trying to install a package.  If
they get a big long traceback from deep inside the guts of Distutils,
they may think the package or the Python installation is broken because
they don't read all the way down to the bottom and see that it's a
permission problem.

  On the other hand, this doesn't help the developer to find the cause
of the failure. For this purpose, the DISTUTILS_DEBUG environment
variable can be set to anything except an empty string, and distutils
will now print detailed information what it is doing, and prints the
full traceback in case an exception occurs.

8.3 Writing the Setup Configuration File
========================================

Often, it's not possible to write down everything needed to build a
distribution _a priori_: you may need to get some information from the
user, or from the user's system, in order to proceed.  As long as that
information is fairly simple--a list of directories to search for C
header files or libraries, for example--then providing a configuration
file, `setup.cfg', for users to edit is a cheap and easy way to solicit
it.  Configuration files also let you provide default values for any
command option, which the installer can then override either on the
command-line or by editing the config file.

  The setup configuration file is a useful middle-ground between the
setup script --which, ideally, would be opaque to installers (1)--and
the command-line to the setup script, which is outside of your control
and entirely up to the installer.  In fact, `setup.cfg' (and any other
Distutils configuration files present on the target system) are
processed after the contents of the setup script, but before the
command-line.  This has  several useful consequences:

   * installers can override some of what you put in `setup.py' by
     editing `setup.cfg'

   * you can provide non-standard defaults for options that are not
     easily set in `setup.py'

   * installers can override anything in `setup.cfg' using the
     command-line options to `setup.py'

  The basic syntax of the configuration file is simple:

    [command]
    option=value
    ...

where _command_ is one of the Distutils commands (e.g. *build_py*,
*install*), and _option_ is one of the options that command supports.
Any number of options can be supplied for each command, and any number
of command sections can be included in the file.  Blank lines are
ignored, as are comments, which run from a `'#'' character until the
end of the line.  Long option values can be split across multiple lines
simply by indenting the continuation lines.

  You can find out the list of options supported by a particular
command with the universal -help option, e.g.

    > python setup.py --help build_ext
    [...]
    Options for 'build_ext' command:
      --build-lib (-b)     directory for compiled extension modules
      --build-temp (-t)    directory for temporary files (build by-products)
      --inplace (-i)       ignore build-lib and put compiled extensions into the
                           source directory alongside your pure Python modules
      --include-dirs (-I)  list of directories to search for header files
      --define (-D)        C preprocessor macros to define
      --undef (-U)         C preprocessor macros to undefine
      --swig-opts          list of SWIG command line options
    [...]

Note that an option spelled `--foo-bar' on the command-line  is spelled
`foo_bar' in configuration files.

  For example, say you want your extensions to be built
"in-place"--that is, you have an extension `pkg.ext', and you want the
compiled extension file (`ext.so' on Unix, say) to be put in the same
source directory as your pure Python modules `pkg.mod1' and `pkg.mod2'.
You can always use the `--inplace' option on the command-line to ensure
this:

    python setup.py build_ext --inplace

But this requires that you always specify the *build_ext* command
explicitly, and remember to provide `--inplace'. An easier way is to
"set and forget" this option, by encoding it in `setup.cfg', the
configuration file for this distribution:

    [build_ext]
    inplace=1

This will affect all builds of this module distribution, whether or not
you explicitly specify *build_ext*.  If you include `setup.cfg' in your
source distribution, it will also affect end-user builds--which is
probably a bad idea for this option, since always building extensions
in-place would break installation of the module distribution.  In
certain peculiar cases, though, modules are built right in their
installation directory, so this is conceivably a useful ability.
(Distributing extensions that expect to be built in their installation
directory is almost always a bad idea, though.)

  Another example: certain commands take a lot of options that don't
change from run to run; for example, *bdist_rpm* needs to know
everything required to generate a "spec" file for creating an RPM
distribution.  Some of this information comes from the setup script,
and some is automatically generated by the Distutils (such as the list
of files installed).  But some of it has to be supplied as options to
*bdist_rpm*, which would be very tedious to do on the command-line for
every run.  Hence, here is a snippet from the Distutils' own
`setup.cfg':

    [bdist_rpm]
    release = 1
    packager = Greg Ward <gward@python.net>
    doc_files = CHANGES.txt
                README.txt
                USAGE.txt
                doc/
                examples/

Note that the `doc_files' option is simply a whitespace-separated string
split across multiple lines for readability.

Syntax of config files in "Installing Python Modules"
     More information on the configuration files is available in the
     manual for system administrators.

Footnotes
.........

     (1) This ideal probably won't be achieved until auto-configuration
     is fully supported by the Distutils.

8.4 Creating a Source Distribution
==================================

As shown in section A Simple Example, you use the *sdist*
command to create a source distribution.  In the simplest case,

    python setup.py sdist

(assuming you haven't specified any *sdist* options in the setup script
or config file), *sdist* creates the archive of the default format for
the current platform.  The default format is a gzip'ed tar file
(`.tar.gz') on Unix, and ZIP file on Windows.

  You can specify as many formats as you like using the `--formats'
option, for example:

    python setup.py sdist --formats=gztar,zip

to create a gzipped tarball and a zip file.  The available formats are:

Format          Description                   Notes
------------------------------------------------------------
`zip'           zip file (`.zip')             (1),(3)
`gztar'         gzip'ed tar file (`.tar.gz')  (2)
`bztar'         bzip2'ed tar file
                (`.tar.bz2')
`ztar'          compressed tar file           (4)
                (`.tar.Z')
`tar'           tar file (`.tar')

  Notes:

  1. default on Windows

  2. default on Unix

  3. requires either external *zip* utility or zipfile.
     module (part of the standard Python library since Python 1.6)

  4. requires the *compress* program. Notice that this format is now
     pending for deprecation and will be removed in the future versions
     of Python.

  When using any `tar' format (`gztar', `bztar', `ztar' or `tar') under
Unix, you can specify the `owner' and `group' names that will be set
for each member of the archive.

  For example, if you want all files of the archive to be owned by root:

    python setup.py sdist --owner=root --group=root

8.4.1 Specifying the files to distribute
----------------------------------------

If you don't supply an explicit list of files (or instructions on how to
generate one), the *sdist* command puts a minimal default set into the
source distribution:

   * all Python source files implied by the `py_modules' and `packages'
     options

   * all C source files mentioned in the `ext_modules' or `libraries'
     options

   * scripts identified by the `scripts' option See Installing Scripts.

   * anything that looks like a test script: `test/test*.py'
     (currently, the Distutils don't do anything with test scripts
     except include them in source distributions, but in the future
     there will be a standard for testing Python module distributions)

   * `README.txt' (or `README'), `setup.py' (or whatever  you called
     your setup script), and `setup.cfg'

   * all files that matches the `package_data' metadata.  See
     Installing Package Data.

   * all files that matches the `data_files' metadata.  See
     Installing Additional Files.

  Sometimes this is enough, but usually you will want to specify
additional files to distribute.  The typical way to do this is to write
a _manifest template_, called `MANIFEST.in' by default.  The manifest
template is just a list of instructions for how to generate your
manifest file, `MANIFEST', which is the exact list of files to include
in your source distribution.  The *sdist* command processes this
template and generates a manifest based on its instructions and what it
finds in the filesystem.

  If you prefer to roll your own manifest file, the format is simple:
one filename per line, regular files (or symlinks to them) only.  If
you do supply your own `MANIFEST', you must specify everything: the
default set of files described above does not apply in this case.

  New in version 2.7: `MANIFEST' files start with a comment indicating
they are generated.  Files without this comment are not overwritten or
removed.

  See The MANIFEST.in template section for a syntax
reference.

8.4.2 Manifest-related options
------------------------------

The normal course of operations for the *sdist* command is as follows:

   * if the manifest file, `MANIFEST' doesn't exist, read `MANIFEST.in'
     and create the manifest

   * if neither `MANIFEST' nor `MANIFEST.in' exist, create a manifest
     with just the default file set

   * if either `MANIFEST.in' or the setup script (`setup.py') are more
     recent than `MANIFEST', recreate `MANIFEST' by reading
     `MANIFEST.in'

   * use the list of files now in `MANIFEST' (either just generated or
     read in) to create the source distribution archive(s)

  There are a couple of options that modify this behaviour.  First, use
the `--no-defaults' and `--no-prune' to disable the standard "include"
and "exclude" sets.

  Second, you might just want to (re)generate the manifest, but not
create a source distribution:

    python setup.py sdist --manifest-only

`-o' is a shortcut for `--manifest-only'.

8.4.3 The MANIFEST.in template
------------------------------

A `MANIFEST.in' file can be added in a project to define the list of
files to include in the distribution built by the *sdist* command.

  When *sdist* is run, it will look for the `MANIFEST.in' file and
interpret it to generate the `MANIFEST' file that contains the list of
files that will be included in the package.

  This mechanism can be used when the default list of files is not
enough.  (See Specifying the files to distribute).

8.4.3.1 Principle
.................

The manifest template has one command per line, where each command
specifies a set of files to include or exclude from the source
distribution.  For an example, let's look at the Distutils' own
manifest template:

    include *.txt
    recursive-include examples *.txt *.py
    prune examples/sample?/build

The meanings should be fairly clear: include all files in the
distribution root matching `*.txt', all files anywhere under the
`examples' directory matching `*.txt' or `*.py', and exclude all
directories matching `examples/sample?/build'.  All of this is done
_after_ the standard include set, so you can exclude files from the
standard set with explicit instructions in the manifest template.  (Or,
you can use the `--no-defaults' option to disable the standard set
entirely.)

  The order of commands in the manifest template matters: initially, we
have the list of default files as described above, and each command in
the template adds to or removes from that list of files.  Once we have
fully processed the manifest template, we remove files that should not
be included in the source distribution:

   * all files in the Distutils "build" tree (default `build/')

   * all files in directories named `RCS', `CVS', `.svn', `.hg',
     `.git', `.bzr' or `_darcs'

  Now we have our complete list of files, which is written to the
manifest for future reference, and then used to build the source
distribution archive(s).

  You can disable the default set of included files with the
`--no-defaults' option, and you can disable the standard exclude set
with `--no-prune'.

  Following the Distutils' own manifest template, let's trace how the
*sdist* command builds the list of files to include in the Distutils
source distribution:

  1. include all Python source files in the `distutils' and
     `distutils/command' subdirectories (because packages corresponding
     to those two directories were mentioned in the `packages' option
     in the setup script--see section Writing the Setup Script)

  2. include `README.txt', `setup.py', and `setup.cfg' (standard files)

  3. include `test/test*.py' (standard files)

  4. include `*.txt' in the distribution root (this will find
     `README.txt' a second time, but such redundancies are weeded out
     later)

  5. include anything matching `*.txt' or `*.py' in the sub-tree under
     `examples',

  6. exclude all files in the sub-trees starting at directories matching
     `examples/sample?/build'--this may exclude files included by the
     previous two steps, so it's important that the `prune' command in
     the manifest template comes after the `recursive-include' command

  7. exclude the entire `build' tree, and any `RCS', `CVS', `.svn',
     `.hg', `.git', `.bzr' and `_darcs' directories

  Just like in the setup script, file and directory names in the
manifest template should always be slash-separated; the Distutils will
take care of converting them to the standard representation on your
platform. That way, the manifest template is portable across operating
systems.

8.4.3.2 Commands
................

The manifest template commands are:

Command                                         Description
----------------------------------------------------------------------------------------------------
*include pat1 pat2 ...*                         include all files matching any of the listed
                                                patterns
*exclude pat1 pat2 ...*                         exclude all files matching any of the listed
                                                patterns
*recursive-include dir pat1 pat2 ...*           include all files under _dir_ matching any of the
                                                listed patterns
*recursive-exclude dir pat1 pat2 ...*           exclude all files under _dir_ matching any of the
                                                listed patterns
*global-include pat1 pat2 ...*                  include all files anywhere in the source tree
                                                matching -- & any of the listed patterns
*global-exclude pat1 pat2 ...*                  exclude all files anywhere in the source tree
                                                matching -- & any of the listed patterns
*prune dir*                                     exclude all files under _dir_
*graft dir*                                     include all files under _dir_

  The patterns here are Unix-style "glob" patterns: `*' matches any
sequence of regular filename characters, `?' matches any single regular
filename character, and `[range]' matches any of the characters in
_range_ (e.g., `a-z', `a-zA-Z', `a-f0-9_.').  The definition of
"regular filename character" is platform-specific: on Unix it is
anything except slash; on Windows anything except backslash or colon.

  Changed in version 2.7: An existing generated `MANIFEST' will be
regenerated without *sdist* comparing its modification time to the one
of `MANIFEST.in' or `setup.py'.

8.5 Creating Built Distributions
================================

A "built distribution" is what you're probably used to thinking of
either as a "binary package" or an "installer" (depending on your
background).  It's not necessarily binary, though, because it might
contain only Python source code and/or byte-code; and we don't call it
a package, because that word is already spoken for in Python.  (And
"installer" is a term specific to the world of mainstream desktop
systems.)

  A built distribution is how you make life as easy as possible for
installers of your module distribution: for users of RPM-based Linux
systems, it's a binary RPM; for Windows users, it's an executable
installer; for Debian-based Linux users, it's a Debian package; and so
forth.  Obviously, no one person will be able to create built
distributions for every platform under the sun, so the Distutils are
designed to enable module developers to concentrate on their
specialty--writing code and creating source distributions--while an
intermediary species called _packagers_ springs up to turn source
distributions into built distributions for as many platforms as there
are packagers.

  Of course, the module developer could be his own packager; or the
packager could be a volunteer "out there" somewhere who has access to a
platform which the original developer does not; or it could be software
periodically grabbing new source distributions and turning them into
built distributions for as many platforms as the software has access
to.  Regardless of who they are, a packager uses the setup script and
the *bdist* command family to generate built distributions.

  As a simple example, if I run the following command in the Distutils
source tree:

    python setup.py bdist

then the Distutils builds my module distribution (the Distutils itself
in this case), does a "fake" installation (also in the `build'
directory), and creates the default type of built distribution for my
platform.  The default format for built distributions is a "dumb" tar
file on Unix, and a simple executable installer on Windows.  (That tar
file is considered "dumb" because it has to be unpacked in a specific
location to work.)

  Thus, the above command on a Unix system creates
`Distutils-1.0._plat_.tar.gz'; unpacking this tarball from the right
place installs the Distutils just as though you had downloaded the
source distribution and run `python setup.py install'.  (The "right
place" is either the root of the filesystem or  Python's `_prefix_'
directory, depending on the options given to the *bdist_dumb* command;
the default is to make dumb distributions relative to `_prefix_'.)

  Obviously, for pure Python distributions, this isn't any simpler than
just running `python setup.py install'--but for non-pure distributions,
which include extensions that would need to be compiled, it can mean
the difference between someone being able to use your extensions or
not.  And creating "smart" built distributions, such as an RPM package
or an executable installer for Windows, is far more convenient for
users even if your distribution doesn't include any extensions.

  The *bdist* command has a `--formats' option, similar to the *sdist*
command, which you can use to select the types of built distribution to
generate: for example,

    python setup.py bdist --format=zip

would, when run on a Unix system, create
`Distutils-1.0._plat_.zip'--again, this archive would be unpacked from
the root directory to install the Distutils.

  The available formats for built distributions are:

Format            Description                        Notes
-------------------------------------------------------------------
`gztar'           gzipped tar file (`.tar.gz')       (1),(3)
`ztar'            compressed tar file (`.tar.Z')     (3)
`tar'             tar file (`.tar')                  (3)
`zip'             zip file (`.zip')                  (2),(4)
`rpm'             RPM                                (5)
`pkgtool'         Solaris *pkgtool*
`sdux'            HP-UX *swinstall*
`rpm'             RPM                                (5)
`wininst'         self-extracting ZIP file for       (4)
                  Windows
`msi'             Microsoft Installer.

  Notes:

  1. default on Unix

  2. default on Windows

  3. requires external utilities: *tar* and possibly one of *gzip*,
     *bzip2*, or *compress*

  4. requires either external *zip* utility or zipfile.
     module (part of the standard Python library since Python 1.6)

  5. requires external *rpm* utility, version 3.0.4 or better (use `rpm
     --version' to find out which version you have)

  You don't have to use the *bdist* command with the `--formats'
option; you can also use the command that directly implements the
format you're interested in.  Some of these *bdist* "sub-commands"
actually generate several similar formats; for instance, the
*bdist_dumb* command generates all the "dumb" archive formats (`tar',
`ztar', `gztar', and `zip'), and *bdist_rpm* generates both binary and
source RPMs.  The *bdist* sub-commands, and the formats generated by
each, are:

Command                        Formats
-----------------------------------------------------------
*bdist_dumb*                   tar, ztar, gztar, zip
*bdist_rpm*                    rpm, srpm
*bdist_wininst*                wininst
*bdist_msi*                    msi

  The following sections give details on the individual *bdist_**
commands.

8.5.1 Creating dumb built distributions
---------------------------------------

8.5.2 Creating RPM packages
---------------------------

The RPM format is used by many popular Linux distributions, including
Red Hat, SuSE, and Mandrake.  If one of these (or any of the other
RPM-based Linux distributions) is your usual environment, creating RPM
packages for other users of that same distribution is trivial.
Depending on the complexity of your module distribution and differences
between Linux distributions, you may also be able to create RPMs that
work on different RPM-based distributions.

  The usual way to create an RPM of your module distribution is to run
the *bdist_rpm* command:

    python setup.py bdist_rpm

or the *bdist* command with the `--format' option:

    python setup.py bdist --formats=rpm

The former allows you to specify RPM-specific options; the latter
allows  you to easily specify multiple formats in one run.  If you need
to do both, you can explicitly specify multiple *bdist_** commands and
their options:

    python setup.py bdist_rpm --packager="John Doe <jdoe@example.org>" \
                    bdist_wininst --target-version="2.0"

Creating RPM packages is driven by a `.spec' file, much as using the
Distutils is driven by the setup script.  To make your life easier, the
*bdist_rpm* command normally creates a `.spec' file based on the
information you supply in the setup script, on the command line, and in
any Distutils configuration files.  Various options and sections in the
`.spec' file are derived from options in the setup script as follows:

RPM `.spec' file option or section             Distutils setup script option
--------------------------------------------------------------------------------------------------
Name                                           `name'
Summary (in preamble)                          `description'
Version                                        `version'
Vendor                                         `author' and `author_email', or  -- &
                                               `maintainer' and `maintainer_email'
Copyright                                      `license'
Url                                            `url'
%description (section)                         `long_description'

  Additionally, there are many options in `.spec' files that don't have
corresponding options in the setup script.  Most of these are handled
through options to the *bdist_rpm* command as follows:

RPM `.spec' file option or section  *bdist_rpm* option                default value
----------------------------------------------------------------------------------------------------
Release                             `release'                         "1"
Group                               `group'                           "Development/Libraries"
Vendor                              `vendor'                          (see above)
Packager                            `packager'                        (none)
Provides                            `provides'                        (none)
Requires                            `requires'                        (none)
Conflicts                           `conflicts'                       (none)
Obsoletes                           `obsoletes'                       (none)
Distribution                        `distribution_name'               (none)
BuildRequires                       `build_requires'                  (none)
Icon                                `icon'                            (none)

  Obviously, supplying even a few of these options on the command-line
would be tedious and error-prone, so it's usually best to put them in
the setup configuration file, `setup.cfg'--see section Writing the Setup Configuration File.
If you distribute or package many
Python module distributions, you might want to put options that apply
to all of them in your personal Distutils configuration file
(`~/.pydistutils.cfg').  If you want to temporarily disable this file,
you can pass the -no-user-cfg option to setup.py.

  There are three steps to building a binary RPM package, all of which
are handled automatically by the Distutils:

  1. create a `.spec' file, which describes the package (analogous  to
     the Distutils setup script; in fact, much of the information in
     the  setup script winds up in the `.spec' file)

  2. create the source RPM

  3. create the "binary" RPM (which may or may not contain binary code,
     depending on whether your module distribution contains Python
     extensions)

  Normally, RPM bundles the last two steps together; when you use the
Distutils, all three steps are typically bundled together.

  If you wish, you can separate these three steps.  You can use the
`--spec-only' option to make *bdist_rpm* just create the `.spec' file
and exit; in this case, the `.spec' file will be written to the
"distribution directory"--normally `dist/', but customizable with the
`--dist-dir' option.  (Normally, the `.spec' file winds up deep in the
"build tree," in a temporary directory created by *bdist_rpm*.)

8.5.3 Creating Windows Installers
---------------------------------

Executable installers are the natural format for binary distributions on
Windows.  They display a nice graphical user interface, display some
information about the module distribution to be installed taken from
the metadata in the setup script, let the user select a few options,
and start or cancel the installation.

  Since the metadata is taken from the setup script, creating Windows
installers is usually as easy as running:

    python setup.py bdist_wininst

or the *bdist* command with the `--formats' option:

    python setup.py bdist --formats=wininst

If you have a pure module distribution (only containing pure Python
modules and packages), the resulting installer will be version
independent and have a name like `foo-1.0.win32.exe'.  These installers
can even be created on Unix platforms or Mac OS X.

  If you have a non-pure distribution, the extensions can only be
created on a Windows platform, and will be Python version dependent.
The installer filename will reflect this and now has the form
`foo-1.0.win32-py2.0.exe'.  You have to create a separate installer for
every Python version you want to support.

  The installer will try to compile pure modules into bytecode
after installation on the target system in normal and optimizing
mode.  If you don't want this to happen for some reason, you can run
the *bdist_wininst* command with the `--no-target-compile' and/or the
`--no-target-optimize' option.

  By default the installer will display the cool "Python Powered" logo
when it is run, but you can also supply your own 152x261 bitmap which
must be a Windows `.bmp' file with the `--bitmap' option.

  The installer will also display a large title on the desktop
background window when it is run, which is constructed from the name of
your distribution and the version number.  This can be changed to
another text by using the `--title' option.

  The installer file will be written to the "distribution directory" --
normally `dist/', but customizable with the `--dist-dir' option.

8.5.4 Cross-compiling on Windows
--------------------------------

Starting with Python 2.6, distutils is capable of cross-compiling
between Windows platforms.  In practice, this means that with the
correct tools installed, you can use a 32bit version of Windows to
create 64bit extensions and vice-versa.

  To build for an alternate platform, specify the `--plat-name' option
to the build command.  Valid values are currently 'win32', 'win-amd64'
and 'win-ia64'.  For example, on a 32bit version of Windows, you could
execute:

    python setup.py build --plat-name=win-amd64

to build a 64bit version of your extension.  The Windows Installers also
support this option, so the command:

    python setup.py build --plat-name=win-amd64 bdist_wininst

would create a 64bit installation executable on your 32bit version of
Windows.

  To cross-compile, you must download the Python source code and
cross-compile Python itself for the platform you are targetting - it is
not possible from a binary installation of Python (as the .lib etc file
for other platforms are not included.)  In practice, this means the
user of a 32 bit operating system will need to use Visual Studio 2008
to open the `PCBuild/PCbuild.sln' solution in the Python source tree
and build the "x64" configuration of the 'pythoncore' project before
cross-compiling extensions is possible.

  Note that by default, Visual Studio 2008 does not install 64bit
compilers or tools.  You may need to reexecute the Visual Studio setup
process and select these tools (using Control Panel->[Add/Remove]
Programs is a convenient way to check or modify your existing install.)

8.5.4.1 The Postinstallation script
...................................

Starting with Python 2.3, a postinstallation script can be specified
with the `--install-script' option.  The basename of the script must be
specified, and the script filename must also be listed in the scripts
argument to the setup function.

  This script will be run at installation time on the target system
after all the files have been copied, with `argv[1]' set to `-install',
and again at uninstallation time before the files are removed with
`argv[1]' set to `-remove'.

  The installation script runs embedded in the windows installer, every
output (`sys.stdout', `sys.stderr') is redirected into a buffer and
will be displayed in the GUI after the script has finished.

  Some functions especially useful in this context are available as
additional built-in functions in the installation script.

 -- Function: directory_created (path)
 -- Function: file_created (path)
     These functions should be called when a directory or file is
     created by the postinstall script at installation time.  It will
     register _path_ with the uninstaller, so that it will be removed
     when the distribution is uninstalled.  To be safe, directories are
     only removed if they are empty.

 -- Function: get_special_folder_path (csidl_string)
     This function can be used to retrieve special folder locations on
     Windows like the Start Menu or the Desktop.  It returns the full
     path to the folder.  _csidl_string_ must be one of the following
     strings:

         "CSIDL_APPDATA"

         "CSIDL_COMMON_STARTMENU"
         "CSIDL_STARTMENU"

         "CSIDL_COMMON_DESKTOPDIRECTORY"
         "CSIDL_DESKTOPDIRECTORY"

         "CSIDL_COMMON_STARTUP"
         "CSIDL_STARTUP"

         "CSIDL_COMMON_PROGRAMS"
         "CSIDL_PROGRAMS"

         "CSIDL_FONTS"

     If the folder cannot be retrieved, see OSError is raised.

     Which folders are available depends on the exact Windows version,
     and probably also the configuration.  For details refer to
     Microsoft's documentation of the `SHGetSpecialFolderPath()'
     function.

 -- Function: create_shortcut (target, description, filename[,
          arguments[, workdir[, iconpath[, iconindex]]]])
     This function creates a shortcut. _target_ is the path to the
     program to be started by the shortcut. _description_ is the
     description of the shortcut.  _filename_ is the title of the
     shortcut that the user will see. _arguments_ specifies the command
     line arguments, if any. _workdir_ is the working directory for the
     program. _iconpath_ is the file containing the icon for the
     shortcut, and _iconindex_ is the index of the icon in the file
     _iconpath_.  Again, for details consult the Microsoft
     documentation for the `IShellLink' interface.

8.5.5 Vista User Access Control (UAC)
-------------------------------------

Starting with Python 2.6, bdist_wininst supports a
`--user-access-control' option.  The default is 'none' (meaning no UAC
handling is done), and other valid values are 'auto' (meaning prompt
for UAC elevation if Python was installed for all users) and 'force'
(meaning always prompt for elevation).

8.6 Registering with the Package Index
======================================

The Python Package Index (PyPI) holds meta-data describing distributions
packaged with distutils. The distutils command *register* is used to
submit your distribution's meta-data to the index. It is invoked as
follows:

    python setup.py register

Distutils will respond with the following prompt:

    running register
    We need to know who you are, so please choose either:
        1. use your existing login,
        2. register as a new user,
        3. have the server generate a new password for you (and email it to you), or
        4. quit
    Your selection [default 1]:

Note: if your username and password are saved locally, you will not see
this menu.

  If you have not registered with PyPI, then you will need to do so
now. You should choose option 2, and enter your details as required.
Soon after submitting your details, you will receive an email which
will be used to confirm your registration.

  Once you are registered, you may choose option 1 from the menu. You
will be prompted for your PyPI username and password, and *register*
will then submit your meta-data to the index.

  You may submit any number of versions of your distribution to the
index. If you alter the meta-data for a particular version, you may
submit it again and the index will be updated.

  PyPI holds a record for each (name, version) combination submitted.
The first user to submit information for a given name is designated the
Owner of that name. They may submit changes through the *register*
command or through the web interface. They may also designate other
users as Owners or Maintainers.  Maintainers may edit the package
information, but not designate other Owners or Maintainers.

  By default PyPI will list all versions of a given package. To hide
certain versions, the Hidden property should be set to yes. This must
be edited through the web interface.

8.6.1 The .pypirc file
----------------------

The format of the `.pypirc' file is as follows:

    [distutils]
    index-servers =
        pypi

    [pypi]
    repository: <repository-url>
    username: <username>
    password: <password>

The _distutils_ section defines a _index-servers_ variable that lists
the name of all sections describing a repository.

  Each section describing a repository defines three variables:

   -
    _repository_, that defines the url of the PyPI server. Defaults to
          `http://www.python.org/pypi'.

   - _username_, which is the registered username on the PyPI server.

   -
    _password_, that will be used to authenticate. If omitted the user
          will be prompt to type it when needed.

  If you want to define another server a new section can be created and
listed in the _index-servers_ variable:

    [distutils]
    index-servers =
        pypi
        other

    [pypi]
    repository: <repository-url>
    username: <username>
    password: <password>

    [other]
    repository: http://example.com/pypi
    username: <username>
    password: <password>

*register* can then be called with the -r option to point the
repository to work with:

    python setup.py register -r http://example.com/pypi

For convenience, the name of the section that describes the repository
may also be used:

    python setup.py register -r other

8.7 Uploading Packages to the Package Index
===========================================

New in version 2.5.

  The Python Package Index (PyPI) not only stores the package info, but
also  the package data if the author of the package wishes to. The
distutils command *upload* pushes the distribution files to PyPI.

  The command is invoked immediately after building one or more
distribution files.  For example, the command

    python setup.py sdist bdist_wininst upload

will cause the source distribution and the Windows installer to be
uploaded to PyPI.  Note that these will be uploaded even if they are
built using an earlier invocation of `setup.py', but that only
distributions named on the command line for the invocation including
the *upload* command are uploaded.

  The *upload* command uses the username, password, and repository URL
from the `$HOME/.pypirc' file (see section The .pypirc file
for more on this file). If a *register* command was previously
called in the same command, and if the password was entered in the
prompt, *upload* will reuse the entered password. This is useful if you
do not want to store a clear text password in the `$HOME/.pypirc' file.

  You can specify another PyPI server with the `--repository=*url*'
option:

    python setup.py sdist bdist_wininst upload -r http://example.com/pypi

See section The .pypirc file for more on defining several
servers.

  You can use the `--sign' option to tell *upload* to sign each
uploaded file using GPG (GNU Privacy Guard).  The  *gpg* program must
be available for execution on the system `PATH'.  You can also specify
which key to use for signing using the `--identity=*name*' option.

  Other *upload* options include `--repository=' or `--repository='
where _url_ is the url of the server and _section_ the name of the
section in `$HOME/.pypirc', and `--show-response' (which displays the
full response text from the PyPI server for help in debugging upload
problems).

8.7.1 PyPI package display
--------------------------

The `long_description' field plays a special role at PyPI. It is used by
the server to display a home page for the registered package.

  If you use the reStructuredText
(http://docutils.sourceforge.net/rst.html)syntax for this field, PyPI
will parse it and display an HTML output for the package home page.

  The `long_description' field can be attached to a text file located
in the package:

    from distutils.core import setup

    setup(name='Distutils',
          long_description=open('README.txt'))

In that case, `README.txt' is a regular reStructuredText text file
located in the root of the package besides `setup.py'.

  To prevent registering broken reStructuredText content, you can use
the *rst2html* program that is provided by the `docutils' package and
check the `long_description' from the command line:

    $ python setup.py --long-description | rst2html.py > output.html

`docutils' will display a warning if there's something wrong with your
syntax.

8.8 Examples
============

This chapter provides a number of basic examples to help get started
with distutils.  Additional information about using distutils can be
found in the Distutils Cookbook.

Distutils Cookbook (http://wiki.python.org/moin/Distutils/Cookbook)
     Collection of recipes showing how to achieve more control over
     distutils.

8.8.1 Pure Python distribution (by module)
------------------------------------------

If you're just distributing a couple of modules, especially if they
don't live in a particular package, you can specify them individually
using the `py_modules' option in the setup script.

  In the simplest case, you'll have two files to worry about: a setup
script and the single module you're distributing, `foo.py' in this
example:

    <root>/
            setup.py
            foo.py

(In all diagrams in this section, _<root>_ will refer to the
distribution root directory.)  A minimal setup script to describe this
situation would be:

    from distutils.core import setup
    setup(name='foo',
          version='1.0',
          py_modules=['foo'],
          )

Note that the name of the distribution is specified independently with
the `name' option, and there's no rule that says it has to be the same
as the name of the sole module in the distribution (although that's
probably a good convention to follow).  However, the distribution name
is used to generate filenames, so you should stick to letters, digits,
underscores, and hyphens.

  Since `py_modules' is a list, you can of course specify multiple
modules, eg. if you're distributing modules `foo' and `bar', your setup
might look like this:

    <root>/
            setup.py
            foo.py
            bar.py

and the setup script might be

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          py_modules=['foo', 'bar'],
          )

You can put module source files into another directory, but if you have
enough modules to do that, it's probably easier to specify modules by
package rather than listing them individually.

8.8.2 Pure Python distribution (by package)
-------------------------------------------

If you have more than a couple of modules to distribute, especially if
they are in multiple packages, it's probably easier to specify whole
packages rather than individual modules.  This works even if your
modules are not in a package; you can just tell the Distutils to
process modules from the root package, and that works the same as any
other package (except that you don't have to have an `__init__.py'
file).

  The setup script from the last example could also be written as

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          packages=[''],
          )

(The empty string stands for the root package.)

  If those two files are moved into a subdirectory, but remain in the
root package, e.g.:

    <root>/
            setup.py
            src/      foo.py
                      bar.py

then you would still specify the root package, but you have to tell the
Distutils where source files in the root package live:

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          package_dir={'': 'src'},
          packages=[''],
          )

More typically, though, you will want to distribute multiple modules in
the same package (or in sub-packages).  For example, if the `foo'  and
`bar' modules belong in package `foobar', one way to layout your source
tree is

    <root>/
            setup.py
            foobar/
                     __init__.py
                     foo.py
                     bar.py

This is in fact the default layout expected by the Distutils, and the
one that requires the least work to describe in your setup script:

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          packages=['foobar'],
          )

If you want to put modules in directories not named for their package,
then you need to use the `package_dir' option again.  For example, if
the `src' directory holds modules in the `foobar' package:

    <root>/
            setup.py
            src/
                     __init__.py
                     foo.py
                     bar.py

an appropriate setup script would be

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          package_dir={'foobar': 'src'},
          packages=['foobar'],
          )

Or, you might put modules from your main package right in the
distribution root:

    <root>/
            setup.py
            __init__.py
            foo.py
            bar.py

in which case your setup script would be

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          package_dir={'foobar': ''},
          packages=['foobar'],
          )

(The empty string also stands for the current directory.)

  If you have sub-packages, they must be explicitly listed in
`packages', but any entries in `package_dir' automatically extend to
sub-packages.  (In other words, the Distutils does _not_ scan your
source tree, trying to figure out which directories correspond to
Python packages by looking for `__init__.py' files.)  Thus, if the
default layout grows a sub-package:

    <root>/
            setup.py
            foobar/
                     __init__.py
                     foo.py
                     bar.py
                     subfoo/
                               __init__.py
                               blah.py

then the corresponding setup script would be

    from distutils.core import setup
    setup(name='foobar',
          version='1.0',
          packages=['foobar', 'foobar.subfoo'],
          )

(Again, the empty string in `package_dir' stands for the current
directory.)

8.8.3 Single extension module
-----------------------------

Extension modules are specified using the `ext_modules' option.
`package_dir' has no effect on where extension source files are found;
it only affects the source for pure Python modules.  The simplest
case, a single extension module in a single C source file, is:

    <root>/
            setup.py
            foo.c

If the `foo' extension belongs in the root package, the setup script for
this could be

    from distutils.core import setup
    from distutils.extension import Extension
    setup(name='foobar',
          version='1.0',
          ext_modules=[Extension('foo', ['foo.c'])],
          )

If the extension actually belongs in a package, say `foopkg', then

  With exactly the same source tree layout, this extension can be put
in the `foopkg' package simply by changing the name of the extension:

    from distutils.core import setup
    from distutils.extension import Extension
    setup(name='foobar',
          version='1.0',
          ext_modules=[Extension('foopkg.foo', ['foo.c'])],
          )

8.9 Extending Distutils
=======================

Distutils can be extended in various ways.  Most extensions take the
form of new commands or replacements for existing commands.  New
commands may be written to support new types of platform-specific
packaging, for example, while replacements for existing commands may be
made to modify details of how the command operates on a package.

  Most extensions of the distutils are made within `setup.py' scripts
that want to modify existing commands; many simply add a few file
extensions that should be copied into packages in addition to `.py'
files as a convenience.

  Most distutils command implementations are subclasses of the
distutils.cmd.Command class.  New commands may directly inherit
from `Command', while replacements often derive from `Command'
indirectly, directly subclassing the command they are replacing.
Commands are required to derive from `Command'.

8.9.1 Integrating new commands
------------------------------

There are different ways to integrate new command implementations into
distutils.  The most difficult is to lobby for the inclusion of the new
features in distutils itself, and wait for (and require) a version of
Python that provides that support.  This is really hard for many
reasons.

  The most common, and possibly the most reasonable for most needs, is
to include the new implementations with your `setup.py' script, and
cause the distutils.core.setup() function use them:

    from distutils.command.build_py import build_py as _build_py
    from distutils.core import setup

    class build_py(_build_py):
        """Specialized Python source builder."""

        # implement whatever needs to be different...

    setup(cmdclass={'build_py': build_py},
          ...)

This approach is most valuable if the new implementations must be used
to use a particular package, as everyone interested in the package will
need to have the new command implementation.

  Beginning with Python 2.4, a third option is available, intended to
allow new commands to be added which can support existing `setup.py'
scripts without requiring modifications to the Python installation.
This is expected to allow third-party extensions to provide support for
additional packaging systems, but the commands can be used for anything
distutils commands can be used for.  A new configuration option,
`command_packages' (command-line option `--command-packages'), can be
used to specify additional packages to be searched for modules
implementing commands.  Like all distutils options, this can be
specified on the command line or in a configuration file.  This option
can only be set in the `[global]' section of a configuration file, or
before any commands on the command line.  If set in a configuration
file, it can be overridden from the command line; setting it to an
empty string on the command line causes the default to be used.  This
should never be set in a configuration file provided with a package.

  This new option can be used to add any number of packages to the list
of packages searched for command implementations; multiple package
names should be separated by commas.  When not specified, the search is
only performed in the distutils.command package.  When
`setup.py' is run with the option `--command-packages'
`distcmds,buildcmds', however, the packages distutils.command,
`distcmds', and `buildcmds' will be searched in that order.  New
commands are expected to be implemented in modules of the same name as
the command by classes sharing the same name.  Given the example
command line option above, the command *bdist_openpkg* could be
implemented by the class `distcmds.bdist_openpkg.bdist_openpkg' or
`buildcmds.bdist_openpkg.bdist_openpkg'.

8.9.2 Adding new distribution types
-----------------------------------

Commands that create distributions (files in the `dist/' directory) need
to add `(command, filename)' pairs to `self.distribution.dist_files' so
that *upload* can upload it to PyPI.  The _filename_ in the pair
contains no path information, only the name of the file itself.  In
dry-run mode, pairs should still be added to represent what would have
been created.

8.10 Command Reference
======================

8.10.1 Installing modules: the *install* command family
-------------------------------------------------------

The install command ensures that the build commands have been run and
then runs the subcommands *install_lib*, *install_data* and
*install_scripts*.

8.10.1.1 *install_data*
.......................

This command installs all data files provided with the distribution.

8.10.1.2 *install_scripts*
..........................

This command installs all (Python) scripts in the distribution.

8.11 API Reference
==================

8.11.1 `distutils.core' -- Core Distutils functionality
-------------------------------------------------------

The distutils.core module is the only module that needs to
be installed to use the Distutils. It provides the setup().
(which is called from the setup script). Indirectly provides the
`distutils.dist.Distribution' and distutils.cmd.Command.
class.

 -- Function: distutils.core.setup (arguments)
     The basic do-everything function that does most everything you
     could ever ask for from a Distutils method.

     The setup function takes a large number of arguments. These are
     laid out in the following table.

     argument name            value                                type
     --------------------------------------------------------------------------------------------------------------------------------
     _name_                   The name of the package              a string
     _version_                The version number of the package    See distutils.version.
     _description_            A single line describing the package a string
     _long_description_       Longer description of the package    a string
     _author_                 The name of the package author       a string
     _author_email_           The email address of the package     a string
                              author
     _maintainer_             The name of the current maintainer,  a string
                              if different from the author
     _maintainer_email_       The email address of the current
                              maintainer, if different from the
                              author
     _url_                    A URL for the package (homepage)     a URL
     _download_url_           A URL to download the package        a URL
     _packages_               A list of Python packages that       a list of strings
                              distutils will manipulate
     _py_modules_             A list of Python modules that        a list of strings
                              distutils will manipulate
     _scripts_                A list of standalone script files    a list of strings
                              to be built and installed
     _ext_modules_            A list of Python extensions to be    A list of  instances of distutils.core.Extension.
                              built
     _classifiers_            A list of categories for the package The list of available categorizations is at
                                                                   <http://pypi.python.org/pypi?:action=list_classifiers>.
     _distclass_              the Distribution.        A subclass of distutils.core.Distribution.
                              class to use
     _script_name_            The name of the setup.py script -    a string
                              defaults to `sys.argv[0]'
     _script_args_            Arguments to supply to the setup     a list of strings
                              script
     _options_                default options for the setup script a string
     _license_                The license for the package          a string
     _keywords_               Descriptive meta-data, see PEP 314
                              (http://www.python.org/dev/peps/pep-0314)
     _platforms_
     _cmdclass_               A mapping of command names to a dictionary Command
                              subclasses
     _data_files_             A list of data files to install      a list
     _package_dir_            A mapping of package to directory    a dictionary
                              names

 -- Function: distutils.core.run_setup (script_name[, script_args=None,
          stop_after='run'])
     Run a setup script in a somewhat controlled environment, and
     return  the `distutils.dist.Distribution' instance that drives
     things.   This is useful if you need to find out the distribution
     meta-data  (passed as keyword args from _script_ to setup()),
     or  the contents of the config files or command-line.

     _script_name_ is a file that will be run with execfile().
     `sys.argv[0]' will be replaced with _script_ for the duration
     of the call.  _script_args_ is a list of strings; if supplied,
     `sys.argv[1:]' will be replaced by _script_args_ for the duration
     of the call.

     _stop_after_ tells setup() when to stop processing;
     possible  values:

     value               description
     ----------------------------------------------------------------------
     _init_              Stop after the Distribution.
                         instance has been created  and populated with
                         the keyword arguments to setup().
     _config_            Stop after config files have been parsed (and
                         their data stored in the Distribution
                         instance)
     _commandline_       Stop after the command-line (`sys.argv[1:]' or
                         _script_args_) have been parsed (and the data
                         stored in the Distribution.
                         instance.)
     _run_               Stop after all commands have been run (the same
                         as  if setup() had been called in
                         the usual way). This is the default value.

  In addition, the distutils.core module exposed a number of
classes that live elsewhere.

   * Extension from distutils.extension.

   * Command from distutils.cmd.

   * Distribution from distutils.dist.

  A short description of each of these follows, but see the relevant
module for the full reference.

 -- Class: distutils.core.Extension
     The Extension class describes a single C or C++extension module in
     a setup script. It accepts the following keyword arguments in its
     constructor

     argument name                value                                type
     --------------------------------------------------------------------------------------------------
     _name_                       the full name of the extension,      string
                                  including any packages -- ie. _not_
                                  a filename or pathname, but Python
                                  dotted name
     _sources_                    list of source filenames, relative   string
                                  to the distribution root (where the
                                  setup script lives), in Unix form
                                  (slash- separated) for portability.
                                  Source files may be C, C++, SWIG
                                  (.i), platform-specific resource
                                  files, or whatever else is
                                  recognized by the *build_ext*
                                  command as source for a Python
                                  extension.
     _include_dirs_               list of directories to search for    string
                                  C/C++ header files (in Unix form
                                  for portability)
     _define_macros_              list of macros to define; each       (string, string) tuple or
                                  macro is defined using a 2-tuple     (name, `None')
                                  `(name, value)', where _value_ is
                                  either the string to define it to
                                  or `None' to define it without a
                                  particular value (equivalent of
                                  `#define FOO' in source or `-DFOO'
                                  on Unix C compiler command line)
     _undef_macros_               list of macros to undefine           string
                                  explicitly
     _library_dirs_               list of directories to search for    string
                                  C/C++ libraries at link time
     _libraries_                  list of library names (not           string
                                  filenames or paths) to link against
     _runtime_library_dirs_       list of directories to search for    string
                                  C/C++ libraries at run time (for
                                  shared extensions, this is when the
                                  extension is loaded)
     _extra_objects_              list of extra files to link with     string
                                  (eg. object files not implied by
                                  'sources', static library that must
                                  be explicitly specified, binary
                                  resource files, etc.)
     _extra_compile_args_         any extra platform- and              string
                                  compiler-specific information to
                                  use when compiling the source files
                                  in 'sources'. For platforms and
                                  compilers where a command line
                                  makes sense, this is typically a
                                  list of command-line arguments, but
                                  for other platforms it could be
                                  anything.
     _extra_link_args_            any extra platform- and              string
                                  compiler-specific information to
                                  use when linking object files
                                  together to create the extension
                                  (or to create a new static Python
                                  interpreter).  Similar
                                  interpretation as for
                                  'extra_compile_args'.
     _export_symbols_             list of symbols to be exported from  string
                                  a shared extension. Not used on all
                                  platforms, and not generally
                                  necessary for Python extensions,
                                  which typically export exactly one
                                  symbol: `init' + extension_name.
     _depends_                    list of files that the extension     string
                                  depends on
     _language_                   extension language (i.e.  `'c'',     string
                                  `'c++'', `'objc''). Will be detected
                                  from the source extensions if not
                                  provided.

 -- Class: distutils.core.Distribution
     A Distribution describes how to build, install and
     package up a Python software package.

     See the setup() function for a list of keyword
     arguments accepted  by the Distribution constructor.
     setup() creates a Distribution instance.

 -- Class: distutils.core.Command
     A Command class (or rather, an instance of one of its
     subclasses) implement a single distutils command.

8.11.2 `distutils.ccompiler' -- CCompiler base class
----------------------------------------------------

This module provides the abstract base class for the CCompiler.
classes.  A CCompiler instance can be used for all
the compile  and link steps needed to build a single project. Methods
are provided to  set options for the compiler -- macro definitions,
include directories,  link path, libraries and the like.

  This module provides the following functions.

 -- Function: distutils.ccompiler.gen_lib_options (compiler,
          library_dirs, runtime_library_dirs, libraries)
     Generate linker options for searching library directories and
     linking with specific libraries.  _libraries_ and _library_dirs_
     are, respectively, lists of library names (not filenames!) and
     search directories.  Returns a list of command-line options
     suitable for use with some compiler (depending on the two format
     strings passed in).

 -- Function: distutils.ccompiler.gen_preprocess_options (macros,
          include_dirs)
     Generate C pre-processor options (`-D', see -U, `-I') as
     used by at least two types of compilers: the typical Unix compiler
     and Visual C++. _macros_ is the usual thing, a list of 1- or
     2-tuples, where `(name,)' means undefine (see -U) macro
     _name_, and `(name, value)' means define (`-D') macro _name_ to
     _value_.  _include_dirs_ is just a list of directory names to be
     added to the header file search path (`-I').  Returns a list of
     command-line options suitable for either Unix compilers or Visual
     C++.

 -- Function: distutils.ccompiler.get_default_compiler (osname,
          platform)
     Determine the default compiler to use for the given platform.

     _osname_ should be one of the standard Python OS names (i.e. the
     ones returned by `os.name') and _platform_ the common value
     returned by `sys.platform' for the platform in question.

     The default values are `os.name' and `sys.platform' in case the
     parameters are not given.

 -- Function: distutils.ccompiler.new_compiler (plat=None,
          compiler=None, verbose=0, dry_run=0, force=0)
     Factory function to generate an instance of some CCompiler
     subclass for the supplied platform/compiler combination. _plat_
     defaults to `os.name' (eg.  `'posix'', `'nt''), and _compiler_
     defaults to the default compiler for that platform. Currently only
     `'posix'' and `'nt'' are supported, and the default compilers are
     "traditional Unix interface" (`UnixCCompiler' class) and Visual
     C++ (`MSVCCompiler' class).  Note that it's perfectly possible to
     ask for a Unix compiler object under Windows, and a Microsoft
     compiler object under Unix--if you supply a value for _compiler_,
     _plat_ is ignored.

 -- Function: distutils.ccompiler.show_compilers ()
     Print list of available compilers (used by the `--help-compiler'
     options to *build*, *build_ext*, *build_clib*).

 -- Class: distutils.ccompiler.CCompiler ([verbose=0, dry_run=0,
          force=0])
     The abstract base class CCompiler defines the
     interface that  must be implemented by real compiler classes.  The
     class also has  some utility methods used by several compiler
     classes.

     The basic idea behind a compiler abstraction class is that each
     instance can be used for all the compile/link steps in building a
     single project.  Thus, attributes common to all of those compile
     and link steps -- include directories, macros to define, libraries
     to link against, etc. -- are attributes of the compiler instance.
     To allow for variability in how individual files are treated, most
     of those attributes may be varied on a per-compilation or per-link
     basis.

     The constructor for each subclass creates an instance of the
     Compiler object.  Flags are _verbose_ (show verbose output),
     _dry_run_ (don't actually execute the steps) and _force_ (rebuild
     everything, regardless of dependencies). All of these flags
     default to `0' (off). Note that you probably don't want to
     instantiate CCompiler or one of its subclasses
     directly - use the `distutils.CCompiler.new_compiler()' factory
     function instead.

     The following methods allow you to manually alter compiler options
     for  the instance of the Compiler class.

      -- Method: add_include_dir (dir)
          Add _dir_ to the list of directories that will be searched
          for header files.  The compiler is instructed to search
          directories in the order in which they are supplied by
          successive calls to add_include_dir().

      -- Method: set_include_dirs (dirs)
          Set the list of directories that will be searched to _dirs_
          (a list of strings).  Overrides any preceding calls to
          add_include_dir().; subsequent calls to
          add_include_dir() add to the list passed to
          set_include_dirs().  This does not affect any list of
          standard include directories that the compiler may search by
          default.

      -- Method: add_library (libname)
          Add _libname_ to the list of libraries that will be included
          in all links driven by this compiler object.  Note that
          _libname_ should *not* be the name of a file containing a
          library, but the name of the library itself: the actual
          filename will be inferred by the linker, the compiler, or the
          compiler class (depending on the platform).

          The linker will be instructed to link against libraries in
          the order they were supplied to add_library().
          and/or set_libraries().  It is perfectly valid to
          duplicate library names; the linker will be instructed to
          link against libraries as many times as they are mentioned.

      -- Method: set_libraries (libnames)
          Set the list of libraries to be included in all links driven
          by this compiler object to _libnames_ (a list of strings).
          This does not affect any standard system libraries that the
          linker may include by default.

      -- Method: add_library_dir (dir)
          Add _dir_ to the list of directories that will be searched
          for libraries specified to add_library() and
          set_libraries().  The linker will be instructed
          to search for libraries in the order they are supplied to
          add_library_dir() and/or
          set_library_dirs().

      -- Method: set_library_dirs (dirs)
          Set the list of library search directories to _dirs_ (a list
          of strings).  This does not affect any standard library
          search path that the linker may search by default.

      -- Method: add_runtime_library_dir (dir)
          Add _dir_ to the list of directories that will be searched
          for shared libraries at runtime.

      -- Method: set_runtime_library_dirs (dirs)
          Set the list of directories to search for shared libraries at
          runtime to _dirs_ (a list of strings).  This does not affect
          any standard search path that the runtime linker may search
          by default.

      -- Method: define_macro (name[, value=None])
          Define a preprocessor macro for all compilations driven by
          this compiler object.  The optional parameter _value_ should
          be a string; if it is not supplied, then the macro will be
          defined without an explicit value and the exact outcome
          depends on the compiler used (XXX true? does ANSI say
          anything about this?)

      -- Method: undefine_macro (name)
          Undefine a preprocessor macro for all compilations driven by
          this compiler object.  If the same macro is defined by
          define_macro() and undefined by
          undefine_macro() the last call takes precedence
          (including multiple redefinitions or undefinitions).  If the
          macro is redefined/undefined on a per-compilation basis (ie.
          in the call to compile()), then that takes
          precedence.

      -- Method: add_link_object (object)
          Add _object_ to the list of object files (or analogues, such
          as explicitly named library files or the output of "resource
          compilers") to be included in every link driven by this
          compiler object.

      -- Method: set_link_objects (objects)
          Set the list of object files (or analogues) to be included in
          every link to _objects_.  This does not affect any standard
          object files that the linker may include by default (such as
          system libraries).

     The following methods implement methods for autodetection of
     compiler  options, providing some functionality similar to GNU
     *autoconf*.

      -- Method: detect_language (sources)
          Detect the language of a given file, or list of files. Uses
          the  instance attributes `language_map' (a dictionary), and
          `language_order' (a list) to do the job.

      -- Method: find_library_file (dirs, lib[, debug=0])
          Search the specified list of directories for a static or
          shared library file _lib_ and return the full path to that
          file.  If _debug_ is true, look for a debugging version (if
          that makes sense on the current platform).  Return `None' if
          _lib_ wasn't found in any of the specified directories.

      -- Method: has_function (funcname[, includes=None,
               include_dirs=None, libraries=None, library_dirs=None])
          Return a boolean indicating whether _funcname_ is supported
          on the current platform.  The optional arguments can be used
          to augment the compilation environment by providing
          additional include files and paths and libraries and paths.

      -- Method: library_dir_option (dir)
          Return the compiler option to add _dir_ to the list of
          directories searched for libraries.

      -- Method: library_option (lib)
          Return the compiler option to add _dir_ to the list of
          libraries linked into the shared library or executable.

      -- Method: runtime_library_dir_option (dir)
          Return the compiler option to add _dir_ to the list of
          directories searched for runtime libraries.

      -- Method: set_executables (**args)
          Define the executables (and options for them) that will be
          run to perform the various stages of compilation.  The exact
          set of executables that may be specified here depends on the
          compiler class (via the 'executables' class attribute), but
          most will have:

          attribute          description
          ------------------------------------------------------------------
          _compiler_         the C/C++ compiler
          _linker_so_        linker used to create shared objects and
                             libraries
          _linker_exe_       linker used to create binary executables
          _archiver_         static library creator

          On platforms with a command-line (Unix, DOS/Windows), each of
          these is a string that will be split into executable name and
          (optional) list of arguments.  (Splitting the string is done
          similarly to how Unix shells operate: words are delimited by
          spaces, but quotes and backslashes can override this.  See
          distutils.util.split_quoted())

     The following methods invoke stages in the build process.

      -- Method: compile (sources[, output_dir=None, macros=None,
               include_dirs=None, debug=0, extra_preargs=None,
               extra_postargs=None, depends=None])
          Compile one or more source files. Generates object files
          (e.g.  transforms a `.c' file to a `.o' file.)

          _sources_ must be a list of filenames, most likely C/C++
          files, but in reality anything that can be handled by a
          particular compiler and compiler class (eg.  `MSVCCompiler'
          can handle resource files in _sources_).  Return a list of
          object filenames, one per source filename in _sources_.
          Depending on the implementation, not all source files will
          necessarily be compiled, but all corresponding object
          filenames will be returned.

          If _output_dir_ is given, object files will be put under it,
          while retaining their original path component.  That is,
          `foo/bar.c' normally compiles to `foo/bar.o' (for a Unix
          implementation); if _output_dir_ is _build_, then it would
          compile to `build/foo/bar.o'.

          _macros_, if given, must be a list of macro definitions.  A
          macro definition is either a `(name, value)' 2-tuple or a
          `(name,)' 1-tuple. The former defines a macro; if the value
          is `None', the macro is defined without an explicit value.
          The 1-tuple case undefines a macro.  Later
          definitions/redefinitions/undefinitions take precedence.

          _include_dirs_, if given, must be a list of strings, the
          directories to add to the default include file search path
          for this compilation only.

          _debug_ is a boolean; if true, the compiler will be
          instructed to output debug symbols in (or alongside) the
          object file(s).

          _extra_preargs_ and _extra_postargs_ are
          implementation-dependent. On platforms that have the notion
          of a command-line (e.g. Unix, DOS/Windows), they are most
          likely lists of strings: extra command-line arguments to
          prepend/append to the compiler command line.  On other
          platforms, consult the implementation class documentation.
          In any event, they are intended as an escape hatch for those
          occasions when the abstract compiler framework doesn't cut
          the mustard.

          _depends_, if given, is a list of filenames that all targets
          depend on.  If a source file is older than any file in
          depends, then the source file will be recompiled.  This
          supports dependency tracking, but only at a coarse
          granularity.

          Raises `CompileError' on failure.

      -- Method: create_static_lib (objects, output_libname[,
               output_dir=None, debug=0, target_lang=None])
          Link a bunch of stuff together to create a static library
          file. The "bunch of stuff" consists of the list of object
          files supplied as _objects_, the extra object files supplied
          to add_link_object() and/or
          set_link_objects(), the libraries supplied to
          add_library() and/or set_libraries(), and
          the libraries supplied as _libraries_ (if any).

          _output_libname_ should be a library name, not a filename;
          the filename will be inferred from the library name.
          _output_dir_ is the directory where the library file will be
          put. XXX defaults to what?

          _debug_ is a boolean; if true, debugging information will be
          included in the library (note that on most platforms, it is
          the compile step where this matters: the _debug_ flag is
          included here just for consistency).

          _target_lang_ is the target language for which the given
          objects are being compiled. This allows specific linkage time
          treatment of certain languages.

          Raises `LibError' on failure.

      -- Method: link (target_desc, objects, output_filename[,
               output_dir=None, libraries=None, library_dirs=None,
               runtime_library_dirs=None, export_symbols=None, debug=0,
               extra_preargs=None, extra_postargs=None,
               build_temp=None, target_lang=None])
          Link a bunch of stuff together to create an executable or
          shared library file.

          The "bunch of stuff" consists of the list of object files
          supplied as _objects_.  _output_filename_ should be a
          filename.  If _output_dir_ is supplied, _output_filename_ is
          relative to it (i.e. _output_filename_ can provide directory
          components if needed).

          _libraries_ is a list of libraries to link against.  These
          are library names, not filenames, since they're translated
          into filenames in a platform-specific way (eg. _foo_ becomes
          `libfoo.a' on Unix and `foo.lib' on DOS/Windows).  However,
          they can include a directory component, which means the
          linker will look in that specific directory rather than
          searching all the normal locations.

          _library_dirs_, if supplied, should be a list of directories
          to search for libraries that were specified as bare library
          names (ie. no directory component).  These are on top of the
          system default and those supplied to add_library_dir()
          and/or set_library_dirs().
          _runtime_library_dirs_ is a list of directories that will be
          embedded into the shared library and used to search for other
          shared libraries that *it* depends on at run-time.  (This may
          only be relevant on Unix.)

          _export_symbols_ is a list of symbols that the shared library
          will export.  (This appears to be relevant only on Windows.)

          _debug_ is as for compile() and
          create_static_lib(),  with the slight distinction that
          it actually matters on most platforms (as opposed to
          create_static_lib(), which includes a _debug_ flag
          mostly for form's sake).

          _extra_preargs_ and _extra_postargs_ are as for
          compile().  (except of course that they supply
          command-line arguments for the particular linker being used).

          _target_lang_ is the target language for which the given
          objects are being compiled. This allows specific linkage time
          treatment of certain languages.

          Raises `LinkError' on failure.

      -- Method: link_executable (objects, output_progname[,
               output_dir=None, libraries=None, library_dirs=None,
               runtime_library_dirs=None, debug=0, extra_preargs=None,
               extra_postargs=None, target_lang=None])
          Link an executable.  _output_progname_ is the name of the
          file executable, while _objects_ are a list of object
          filenames to link in. Other arguments  are as for the
          link() method.

      -- Method: link_shared_lib (objects, output_libname[,
               output_dir=None, libraries=None, library_dirs=None,
               runtime_library_dirs=None, export_symbols=None, debug=0,
               extra_preargs=None, extra_postargs=None,
               build_temp=None, target_lang=None])
          Link a shared library. _output_libname_ is the name of the
          output  library, while _objects_ is a list of object
          filenames to link in.  Other arguments are as for the
          link() method.

      -- Method: link_shared_object (objects, output_filename[,
               output_dir=None, libraries=None, library_dirs=None,
               runtime_library_dirs=None, export_symbols=None, debug=0,
               extra_preargs=None, extra_postargs=None,
               build_temp=None, target_lang=None])
          Link a shared object. _output_filename_ is the name of the
          shared object that will be created, while _objects_ is a list
          of object filenames  to link in.  Other arguments are as for
          the link() method.

      -- Method: preprocess (source[, output_file=None, macros=None,
               include_dirs=None, extra_preargs=None,
               extra_postargs=None])
          Preprocess a single C/C++ source file, named in _source_.
          Output will be written to file named _output_file_, or
          _stdout_ if _output_file_ not supplied.  _macros_ is a list
          of macro definitions as for compile(), which will
          augment the macros set with define_macro() and
          undefine_macro().  _include_dirs_ is a list of
          directory names that will be added to the  default list, in
          the same way as add_include_dir().

          Raises `PreprocessError' on failure.

     The following utility methods are defined by the CCompiler
     class, for use by the various concrete subclasses.

      -- Method: executable_filename (basename[, strip_dir=0,
               output_dir="])
          Returns the filename of the executable for the given
          _basename_.  Typically for non-Windows platforms this is the
          same as the basename,  while Windows will get a `.exe' added.

      -- Method: library_filename (libname[, lib_type='static',
               strip_dir=0, output_dir="])
          Returns the filename for the given library name on the
          current platform. On Unix a library with _lib_type_ of
          `'static'' will typically  be of the form `liblibname.a',
          while a _lib_type_ of `'dynamic''  will be of the form
          `liblibname.so'.

      -- Method: object_filenames (source_filenames[, strip_dir=0,
               output_dir="])
          Returns the name of the object files for the given source
          files.  _source_filenames_ should be a list of filenames.

      -- Method: shared_object_filename (basename[, strip_dir=0,
               output_dir="])
          Returns the name of a shared object file for the given file
          name _basename_.

      -- Method: execute (func, args[, msg=None, level=1])
          Invokes distutils.util.execute(). This method
          invokes a  Python function _func_ with the given arguments
          _args_, after  logging and taking into account the _dry_run_
          flag. XXX see also.

      -- Method: spawn (cmd)
          Invokes `distutils.util.spawn()'. This invokes an external
          process to run the given command. XXX see also.

      -- Method: mkpath (name[, mode=511])
          Invokes distutils.dir_util.mkpath(). This creates
          a directory  and any missing ancestor directories. XXX see
          also.

      -- Method: move_file (src, dst)
          Invokes distutils.file_util.move_file(). Renames
          _src_ to  _dst_.  XXX see also.

      -- Method: announce (msg[, level=1])
          Write a message using `distutils.log.debug()'. XXX see also.

      -- Method: warn (msg)
          Write a warning message _msg_ to standard error.

      -- Method: debug_print (msg)
          If the _debug_ flag is set on this CCompiler.
          instance, print  _msg_ to standard output, otherwise do
          nothing.

8.11.3 `distutils.unixccompiler' -- Unix C Compiler
---------------------------------------------------

This module provides the `UnixCCompiler' class, a subclass of
`CCompiler' that handles the typical Unix-style command-line  C
compiler:

   * macros defined with `-Dname[=value]'

   * macros undefined with `-Uname'

   * include search directories specified with `-Idir'

   * libraries specified with `-llib'

   * library search directories specified with `-Ldir'

   * compile handled by *cc* (or similar) executable with -c.
     option: compiles `.c' to `.o'

   * link static library handled by *ar* command (possibly with
     *ranlib*)

   * link shared library handled by *cc* `-shared'

8.11.4 `distutils.msvccompiler' -- Microsoft Compiler
-----------------------------------------------------

This module provides `MSVCCompiler', an implementation of the abstract
`CCompiler' class for Microsoft Visual Studio. Typically, extension
modules need to be compiled with the same compiler that was used to
compile Python. For Python 2.3 and earlier, the compiler was Visual
Studio 6. For Python 2.4 and 2.5, the compiler is Visual Studio .NET
2003. The AMD64 and Itanium binaries are created using the Platform SDK.

  `MSVCCompiler' will normally choose the right compiler, linker etc. on
its own. To override this choice, the environment variables
_DISTUTILS_USE_SDK_ and _MSSdk_ must be both set. _MSSdk_ indicates
that the current environment has been setup by the SDK's `SetEnv.Cmd'
script, or that the environment variables had been registered when the
SDK was installed; _DISTUTILS_USE_SDK_ indicates that the distutils
user has made an explicit choice to override the compiler selection by
`MSVCCompiler'.

8.11.5 `distutils.bcppcompiler' -- Borland Compiler
---------------------------------------------------

This module provides `BorlandCCompiler', an subclass of the abstract
`CCompiler' class for the Borland C++ compiler.

8.11.6 `distutils.cygwincompiler' -- Cygwin Compiler
----------------------------------------------------

This module provides the `CygwinCCompiler' class, a subclass of
`UnixCCompiler' that handles the Cygwin port of the GNU C compiler to
Windows.  It also contains the Mingw32CCompiler class which handles the
mingw32 port of GCC (same as cygwin in no-cygwin mode).

8.11.7 `distutils.emxccompiler' -- OS/2 EMX Compiler
----------------------------------------------------

This module provides the EMXCCompiler class, a subclass of
`UnixCCompiler' that handles the EMX port of the GNU C compiler to OS/2.

8.11.8 `distutils.archive_util' --  Archiving utilities
-------------------------------------------------------

This module provides a few functions for creating archive files, such as
tarballs or zipfiles.

 -- Function: distutils.archive_util.make_archive (base_name, format[,
          root_dir=None, base_dir=None, verbose=0, dry_run=0])
     Create an archive file (eg. `zip' or `tar').  _base_name_  is the
     name of the file to create, minus any format-specific extension;
     _format_ is the archive format: one of `zip', `tar',  `ztar', or
     `gztar'. _root_dir_ is a directory that will be the root directory
     of the archive; ie. we typically `chdir' into _root_dir_ before
     creating the archive.  _base_dir_ is the directory where we start
     archiving from; ie. _base_dir_ will be the common prefix of all
     files and directories in the archive.  _root_dir_ and _base_dir_
     both default to the current directory.  Returns the name of the
     archive file.

 -- Function: distutils.archive_util.make_tarball (base_name,
          base_dir[, compress='gzip', verbose=0, dry_run=0])
     'Create an (optional compressed) archive as a tar file from all
     files in and under _base_dir_. _compress_ must be `'gzip'' (the
     default),  `'compress'', `'bzip2'', or `None'.  Both *tar* and the
     compression utility named by _compress_ must be on the  default
     program search path, so this is probably Unix-specific.  The
     output tar file will be named `base_dir.tar', possibly plus the
     appropriate compression extension (`.gz', `.bz2' or `.Z').  Return
     the output filename.

 -- Function: distutils.archive_util.make_zipfile (base_name,
          base_dir[, verbose=0, dry_run=0])
     Create a zip file from all files in and under _base_dir_.  The
     output zip file will be named _base_dir_ + `.zip'.  Uses either
     the  zipfile. Python module (if available) or the
     InfoZIP `zip'  utility (if installed and found on the default
     search path).  If neither  tool is available, raises
     `DistutilsExecError'.   Returns the name of the output zip file.

8.11.9 `distutils.dep_util' -- Dependency checking
--------------------------------------------------

This module provides functions for performing simple, timestamp-based
dependency of files and groups of files; also, functions based entirely
on such timestamp dependency analysis.

 -- Function: distutils.dep_util.newer (source, target)
     Return true if _source_ exists and is more recently modified than
     _target_, or if _source_ exists and _target_ doesn't. Return false
     if both exist and _target_ is the same age or newer  than
     _source_. Raise `DistutilsFileError' if _source_ does not exist.

 -- Function: distutils.dep_util.newer_pairwise (sources, targets)
     Walk two filename lists in parallel, testing if each source is
     newer than its corresponding target.  Return a pair of lists
     (_sources_, _targets_) where source is newer than target,
     according to the semantics of newer().

 -- Function: distutils.dep_util.newer_group (sources, target[,
          missing='error'])
     Return true if _target_ is out-of-date with respect to any file
     listed in _sources_  In other words, if _target_ exists and is
     newer than every file in _sources_, return false; otherwise return
     true. _missing_ controls what we do when a source file is missing;
     the default (`'error'') is to blow up with an OSError.
     from  inside os.stat().; if it is `'ignore'', we
     silently drop any missing source files; if it is `'newer'', any
     missing source files make us assume that _target_ is out-of-date
     (this is handy in "dry-run" mode: it'll make you pretend to carry
     out commands that wouldn't work because inputs are missing, but
     that doesn't matter because you're not actually going to run the
     commands).

8.11.10 `distutils.dir_util' -- Directory tree operations
---------------------------------------------------------

This module provides functions for operating on directories and trees of
directories.

 -- Function: distutils.dir_util.mkpath (name[, mode=0777, verbose=0,
          dry_run=0])
     Create a directory and any missing ancestor directories.  If the
     directory already exists (or if _name_ is the empty string, which
     means the current directory, which of course exists), then do
     nothing.  Raise `DistutilsFileError' if unable to create some
     directory along the way (eg.  some sub-path exists, but is a file
     rather than a directory).  If _verbose_ is true, print a one-line
     summary of each mkdir to stdout.  Return the list of directories
     actually created.

 -- Function: distutils.dir_util.create_tree (base_dir, files[,
          mode=0777, verbose=0, dry_run=0])
     Create all the empty directories under _base_dir_ needed to put
     _files_ there.  _base_dir_ is just the a name of a directory which
     doesn't necessarily exist yet; _files_ is a list of filenames to
     be interpreted relative to _base_dir_.  _base_dir_ + the directory
     portion of every file in _files_ will be created if it doesn't
     already exist.  _mode_, _verbose_ and _dry_run_ flags  are as for
     mkpath().

 -- Function: distutils.dir_util.copy_tree (src, dst[, preserve_mode=1,
          preserve_times=1, preserve_symlinks=0, update=0, verbose=0,
          dry_run=0])
     Copy an entire directory tree _src_ to a new location _dst_.  Both
     _src_ and _dst_ must be directory names.  If _src_ is not a
     directory, raise `DistutilsFileError'.  If _dst_ does  not exist,
     it is created with mkpath().  The end result of the
     copy is that every file in _src_ is copied to _dst_, and
     directories under _src_ are recursively copied to _dst_.  Return
     the list of files that were copied or might have been copied,
     using their output name. The return value is unaffected by
     _update_ or _dry_run_: it is simply the list of all files under
     _src_, with the names changed to be under _dst_.

     _preserve_mode_ and _preserve_times_ are the same as for
     `copy_file()' in distutils.file_util.; note that they
     only apply to regular files, not to directories.  If
     _preserve_symlinks_ is true, symlinks will be copied as symlinks
     (on platforms that support them!); otherwise (the default), the
     destination of the symlink will be copied.  _update_ and _verbose_
     are the same as for `copy_file()'.

 -- Function: distutils.dir_util.remove_tree (directory[, verbose=0,
          dry_run=0])
     Recursively remove _directory_ and all files and directories
     underneath it. Any errors are ignored (apart from being reported
     to `sys.stdout' if _verbose_ is true).

8.11.11 `distutils.file_util' -- Single file operations
-------------------------------------------------------

This module contains some utility functions for operating on individual
files.

 -- Function: distutils.file_util.copy_file (src, dst[,
          preserve_mode=1, preserve_times=1, update=0, link=None,
          verbose=0, dry_run=0])
     Copy file _src_ to _dst_. If _dst_ is a directory, then _src_ is
     copied there with the same name; otherwise, it must be a filename.
     (If the file exists, it will be ruthlessly clobbered.) If
     _preserve_mode_ is true (the default), the file's mode (type and
     permission bits, or whatever is analogous on the current platform)
     is copied. If _preserve_times_ is true (the default), the
     last-modified and last-access times are copied as well. If
     _update_ is true, _src_ will only be copied if _dst_ does not
     exist, or if _dst_ does exist but is older than _src_.

     _link_ allows you to make hard links (using os.link())
     or symbolic links (using os.symlink()) instead
     of copying: set it to `'hard'' or `'sym''; if it is `None' (the
     default), files are copied. Don't set _link_ on systems that don't
     support it: copy_file() doesn't check if hard or
     symbolic linking is available.  It uses `_copy_file_contents()' to
     copy file contents.

     Return a tuple `(dest_name, copied)': _dest_name_ is the actual
     name of the output file, and _copied_ is true if the file was
     copied  (or would have been copied, if _dry_run_ true).

 -- Function: distutils.file_util.move_file (src, dst[, verbose,
          dry_run])
     Move file _src_ to _dst_. If _dst_ is a directory, the file will
     be moved into it with the same name; otherwise, _src_ is just
     renamed to _dst_.  Returns the new full name of the file.

          Warning: Handles cross-device moves on Unix using
          copy_file().  What about other systems?

 -- Function: distutils.file_util.write_file (filename, contents)
     Create a file called _filename_ and write _contents_ (a sequence
     of strings without line terminators) to it.

8.11.12 `distutils.util' -- Miscellaneous other utility functions
-----------------------------------------------------------------

This module contains other assorted bits and pieces that don't fit into
any other utility module.

 -- Function: distutils.util.get_platform ()
     Return a string that identifies the current platform.  This is
     used mainly to distinguish platform-specific build directories and
     platform-specific built distributions.  Typically includes the OS
     name and version and the architecture (as supplied by
     'os.uname()'), although the exact information included depends on
     the OS; eg. for IRIX the architecture isn't particularly important
     (IRIX only runs on SGI hardware), but for Linux the kernel version
     isn't particularly important.

     Examples of returned values:

        * `linux-i586'

        * `linux-alpha'

        * `solaris-2.6-sun4u'

        * `irix-5.3'

        * `irix64-6.2'

     For non-POSIX platforms, currently just returns `sys.platform'.

     For Mac OS X systems the OS version reflects the minimal version
     on which binaries will run (that is, the value of
     `MACOSX_DEPLOYMENT_TARGET' during the build of Python), not the OS
     version of the current system.

     For universal binary builds on Mac OS X the architecture value
     reflects the univeral binary status instead of the architecture of
     the current processor. For 32-bit universal binaries the
     architecture is `fat', for 64-bit universal binaries the
     architecture is `fat64', and for 4-way universal binaries the
     architecture is `universal'. Starting from Python 2.7 and Python
     3.2 the architecture `fat3' is used for a 3-way universal build
     (ppc, i386, x86_64) and `intel' is used for a univeral build with
     the i386 and x86_64 architectures

     Examples of returned values on Mac OS X:

        * `macosx-10.3-ppc'

        * `macosx-10.3-fat'

        * `macosx-10.5-universal'

        * `macosx-10.6-intel'

 -- Function: distutils.util.convert_path (pathname)
     Return 'pathname' as a name that will work on the native
     filesystem, i.e. split it on '/' and put it back together again
     using the current directory separator.  Needed because filenames
     in the setup script are always supplied in Unix style, and have to
     be converted to the local convention before we can actually use
     them in the filesystem.  Raises ValueError on
     non-Unix-ish systems if _pathname_ either  starts or ends with a
     slash.

 -- Function: distutils.util.change_root (new_root, pathname)
     Return _pathname_ with _new_root_ prepended.  If _pathname_ is
     relative, this is equivalent to `os.path.join(new_root,pathname)'
     Otherwise, it requires making _pathname_ relative and then joining
     the two, which is tricky on DOS/Windows.

 -- Function: distutils.util.check_environ ()
     Ensure that 'os.environ' has all the environment variables we
     guarantee that users can use in config files, command-line
     options, etc.  Currently this includes:

        * `HOME' - user's home directory (Unix only)

        * `PLAT' - description of the current platform, including
          hardware and OS (see get_platform())

 -- Function: distutils.util.subst_vars (s, local_vars)
     Perform shell/Perl-style variable substitution on _s_.  Every
     occurrence of `$' followed by a name is considered a variable, and
     variable is substituted by the value found in the _local_vars_
     dictionary, or in `os.environ' if it's not in _local_vars_.
     _os.environ_ is first checked/augmented to guarantee that it
     contains certain values: see check_environ().  Raise
     ValueError.  for any variables not found in either
     _local_vars_ or `os.environ'.

     Note that this is not a fully-fledged string interpolation
     function. A valid `$variable' can consist only of upper and lower
     case letters, numbers and an underscore. No { } or ( ) style
     quoting is available.

 -- Function: distutils.util.grok_environment_error (exc[,
          prefix='error: '])
     Generate a useful error message from an EnvironmentError.
     (see IOError.  or OSError) exception
     object.   Handles Python 1.5.1 and later styles, and does what it
     can to deal with  exception objects that don't have a filename
     (which happens when the error  is due to a two-file operation,
     such as `rename()' or  `link()').  Returns the error message as a
     string prefixed  with _prefix_.

 -- Function: distutils.util.split_quoted (s)
     Split a string up according to Unix shell-like rules for quotes
     and backslashes.  In short: words are delimited by spaces, as long
     as those spaces are not escaped by a backslash, or inside a quoted
     string. Single and double quotes are equivalent, and the quote
     characters can be backslash-escaped.  The backslash is stripped
     from any two-character escape sequence, leaving only the escaped
     character.  The quote characters are stripped from any quoted
     string.  Returns a list of words.

 -- Function: distutils.util.execute (func, args[, msg=None, verbose=0,
          dry_run=0])
     Perform some action that affects the outside world (for instance,
     writing to the filesystem).  Such actions are special because they
     are disabled by the _dry_run_ flag.  This method takes  care of
     all that bureaucracy for you; all you have to do is supply the
     function to call and an argument tuple for it (to embody the
     "external action" being performed), and an optional message to
     print.

 -- Function: distutils.util.strtobool (val)
     Convert a string representation of truth to true (1) or false (0).

     True values are `y', `yes', `t', `true', `on'  and `1'; false
     values are `n', `no', `f', `false',  `off' and `0'.  Raises
     ValueError if _val_  is anything else.

 -- Function: distutils.util.byte_compile (py_files[, optimize=0,
          force=0, prefix=None, base_dir=None, verbose=1, dry_run=0,
          direct=None])
     Byte-compile a collection of Python source files to either `.pyc'
     or `.pyo' files in the same directory.  _py_files_ is a list of
     files to compile; any files that don't end in `.py' are silently
     skipped.  _optimize_ must be one of the following:

        * `0' - don't optimize (generate `.pyc')

        * `1' - normal optimization (like `python -O')

        * `2' - extra optimization (like `python -OO')

     If _force_ is true, all files are recompiled regardless of
     timestamps.

     The source filename encoded in each bytecode file
     defaults to the filenames listed in _py_files_; you can modify
     these with _prefix_ and _basedir_.  _prefix_ is a string that will
     be stripped off of each source filename, and _base_dir_ is a
     directory name that will be prepended (after _prefix_ is
     stripped).  You can supply either or both (or neither) of _prefix_
     and _base_dir_, as you wish.

     If _dry_run_ is true, doesn't actually do anything that would
     affect the filesystem.

     Byte-compilation is either done directly in this interpreter
     process with the standard py_compile module, or
     indirectly by writing a temporary script and executing it.
     Normally, you should let byte_compile() figure out to
     use direct compilation or not (see the source for details).  The
     _direct_ flag is used by the script generated in indirect mode;
     unless you know what you're doing, leave it set to `None'.

 -- Function: distutils.util.rfc822_escape (header)
     Return a version of _header_ escaped for inclusion in an RFC 822
     (http://tools.ietf.org/html/rfc822.html) header, by ensuring there
     are 8 spaces space after each newline. Note that it does no other
     modification of the string.

8.11.13 `distutils.dist' -- The Distribution class
--------------------------------------------------

This module provides the `Distribution' class, which represents the
module distribution being built/installed/distributed.

8.11.14 `distutils.extension' -- The Extension class
----------------------------------------------------

This module provides the `Extension' class, used to describe C/C++
extension modules in setup scripts.

8.11.15 `distutils.debug' -- Distutils debug mode
-------------------------------------------------

This module provides the DEBUG flag.

8.11.16 `distutils.errors' -- Distutils exceptions
--------------------------------------------------

Provides exceptions used by the Distutils modules.  Note that Distutils
modules may raise standard exceptions; in particular, SystemExit is
usually raised for errors that are obviously the end-user's fault (eg.
bad command-line arguments).

  This module is safe to use in `from ... import *' mode; it only
exports symbols whose names start with `Distutils' and end with `Error'.

8.11.17 `distutils.fancy_getopt' -- Wrapper around the standard getopt module
-----------------------------------------------------------------------------

This module provides a wrapper around the standard getopt.
module that provides the following additional features:

   * short and long options are tied together

   * options have help strings, so fancy_getopt() could
     potentially  create a complete usage summary

   * options set attributes of a passed-in object

   * boolean options can have "negative aliases" -- eg. if `--quiet' is
     the "negative alias" of `--verbose', then `--quiet' on the command
     line sets _verbose_ to false.

 -- Function: distutils.fancy_getopt.fancy_getopt (options,
          negative_opt, object, args)
     Wrapper function. _options_ is a list of `(long_option,
     short_option, help_string)' 3-tuples as described in the
     constructor for FancyGetopt. _negative_opt_ should be
     a dictionary mapping option names to option names, both the key
     and value should be in the _options_ list.  _object_ is an object
     which will be used to store values (see the getopt().
     method of the FancyGetopt class). _args_ is the
     argument list. Will use `sys.argv[1:]' if you  pass `None' as
     _args_.

 -- Function: distutils.fancy_getopt.wrap_text (text, width)
     Wraps _text_ to less than _width_ wide.

 -- Class: distutils.fancy_getopt.FancyGetopt ([option_table=None])
     The option_table is a list of 3-tuples: `(long_option,
     short_option, help_string)'

     If an option takes an argument, its _long_option_ should have
     `'='' appended; _short_option_ should just be a single character,
     no `':'' in any case.  _short_option_ should be `None' if a
     _long_option_  doesn't have a corresponding _short_option_. All
     option tuples must have long options.

  The FancyGetopt class provides the following methods:

 -- Method: FancyGetopt.getopt ([args=None, object=None])
     Parse command-line options in args. Store as attributes on
     _object_.

     If _args_ is `None' or not supplied, uses `sys.argv[1:]'.  If
     _object_ is `None' or not supplied, creates a new `OptionDummy'
     instance, stores option values there, and returns a tuple `(args,
     object)'.  If _object_ is supplied, it is modified in place and
     getopt() just returns _args_; in both cases, the
     returned _args_ is a modified copy of the passed-in _args_ list,
     which is left untouched.

 -- Method: FancyGetopt.get_option_order ()
     Returns the list of `(option, value)' tuples processed by the
     previous run of getopt().  Raises RuntimeError
     if getopt() hasn't been called yet.

 -- Method: FancyGetopt.generate_help ([header=None])
     Generate help text (a list of strings, one per suggested line of
     output) from the option table for this FancyGetopt.
     object.

     If supplied, prints the supplied _header_ at the top of the help.

8.11.18 `distutils.filelist' -- The FileList class
--------------------------------------------------

This module provides the `FileList' class, used for poking about the
filesystem and building lists of files.

8.11.19 `distutils.log' -- Simple PEP 282-style logging
-------------------------------------------------------

8.11.20 `distutils.spawn' -- Spawn a sub-process
------------------------------------------------

This module provides the `spawn()' function, a front-end to  various
platform-specific functions for launching another program in a
sub-process.  Also provides `find_executable()' to search the path for
a given executable name.

8.11.21 `distutils.sysconfig' -- System configuration information
-----------------------------------------------------------------

The distutils.sysconfig module provides access to Python's
low-level configuration information.  The specific configuration
variables available depend heavily on the platform and configuration.
The specific variables depend on the build process for the specific
version of Python being run; the variables are those found in the
`Makefile' and configuration header that are installed with Python on
Unix systems.  The configuration header is called `pyconfig.h' for
Python versions starting with 2.2, and `config.h' for earlier versions
of Python.

  Some additional functions are provided which perform some useful
manipulations for other parts of the distutils package.

 -- Data: distutils.sysconfig.PREFIX
     The result of `os.path.normpath(sys.prefix)'.

 -- Data: distutils.sysconfig.EXEC_PREFIX
     The result of `os.path.normpath(sys.exec_prefix)'.

 -- Function: distutils.sysconfig.get_config_var (name)
     Return the value of a single variable.  This is equivalent to
     `get_config_vars().get(name)'.

 -- Function: distutils.sysconfig.get_config_vars (...)
     Return a set of variable definitions.  If there are no arguments,
     this returns a dictionary mapping names of configuration variables
     to values.  If arguments are provided, they should be strings, and
     the return value will be a sequence giving the associated values.
     If a given name does not have a corresponding value, `None' will
     be included for that variable.

 -- Function: distutils.sysconfig.get_config_h_filename ()
     Return the full path name of the configuration header.  For Unix,
     this will be the header generated by the *configure* script; for
     other platforms the header will have been supplied directly by the
     Python source distribution.  The file is a platform-specific text
     file.

 -- Function: distutils.sysconfig.get_makefile_filename ()
     Return the full path name of the `Makefile' used to build Python.
     For Unix, this will be a file generated by the *configure* script;
     the meaning for other platforms will vary.  The file is a
     platform-specific text file, if it exists. This function is only
     useful on POSIX platforms.

 -- Function: distutils.sysconfig.get_python_inc ([plat_specific[,
          prefix]])
     Return the directory for either the general or platform-dependent
     C include files.  If _plat_specific_ is true, the
     platform-dependent include directory is returned; if false or
     omitted, the platform-independent directory is returned.  If
     _prefix_ is given, it is used as either the prefix instead of
     PREFIX, or as the exec-prefix instead of
     EXEC_PREFIX if _plat_specific_ is true.

 -- Function: distutils.sysconfig.get_python_lib ([plat_specific[,
          standard_lib[, prefix]]])
     Return the directory for either the general or platform-dependent
     library installation.  If _plat_specific_ is true, the
     platform-dependent include directory is returned; if false or
     omitted, the platform-independent directory is returned.  If
     _prefix_ is given, it is used as either the prefix instead of
     PREFIX, or as the exec-prefix instead of
     EXEC_PREFIX if _plat_specific_ is true.  If _standard_lib_
     is true, the directory for the standard library is returned rather
     than the directory for the installation of third-party extensions.

  The following function is only intended for use within the
distutils.  package.

 -- Function: distutils.sysconfig.customize_compiler (compiler)
     Do any platform-specific customization of a
     distutils.ccompiler.CCompiler instance.

     This function is only needed on Unix at this time, but should be
     called consistently to support forward-compatibility.  It inserts
     the information that varies across Unix flavors and is stored in
     Python's `Makefile'.  This information includes the selected
     compiler, compiler and linker options, and the extension used by
     the linker for shared objects.

  This function is even more special-purpose, and should only be used
from Python's own build procedures.

 -- Function: distutils.sysconfig.set_python_build ()
     Inform the distutils.sysconfig module that it is being
     used as part of the build process for Python.  This changes a lot
     of relative locations for files, allowing them to be located in
     the build area rather than in an installed Python.

8.11.22 `distutils.text_file' -- The TextFile class
---------------------------------------------------

This module provides the TextFile class, which gives an
interface  to text files that (optionally) takes care of stripping
comments, ignoring  blank lines, and joining lines with backslashes.

 -- Class: distutils.text_file.TextFile ([filename=None, file=None,
          **options])
     This class provides a file-like object that takes care of all  the
     things you commonly want to do when processing a text file  that
     has some line-by-line syntax: strip comments (as long as `#'  is
     your comment character), skip blank lines, join adjacent lines by
     escaping the newline (ie. backslash at end of line), strip leading
     and/or trailing whitespace.  All of these are optional and
     independently controllable.

     The class provides a warn() method so you can
     generate  warning messages that report physical line number, even
     if the  logical line in question spans multiple physical lines.
     Also  provides unreadline() for implementing
     line-at-a-time lookahead.

     TextFile instances are create with either _filename_,
     _file_, or both.  RuntimeError is raised if both are
     `None'. _filename_ should be a string, and _file_ a file object
     (or something that provides readline().  and
     close().  methods).  It is recommended that you supply at
     least _filename_,  so that TextFile can include it in
     warning messages.  If _file_ is not supplied, see TextFile
     creates its own using the open() built-in
     function.

     The options are all boolean, and affect the values returned by
     readline().

     option name            description                          default
     --------------------------------------------------------------------------
     _strip_comments_       strip from `'#'' to end-of- line,    true
                            as well as any whitespace leading
                            up to the `'#''--unless it is
                            escaped by a backslash
     _lstrip_ws_            strip leading whitespace from each   false
                            line before returning it
     _rstrip_ws_            strip trailing whitespace            true
                            (including line terminator!)  from
                            each line before returning it.
     _skip_blanks_          skip lines that are empty *after*    true
                            stripping comments and whitespace.
                            (If both lstrip_ws and rstrip_ws are
                            false, then some lines may consist
                            of solely whitespace: these will
                            *not* be skipped, even if
                            _skip_blanks_ is true.)
     _join_lines_           if a backslash is the last           false
                            non-newline character on a line
                            after stripping comments and
                            whitespace, join the following line
                            to it to form one logical line; if N
                            consecutive lines end with a
                            backslash, then N+1 physical lines
                            will be joined to form one logical
                            line.
     _collapse_join_        strip leading whitespace from lines  false
                            that are joined to their
                            predecessor; only matters if
                            `(join_lines and not lstrip_ws)'

     Note that since _rstrip_ws_ can strip the trailing newline, the
     semantics of readline() must differ from those of the
     built-in file object's readline() method!  In
     particular, see readline().  returns `None' for
     end-of-file: an empty string might just be a  blank line (or an
     all-whitespace line), if _rstrip_ws_ is true  but _skip_blanks_ is
     not.

      -- Method: open (filename)
          Open a new file _filename_.  This overrides any _file_ or
          _filename_ constructor arguments.

      -- Method: close ()
          Close the current file and forget everything we know about it
          (including the filename and the current line number).

      -- Method: warn (msg[, line=None])
          Print (to stderr) a warning message tied to the current
          logical line in the current file.  If the current logical
          line in the file spans multiple physical lines, the warning
          refers to the whole range, such as `"lines 3-5"'.  If _line_
          is supplied,  it overrides the current line number; it may be
          a list or tuple  to indicate a range of physical lines, or an
          integer for a  single physical line.

      -- Method: readline ()
          Read and return a single logical line from the current file
          (or from an internal buffer if lines have previously been
          "unread" with unreadline()).  If the
          _join_lines_ option  is true, this may involve reading
          multiple physical lines concatenated into a single string.
          Updates the current line number,  so calling warn()
          after readline() emits a warning  about the
          physical line(s) just read.  Returns `None' on end-of-file,
          since the empty string can occur if _rstrip_ws_ is true but
          _strip_blanks_ is not.

      -- Method: readlines ()
          Read and return the list of all logical lines remaining in
          the current file.  This updates the current line number to
          the last line of the file.

      -- Method: unreadline (line)
          Push _line_ (a string) onto an internal buffer that will be
          checked by future readline() calls.  Handy for
          implementing a parser with line-at-a-time lookahead. Note
          that lines that are "unread" with unreadline().
          are not subsequently re-cleansed (whitespace  stripped, or
          whatever) when read with readline(). If multiple
          calls are made to unreadline() before a call to
          readline(), the lines will be returned most in
          most recent first order.

8.11.23 `distutils.version' -- Version number classes
-----------------------------------------------------

8.11.24 `distutils.cmd' -- Abstract base class for Distutils commands
---------------------------------------------------------------------

This module supplies the abstract base class Command.

 -- Class: distutils.cmd.Command (dist)
     Abstract base class for defining command classes, the "worker
     bees" of the Distutils.  A useful analogy for command classes is
     to think of them as subroutines with local variables called
     _options_.  The options are declared in `initialize_options()' and
     defined (given their final values) in `finalize_options()', both
     of which must be defined by every command class.  The distinction
     between the two is necessary because option values might come from
     the outside world (command line, config file, ...), and any options
     dependent on other options must be computed after these outside
     influences have been processed -- hence `finalize_options()'.  The
     body of the subroutine, where it does all its work based on the
     values of its options, is the `run()' method, which must also be
     implemented by every command class.

     The class constructor takes a single argument _dist_, a
     `Distribution' instance.

8.11.25 `distutils.command' -- Individual Distutils commands
------------------------------------------------------------

8.11.26 `distutils.command.bdist' -- Build a binary installer
-------------------------------------------------------------

8.11.27 `distutils.command.bdist_packager' -- Abstract base class for packagers
-------------------------------------------------------------------------------

8.11.28 `distutils.command.bdist_dumb' -- Build a "dumb" installer
------------------------------------------------------------------

8.11.29 `distutils.command.bdist_msi' -- Build a Microsoft Installer binary package
-----------------------------------------------------------------------------------

 -- Class: distutils.command.bdist_msi.bdist_msi (Command)
     Builds a Windows Installer
     (http://msdn.microsoft.com/en-us/library/cc185688(VS.85).aspx)
     (.msi) binary package.

     In most cases, the `bdist_msi' installer is a better choice than
     the `bdist_wininst' installer, because it provides better support
     for Win64 platforms, allows administrators to perform
     non-interactive installations, and allows installation through
     group policies.

8.11.30 `distutils.command.bdist_rpm' -- Build a binary distribution as a Redhat RPM and SRPM
---------------------------------------------------------------------------------------------

8.11.31 `distutils.command.bdist_wininst' -- Build a Windows installer
----------------------------------------------------------------------

8.11.32 `distutils.command.sdist' -- Build a source distribution
----------------------------------------------------------------

8.11.33 `distutils.command.build' -- Build all files of a package
-----------------------------------------------------------------

8.11.34 `distutils.command.build_clib' -- Build any C libraries in a package
----------------------------------------------------------------------------

8.11.35 `distutils.command.build_ext' -- Build any extensions in a package
--------------------------------------------------------------------------

8.11.36 `distutils.command.build_py' -- Build the .py/.pyc files of a package
-----------------------------------------------------------------------------

8.11.37 `distutils.command.build_scripts' -- Build the scripts of a package
---------------------------------------------------------------------------

8.11.38 `distutils.command.clean' -- Clean a package build area
---------------------------------------------------------------

8.11.39 `distutils.command.config' -- Perform package configuration
-------------------------------------------------------------------

8.11.40 `distutils.command.install' -- Install a package
--------------------------------------------------------

8.11.41 `distutils.command.install_data' -- Install data files from a package
-----------------------------------------------------------------------------

8.11.42 `distutils.command.install_headers' -- Install C/C++ header files from a package
----------------------------------------------------------------------------------------

8.11.43 `distutils.command.install_lib' -- Install library files from a package
-------------------------------------------------------------------------------

8.11.44 `distutils.command.install_scripts' -- Install script files from a package
----------------------------------------------------------------------------------

8.11.45 `distutils.command.register' -- Register a module with the Python Package Index
---------------------------------------------------------------------------------------

The `register' command registers the package with the Python Package
Index.  This is described in more detail in PEP 301
(http://www.python.org/dev/peps/pep-0301).

8.11.46 Creating a new Distutils command
----------------------------------------

This section outlines the steps to create a new Distutils command.

  A new command lives in a module in the distutils.command.
package. There is a sample template in that directory called
`command_template'. Copy this file to a new module with the same name
as the new command you're implementing. This module should implement a
class with the same name as the module (and the command). So, for
instance, to create the command `peel_banana' (so that users can run
`setup.py peel_banana'), you'd copy `command_template'  to
`distutils/command/peel_banana.py', then edit it so that it's
implementing the class `peel_banana', a subclass of
distutils.cmd.Command.

  Subclasses of `Command' must define the following methods.

 -- Method: Command.initialize_options ()
     Set default values for all the options that this command supports.
     Note that these defaults may be overridden by other commands, by
     the setup script, by config files, or by the command-line.  Thus,
     this is not the place to code dependencies between options;
     generally, see initialize_options().  implementations are
     just a bunch of `self.foo = None' assignments.

 -- Method: Command.finalize_options ()
     Set final values for all the options that this command supports.
     This is always called as late as possible, ie.  after any option
     assignments from the command-line or from other commands have been
     done.  Thus, this is the place to to code option dependencies: if
     _foo_ depends on _bar_, then it is safe to set _foo_ from _bar_ as
     long as _foo_ still has the same value it was assigned in
     initialize_options().

 -- Method: Command.run ()
     A command's raison d'etre: carry out the action it exists to
     perform, controlled by the options initialized in
     initialize_options(), customized by other commands, the
     setup script, the command-line, and config files, and finalized in
     finalize_options().  All terminal output and
     filesystem interaction should be done by run().

  _sub_commands_ formalizes the notion of a "family" of commands, eg.
`install' as the parent with sub-commands `install_lib',
`install_headers', etc.  The parent of a family of commands defines
_sub_commands_ as a class attribute; it's a list of 2-tuples
`(command_name, predicate)', with _command_name_ a string and
_predicate_ an unbound method, a string or None. _predicate_ is a
method of the parent command that determines whether the corresponding
command is applicable in the current situation.  (Eg. we
`install_headers' is only applicable if we have any C header files to
install.)  If _predicate_ is None, that command is always applicable.

  _sub_commands_ is usually defined at the *end* of a class, because
predicates can be unbound methods, so they must already have been
defined.  The canonical example is the *install* command.

9 Installing Python Modules
***************************

     Author: Greg Ward

     Release: 2.7

     Date: February 08, 2011

Abstract
--------

     This document describes the Python Distribution Utilities
     ("Distutils") from the end-user's point-of-view, describing how to
     extend the capabilities of a standard Python installation by
     building and installing third-party Python modules and extensions.

9.1 Introduction
================

Although Python's extensive standard library covers many programming
needs, there often comes a time when you need to add some new
functionality to your Python installation in the form of third-party
modules.  This might be necessary to support your own programming, or
to support an application that you want to use and that happens to be
written in Python.

  In the past, there has been little support for adding third-party
modules to an existing Python installation.  With the introduction of
the Python Distribution Utilities (Distutils for short) in Python 2.0,
this changed.

  This document is aimed primarily at the people who need to install
third-party Python modules: end-users and system administrators who
just need to get some Python application running, and existing Python
programmers who want to add some new goodies to their toolbox.  You
don't need to know Python to read this document; there will be some
brief forays into using Python's interactive mode to explore your
installation, but that's it.  If you're looking for information on how
to distribute your own Python modules so that others may use them, see
the Distributing Python Modules manual.

9.1.1 Best case: trivial installation
-------------------------------------

In the best case, someone will have prepared a special version of the
module distribution you want to install that is targeted specifically
at your platform and is installed just like any other software on your
platform.  For example, the module developer might make an executable
installer available for Windows users, an RPM package for users of
RPM-based Linux systems (Red Hat, SuSE, Mandrake, and many others), a
Debian package for users of Debian-based Linux systems, and so forth.

  In that case, you would download the installer appropriate to your
platform and do the obvious thing with it: run it if it's an executable
installer, `rpm --install' it if it's an RPM, etc.  You don't need to
run Python or a setup script, you don't need to compile anything--you
might not even need to read any instructions (although it's always a
good idea to do so anyways).

  Of course, things will not always be that easy.  You might be
interested in a module distribution that doesn't have an easy-to-use
installer for your platform.  In that case, you'll have to start with
the source distribution released by the module's author/maintainer.
Installing from a source distribution is not too hard, as long as the
modules are packaged in the standard way.  The bulk of this document is
about building and installing modules from standard source
distributions.

9.1.2 The new standard: Distutils
---------------------------------

If you download a module source distribution, you can tell pretty
quickly if it was packaged and distributed in the standard way, i.e.
using the Distutils.  First, the distribution's name and version number
will be featured prominently in the name of the downloaded archive,
e.g. `foo-1.0.tar.gz' or `widget-0.9.7.zip'.  Next, the archive will
unpack into a similarly-named directory: `foo-1.0' or `widget-0.9.7'.
Additionally, the distribution will contain a setup script `setup.py',
and a file named `README.txt' or possibly just `README', which should
explain that building and installing the module distribution is a
simple matter of running

    python setup.py install

If all these things are true, then you already know how to build and
install the modules you've just downloaded:  Run the command above.
Unless you need to install things in a non-standard way or customize
the build process, you don't really need this manual.  Or rather, the
above command is everything you need to get out of this manual.

9.2 Standard Build and Install
==============================

As described in section The new standard; Distutils,
building and installing a module distribution using the Distutils is
usually one simple command:

    python setup.py install

On Unix, you'd run this command from a shell prompt; on Windows, you
have to open a command prompt window ("DOS box") and do it there; on
Mac OS X, you open a *Terminal* window to get a shell prompt.

9.2.1 Platform variations
-------------------------

You should always run the setup command from the distribution root
directory, i.e. the top-level subdirectory that the module source
distribution unpacks into.  For example, if you've just downloaded a
module source distribution `foo-1.0.tar.gz' onto a Unix system, the
normal thing to do is:

    gunzip -c foo-1.0.tar.gz | tar xf -    # unpacks into directory foo-1.0
    cd foo-1.0
    python setup.py install

On Windows, you'd probably download `foo-1.0.zip'.  If you downloaded
the archive file to `C:\Temp', then it would unpack into
`C:\Temp\foo-1.0'; you can use either a archive manipulator with a
graphical user interface (such as WinZip) or a command-line tool (such
as *unzip* or *pkunzip*) to unpack the archive.  Then, open a command
prompt window ("DOS box"), and run:

    cd c:\Temp\foo-1.0
    python setup.py install

9.2.2 Splitting the job up
--------------------------

Running `setup.py install' builds and installs all modules in one run.
If you prefer to work incrementally--especially useful if you want to
customize the build process, or if things are going wrong--you can use
the setup script to do one thing at a time.  This is particularly
helpful when the build and install will be done by different users--for
example, you might want to build a module distribution and hand it off
to a system administrator for installation (or do it yourself, with
super-user privileges).

  For example, you can build everything in one step, and then install
everything in a second step, by invoking the setup script twice:

    python setup.py build
    python setup.py install

If you do this, you will notice that running the *install* command
first runs the *build* command, which--in this case--quickly notices
that it has nothing to do, since everything in the `build' directory is
up-to-date.

  You may not need this ability to break things down often if all you
do is install modules downloaded off the 'net, but it's very handy for
more advanced tasks.  If you get into distributing your own Python
modules and extensions, you'll run lots of individual Distutils
commands on their own.

9.2.3 How building works
------------------------

As implied above, the *build* command is responsible for putting the
files to install into a _build directory_.  By default, this is `build'
under the distribution root; if you're excessively concerned with
speed, or want to keep the source tree pristine, you can change the
build directory with the `--build-base' option. For example:

    python setup.py build --build-base=/tmp/pybuild/foo-1.0

(Or you could do this permanently with a directive in your system or
personal Distutils configuration file; see section Distutils Configuration Files)
Normally, this isn't necessary.

  The default layout for the build tree is as follows:

    --- build/ --- lib/
    or
    --- build/ --- lib.<plat>/
                   temp.<plat>/

where `<plat>' expands to a brief description of the current OS/hardware
platform and Python version.  The first form, with just a `lib'
directory, is used for "pure module distributions"--that is, module
distributions that include only pure Python modules.  If a module
distribution contains any extensions (modules written in C/C++), then
the second form, with two `<plat>' directories, is used.  In that case,
the `temp._plat_' directory holds temporary files generated by the
compile/link process that don't actually get installed.  In either
case, the `lib' (or `lib._plat_') directory contains all Python modules
(pure Python and extensions) that will be installed.

  In the future, more directories will be added to handle Python
scripts, documentation, binary executables, and whatever else is needed
to handle the job of installing Python modules and applications.

9.2.4 How installation works
----------------------------

After the *build* command runs (whether you run it explicitly, or the
*install* command does it for you), the work of the *install* command
is relatively simple: all it has to do is copy everything under
`build/lib' (or `build/lib._plat_') to your chosen installation
directory.

  If you don't choose an installation directory--i.e., if you just run
`setup.py install'--then the *install* command installs to the standard
location for third-party Python modules.  This location varies by
platform and by how you built/installed Python itself.  On Unix (and
Mac OS X, which is also Unix-based), it also depends on whether the
module distribution being installed is pure Python or contains
extensions ("non-pure"):

Platform              Standard installation location                            Default value                                          Notes
---------------------------------------------------------------------------------------------------------------------------------------------------
Unix (pure)           `_prefix_/lib/python_X.Y_/site-packages'                  `/usr/local/lib/python_X.Y_/site-packages'             (1)
Unix (non-pure)       `_exec-prefix_/lib/python_X.Y_/site-packages'             `/usr/local/lib/python_X.Y_/site-packages'             (1)
Windows               `_prefix_'                                                `C:\Python'                                            (2)

  Notes:

  1. Most Linux distributions include Python as a standard part of the
     system, so `_prefix_' and `_exec-prefix_' are usually both `/usr'
     on Linux.  If you build Python yourself on Linux (or any Unix-like
     system), the default `_prefix_' and `_exec-prefix_' are
     `/usr/local'.

  2. The default installation directory on Windows was `C:\Program
     Files\Python' under Python 1.6a1, 1.5.2, and earlier.

  `_prefix_' and `_exec-prefix_' stand for the directories that Python
is installed to, and where it finds its libraries at run-time.  They
are always the same under Windows, and very often the same under Unix
and Mac OS X.  You can find out what your Python installation uses for
`_prefix_' and `_exec-prefix_' by running Python in interactive mode
and typing a few simple commands. Under Unix, just type `python' at the
shell prompt.  Under Windows, choose _Start ‣ Programs ‣ Python X.Y ‣
Python (command line)_.   Once the interpreter is started, you type
Python code at the prompt.  For example, on my Linux system, I type the
three Python statements shown below, and get the output as shown, to
find out my `_prefix_' and `_exec-prefix_':

    Python 2.4 (#26, Aug  7 2004, 17:19:02)
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.prefix
    '/usr'
    >>> sys.exec_prefix
    '/usr'

If you don't want to install modules to the standard location, or if
you don't have permission to write there, then you need to read about
alternate installations in section Alternate Installation.
If you want to customize your installation directories more heavily,
see section Custom Installation on custom installations.

9.3 Alternate Installation
==========================

Often, it is necessary or desirable to install modules to a location
other than the standard location for third-party Python modules.  For
example, on a Unix system you might not have permission to write to the
standard third-party module directory.  Or you might wish to try out a
module before making it a standard part of your local Python
installation.  This is especially true when upgrading a distribution
already present: you want to make sure your existing base of scripts
still works with the new version before actually upgrading.

  The Distutils *install* command is designed to make installing module
distributions to an alternate location simple and painless.  The basic
idea is that you supply a base directory for the installation, and the
*install* command picks a set of directories (called an _installation
scheme_) under this base directory in which to install files.  The
details differ across platforms, so read whichever of the following
sections applies to you.

9.3.1 Alternate installation: the home scheme
---------------------------------------------

The idea behind the "home scheme" is that you build and maintain a
personal stash of Python modules.  This scheme's name is derived from
the idea of a "home" directory on Unix, since it's not unusual for a
Unix user to make their home directory have a layout similar to `/usr/'
or `/usr/local/'.  This scheme can be used by anyone, regardless of the
operating system they are installing for.

  Installing a new module distribution is as simple as

    python setup.py install --home=<dir>

where you can supply any directory you like for the `--home' option.  On
Unix, lazy typists can just type a tilde (`~'); the *install* command
will expand this to your home directory:

    python setup.py install --home=~

The `--home' option defines the installation base directory.  Files are
installed to the following directories under the installation base as
follows:

Type of file                       Installation Directory          Override option
-----------------------------------------------------------------------------------------------------
pure module distribution           `_home_/lib/python'             `--install-purelib'
non-pure module distribution       `_home_/lib/python'             `--install-platlib'
scripts                            `_home_/bin'                    `--install-scripts'
data                               `_home_/share'                  `--install-data'

  Changed in version 2.4: The `--home' option used to be supported only
on Unix.

9.3.2 Alternate installation: Unix (the prefix scheme)
------------------------------------------------------

The "prefix scheme" is useful when you wish to use one Python
installation to perform the build/install (i.e., to run the setup
script), but install modules into the third-party module directory of a
different Python installation (or something that looks like a different
Python installation).  If this sounds a trifle unusual, it is--that's
why the "home scheme" comes first.  However, there are at least two
known cases where the prefix scheme will be useful.

  First, consider that many Linux distributions put Python in `/usr',
rather than the more traditional `/usr/local'.  This is entirely
appropriate, since in those cases Python is part of "the system" rather
than a local add-on.  However, if you are installing Python modules
from source, you probably want them to go in
`/usr/local/lib/python2._X_' rather than `/usr/lib/python2._X_'.  This
can be done with

    /usr/bin/python setup.py install --prefix=/usr/local

Another possibility is a network filesystem where the name used to
write to a remote directory is different from the name used to read it:
for example, the Python interpreter accessed as `/usr/local/bin/python'
might search for modules in `/usr/local/lib/python2._X_', but those
modules would have to be installed to, say,
`/mnt/_@server_/export/lib/python2._X_'.  This could be done with

    /usr/local/bin/python setup.py install --prefix=/mnt/@server/export

In either case, the `--prefix' option defines the installation base, and
the `--exec-prefix' option defines the platform-specific installation
base, which is used for platform-specific files.  (Currently, this just
means non-pure module distributions, but could be expanded to C
libraries, binary executables, etc.)  If `--exec-prefix' is not
supplied, it defaults to `--prefix'.  Files are installed as follows:

Type of file                       Installation Directory                                    Override option
-------------------------------------------------------------------------------------------------------------------------------
pure module distribution           `_prefix_/lib/python_X.Y_/site-packages'                  `--install-purelib'
non-pure module distribution       `_exec-prefix_/lib/python_X.Y_/site-packages'             `--install-platlib'
scripts                            `_prefix_/bin'                                            `--install-scripts'
data                               `_prefix_/share'                                          `--install-data'

  There is no requirement that `--prefix' or `--exec-prefix' actually
point to an alternate Python installation; if the directories listed
above do not already exist, they are created at installation time.

  Incidentally, the real reason the prefix scheme is important is
simply that a standard Unix installation uses the prefix scheme, but
with `--prefix' and `--exec-prefix' supplied by Python itself as
`sys.prefix' and `sys.exec_prefix'.  Thus, you might think you'll never
use the prefix scheme, but every time you run `python setup.py install'
without any other options, you're using it.

  Note that installing extensions to an alternate Python installation
has no effect on how those extensions are built: in particular, the
Python header files (`Python.h' and friends) installed with the Python
interpreter used to run the setup script will be used in compiling
extensions.  It is your responsibility to ensure that the interpreter
used to run extensions installed in this way is compatible with the
interpreter used to build them.  The best way to do this is to ensure
that the two interpreters are the same version of Python (possibly
different builds, or possibly copies of the same build).  (Of course,
if your `--prefix' and `--exec-prefix' don't even point to an alternate
Python installation, this is immaterial.)

9.3.3 Alternate installation: Windows (the prefix scheme)
---------------------------------------------------------

Windows has no concept of a user's home directory, and since the
standard Python installation under Windows is simpler than under Unix,
the `--prefix' option has traditionally been used to install additional
packages in separate locations on Windows.

    python setup.py install --prefix="\Temp\Python"

to install modules to the `\Temp\Python' directory on the current drive.

  The installation base is defined by the `--prefix' option; the
`--exec-prefix' option is not supported under Windows. Files are
installed as follows:

Type of file                       Installation Directory          Override option
-----------------------------------------------------------------------------------------------------
pure module distribution           `_prefix_'                      `--install-purelib'
non-pure module distribution       `_prefix_'                      `--install-platlib'
scripts                            `_prefix_\Scripts'              `--install-scripts'
data                               `_prefix_\Data'                 `--install-data'

9.4 Custom Installation
=======================

Sometimes, the alternate installation schemes described in section
Alternate Installation just don't do what you want.  You
might want to tweak just one or two directories while keeping
everything under the same base directory, or you might want to
completely redefine the installation scheme.  In either case, you're
creating a _custom installation scheme_.

  You probably noticed the column of "override options" in the tables
describing the alternate installation schemes above.  Those options are
how you define a custom installation scheme.  These override options
can be relative, absolute, or explicitly defined in terms of one of the
installation base directories.  (There are two installation base
directories, and they are normally the same-- they only differ when you
use the Unix "prefix scheme" and supply different `--prefix' and
`--exec-prefix' options.)

  For example, say you're installing a module distribution to your home
directory under Unix--but you want scripts to go in `~/scripts' rather
than `~/bin'. As you might expect, you can override this directory with
the `--install-scripts' option; in this case, it makes most sense to
supply a relative path, which will be interpreted relative to the
installation base directory (your home directory, in this case):

    python setup.py install --home=~ --install-scripts=scripts

Another Unix example: suppose your Python installation was built and
installed with a prefix of `/usr/local/python', so under a standard
installation scripts will wind up in `/usr/local/python/bin'.  If you
want them in `/usr/local/bin' instead, you would supply this absolute
directory for the `--install-scripts' option:

    python setup.py install --install-scripts=/usr/local/bin

(This performs an installation using the "prefix scheme," where the
prefix is whatever your Python interpreter was installed with--
`/usr/local/python' in this case.)

  If you maintain Python on Windows, you might want third-party modules
to live in a subdirectory of `_prefix_', rather than right in `_prefix_'
itself.  This is almost as easy as customizing the script installation
directory --you just have to remember that there are two types of
modules to worry about, pure modules and non-pure modules (i.e.,
modules from a non-pure distribution).  For example:

    python setup.py install --install-purelib=Site --install-platlib=Site

The specified installation directories are relative to `_prefix_'.  Of
course, you also have to ensure that these directories are in Python's
module search path, such as by putting a `.pth' file in `_prefix_'.  See
section Modifying Python's Search Path to find out how to
modify Python's search path.

  If you want to define an entire installation scheme, you just have to
supply all of the installation directory options.  The recommended way
to do this is to supply relative paths; for example, if you want to
maintain all Python module-related files under `python' in your home
directory, and you want a separate directory for each platform that you
use your home directory from, you might define the following
installation scheme:

    python setup.py install --home=~ \
                            --install-purelib=python/lib \
                            --install-platlib=python/lib.$PLAT \
                            --install-scripts=python/scripts
                            --install-data=python/data

or, equivalently,

    python setup.py install --home=~/python \
                            --install-purelib=lib \
                            --install-platlib='lib.$PLAT' \
                            --install-scripts=scripts
                            --install-data=data

`$PLAT' is not (necessarily) an environment variable--it will be
expanded by the Distutils as it parses your command line options, just
as it does when parsing your configuration file(s).

  Obviously, specifying the entire installation scheme every time you
install a new module distribution would be very tedious.  Thus, you can
put these options into your Distutils config file (see section
Distutils Configuration Files):

    [install]
    install-base=$HOME
    install-purelib=python/lib
    install-platlib=python/lib.$PLAT
    install-scripts=python/scripts
    install-data=python/data

or, equivalently,

    [install]
    install-base=$HOME/python
    install-purelib=lib
    install-platlib=lib.$PLAT
    install-scripts=scripts
    install-data=data

Note that these two are _not_ equivalent if you supply a different
installation base directory when you run the setup script.  For example,

    python setup.py install --install-base=/tmp

would install pure modules to `_/tmp/python/lib_' in the first case, and
to `_/tmp/lib_' in the second case.  (For the second case, you probably
want to supply an installation base of `/tmp/python'.)

  You probably noticed the use of `$HOME' and `$PLAT' in the sample
configuration file input.  These are Distutils configuration variables,
which bear a strong resemblance to environment variables. In fact, you
can use environment variables in config files on platforms that have
such a notion but the Distutils additionally define a few extra
variables that may not be in your environment, such as `$PLAT'.  (And
of course, on systems that don't have environment variables, such as
Mac OS 9, the configuration variables supplied by the Distutils are the
only ones you can use.) See section Distutils Configuration Files.
for details.

9.4.1 Modifying Python's Search Path
------------------------------------

When the Python interpreter executes an import statement,
it searches for both Python code and extension modules along a search
path.  A default value for the path is configured into the Python
binary when the interpreter is built.  You can determine the path by
importing the sys module and printing the value of
`sys.path'.

    $ python
    Python 2.2 (#11, Oct  3 2002, 13:31:27)
    [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.path
    ['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2',
     '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload',
     '/usr/local/lib/python2.3/site-packages']
    >>>

The null string in `sys.path' represents the current working directory.

  The expected convention for locally installed packages is to put them
in the `_..._/site-packages/' directory, but you may want to install
Python modules into some arbitrary directory.  For example, your site
may have a convention of keeping all software related to the web server
under `/www'.  Add-on Python modules might then belong in
`/www/python', and in order to import them, this directory must be
added to `sys.path'.  There are several different ways to add the
directory.

  The most convenient way is to add a path configuration file to a
directory that's already on Python's path, usually to the
`.../site-packages/' directory.  Path configuration files have an
extension of `.pth', and each line must contain a single path that will
be appended to `sys.path'.  (Because the new paths are appended to
`sys.path', modules in the added directories will not override standard
modules.  This means you can't use this mechanism for installing fixed
versions of standard modules.)

  Paths can be absolute or relative, in which case they're relative to
the directory containing the `.pth' file.  See the documentation of the
site module for more information.

  A slightly less convenient way is to edit the `site.py' file in
Python's standard library, and modify `sys.path'.  `site.py' is
automatically imported when the Python interpreter is executed, unless
the -S switch is supplied to suppress this behaviour.  So
you could simply edit `site.py' and add two lines to it:

    import sys
    sys.path.append('/www/python/')

However, if you reinstall the same major version of Python (perhaps when
upgrading from 2.2 to 2.2.2, for example) `site.py' will be overwritten
by the stock version.  You'd have to remember that it was modified and
save a copy before doing the installation.

  There are two environment variables that can modify `sys.path'.
PYTHONHOME sets an alternate value for the prefix of the Python
installation.  For example, if PYTHONHOME is set to
`/www/python', the search path will be set to `['',
'/www/python/lib/pythonX.Y/', '/www/python/lib/pythonX.Y/plat-linux2',
...]'.

  The PYTHONPATH variable can be set to a list of paths
that will be added to the beginning of `sys.path'.  For example, if
PYTHONPATH is set to `/www/python:/opt/py', the search path will
begin with `['/www/python', '/opt/py']'.  (Note that directories must
exist in order to be added to `sys.path'; the site module
removes paths that don't exist.)

  Finally, `sys.path' is just a regular Python list, so any Python
application can modify it by adding or removing entries.

9.5 Distutils Configuration Files
=================================

As mentioned above, you can use Distutils configuration files to record
personal or site preferences for any Distutils options.  That is, any
option to any command can be stored in one of two or three (depending
on your platform) configuration files, which will be consulted before
the command-line is parsed.  This means that configuration files will
override default values, and the command-line will in turn override
configuration files.  Furthermore, if multiple configuration files
apply, values from "earlier" files are overridden by "later" files.

9.5.1 Location and names of config files
----------------------------------------

The names and locations of the configuration files vary slightly across
platforms.  On Unix and Mac OS X, the three configuration files (in the
order they are processed) are:

Type of file       Location and filename                                          Notes
----------------------------------------------------------------------------------------------
system             `_prefix_/lib/python_ver_/distutils/distutils.cfg'             (1)
personal           `$HOME/.pydistutils.cfg'                                       (2)
local              `setup.cfg'                                                    (3)

  And on Windows, the configuration files are:

Type of file       Location and filename                                 Notes
-------------------------------------------------------------------------------------
system             `_prefix_\Lib\distutils\distutils.cfg'                (4)
personal           `%HOME%\pydistutils.cfg'                              (5)
local              `setup.cfg'                                           (3)

  On all platforms, the "personal" file can be temporarily disabled by
passing the `-no-user-cfg' option.

  Notes:

  1. Strictly speaking, the system-wide configuration file lives in the
     directory where the Distutils are installed; under Python 1.6 and
     later on Unix, this is as shown. For Python 1.5.2, the Distutils
     will normally be installed to
     `_prefix_/lib/python1.5/site-packages/distutils', so the system
     configuration file should be put there under Python 1.5.2.

  2. On Unix, if the `HOME' environment variable is not defined, the
     user's home directory will be determined with the `getpwuid()'
     function from the standard pwd module. This is done by
     the os.path.expanduser().  function used by Distutils.

  3. I.e., in the current directory (usually the location of the setup
     script).

  4. (See also note (1).)  Under Python 1.6 and later, Python's default
     "installation prefix" is `C:\Python', so the system configuration
     file is normally `C:\Python\Lib\distutils\distutils.cfg'. Under
     Python 1.5.2, the default prefix was `C:\Program Files\Python',
     and the Distutils were not part of the standard library--so the
     system configuration file would be `C:\Program
     Files\Python\distutils\distutils.cfg' in a standard Python 1.5.2
     installation under Windows.

  5. On Windows, if the `HOME' environment variable is not defined, `USERPROFILE'
     then `HOMEDRIVE' and `HOMEPATH' will be tried. This is done by the
     os.path.expanduser() function used by Distutils.

9.5.2 Syntax of config files
----------------------------

The Distutils configuration files all have the same syntax.  The config
files are grouped into sections.  There is one section for each
Distutils command, plus a `global' section for global options that
affect every command.  Each section consists of one option per line,
specified as `option=value'.

  For example, the following is a complete config file that just forces
all commands to run quietly by default:

    [global]
    verbose=0

If this is installed as the system config file, it will affect all
processing of any Python module distribution by any user on the current
system.  If it is installed as your personal config file (on systems
that support them), it will affect only module distributions processed
by you.  And if it is used as the `setup.cfg' for a particular module
distribution, it affects only that distribution.

  You could override the default "build base" directory and make the
*build** commands always forcibly rebuild all files with the following:

    [build]
    build-base=blib
    force=1

which corresponds to the command-line arguments

    python setup.py build --build-base=blib --force

except that including the *build* command on the command-line means
that command will be run.  Including a particular command in config
files has no such implication; it only means that if the command is
run, the options in the config file will apply.  (Or if other commands
that derive values from it are run, they will use the values in the
config file.)

  You can find out the complete list of options for any command using
the -help option, e.g.:

    python setup.py build --help

and you can find out the complete list of global options by using
-help without a command:

    python setup.py --help

See also the "Reference" section of the "Distributing Python Modules"
manual.

9.6 Building Extensions: Tips and Tricks
========================================

Whenever possible, the Distutils try to use the configuration
information made available by the Python interpreter used to run the
`setup.py' script.  For example, the same compiler and linker flags
used to compile Python will also be used for compiling extensions.
Usually this will work well, but in complicated situations this might
be inappropriate.  This section discusses how to override the usual
Distutils behaviour.

9.6.1 Tweaking compiler/linker flags
------------------------------------

Compiling a Python extension written in C or C++ will sometimes require
specifying custom flags for the compiler and linker in order to use a
particular library or produce a special kind of object code. This is
especially true if the extension hasn't been tested on your platform,
or if you're trying to cross-compile Python.

  In the most general case, the extension author might have foreseen
that compiling the extensions would be complicated, and provided a
`Setup' file for you to edit.  This will likely only be done if the
module distribution contains many separate extension modules, or if
they often require elaborate sets of compiler flags in order to work.

  A `Setup' file, if present, is parsed in order to get a list of
extensions to build.  Each line in a `Setup' describes a single module.
Lines have the following structure:

    module ... [sourcefile ...] [cpparg ...] [library ...]

Let's examine each of the fields in turn.

   * _module_ is the name of the extension module to be built, and
     should be a valid Python identifier.  You can't just change this
     in order to rename a module (edits to the source code would also
     be needed), so this should be left alone.

   * _sourcefile_ is anything that's likely to be a source code file,
     at least judging by the filename.  Filenames ending in `.c' are
     assumed to be written in C, filenames ending in `.C', `.cc', and
     `.c++' are assumed to be C++, and filenames ending in `.m' or
     `.mm' are assumed to be in Objective C.

   * _cpparg_ is an argument for the C preprocessor,  and is anything
     starting with `-I', `-D', see -U or `-C'.

   * _library_ is anything ending in `.a' or beginning with `-l' or
     `-L'.

  If a particular platform requires a special library on your platform,
you can add it by editing the `Setup' file and running `python setup.py
build'.  For example, if the module defined by the line

    foo foomodule.c

must be linked with the math library `libm.a' on your platform, simply
add `-lm' to the line:

    foo foomodule.c -lm

Arbitrary switches intended for the compiler or the linker can be
supplied with the `-Xcompiler' _arg_ and `-Xlinker' _arg_ options:

    foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm

The next option after `-Xcompiler' and `-Xlinker' will be appended to
the proper command line, so in the above example the compiler will be
passed the `-o32' option, and the linker will be passed `-shared'.  If
a compiler option requires an argument, you'll have to supply multiple
`-Xcompiler' options; for example, to pass `-x c++' the `Setup' file
would have to contain `-Xcompiler -x -Xcompiler c++'.

  Compiler flags can also be supplied through setting the `CFLAGS'
environment variable.  If set, the contents of `CFLAGS' will be added to
the compiler flags specified in the  `Setup' file.

9.6.2 Using non-Microsoft compilers on Windows
----------------------------------------------

9.6.2.1 Borland/CodeGear C++
............................

This subsection describes the necessary steps to use Distutils with the
Borland C++ compiler version 5.5.  First you have to know that
Borland's object file format (OMF) is different from the format used by
the Python version you can download from the Python or ActiveState Web
site.  (Python is built with Microsoft Visual C++, which uses COFF as
the object file format.) For this reason you have to convert Python's
library `python25.lib' into the Borland format.  You can do this as
follows:

    coff2omf python25.lib python25_bcpp.lib

The `coff2omf' program comes with the Borland compiler.  The file
`python25.lib' is in the `Libs' directory of your Python installation.
If your extension uses other libraries (zlib, ...) you have to convert
them too.

  The converted files have to reside in the same directories as the
normal libraries.

  How does Distutils manage to use these libraries with their changed
names?  If the extension needs a library (eg. `foo') Distutils checks
first if it finds a library with suffix `_bcpp' (eg. `foo_bcpp.lib')
and then uses this library.  In the case it doesn't find such a special
library it uses the default name (`foo.lib'.) (1)

  To let Distutils compile your extension with Borland C++ you now have
to type:

    python setup.py build --compiler=bcpp

If you want to use the Borland C++ compiler as the default, you could
specify this in your personal or system-wide configuration file for
Distutils (see section Distutils Configuration Files)

C++Builder Compiler (http://www.codegear.com/downloads/free/cppbuilder)
     Information about the free C++ compiler from Borland, including
     links to the download pages.

Creating Python Extensions Using Borland's Free Compiler (http://www.cyberus.ca/~g_will/pyExtenDL.shtml)
     Document describing how to use Borland's free command-line C++
     compiler to build Python.

9.6.2.2 GNU C / Cygwin / MinGW
..............................

These instructions only apply if you're using a version of Python prior
to 2.4.1 with a MinGW prior to 3.0.0 (with binutils-2.13.90-20030111-1).

  This section describes the necessary steps to use Distutils with the
GNU C/C++ compilers in their Cygwin and MinGW distributions. (2) For a
Python interpreter that was built with Cygwin, everything should work
without any of these following steps.

  These compilers require some special libraries. This task is more
complex than for Borland's C++, because there is no program to convert
the library.  First you have to create a list of symbols which the
Python DLL exports. (You can find a good program for this task at
<http://www.emmestech.com/software/pexports-0.43/download_pexports.html>).

    pexports python25.dll >python25.def

The location of an installed `python25.dll' will depend on the
installation options and the version and language of Windows.  In a
"just for me" installation, it will appear in the root of the
installation directory.  In a shared installation, it will be located
in the system directory.

  Then you can create from these information an import library for gcc.

    /cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a

The resulting library has to be placed in the same directory as
`python25.lib'. (Should be the `libs' directory under your Python
installation directory.)

  If your extension uses other libraries (zlib,...) you might  have to
convert them too. The converted files have to reside in the same
directories as the normal libraries do.

  To let Distutils compile your extension with Cygwin you now have to
type

    python setup.py build --compiler=cygwin

and for Cygwin in no-cygwin mode (3) or for MinGW type:

    python setup.py build --compiler=mingw32

If you want to use any of these options/compilers as default, you should
consider to write it in your personal or system-wide configuration file
for Distutils (see section Distutils Configuration Files)

Building Python modules on MS Windows platform with MinGW (http://www.zope.org/Members/als/tips/win32_mingw_modules)
     Information about building the required libraries for the MinGW
     environment.

Footnotes
.........

     (1) This also means you could replace all existing COFF-libraries
     with OMF-libraries of the same name.

     (2) Check <http://sources.redhat.com/cygwin/> and
     <http://www.mingw.org/> for more information

     (3) Then you have no POSIX emulation available, but you also don't
     need `cygwin1.dll'.

10 Documenting Python
*********************

The Python language has a substantial body of documentation, much of it
contributed by various authors. The markup used for the Python
documentation is reStructuredText (http://docutils.sf.net/rst.html),
developed by the docutils (http://docutils.sf.net/) project, amended by
custom directives and using a toolset named Sphinx
(http://sphinx.pocoo.org/) to postprocess the HTML output.

  This document describes the style guide for our documentation as well
as the custom reStructuredText markup introduced by Sphinx to support
Python documentation and how it should be used.

     Note: If you're interested in contributing to Python's
     documentation, there's no need to write reStructuredText if you're
     not so inclined; plain text contributions are more than welcome as
     well.  Send an e-mail to <docs@python.org> or open an issue on the
     tracker.

10.1 Introduction
=================

Python's documentation has long been considered to be good for a free
programming language.  There are a number of reasons for this, the most
important being the early commitment of Python's creator, Guido van
Rossum, to providing documentation on the language and its libraries,
and the continuing involvement of the user community in providing
assistance for creating and maintaining documentation.

  The involvement of the community takes many forms, from authoring to
bug reports to just plain complaining when the documentation could be
more complete or easier to use.

  This document is aimed at authors and potential authors of
documentation for Python.  More specifically, it is for people
contributing to the standard documentation and developing additional
documents using the same tools as the standard documents.  This guide
will be less useful for authors using the Python documentation tools
for topics other than Python, and less useful still for authors not
using the tools at all.

  If your interest is in contributing to the Python documentation, but
you don't have the time or inclination to learn reStructuredText and
the markup structures documented here, there's a welcoming place for
you among the Python contributors as well.  Any time you feel that you
can clarify existing documentation or provide documentation that's
missing, the existing documentation team will gladly work with you to
integrate your text, dealing with the markup for you.  Please don't let
the material in this document stand between the documentation and your
desire to help out!

10.2 Style Guide
================

The Python documentation should follow the Apple Publications Style
Guide
(http://developer.apple.com/mac/library/documentation/UserExperience/Conceptual/APStyleGuide/APSG_2009.pdf)
wherever possible. This particular style guide was selected mostly
because it seems reasonable and is easy to get online.

  Topics which are not covered in Apple's style guide will be discussed
in this document.

  All reST files use an indentation of 3 spaces.  The maximum line
length is 80 characters for normal text, but tables, deeply indented
code samples and long links may extend beyond that.

  Make generous use of blank lines where applicable; they help grouping
things together.

  A sentence-ending period may be followed by one or two spaces; while
reST ignores the second space, it is customarily put in by some users,
for example to aid Emacs' auto-fill mode.

  Footnotes are generally discouraged, though they may be used when
they are the best way to present specific information. When a footnote
reference is added at the end of the sentence, it should follow the
sentence-ending punctuation. The reST markup should appear something
like this:

    This sentence has a footnote reference. [#]_ This is the next sentence.

Footnotes should be gathered at the end of a file, or if the file is
very long, at the end of a section. The docutils will automatically
create backlinks to the footnote reference.

  Footnotes may appear in the middle of sentences where appropriate.

  Many special names are used in the Python documentation, including
the names of operating systems, programming languages, standards
bodies, and the like. Most of these entities are not assigned any
special markup, but the preferred spellings are given here to aid
authors in maintaining the consistency of presentation in the Python
documentation.

  Other terms and words deserve special mention as well; these
conventions should be used to ensure consistency throughout the
documentation:

CPU
     For "central processing unit." Many style guides say this should
     be spelled out on the first use (and if you must use it, do so!).
     For the Python documentation, this abbreviation should be avoided
     since there's no reasonable way to predict which occurrence will
     be the first seen by the reader. It is better to use the word
     "processor" instead.

POSIX
     The name assigned to a particular group of standards. This is
     always uppercase.

Python
     The name of our favorite programming language is always
     capitalized.

Unicode
     The name of a character set and matching encoding. This is always
     written capitalized.

Unix
     The name of the operating system developed at AT&T Bell Labs in
     the early 1970s.

10.3 reStructuredText Primer
============================

This section is a brief introduction to reStructuredText (reST)
concepts and syntax, intended to provide authors with enough
information to author documents productively.  Since reST was designed
to be a simple, unobtrusive markup language, this will not take too
long.

  The authoritative reStructuredText User Documentation
(http://docutils.sourceforge.net/rst.html).

10.3.1 Paragraphs
-----------------

The paragraph is the most basic block in a reST document.  Paragraphs
are simply chunks of text separated by one or more blank lines.  As in
Python, indentation is significant in reST, so all lines of the same
paragraph must be left-aligned to the same level of indentation.

10.3.2 Inline markup
--------------------

The standard reST inline markup is quite simple: use

   * one asterisk: `*text*' for emphasis (italics),

   * two asterisks: `**text**' for strong emphasis (boldface), and

   * backquotes: ```text``' for code samples.

  If asterisks or backquotes appear in running text and could be
confused with inline markup delimiters, they have to be escaped with a
backslash.

  Be aware of some restrictions of this markup:

   * it may not be nested,

   * content may not start or end with whitespace: `* text*' is wrong,

   * it must be separated from surrounding text by non-word characters.
     Use a backslash escaped space to work around that: `thisis\ *one*\
     word'.

  These restrictions may be lifted in future versions of the docutils.

  reST also allows for custom "interpreted text roles"', which signify
that the enclosed text should be interpreted in a specific way.  Sphinx
uses this to provide semantic markup and cross-referencing of
identifiers, as described in the appropriate section.  The general
syntax is `:rolename:`content`'.

10.3.3 Lists and Quotes
-----------------------

List markup is natural: just place an asterisk at the start of a
paragraph and indent properly.  The same goes for numbered lists; they
can also be autonumbered using a `#' sign:

    * This is a bulleted list.
    * It has two items, the second
      item uses two lines.

    1. This is a numbered list.
    2. It has two items too.

    #. This is a numbered list.
    #. It has two items too.

Nested lists are possible, but be aware that they must be separated
from the parent list items by blank lines:

    * this is
    * a list

      * with a nested list
      * and some subitems

    * and here the parent list continues

Definition lists are created as follows:

    term (up to a line of text)
       Definition of the term, which must be indented

       and can even consist of multiple paragraphs

    next term
       Description.

Paragraphs are quoted by just indenting them more than the surrounding
paragraphs.

10.3.4 Source Code
------------------

Literal code blocks are introduced by ending a paragraph with the
special marker `::'.  The literal block must be indented:

    This is a normal text paragraph. The next paragraph is a code sample::

       It is not processed in any way, except
       that the indentation is removed.

       It can span multiple lines.

    This is a normal text paragraph again.

The handling of the `::' marker is smart:

   * If it occurs as a paragraph of its own, that paragraph is
     completely left out of the document.

   * If it is preceded by whitespace, the marker is removed.

   * If it is preceded by non-whitespace, the marker is replaced by a
     single colon.

  That way, the second sentence in the above example's first paragraph
would be rendered as "The next paragraph is a code sample:".

10.3.5 Hyperlinks
-----------------

10.3.5.1 External links
.......................

Use ``Link text <http://target>`_' for inline web links.  If the link
text should be the web address, you don't need special markup at all,
the parser finds links and mail addresses in ordinary text.

10.3.5.2 Internal links
.......................

Internal linking is done via a special reST role, see the section on
specific markup, see Cross-linking markup.

10.3.6 Sections
---------------

Section headers are created by underlining (and optionally overlining)
the section title with a punctuation character, at least as long as the
text:

    =================
    This is a heading
    =================

Normally, there are no heading levels assigned to certain characters as
the structure is determined from the succession of headings.  However,
for the Python documentation, we use this convention:

   * `#' with overline, for parts

   * `*' with overline, for chapters

   * `=', for sections

   * `-', for subsections

   * `^', for subsubsections

   * `"', for paragraphs

10.3.7 Explicit Markup
----------------------

"Explicit markup" is used in reST for most constructs that need special
handling, such as footnotes, specially-highlighted paragraphs,
comments, and generic directives.

  An explicit markup block begins with a line starting with `..'
followed by whitespace and is terminated by the next paragraph at the
same level of indentation.  (There needs to be a blank line between
explicit markup and normal paragraphs.  This may all sound a bit
complicated, but it is intuitive enough when you write it.)

10.3.8 Directives
-----------------

A directive is a generic block of explicit markup.  Besides roles, it
is one of the extension mechanisms of reST, and Sphinx makes heavy use
of it.

  Basically, a directive consists of a name, arguments, options and
content. (Keep this terminology in mind, it is used in the next chapter
describing custom directives.)  Looking at this example,

    .. function:: foo(x)
                  foo(y, z)
       :bar: no

       Return a line of text input from the user.

`function' is the directive name.  It is given two arguments here, the
remainder of the first line and the second line, as well as one option
`bar' (as you can see, options are given in the lines immediately
following the arguments and indicated by the colons).

  The directive content follows after a blank line and is indented
relative to the directive start.

10.3.9 Footnotes
----------------

For footnotes, use `[#]_' to mark the footnote location, and add the
footnote body at the bottom of the document after a "Footnotes" rubric
heading, like so:

    Lorem ipsum [#]_ dolor sit amet ... [#]_

    .. rubric:: Footnotes

    .. [#] Text of the first footnote.
    .. [#] Text of the second footnote.

You can also explicitly number the footnotes for better context.

10.3.10 Comments
----------------

Every explicit markup block which isn't a valid markup construct (like
the footnotes above) is regarded as a comment.

10.3.11 Source encoding
-----------------------

Since the easiest way to include special characters like em dashes or
copyright signs in reST is to directly write them as Unicode
characters, one has to specify an encoding:

  All Python documentation source files must be in UTF-8 encoding, and
the HTML documents written from them will be in that encoding as well.

10.3.12 Gotchas
---------------

There are some problems one commonly runs into while authoring reST
documents:

   * *Separation of inline markup:* As said above, inline markup spans
     must be separated from the surrounding text by non-word
     characters, you have to use an escaped space to get around that.

10.4 Additional Markup Constructs
=================================

Sphinx adds a lot of new directives and interpreted text roles to
standard reST markup.  This section contains the reference material for
these facilities.  Documentation for "standard" reST constructs is not
included here, though they are used in the Python documentation.

     Note: This is just an overview of Sphinx' extended markup
     capabilities; full coverage can be found in its own documentation
     (http://sphinx.pocoo.org/contents.html).

10.4.1 Meta-information markup
------------------------------

 -- Description: sectionauthor
     Identifies the author of the current section.  The argument should
     include the author's name such that it can be used for
     presentation (though it isn't) and email address.  The domain name
     portion of the address should be lower case.  Example:

         .. sectionauthor:: Guido van Rossum <guido@python.org>

     Currently, this markup isn't reflected in the output in any way,
     but it helps keep track of contributions.

10.4.2 Module-specific markup
-----------------------------

The markup described in this section is used to provide information
about a module being documented.  Each module should be documented in
its own file.  Normally this markup appears after the title heading of
that file; a typical file might start like this:

    :mod:`parrot` -- Dead parrot access
    ===================================

    .. module:: parrot
       :platform: Unix, Windows
       :synopsis: Analyze and reanimate dead parrots.
    .. moduleauthor:: Eric Cleese <eric@python.invalid>
    .. moduleauthor:: John Idle <john@python.invalid>

As you can see, the module-specific markup consists of two directives,
the `module' directive and the `moduleauthor' directive.

 -- Description: module
     This directive marks the beginning of the description of a module
     (or package submodule, in which case the name should be fully
     qualified, including the package name).

     The `platform' option, if present, is a comma-separated list of the
     platforms on which the module is available (if it is available on
     all platforms, the option should be omitted).  The keys are short
     identifiers; examples that are in use include "IRIX", "Mac",
     "Windows", and "Unix".  It is important to use a key which has
     already been used when applicable.

     The `synopsis' option should consist of one sentence describing the
     module's purpose - it is currently only used in the Global Module
     Index.

     The `deprecated' option can be given (with no value) to mark a
     module as deprecated; it will be designated as such in various
     locations then.

 -- Description: moduleauthor
     The `moduleauthor' directive, which can appear multiple times,
     names the authors of the module code, just like `sectionauthor'
     names the author(s) of a piece of documentation.  It too does not
     result in any output currently.

     Note: It is important to make the section title of a
     module-describing file meaningful since that value will be
     inserted in the table-of-contents trees in overview files.

10.4.3 Information units
------------------------

There are a number of directives used to describe specific features
provided by modules.  Each directive requires one or more signatures to
provide basic information about what is being described, and the
content should be the description.  The basic version makes entries in
the general index; if no index entry is desired, you can give the
directive option flag `:noindex:'.  The following example shows all of
the features of this directive type:

    .. function:: spam(eggs)
                  ham(eggs)
       :noindex:

       Spam or ham the foo.

The signatures of object methods or data attributes should always
include the type name (`.. method:: FileInput.input(...)'), even if it
is obvious from the context which type they belong to; this is to
enable consistent cross-references.  If you describe methods belonging
to an abstract protocol, such as "context managers", include a
(pseudo-)type name too to make the index entries more informative.

  The directives are:

 -- Description: cfunction
     Describes a C function. The signature should be given as in C,
     e.g.:

         .. cfunction:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)

     This is also used to describe function-like preprocessor macros.
     The names of the arguments should be given so they may be used in
     the description.

     Note that you don't have to backslash-escape asterisks in the
     signature, as it is not parsed by the reST inliner.

 -- Description: cmember
     Describes a C struct member. Example signature:

         .. cmember:: PyObject* PyTypeObject.tp_bases

     The text of the description should include the range of values
     allowed, how the value should be interpreted, and whether the
     value can be changed.  References to structure members in text
     should use the `member' role.

 -- Description: cmacro
     Describes a "simple" C macro.  Simple macros are macros which are
     used for code expansion, but which do not take arguments so cannot
     be described as functions.  This is not to be used for simple
     constant definitions.  Examples of its use in the Python
     documentation include PyObject_HEAD and
     Py_BEGIN_ALLOW_THREADS.

 -- Description: ctype
     Describes a C type. The signature should just be the type name.

 -- Description: cvar
     Describes a global C variable.  The signature should include the
     type, such as:

         .. cvar:: PyObject* PyClass_Type

 -- Description: data
     Describes global data in a module, including both variables and
     values used as "defined constants."  Class and object attributes
     are not documented using this environment.

 -- Description: exception
     Describes an exception class.  The signature can, but need not
     include parentheses with constructor arguments.

 -- Description: function
     Describes a module-level function.  The signature should include
     the parameters, enclosing optional parameters in brackets.
     Default values can be given if it enhances clarity.  For example:

         .. function:: Timer.repeat([repeat=3[, number=1000000]])

     Object methods are not documented using this directive. Bound
     object methods placed in the module namespace as part of the
     public interface of the module are documented using this, as they
     are equivalent to normal functions for most purposes.

     The description should include information about the parameters
     required and how they are used (especially whether mutable objects
     passed as parameters are modified), side effects, and possible
     exceptions.  A small example may be provided.

 -- Description: class
     Describes a class.  The signature can include parentheses with
     parameters which will be shown as the constructor arguments.

 -- Description: attribute
     Describes an object data attribute.  The description should include
     information about the type of the data to be expected and whether
     it may be changed directly.

 -- Description: method
     Describes an object method.  The parameters should not include the
     `self' parameter.  The description should include similar
     information to that described for `function'.

 -- Description: opcode
     Describes a Python bytecode instruction.

 -- Description: cmdoption
     Describes a Python command line option or switch.  Option argument
     names should be enclosed in angle brackets.  Example:

         .. cmdoption:: -m <module>

            Run a module as a script.

 -- Description: envvar
     Describes an environment variable that Python uses or defines.

  There is also a generic version of these directives:

 -- Description: describe
     This directive produces the same formatting as the specific ones
     explained above but does not create index entries or
     cross-referencing targets.  It is used, for example, to describe
     the directives in this document. Example:

         .. describe:: opcode

            Describes a Python bytecode instruction.

10.4.4 Showing code examples
----------------------------

Examples of Python source code or interactive sessions are represented
using standard reST literal blocks.  They are started by a `::' at the
end of the preceding paragraph and delimited by indentation.

  Representing an interactive session requires including the prompts
and output along with the Python code.  No special markup is required
for interactive sessions.  After the last line of input or output
presented, there should not be an "unused" primary prompt; this is an
example of what _not_ to do:

    >>> 1 + 1
    2
    >>>

Syntax highlighting is handled in a smart way:

   * There is a "highlighting language" for each source file.  Per
     default, this is `'python'' as the majority of files will have to
     highlight Python snippets.

   * Within Python highlighting mode, interactive sessions are
     recognized automatically and highlighted appropriately.

   * The highlighting language can be changed using the `highlightlang'
     directive, used as follows:

         .. highlightlang:: c

     This language is used until the next `highlightlang' directive is
     encountered.

   * The values normally used for the highlighting language are:

        * `python' (the default)

        * `c'

        * `rest'

        * `none' (no highlighting)

   * If highlighting with the current language fails, the block is not
     highlighted in any way.

  Longer displays of verbatim text may be included by storing the
example text in an external file containing only plain text.  The file
may be included using the `literalinclude' directive. (1) For example,
to include the Python source file `example.py', use:

    .. literalinclude:: example.py

The file name is relative to the current file's path.
Documentation-specific include files should be placed in the
`Doc/includes' subdirectory.

10.4.5 Inline markup
--------------------

As said before, Sphinx uses interpreted text roles to insert semantic
markup in documents.

  Names of local variables, such as function/method arguments, are an
exception, they should be marked simply with `*var*'.

  For all other roles, you have to write `:rolename:`content`'.

  There are some additional facilities that make cross-referencing
roles more versatile:

   * You may supply an explicit title and reference target, like in
     reST direct hyperlinks: `:role:`title <target>`' will refer to
     _target_, but the link text will be _title_.

   * If you prefix the content with `!', no reference/hyperlink will be
     created.

   * For the Python object roles, if you prefix the content with `~',
     the link text will only be the last component of the target.  For
     example, `:meth:`~Queue.Queue.get`' will refer to
     `Queue.Queue.get' but only display `get' as the link text.

     In HTML output, the link's `title' attribute (that is e.g. shown
     as a tool-tip on mouse-hover) will always be the full target name.

  The following roles refer to objects in modules and are possibly
hyperlinked if a matching identifier is found:

 -- Description: mod
     The name of a module; a dotted name may be used.  This should also
     be used for package names.

 -- Description: func
     The name of a Python function; dotted names may be used.  The role
     text should not include trailing parentheses to enhance
     readability.  The parentheses are stripped when searching for
     identifiers.

 -- Description: data
     The name of a module-level variable or constant.

 -- Description: const
     The name of a "defined" constant.  This may be a C-language
     `#define' or a Python variable that is not intended to be changed.

 -- Description: class
     A class name; a dotted name may be used.

 -- Description: meth
     The name of a method of an object.  The role text should include
     the type name and the method name.  A dotted name may be used.

 -- Description: attr
     The name of a data attribute of an object.

 -- Description: exc
     The name of an exception. A dotted name may be used.

  The name enclosed in this markup can include a module name and/or a
class name.  For example, `:func:`filter`' could refer to a function
named `filter' in the current module, or the built-in function of that
name.  In contrast, `:func:`foo.filter`' clearly refers to the `filter'
function in the `foo' module.

  Normally, names in these roles are searched first without any further
qualification, then with the current module name prepended, then with
the current module and class name (if any) prepended.  If you prefix
the name with a dot, this order is reversed.  For example, in the
documentation of the codecs module, `:func:`open`' always
refers to the built-in function, while `:func:`.open`' refers to
codecs.open().

  A similar heuristic is used to determine whether the name is an
attribute of the currently documented class.

  The following roles create cross-references to C-language constructs
if they are defined in the API documentation:

 -- Description: cdata
     The name of a C-language variable.

 -- Description: cfunc
     The name of a C-language function. Should include trailing
     parentheses.

 -- Description: cmacro
     The name of a "simple" C macro, as defined above.

 -- Description: ctype
     The name of a C-language type.

  The following role does possibly create a cross-reference, but does
not refer to objects:

 -- Description: token
     The name of a grammar token (used in the reference manual to
     create links between production displays).

  The following role creates a cross-reference to the term in the
glossary:

 -- Description: term
     Reference to a term in the glossary.  The glossary is created
     using the `glossary' directive containing a definition list with
     terms and definitions.  It does not have to be in the same file as
     the `term' markup, in fact, by default the Python docs have one
     global glossary in the `glossary.rst' file.

     If you use a term that's not explained in a glossary, you'll get a
     warning during build.

______________________________________________________________________

The following roles don't do anything special except formatting the text
in a different style:

 -- Description: command
     The name of an OS-level command, such as `rm'.

 -- Description: dfn
     Mark the defining instance of a term in the text.  (No index
     entries are generated.)

 -- Description: envvar
     An environment variable.  Index entries are generated.

 -- Description: file
     The name of a file or directory.  Within the contents, you can use
     curly braces to indicate a "variable" part, for example:

         ... is installed in :file:`/usr/lib/python2.{x}/site-packages` ...

     In the built documentation, the `x' will be displayed differently
     to indicate that it is to be replaced by the Python minor version.

 -- Description: guilabel
     Labels presented as part of an interactive user interface should
     be marked using `guilabel'.  This includes labels from text-based
     interfaces such as those created using curses or other
     text-based libraries.  Any label used in the interface should be
     marked with this role, including button labels, window titles,
     field names, menu and menu selection names, and even values in
     selection lists.

 -- Description: kbd
     Mark a sequence of keystrokes.  What form the key sequence takes
     may depend on platform- or application-specific conventions.  When
     there are no relevant conventions, the names of modifier keys
     should be spelled out, to improve accessibility for new users and
     non-native speakers.  For example, an _xemacs_ key sequence may be
     marked like `:kbd:`C-x C-f`', but without reference to a specific
     application or platform, the same sequence should be marked as
     `:kbd:`Control-x Control-f`'.

 -- Description: keyword
     The name of a keyword in Python.

 -- Description: mailheader
     The name of an RFC 822-style mail header.  This markup does not
     imply that the header is being used in an email message, but can
     be used to refer to any header of the same "style."  This is also
     used for headers defined by the various MIME specifications.  The
     header name should be entered in the same way it would normally be
     found in practice, with the camel-casing conventions being
     preferred where there is more than one common usage. For example:
     `:mailheader:`Content-Type`'.

 -- Description: makevar
     The name of a *make* variable.

 -- Description: manpage
     A reference to a Unix manual page including the section, e.g.
     `:manpage:`ls(1)`'.

 -- Description: menuselection
     Menu selections should be marked using the `menuselection' role.
     This is used to mark a complete sequence of menu selections,
     including selecting submenus and choosing a specific operation, or
     any subsequence of such a sequence.  The names of individual
     selections should be separated by `-->'.

     For example, to mark the selection "Start > Programs", use this
     markup:

         :menuselection:`Start --> Programs`

     When including a selection that includes some trailing indicator,
     such as the ellipsis some operating systems use to indicate that
     the command opens a dialog, the indicator should be omitted from
     the selection name.

 -- Description: mimetype
     The name of a MIME type, or a component of a MIME type (the major
     or minor portion, taken alone).

 -- Description: newsgroup
     The name of a Usenet newsgroup.

 -- Description: option
     A command-line option of Python.  The leading hyphen(s) must be
     included.  If a matching `cmdoption' directive exists, it is
     linked to.  For options of other programs or scripts, use simple
     ```code``' markup.

 -- Description: program
     The name of an executable program.  This may differ from the file
     name for the executable for some platforms.  In particular, the
     `.exe' (or other) extension should be omitted for Windows programs.

 -- Description: regexp
     A regular expression. Quotes should not be included.

 -- Description: samp
     A piece of literal text, such as code.  Within the contents, you
     can use curly braces to indicate a "variable" part, as in `:file:'.

     If you don't need the "variable part" indication, use the standard
     ```code``' instead.

  The following roles generate external links:

 -- Description: pep
     A reference to a Python Enhancement Proposal.  This generates
     appropriate index entries. The text "PEP _number_" is generated;
     in the HTML output, this text is a hyperlink to an online copy of
     the specified PEP.

 -- Description: rfc
     A reference to an Internet Request for Comments.  This generates
     appropriate index entries. The text "RFC _number_" is generated;
     in the HTML output, this text is a hyperlink to an online copy of
     the specified RFC.

  Note that there are no special roles for including hyperlinks as you
can use the standard reST markup for that purpose.

10.4.6 Cross-linking markup
---------------------------

To support cross-referencing to arbitrary sections in the
documentation, the standard reST labels are "abused" a bit: Every label
must precede a section title; and every label name must be unique
throughout the entire documentation source.

  You can then reference to these sections using the
`:ref:`label-name`' role.

  Example:

    .. _my-reference-label:

    Section to cross-reference
    --------------------------

    This is the text of the section.

    It refers to the section itself, see :ref:`my-reference-label`.

The `:ref:' invocation is replaced with the section title.

10.4.7 Paragraph-level markup
-----------------------------

These directives create short paragraphs and can be used inside
information units as well as normal text:

 -- Description: note
     An especially important bit of information about an API that a
     user should be aware of when using whatever bit of API the note
     pertains to.  The content of the directive should be written in
     complete sentences and include all appropriate punctuation.

     Example:

         .. note::

            This function is not suitable for sending spam e-mails.

 -- Description: warning
     An important bit of information about an API that a user should be
     aware of when using whatever bit of API the warning pertains to.
     The content of the directive should be written in complete
     sentences and include all appropriate punctuation.  In the
     interest of not scaring users away from pages filled with
     warnings, this directive should only be chosen over `note' for
     information regarding the possibility of crashes, data loss, or
     security implications.

 -- Description: versionadded
     This directive documents the version of Python which added the
     described feature to the library or C API. When this applies to an
     entire module, it should be placed at the top of the module
     section before any prose.

     The first argument must be given and is the version in question;
     you can add a second argument consisting of a _brief_ explanation
     of the change.

     Example:

         .. versionadded:: 2.5
            The *spam* parameter.

     Note that there must be no blank line between the directive head
     and the explanation; this is to make these blocks visually
     continuous in the markup.

 -- Description: versionchanged
     Similar to `versionadded', but describes when and what changed in
     the named feature in some way (new parameters, changed side
     effects, etc.).

______________________________________________________________________

 -- Description: impl-detail
     This directive is used to mark CPython-specific information.  Use
     either with a block content or a single sentence as an argument,
     i.e. either

         .. impl-detail::

            This describes some implementation detail.

            More explanation.

     or

         .. impl-detail:: This shortly mentions an implementation detail.

     "*CPython implementation detail:*" is automatically prepended to
     the content.

 -- Description: seealso
     Many sections include a list of references to module documentation
     or external documents.  These lists are created using the
     `seealso' directive.

     The `seealso' directive is typically placed in a section just
     before any sub-sections.  For the HTML output, it is shown boxed
     off from the main flow of the text.

     The content of the `seealso' directive should be a reST definition
     list.  Example:

         .. seealso::

            Module :mod:`zipfile`
               Documentation of the :mod:`zipfile` standard module.

            `GNU tar manual, Basic Tar Format <http://link>`_
               Documentation for tar archive files, including GNU tar extensions.

 -- Description: rubric
     This directive creates a paragraph heading that is not used to
     create a table of contents node.  It is currently used for the
     "Footnotes" caption.

 -- Description: centered
     This directive creates a centered boldfaced paragraph.  Use it as
     follows:

         .. centered::

            Paragraph contents.

10.4.8 Table-of-contents markup
-------------------------------

Since reST does not have facilities to interconnect several documents,
or split documents into multiple output files, Sphinx uses a custom
directive to add relations between the single files the documentation
is made of, as well as tables of contents.  The `toctree' directive is
the central element.

 -- Description: toctree
     This directive inserts a "TOC tree" at the current location, using
     the individual TOCs (including "sub-TOC trees") of the files given
     in the directive body.  A numeric `maxdepth' option may be given
     to indicate the depth of the tree; by default, all levels are
     included.

     Consider this example (taken from the library reference index):

         .. toctree::
            :maxdepth: 2

            intro
            strings
            datatypes
            numeric
            (many more files listed here)

     This accomplishes two things:

        * Tables of contents from all those files are inserted, with a
          maximum depth of two, that means one nested heading.
          `toctree' directives in those files are also taken into
          account.

        * Sphinx knows that the relative order of the files `intro',
          `strings' and so forth, and it knows that they are children
          of the shown file, the library index.  From this information
          it generates "next chapter", "previous chapter" and "parent
          chapter" links.

     In the end, all files included in the build process must occur in
     one `toctree' directive; Sphinx will emit a warning if it finds a
     file that is not included, because that means that this file will
     not be reachable through standard navigation.

     The special file `contents.rst' at the root of the source
     directory is the "root" of the TOC tree hierarchy; from it the
     "Contents" page is generated.

10.4.9 Index-generating markup
------------------------------

Sphinx automatically creates index entries from all information units
(like functions, classes or attributes) like discussed before.

  However, there is also an explicit directive available, to make the
index more comprehensive and enable index entries in documents where
information is not mainly contained in information units, such as the
language reference.

  The directive is `index' and contains one or more index entries.
Each entry consists of a type and a value, separated by a colon.

  For example:

    .. index::
       single: execution; context
       module: __main__
       module: sys
       triple: module; search; path

This directive contains five entries, which will be converted to
entries in the generated index which link to the exact location of the
index statement (or, in case of offline media, the corresponding page
number).

  The possible entry types are:

single
     Creates a single index entry.  Can be made a subentry by
     separating the subentry text with a semicolon (this notation is
     also used below to describe what entries are created).

pair
     `pair: loop; statement' is a shortcut that creates two index
     entries, namely `loop; statement' and `statement; loop'.

triple
     Likewise, `triple: module; search; path' is a shortcut that
     creates three index entries, which are `module; search path',
     `search; path, module' and `path; module search'.

module, keyword, operator, object, exception, statement, builtin
     These all create two index entries.  For example, `module:
     hashlib' creates the entries `module; hashlib' and `hashlib;
     module'.

  For index directives containing only "single" entries, there is a
shorthand notation:

    .. index:: BNF, grammar, syntax, notation

This creates four index entries.

10.4.10 Grammar production displays
-----------------------------------

Special markup is available for displaying the productions of a formal
grammar.  The markup is simple and does not attempt to model all
aspects of BNF (or any derived forms), but provides enough to allow
context-free grammars to be displayed in a way that causes uses of a
symbol to be rendered as hyperlinks to the definition of the symbol.
There is this directive:

 -- Description: productionlist
     This directive is used to enclose a group of productions.  Each
     production is given on a single line and consists of a name,
     separated by a colon from the following definition.  If the
     definition spans multiple lines, each continuation line must begin
     with a colon placed at the same column as in the first line.

     Blank lines are not allowed within `productionlist' directive
     arguments.

     The definition can contain token names which are marked as
     interpreted text (e.g. `unaryneg ::= "-" `integer`') - this
     generates cross-references to the productions of these tokens.

     Note that no further reST parsing is done in the production, so
     that you don't have to escape `*' or `|' characters.

  The following is an example taken from the Python Reference Manual:

    .. productionlist::
       try_stmt: try1_stmt | try2_stmt
       try1_stmt: "try" ":" `suite`
                : ("except" [`expression` ["," `target`]] ":" `suite`)+
                : ["else" ":" `suite`]
                : ["finally" ":" `suite`]
       try2_stmt: "try" ":" `suite`
                : "finally" ":" `suite`

10.4.11 Substitutions
---------------------

The documentation system provides three substitutions that are defined
by default.  They are set in the build configuration file `conf.py'.

 -- Description: |release|
     Replaced by the Python release the documentation refers to.  This
     is the full version string including alpha/beta/release candidate
     tags, e.g. `2.5.2b3'.

 -- Description: |version|
     Replaced by the Python version the documentation refers to. This
     consists only of the major and minor version parts, e.g. `2.5',
     even for version 2.5.1.

 -- Description: |today|
     Replaced by either today's date, or the date set in the build
     configuration file.  Normally has the format `April 14, 2007'.

Footnotes
.........

     (1) There is a standard `.. include' directive, but it raises
     errors if the file is not found.  This one only emits a warning.

10.5 Differences to the LaTeX markup
====================================

Though the markup language is different, most of the concepts and
markup types of the old LaTeX docs have been kept - environments as
reST directives, inline commands as reST roles and so forth.

  However, there are some differences in the way these work, partly due
to the differences in the markup languages, partly due to improvements
in Sphinx.  This section lists these differences, in order to give
those familiar with the old format a quick overview of what they might
run into.

10.5.1 Inline markup
--------------------

These changes have been made to inline markup:

   * *Cross-reference roles*

     Most of the following semantic roles existed previously as inline
     commands, but didn't do anything except formatting the content as
     code.  Now, they cross-reference to known targets (some names have
     also been shortened):

         _mod_ (previously _refmodule_ or _module_)
         _func_ (previously _function_)
         _data_ (new)
         _const_
         _class_
         _meth_ (previously _method_)
         _attr_ (previously _member_)
         _exc_ (previously _exception_)
         _cdata_
         _cfunc_ (previously _cfunction_)
         _cmacro_ (previously _csimplemacro_)
         _ctype_
     Also different is the handling of _func_ and _meth_: while
     previously parentheses were added to the callable name (like
     `\func{str()}'), they are now appended by the build system -
     appending them in the source will result in double parentheses.
     This also means that `:func:`str(object)`' will not work as
     expected - use ```str(object)``' instead!

   * *Inline commands implemented as directives*

     These were inline commands in LaTeX, but are now directives in
     reST:

         _deprecated_
         _versionadded_
         _versionchanged_
     These are used like so:

         .. deprecated:: 2.5
            Reason of deprecation.

     Also, no period is appended to the text for _versionadded_ and
     _versionchanged_.

         _note_
         _warning_
     These are used like so:

         .. note::

            Content of note.

   * *Otherwise changed commands*

     The _samp_ command previously formatted code and added quotation
     marks around it.  The _samp_ role, however, features a new
     highlighting system just like _file_ does:

          `:samp:`open({filename}, {mode})`' results in
          `open(_filename_, _mode_)'

   * *Dropped commands*

     These were commands in LaTeX, but are not available as roles:

         _bfcode_
         _character_ (use ```'c'``')
         _citetitle_ (use ``Title <URL>`_')
         _code_ (use ```code``')
         _email_ (just write the address in body text)
         _filenq_
         _filevar_ (use the `{...}' highlighting feature of _file_)
         _programopt_, _longprogramopt_ (use _option_)
         _ulink_ (use ``Title <URL>`_')
         _url_ (just write the URL in body text)
         _var_ (use `*var*')
         _infinity_, _plusminus_ (use the Unicode character)
         _shortversion_, _version_ (use the `|version|' and `|release|' substitutions)
         _emph_, _strong_ (use the reST markup)

   * *Backslash escaping*

     In reST, a backslash must be escaped in normal text, and in the
     content of roles.  However, in code literals and literal blocks,
     it must not be escaped.  Example: `:file:`C:\\Temp\\my.tmp`' vs.
     ```open("C:\Temp\my.tmp")``'.

10.5.2 Information units
------------------------

Information units (_...desc_ environments) have been made reST
directives.  These changes to information units should be noted:

   * *New names*

     "desc" has been removed from every name.  Additionally, these
     directives have new names:

         _cfunction_ (previously _cfuncdesc_)
         _cmacro_ (previously _csimplemacrodesc_)
         _exception_ (previously _excdesc_)
         _function_ (previously _funcdesc_)
         _attribute_ (previously _memberdesc_)
     The _classdesc*_ and _excclassdesc_ environments have been
     dropped, the _class_ and _exception_ directives support classes
     documented with and without constructor arguments.

   * *Multiple objects*

     The equivalent of the _...line_ commands is:

         .. function:: do_foo(bar)
                       do_bar(baz)

            Description of the functions.

     IOW, just give one signatures per line, at the same indentation
     level.

   * *Arguments*

     There is no _optional_ command.  Just give function signatures
     like they should appear in the output:

         .. function:: open(filename[, mode[, buffering]])

            Description.

     Note: markup in the signature is not supported.

   * *Indexing*

     The _...descni_ environments have been dropped.  To mark an
     information unit as unsuitable for index entry generation, use the
     _noindex_ option like so:

         .. function:: foo_*
            :noindex:

            Description.

   * *New information units*

     There are new generic information units: One is called "describe"
     and can be used to document things that are not covered by the
     other units:

         .. describe:: a == b

            The equals operator.

     The others are:

         .. cmdoption:: -O

            Describes a command-line option.

         .. envvar:: PYTHONINSPECT

            Describes an environment variable.

10.5.3 Structure
----------------

The LaTeX docs were split in several toplevel manuals.  Now, all files
are part of the same documentation tree, as indicated by the _toctree_
directives in the sources (though individual output formats may choose
to split them up into parts again).  Every _toctree_ directive embeds
other files as subdocuments of the current file (this structure is not
necessarily mirrored in the filesystem layout).  The toplevel file is
`contents.rst'.

  However, most of the old directory structure has been kept, with the
directories renamed as follows:

   * `api' -> `c-api'

   * `dist' -> `distutils', with the single TeX file split up

   * `doc' -> `documenting'

   * `ext' -> `extending'

   * `inst' -> `installing'

   * `lib' -> `library'

   * `mac' -> merged into `library', with `mac/using.tex' moved to
     `using/mac.rst'

   * `ref' -> `reference'

   * `tut' -> `tutorial', with the single TeX file split up

10.6 Building the documentation
===============================

You need to have Python 2.4 or higher installed; the toolset used to
build the docs is written in Python.  It is called _Sphinx_, it is not
included in this tree, but maintained separately.  Also needed are the
docutils, supplying the base markup that Sphinx uses, Jinja, a
templating engine, and optionally Pygments, a code highlighter.

10.6.1 Using make
-----------------

Luckily, a Makefile has been prepared so that on Unix, provided you have
installed Python and Subversion, you can just run

    make html

to check out the necessary toolset in the `tools/' subdirectory and
build the HTML output files.  To view the generated HTML, point your
favorite browser at the top-level index `build/html/index.html' after
running "make".

  Available make targets are:

        * "html", which builds standalone HTML files for offline
          viewing.

        * "htmlhelp", which builds HTML files and a HTML Help project
          file usable to convert them into a single Compiled HTML
          (.chm) file - these are popular under Microsoft Windows, but
          very handy on every platform.

          To create the CHM file, you need to run the Microsoft HTML
          Help Workshop over the generated project (.hhp) file.

        * "latex", which builds LaTeX source files as input to
          "pdflatex" to produce PDF documents.

        * "text", which builds a plain text file for each source file.

        * "linkcheck", which checks all external references to see
          whether they are broken, redirected or malformed, and outputs
          this information to stdout as well as a plain-text (.txt)
          file.

        * "changes", which builds an overview over all
          versionadded/versionchanged/ deprecated items in the current
          version. This is meant as a help for the writer of the
          "What's New" document.

        * "coverage", which builds a coverage overview for standard
          library modules and C API.

        * "pydoc-topics", which builds a Python module containing a
          dictionary with plain text documentation for the labels
          defined in `tools/sphinxext/pyspecific.py' - pydoc needs
          these to show topic and keyword help.

  A "make update" updates the Subversion checkouts in `tools/'.

10.6.2 Without make
-------------------

You'll need to install the Sphinx package, either by checking it out via

    svn co http://svn.python.org/projects/external/Sphinx-0.6.5/sphinx tools/sphinx

or by installing it from PyPI.

  Then, you need to install Docutils, either by checking it out via

    svn co http://svn.python.org/projects/external/docutils-0.6/docutils tools/docutils

or by installing it from <http://docutils.sf.net/>.

  You also need Jinja2, either by checking it out via

    svn co http://svn.python.org/projects/external/Jinja-2.3.1/jinja2 tools/jinja2

or by installing it from PyPI.

  You can optionally also install Pygments, either as a checkout via

    svn co http://svn.python.org/projects/external/Pygments-1.3.1/pygments tools/pygments

or from PyPI at <http://pypi.python.org/pypi/Pygments>.

  Then, make an output directory, e.g. under `build/', and run

    python tools/sphinx-build.py -b<builder> . build/<outputdirectory>

where `<builder>' is one of html, text, latex, or htmlhelp (for
explanations see the make targets above).

11 Python HOWTOs
****************

Python HOWTOs are documents that cover a single, specific topic, and
attempt to cover it fairly completely. Modelled on the Linux
Documentation Project's HOWTO collection, this collection is an effort
to foster documentation that's more detailed than the Python Library
Reference.

  Currently, the HOWTOs are:

11.1 Python Advocacy HOWTO
==========================

     Author: A.M. Kuchling

     Release: 0.03

Abstract
........

     It's usually difficult to get your management to accept open
     source software, and Python is no exception to this rule.  This
     document discusses reasons to use Python, strategies for winning
     acceptance, facts and arguments you can use, and cases where you
     _shouldn't_ try to use Python.

11.1.1 Reasons to Use Python
----------------------------

There are several reasons to incorporate a scripting language into your
development process, and this section will discuss them, and why Python
has some properties that make it a particularly good choice.

11.1.1.1 Programmability
........................

Programs are often organized in a modular fashion.  Lower-level
operations are grouped together, and called by higher-level functions,
which may in turn be used as basic operations by still further upper
levels.

  For example, the lowest level might define a very low-level set of
functions for accessing a hash table.  The next level might use hash
tables to store the headers of a mail message, mapping a header name
like `Date' to a value such as `Tue, 13 May 1997 20:00:54 -0400'.  A
yet higher level may operate on message objects, without knowing or
caring that message headers are stored in a hash table, and so forth.

  Often, the lowest levels do very simple things; they implement a data
structure such as a binary tree or hash table, or they perform some
simple computation, such as converting a date string to a number.  The
higher levels then contain logic connecting these primitive operations.
Using the approach, the primitives can be seen as basic building blocks
which are then glued together to produce the complete product.

  Why is this design approach relevant to Python?  Because Python is
well suited to functioning as such a glue language.  A common approach
is to write a Python module that implements the lower level operations;
for the sake of speed, the implementation might be in C, Java, or even
Fortran.  Once the primitives are available to Python programs, the
logic underlying higher level operations is written in the form of
Python code.  The high-level logic is then more understandable, and
easier to modify.

  John Ousterhout wrote a paper that explains this idea at greater
length, entitled "Scripting: Higher Level Programming for the 21st
Century".  I recommend that you read this paper; see the references for
the URL.  Ousterhout is the inventor of the Tcl language, and therefore
argues that Tcl should be used for this purpose; he only briefly refers
to other languages such as Python, Perl, and Lisp/Scheme, but in
reality, Ousterhout's argument applies to scripting languages in
general, since you could equally write extensions for any of the
languages mentioned above.

11.1.1.2 Prototyping
....................

In _The Mythical Man-Month_, Fredrick Brooks suggests the following
rule when planning software projects: "Plan to throw one away; you will
anyway."  Brooks is saying that the first attempt at a software design
often turns out to be wrong; unless the problem is very simple or
you're an extremely good designer, you'll find that new requirements
and features become apparent once development has actually started.  If
these new requirements can't be cleanly incorporated into the program's
structure, you're presented with two unpleasant choices: hammer the new
features into the program somehow, or scrap everything and write a new
version of the program, taking the new features into account from the
beginning.

  Python provides you with a good environment for quickly developing an
initial prototype.  That lets you get the overall program structure and
logic right, and you can fine-tune small details in the fast
development cycle that Python provides.  Once you're satisfied with the
GUI interface or program output, you can translate the Python code into
C++, Fortran, Java, or some other compiled language.

  Prototyping means you have to be careful not to use too many Python
features that are hard to implement in your other language.  Using
`eval()', or regular expressions, or the pickle module,
means that you're going to need C or Java libraries for formula
evaluation, regular expressions, and serialization, for example.  But
it's not hard to avoid such tricky code, and in the end the translation
usually isn't very difficult.  The resulting code can be rapidly
debugged, because any serious logical errors will have been removed
from the prototype, leaving only more minor slip-ups in the translation
to track down.

  This strategy builds on the earlier discussion of programmability.
Using Python as glue to connect lower-level components has obvious
relevance for constructing prototype systems.  In this way Python can
help you with development, even if end users never come in contact with
Python code at all.  If the performance of the Python version is
adequate and corporate politics allow it, you may not need to do a
translation into C or Java, but it can still be faster to develop a
prototype and then translate it, instead of attempting to produce the
final version immediately.

  One example of this development strategy is Microsoft Merchant
Server. Version 1.0 was written in pure Python, by a company that
subsequently was purchased by Microsoft.  Version 2.0 began to
translate the code into C++, shipping with some C++code and some Python
code.  Version 3.0 didn't contain any Python at all; all the code had
been translated into C++.  Even though the product doesn't contain a
Python interpreter, the Python language has still served a useful
purpose by speeding up development.

  This is a very common use for Python.  Past conference papers have
also described this approach for developing high-level numerical
algorithms; see David M. Beazley and Peter S. Lomdahl's paper "Feeding
a Large-scale Physics Application to Python" in the references for a
good example.  If an algorithm's basic operations are things like "Take
the inverse of this 4000x4000 matrix", and are implemented in some
lower-level language, then Python has almost no additional performance
cost; the extra time required for Python to evaluate an expression like
`m.invert()' is dwarfed by the cost of the actual computation.  It's
particularly good for applications where seemingly endless tweaking is
required to get things right. GUI interfaces and Web sites are prime
examples.

  The Python code is also shorter and faster to write (once you're
familiar with Python), so it's easier to throw it away if you decide
your approach was wrong; if you'd spent two weeks working on it instead
of just two hours, you might waste time trying to patch up what you've
got out of a natural reluctance to admit that those two weeks were
wasted.  Truthfully, those two weeks haven't been wasted, since you've
learnt something about the problem and the technology you're using to
solve it, but it's human nature to view this as a failure of some sort.

11.1.1.3 Simplicity and Ease of Understanding
.............................................

Python is definitely _not_ a toy language that's only usable for small
tasks.  The language features are general and powerful enough to enable
it to be used for many different purposes.  It's useful at the small
end, for 10- or 20-line scripts, but it also scales up to larger
systems that contain thousands of lines of code.

  However, this expressiveness doesn't come at the cost of an obscure
or tricky syntax.  While Python has some dark corners that can lead to
obscure code, there are relatively few such corners, and proper design
can isolate their use to only a few classes or modules.  It's certainly
possible to write confusing code by using too many features with too
little concern for clarity, but most Python code can look a lot like a
slightly-formalized version of human-understandable pseudocode.

  In _The New Hacker's Dictionary_, Eric S. Raymond gives the following
definition for "compact":

     Compact _adj._  Of a design, describes the valuable property that
     it can all be apprehended at once in one's head. This generally
     means the thing created from the design can be used with greater
     facility and fewer errors than an equivalent tool that is not
     compact. Compactness does not imply triviality or lack of power;
     for example, C is compact and FORTRAN is not, but C is more
     powerful than FORTRAN. Designs become non-compact through
     accreting features and cruft that don't merge cleanly into the
     overall design scheme (thus, some fans of Classic C maintain that
     ANSI C is no longer compact).

     (From <http://www.catb.org/~esr/jargon/html/C/compact.html>)

  In this sense of the word, Python is quite compact, because the
language has just a few ideas, which are used in lots of places.  Take
namespaces, for example.  Import a module with `import math', and you
create a new namespace called `math'.  Classes are also namespaces that
share many of the properties of modules, and have a few of their own;
for example, you can create instances of a class. Instances?  They're
yet another namespace.  Namespaces are currently implemented as Python
dictionaries, so they have the same methods as the standard dictionary
data type: .keys() returns all the keys, and so forth.

  This simplicity arises from Python's development history.  The
language syntax derives from different sources; ABC, a relatively
obscure teaching language, is one primary influence, and Modula-3 is
another.  (For more information about ABC and Modula-3, consult their
respective Web sites at <http://www.cwi.nl/~steven/abc/> and
<http://www.m3.org>.)  Other features have come from C, Icon, Algol-68,
and even Perl.  Python hasn't really innovated very much, but instead
has tried to keep the language small and easy to learn, building on
ideas that have been tried in other languages and found useful.

  Simplicity is a virtue that should not be underestimated.  It lets
you learn the language more quickly, and then rapidly write code - code
that often works the first time you run it.

11.1.1.4 Java Integration
.........................

If you're working with Java, Jython (<http://www.jython.org/>) is
definitely worth your attention.  Jython is a re-implementation of
Python in Java that compiles Python code into Java bytecodes.  The
resulting environment has very tight, almost seamless, integration with
Java.  It's trivial to access Java classes from Python, and you can
write Python classes that subclass Java classes.  Jython can be used
for prototyping Java applications in much the same way CPython is used,
and it can also be used for test suites for Java code, or embedded in a
Java application to add scripting capabilities.

11.1.2 Arguments and Rebuttals
------------------------------

Let's say that you've decided upon Python as the best choice for your
application.  How can you convince your management, or your fellow
developers, to use Python?  This section lists some common arguments
against using Python, and provides some possible rebuttals.

  *Python is freely available software that doesn't cost anything. How
good can it be?*

  Very good, indeed.  These days Linux and Apache, two other pieces of
open source software, are becoming more respected as alternatives to
commercial software, but Python hasn't had all the publicity.

  Python has been around for several years, with many users and
developers.  Accordingly, the interpreter has been used by many people,
and has gotten most of the bugs shaken out of it.  While bugs are still
discovered at intervals, they're usually either quite obscure (they'd
have to be, for no one to have run into them before) or they involve
interfaces to external libraries.  The internals of the language itself
are quite stable.

  Having the source code should be viewed as making the software
available for peer review; people can examine the code, suggest (and
implement) improvements, and track down bugs.  To find out more about
the idea of open source code, along with arguments and case studies
supporting it, go to <http://www.opensource.org>.

  *Who's going to support it?*

  Python has a sizable community of developers, and the number is still
growing.  The Internet community surrounding the language is an active
one, and is worth being considered another one of Python's advantages.
Most questions posted to the comp.lang.python newsgroup are quickly
answered by someone.

  Should you need to dig into the source code, you'll find it's clear
and well-organized, so it's not very difficult to write extensions and
track down bugs yourself.  If you'd prefer to pay for support, there
are companies and individuals who offer commercial support for Python.

  *Who uses Python for serious work?*

  Lots of people; one interesting thing about Python is the surprising
diversity of applications that it's been used for.  People are using
Python to:

   * Run Web sites

   * Write GUI interfaces

   * Control number-crunching code on supercomputers

   * Make a commercial application scriptable by embedding the Python
     interpreter inside it

   * Process large XML data sets

   * Build test suites for C or Java code

  Whatever your application domain is, there's probably someone who's
used Python for something similar.  Yet, despite being useable for such
high-end applications, Python's still simple enough to use for little
jobs.

  See <http://wiki.python.org/moin/OrganizationsUsingPython> for a list
of some of the  organizations that use Python.

  *What are the restrictions on Python's use?*

  They're practically nonexistent.  Consult the `Misc/COPYRIGHT' file
in the source distribution, or the section History and License
for the full language, but it boils down to three conditions:

   * You have to leave the copyright notice on the software; if you
     don't include the source code in a product, you have to put the
     copyright notice in the supporting documentation.

   * Don't claim that the institutions that have developed Python
     endorse your product in any way.

   * If something goes wrong, you can't sue for damages.  Practically
     all software licenses contain this condition.

  Notice that you don't have to provide source code for anything that
contains Python or is built with it.  Also, the Python interpreter and
accompanying documentation can be modified and redistributed in any way
you like, and you don't have to pay anyone any licensing fees at all.

  *Why should we use an obscure language like Python instead of
well-known language X?*

  I hope this HOWTO, and the documents listed in the final section,
will help convince you that Python isn't obscure, and has a healthily
growing user base.  One word of advice: always present Python's
positive advantages, instead of concentrating on language X's failings.
People want to know why a solution is good, rather than why all the
other solutions are bad.  So instead of attacking a competing solution
on various grounds, simply show how Python's virtues can help.

11.1.3 Useful Resources
-----------------------

<http://www.pythonology.com/success>
     The Python Success Stories are a collection of stories from
     successful users of Python, with the emphasis on business and
     corporate users.

<http://home.pacbell.net/ouster/scripting.html>
     John Ousterhout's white paper on scripting is a good argument for
     the utility of scripting languages, though naturally enough, he
     emphasizes Tcl, the language he developed.  Most of the arguments
     would apply to any scripting language.

<http://www.python.org/workshops/1997-10/proceedings/beazley.html>
     The authors, David M. Beazley and Peter S. Lomdahl,  describe
     their use of Python at Los Alamos National Laboratory. It's
     another good example of how Python can help get real work done.
     This quotation from the paper has been echoed by many people:

          Originally developed as a large monolithic application for
          massively parallel processing systems, we have used Python to
          transform our application into a flexible, highly modular,
          and extremely powerful system for performing simulation, data
          analysis, and visualization. In addition, we describe how
          Python has solved a number of important problems related to
          the development, debugging, deployment, and maintenance of
          scientific software.

<http://pythonjournal.cognizor.com/pyj1/Everitt-Feit_interview98-V1.html>
     This interview with Andy Feit, discussing Infoseek's use of
     Python, can be used to show that choosing Python didn't introduce
     any difficulties into a company's development process, and
     provided some substantial benefits.

<http://www.python.org/workshops/1997-10/proceedings/stein.ps>
     For the 6th Python conference, Greg Stein presented a paper that
     traced Python's adoption and usage at a startup called eShop, and
     later at Microsoft.

<http://www.opensource.org>
     Management may be doubtful of the reliability and usefulness of
     software that wasn't written commercially.  This site presents
     arguments that show how open source software can have considerable
     advantages over closed-source software.

<http://www.faqs.org/docs/Linux-mini/Advocacy.html>
     The Linux Advocacy mini-HOWTO was the inspiration for this
     document, and is also well worth reading for general suggestions
     on winning acceptance for a new technology, such as Linux or
     Python.  In general, you won't make much progress by simply
     attacking existing systems and complaining about their
     inadequacies; this often ends up looking like unfocused whining.
     It's much better to point out some of the many areas where Python
     is an improvement over other systems.

11.2 Porting Extension Modules to 3.0
=====================================

     author: Benjamin Peterson

Abstract
........

     Although changing the C-API was not one of Python 3.0's
     objectives, the many Python level changes made leaving 2.x's API
     intact impossible.  In fact, some changes such as int()
     and long() unification are more obvious on the C
     level.  This document endeavors to document incompatibilities and
     how they can be worked around.

11.2.1 Conditional compilation
------------------------------

The easiest way to compile only some code for 3.0 is to check if
`PY_MAJOR_VERSION' is greater than or equal to 3.

    #if PY_MAJOR_VERSION >= 3
    #define IS_PY3K
    #endif

API functions that are not present can be aliased to their equivalents
within conditional blocks.

11.2.2 Changes to Object APIs
-----------------------------

Python 3.0 merged together some types with similar functions while
cleanly separating others.

11.2.2.1 str/unicode Unification
................................

Python 3.0's str(). (`PyString_*' functions in C) type is
equivalent to 2.x's unicode(). (`PyUnicode_*').  The old
8-bit string type has become `bytes()'.  Python 2.6 and later provide a
compatibility header, `bytesobject.h', mapping `PyBytes' names to
`PyString' ones.  For best compatibility with 3.0, `PyUnicode' should
be used for textual data and `PyBytes' for binary data.  It's also
important to remember that `PyBytes' and `PyUnicode' in 3.0 are not
interchangeable like `PyString' and `PyUnicode' are in 2.x.  The
following example shows best practices with regards to `PyUnicode',
`PyString', and `PyBytes'.

    #include "stdlib.h"
    #include "Python.h"
    #include "bytesobject.h"

    /* text example */
    static PyObject *
    say_hello(PyObject *self, PyObject *args) {
        PyObject *name, *result;

        if (!PyArg_ParseTuple(args, "U:say_hello", &name))
            return NULL;

        result = PyUnicode_FromFormat("Hello, %S!", name);
        return result;
    }

    /* just a forward */
    static char * do_encode(PyObject *);

    /* bytes example */
    static PyObject *
    encode_object(PyObject *self, PyObject *args) {
        char *encoded;
        PyObject *result, *myobj;

        if (!PyArg_ParseTuple(args, "O:encode_object", &myobj))
            return NULL;

        encoded = do_encode(myobj);
        if (encoded == NULL)
            return NULL;
        result = PyBytes_FromString(encoded);
        free(encoded);
        return result;
    }

11.2.2.2 long/int Unification
.............................

In Python 3.0, there is only one integer type.  It is called
int() on the Python level, but actually corresponds to 2.x's
long() type.  In the C-API, `PyInt_*' functions are
replaced by their `PyLong_*' neighbors.  The best course of action here
is using the `PyInt_*' functions aliased to `PyLong_*' found in
`intobject.h'.  The abstract `PyNumber_*' APIs can also be used in some
cases.

    #include "Python.h"
    #include "intobject.h"

    static PyObject *
    add_ints(PyObject *self, PyObject *args) {
        int one, two;
        PyObject *result;

        if (!PyArg_ParseTuple(args, "ii:add_ints", &one, &two))
            return NULL;

        return PyInt_FromLong(one + two);
    }

11.2.3 Module initialization and state
--------------------------------------

Python 3.0 has a revamped extension module initialization system.  (See PEP
3121 (http://www.python.org/dev/peps/pep-3121).)  Instead of storing
module state in globals, they should be stored in an interpreter
specific structure.  Creating modules that act correctly in both 2.x
and 3.0 is tricky.  The following simple example demonstrates how.

    #include "Python.h"

    struct module_state {
        PyObject *error;
    };

    #if PY_MAJOR_VERSION >= 3
    #define GETSTATE(m) ((struct module_state*)PyModule_GetState(m))
    #else
    #define GETSTATE(m) (&_state)
    static struct module_state _state;
    #endif

    static PyObject *
    error_out(PyObject *m) {
        struct module_state *st = GETSTATE(m);
        PyErr_SetString(st->error, "something bad happened");
        return NULL;
    }

    static PyMethodDef myextension_methods[] = {
        {"error_out", (PyCFunction)error_out, METH_NOARGS, NULL},
        {NULL, NULL}
    };

    #if PY_MAJOR_VERSION >= 3

    static int myextension_traverse(PyObject *m, visitproc visit, void *arg) {
        Py_VISIT(GETSTATE(m)->error);
        return 0;
    }

    static int myextension_clear(PyObject *m) {
        Py_CLEAR(GETSTATE(m)->error);
        return 0;
    }

    static struct PyModuleDef moduledef = {
            PyModuleDef_HEAD_INIT,
            "myextension",
            NULL,
            sizeof(struct module_state),
            myextension_methods,
            NULL,
            myextension_traverse,
            myextension_clear,
            NULL
    };

    #define INITERROR return NULL

    PyObject *
    PyInit_myextension(void)

    #else
    #define INITERROR return

    void
    initmyextension(void)
    #endif
    {
    #if PY_MAJOR_VERSION >= 3
        PyObject *module = PyModule_Create(&moduledef);
    #else
        PyObject *module = Py_InitModule("myextension", myextension_methods);
    #endif

        if (module == NULL)
            INITERROR;
        struct module_state *st = GETSTATE(module);

        st->error = PyErr_NewException("myextension.Error", NULL, NULL);
        if (st->error == NULL) {
            Py_DECREF(module);
            INITERROR;
        }

    #if PY_MAJOR_VERSION >= 3
        return module;
    #endif
    }

11.2.4 Other options
--------------------

If you are writing a new extension module, you might consider Cython
(http://www.cython.org).  It translates a Python-like language to C.
The extension modules it creates are compatible with Python 3.x and 2.x.

11.3 Curses Programming with Python
===================================

     Author: A.M. Kuchling, Eric S. Raymond

     Release: 2.03

Abstract
........

     This document describes how to write text-mode programs with
     Python 2.x, using the curses extension module to
     control the display.

11.3.1 What is curses?
----------------------

The curses library supplies a terminal-independent screen-painting and
keyboard-handling facility for text-based terminals; such terminals
include VT100s, the Linux console, and the simulated terminal provided
by X11 programs such as xterm and rxvt.  Display terminals support
various control codes to perform common operations such as moving the
cursor, scrolling the screen, and erasing areas.  Different terminals
use widely differing codes, and often have their own minor quirks.

  In a world of X displays, one might ask "why bother"?  It's true that
character-cell display terminals are an obsolete technology, but there
are niches in which being able to do fancy things with them are still
valuable.  One is on small-footprint or embedded Unixes that don't
carry an X server.  Another is for tools like OS installers and kernel
configurators that may have to run before X is available.

  The curses library hides all the details of different terminals, and
provides the programmer with an abstraction of a display, containing
multiple non-overlapping windows.  The contents of a window can be
changed in various ways- adding text, erasing it, changing its
appearance-and the curses library will automagically figure out what
control codes need to be sent to the terminal to produce the right
output.

  The curses library was originally written for BSD Unix; the later
System V versions of Unix from AT&T added many enhancements and new
functions. BSD curses is no longer maintained, having been replaced by
ncurses, which is an open-source implementation of the AT&T interface.
If you're using an open-source Unix such as Linux or FreeBSD, your
system almost certainly uses ncurses.  Since most current commercial
Unix versions are based on System V code, all the functions described
here will probably be available.  The older versions of curses carried
by some proprietary Unixes may not support everything, though.

  No one has made a Windows port of the curses module.  On a Windows
platform, try the Console module written by Fredrik Lundh.  The Console
module provides cursor-addressable text output, plus full support for
mouse and keyboard input, and is available from
<http://effbot.org/zone/console-index.htm>.

11.3.1.1 The Python curses module
.................................

Thy Python module is a fairly simple wrapper over the C functions
provided by curses; if you're already familiar with curses programming
in C, it's really easy to transfer that knowledge to Python.  The
biggest difference is that the Python interface makes things simpler,
by merging different C functions such as `addstr()', `mvaddstr()',
`mvwaddstr()', into a single `addstr()' method.  You'll see this
covered in more detail later.

  This HOWTO is simply an introduction to writing text-mode programs
with curses and Python. It doesn't attempt to be a complete guide to
the curses API; for that, see the Python library guide's section on
ncurses, and the C manual pages for ncurses.  It will, however, give
you the basic ideas.

11.3.2 Starting and ending a curses application
-----------------------------------------------

Before doing anything, curses must be initialized.  This is done by
calling the `initscr()' function, which will determine the terminal
type, send any required setup codes to the terminal, and create various
internal data structures.  If successful, `initscr()' returns a window
object representing the entire screen; this is usually called `stdscr',
after the name of the corresponding C variable.

    import curses
    stdscr = curses.initscr()

Usually curses applications turn off automatic echoing of keys to the
screen, in order to be able to read keys and only display them under
certain circumstances.  This requires calling the `noecho()' function.

    curses.noecho()

Applications will also commonly need to react to keys instantly, without
requiring the Enter key to be pressed; this is called cbreak mode, as
opposed to the usual buffered input mode.

    curses.cbreak()

Terminals usually return special keys, such as the cursor keys or
navigation keys such as Page Up and Home, as a multibyte escape
sequence.  While you could write your application to expect such
sequences and process them accordingly, curses can do it for you,
returning a special value such as `curses.KEY_LEFT'.  To get curses to
do the job, you'll have to enable keypad mode.

    stdscr.keypad(1)

Terminating a curses application is much easier than starting one.
You'll need to call

    curses.nocbreak(); stdscr.keypad(0); curses.echo()

to reverse the curses-friendly terminal settings. Then call the
`endwin()' function to restore the terminal to its original operating
mode.

    curses.endwin()

A common problem when debugging a curses application is to get your
terminal messed up when the application dies without restoring the
terminal to its previous state.  In Python this commonly happens when
your code is buggy and raises an uncaught exception.  Keys are no
longer be echoed to the screen when you type them, for example, which
makes using the shell difficult.

  In Python you can avoid these complications and make debugging much
easier by importing the module curses.wrapper.  It supplies a
`wrapper()' function that takes a callable.  It does the
initializations described above, and also initializes colors if color
support is present.  It then runs your provided callable and finally
deinitializes appropriately.  The callable is called inside a try-catch
clause which catches exceptions, performs curses deinitialization, and
then passes the exception upwards.  Thus, your terminal won't be left
in a funny state on exception.

11.3.3 Windows and Pads
-----------------------

Windows are the basic abstraction in curses.  A window object
represents a rectangular area of the screen, and supports various
methods to display text, erase it, allow the user to input strings, and
so forth.

  The `stdscr' object returned by the `initscr()' function is a window
object that covers the entire screen.  Many programs may need only this
single window, but you might wish to divide the screen into smaller
windows, in order to redraw or clear them separately. The `newwin()'
function creates a new window of a given size, returning the new window
object.

    begin_x = 20 ; begin_y = 7
    height = 5 ; width = 40
    win = curses.newwin(height, width, begin_y, begin_x)

A word about the coordinate system used in curses: coordinates are
always passed in the order _y,x_, and the top-left corner of a window
is coordinate (0,0).  This breaks a common convention for handling
coordinates, where the _x_ coordinate usually comes first.  This is an
unfortunate difference from most other computer applications, but it's
been part of curses since it was first written, and it's too late to
change things now.

  When you call a method to display or erase text, the effect doesn't
immediately show up on the display.  This is because curses was
originally written with slow 300-baud terminal connections in mind;
with these terminals, minimizing the time required to redraw the screen
is very important.  This lets curses accumulate changes to the screen,
and display them in the most efficient manner.  For example, if your
program displays some characters in a window, and then clears the
window, there's no need to send the original characters because they'd
never be visible.

  Accordingly, curses requires that you explicitly tell it to redraw
windows, using the `refresh()' method of window objects.  In practice,
this doesn't really complicate programming with curses much. Most
programs go into a flurry of activity, and then pause waiting for a
keypress or some other action on the part of the user.  All you have to
do is to be sure that the screen has been redrawn before pausing to
wait for user input, by simply calling `stdscr.refresh()' or the
`refresh()' method of some other relevant window.

  A pad is a special case of a window; it can be larger than the actual
display screen, and only a portion of it displayed at a time. Creating
a pad simply requires the pad's height and width, while refreshing a
pad requires giving the coordinates of the on-screen area where a
subsection of the pad will be displayed.

    pad = curses.newpad(100, 100)
    #  These loops fill the pad with letters; this is
    # explained in the next section
    for y in range(0, 100):
        for x in range(0, 100):
            try: pad.addch(y,x, ord('a') + (x*x+y*y) % 26 )
            except curses.error: pass

    #  Displays a section of the pad in the middle of the screen
    pad.refresh( 0,0, 5,5, 20,75)

The `refresh()' call displays a section of the pad in the rectangle
extending from coordinate (5,5) to coordinate (20,75) on the screen;
the upper left corner of the displayed section is coordinate (0,0) on
the pad.  Beyond that difference, pads are exactly like ordinary
windows and support the same methods.

  If you have multiple windows and pads on screen there is a more
efficient way to go, which will prevent annoying screen flicker at
refresh time.  Use the `noutrefresh()' method of each window to update
the data structure representing the desired state of the screen; then
change the physical screen to match the desired state in one go with
the function `doupdate()'.  The normal `refresh()' method calls
`doupdate()' as its last act.

11.3.4 Displaying Text
----------------------

From a C programmer's point of view, curses may sometimes look like a
twisty maze of functions, all subtly different.  For example,
`addstr()' displays a string at the current cursor location in the
`stdscr' window, while `mvaddstr()' moves to a given y,x coordinate
first before displaying the string. `waddstr()' is just like
`addstr()', but allows specifying a window to use, instead of using
`stdscr' by default. `mvwaddstr()' follows similarly.

  Fortunately the Python interface hides all these details; `stdscr' is
a window object like any other, and methods like `addstr()' accept
multiple argument forms.  Usually there are four different forms.

Form                                  Description
------------------------------------------------------------------------------------------
_str_ or _ch_                         Display the string _str_ or character _ch_ at the
                                      current position
_str_ or _ch_, _attr_                 Display the string _str_ or character _ch_, using
                                      attribute _attr_ at the current position
_y_, _x_, _str_ or _ch_               Move to position _y,x_ within the window, and
                                      display _str_ or _ch_
_y_, _x_, _str_ or _ch_, _attr_       Move to position _y,x_ within the window, and
                                      display _str_ or _ch_, using attribute _attr_

  Attributes allow displaying text in highlighted forms, such as in
boldface, underline, reverse code, or in color.  They'll be explained
in more detail in the next subsection.

  The `addstr()' function takes a Python string as the value to be
displayed, while the `addch()' functions take a character, which can be
either a Python string of length 1 or an integer.  If it's a string,
you're limited to displaying characters between 0 and 255.  SVr4 curses
provides constants for extension characters; these constants are
integers greater than 255.  For example, `ACS_PLMINUS' is a +/- symbol,
and `ACS_ULCORNER' is the upper left corner of a box (handy for drawing
borders).

  Windows remember where the cursor was left after the last operation,
so if you leave out the _y,x_ coordinates, the string or character will
be displayed wherever the last operation left off.  You can also move
the cursor with the `move(y,x)' method.  Because some terminals always
display a flashing cursor, you may want to ensure that the cursor is
positioned in some location where it won't be distracting; it can be
confusing to have the cursor blinking at some apparently random
location.

  If your application doesn't need a blinking cursor at all, you can
call `curs_set(0)' to make it invisible.  Equivalently, and for
compatibility with older curses versions, there's a `leaveok(bool)'
function.  When _bool_ is true, the curses library will attempt to
suppress the flashing cursor, and you won't need to worry about leaving
it in odd locations.

11.3.4.1 Attributes and Color
.............................

Characters can be displayed in different ways.  Status lines in a
text-based application are commonly shown in reverse video; a text
viewer may need to highlight certain words.  curses supports this by
allowing you to specify an attribute for each cell on the screen.

  An attribute is a integer, each bit representing a different
attribute.  You can try to display text with multiple attribute bits
set, but curses doesn't guarantee that all the possible combinations
are available, or that they're all visually distinct.  That depends on
the ability of the terminal being used, so it's safest to stick to the
most commonly available attributes, listed here.

Attribute                  Description
----------------------------------------------------------------------
`A_BLINK'                  Blinking text
`A_BOLD'                   Extra bright or bold text
`A_DIM'                    Half bright text
`A_REVERSE'                Reverse-video text
`A_STANDOUT'               The best highlighting mode available
`A_UNDERLINE'              Underlined text

  So, to display a reverse-video status line on the top line of the
screen, you could code:

    stdscr.addstr(0, 0, "Current mode: Typing mode",
                  curses.A_REVERSE)
    stdscr.refresh()

The curses library also supports color on those terminals that provide
it, The most common such terminal is probably the Linux console,
followed by color xterms.

  To use color, you must call the `start_color()' function soon after
calling `initscr()', to initialize the default color set (the
curses.wrapper.wrapper() function does this automatically).
Once that's done, the `has_colors()' function returns TRUE if the
terminal in use can actually display color.  (Note: curses uses the
American spelling 'color', instead of the Canadian/British spelling
'colour'.  If you're used to the British spelling, you'll have to
resign yourself to misspelling it for the sake of these functions.)

  The curses library maintains a finite number of color pairs,
containing a foreground (or text) color and a background color.  You
can get the attribute value corresponding to a color pair with the
`color_pair()' function; this can be bitwise-OR'ed with other
attributes such as `A_REVERSE', but again, such combinations are not
guaranteed to work on all terminals.

  An example, which displays a line of text using color pair 1:

    stdscr.addstr( "Pretty text", curses.color_pair(1) )
    stdscr.refresh()

As I said before, a color pair consists of a foreground and background
color.  `start_color()' initializes 8 basic colors when it activates
color mode.  They are: 0:black, 1:red, 2:green, 3:yellow, 4:blue,
5:magenta, 6:cyan, and 7:white.  The curses module defines named
constants for each of these colors: `curses.COLOR_BLACK',
`curses.COLOR_RED', and so forth.

  The `init_pair(n, f, b)' function changes the definition of color
pair _n_, to foreground color f and background color b.  Color pair 0
is hard-wired to white on black, and cannot be changed.

  Let's put all this together. To change color 1 to red text on a white
background, you would call:

    curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)

When you change a color pair, any text already displayed using that
color pair will change to the new colors.  You can also display new
text in this color with:

    stdscr.addstr(0,0, "RED ALERT!", curses.color_pair(1) )

Very fancy terminals can change the definitions of the actual colors to
a given RGB value.  This lets you change color 1, which is usually red,
to purple or blue or any other color you like.  Unfortunately, the
Linux console doesn't support this, so I'm unable to try it out, and
can't provide any examples.  You can check if your terminal can do this
by calling `can_change_color()', which returns TRUE if the capability
is there.  If you're lucky enough to have such a talented terminal,
consult your system's man pages for more information.

11.3.5 User Input
-----------------

The curses library itself offers only very simple input mechanisms.
Python's support adds a text-input widget that makes up some of the
lack.

  The most common way to get input to a window is to use its `getch()'
method.  `getch()' pauses and waits for the user to hit a key,
displaying it if `echo()' has been called earlier.  You can optionally
specify a coordinate to which the cursor should be moved before pausing.

  It's possible to change this behavior with the method `nodelay()'.
After `nodelay(1)', `getch()' for the window becomes non-blocking and
returns `curses.ERR' (a value of -1) when no input is ready.  There's
also a `halfdelay()' function, which can be used to (in effect) set a
timer on each `getch()'; if no input becomes available within a
specified delay (measured in tenths of a second), curses raises an
exception.

  The `getch()' method returns an integer; if it's between 0 and 255, it
represents the ASCII code of the key pressed.  Values greater than 255
are special keys such as Page Up, Home, or the cursor keys. You can
compare the value returned to constants such as `curses.KEY_PPAGE',
`curses.KEY_HOME', or `curses.KEY_LEFT'.  Usually the main loop of your
program will look something like this:

    while 1:
        c = stdscr.getch()
        if c == ord('p'): PrintDocument()
        elif c == ord('q'): break  # Exit the while()
        elif c == curses.KEY_HOME: x = y = 0

The curses.ascii module supplies ASCII class membership
functions that take either integer or 1-character-string arguments;
these may be useful in writing more readable tests for your command
interpreters.  It also supplies conversion functions  that take either
integer or 1-character-string arguments and return the same type.  For
example, see curses.ascii.ctrl() returns the control character
corresponding to its argument.

  There's also a method to retrieve an entire string, `getstr()'.  It
isn't used very often, because its functionality is quite limited; the
only editing keys available are the backspace key and the Enter key,
which terminates the string.  It can optionally be limited to a fixed
number of characters.

    curses.echo()            # Enable echoing of characters

    # Get a 15-character string, with the cursor on the top line
    s = stdscr.getstr(0,0, 15)

The Python curses.textpad module supplies something better.
With it, you can turn a window into a text box that supports an
Emacs-like set of keybindings.  Various methods of `Textbox' class
support editing with input validation and gathering the edit results
either with or without trailing spaces.   See the library documentation
on curses.textpad for the details.

11.3.6 For More Information
---------------------------

This HOWTO didn't cover some advanced topics, such as screen-scraping or
capturing mouse events from an xterm instance.  But the Python library
page for the curses modules is now pretty complete.  You should browse
it next.

  If you're in doubt about the detailed behavior of any of the ncurses
entry points, consult the manual pages for your curses implementation,
whether it's ncurses or a proprietary Unix vendor's.  The manual pages
will document any quirks, and provide complete lists of all the
functions, attributes, and `ACS_*' characters available to you.

  Because the curses API is so large, some functions aren't supported
in the Python interface, not because they're difficult to implement,
but because no one has needed them yet.  Feel free to add them and then
submit a patch.  Also, we don't yet have support for the menu library
associated with ncurses; feel free to add that.

  If you write an interesting little program, feel free to contribute
it as another demo.  We can always use more of them!

  The ncurses FAQ:
<http://invisible-island.net/ncurses/ncurses.faq.html>

11.4 Descriptor HowTo Guide
===========================

     Author: Raymond Hettinger

     Contact: <python at rcn dot com>

11.4.1 Abstract
---------------

Defines descriptors, summarizes the protocol, and shows how descriptors
are called.  Examines a custom descriptor and several built-in python
descriptors including functions, properties, static methods, and class
methods.  Shows how each works by giving a pure Python equivalent and a
sample application.

  Learning about descriptors not only provides access to a larger
toolset, it creates a deeper understanding of how Python works and an
appreciation for the elegance of its design.

11.4.2 Definition and Introduction
----------------------------------

In general, a descriptor is an object attribute with "binding
behavior", one whose attribute access has been overridden by methods in
the descriptor protocol.  Those methods are __get__(),
__set__(), and __delete__().  If any of those methods
are defined for an object, it is said to be a descriptor.

  The default behavior for attribute access is to get, set, or delete
the attribute from an object's dictionary.  For instance, `a.x' has a
lookup chain starting with `a.__dict__['x']', then
`type(a).__dict__['x']', and continuing through the base classes of
`type(a)' excluding metaclasses. If the looked-up value is an object
defining one of the descriptor methods, then Python may override the
default behavior and invoke the descriptor method instead.  Where this
occurs in the precedence chain depends on which descriptor methods were
defined.  Note that descriptors are only invoked for new style objects
or classes (a class is new style if it inherits from object.
or type).

  Descriptors are a powerful, general purpose protocol.  They are the
mechanism behind properties, methods, static methods, class methods,
and super().  They are used used throughout Python itself to
implement the new style classes introduced in version 2.2.  Descriptors
simplify the underlying C-code and offer a flexible set of new tools
for everyday Python programs.

11.4.3 Descriptor Protocol
--------------------------

`descr.__get__(self, obj, type=None) --> value'

  `descr.__set__(self, obj, value) --> None'

  `descr.__delete__(self, obj) --> None'

  That is all there is to it.  Define any of these methods and an
object is considered a descriptor and can override default behavior
upon being looked up as an attribute.

  If an object defines both __get__() and __set__(),
it is considered a data descriptor.  Descriptors that only define
__get__() are called non-data descriptors (they are
typically used for methods but other uses are possible).

  Data and non-data descriptors differ in how overrides are calculated
with respect to entries in an instance's dictionary.  If an instance's
dictionary has an entry with the same name as a data descriptor, the
data descriptor takes precedence.  If an instance's dictionary has an
entry with the same name as a non-data descriptor, the dictionary entry
takes precedence.

  To make a read-only data descriptor, define both __get__()
and __set__() with the __set__() raising
an AttributeError when called.  Defining the
__set__() method with an exception raising placeholder is enough
to make it a data descriptor.

11.4.4 Invoking Descriptors
---------------------------

A descriptor can be called directly by its method name.  For example,
`d.__get__(obj)'.

  Alternatively, it is more common for a descriptor to be invoked
automatically upon attribute access.  For example, `obj.d' looks up `d'
in the dictionary of `obj'.  If `d' defines the method __get__(),
then `d.__get__(obj)' is invoked according to the precedence rules
listed below.

  The details of invocation depend on whether `obj' is an object or a
class.  Either way, descriptors only work for new style objects and
classes.  A class is new style if it is a subclass of object.

  For objects, the machinery is in object.__getattribute__()
which transforms `b.x' into `type(b).__dict__['x'].__get__(b,
type(b))'.  The implementation works through a precedence chain that
gives data descriptors priority over instance variables, instance
variables priority over non-data descriptors, and assigns lowest
priority to __getattr__() if provided.  The full C
implementation can be found in PyObject_GenericGetAttr() in
Objects/object.c
(http://svn.python.org/view/python/trunk/Objects/object.c?view=markup).

  For classes, the machinery is in `type.__getattribute__()' which
transforms `B.x' into `B.__dict__['x'].__get__(None, B)'.  In pure
Python, it looks like:

    def __getattribute__(self, key):
        "Emulate type_getattro() in Objects/typeobject.c"
        v = object.__getattribute__(self, key)
        if hasattr(v, '__get__'):
           return v.__get__(None, self)
        return v

The important points to remember are:

   * descriptors are invoked by the __getattribute__().
     method

   * overriding __getattribute__() prevents automatic
     descriptor calls

   * __getattribute__() is only available with new style
     classes and objects

   * object.__getattribute__() and
     `type.__getattribute__()' make different calls to __get__().

   * data descriptors always override instance dictionaries.

   * non-data descriptors may be overridden by instance dictionaries.

  The object returned by `super()' also has a custom
__getattribute__().  method for invoking descriptors.  The call
`super(B, obj).m()' searches `obj.__class__.__mro__' for the base class
`A' immediately following `B' and then returns
`A.__dict__['m'].__get__(obj, A)'.  If not a descriptor, `m' is
returned unchanged.  If not in the dictionary, `m' reverts to a search
using object.__getattribute__().

  Note, in Python 2.2, `super(B, obj).m()' would only invoke
__get__() if `m' was a data descriptor.  In Python 2.3, non-data
descriptors also get invoked unless an old-style class is involved.
The implementation details are in `super_getattro()' in
Objects/typeobject.c
(http://svn.python.org/view/python/trunk/Objects/typeobject.c?view=markup)and
a pure Python equivalent can be found in Guido's Tutorial
(http://www.python.org/2.2.3/descrintro.html#cooperation).

  The details above show that the mechanism for descriptors is embedded
in the __getattribute__() methods for object,
type, and super().  Classes inherit this
machinery when they derive from object or if they have a
meta-class providing similar functionality.  Likewise, classes can
turn-off descriptor invocation by overriding __getattribute__().

11.4.5 Descriptor Example
-------------------------

The following code creates a class whose objects are data descriptors
which print a message for each get or set.  Overriding
__getattribute__() is alternate approach that could do this for
every attribute.  However, this descriptor is useful for monitoring
just a few chosen attributes:

    class RevealAccess(object):
        """A data descriptor that sets and returns values
           normally and prints a message logging their access.
        """

        def __init__(self, initval=None, name='var'):
            self.val = initval
            self.name = name

        def __get__(self, obj, objtype):
            print 'Retrieving', self.name
            return self.val

        def __set__(self, obj, val):
            print 'Updating' , self.name
            self.val = val

    >>> class MyClass(object):
        x = RevealAccess(10, 'var "x"')
        y = 5

    >>> m = MyClass()
    >>> m.x
    Retrieving var "x"
    10
    >>> m.x = 20
    Updating var "x"
    >>> m.x
    Retrieving var "x"
    20
    >>> m.y
    5

The protocol is simple and offers exciting possibilities.  Several use
cases are so common that they have been packaged into individual
function calls.  Properties, bound and unbound methods, static methods,
and class methods are all based on the descriptor protocol.

11.4.6 Properties
-----------------

Calling property() is a succinct way of building a data
descriptor that triggers function calls upon access to an attribute.
Its signature is:

    property(fget=None, fset=None, fdel=None, doc=None) -> property attribute

The documentation shows a typical use to define a managed attribute `x':

    class C(object):
        def getx(self): return self.__x
        def setx(self, value): self.__x = value
        def delx(self): del self.__x
        x = property(getx, setx, delx, "I'm the 'x' property.")

To see how property() is implemented in terms of the
descriptor protocol, here is a pure Python equivalent:

    class Property(object):
        "Emulate PyProperty_Type() in Objects/descrobject.c"

        def __init__(self, fget=None, fset=None, fdel=None, doc=None):
            self.fget = fget
            self.fset = fset
            self.fdel = fdel
            self.__doc__ = doc

        def __get__(self, obj, objtype=None):
            if obj is None:
                return self
            if self.fget is None:
                raise AttributeError, "unreadable attribute"
            return self.fget(obj)

        def __set__(self, obj, value):
            if self.fset is None:
                raise AttributeError, "can't set attribute"
            self.fset(obj, value)

        def __delete__(self, obj):
            if self.fdel is None:
                raise AttributeError, "can't delete attribute"
            self.fdel(obj)

The property() builtin helps whenever a user interface has
granted attribute access and then subsequent changes require the
intervention of a method.

  For instance, a spreadsheet class may grant access to a cell value
through `Cell('b10').value'. Subsequent improvements to the program
require the cell to be recalculated on every access; however, the
programmer does not want to affect existing client code accessing the
attribute directly.  The solution is to wrap access to the value
attribute in a property data descriptor:

    class Cell(object):
        . . .
        def getvalue(self, obj):
            "Recalculate cell before returning value"
            self.recalc()
            return obj._value
        value = property(getvalue)

11.4.7 Functions and Methods
----------------------------

Python's object oriented features are built upon a function based
environment.  Using non-data descriptors, the two are merged seamlessly.

  Class dictionaries store methods as functions.  In a class
definition, methods are written using def and lambda,
the usual tools for creating functions.  The only difference from
regular functions is that the first argument is reserved for the object
instance.  By Python convention, the instance reference is called
_self_ but may be called _this_ or any other variable name.

  To support method calls, functions include the __get__().
method for binding methods during attribute access.  This means that
all functions are non-data descriptors which return bound or unbound
methods depending whether they are invoked from an object or a class.
In pure python, it works like this:

    class Function(object):
        . . .
        def __get__(self, obj, objtype=None):
            "Simulate func_descr_get() in Objects/funcobject.c"
            return types.MethodType(self, obj, objtype)

Running the interpreter shows how the function descriptor works in
practice:

    >>> class D(object):
         def f(self, x):
              return x

    >>> d = D()
    >>> D.__dict__['f'] # Stored internally as a function
    <function f at 0x00C45070>
    >>> D.f             # Get from a class becomes an unbound method
    <unbound method D.f>
    >>> d.f             # Get from an instance becomes a bound method
    <bound method D.f of <__main__.D object at 0x00B18C90>>

The output suggests that bound and unbound methods are two different
types.  While they could have been implemented that way, the actual C
implementation of PyMethod_Type in Objects/classobject.c
(http://svn.python.org/view/python/trunk/Objects/classobject.c?view=markup)is
a single object with two different representations depending on whether
the `im_self' field is set or is _NULL_ (the C equivalent of _None_).

  Likewise, the effects of calling a method object depend on the
`im_self' field. If set (meaning bound), the original function (stored
in the `im_func' field) is called as expected with the first argument
set to the instance.  If unbound, all of the arguments are passed
unchanged to the original function. The actual C implementation of
`instancemethod_call()' is only slightly more complex in that it
includes some type checking.

11.4.8 Static Methods and Class Methods
---------------------------------------

Non-data descriptors provide a simple mechanism for variations on the
usual patterns of binding functions into methods.

  To recap, functions have a __get__() method so that they
can be converted to a method when accessed as attributes.  The non-data
descriptor transforms a `obj.f(*args)' call into `f(obj, *args)'.
Calling `klass.f(*args)' becomes `f(*args)'.

  This chart summarizes the binding and its two most useful variants:

     Transformation        Called from an Object      Called from a Class
     ------------------------------------------------------------------------
     function              f(obj, *args)              f(*args)
     staticmethod          f(*args)                   f(*args)
     classmethod           f(type(obj), *args)        f(klass, *args)

  Static methods return the underlying function without changes.
Calling either `c.f' or `C.f' is the equivalent of a direct lookup into
`object.__getattribute__(c, "f")' or `object.__getattribute__(C, "f")'.
As a result, the function becomes identically accessible from either an
object or a class.

  Good candidates for static methods are methods that do not reference
the `self' variable.

  For instance, a statistics package may include a container class for
experimental data.  The class provides normal methods for computing the
average, mean, median, and other descriptive statistics that depend on
the data. However, there may be useful functions which are conceptually
related but do not depend on the data.  For instance, `erf(x)' is handy
conversion routine that comes up in statistical work but does not
directly depend on a particular dataset.  It can be called either from
an object or the class:  `s.erf(1.5) --> .9332' or `Sample.erf(1.5) -->
.9332'.

  Since staticmethods return the underlying function with no changes,
the example calls are unexciting:

    >>> class E(object):
         def f(x):
              print x
         f = staticmethod(f)

    >>> print E.f(3)
    3
    >>> print E().f(3)
    3

Using the non-data descriptor protocol, a pure Python version of
staticmethod() would look like this:

    class StaticMethod(object):
     "Emulate PyStaticMethod_Type() in Objects/funcobject.c"

     def __init__(self, f):
          self.f = f

     def __get__(self, obj, objtype=None):
          return self.f

Unlike static methods, class methods prepend the class reference to the
argument list before calling the function.  This format is the same for
whether the caller is an object or a class:

    >>> class E(object):
         def f(klass, x):
              return klass.__name__, x
         f = classmethod(f)

    >>> print E.f(3)
    ('E', 3)
    >>> print E().f(3)
    ('E', 3)

This behavior is useful whenever the function only needs to have a class
reference and does not care about any underlying data.  One use for
classmethods is to create alternate class constructors.  In Python 2.3,
the classmethod dict.fromkeys() creates a new dictionary
from a list of keys.  The pure Python equivalent is:

    class Dict:
        . . .
        def fromkeys(klass, iterable, value=None):
            "Emulate dict_fromkeys() in Objects/dictobject.c"
            d = klass()
            for key in iterable:
                d[key] = value
            return d
        fromkeys = classmethod(fromkeys)

Now a new dictionary of unique keys can be constructed like this:

    >>> Dict.fromkeys('abracadabra')
    {'a': None, 'r': None, 'b': None, 'c': None, 'd': None}

Using the non-data descriptor protocol, a pure Python version of
classmethod() would look like this:

    class ClassMethod(object):
         "Emulate PyClassMethod_Type() in Objects/funcobject.c"

         def __init__(self, f):
              self.f = f

         def __get__(self, obj, klass=None):
              if klass is None:
                   klass = type(obj)
              def newfunc(*args):
                   return self.f(klass, *args)
              return newfunc

11.5 Idioms and Anti-Idioms in Python
=====================================

     Author: Moshe Zadka

  This document is placed in the public domain.

Abstract
........

     This document can be considered a companion to the tutorial. It
     shows how to use Python, and even more importantly, how _not_ to
     use Python.

11.5.1 Language Constructs You Should Not Use
---------------------------------------------

While Python has relatively few gotchas compared to other languages, it
still has some constructs which are only useful in corner cases, or are
plain dangerous.

11.5.1.1 from module import *
.............................

11.5.1.2 Inside Function Definitions
....................................

`from module import *' is _invalid_ inside function definitions. While
many versions of Python do not check for the invalidity, it does not
make it more valid, no more than having a smart lawyer makes a man
innocent. Do not use it like that ever. Even in versions where it was
accepted, it made the function execution slower, because the compiler
could not be certain which names are local and which are global. In
Python 2.1 this construct causes warnings, and sometimes even errors.

11.5.1.3 At Module Level
........................

While it is valid to use `from module import *' at module level it is
usually a bad idea. For one, this loses an important property Python
otherwise has -- you can know where each toplevel name is defined by a
simple "search" function in your favourite editor. You also open
yourself to trouble in the future, if some module grows additional
functions or classes.

  One of the most awful question asked on the newsgroup is why this
code:

    f = open("www")
    f.read()

does not work. Of course, it works just fine (assuming you have a file
called "www".) But it does not work if somewhere in the module, the
statement `from os import *' is present. The os module has
a function called open() which returns an integer. While it
is very useful, shadowing a builtin is one of its least useful
properties.

  Remember, you can never know for sure what names a module exports, so
either take what you need -- `from module import name1, name2', or keep
them in the module and access on a per-need basis --  `import
module;print module.name'.

11.5.1.4 When It Is Just Fine
.............................

There are situations in which `from module import *' is just fine:

   * The interactive prompt. For example, `from math import *' makes
     Python an amazing scientific calculator.

   * When extending a module in C with a module in Python.

   * When the module advertises itself as `from import *' safe.

11.5.1.5 Unadorned `exec', `execfile()' and friends
...................................................

The word "unadorned" refers to the use without an explicit dictionary,
in which case those constructs evaluate code in the _current_
environment. This is dangerous for the same reasons `from import *' is
dangerous -- it might step over variables you are counting on and mess
up things for the rest of your code.  Simply do not do that.

  Bad examples:

    >>> for name in sys.argv[1:]:
    >>>     exec "%s=1" % name
    >>> def func(s, **kw):
    >>>     for var, val in kw.items():
    >>>         exec "s.%s=val" % var  # invalid!
    >>> execfile("handler.py")
    >>> handle()

Good examples:

    >>> d = {}
    >>> for name in sys.argv[1:]:
    >>>     d[name] = 1
    >>> def func(s, **kw):
    >>>     for var, val in kw.items():
    >>>         setattr(s, var, val)
    >>> d={}
    >>> execfile("handle.py", d, d)
    >>> handle = d['handle']
    >>> handle()

11.5.1.6 from module import name1, name2
........................................

This is a "don't" which is much weaker than the previous "don't"s but
is still something you should not do if you don't have good reasons to
do that. The reason it is usually bad idea is because you suddenly have
an object which lives in two separate namespaces. When the binding in
one namespace changes, the binding in the other will not, so there will
be a discrepancy between them. This happens when, for example, one
module is reloaded, or changes the definition of a function at runtime.

  Bad example:

    # foo.py
    a = 1

    # bar.py
    from foo import a
    if something():
        a = 2 # danger: foo.a != a

Good example:

    # foo.py
    a = 1

    # bar.py
    import foo
    if something():
        foo.a = 2

11.5.1.7 except:
................

Python has the `except:' clause, which catches all exceptions. Since
_every_ error in Python raises an exception, using `except:' can make
many programming errors look like runtime problems, which hinders the
debugging process.

  The following code shows a great example of why this is bad:

    try:
        foo = opne("file") # misspelled "open"
    except:
        sys.exit("could not open file!")

The second line triggers a NameError, which is caught by the
except clause. The program will exit, and the error message the program
prints will make you think the problem is the readability of `"file"'
when in fact the real error has nothing to do with `"file"'.

  A better way to write the above is

    try:
        foo = opne("file")
    except IOError:
        sys.exit("could not open file")

When this is run, Python will produce a traceback showing the
NameError, and it will be immediately apparent what needs to be
fixed.

  Because `except:' catches _all_ exceptions, including
SystemExit, KeyboardInterrupt, and GeneratorExit.
(which is not an error and should not normally be caught by user
code), using a bare `except:' is almost never a good idea.  In
situations where you need to catch all "normal" errors, such as in a
framework that runs callbacks, you can catch the base class for all
normal exceptions, see Exception.  Unfortunately in Python 2.x
it is possible for third-party code to raise exceptions that do not
inherit from Exception, so in Python 2.x there are some
cases where you may have to use a bare `except:' and manually re-raise
the exceptions you don't want to catch.

11.5.2 Exceptions
-----------------

Exceptions are a useful feature of Python. You should learn to raise
them whenever something unexpected occurs, and catch them only where
you can do something about them.

  The following is a very popular anti-idiom

    def get_status(file):
        if not os.path.exists(file):
            print "file not found"
            sys.exit(1)
        return open(file).readline()

Consider the case where the file gets deleted between the time the call
to os.path.exists() is made and the time open().
is called. In that case the last line will raise an IOError.
The same thing would happen if _file_ exists but has no read
permission.  Since testing this on a normal machine on existent and
non-existent files makes it seem bugless, the test results will seem
fine, and the code will get shipped.  Later an unhandled IOError.
(or perhaps some other EnvironmentError) escapes to the
user, who gets to watch the ugly traceback.

  Here is a somewhat better way to do it.

    def get_status(file):
        try:
            return open(file).readline()
        except EnvironmentError as err:
            print "Unable to open file: {}".format(err)
            sys.exit(1)

In this version, _either_ the file gets opened and the line is read (so
it works even on flaky NFS or SMB connections), or an error message is
printed that provides all the available information on why the open
failed, and the application is aborted.

  However, even this version of `get_status()' makes too many
assumptions -- that it will only be used in a short running script, and
not, say, in a long running server. Sure, the caller could do something
like

    try:
        status = get_status(log)
    except SystemExit:
        status = None

But there is a better way.  You should try to use as few `except'
clauses in your code as you can -- the ones you do use will usually be
inside calls which should always succeed, or a catch-all in a main
function.

  So, an even better version of `get_status()' is probably

    def get_status(file):
        return open(file).readline()

The caller can deal with the exception if it wants (for example, if it
tries several files in a loop), or just let the exception filter
upwards to _its_ caller.

  But the last version still has a serious problem -- due to
implementation details in CPython, the file would not be closed when an
exception is raised until the exception handler finishes; and, worse,
in other implementations (e.g., Jython) it might not be closed at all
regardless of whether or not an exception is raised.

  The best version of this function uses the `open()' call as a context
manager, which will ensure that the file gets closed as soon as the
function returns:

    def get_status(file):
        with open(file) as fp:
            return fp.readline()

11.5.3 Using the Batteries
--------------------------

Every so often, people seem to be writing stuff in the Python library
again, usually poorly. While the occasional module has a poor
interface, it is usually much better to use the rich standard library
and data types that come with Python than inventing your own.

  A useful module very few people know about is os.path. It
always has the correct path arithmetic for your operating system, and
will usually be much better than whatever you come up with yourself.

  Compare:

    # ugh!
    return dir+"/"+file
    # better
    return os.path.join(dir, file)

More useful functions in os.path.: `basename()',
`dirname()' and `splitext()'.

  There are also many useful built-in functions people seem not to be
aware of for some reason: min() and max() can
find the minimum/maximum of any sequence with comparable semantics, for
example, yet many people write their own max()./see min().
Another highly useful function is reduce() which can
be used to repeatly apply a binary operation to a sequence, reducing it
to a single value.  For example, compute a factorial with a series of
multiply operations:

    >>> n = 4
    >>> import operator
    >>> reduce(operator.mul, range(1, n+1))
    24

When it comes to parsing numbers, note that float(),
int() and long() all accept string arguments and will
reject ill-formed strings by raising an ValueError.

11.5.4 Using Backslash to Continue Statements
---------------------------------------------

Since Python treats a newline as a statement terminator, and since
statements are often more than is comfortable to put in one line, many
people do:

    if foo.bar()['first'][0] == baz.quux(1, 2)[5:9] and \
       calculate_number(10, 20) != forbulate(500, 360):
          pass

You should realize that this is dangerous: a stray space after the `\'
would make this line wrong, and stray spaces are notoriously hard to
see in editors.  In this case, at least it would be a syntax error, but
if the code was:

    value = foo.bar()['first'][0]*baz.quux(1, 2)[5:9] \
            + calculate_number(10, 20)*forbulate(500, 360)

then it would just be subtly wrong.

  It is usually much better to use the implicit continuation inside
parenthesis:

  This version is bulletproof:

    value = (foo.bar()['first'][0]*baz.quux(1, 2)[5:9]
            + calculate_number(10, 20)*forbulate(500, 360))

11.6 Functional Programming HOWTO
=================================

     Author: A. M. Kuchling

     Release: 0.31

  In this document, we'll take a tour of Python's features suitable for
implementing programs in a functional style.  After an introduction to
the concepts of functional programming, we'll look at language features
such as iterator.s and generator.s and relevant
library modules such as itertools and functools.

11.6.1 Introduction
-------------------

This section explains the basic concept of functional programming; if
you're just interested in learning about Python language features, skip
to the next section.

  Programming languages support decomposing problems in several
different ways:

   * Most programming languages are *procedural*: programs are lists of
     instructions that tell the computer what to do with the program's
     input.  C, Pascal, and even Unix shells are procedural languages.

   * In *declarative* languages, you write a specification that
     describes the problem to be solved, and the language
     implementation figures out how to perform the computation
     efficiently.  SQL is the declarative language you're most likely
     to be familiar with; a SQL query describes the data set you want
     to retrieve, and the SQL engine decides whether to scan tables or
     use indexes, which subclauses should be performed first, etc.

   * *Object-oriented* programs manipulate collections of objects.
     Objects have internal state and support methods that query or
     modify this internal state in some way. Smalltalk and Java are
     object-oriented languages.  C++ and Python are languages that
     support object-oriented programming, but don't force the use of
     object-oriented features.

   * *Functional* programming decomposes a problem into a set of
     functions.  Ideally, functions only take inputs and produce
     outputs, and don't have any internal state that affects the output
     produced for a given input.  Well-known functional languages
     include the ML family (Standard ML, OCaml, and other variants) and
     Haskell.

  The designers of some computer languages choose to emphasize one
particular approach to programming.  This often makes it difficult to
write programs that use a different approach.  Other languages are
multi-paradigm languages that support several different approaches.
Lisp, C++, and Python are multi-paradigm; you can write programs or
libraries that are largely procedural, object-oriented, or functional
in all of these languages.  In a large program, different sections
might be written using different approaches; the GUI might be
object-oriented while the processing logic is procedural or functional,
for example.

  In a functional program, input flows through a set of functions. Each
function operates on its input and produces some output.  Functional
style discourages functions with side effects that modify internal
state or make other changes that aren't visible in the function's
return value.  Functions that have no side effects at all are called
*purely functional*.  Avoiding side effects means not using data
structures that get updated as a program runs; every function's output
must only depend on its input.

  Some languages are very strict about purity and don't even have
assignment statements such as `a=3' or `c = a + b', but it's difficult
to avoid all side effects.  Printing to the screen or writing to a disk
file are side effects, for example.  For example, in Python a `print'
statement or a `time.sleep(1)' both return no useful value; they're
only called for their side effects of sending some text to the screen
or pausing execution for a second.

  Python programs written in functional style usually won't go to the
extreme of avoiding all I/O or all assignments; instead, they'll
provide a functional-appearing interface but will use non-functional
features internally.  For example, the implementation of a function
will still use assignments to local variables, but won't modify global
variables or have other side effects.

  Functional programming can be considered the opposite of
object-oriented programming.  Objects are little capsules containing
some internal state along with a collection of method calls that let
you modify this state, and programs consist of making the right set of
state changes.  Functional programming wants to avoid state changes as
much as possible and works with data flowing between functions.  In
Python you might combine the two approaches by writing functions that
take and return instances representing objects in your application
(e-mail messages, transactions, etc.).

  Functional design may seem like an odd constraint to work under.  Why
should you avoid objects and side effects?  There are theoretical and
practical advantages to the functional style:

   * Formal provability.

   * Modularity.

   * Composability.

   * Ease of debugging and testing.

11.6.1.1 Formal provability
...........................

A theoretical benefit is that it's easier to construct a mathematical
proof that a functional program is correct.

  For a long time researchers have been interested in finding ways to
mathematically prove programs correct.  This is different from testing
a program on numerous inputs and concluding that its output is usually
correct, or reading a program's source code and concluding that the
code looks right; the goal is instead a rigorous proof that a program
produces the right result for all possible inputs.

  The technique used to prove programs correct is to write down
*invariants*, properties of the input data and of the program's
variables that are always true.  For each line of code, you then show
that if invariants X and Y are true *before* the line is executed, the
slightly different invariants X' and Y' are true *after* the line is
executed.  This continues until you reach the end of the program, at
which point the invariants should match the desired conditions on the
program's output.

  Functional programming's avoidance of assignments arose because
assignments are difficult to handle with this technique; assignments
can break invariants that were true before the assignment without
producing any new invariants that can be propagated onward.

  Unfortunately, proving programs correct is largely impractical and
not relevant to Python software. Even trivial programs require proofs
that are several pages long; the proof of correctness for a moderately
complicated program would be enormous, and few or none of the programs
you use daily (the Python interpreter, your XML parser, your web
browser) could be proven correct.  Even if you wrote down or generated
a proof, there would then be the question of verifying the proof; maybe
there's an error in it, and you wrongly believe you've proved the
program correct.

11.6.1.2 Modularity
...................

A more practical benefit of functional programming is that it forces
you to break apart your problem into small pieces.  Programs are more
modular as a result.  It's easier to specify and write a small function
that does one thing than a large function that performs a complicated
transformation.  Small functions are also easier to read and to check
for errors.

11.6.1.3 Ease of debugging and testing
......................................

Testing and debugging a functional-style program is easier.

  Debugging is simplified because functions are generally small and
clearly specified.  When a program doesn't work, each function is an
interface point where you can check that the data are correct.  You can
look at the intermediate inputs and outputs to quickly isolate the
function that's responsible for a bug.

  Testing is easier because each function is a potential subject for a
unit test.  Functions don't depend on system state that needs to be
replicated before running a test; instead you only have to synthesize
the right input and then check that the output matches expectations.

11.6.1.4 Composability
......................

As you work on a functional-style program, you'll write a number of
functions with varying inputs and outputs.  Some of these functions
will be unavoidably specialized to a particular application, but others
will be useful in a wide variety of programs.  For example, a function
that takes a directory path and returns all the XML files in the
directory, or a function that takes a filename and returns its
contents, can be applied to many different situations.

  Over time you'll form a personal library of utilities.  Often you'll
assemble new programs by arranging existing functions in a new
configuration and writing a few functions specialized for the current
task.

11.6.2 Iterators
----------------

I'll start by looking at a Python language feature that's an important
foundation for writing functional-style programs: iterators.

  An iterator is an object representing a stream of data; this object
returns the data one element at a time.  A Python iterator must support
a method called `next()' that takes no arguments and always returns the
next element of the stream.  If there are no more elements in the
stream, `next()' must raise the `StopIteration' exception.  Iterators
don't have to be finite, though; it's perfectly reasonable to write an
iterator that produces an infinite stream of data.

  The built-in iter() function takes an arbitrary object
and tries to return an iterator that will return the object's contents
or elements, raising TypeError if the object doesn't
support iteration.  Several of Python's built-in data types support
iteration, the most common being lists and dictionaries.  An object is
called an *iterable* object if you can get an iterator for it.

  You can experiment with the iteration interface manually:

    >>> L = [1,2,3]
    >>> it = iter(L)
    >>> print it
    <...iterator object at ...>
    >>> it.next()
    1
    >>> it.next()
    2
    >>> it.next()
    3
    >>> it.next()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    StopIteration
    >>>

Python expects iterable objects in several different contexts, the most
important being the `for' statement.  In the statement `for X in Y', Y
must be an iterator or some object for which `iter()' can create an
iterator.  These two statements are equivalent:

    for i in iter(obj):
        print i

    for i in obj:
        print i

Iterators can be materialized as lists or tuples by using the
list() or tuple() constructor functions:

    >>> L = [1,2,3]
    >>> iterator = iter(L)
    >>> t = tuple(iterator)
    >>> t
    (1, 2, 3)

Sequence unpacking also supports iterators: if you know an iterator
will return N elements, you can unpack them into an N-tuple:

    >>> L = [1,2,3]
    >>> iterator = iter(L)
    >>> a,b,c = iterator
    >>> a,b,c
    (1, 2, 3)

Built-in functions such as max() and min() can
take a single iterator argument and will return the largest or smallest
element.  The `"in"' and `"not in"' operators also support iterators:
`X in iterator' is true if X is found in the stream returned by the
iterator.  You'll run into obvious problems if the iterator is
infinite; `max()', `min()', and `"not in"' will never return, and if
the element X never appears in the stream, the `"in"' operator won't
return either.

  Note that you can only go forward in an iterator; there's no way to
get the previous element, reset the iterator, or make a copy of it.
Iterator objects can optionally provide these additional capabilities,
but the iterator protocol only specifies the `next()' method.
Functions may therefore consume all of the iterator's output, and if
you need to do something different with the same stream, you'll have to
create a new iterator.

11.6.2.1 Data Types That Support Iterators
..........................................

We've already seen how lists and tuples support iterators.  In fact,
any Python sequence type, such as strings, will automatically support
creation of an iterator.

  Calling iter() on a dictionary returns an iterator that
will loop over the dictionary's keys:

    >>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
    ...      'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
    >>> for key in m:
    ...     print key, m[key]
    Mar 3
    Feb 2
    Aug 8
    Sep 9
    Apr 4
    Jun 6
    Jul 7
    Jan 1
    May 5
    Nov 11
    Dec 12
    Oct 10

Note that the order is essentially random, because it's based on the
hash ordering of the objects in the dictionary.

  Applying `iter()' to a dictionary always loops over the keys, but
dictionaries have methods that return other iterators.  If you want to
iterate over keys, values, or key/value pairs, you can explicitly call
the `iterkeys()', `itervalues()', or `iteritems()' methods to get an
appropriate iterator.

  The dict() constructor can accept an iterator that
returns a finite stream of `(key, value)' tuples:

    >>> L = [('Italy', 'Rome'), ('France', 'Paris'), ('US', 'Washington DC')]
    >>> dict(iter(L))
    {'Italy': 'Rome', 'US': 'Washington DC', 'France': 'Paris'}

Files also support iteration by calling the `readline()' method until
there are no more lines in the file.  This means you can read each line
of a file like this:

    for line in file:
        # do something for each line
        ...

Sets can take their contents from an iterable and let you iterate over
the set's elements:

    S = set((2, 3, 5, 7, 11, 13))
    for i in S:
        print i

11.6.3 Generator expressions and list comprehensions
----------------------------------------------------

Two common operations on an iterator's output are 1) performing some
operation for every element, 2) selecting a subset of elements that
meet some condition.  For example, given a list of strings, you might
want to strip off trailing whitespace from each line or extract all the
strings containing a given substring.

  List comprehensions and generator expressions (short form:
"listcomps" and "genexps") are a concise notation for such operations,
borrowed from the functional programming language Haskell
(<http://www.haskell.org/>).  You can strip all the whitespace from a
stream of strings with the following code:

    line_list = ['  line 1\n', 'line 2  \n', ...]

    # Generator expression -- returns iterator
    stripped_iter = (line.strip() for line in line_list)

    # List comprehension -- returns list
    stripped_list = [line.strip() for line in line_list]

You can select only certain elements by adding an `"if"' condition:

    stripped_list = [line.strip() for line in line_list
                     if line != ""]

With a list comprehension, you get back a Python list; `stripped_list'
is a list containing the resulting lines, not an iterator.  Generator
expressions return an iterator that computes the values as necessary,
not needing to materialize all the values at once.  This means that
list comprehensions aren't useful if you're working with iterators that
return an infinite stream or a very large amount of data.  Generator
expressions are preferable in these situations.

  Generator expressions are surrounded by parentheses ("()") and list
comprehensions are surrounded by square brackets ("[]").  Generator
expressions have the form:

    ( expression for expr in sequence1
                 if condition1
                 for expr2 in sequence2
                 if condition2
                 for expr3 in sequence3 ...
                 if condition3
                 for exprN in sequenceN
                 if conditionN )

Again, for a list comprehension only the outside brackets are different
(square brackets instead of parentheses).

  The elements of the generated output will be the successive values of
`expression'.  The `if' clauses are all optional; if present,
`expression' is only evaluated and added to the result when `condition'
is true.

  Generator expressions always have to be written inside parentheses,
but the parentheses signalling a function call also count.  If you want
to create an iterator that will be immediately passed to a function you
can write:

    obj_total = sum(obj.count for obj in list_all_objects())

The `for...in' clauses contain the sequences to be iterated over.  The
sequences do not have to be the same length, because they are iterated
over from left to right, *not* in parallel.  For each element in
`sequence1', `sequence2' is looped over from the beginning.
`sequence3' is then looped over for each resulting pair of elements
from `sequence1' and `sequence2'.

  To put it another way, a list comprehension or generator expression is
equivalent to the following Python code:

    for expr1 in sequence1:
        if not (condition1):
            continue   # Skip this element
        for expr2 in sequence2:
            if not (condition2):
                continue    # Skip this element
            ...
            for exprN in sequenceN:
                 if not (conditionN):
                     continue   # Skip this element

                 # Output the value of
                 # the expression.

This means that when there are multiple `for...in' clauses but no `if'
clauses, the length of the resulting output will be equal to the
product of the lengths of all the sequences.  If you have two lists of
length 3, the output list is 9 elements long:

    >>> seq1 = 'abc'
    >>> seq2 = (1,2,3)
    >>> [(x,y) for x in seq1 for y in seq2]
    [('a', 1), ('a', 2), ('a', 3),
     ('b', 1), ('b', 2), ('b', 3),
     ('c', 1), ('c', 2), ('c', 3)]

To avoid introducing an ambiguity into Python's grammar, if
`expression' is creating a tuple, it must be surrounded with
parentheses.  The first list comprehension below is a syntax error,
while the second one is correct:

    # Syntax error
    [ x,y for x in seq1 for y in seq2]
    # Correct
    [ (x,y) for x in seq1 for y in seq2]

11.6.4 Generators
-----------------

Generators are a special class of functions that simplify the task of
writing iterators.  Regular functions compute a value and return it,
but generators return an iterator that returns a stream of values.

  You're doubtless familiar with how regular function calls work in
Python or C.  When you call a function, it gets a private namespace
where its local variables are created.  When the function reaches a
`return' statement, the local variables are destroyed and the value is
returned to the caller.  A later call to the same function creates a
new private namespace and a fresh set of local variables. But, what if
the local variables weren't thrown away on exiting a function?  What if
you could later resume the function where it left off?  This is what
generators provide; they can be thought of as resumable functions.

  Here's the simplest example of a generator function:

    def generate_ints(N):
        for i in range(N):
            yield i

Any function containing a `yield' keyword is a generator function; this
is detected by Python's bytecode compiler which compiles
the function specially as a result.

  When you call a generator function, it doesn't return a single value;
instead it returns a generator object that supports the iterator
protocol.  On executing the `yield' expression, the generator outputs
the value of `i', similar to a `return' statement.  The big difference
between `yield' and a `return' statement is that on reaching a `yield'
the generator's state of execution is suspended and local variables are
preserved.  On the next call to the generator's `.next()' method, the
function will resume executing.

  Here's a sample usage of the `generate_ints()' generator:

    >>> gen = generate_ints(3)
    >>> gen
    <generator object generate_ints at ...>
    >>> gen.next()
    0
    >>> gen.next()
    1
    >>> gen.next()
    2
    >>> gen.next()
    Traceback (most recent call last):
      File "stdin", line 1, in ?
      File "stdin", line 2, in generate_ints
    StopIteration

You could equally write `for i in generate_ints(5)', or `a,b,c =
generate_ints(3)'.

  Inside a generator function, the `return' statement can only be used
without a value, and signals the end of the procession of values; after
executing a `return' the generator cannot return any further values.
`return' with a value, such as `return 5', is a syntax error inside a
generator function.  The end of the generator's results can also be
indicated by raising `StopIteration' manually, or by just letting the
flow of execution fall off the bottom of the function.

  You could achieve the effect of generators manually by writing your
own class and storing all the local variables of the generator as
instance variables.  For example, returning a list of integers could be
done by setting `self.count' to 0, and having the `next()' method
increment `self.count' and return it.  However, for a moderately
complicated generator, writing a corresponding class can be much
messier.

  The test suite included with Python's library, `test_generators.py',
contains a number of more interesting examples.  Here's one generator
that implements an in-order traversal of a tree using generators
recursively.

    # A recursive generator that generates Tree leaves in in-order.
    def inorder(t):
        if t:
            for x in inorder(t.left):
                yield x

            yield t.label

            for x in inorder(t.right):
                yield x

Two other examples in `test_generators.py' produce solutions for the
N-Queens problem (placing N queens on an NxN chess board so that no
queen threatens another) and the Knight's Tour (finding a route that
takes a knight to every square of an NxN chessboard without visiting
any square twice).

11.6.4.1 Passing values into a generator
........................................

In Python 2.4 and earlier, generators only produced output.  Once a
generator's code was invoked to create an iterator, there was no way to
pass any new information into the function when its execution is
resumed.  You could hack together this ability by making the generator
look at a global variable or by passing in some mutable object that
callers then modify, but these approaches are messy.

  In Python 2.5 there's a simple way to pass values into a generator.
yield became an expression, returning a value that can be
assigned to a variable or otherwise operated on:

    val = (yield i)

I recommend that you *always* put parentheses around a `yield'
expression when you're doing something with the returned value, as in
the above example.  The parentheses aren't always necessary, but it's
easier to always add them instead of having to remember when they're
needed.

  (PEP 342 explains the exact rules, which are that a
`yield'-expression must always be parenthesized except when it occurs
at the top-level expression on the right-hand side of an assignment.
This means you can write `val = yield i' but have to use parentheses
when there's an operation, as in `val = (yield i) + 12'.)

  Values are sent into a generator by calling its `send(value)' method.
This method resumes the generator's code and the `yield' expression
returns the specified value.  If the regular `next()' method is called,
the `yield' returns `None'.

  Here's a simple counter that increments by 1 and allows changing the
value of the internal counter.

    def counter (maximum):
        i = 0
        while i < maximum:
            val = (yield i)
            # If value provided, change counter
            if val is not None:
                i = val
            else:
                i += 1

And here's an example of changing the counter:

    >>> it = counter(10)
    >>> print it.next()
    0
    >>> print it.next()
    1
    >>> print it.send(8)
    8
    >>> print it.next()
    9
    >>> print it.next()
    Traceback (most recent call last):
      File "t.py", line 15, in ?
        print it.next()
    StopIteration

Because `yield' will often be returning `None', you should always check
for this case.  Don't just use its value in expressions unless you're
sure that the `send()' method will be the only method used resume your
generator function.

  In addition to `send()', there are two other new methods on
generators:

   * `throw(type, value=None, traceback=None)' is used to raise an
     exception inside the generator; the exception is raised by the
     `yield' expression where the generator's execution is paused.

   * `close()' raises a GeneratorExit exception inside the
     generator to terminate the iteration.  On receiving this
     exception, the generator's code must either raise
     GeneratorExit or StopIteration.; catching the
     exception and doing anything else is illegal and will trigger a
     RuntimeError.  `close()' will also be called by
     Python's garbage collector when the generator is garbage-collected.

     If you need to run cleanup code when a GeneratorExit.
     occurs, I suggest using a `try: ... finally:' suite instead of
     catching GeneratorExit.

  The cumulative effect of these changes is to turn generators from
one-way producers of information into both producers and consumers.

  Generators also become *coroutines*, a more generalized form of
subroutines.  Subroutines are entered at one point and exited at
another point (the top of the function, and a `return' statement), but
coroutines can be entered, exited, and resumed at many different points
(the `yield' statements).

11.6.5 Built-in functions
-------------------------

Let's look in more detail at built-in functions often used with
iterators.

  Two of Python's built-in functions, see map() and
filter(), are somewhat obsolete; they duplicate the features of
list comprehensions but return actual lists instead of iterators.

  `map(f, iterA, iterB, ...)' returns a list containing `f(iterA[0],
iterB[0]), f(iterA[1], iterB[1]), f(iterA[2], iterB[2]), ...'.

    >>> def upper(s):
    ...     return s.upper()

    >>> map(upper, ['sentence', 'fragment'])
    ['SENTENCE', 'FRAGMENT']

    >>> [upper(s) for s in ['sentence', 'fragment']]
    ['SENTENCE', 'FRAGMENT']

As shown above, you can achieve the same effect with a list
comprehension.  The itertools.imap() function does the same
thing but can handle infinite iterators; it'll be discussed later, in
the section on the itertools module.

  `filter(predicate, iter)' returns a list that contains all the
sequence elements that meet a certain condition, and is similarly
duplicated by list comprehensions.  A *predicate* is a function that
returns the truth value of some condition; for use with filter(),
the predicate must take a single value.

    >>> def is_even(x):
    ...     return (x % 2) == 0

    >>> filter(is_even, range(10))
    [0, 2, 4, 6, 8]

This can also be written as a list comprehension:

    >>> [x for x in range(10) if is_even(x)]
    [0, 2, 4, 6, 8]

filter() also has a counterpart in the itertools.
module, see itertools.ifilter(), that returns an iterator and
can therefore handle infinite sequences just as itertools.imap()
can.

  `reduce(func, iter, [initial_value])' doesn't have a counterpart in
the itertools module because it cumulatively performs an
operation on all the iterable's elements and therefore can't be applied
to infinite iterables.  `func' must be a function that takes two
elements and returns a single value.  reduce() takes the
first two elements A and B returned by the iterator and calculates
`func(A, B)'.  It then requests the third element, C, calculates
`func(func(A, B), C)', combines this result with the fourth element
returned, and continues until the iterable is exhausted.  If the
iterable returns no values at all, a TypeError exception is
raised.  If the initial value is supplied, it's used as a starting
point and `func(initial_value, A)' is the first calculation.

    >>> import operator
    >>> reduce(operator.concat, ['A', 'BB', 'C'])
    'ABBC'
    >>> reduce(operator.concat, [])
    Traceback (most recent call last):
      ...
    TypeError: reduce() of empty sequence with no initial value
    >>> reduce(operator.mul, [1,2,3], 1)
    6
    >>> reduce(operator.mul, [], 1)
    1

If you use operator.add() with reduce(), you'll
add up all the elements of the iterable.  This case is so common that
there's a special built-in called sum() to compute it:

    >>> reduce(operator.add, [1,2,3,4], 0)
    10
    >>> sum([1,2,3,4])
    10
    >>> sum([])
    0

For many uses of reduce(), though, it can be clearer to just
write the obvious for loop:

    # Instead of:
    product = reduce(operator.mul, [1,2,3], 1)

    # You can write:
    product = 1
    for i in [1,2,3]:
        product *= i

`enumerate(iter)' counts off the elements in the iterable, returning
2-tuples containing the count and each element.

    >>> for item in enumerate(['subject', 'verb', 'object']):
    ...     print item
    (0, 'subject')
    (1, 'verb')
    (2, 'object')

enumerate() is often used when looping through a list and
recording the indexes at which certain conditions are met:

    f = open('data.txt', 'r')
    for i, line in enumerate(f):
        if line.strip() == '':
            print 'Blank line at line #%i' % i

`sorted(iterable, [cmp=None], [key=None], [reverse=False])' collects
all the elements of the iterable into a list, sorts the list, and
returns the sorted result.  The `cmp', `key', and `reverse' arguments
are passed through to the constructed list's `.sort()' method.

    >>> import random
    >>> # Generate 8 random numbers between [0, 10000)
    >>> rand_list = random.sample(range(10000), 8)
    >>> rand_list
    [769, 7953, 9828, 6431, 8442, 9878, 6213, 2207]
    >>> sorted(rand_list)
    [769, 2207, 6213, 6431, 7953, 8442, 9828, 9878]
    >>> sorted(rand_list, reverse=True)
    [9878, 9828, 8442, 7953, 6431, 6213, 2207, 769]

(For a more detailed discussion of sorting, see the Sorting mini-HOWTO
in the Python wiki at <http://wiki.python.org/moin/HowTo/Sorting>.)

  The `any(iter)' and `all(iter)' built-ins look at the truth values of
an iterable's contents.  any() returns True if any element
in the iterable is a true value, and all() returns True if
all of the elements are true values:

    >>> any([0,1,0])
    True
    >>> any([0,0,0])
    False
    >>> any([1,1,1])
    True
    >>> all([0,1,0])
    False
    >>> all([0,0,0])
    False
    >>> all([1,1,1])
    True

11.6.6 Small functions and the lambda expression
------------------------------------------------

When writing functional-style programs, you'll often need little
functions that act as predicates or that combine elements in some way.

  If there's a Python built-in or a module function that's suitable,
you don't need to define a new function at all:

    stripped_lines = [line.strip() for line in lines]
    existing_files = filter(os.path.exists, file_list)

If the function you need doesn't exist, you need to write it.  One way
to write small functions is to use the `lambda' statement.  `lambda'
takes a number of parameters and an expression combining these
parameters, and creates a small function that returns the value of the
expression:

    lowercase = lambda x: x.lower()

    print_assign = lambda name, value: name + '=' + str(value)

    adder = lambda x, y: x+y

An alternative is to just use the `def' statement and define a function
in the usual way:

    def lowercase(x):
        return x.lower()

    def print_assign(name, value):
        return name + '=' + str(value)

    def adder(x,y):
        return x + y

Which alternative is preferable?  That's a style question; my usual
course is to avoid using `lambda'.

  One reason for my preference is that `lambda' is quite limited in the
functions it can define.  The result has to be computable as a single
expression, which means you can't have multiway `if... elif... else'
comparisons or `try... except' statements.  If you try to do too much
in a `lambda' statement, you'll end up with an overly complicated
expression that's hard to read.  Quick, what's the following code doing?

    total = reduce(lambda a, b: (0, a[1] + b[1]), items)[1]

You can figure it out, but it takes time to disentangle the expression
to figure out what's going on.  Using a short nested `def' statements
makes things a little bit better:

    def combine (a, b):
        return 0, a[1] + b[1]

    total = reduce(combine, items)[1]

But it would be best of all if I had simply used a `for' loop:

    total = 0
    for a, b in items:
        total += b

Or the sum() built-in and a generator expression:

    total = sum(b for a,b in items)

Many uses of reduce() are clearer when written as `for'
loops.

  Fredrik Lundh once suggested the following set of rules for
refactoring uses of `lambda':

  1. Write a lambda function.

  2. Write a comment explaining what the heck that lambda does.

  3. Study the comment for a while, and think of a name that captures
     the essence of the comment.

  4. Convert the lambda to a def statement, using that name.

  5. Remove the comment.

  I really like these rules, but you're free to disagree about whether
this lambda-free style is better.

11.6.7 The itertools module
---------------------------

The itertools module contains a number of commonly-used
iterators as well as functions for combining several iterators.  This
section will introduce the module's contents by showing small examples.

  The module's functions fall into a few broad classes:

   * Functions that create a new iterator based on an existing iterator.

   * Functions for treating an iterator's elements as function
     arguments.

   * Functions for selecting portions of an iterator's output.

   * A function for grouping an iterator's output.

11.6.7.1 Creating new iterators
...............................

`itertools.count(n)' returns an infinite stream of integers, increasing
by 1 each time.  You can optionally supply the starting number, which
defaults to 0:

    itertools.count() =>
      0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...
    itertools.count(10) =>
      10, 11, 12, 13, 14, 15, 16, 17, 18, 19, ...

`itertools.cycle(iter)' saves a copy of the contents of a provided
iterable and returns a new iterator that returns its elements from
first to last.  The new iterator will repeat these elements infinitely.

    itertools.cycle([1,2,3,4,5]) =>
      1, 2, 3, 4, 5, 1, 2, 3, 4, 5, ...

`itertools.repeat(elem, [n])' returns the provided element `n' times, or
returns the element endlessly if `n' is not provided.

    itertools.repeat('abc') =>
      abc, abc, abc, abc, abc, abc, abc, abc, abc, abc, ...
    itertools.repeat('abc', 5) =>
      abc, abc, abc, abc, abc

`itertools.chain(iterA, iterB, ...)' takes an arbitrary number of
iterables as input, and returns all the elements of the first iterator,
then all the elements of the second, and so on, until all of the
iterables have been exhausted.

    itertools.chain(['a', 'b', 'c'], (1, 2, 3)) =>
      a, b, c, 1, 2, 3

`itertools.izip(iterA, iterB, ...)' takes one element from each
iterable and returns them in a tuple:

    itertools.izip(['a', 'b', 'c'], (1, 2, 3)) =>
      ('a', 1), ('b', 2), ('c', 3)

It's similar to the built-in zip() function, but doesn't
construct an in-memory list and exhaust all the input iterators before
returning; instead tuples are constructed and returned only if they're
requested.  (The technical term for this behaviour is lazy evaluation
(http://en.wikipedia.org/wiki/Lazy_evaluation).)

  This iterator is intended to be used with iterables that are all of
the same length.  If the iterables are of different lengths, the
resulting stream will be the same length as the shortest iterable.

    itertools.izip(['a', 'b'], (1, 2, 3)) =>
      ('a', 1), ('b', 2)

You should avoid doing this, though, because an element may be taken
from the longer iterators and discarded.  This means you can't go on to
use the iterators further because you risk skipping a discarded element.

  `itertools.islice(iter, [start], stop, [step])' returns a stream
that's a slice of the iterator.  With a single `stop' argument, it will
return the first `stop' elements.  If you supply a starting index,
you'll get `stop-start' elements, and if you supply a value for `step',
elements will be skipped accordingly.  Unlike Python's string and list
slicing, you can't use negative values for `start', `stop', or `step'.

    itertools.islice(range(10), 8) =>
      0, 1, 2, 3, 4, 5, 6, 7
    itertools.islice(range(10), 2, 8) =>
      2, 3, 4, 5, 6, 7
    itertools.islice(range(10), 2, 8, 2) =>
      2, 4, 6

`itertools.tee(iter, [n])' replicates an iterator; it returns `n'
independent iterators that will all return the contents of the source
iterator.  If you don't supply a value for `n', the default is 2.
Replicating iterators requires saving some of the contents of the
source iterator, so this can consume significant memory if the iterator
is large and one of the new iterators is consumed more than the others.

    itertools.tee( itertools.count() ) =>
       iterA, iterB

    where iterA ->
       0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...

    and   iterB ->
       0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...

11.6.7.2 Calling functions on elements
......................................

Two functions are used for calling other functions on the contents of an
iterable.

  `itertools.imap(f, iterA, iterB, ...)' returns a stream containing
`f(iterA[0], iterB[0]), f(iterA[1], iterB[1]), f(iterA[2], iterB[2]),
...':

    itertools.imap(operator.add, [5, 6, 5], [1, 2, 3]) =>
      6, 8, 8

The `operator' module contains a set of functions corresponding to
Python's operators.  Some examples are `operator.add(a, b)' (adds two
values), `operator.ne(a, b)' (same as `a!=b'), and
`operator.attrgetter('id')' (returns a callable that fetches the `"id"'
attribute).

  `itertools.starmap(func, iter)' assumes that the iterable will return
a stream of tuples, and calls `f()' using these tuples as the arguments:

    itertools.starmap(os.path.join,
                      [('/usr', 'bin', 'java'), ('/bin', 'python'),
                       ('/usr', 'bin', 'perl'),('/usr', 'bin', 'ruby')])
    =>
      /usr/bin/java, /bin/python, /usr/bin/perl, /usr/bin/ruby

11.6.7.3 Selecting elements
...........................

Another group of functions chooses a subset of an iterator's elements
based on a predicate.

  `itertools.ifilter(predicate, iter)' returns all the elements for
which the predicate returns true:

    def is_even(x):
        return (x % 2) == 0

    itertools.ifilter(is_even, itertools.count()) =>
      0, 2, 4, 6, 8, 10, 12, 14, ...

`itertools.ifilterfalse(predicate, iter)' is the opposite, returning all
elements for which the predicate returns false:

    itertools.ifilterfalse(is_even, itertools.count()) =>
      1, 3, 5, 7, 9, 11, 13, 15, ...

`itertools.takewhile(predicate, iter)' returns elements for as long as
the predicate returns true.  Once the predicate returns false, the
iterator will signal the end of its results.

    def less_than_10(x):
        return (x < 10)

    itertools.takewhile(less_than_10, itertools.count()) =>
      0, 1, 2, 3, 4, 5, 6, 7, 8, 9

    itertools.takewhile(is_even, itertools.count()) =>
      0

`itertools.dropwhile(predicate, iter)' discards elements while the
predicate returns true, and then returns the rest of the iterable's
results.

    itertools.dropwhile(less_than_10, itertools.count()) =>
      10, 11, 12, 13, 14, 15, 16, 17, 18, 19, ...

    itertools.dropwhile(is_even, itertools.count()) =>
      1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...

11.6.7.4 Grouping elements
..........................

The last function I'll discuss, `itertools.groupby(iter,
key_func=None)', is the most complicated.  `key_func(elem)' is a
function that can compute a key value for each element returned by the
iterable.  If you don't supply a key function, the key is simply each
element itself.

  `groupby()' collects all the consecutive elements from the underlying
iterable that have the same key value, and returns a stream of 2-tuples
containing a key value and an iterator for the elements with that key.

    city_list = [('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL'),
                 ('Anchorage', 'AK'), ('Nome', 'AK'),
                 ('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ'),
                 ...
                ]

    def get_state ((city, state)):
        return state

    itertools.groupby(city_list, get_state) =>
      ('AL', iterator-1),
      ('AK', iterator-2),
      ('AZ', iterator-3), ...

    where
    iterator-1 =>
      ('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL')
    iterator-2 =>
      ('Anchorage', 'AK'), ('Nome', 'AK')
    iterator-3 =>
      ('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ')

`groupby()' assumes that the underlying iterable's contents will
already be sorted based on the key.  Note that the returned iterators
also use the underlying iterable, so you have to consume the results of
iterator-1 before requesting iterator-2 and its corresponding key.

11.6.8 The functools module
---------------------------

The functools module in Python 2.5 contains some
higher-order functions.  A *higher-order function* takes one or more
functions as input and returns a new function.  The most useful tool in
this module is the functools.partial() function.

  For programs written in a functional style, you'll sometimes want to
construct variants of existing functions that have some of the
parameters filled in.  Consider a Python function `f(a, b, c)'; you may
wish to create a new function `g(b, c)' that's equivalent to `f(1, b,
c)'; you're filling in a value for one of `f()''s parameters.  This is
called "partial function application".

  The constructor for `partial' takes the arguments `(function, arg1,
arg2, ... kwarg1=value1, kwarg2=value2)'.  The resulting object is
callable, so you can just call it to invoke `function' with the
filled-in arguments.

  Here's a small but realistic example:

    import functools

    def log (message, subsystem):
        "Write the contents of 'message' to the specified subsystem."
        print '%s: %s' % (subsystem, message)
        ...

    server_log = functools.partial(log, subsystem='server')
    server_log('Unable to open socket')

11.6.8.1 The operator module
............................

The operator module was mentioned earlier.  It contains a
set of functions corresponding to Python's operators.  These functions
are often useful in functional-style code because they save you from
writing trivial functions that perform a single operation.

  Some of the functions in this module are:

   * Math operations: `add()', `sub()', `mul()', `div()', `floordiv()',
     `abs()', ...

   * Logical operations: `not_()', `truth()'.

   * Bitwise operations: `and_()', `or_()', `invert()'.

   * Comparisons: `eq()', `ne()', `lt()', `le()', `gt()', and `ge()'.

   * Object identity: `is_()', `is_not()'.

  Consult the operator module's documentation for a complete list.

11.6.8.2 The functional module
..............................

Collin Winter's functional module
(http://oakwinter.com/code/functional/) provides a number of more
advanced tools for functional programming. It also reimplements several
Python built-ins, trying to make them more intuitive to those used to
functional programming in other languages.

  This section contains an introduction to some of the most important
functions in `functional'; full documentation can be found at the
project's website (http://oakwinter.com/code/functional/documentation/).

  `compose(outer, inner, unpack=False)'

  The `compose()' function implements function composition.  In other
words, it returns a wrapper around the `outer' and `inner' callables,
such that the return value from `inner' is fed directly to `outer'.
That is,

    >>> def add(a, b):
    ...     return a + b
    ...
    >>> def double(a):
    ...     return 2 * a
    ...
    >>> compose(double, add)(5, 6)
    22

is equivalent to

    >>> double(add(5, 6))
    22

The `unpack' keyword is provided to work around the fact that Python
functions are not always fully curried
(http://en.wikipedia.org/wiki/Currying).  By default, it is expected
that the `inner' function will return a single object and that the
`outer' function will take a single argument. Setting the `unpack'
argument causes `compose' to expect a tuple from `inner' which will be
expanded before being passed to `outer'. Put simply,

    compose(f, g)(5, 6)

is equivalent to:

    f(g(5, 6))

while

    compose(f, g, unpack=True)(5, 6)

is equivalent to:

    f(*g(5, 6))

Even though `compose()' only accepts two functions, it's trivial to
build up a version that will compose any number of functions. We'll use
`reduce()', `compose()' and `partial()' (the last of which is provided
by both `functional' and `functools').

    from functional import compose, partial

    multi_compose = partial(reduce, compose)

We can also use `map()', `compose()' and `partial()' to craft a version
of `"".join(...)' that converts its arguments to string:

    from functional import compose, partial

    join = compose("".join, partial(map, str))

`flip(func)'

  `flip()' wraps the callable in `func' and causes it to receive its
non-keyword arguments in reverse order.

    >>> def triple(a, b, c):
    ...     return (a, b, c)
    ...
    >>> triple(5, 6, 7)
    (5, 6, 7)
    >>>
    >>> flipped_triple = flip(triple)
    >>> flipped_triple(5, 6, 7)
    (7, 6, 5)

`foldl(func, start, iterable)'

  `foldl()' takes a binary function, a starting value (usually some
kind of 'zero'), and an iterable.  The function is applied to the
starting value and the first element of the list, then the result of
that and the second element of the list, then the result of that and
the third element of the list, and so on.

  This means that a call such as:

    foldl(f, 0, [1, 2, 3])

is equivalent to:

    f(f(f(0, 1), 2), 3)

`foldl()' is roughly equivalent to the following recursive function:

    def foldl(func, start, seq):
        if len(seq) == 0:
            return start

        return foldl(func, func(start, seq[0]), seq[1:])

Speaking of equivalence, the above `foldl' call can be expressed in
terms of the built-in `reduce' like so:

    reduce(f, [1, 2, 3], 0)

We can use `foldl()', `operator.concat()' and `partial()' to write a
cleaner, more aesthetically-pleasing version of Python's `"".join(...)'
idiom:

    from functional import foldl, partial from operator import concat

    join = partial(foldl, concat, "")

11.6.9 Revision History and Acknowledgements
--------------------------------------------

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Ian Bicking, Nick Coghlan, Nick Efford, Raymond Hettinger, Jim
Jewett, Mike Krell, Leandro Lameiro, Jussi Salmela, Collin Winter,
Blake Winton.

  Version 0.1: posted June 30 2006.

  Version 0.11: posted July 1 2006.  Typo fixes.

  Version 0.2: posted July 10 2006.  Merged genexp and listcomp
sections into one.  Typo fixes.

  Version 0.21: Added more references suggested on the tutor mailing
list.

  Version 0.30: Adds a section on the `functional' module written by
Collin Winter; adds short section on the operator module; a few other
edits.

11.6.10 References
------------------

11.6.10.1 General
.................

*Structure and Interpretation of Computer Programs*, by Harold Abelson
and Gerald Jay Sussman with Julie Sussman.  Full text at
<http://mitpress.mit.edu/sicp/>.  In this classic textbook of computer
science, chapters 2 and 3 discuss the use of sequences and streams to
organize the data flow inside a program.  The book uses Scheme for its
examples, but many of the design approaches described in these chapters
are applicable to functional-style Python code.

  <http://www.defmacro.org/ramblings/fp.html>: A general introduction
to functional programming that uses Java examples and has a lengthy
historical introduction.

  <http://en.wikipedia.org/wiki/Functional_programming>: General
Wikipedia entry describing functional programming.

  <http://en.wikipedia.org/wiki/Coroutine>: Entry for coroutines.

  <http://en.wikipedia.org/wiki/Currying>: Entry for the concept of
currying.

11.6.10.2 Python-specific
.........................

<http://gnosis.cx/TPiP/>: The first chapter of David Mertz's book `Text
Processing in Python' discusses functional programming for text
processing, in the section titled "Utilizing Higher-Order Functions in
Text Processing".

  Mertz also wrote a 3-part series of articles on functional programming
for IBM's DeveloperWorks site; see part 1
(http://www-128.ibm.com/developerworks/library/l-prog.html), part 2
(http://www-128.ibm.com/developerworks/library/l-prog2.html), and part
3 (http://www-128.ibm.com/developerworks/linux/library/l-prog3.html),

11.6.10.3 Python documentation
..............................

Documentation for the itertools module.

  Documentation for the operator module.

  PEP 289 (http://www.python.org/dev/peps/pep-0289): "Generator
Expressions"

  PEP 342 (http://www.python.org/dev/peps/pep-0342): "Coroutines via
Enhanced Generators" describes the new generator features in Python 2.5.

11.7 Regular Expression HOWTO
=============================

     Author: A.M. Kuchling <<amk@amk.ca>>

     Release: 0.05

Abstract
........

     This document is an introductory tutorial to using regular
     expressions in Python with the re module.  It provides
     a gentler introduction than the corresponding section in the
     Library Reference.

11.7.1 Introduction
-------------------

The re module was added in Python 1.5, and provides
Perl-style regular expression patterns.  Earlier versions of Python
came with the `regex' module, which provided Emacs-style patterns.  The
`regex' module was removed completely in Python 2.5.

  Regular expressions (called REs, or regexes, or regex patterns) are
essentially a tiny, highly specialized programming language embedded
inside Python and made available through the re module.
Using this little language, you specify the rules for the set of
possible strings that you want to match; this set might contain English
sentences, or e-mail addresses, or TeX commands, or anything you like.
You can then ask questions such as "Does this string match the
pattern?", or "Is there a match for the pattern anywhere in this
string?".  You can also use REs to modify a string or to split it apart
in various ways.

  Regular expression patterns are compiled into a series of bytecodes
which are then executed by a matching engine written in C.  For
advanced use, it may be necessary to pay careful attention to how the
engine will execute a given RE, and write the RE in a certain way in
order to produce bytecode that runs faster.  Optimization isn't covered
in this document, because it requires that you have a good
understanding of the matching engine's internals.

  The regular expression language is relatively small and restricted,
so not all possible string processing tasks can be done using regular
expressions.  There are also tasks that _can_ be done with regular
expressions, but the expressions turn out to be very complicated.  In
these cases, you may be better off writing Python code to do the
processing; while Python code will be slower than an elaborate regular
expression, it will also probably be more understandable.

11.7.2 Simple Patterns
----------------------

We'll start by learning about the simplest possible regular
expressions.  Since regular expressions are used to operate on strings,
we'll begin with the most common task: matching characters.

  For a detailed explanation of the computer science underlying regular
expressions (deterministic and non-deterministic finite automata), you
can refer to almost any textbook on writing compilers.

11.7.2.1 Matching Characters
............................

Most letters and characters will simply match themselves.  For example,
the regular expression `test' will match the string `test' exactly.
(You can enable a case-insensitive mode that would let this RE match
`Test' or `TEST' as well; more about this later.)

  There are exceptions to this rule; some characters are special
_metacharacters_, and don't match themselves.  Instead, they signal that
some out-of-the-ordinary thing should be matched, or they affect other
portions of the RE by repeating them or changing their meaning.  Much
of this document is devoted to discussing various metacharacters and
what they do.

  Here's a complete list of the metacharacters; their meanings will be
discussed in the rest of this HOWTO.

    . ^ $ * + ? { [ ] \ | ( )

The first metacharacters we'll look at are `[' and `]'. They're used for
specifying a character class, which is a set of characters that you
wish to match.  Characters can be listed individually, or a range of
characters can be indicated by giving two characters and separating
them by a `'-''.  For example, `[abc]' will match any of the characters
`a', `b', or `c'; this is the same as `[a-c]', which uses a range to
express the same set of characters.  If you wanted to match only
lowercase letters, your RE would be `[a-z]'.

  Metacharacters are not active inside classes.  For example, `[akm$]'
will match any of the characters `'a'', `'k'', `'m'', or `'$''; `'$'' is
usually a metacharacter, but inside a character class it's stripped of
its special nature.

  You can match the characters not listed within the class by
_complementing_ the set.  This is indicated by including a `'^'' as the
first character of the class; `'^'' outside a character class will
simply match the `'^'' character.  For example, `[^5]' will match any
character except `'5''.

  Perhaps the most important metacharacter is the backslash, `\'.   As
in Python string literals, the backslash can be followed by various
characters to signal various special sequences.  It's also used to
escape all the metacharacters so you can still match them in patterns;
for example, if you need to match a `[' or  `\', you can precede them
with a backslash to remove their special meaning: `\[' or `\\'.

  Some of the special sequences beginning with `'\'' represent
predefined sets of characters that are often useful, such as the set of
digits, the set of letters, or the set of anything that isn't
whitespace.  The following predefined special sequences are available:

`\d'
     Matches any decimal digit; this is equivalent to the class `[0-9]'.

`\D'
     Matches any non-digit character; this is equivalent to the class
     `[^0-9]'.

`\s'
     Matches any whitespace character; this is equivalent to the class
     `[ \t\n\r\f\v]'.

`\S'
     Matches any non-whitespace character; this is equivalent to the
     class `[^ \t\n\r\f\v]'.

`\w'
     Matches any alphanumeric character; this is equivalent to the class
     `[a-zA-Z0-9_]'.

`\W'
     Matches any non-alphanumeric character; this is equivalent to the
     class `[^a-zA-Z0-9_]'.

  These sequences can be included inside a character class.  For
example, `[\s,.]' is a character class that will match any whitespace
character, or `','' or `'.''.

  The final metacharacter in this section is `.'.  It matches anything
except a newline character, and there's an alternate mode (`re.DOTALL')
where it will match even a newline.  `'.'' is often used where you want
to match "any character".

11.7.2.2 Repeating Things
.........................

Being able to match varying sets of characters is the first thing
regular expressions can do that isn't already possible with the methods
available on strings.  However, if that was the only additional
capability of regexes, they wouldn't be much of an advance. Another
capability is that you can specify that portions of the RE must be
repeated a certain number of times.

  The first metacharacter for repeating things that we'll look at is
`*'.  `*' doesn't match the literal character `*'; instead, it
specifies that the previous character can be matched zero or more
times, instead of exactly once.

  For example, `ca*t' will match `ct' (0 `a' characters), `cat' (1 `a'),
`caaat' (3 `a' characters), and so forth.  The RE engine has various
internal limitations stemming from the size of C's `int' type that will
prevent it from matching over 2 billion `a' characters; you probably
don't have enough memory to construct a string that large, so you
shouldn't run into that limit.

  Repetitions such as `*' are _greedy_; when repeating a RE, the
matching engine will try to repeat it as many times as possible. If
later portions of the pattern don't match, the matching engine will
then back up and try again with few repetitions.

  A step-by-step example will make this more obvious.  Let's consider
the expression `a[bcd]*b'.  This matches the letter `'a'', zero or more
letters from the class `[bcd]', and finally ends with a `'b''.  Now
imagine matching this RE against the string `abcbd'.

Step       Matched         Explanation
-----------------------------------------------------------------
1          `a'             The `a' in the RE matches.
2          `abcbd'         The engine matches `[bcd]*', going
                           as far as it can, which is to the
                           end of the string.
3          _Failure_       The engine tries to match `b', but
                           the current position is at the end
                           of the string, so it fails.
4          `abcb'          Back up, so that  `[bcd]*' matches
                           one less character.
5          _Failure_       Try `b' again, but the current
                           position is at the last character,
                           which is a `'d''.
6          `abc'           Back up again, so that `[bcd]*' is
                           only matching `bc'.
6          `abcb'          Try `b' again.  This time the
                           character at the current position is
                           `'b'', so it succeeds.

  The end of the RE has now been reached, and it has matched `abcb'.
This demonstrates how the matching engine goes as far as it can at
first, and if no match is found it will then progressively back up and
retry the rest of the RE again and again.  It will back up until it has
tried zero matches for `[bcd]*', and if that subsequently fails, the
engine will conclude that the string doesn't match the RE at all.

  Another repeating metacharacter is `+', which matches one or more
times.  Pay careful attention to the difference between `*' and `+';
`*' matches _zero_ or more times, so whatever's being repeated may not
be present at all, while `+' requires at least _one_ occurrence.  To
use a similar example, `ca+t' will match `cat' (1 `a'), `caaat' (3
`a''s), but won't match `ct'.

  There are two more repeating qualifiers.  The question mark
character, `?', matches either once or zero times; you can think of it
as marking something as being optional.  For example, `home-?brew'
matches either `homebrew' or `home-brew'.

  The most complicated repeated qualifier is `{m,n}', where _m_ and _n_
are decimal integers.  This qualifier means there must be at least _m_
repetitions, and at most _n_.  For example, `a/{1,3}b' will match
`a/b', `a//b', and `a///b'.  It won't match `ab', which has no slashes,
or `a////b', which has four.

  You can omit either _m_ or _n_; in that case, a reasonable value is
assumed for the missing value.  Omitting _m_ is interpreted as a lower
limit of 0, while omitting _n_ results in an upper bound of infinity --
actually, the upper bound is the 2-billion limit mentioned earlier, but
that might as well be infinity.

  Readers of a reductionist bent may notice that the three other
qualifiers can all be expressed using this notation.  `{0,}' is the
same as `*', `{1,}' is equivalent to `+', and `{0,1}' is the same as
`?'.  It's better to use `*', `+', or `?' when you can, simply because
they're shorter and easier to read.

11.7.3 Using Regular Expressions
--------------------------------

Now that we've looked at some simple regular expressions, how do we
actually use them in Python?  The re module provides an
interface to the regular expression engine, allowing you to compile REs
into objects and then perform matches with them.

11.7.3.1 Compiling Regular Expressions
......................................

Regular expressions are compiled into pattern objects, which have
methods for various operations such as searching for pattern matches or
performing string substitutions.

    >>> import re
    >>> p = re.compile('ab*')
    >>> print p
    <_sre.SRE_Pattern object at 80b4150>

re.compile() also accepts an optional _flags_ argument,
used to enable various special features and syntax variations.  We'll
go over the available settings later, but for now a single example will
do:

    >>> p = re.compile('ab*', re.IGNORECASE)

The RE is passed to re.compile() as a string.  REs are
handled as strings because regular expressions aren't part of the core
Python language, and no special syntax was created for expressing them.
(There are applications that don't need REs at all, so there's no need
to bloat the language specification by including them.) Instead, the
re module is simply a C extension module included with
Python, just like the socket or zlib modules.

  Putting REs in strings keeps the Python language simpler, but has one
disadvantage which is the topic of the next section.

11.7.3.2 The Backslash Plague
.............................

As stated earlier, regular expressions use the backslash character
(`'\'') to indicate special forms or to allow special characters to be
used without invoking their special meaning. This conflicts with
Python's usage of the same character for the same purpose in string
literals.

  Let's say you want to write a RE that matches the string `\section',
which might be found in a LaTeX file.  To figure out what to write in
the program code, start with the desired string to be matched.  Next,
you must escape any backslashes and other metacharacters by preceding
them with a backslash, resulting in the string `\\section'.  The
resulting string that must be passed to re.compile() must
be `\\section'.  However, to express this as a Python string literal,
both backslashes must be escaped _again_.

Characters              Stage
-----------------------------------------------------------------------
`\section'              Text string to be matched
`\\section'             Escaped backslash for re.compile().
`"\\\\section"'         Escaped backslashes for a string literal

  In short, to match a literal backslash, one has to write `'\\\\'' as
the RE string, because the regular expression must be `\\', and each
backslash must be expressed as `\\' inside a regular Python string
literal.  In REs that feature backslashes repeatedly, this leads to
lots of repeated backslashes and makes the resulting strings difficult
to understand.

  The solution is to use Python's raw string notation for regular
expressions; backslashes are not handled in any special way in a string
literal prefixed with `'r'', so `r"\n"' is a two-character string
containing `'\'' and `'n'', while `"\n"' is a one-character string
containing a newline. Regular expressions will often be written in
Python code using this raw string notation.

Regular String          Raw string
-----------------------------------------------
`"ab*"'                 `r"ab*"'
`"\\\\section"'         `r"\\section"'
`"\\w+\\s+\\1"'         `r"\w+\s+\1"'

11.7.3.3 Performing Matches
...........................

Once you have an object representing a compiled regular expression,
what do you do with it?  Pattern objects have several methods and
attributes.  Only the most significant ones will be covered here;
consult the re docs for a complete listing.

Method/Attribute       Purpose
---------------------------------------------------------------------------
`match()'              Determine if the RE matches at the beginning of
                       the string.
`search()'             Scan through a string, looking for any location
                       where this RE matches.
`findall()'            Find all substrings where the RE matches, and
                       returns them as a list.
`finditer()'           Find all substrings where the RE matches, and
                       returns them as an iterator.

  `match()' and `search()' return `None' if no match can be found.  If
they're successful, a `MatchObject' instance is returned, containing
information about the match: where it starts and ends, the substring it
matched, and more.

  You can learn about this by interactively experimenting with the
re.  module.  If you have Tkinter available, you may also
want to look at `Tools/scripts/redemo.py', a demonstration program
included with the Python distribution.  It allows you to enter REs and
strings, and displays whether the RE matches or fails. `redemo.py' can
be quite useful when trying to debug a complicated RE.  Phil Schwartz's
Kodos (http://kodos.sourceforge.net/)is also an interactive tool for
developing and testing RE patterns.

  This HOWTO uses the standard Python interpreter for its examples.
First, run the Python interpreter, import the re module,
and compile a RE:

    Python 2.2.2 (#1, Feb 10 2003, 12:57:01)
    >>> import re
    >>> p = re.compile('[a-z]+')
    >>> p
    <_sre.SRE_Pattern object at 80c3c28>

Now, you can try matching various strings against the RE `[a-z]+'.  An
empty string shouldn't match at all, since `+' means 'one or more
repetitions'.  `match()' should return `None' in this case, which will
cause the interpreter to print no output.  You can explicitly print the
result of `match()' to make this clear.

    >>> p.match("")
    >>> print p.match("")
    None

Now, let's try it on a string that it should match, such as `tempo'.
In this case, `match()' will return a `MatchObject', so you should
store the result in a variable for later use.

    >>> m = p.match('tempo')
    >>> print m
    <_sre.SRE_Match object at 80c4f68>

Now you can query the `MatchObject' for information about the matching
string.   `MatchObject' instances also have several methods and
attributes; the most important ones are:

Method/Attribute       Purpose
------------------------------------------------------------------------
`group()'              Return the string matched by the RE
`start()'              Return the starting position of the match
`end()'                Return the ending position of the match
`span()'               Return a tuple containing the (start, end)
                       positions  of the match

  Trying these methods will soon clarify their meaning:

    >>> m.group()
    'tempo'
    >>> m.start(), m.end()
    (0, 5)
    >>> m.span()
    (0, 5)

`group()' returns the substring that was matched by the RE.  `start()'
and `end()' return the starting and ending index of the match. `span()'
returns both start and end indexes in a single tuple.  Since the
`match()' method only checks if the RE matches at the start of a
string, `start()' will always be zero.  However, the `search()' method
of patterns scans through the string, so  the match may not start at
zero in that case.

    >>> print p.match('::: message')
    None
    >>> m = p.search('::: message') ; print m
    <re.MatchObject instance at 80c9650>
    >>> m.group()
    'message'
    >>> m.span()
    (4, 11)

In actual programs, the most common style is to store the `MatchObject'
in a variable, and then check if it was `None'.  This usually looks
like:

    p = re.compile( ... )
    m = p.match( 'string goes here' )
    if m:
        print 'Match found: ', m.group()
    else:
        print 'No match'

Two pattern methods return all of the matches for a pattern.
`findall()' returns a list of matching strings:

    >>> p = re.compile('\d+')
    >>> p.findall('12 drummers drumming, 11 pipers piping, 10 lords a-leaping')
    ['12', '11', '10']

`findall()' has to create the entire list before it can be returned as
the result.  The `finditer()' method returns a sequence of `MatchObject'
instances as an iterator. (1)

    >>> iterator = p.finditer('12 drummers drumming, 11 ... 10 ...')
    >>> iterator
    <callable-iterator object at 0x401833ac>
    >>> for match in iterator:
    ...     print match.span()
    ...
    (0, 2)
    (22, 24)
    (29, 31)

11.7.3.4 Module-Level Functions
...............................

You don't have to create a pattern object and call its methods; the
re module also provides top-level functions called
`match()', `search()', `findall()', `sub()', and so forth.  These
functions take the same arguments as the corresponding pattern method,
with the RE string added as the first argument, and still return either
`None' or a `MatchObject' instance.

    >>> print re.match(r'From\s+', 'Fromage amk')
    None
    >>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998')
    <re.MatchObject instance at 80c5978>

Under the hood, these functions simply create a pattern object for you
and call the appropriate method on it.  They also store the compiled
object in a cache, so future calls using the same RE are faster.

  Should you use these module-level functions, or should you get the
pattern and call its methods yourself?  That choice depends on how
frequently the RE will be used, and on your personal coding style.  If
the RE is being used at only one point in the code, then the module
functions are probably more convenient.  If a program contains a lot of
regular expressions, or re-uses the same ones in several locations,
then it might be worthwhile to collect all the definitions in one
place, in a section of code that compiles all the REs ahead of time.
To take an example from the standard library, here's an extract from
`xmllib.py':

    ref = re.compile( ... )
    entityref = re.compile( ... )
    charref = re.compile( ... )
    starttagopen = re.compile( ... )

I generally prefer to work with the compiled object, even for one-time
uses, but few people will be as much of a purist about this as I am.

11.7.3.5 Compilation Flags
..........................

Compilation flags let you modify some aspects of how regular
expressions work.  Flags are available in the re module
under two names, a long name such as `IGNORECASE' and a short,
one-letter form such as `I'.  (If you're familiar with Perl's pattern
modifiers, the one-letter forms use the same letters; the short form of
re.VERBOSE is re.X, for example.)  Multiple
flags can be specified by bitwise OR-ing them; `re.I | re.M' sets both
the `I' and `M' flags, for example.

  Here's a table of the available flags, followed by a more detailed
explanation of each one.

Flag                                  Meaning
---------------------------------------------------------------------------------------
`DOTALL', `S'                         Make `.' match any character, including newlines
`IGNORECASE', `I'                     Do case-insensitive matches
`LOCALE', `L'                         Do a locale-aware match
`MULTILINE', `M'                      Multi-line matching, affecting `^' and `$'
`VERBOSE', `X'                        Enable verbose REs, which can be organized more
                                      cleanly and understandably.
`UNICODE', `U'                        Makes several escapes like `\w', `\b', `\s' and
                                      `\d' dependent on the Unicode character
                                      database.

 -- Data: I
 -- Data: IGNORECASE
     Perform case-insensitive matching; character class and literal
     strings will match letters by ignoring case.  For example, `[A-Z]'
     will match lowercase letters, too, and `Spam' will match `Spam',
     `spam', or `spAM'. This lowercasing doesn't take the current
     locale into account; it will if you also set the `LOCALE' flag.

 -- Data: L
 -- Data: LOCALE
     Make `\w', `\W', `\b', and `\B', dependent on the current locale.

     Locales are a feature of the C library intended to help in writing
     programs that take account of language differences.  For example,
     if you're processing French text, you'd want to be able to write
     `\w+' to match words, but `\w' only matches the character class
     `[A-Za-z]'; it won't match `'é'' or `'ç''.  If your system is
     configured properly and a French locale is selected, certain C
     functions will tell the program that `'é'' should also be
     considered a letter.  Setting the `LOCALE' flag when compiling a
     regular expression will cause the resulting compiled object to use
     these C functions for `\w'; this is slower, but also enables `\w+'
     to match French words as you'd expect.

 -- Data: M
 -- Data: MULTILINE
     (`^' and `$' haven't been explained yet;  they'll be introduced in
     section More Metacharacters)

     Usually `^' matches only at the beginning of the string, and `$'
     matches only at the end of the string and immediately before the
     newline (if any) at the end of the string. When this flag is
     specified, `^' matches at the beginning of the string and at the
     beginning of each line within the string, immediately following
     each newline.  Similarly, the `$' metacharacter matches either at
     the end of the string and at the end of each line (immediately
     preceding each newline).

 -- Data: S
 -- Data: DOTALL
     Makes the `'.'' special character match any character at all,
     including a newline; without this flag, `'.'' will match anything
     _except_ a newline.

 -- Data: U
 -- Data: UNICODE
     Make `\w', `\W', `\b', `\B', `\d', `\D', `\s' and `\S' dependent
     on the Unicode character properties database.

 -- Data: X
 -- Data: VERBOSE
     This flag allows you to write regular expressions that are more
     readable by granting you more flexibility in how you can format
     them.  When this flag has been specified, whitespace within the RE
     string is ignored, except when the whitespace is in a character
     class or preceded by an unescaped backslash; this lets you
     organize and indent the RE more clearly.  This flag also lets you
     put comments within a RE that will be ignored by the engine;
     comments are marked by a `'#'' that's neither in a character class
     or preceded by an unescaped backslash.

     For example, here's a RE that uses re.VERBOSE.; see how
     much easier it is to read?

         charref = re.compile(r"""
          &[#]                # Start of a numeric entity reference
          (
              0[0-7]+         # Octal form
            | [0-9]+          # Decimal form
            | x[0-9a-fA-F]+   # Hexadecimal form
          )
          ;                   # Trailing semicolon
         """, re.VERBOSE)

     Without the verbose setting, the RE would look like this:

         charref = re.compile("&#(0[0-7]+"
                              "|[0-9]+"
                              "|x[0-9a-fA-F]+);")

     In the above example, Python's automatic concatenation of string
     literals has been used to break up the RE into smaller pieces, but
     it's still more difficult to understand than the version using
     re.VERBOSE.

11.7.4 More Pattern Power
-------------------------

So far we've only covered a part of the features of regular
expressions.  In this section, we'll cover some new metacharacters, and
how to use groups to retrieve portions of the text that was matched.

11.7.4.1 More Metacharacters
............................

There are some metacharacters that we haven't covered yet.  Most of
them will be covered in this section.

  Some of the remaining metacharacters to be discussed are _zero-width
assertions_.  They don't cause the engine to advance through the string;
instead, they consume no characters at all, and simply succeed or fail.
For example, `\b' is an assertion that the current position is located
at a word boundary; the position isn't changed by the `\b' at all.
This means that zero-width assertions should never be repeated, because
if they match once at a given location, they can obviously be matched
an infinite number of times.

`|'
     Alternation, or the "or" operator.   If A and B are regular
     expressions, `A|B' will match any string that matches either `A'
     or `B'. `|' has very low precedence in order to make it work
     reasonably when you're alternating multi-character strings.
     `Crow|Servo' will match either `Crow' or `Servo', not `Cro', a
     `'w'' or an `'S'', and `ervo'.

     To match a literal `'|'', use `\|', or enclose it inside a
     character class, as in `[|]'.

`^'
     Matches at the beginning of lines.  Unless the `MULTILINE' flag
     has been set, this will only match at the beginning of the string.
     In `MULTILINE' mode, this also matches immediately after each
     newline within the string.

     For example, if you wish to match the word `From' only at the
     beginning of a line, the RE to use is `^From'.

         >>> print re.search('^From', 'From Here to Eternity')
         <re.MatchObject instance at 80c1520>
         >>> print re.search('^From', 'Reciting From Memory')
         None

`$'
     Matches at the end of a line, which is defined as either the end
     of the string, or any location followed by a newline character.

         >>> print re.search('}$', '{block}')
         <re.MatchObject instance at 80adfa8>
         >>> print re.search('}$', '{block} ')
         None
         >>> print re.search('}$', '{block}\n')
         <re.MatchObject instance at 80adfa8>

     To match a literal `'$'', use `\$' or enclose it inside a
     character class, as in  `[$]'.

`\A'
     Matches only at the start of the string.  When not in `MULTILINE'
     mode, `\A' and `^' are effectively the same.  In `MULTILINE' mode,
     they're different: `\A' still matches only at the beginning of the
     string, but `^' may match at any location inside the string that
     follows a newline character.

`\Z'
     Matches only at the end of the string.

`\b'
     Word boundary.  This is a zero-width assertion that matches only
     at the beginning or end of a word.  A word is defined as a
     sequence of alphanumeric characters, so the end of a word is
     indicated by whitespace or a non-alphanumeric character.

     The following example matches `class' only when it's a complete
     word; it won't match when it's contained inside another word.

         >>> p = re.compile(r'\bclass\b')
         >>> print p.search('no class at all')
         <re.MatchObject instance at 80c8f28>
         >>> print p.search('the declassified algorithm')
         None
         >>> print p.search('one subclass is')
         None

     There are two subtleties you should remember when using this
     special sequence.  First, this is the worst collision between
     Python's string literals and regular expression sequences.  In
     Python's string literals, `\b' is the backspace character, ASCII
     value 8.  If you're not using raw strings, then Python will
     convert the `\b' to a backspace, and your RE won't match as you
     expect it to.  The following example looks the same as our
     previous RE, but omits the `'r'' in front of the RE string.

         >>> p = re.compile('\bclass\b')
         >>> print p.search('no class at all')
         None
         >>> print p.search('\b' + 'class' + '\b')
         <re.MatchObject instance at 80c3ee0>

     Second, inside a character class, where there's no use for this
     assertion, `\b' represents the backspace character, for
     compatibility with Python's string literals.

`\B'
     Another zero-width assertion, this is the opposite of `\b', only
     matching when the current position is not at a word boundary.

11.7.4.2 Grouping
.................

Frequently you need to obtain more information than just whether the RE
matched or not.  Regular expressions are often used to dissect strings
by writing a RE divided into several subgroups which match different
components of interest.  For example, an RFC-822 header line is divided
into a header name and a value, separated by a `':'', like this:

    From: author@example.com
    User-Agent: Thunderbird 1.5.0.9 (X11/20061227)
    MIME-Version: 1.0
    To: editor@example.com

This can be handled by writing a regular expression which matches an
entire header line, and has one group which matches the header name,
and another group which matches the header's value.

  Groups are marked by the `'('', `')'' metacharacters. `'('' and `')''
have much the same meaning as they do in mathematical expressions; they
group together the expressions contained inside them, and you can
repeat the contents of a group with a repeating qualifier, such as `*',
`+', `?', or `{m,n}'.  For example, `(ab)*' will match zero or more
repetitions of `ab'.

    >>> p = re.compile('(ab)*')
    >>> print p.match('ababababab').span()
    (0, 10)

Groups indicated with `'('', `')'' also capture the starting and ending
index of the text that they match; this can be retrieved by passing an
argument to `group()', `start()', `end()', and `span()'.  Groups are
numbered starting with 0.  Group 0 is always present; it's the whole
RE, so `MatchObject' methods all have group 0 as their default
argument.  Later we'll see how to express groups that don't capture the
span of text that they match.

    >>> p = re.compile('(a)b')
    >>> m = p.match('ab')
    >>> m.group()
    'ab'
    >>> m.group(0)
    'ab'

Subgroups are numbered from left to right, from 1 upward.  Groups can
be nested; to determine the number, just count the opening parenthesis
characters, going from left to right.

    >>> p = re.compile('(a(b)c)d')
    >>> m = p.match('abcd')
    >>> m.group(0)
    'abcd'
    >>> m.group(1)
    'abc'
    >>> m.group(2)
    'b'

`group()' can be passed multiple group numbers at a time, in which case
it will return a tuple containing the corresponding values for those
groups.

    >>> m.group(2,1,2)
    ('b', 'abc', 'b')

The `groups()' method returns a tuple containing the strings for all the
subgroups, from 1 up to however many there are.

    >>> m.groups()
    ('abc', 'b')

Backreferences in a pattern allow you to specify that the contents of
an earlier capturing group must also be found at the current location
in the string.  For example, `\1' will succeed if the exact contents of
group 1 can be found at the current position, and fails otherwise.
Remember that Python's string literals also use a backslash followed by
numbers to allow including arbitrary characters in a string, so be sure
to use a raw string when incorporating backreferences in a RE.

  For example, the following RE detects doubled words in a string.

    >>> p = re.compile(r'(\b\w+)\s+\1')
    >>> p.search('Paris in the the spring').group()
    'the the'

Backreferences like this aren't often useful for just searching through
a string -- there are few text formats which repeat data in this way --
but you'll soon find out that they're _very_ useful when performing
string substitutions.

11.7.4.3 Non-capturing and Named Groups
.......................................

Elaborate REs may use many groups, both to capture substrings of
interest, and to group and structure the RE itself.  In complex REs, it
becomes difficult to keep track of the group numbers.  There are two
features which help with this problem.  Both of them use a common
syntax for regular expression extensions, so we'll look at that first.

  Perl 5 added several additional features to standard regular
expressions, and the Python re module supports most of
them.   It would have been difficult to choose new single-keystroke
metacharacters or new special sequences beginning with `\' to represent
the new features without making Perl's regular expressions confusingly
different from standard REs.  If you chose `&' as a new metacharacter,
for example, old expressions would be assuming that `&' was a regular
character and wouldn't have escaped it by writing `\&' or `[&]'.

  The solution chosen by the Perl developers was to use `(?...)' as the
extension syntax.  `?' immediately after a parenthesis was a syntax
error because the `?' would have nothing to repeat, so this didn't
introduce any compatibility problems.  The characters immediately after
the `?'  indicate what extension is being used, so `(?=foo)' is one
thing (a positive lookahead assertion) and `(?:foo)' is something else
(a non-capturing group containing the subexpression `foo').

  Python adds an extension syntax to Perl's extension syntax.  If the
first character after the question mark is a `P', you know that it's an
extension that's specific to Python.  Currently there are two such
extensions: `(?P<name>...)' defines a named group, and `(?P=name)' is a
backreference to a named group.  If future versions of Perl 5 add
similar features using a different syntax, the re module
will be changed to support the new syntax, while preserving the
Python-specific syntax for compatibility's sake.

  Now that we've looked at the general extension syntax, we can return
to the features that simplify working with groups in complex REs. Since
groups are numbered from left to right and a complex expression may use
many groups, it can become difficult to keep track of the correct
numbering.  Modifying such a complex RE is annoying, too: insert a new
group near the beginning and you change the numbers of everything that
follows it.

  Sometimes you'll want to use a group to collect a part of a regular
expression, but aren't interested in retrieving the group's contents.
You can make this fact explicit by using a non-capturing group:
`(?:...)', where you can replace the `...' with any other regular
expression.

    >>> m = re.match("([abc])+", "abc")
    >>> m.groups()
    ('c',)
    >>> m = re.match("(?:[abc])+", "abc")
    >>> m.groups()
    ()

Except for the fact that you can't retrieve the contents of what the
group matched, a non-capturing group behaves exactly the same as a
capturing group; you can put anything inside it, repeat it with a
repetition metacharacter such as `*', and nest it within other groups
(capturing or non-capturing).  `(?:...)' is particularly useful when
modifying an existing pattern, since you can add new groups without
changing how all the other groups are numbered.  It should be mentioned
that there's no performance difference in searching between capturing
and non-capturing groups; neither form is any faster than the other.

  A more significant feature is named groups: instead of referring to
them by numbers, groups can be referenced by a name.

  The syntax for a named group is one of the Python-specific extensions:
`(?P<name>...)'.  _name_ is, obviously, the name of the group.  Named
groups also behave exactly like capturing groups, and additionally
associate a name with a group.  The `MatchObject' methods that deal
with capturing groups all accept either integers that refer to the
group by number or strings that contain the desired group's name.
Named groups are still given numbers, so you can retrieve information
about a group in two ways:

    >>> p = re.compile(r'(?P<word>\b\w+\b)')
    >>> m = p.search( '(((( Lots of punctuation )))' )
    >>> m.group('word')
    'Lots'
    >>> m.group(1)
    'Lots'

Named groups are handy because they let you use easily-remembered
names, instead of having to remember numbers.  Here's an example RE
from the imaplib.  module:

    InternalDate = re.compile(r'INTERNALDATE "'
            r'(?P<day>[ 123][0-9])-(?P<mon>[A-Z][a-z][a-z])-'
            r'(?P<year>[0-9][0-9][0-9][0-9])'
            r' (?P<hour>[0-9][0-9]):(?P<min>[0-9][0-9]):(?P<sec>[0-9][0-9])'
            r' (?P<zonen>[-+])(?P<zoneh>[0-9][0-9])(?P<zonem>[0-9][0-9])'
            r'"')

It's obviously much easier to retrieve `m.group('zonem')', instead of
having to remember to retrieve group 9.

  The syntax for backreferences in an expression such as `(...)\1'
refers to the number of the group.  There's naturally a variant that
uses the group name instead of the number. This is another Python
extension: `(?P=name)' indicates that the contents of the group called
_name_ should again be matched at the current point.  The regular
expression for finding doubled words, `(\b\w+)\s+\1' can also be
written as `(?P<word>\b\w+)\s+(?P=word)':

    >>> p = re.compile(r'(?P<word>\b\w+)\s+(?P=word)')
    >>> p.search('Paris in the the spring').group()
    'the the'

11.7.4.4 Lookahead Assertions
.............................

Another zero-width assertion is the lookahead assertion.  Lookahead
assertions are available in both positive and negative form, and  look
like this:

`(?=...)'
     Positive lookahead assertion.  This succeeds if the contained
     regular expression, represented here by `...', successfully
     matches at the current location, and fails otherwise. But, once
     the contained expression has been tried, the matching engine
     doesn't advance at all; the rest of the pattern is tried right
     where the assertion started.

`(?!...)'
     Negative lookahead assertion.  This is the opposite of the
     positive assertion; it succeeds if the contained expression
     _doesn't_ match at the current position in the string.

  To make this concrete, let's look at a case where a lookahead is
useful.  Consider a simple pattern to match a filename and split it
apart into a base name and an extension, separated by a `.'.  For
example, in `news.rc', `news' is the base name, and `rc' is the
filename's extension.

  The pattern to match this is quite simple:

  `.*[.].*$'

  Notice that the `.' needs to be treated specially because it's a
metacharacter; I've put it inside a character class.  Also notice the
trailing `$'; this is added to ensure that all the rest of the string
must be included in the extension.  This regular expression matches
`foo.bar' and `autoexec.bat' and `sendmail.cf' and `printers.conf'.

  Now, consider complicating the problem a bit; what if you want to
match filenames where the extension is not `bat'? Some incorrect
attempts:

  `.*[.][^b].*$'  The first attempt above tries to exclude `bat' by
requiring that the first character of the extension is not a `b'.  This
is wrong, because the pattern also doesn't match `foo.bar'.

  `.*[.]([^b]..|.[^a].|..[^t])$'

  The expression gets messier when you try to patch up the first
solution by requiring one of the following cases to match: the first
character of the extension isn't `b'; the second character isn't `a';
or the third character isn't `t'.  This accepts `foo.bar' and rejects
`autoexec.bat', but it requires a three-letter extension and won't
accept a filename with a two-letter extension such as `sendmail.cf'.
We'll complicate the pattern again in an effort to fix it.

  `.*[.]([^b].?.?|.[^a]?.?|..?[^t]?)$'

  In the third attempt, the second and third letters are all made
optional in order to allow matching extensions shorter than three
characters, such as `sendmail.cf'.

  The pattern's getting really complicated now, which makes it hard to
read and understand.  Worse, if the problem changes and you want to
exclude both `bat' and `exe' as extensions, the pattern would get even
more complicated and confusing.

  A negative lookahead cuts through all this confusion:

  `.*[.](?!bat$).*$'  The negative lookahead means: if the expression
`bat' doesn't match at this point, try the rest of the pattern; if
`bat$' does match, the whole pattern will fail.  The trailing `$' is
required to ensure that something like `sample.batch', where the
extension only starts with `bat', will be allowed.

  Excluding another filename extension is now easy; simply add it as an
alternative inside the assertion.  The following pattern excludes
filenames that end in either `bat' or `exe':

  `.*[.](?!bat$|exe$).*$'

11.7.5 Modifying Strings
------------------------

Up to this point, we've simply performed searches against a static
string.  Regular expressions are also commonly used to modify strings
in various ways, using the following pattern methods:

Method/Attribute       Purpose
---------------------------------------------------------------------------
`split()'              Split the string into a list, splitting it
                       wherever the RE matches
`sub()'                Find all substrings where the RE matches, and
                       replace them with a different string
`subn()'               Does the same thing as `sub()',  but returns the
                       new string and the number of replacements

11.7.5.1 Splitting Strings
..........................

The `split()' method of a pattern splits a string apart wherever the RE
matches, returning a list of the pieces. It's similar to the `split()'
method of strings but provides much more generality in the delimiters
that you can split by; `split()' only supports splitting by whitespace
or by a fixed string.  As you'd expect, there's a module-level
re.split() function, too.

 -- Method: .split (string[, maxsplit=0])
     Split _string_ by the matches of the regular expression.  If
     capturing parentheses are used in the RE, then their contents will
     also be returned as part of the resulting list.  If _maxsplit_ is
     nonzero, at most _maxsplit_ splits are performed.

  You can limit the number of splits made, by passing a value for
_maxsplit_.  When _maxsplit_ is nonzero, at most _maxsplit_ splits will
be made, and the remainder of the string is returned as the final
element of the list.  In the following example, the delimiter is any
sequence of non-alphanumeric characters.

    >>> p = re.compile(r'\W+')
    >>> p.split('This is a test, short and sweet, of split().')
    ['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', '']
    >>> p.split('This is a test, short and sweet, of split().', 3)
    ['This', 'is', 'a', 'test, short and sweet, of split().']

Sometimes you're not only interested in what the text between
delimiters is, but also need to know what the delimiter was.  If
capturing parentheses are used in the RE, then their values are also
returned as part of the list.  Compare the following calls:

    >>> p = re.compile(r'\W+')
    >>> p2 = re.compile(r'(\W+)')
    >>> p.split('This... is a test.')
    ['This', 'is', 'a', 'test', '']
    >>> p2.split('This... is a test.')
    ['This', '... ', 'is', ' ', 'a', ' ', 'test', '.', '']

The module-level function re.split() adds the RE to be used
as the first argument, but is otherwise the same.

    >>> re.split('[\W]+', 'Words, words, words.')
    ['Words', 'words', 'words', '']
    >>> re.split('([\W]+)', 'Words, words, words.')
    ['Words', ', ', 'words', ', ', 'words', '.', '']
    >>> re.split('[\W]+', 'Words, words, words.', 1)
    ['Words', 'words, words.']

11.7.5.2 Search and Replace
...........................

Another common task is to find all the matches for a pattern, and
replace them with a different string.  The `sub()' method takes a
replacement value, which can be either a string or a function, and the
string to be processed.

 -- Method: .sub (replacement, string[, count=0])
     Returns the string obtained by replacing the leftmost
     non-overlapping occurrences of the RE in _string_ by the
     replacement _replacement_.  If the pattern isn't found, _string_
     is returned unchanged.

     The optional argument _count_ is the maximum number of pattern
     occurrences to be replaced; _count_ must be a non-negative
     integer.  The default value of 0 means to replace all occurrences.

  Here's a simple example of using the `sub()' method.  It replaces
colour names with the word `colour':

    >>> p = re.compile( '(blue|white|red)')
    >>> p.sub( 'colour', 'blue socks and red shoes')
    'colour socks and colour shoes'
    >>> p.sub( 'colour', 'blue socks and red shoes', count=1)
    'colour socks and red shoes'

The `subn()' method does the same work, but returns a 2-tuple
containing the new string value and the number of replacements  that
were performed:

    >>> p = re.compile( '(blue|white|red)')
    >>> p.subn( 'colour', 'blue socks and red shoes')
    ('colour socks and colour shoes', 2)
    >>> p.subn( 'colour', 'no colours at all')
    ('no colours at all', 0)

Empty matches are replaced only when they're not adjacent to a previous
match.

    >>> p = re.compile('x*')
    >>> p.sub('-', 'abxd')
    '-a-b-d-'

If _replacement_ is a string, any backslash escapes in it are
processed.  That is, `\n' is converted to a single newline character,
`\r' is converted to a carriage return, and so forth. Unknown escapes
such as `\j' are left alone.  Backreferences, such as `\6', are
replaced with the substring matched by the corresponding group in the
RE.  This lets you incorporate portions of the original text in the
resulting replacement string.

  This example matches the word `section' followed by a string enclosed
in `{', `}', and changes `section' to `subsection':

    >>> p = re.compile('section{ ( [^}]* ) }', re.VERBOSE)
    >>> p.sub(r'subsection{\1}','section{First} section{second}')
    'subsection{First} subsection{second}'

There's also a syntax for referring to named groups as defined by the
`(?P<name>...)' syntax.  `\g<name>' will use the substring matched by
the group named `name', and  `\g<number>'  uses the corresponding group
number.  `\g<2>' is therefore equivalent to `\2',  but isn't ambiguous
in a replacement string such as `\g<2>0'.  (`\20' would be interpreted
as a reference to group 20, not a reference to group 2 followed by the
literal character `'0''.)  The following substitutions are all
equivalent, but use all three variations of the replacement string.

    >>> p = re.compile('section{ (?P<name> [^}]* ) }', re.VERBOSE)
    >>> p.sub(r'subsection{\1}','section{First}')
    'subsection{First}'
    >>> p.sub(r'subsection{\g<1>}','section{First}')
    'subsection{First}'
    >>> p.sub(r'subsection{\g<name>}','section{First}')
    'subsection{First}'

_replacement_ can also be a function, which gives you even more
control.  If _replacement_ is a function, the function is called for
every non-overlapping occurrence of _pattern_.  On each call, the
function is  passed a `MatchObject' argument for the match and can use
this information to compute the desired replacement string and return
it.

  In the following example, the replacement function translates
decimals into hexadecimal:

    >>> def hexrepl( match ):
    ...     "Return the hex string for a decimal number"
    ...     value = int( match.group() )
    ...     return hex(value)
    ...
    >>> p = re.compile(r'\d+')
    >>> p.sub(hexrepl, 'Call 65490 for printing, 49152 for user code.')
    'Call 0xffd2 for printing, 0xc000 for user code.'

When using the module-level re.sub() function, the pattern
is passed as the first argument.  The pattern may be provided as an
object or as a string; if you need to specify regular expression flags,
you must either use a pattern object as the first parameter, or use
embedded modifiers in the pattern string, e.g. `sub("(?i)b+", "x",
"bbbb BBBB")' returns `'x x''.

11.7.6 Common Problems
----------------------

Regular expressions are a powerful tool for some applications, but in
some ways their behaviour isn't intuitive and at times they don't
behave the way you may expect them to.  This section will point out
some of the most common pitfalls.

11.7.6.1 Use String Methods
...........................

Sometimes using the re module is a mistake.  If you're
matching a fixed string, or a single character class, and you're not
using any re features such as the `IGNORECASE' flag, then
the full power of regular expressions may not be required. Strings have
several methods for performing operations with fixed strings and
they're usually much faster, because the implementation is a single
small C loop that's been optimized for the purpose, instead of the
large, more generalized regular expression engine.

  One example might be replacing a single fixed string with another
one; for example, you might replace `word' with `deed'.  `re.sub()'
seems like the function to use for this, but consider the `replace()'
method.  Note that `replace()' will also replace `word' inside words,
turning `swordfish' into `sdeedfish', but the  naive RE `word' would
have done that, too.  (To avoid performing the substitution on parts of
words, the pattern would have to be `\bword\b', in order to require
that `word' have a word boundary on either side.  This takes the job
beyond  `replace()''s abilities.)

  Another common task is deleting every occurrence of a single
character from a string or replacing it with another single character.
You might do this with something like `re.sub('\n', ' ', S)', but
`translate()' is capable of doing both tasks and will be faster than
any regular expression operation can be.

  In short, before turning to the re module, consider
whether your problem can be solved with a faster and simpler string
method.

11.7.6.2 match() versus search()
................................

The `match()' function only checks if the RE matches at the beginning
of the string while `search()' will scan forward through the string for
a match.  It's important to keep this distinction in mind.  Remember,
`match()' will only report a successful match which will start at 0; if
the match wouldn't start at zero,  `match()' will _not_ report it.

    >>> print re.match('super', 'superstition').span()
    (0, 5)
    >>> print re.match('super', 'insuperable')
    None

On the other hand, `search()' will scan forward through the string,
reporting the first match it finds.

    >>> print re.search('super', 'superstition').span()
    (0, 5)
    >>> print re.search('super', 'insuperable').span()
    (2, 7)

Sometimes you'll be tempted to keep using re.match(), and
just add `.*' to the front of your RE.  Resist this temptation and use
re.search().  instead.  The regular expression compiler does
some analysis of REs in order to speed up the process of looking for a
match.  One such analysis figures out what the first character of a
match must be; for example, a pattern starting with `Crow' must match
starting with a `'C''.  The analysis lets the engine quickly scan
through the string looking for the starting character, only trying the
full match if a `'C'' is found.

  Adding `.*' defeats this optimization, requiring scanning to the end
of the string and then backtracking to find a match for the rest of the
RE.  Use re.search() instead.

11.7.6.3 Greedy versus Non-Greedy
.................................

When repeating a regular expression, as in `a*', the resulting action
is to consume as much of the pattern as possible.  This fact often
bites you when you're trying to match a pair of balanced delimiters,
such as the angle brackets surrounding an HTML tag.  The naive pattern
for matching a single HTML tag doesn't work because of the greedy
nature of `.*'.

    >>> s = '<html><head><title>Title</title>'
    >>> len(s)
    32
    >>> print re.match('<.*>', s).span()
    (0, 32)
    >>> print re.match('<.*>', s).group()
    <html><head><title>Title</title>

The RE matches the `'<'' in `<html>', and the `.*' consumes the rest of
the string.  There's still more left in the RE, though, and the `>'
can't match at the end of the string, so the regular expression engine
has to backtrack character by character until it finds a match for the
`>'.   The final match extends from the `'<'' in `<html>' to the `'>''
in `</title>', which isn't what you want.

  In this case, the solution is to use the non-greedy qualifiers `*?',
`+?', `??', or `{m,n}?', which match as _little_ text as possible.  In
the above example, the `'>'' is tried immediately after the first `'<''
matches, and when it fails, the engine advances a character at a time,
retrying the `'>'' at every step.  This produces just the right result:

    >>> print re.match('<.*?>', s).group()
    <html>

(Note that parsing HTML or XML with regular expressions is painful.
Quick-and-dirty patterns will handle common cases, but HTML and XML
have special cases that will break the obvious regular expression; by
the time you've written a regular expression that handles all of the
possible cases, the patterns will be _very_ complicated.  Use an HTML
or XML parser module for such tasks.)

11.7.6.4 Not Using re.VERBOSE
.............................

By now you've probably noticed that regular expressions are a very
compact notation, but they're not terribly readable.  REs of moderate
complexity can become lengthy collections of backslashes, parentheses,
and metacharacters, making them difficult to read and understand.

  For such REs, specifying the `re.VERBOSE' flag when compiling the
regular expression can be helpful, because it allows you to format the
regular expression more clearly.

  The `re.VERBOSE' flag has several effects.  Whitespace in the regular
expression that _isn't_ inside a character class is ignored.  This
means that an expression such as `dog | cat' is equivalent to the less
readable `dog|cat', but `[a b]' will still match the characters `'a'',
`'b'', or a space.  In addition, you can also put comments inside a RE;
comments extend from a `#' character to the next newline.  When used
with triple-quoted strings, this enables REs to be formatted more
neatly:

    pat = re.compile(r"""
     \s*                 # Skip leading whitespace
     (?P<header>[^:]+)   # Header name
     \s* :               # Whitespace, and a colon
     (?P<value>.*?)      # The header's value -- *? used to
                         # lose the following trailing whitespace
     \s*$                # Trailing whitespace to end-of-line
    """, re.VERBOSE)

This is far more readable than:

    pat = re.compile(r"\s*(?P<header>[^:]+)\s*:(?P<value>.*?)\s*$")

11.7.7 Feedback
---------------

Regular expressions are a complicated topic.  Did this document help you
understand them?  Were there parts that were unclear, or Problems you
encountered that weren't covered here?  If so, please send suggestions
for improvements to the author.

  The most complete book on regular expressions is almost certainly
Jeffrey Friedl's Mastering Regular Expressions, published by O'Reilly.
Unfortunately, it exclusively concentrates on Perl and Java's flavours
of regular expressions, and doesn't contain any Python material at all,
so it won't be useful as a reference for programming in Python.  (The
first edition covered Python's now-removed `regex' module, which won't
help you much.)  Consider checking it out from your library.

Footnotes
.........

     (1) Introduced in Python 2.2.2.

11.8 Socket Programming HOWTO
=============================

     Author: Gordon McMillan

Abstract
........

     Sockets are used nearly everywhere, but are one of the most
     severely misunderstood technologies around. This is a 10,000 foot
     overview of sockets.  It's not really a tutorial - you'll still
     have work to do in getting things operational. It doesn't cover
     the fine points (and there are a lot of them), but I hope it will
     give you enough background to begin using them decently.

11.8.1 Sockets
--------------

Sockets are used nearly everywhere, but are one of the most severely
misunderstood technologies around. This is a 10,000 foot overview of
sockets.  It's not really a tutorial - you'll still have work to do in
getting things working. It doesn't cover the fine points (and there are
a lot of them), but I hope it will give you enough background to begin
using them decently.

  I'm only going to talk about INET sockets, but they account for at
least 99% of the sockets in use. And I'll only talk about STREAM
sockets - unless you really know what you're doing (in which case this
HOWTO isn't for you!), you'll get better behavior and performance from
a STREAM socket than anything else. I will try to clear up the mystery
of what a socket is, as well as some hints on how to work with blocking
and non-blocking sockets. But I'll start by talking about blocking
sockets. You'll need to know how they work before dealing with
non-blocking sockets.

  Part of the trouble with understanding these things is that "socket"
can mean a number of subtly different things, depending on context. So
first, let's make a distinction between a "client" socket - an endpoint
of a conversation, and a "server" socket, which is more like a
switchboard operator. The client application (your browser, for
example) uses "client" sockets exclusively; the web server it's talking
to uses both "server" sockets and "client" sockets.

11.8.1.1 History
................

Of the various forms of IPC (_Inter Process Communication_), sockets
are by far the most popular.  On any given platform, there are likely
to be other forms of IPC that are faster, but for cross-platform
communication, sockets are about the only game in town.

  They were invented in Berkeley as part of the BSD flavor of Unix.
They spread like wildfire with the Internet. With good reason -- the
combination of sockets with INET makes talking to arbitrary machines
around the world unbelievably easy (at least compared to other schemes).

11.8.2 Creating a Socket
------------------------

Roughly speaking, when you clicked on the link that brought you to this
page, your browser did something like the following:

    #create an INET, STREAMing socket
    s = socket.socket(
        socket.AF_INET, socket.SOCK_STREAM)
    #now connect to the web server on port 80
    # - the normal http port
    s.connect(("www.mcmillan-inc.com", 80))

When the `connect' completes, the socket `s' can now be used to send in
a request for the text of this page. The same socket will read the
reply, and then be destroyed. That's right - destroyed. Client sockets
are normally only used for one exchange (or a small set of sequential
exchanges).

  What happens in the web server is a bit more complex. First, the web
server creates a "server socket".

    #create an INET, STREAMing socket
    serversocket = socket.socket(
        socket.AF_INET, socket.SOCK_STREAM)
    #bind the socket to a public host,
    # and a well-known port
    serversocket.bind((socket.gethostname(), 80))
    #become a server socket
    serversocket.listen(5)

A couple things to notice: we used `socket.gethostname()' so that the
socket would be visible to the outside world. If we had used
`s.bind(('', 80))' or `s.bind(('localhost', 80))' or
`s.bind(('127.0.0.1', 80))' we would still have a "server" socket, but
one that was only visible within the same machine.

  A second thing to note: low number ports are usually reserved for
"well known" services (HTTP, SNMP etc). If you're playing around, use a
nice high number (4 digits).

  Finally, the argument to `listen' tells the socket library that we
want it to queue up as many as 5 connect requests (the normal max)
before refusing outside connections. If the rest of the code is written
properly, that should be plenty.

  OK, now we have a "server" socket, listening on port 80. Now we enter
the mainloop of the web server:

    while 1:
        #accept connections from outside
        (clientsocket, address) = serversocket.accept()
        #now do something with the clientsocket
        #in this case, we'll pretend this is a threaded server
        ct = client_thread(clientsocket)
        ct.run()

There's actually 3 general ways in which this loop could work -
dispatching a thread to handle `clientsocket', create a new process to
handle `clientsocket', or restructure this app to use non-blocking
sockets, and mulitplex between our "server" socket and any active
`clientsocket's using `select'. More about that later. The important
thing to understand now is this: this is _all_ a "server" socket does.
It doesn't send any data. It doesn't receive any data. It just produces
"client" sockets. Each `clientsocket' is created in response to some
_other_ "client" socket doing a `connect()' to the host and port we're
bound to. As soon as we've created that `clientsocket', we go back to
listening for more connections. The two "clients" are free to chat it
up - they are using some dynamically allocated port which will be
recycled when the conversation ends.

11.8.2.1 IPC
............

If you need fast IPC between two processes on one machine, you should
look into whatever form of shared memory the platform offers. A simple
protocol based around shared memory and locks or semaphores is by far
the fastest technique.

  If you do decide to use sockets, bind the "server" socket to
`'localhost''. On most platforms, this will take a shortcut around a
couple of layers of network code and be quite a bit faster.

11.8.3 Using a Socket
---------------------

The first thing to note, is that the web browser's "client" socket and
the web server's "client" socket are identical beasts. That is, this is
a "peer to peer" conversation. Or to put it another way, _as the
designer, you will have to decide what the rules of etiquette are for a
conversation_. Normally, the `connect'ing socket starts the
conversation, by sending in a request, or perhaps a signon. But that's
a design decision - it's not a rule of sockets.

  Now there are two sets of verbs to use for communication. You can use
`send' and `recv', or you can transform your client socket into a
file-like beast and use `read' and `write'. The latter is the way Java
presents their sockets.  I'm not going to talk about it here, except to
warn you that you need to use `flush' on sockets. These are buffered
"files", and a common mistake is to `write' something, and then `read'
for a reply. Without a `flush' in there, you may wait forever for the
reply, because the request may still be in your output buffer.

  Now we come the major stumbling block of sockets - `send' and `recv'
operate on the network buffers. They do not necessarily handle all the
bytes you hand them (or expect from them), because their major focus is
handling the network buffers. In general, they return when the
associated network buffers have been filled (`send') or emptied
(`recv'). They then tell you how many bytes they handled. It is _your_
responsibility to call them again until your message has been
completely dealt with.

  When a `recv' returns 0 bytes, it means the other side has closed (or
is in the process of closing) the connection.  You will not receive any
more data on this connection. Ever.  You may be able to send data
successfully; I'll talk about that some on the next page.

  A protocol like HTTP uses a socket for only one transfer. The client
sends a request, the reads a reply.  That's it. The socket is
discarded. This means that a client can detect the end of the reply by
receiving 0 bytes.

  But if you plan to reuse your socket for further transfers, you need
to realize that _there is no "EOT" (End of Transfer) on a socket._ I
repeat: if a socket `send' or `recv' returns after handling 0 bytes,
the connection has been broken.  If the connection has _not_ been
broken, you may wait on a `recv' forever, because the socket will _not_
tell you that there's nothing more to read (for now).  Now if you think
about that a bit, you'll come to realize a fundamental truth of
sockets: _messages must either be fixed length_ (yuck), _or be
delimited_ (shrug), _or indicate how long they are_ (much better), _or
end by shutting down the connection_. The choice is entirely yours,
(but some ways are righter than others).

  Assuming you don't want to end the connection, the simplest solution
is a fixed length message:

    class mysocket:
        '''demonstration class only
          - coded for clarity, not efficiency
        '''

        def __init__(self, sock=None):
            if sock is None:
                self.sock = socket.socket(
                    socket.AF_INET, socket.SOCK_STREAM)
            else:
                self.sock = sock

        def connect(self, host, port):
            self.sock.connect((host, port))

        def mysend(self, msg):
            totalsent = 0
            while totalsent < MSGLEN:
                sent = self.sock.send(msg[totalsent:])
                if sent == 0:
                    raise RuntimeError("socket connection broken")
                totalsent = totalsent + sent

        def myreceive(self):
            msg = ''
            while len(msg) < MSGLEN:
                chunk = self.sock.recv(MSGLEN-len(msg))
                if chunk == '':
                    raise RuntimeError("socket connection broken")
                msg = msg + chunk
            return msg

The sending code here is usable for almost any messaging scheme - in
Python you send strings, and you can use `len()' to determine its
length (even if it has embedded `\0' characters). It's mostly the
receiving code that gets more complex. (And in C, it's not much worse,
except you can't use `strlen' if the message has embedded `\0's.)

  The easiest enhancement is to make the first character of the message
an indicator of message type, and have the type determine the length.
Now you have two `recv's - the first to get (at least) that first
character so you can look up the length, and the second in a loop to
get the rest. If you decide to go the delimited route, you'll be
receiving in some arbitrary chunk size, (4096 or 8192 is frequently a
good match for network buffer sizes), and scanning what you've received
for a delimiter.

  One complication to be aware of: if your conversational protocol
allows multiple messages to be sent back to back (without some kind of
reply), and you pass `recv' an arbitrary chunk size, you may end up
reading the start of a following message. You'll need to put that aside
and hold onto it, until it's needed.

  Prefixing the message with it's length (say, as 5 numeric characters)
gets more complex, because (believe it or not), you may not get all 5
characters in one `recv'. In playing around, you'll get away with it;
but in high network loads, your code will very quickly break unless you
use two `recv' loops - the first to determine the length, the second to
get the data part of the message. Nasty.  This is also when you'll
discover that `send' does not always manage to get rid of everything in
one pass. And despite having read this, you will eventually get bit by
it!

  In the interests of space, building your character, (and preserving my
competitive position), these enhancements are left as an exercise for
the reader. Lets move on to cleaning up.

11.8.3.1 Binary Data
....................

It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00.  Socket libraries have calls for
converting 16 and 32 bit integers - `ntohl, htonl, ntohs, htons' where
"n" means _network_ and "h" means _host_, "s" means _short_ and "l"
means _long_. Where network order is host order, these do nothing, but
where the machine is byte-reversed, these swap the bytes around
appropriately.

  In these days of 32 bit machines, the ascii representation of binary
data is frequently smaller than the binary representation. That's
because a surprising amount of the time, all those longs have the value
0, or maybe 1. The string "0" would be two bytes, while binary is four.
Of course, this doesn't fit well with fixed-length messages. Decisions,
decisions.

11.8.4 Disconnecting
--------------------

Strictly speaking, you're supposed to use `shutdown' on a socket before
you `close' it.  The `shutdown' is an advisory to the socket at the
other end.  Depending on the argument you pass it, it can mean "I'm not
going to send anymore, but I'll still listen", or "I'm not listening,
good riddance!".  Most socket libraries, however, are so used to
programmers neglecting to use this piece of etiquette that normally a
`close' is the same as `shutdown(); close()'.  So in most situations,
an explicit `shutdown' is not needed.

  One way to use `shutdown' effectively is in an HTTP-like exchange.
The client sends a request and then does a `shutdown(1)'. This tells
the server "This client is done sending, but can still receive."  The
server can detect "EOF" by a receive of 0 bytes. It can assume it has
the complete request.  The server sends a reply. If the `send'
completes successfully then, indeed, the client was still receiving.

  Python takes the automatic shutdown a step further, and says that
when a socket is garbage collected, it will automatically do a `close'
if it's needed. But relying on this is a very bad habit. If your socket
just disappears without doing a `close', the socket at the other end
may hang indefinitely, thinking you're just being slow. _Please_
`close' your sockets when you're done.

11.8.4.1 When Sockets Die
.........................

Probably the worst thing about using blocking sockets is what happens
when the other side comes down hard (without doing a `close'). Your
socket is likely to hang. SOCKSTREAM is a reliable protocol, and it
will wait a long, long time before giving up on a connection. If you're
using threads, the entire thread is essentially dead. There's not much
you can do about it. As long as you aren't doing something dumb, like
holding a lock while doing a blocking read, the thread isn't really
consuming much in the way of resources. Do _not_ try to kill the thread
- part of the reason that threads are more efficient than processes is
that they avoid the overhead associated with the automatic recycling of
resources. In other words, if you do manage to kill the thread, your
whole process is likely to be screwed up.

11.8.5 Non-blocking Sockets
---------------------------

If you've understood the preceding, you already know most of what you
need to know about the mechanics of using sockets. You'll still use the
same calls, in much the same ways. It's just that, if you do it right,
your app will be almost inside-out.

  In Python, you use `socket.setblocking(0)' to make it non-blocking.
In C, it's more complex, (for one thing, you'll need to choose between
the BSD flavor `O_NONBLOCK' and the almost indistinguishable Posix
flavor `O_NDELAY', which is completely different from `TCP_NODELAY'),
but it's the exact same idea. You do this after creating the socket,
but before using it. (Actually, if you're nuts, you can switch back and
forth.)

  The major mechanical difference is that `send', `recv', `connect' and
`accept' can return without having done anything. You have (of course) a
number of choices. You can check return code and error codes and
generally drive yourself crazy. If you don't believe me, try it
sometime. Your app will grow large, buggy and suck CPU. So let's skip
the brain-dead solutions and do it right.

  Use `select'.

  In C, coding `select' is fairly complex. In Python, it's a piece of
cake, but it's close enough to the C version that if you understand
`select' in Python, you'll have little trouble with it in C.

    ready_to_read, ready_to_write, in_error = \
                   select.select(
                      potential_readers,
                      potential_writers,
                      potential_errs,
                      timeout)

You pass `select' three lists: the first contains all sockets that you
might want to try reading; the second all the sockets you might want to
try writing to, and the last (normally left empty) those that you want
to check for errors.  You should note that a socket can go into more
than one list. The `select' call is blocking, but you can give it a
timeout. This is generally a sensible thing to do - give it a nice long
timeout (say a minute) unless you have good reason to do otherwise.

  In return, you will get three lists. They have the sockets that are
actually readable, writable and in error. Each of these lists is a
subset (possibly empty) of the corresponding list you passed in. And if
you put a socket in more than one input list, it will only be (at most)
in one output list.

  If a socket is in the output readable list, you can be
as-close-to-certain-as-we-ever-get-in-this-business that a `recv' on
that socket will return _something_. Same idea for the writable list.
You'll be able to send _something_. Maybe not all you want to, but
_something_ is better than nothing.  (Actually, any reasonably healthy
socket will return as writable - it just means outbound network buffer
space is available.)

  If you have a "server" socket, put it in the potential_readers list.
If it comes out in the readable list, your `accept' will (almost
certainly) work. If you have created a new socket to `connect' to
someone else, put it in the potential_writers list. If it shows up in
the writable list, you have a decent chance that it has connected.

  One very nasty problem with `select': if somewhere in those input
lists of sockets is one which has died a nasty death, the `select' will
fail. You then need to loop through every single damn socket in all
those lists and do a `select([sock],[],[],0)' until you find the bad
one. That timeout of 0 means it won't take long, but it's ugly.

  Actually, `select' can be handy even with blocking sockets. It's one
way of determining whether you will block - the socket returns as
readable when there's something in the buffers.  However, this still
doesn't help with the problem of determining whether the other end is
done, or just busy with something else.

  *Portability alert*: On Unix, `select' works both with the sockets and
files. Don't try this on Windows. On Windows, `select' works with
sockets only. Also note that in C, many of the more advanced socket
options are done differently on Windows. In fact, on Windows I usually
use threads (which work very, very well) with my sockets. Face it, if
you want any kind of performance, your code will look very different on
Windows than on Unix.

11.8.5.1 Performance
....................

There's no question that the fastest sockets code uses non-blocking
sockets and select to multiplex them. You can put together something
that will saturate a LAN connection without putting any strain on the
CPU. The trouble is that an app written this way can't do much of
anything else - it needs to be ready to shuffle bytes around at all
times.

  Assuming that your app is actually supposed to do something more than
that, threading is the optimal solution, (and using non-blocking
sockets will be faster than using blocking sockets). Unfortunately,
threading support in Unixes varies both in API and quality. So the
normal Unix solution is to fork a subprocess to deal with each
connection. The overhead for this is significant (and don't do this on
Windows - the overhead of process creation is enormous there). It also
means that unless each subprocess is completely independent, you'll
need to use another form of IPC, say a pipe, or shared memory and
semaphores, to communicate between the parent and child processes.

  Finally, remember that even though blocking sockets are somewhat
slower than non-blocking, in many cases they are the "right" solution.
After all, if your app is driven by the data it receives over a socket,
there's not much sense in complicating the logic just so your app can
wait on `select' instead of `recv'.

11.9 Sorting HOW TO
===================

     Author: Andrew Dalke and Raymond Hettinger

     Release: 0.1

  Python lists have a built-in `list.sort()' method that modifies the
list in-place and a sorted() built-in function that builds
a new sorted list from an iterable.

  In this document, we explore the various techniques for sorting data
using Python.

11.9.1 Sorting Basics
---------------------

A simple ascending sort is very easy: just call the sorted()
function. It returns a new sorted list:

    >>> sorted([5, 2, 3, 1, 4])
    [1, 2, 3, 4, 5]

You can also use the `list.sort()' method of a list. It modifies the
list in-place (and returns _None_ to avoid confusion). Usually it's
less convenient than sorted(). - but if you don't need the
original list, it's slightly more efficient.

    >>> a = [5, 2, 3, 1, 4]
    >>> a.sort()
    >>> a
    [1, 2, 3, 4, 5]

Another difference is that the `list.sort()' method is only defined for
lists. In contrast, the sorted() function accepts any
iterable.

    >>> sorted({1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'})
    [1, 2, 3, 4, 5]

11.9.2 Key Functions
--------------------

Starting with Python 2.4, both `list.sort()' and sorted().
added a _key_ parameter to specify a function to be called on each list
element prior to making comparisons.

  For example, here's a case-insensitive string comparison:

    >>> sorted("This is a test string from Andrew".split(), key=str.lower)
    ['a', 'Andrew', 'from', 'is', 'string', 'test', 'This']

The value of the _key_ parameter should be a function that takes a
single argument and returns a key to use for sorting purposes. This
technique is fast because the key function is called exactly once for
each input record.

  A common pattern is to sort complex objects using some of the
object's indices as keys. For example:

    >>> student_tuples = [
        ('john', 'A', 15),
        ('jane', 'B', 12),
        ('dave', 'B', 10),
    ]
    >>> sorted(student_tuples, key=lambda student: student[2])   # sort by age
    [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

The same technique works for objects with named attributes. For example:

    >>> class Student:
            def __init__(self, name, grade, age):
                self.name = name
                self.grade = grade
                self.age = age
            def __repr__(self):
                return repr((self.name, self.grade, self.age))

    >>> student_objects = [
        Student('john', 'A', 15),
        Student('jane', 'B', 12),
        Student('dave', 'B', 10),
    ]
    >>> sorted(student_objects, key=lambda student: student.age)   # sort by age
    [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

11.9.3 Operator Module Functions
--------------------------------

The key-function patterns shown above are very common, so Python
provides convenience functions to make accessor functions easier and
faster. The operator module has operator.itemgetter(),
operator.attrgetter(), and starting in Python 2.5 a
operator.methodcaller() function.

  Using those functions, the above examples become simpler and faster:

    >>> from operator import itemgetter, attrgetter

    >>> sorted(student_tuples, key=itemgetter(2))
    [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

    >>> sorted(student_objects, key=attrgetter('age'))
    [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

The operator module functions allow multiple levels of sorting. For
example, to sort by _grade_ then by _age_:

    >>> sorted(student_tuples, key=itemgetter(1,2))
    [('john', 'A', 15), ('dave', 'B', 10), ('jane', 'B', 12)]

    >>> sorted(student_objects, key=attrgetter('grade', 'age'))
    [('john', 'A', 15), ('dave', 'B', 10), ('jane', 'B', 12)]

11.9.4 Ascending and Descending
-------------------------------

Both `list.sort()' and sorted() accept a _reverse_
parameter with a boolean value. This is using to flag descending sorts.
For example, to get the student data in reverse _age_ order:

    >>> sorted(student_tuples, key=itemgetter(2), reverse=True)
    [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]

    >>> sorted(student_objects, key=attrgetter('age'), reverse=True)
    [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]

11.9.5 Sort Stability and Complex Sorts
---------------------------------------

Starting with Python 2.2, sorts are guaranteed to be stable
(http://en.wikipedia.org/wiki/Sorting_algorithm#Stability). That means
that when multiple records have the same key, their original order is
preserved.

    >>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)]
    >>> sorted(data, key=itemgetter(0))
    [('blue', 1), ('blue', 2), ('red', 1), ('red', 2)]

Notice how the two records for _blue_ retain their original order so
that `('blue', 1)' is guaranteed to precede `('blue', 2)'.

  This wonderful property lets you build complex sorts in a series of
sorting steps. For example, to sort the student data by descending
_grade_ and then ascending _age_, do the _age_ sort first and then sort
again using _grade_:

    >>> s = sorted(student_objects, key=attrgetter('age'))     # sort on secondary key
    >>> sorted(s, key=attrgetter('grade'), reverse=True)       # now sort on primary key, descending
    [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

The Timsort (http://en.wikipedia.org/wiki/Timsort)algorithm used in
Python does multiple sorts efficiently because it can take advantage of
any ordering already present in a dataset.

11.9.6 The Old Way Using Decorate-Sort-Undecorate
-------------------------------------------------

This idiom is called Decorate-Sort-Undecorate after its three steps:

   * First, the initial list is decorated with new values that control
     the sort order.

   * Second, the decorated list is sorted.

   * Finally, the decorations are removed, creating a list that
     contains only the initial values in the new order.

  For example, to sort the student data by _grade_ using the DSU
approach:

    >>> decorated = [(student.grade, i, student) for i, student in enumerate(student_objects)]
    >>> decorated.sort()
    >>> [student for grade, i, student in decorated]               # undecorate
    [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]

This idiom works because tuples are compared lexicographically; the
first items are compared; if they are the same then the second items
are compared, and so on.

  It is not strictly necessary in all cases to include the index _i_ in
the decorated list, but including it gives two benefits:

   * The sort is stable - if two items have the same key, their order
     will be preserved in the sorted list.

   * The original items do not have to be comparable because the
     ordering of the decorated tuples will be determined by at most the
     first two items. So for example the original list could contain
     complex numbers which cannot be sorted directly.

  Another name for this idiom is Schwartzian transform
(http://en.wikipedia.org/wiki/Schwartzian_transform), after Randal L.
Schwartz, who popularized it among Perl programmers.

  For large lists and lists where the comparison information is
expensive to calculate, and Python versions before 2.4, DSU is likely
to be the fastest way to sort the list. For 2.4 and later, key
functions provide the same functionality.

11.9.7 The Old Way Using the _cmp_ Parameter
--------------------------------------------

Many constructs given in this HOWTO assume Python 2.4 or later. Before
that, there was no sorted() builtin and `list.sort()' took
no keyword arguments. Instead, all of the Py2.x versions supported a
_cmp_ parameter to handle user specified comparison functions.

  In Py3.0, the _cmp_ parameter was removed entirely (as part of a
larger effort to simplify and unify the language, eliminating the
conflict between rich comparisons and the __cmp__() magic
method).

  In Py2.x, sort allowed an optional function which can be called for
doing the comparisons. That function should take two arguments to be
compared and then return a negative value for less-than, return zero if
they are equal, or return a positive value for greater-than. For
example, we can do:

    >>> def numeric_compare(x, y):
            return x - y
    >>> sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
    [1, 2, 3, 4, 5]

Or you can reverse the order of comparison with:

    >>> def reverse_numeric(x, y):
            return y - x
    >>> sorted([5, 2, 4, 1, 3], cmp=reverse_numeric)
    [5, 4, 3, 2, 1]

When porting code from Python 2.x to 3.x, the situation can arise when
you have the user supplying a comparison function and you need to
convert that to a key function. The following wrapper makes that easy
to do:

    def cmp_to_key(mycmp):
        'Convert a cmp= function into a key= function'
        class K(object):
            def __init__(self, obj, *args):
                self.obj = obj
            def __lt__(self, other):
                return mycmp(self.obj, other.obj) < 0
            def __gt__(self, other):
                return mycmp(self.obj, other.obj) > 0
            def __eq__(self, other):
                return mycmp(self.obj, other.obj) == 0
            def __le__(self, other):
                return mycmp(self.obj, other.obj) <= 0
            def __ge__(self, other):
                return mycmp(self.obj, other.obj) >= 0
            def __ne__(self, other):
                return mycmp(self.obj, other.obj) != 0
        return K

To convert to a key function, just wrap the old comparison function:

    >>> sorted([5, 2, 4, 1, 3], key=cmp_to_key(reverse_numeric))
    [5, 4, 3, 2, 1]

In Python 2.7, the functools.cmp_to_key() function was
added to the functools module.

11.9.8 Odd and Ends
-------------------

   * For locale aware sorting, use locale.strxfrm() for a
     key function or locale.strcoll() for a comparison
     function.

   * The _reverse_ parameter still maintains sort stability (i.e.
     records with equal keys retain the original order). Interestingly,
     that effect can be simulated without the parameter by using the
     builtin reversed() function twice:

         >>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)]
         >>> assert sorted(data, reverse=True) == list(reversed(sorted(reversed(data))))

   * The sort routines are guaranteed to use __lt__() when
     making comparisons between two objects. So, it is easy to add a
     standard sort order to a class by defining an __lt__().
     method:

         >>> Student.__lt__ = lambda self, other: self.age < other.age
         >>> sorted(student_objects)
         [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

   * Key functions need not depend directly on the objects being
     sorted. A key function can also access external resources. For
     instance, if the student grades are stored in a dictionary, they
     can be used to sort a separate list of student names:

         >>> students = ['dave', 'john', 'jane']
         >>> newgrades = {'john': 'F', 'jane':'A', 'dave': 'C'}
         >>> sorted(students, key=newgrades.__getitem__)
         ['jane', 'dave', 'john']

11.10 Unicode HOWTO
===================

     Release: 1.03

  This HOWTO discusses Python 2.x's support for Unicode, and explains
various problems that people commonly encounter when trying to work
with Unicode.  (This HOWTO has not yet been updated to cover the 3.x
versions of Python.)

11.10.1 Introduction to Unicode
-------------------------------

11.10.1.1 History of Character Codes
....................................

In 1968, the American Standard Code for Information Interchange, better
known by its acronym ASCII, was standardized.  ASCII defined numeric
codes for various characters, with the numeric values running from 0 to
127.  For example, the lowercase letter 'a' is assigned 97 as its code
value.

  ASCII was an American-developed standard, so it only defined
unaccented characters.  There was an 'e', but no 'é' or 'Í'.  This
meant that languages which required accented characters couldn't be
faithfully represented in ASCII.  (Actually the missing accents matter
for English, too, which contains words such as 'naïve' and 'café', and
some publications have house styles which require spellings such as
'coöperate'.)

  For a while people just wrote programs that didn't display accents.
I remember looking at Apple ][ BASIC programs, published in
French-language publications in the mid-1980s, that had lines like
these:

    PRINT "FICHIER EST COMPLETE."
    PRINT "CARACTERE NON ACCEPTE."

Those messages should contain accents, and they just look wrong to
someone who can read French.

  In the 1980s, almost all personal computers were 8-bit, meaning that
bytes could hold values ranging from 0 to 255.  ASCII codes only went
up to 127, so some machines assigned values between 128 and 255 to
accented characters.  Different machines had different codes, however,
which led to problems exchanging files.  Eventually various commonly
used sets of values for the 128-255 range emerged.  Some were true
standards, defined by the International Standards Organization, and
some were *de facto* conventions that were invented by one company or
another and managed to catch on.

  255 characters aren't very many.  For example, you can't fit both the
accented characters used in Western Europe and the Cyrillic alphabet
used for Russian into the 128-255 range because there are more than 127
such characters.

  You could write files using different codes (all your Russian files
in a coding system called KOI8, all your French files in a different
coding system called Latin1), but what if you wanted to write a French
document that quotes some Russian text?  In the 1980s people began to
want to solve this problem, and the Unicode standardization effort
began.

  Unicode started out using 16-bit characters instead of 8-bit
characters.  16 bits means you have 2^16 = 65,536 distinct values
available, making it possible to represent many different characters
from many different alphabets; an initial goal was to have Unicode
contain the alphabets for every single human language.  It turns out
that even 16 bits isn't enough to meet that goal, and the modern
Unicode specification uses a wider range of codes, 0-1,114,111
(0x10ffff in base-16).

  There's a related ISO standard, ISO 10646.  Unicode and ISO 10646 were
originally separate efforts, but the specifications were merged with
the 1.1 revision of Unicode.

  (This discussion of Unicode's history is highly simplified.  I don't
think the average Python programmer needs to worry about the historical
details; consult the Unicode consortium site listed in the References
for more information.)

11.10.1.2 Definitions
.....................

A *character* is the smallest possible component of a text.  'A', 'B',
'C', etc., are all different characters.  So are 'È' and 'Í'.
Characters are abstractions, and vary depending on the language or
context you're talking about.  For example, the symbol for ohms (Ω) is
usually drawn much like the capital letter omega (Ω) in the Greek
alphabet (they may even be the same in some fonts), but these are two
different characters that have different meanings.

  The Unicode standard describes how characters are represented by *code
points*.  A code point is an integer value, usually denoted in base 16.
In the standard, a code point is written using the notation U+12ca to
mean the character with value 0x12ca (4810 decimal).  The Unicode
standard contains a lot of tables listing characters and their
corresponding code points:

    0061    'a'; LATIN SMALL LETTER A
    0062    'b'; LATIN SMALL LETTER B
    0063    'c'; LATIN SMALL LETTER C
    ...
    007B    '{'; LEFT CURLY BRACKET

Strictly, these definitions imply that it's meaningless to say 'this is
character U+12ca'.  U+12ca is a code point, which represents some
particular character; in this case, it represents the character
'ETHIOPIC SYLLABLE WI'.  In informal contexts, this distinction between
code points and characters will sometimes be forgotten.

  A character is represented on a screen or on paper by a set of
graphical elements that's called a *glyph*.  The glyph for an uppercase
A, for example, is two diagonal strokes and a horizontal stroke, though
the exact details will depend on the font being used.  Most Python code
doesn't need to worry about glyphs; figuring out the correct glyph to
display is generally the job of a GUI toolkit or a terminal's font
renderer.

11.10.1.3 Encodings
...................

To summarize the previous section: a Unicode string is a sequence of
code points, which are numbers from 0 to 0x10ffff.  This sequence needs
to be represented as a set of bytes (meaning, values from 0-255) in
memory.  The rules for translating a Unicode string into a sequence of
bytes are called an *encoding*.

  The first encoding you might think of is an array of 32-bit integers.
In this representation, the string "Python" would look like this:

       P           y           t           h           o           n
    0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00
       0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

This representation is straightforward but using it presents a number of
problems.

  1. It's not portable; different processors order the bytes
     differently.

  2. It's very wasteful of space.  In most texts, the majority of the
     code points are less than 127, or less than 255, so a lot of space
     is occupied by zero bytes.  The above string takes 24 bytes
     compared to the 6 bytes needed for an ASCII representation.
     Increased RAM usage doesn't matter too much (desktop computers
     have megabytes of RAM, and strings aren't usually that large), but
     expanding our usage of disk and network bandwidth by a factor of 4
     is intolerable.

  3. It's not compatible with existing C functions such as `strlen()',
     so a new family of wide string functions would need to be used.

  4. Many Internet standards are defined in terms of textual data, and
     can't handle content with embedded zero bytes.

  Generally people don't use this encoding, instead choosing other
encodings that are more efficient and convenient.  UTF-8 is probably
the most commonly supported encoding; it will be discussed below.

  Encodings don't have to handle every possible Unicode character, and
most encodings don't.  For example, Python's default encoding is the
'ascii' encoding.  The rules for converting a Unicode string into the
ASCII encoding are simple; for each code point:

  1. If the code point is < 128, each byte is the same as the value of
     the code point.

  2. If the code point is 128 or greater, the Unicode string can't be
     represented in this encoding.  (Python raises a
     UnicodeEncodeError exception in this case.)

  Latin-1, also known as ISO-8859-1, is a similar encoding.  Unicode
code points 0-255 are identical to the Latin-1 values, so converting to
this encoding simply requires converting code points to byte values; if
a code point larger than 255 is encountered, the string can't be
encoded into Latin-1.

  Encodings don't have to be simple one-to-one mappings like Latin-1.
Consider IBM's EBCDIC, which was used on IBM mainframes.  Letter values
weren't in one block: 'a' through 'i' had values from 129 to 137, but
'j' through 'r' were 145 through 153.  If you wanted to use EBCDIC as
an encoding, you'd probably use some sort of lookup table to perform
the conversion, but this is largely an internal detail.

  UTF-8 is one of the most commonly used encodings.  UTF stands for
"Unicode Transformation Format", and the '8' means that 8-bit numbers
are used in the encoding.  (There's also a UTF-16 encoding, but it's
less frequently used than UTF-8.)  UTF-8 uses the following rules:

  1. If the code point is <128, it's represented by the corresponding
     byte value.

  2. If the code point is between 128 and 0x7ff, it's turned into two
     byte values between 128 and 255.

  3. Code points >0x7ff are turned into three- or four-byte sequences,
     where each byte of the sequence is between 128 and 255.

  UTF-8 has several convenient properties:

  1. It can handle any Unicode code point.

  2. A Unicode string is turned into a string of bytes containing no
     embedded zero bytes.  This avoids byte-ordering issues, and means
     UTF-8 strings can be processed by C functions such as `strcpy()'
     and sent through protocols that can't handle zero bytes.

  3. A string of ASCII text is also valid UTF-8 text.

  4. UTF-8 is fairly compact; the majority of code points are turned
     into two bytes, and values less than 128 occupy only a single byte.

  5. If bytes are corrupted or lost, it's possible to determine the
     start of the next UTF-8-encoded code point and resynchronize.
     It's also unlikely that random 8-bit data will look like valid
     UTF-8.

11.10.1.4 References
....................

The Unicode Consortium site at <<http://www.unicode.org>> has character
charts, a glossary, and PDF versions of the Unicode specification.  Be
prepared for some difficult reading.
<<http://www.unicode.org/history/>> is a chronology of the origin and
development of Unicode.

  To help understand the standard, Jukka Korpela has written an
introductory guide to reading the Unicode character tables, available at
<<http://www.cs.tut.fi/~jkorpela/unicode/guide.html>>.

  Another good introductory article was written by Joel Spolsky
<<http://www.joelonsoftware.com/articles/Unicode.html>>.  If this
introduction didn't make things clear to you, you should try reading
this alternate article before continuing.

  Wikipedia entries are often helpful; see the entries for "character
encoding" <<http://en.wikipedia.org/wiki/Character_encoding>> and UTF-8
<<http://en.wikipedia.org/wiki/UTF-8>>, for example.

11.10.2 Python 2.x's Unicode Support
------------------------------------

Now that you've learned the rudiments of Unicode, we can look at
Python's Unicode features.

11.10.2.1 The Unicode Type
..........................

Unicode strings are expressed as instances of the unicode.
type, one of Python's repertoire of built-in types.  It derives from an
abstract type called basestring, which is also an ancestor
of the str type; you can therefore check if a value is a
string type with `isinstance(value, basestring)'.  Under the hood,
Python represents Unicode strings as either 16- or 32-bit integers,
depending on how the Python interpreter was compiled.

  The unicode() constructor has the signature
`unicode(string[, encoding, errors])'.  All of its arguments should be
8-bit strings.  The first argument is converted to Unicode using the
specified encoding; if you leave off the `encoding' argument, the ASCII
encoding is used for the conversion, so characters greater than 127
will be treated as errors:

    >>> unicode('abcdef')
    u'abcdef'
    >>> s = unicode('abcdef')
    >>> type(s)
    <type 'unicode'>
    >>> unicode('abcdef' + chr(255))
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:
                        ordinal not in range(128)

The `errors' argument specifies the response when the input string
can't be converted according to the encoding's rules.  Legal values for
this argument are 'strict' (raise a `UnicodeDecodeError' exception),
'replace' (add U+FFFD, 'REPLACEMENT CHARACTER'), or 'ignore' (just
leave the character out of the Unicode result).  The following examples
show the differences:

    >>> unicode('\x80abc', errors='strict')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
                        ordinal not in range(128)
    >>> unicode('\x80abc', errors='replace')
    u'\ufffdabc'
    >>> unicode('\x80abc', errors='ignore')
    u'abc'

Encodings are specified as strings containing the encoding's name.
Python 2.7 comes with roughly 100 different encodings; see the Python
Library Reference at Standard Encodings for a list.  Some
encodings have multiple names; for example, 'latin-1', 'iso_8859_1' and
'8859' are all synonyms for the same encoding.

  One-character Unicode strings can also be created with the
unichr().  built-in function, which takes integers and returns a
Unicode string of length 1 that contains the corresponding code point.
The reverse operation is the built-in ord() function that
takes a one-character Unicode string and returns the code point value:

    >>> unichr(40960)
    u'\ua000'
    >>> ord(u'\ua000')
    40960

Instances of the unicode type have many of the same methods
as the 8-bit string type for operations such as searching and
formatting:

    >>> s = u'Was ever feather so lightly blown to and fro as this multitude?'
    >>> s.count('e')
    5
    >>> s.find('feather')
    9
    >>> s.find('bird')
    -1
    >>> s.replace('feather', 'sand')
    u'Was ever sand so lightly blown to and fro as this multitude?'
    >>> s.upper()
    u'WAS EVER FEATHER SO LIGHTLY BLOWN TO AND FRO AS THIS MULTITUDE?'

Note that the arguments to these methods can be Unicode strings or 8-bit
strings.  8-bit strings will be converted to Unicode before carrying
out the operation; Python's default ASCII encoding will be used, so
characters greater than 127 will cause an exception:

    >>> s.find('Was\x9f')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 3: ordinal not in range(128)
    >>> s.find(u'Was\x9f')
    -1

Much Python code that operates on strings will therefore work with
Unicode strings without requiring any changes to the code.  (Input and
output code needs more updating for Unicode; more on this later.)

  Another important method is `.encode([encoding], [errors='strict'])',
which returns an 8-bit string version of the Unicode string, encoded in
the requested encoding.  The `errors' parameter is the same as the
parameter of the `unicode()' constructor, with one additional
possibility; as well as 'strict', 'ignore', and 'replace', you can also
pass 'xmlcharrefreplace' which uses XML's character references.  The
following example shows the different results:

    >>> u = unichr(40960) + u'abcd' + unichr(1972)
    >>> u.encode('utf-8')
    '\xea\x80\x80abcd\xde\xb4'
    >>> u.encode('ascii')
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in position 0: ordinal not in range(128)
    >>> u.encode('ascii', 'ignore')
    'abcd'
    >>> u.encode('ascii', 'replace')
    '?abcd?'
    >>> u.encode('ascii', 'xmlcharrefreplace')
    '&#40960;abcd&#1972;'

Python's 8-bit strings have a `.decode([encoding], [errors])' method
that interprets the string using the given encoding:

    >>> u = unichr(40960) + u'abcd' + unichr(1972)   # Assemble a string
    >>> utf8_version = u.encode('utf-8')             # Encode as UTF-8
    >>> type(utf8_version), utf8_version
    (<type 'str'>, '\xea\x80\x80abcd\xde\xb4')
    >>> u2 = utf8_version.decode('utf-8')            # Decode using UTF-8
    >>> u == u2                                      # The two strings match
    True

The low-level routines for registering and accessing the available
encodings are found in the codecs module.  However, the
encoding and decoding functions returned by this module are usually
more low-level than is comfortable, so I'm not going to describe the
codecs module here.  If you need to implement a completely
new encoding, you'll need to learn about the codecs module
interfaces, but implementing encodings is a specialized task that also
won't be covered here.  Consult the Python documentation to learn more
about this module.

  The most commonly used part of the codecs module is the
codecs.open() function which will be discussed in the
section on input and output.

11.10.2.2 Unicode Literals in Python Source Code
................................................

In Python source code, Unicode literals are written as strings prefixed
with the 'u' or 'U' character: `u'abcdefghijk''.  Specific code points
can be written using the `\u' escape sequence, which is followed by
four hex digits giving the code point.  The `\U' escape sequence is
similar, but expects 8 hex digits, not 4.

  Unicode literals can also use the same escape sequences as 8-bit
strings, including `\x', but `\x' only takes two hex digits so it can't
express an arbitrary code point.  Octal escapes can go up to U+01ff,
which is octal 777.

    >>> s = u"a\xac\u1234\u20ac\U00008000"
               ^^^^ two-digit hex escape
                   ^^^^^^ four-digit Unicode escape
                               ^^^^^^^^^^ eight-digit Unicode escape
    >>> for c in s:  print ord(c),
    ...
    97 172 4660 8364 32768

Using escape sequences for code points greater than 127 is fine in
small doses, but becomes an annoyance if you're using many accented
characters, as you would in a program with messages in French or some
other accent-using language.  You can also assemble strings using the
unichr() built-in function, but this is even more tedious.

  Ideally, you'd want to be able to write literals in your language's
natural encoding.  You could then edit Python source code with your
favorite editor which would display the accented characters naturally,
and have the right characters used at runtime.

  Python supports writing Unicode literals in any encoding, but you
have to declare the encoding being used.  This is done by including a
special comment as either the first or second line of the source file:

    #!/usr/bin/env python
    # -*- coding: latin-1 -*-

    u = u'abcdé'
    print ord(u[-1])

The syntax is inspired by Emacs's notation for specifying variables
local to a file.  Emacs supports many different variables, but Python
only supports 'coding'.  The `-*-' symbols indicate to Emacs that the
comment is special; they have no significance to Python but are a
convention.  Python looks for `coding: name' or `coding=name' in the
comment.

  If you don't include such a comment, the default encoding used will
be ASCII.  Versions of Python before 2.4 were Euro-centric and assumed
Latin-1 as a default encoding for string literals; in Python 2.4,
characters greater than 127 still work but result in a warning.  For
example, the following program has no encoding declaration:

    #!/usr/bin/env python
    u = u'abcdé'
    print ord(u[-1])

When you run it with Python 2.4, it will output the following warning:

    amk:~$ python2.4 p263.py
    sys:1: DeprecationWarning: Non-ASCII character '\xe9'
         in file p263.py on line 2, but no encoding declared;
         see http://www.python.org/peps/pep-0263.html for details

Python 2.5 and higher are stricter and will produce a syntax error:

    amk:~$ python2.5 p263.py
    File "/tmp/p263.py", line 2
    SyntaxError: Non-ASCII character '\xc3' in file /tmp/p263.py
      on line 2, but no encoding declared; see
      http://www.python.org/peps/pep-0263.html for details

11.10.2.3 Unicode Properties
............................

The Unicode specification includes a database of information about code
points.  For each code point that's defined, the information includes
the character's name, its category, the numeric value if applicable
(Unicode has characters representing the Roman numerals and fractions
such as one-third and four-fifths).  There are also properties related
to the code point's use in bidirectional text and other display-related
properties.

  The following program displays some information about several
characters, and prints the numeric value of one particular character:

    import unicodedata

    u = unichr(233) + unichr(0x0bf2) + unichr(3972) + unichr(6000) + unichr(13231)

    for i, c in enumerate(u):
        print i, '%04x' % ord(c), unicodedata.category(c),
        print unicodedata.name(c)

    # Get numeric value of second character
    print unicodedata.numeric(u[1])

When run, this prints:

    0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
    1 0bf2 No TAMIL NUMBER ONE THOUSAND
    2 0f84 Mn TIBETAN MARK HALANTA
    3 1770 Lo TAGBANWA LETTER SA
    4 33af So SQUARE RAD OVER S SQUARED
    1000.0

The category codes are abbreviations describing the nature of the
character.  These are grouped into categories such as "Letter",
"Number", "Punctuation", or "Symbol", which in turn are broken up into
subcategories.  To take the codes from the above output, `'Ll'' means
'Letter, lowercase', `'No'' means "Number, other", `'Mn'' is "Mark,
nonspacing", and `'So'' is "Symbol, other".  See
<<http://www.unicode.org/reports/tr44/#General_Category_Values>> for a
list of category codes.

11.10.2.4 References
....................

The Unicode and 8-bit string types are described in the Python library
reference at Sequence Types -- str, unicode, list, tuple, bytearray, buffer, xrange.

  The documentation for the unicodedata module.

  The documentation for the codecs module.

  Marc-André Lemburg gave a presentation at EuroPython 2002 titled
"Python and Unicode".  A PDF version of his slides is available at
<<http://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf>>, and is
an excellent overview of the design of Python's Unicode features.

11.10.3 Reading and Writing Unicode Data
----------------------------------------

Once you've written some code that works with Unicode data, the next
problem is input/output.  How do you get Unicode strings into your
program, and how do you convert Unicode into a form suitable for
storage or transmission?

  It's possible that you may not need to do anything depending on your
input sources and output destinations; you should check whether the
libraries used in your application support Unicode natively.  XML
parsers often return Unicode data, for example.  Many relational
databases also support Unicode-valued columns and can return Unicode
values from an SQL query.

  Unicode data is usually converted to a particular encoding before it
gets written to disk or sent over a socket.  It's possible to do all
the work yourself: open a file, read an 8-bit string from it, and
convert the string with `unicode(str, encoding)'.  However, the manual
approach is not recommended.

  One problem is the multi-byte nature of encodings; one Unicode
character can be represented by several bytes.  If you want to read the
file in arbitrary-sized chunks (say, 1K or 4K), you need to write
error-handling code to catch the case where only part of the bytes
encoding a single Unicode character are read at the end of a chunk.
One solution would be to read the entire file into memory and then
perform the decoding, but that prevents you from working with files that
are extremely large; if you need to read a 2Gb file, you need 2Gb of
RAM.  (More, really, since for at least a moment you'd need to have
both the encoded string and its Unicode version in memory.)

  The solution would be to use the low-level decoding interface to
catch the case of partial coding sequences.  The work of implementing
this has already been done for you: the codecs module
includes a version of the open().  function that returns a
file-like object that assumes the file's contents are in a specified
encoding and accepts Unicode parameters for methods such as `.read()'
and `.write()'.

  The function's parameters are `open(filename, mode='rb',
encoding=None, errors='strict', buffering=1)'.  `mode' can be `'r'',
`'w'', or `'a'', just like the corresponding parameter to the regular
built-in `open()' function; add a `'+'' to update the file.
`buffering' is similarly parallel to the standard function's parameter.
`encoding' is a string giving the encoding to use; if it's left as
`None', a regular Python file object that accepts 8-bit strings is
returned.  Otherwise, a wrapper object is returned, and data written to
or read from the wrapper object will be converted as needed.  `errors'
specifies the action for encoding errors and can be one of the usual
values of 'strict', 'ignore', and 'replace'.

  Reading Unicode from a file is therefore simple:

    import codecs
    f = codecs.open('unicode.rst', encoding='utf-8')
    for line in f:
        print repr(line)

It's also possible to open files in update mode, allowing both reading
and writing:

    f = codecs.open('test', encoding='utf-8', mode='w+')
    f.write(u'\u4500 blah blah blah\n')
    f.seek(0)
    print repr(f.readline()[:1])
    f.close()

Unicode character U+FEFF is used as a byte-order mark (BOM), and is
often written as the first character of a file in order to assist with
autodetection of the file's byte ordering.  Some encodings, such as
UTF-16, expect a BOM to be present at the start of a file; when such an
encoding is used, the BOM will be automatically written as the first
character and will be silently dropped when the file is read.  There
are variants of these encodings, such as 'utf-16-le' and 'utf-16-be'
for little-endian and big-endian encodings, that specify one particular
byte ordering and don't skip the BOM.

11.10.3.1 Unicode filenames
...........................

Most of the operating systems in common use today support filenames
that contain arbitrary Unicode characters.  Usually this is implemented
by converting the Unicode string into some encoding that varies
depending on the system.  For example, Mac OS X uses UTF-8 while
Windows uses a configurable encoding; on Windows, Python uses the name
"mbcs" to refer to whatever the currently configured encoding is.  On
Unix systems, there will only be a filesystem encoding if you've set
the `LANG' or `LC_CTYPE' environment variables; if you haven't, the
default encoding is ASCII.

  The sys.getfilesystemencoding() function returns the
encoding to use on your current system, in case you want to do the
encoding manually, but there's not much reason to bother.  When opening
a file for reading or writing, you can usually just provide the Unicode
string as the filename, and it will be automatically converted to the
right encoding for you:

    filename = u'filename\u4500abc'
    f = open(filename, 'w')
    f.write('blah\n')
    f.close()

Functions in the os module such as os.stat().
will also accept Unicode filenames.

  See os.listdir(), which returns filenames, raises an issue:
should it return the Unicode version of filenames, or should it return
8-bit strings containing the encoded versions?  os.listdir()
will do both, depending on whether you provided the directory path
as an 8-bit string or a Unicode string.  If you pass a Unicode string
as the path, filenames will be decoded using the filesystem's encoding
and a list of Unicode strings will be returned, while passing an 8-bit
path will return the 8-bit versions of the filenames.  For example,
assuming the default filesystem encoding is UTF-8, running the
following program:

    fn = u'filename\u4500abc'
    f = open(fn, 'w')
    f.close()

    import os
    print os.listdir('.')
    print os.listdir(u'.')

will produce the following output:

    amk:~$ python t.py
    ['.svn', 'filename\xe4\x94\x80abc', ...]
    [u'.svn', u'filename\u4500abc', ...]

The first list contains UTF-8-encoded filenames, and the second list
contains the Unicode versions.

11.10.3.2 Tips for Writing Unicode-aware Programs
.................................................

This section provides some suggestions on writing software that deals
with Unicode.

  The most important tip is:

     Software should only work with Unicode strings internally,
     converting to a particular encoding on output.

  If you attempt to write processing functions that accept both Unicode
and 8-bit strings, you will find your program vulnerable to bugs
wherever you combine the two different kinds of strings.  Python's
default encoding is ASCII, so whenever a character with an ASCII value
> 127 is in the input data, you'll get a UnicodeDecodeError.
because that character can't be handled by the ASCII encoding.

  It's easy to miss such problems if you only test your software with
data that doesn't contain any accents; everything will seem to work,
but there's actually a bug in your program waiting for the first user
who attempts to use characters > 127.  A second tip, therefore, is:

     Include characters > 127 and, even better, characters > 255 in
     your test data.

  When using data coming from a web browser or some other untrusted
source, a common technique is to check for illegal characters in a
string before using the string in a generated command line or storing
it in a database.  If you're doing this, be careful to check the string
once it's in the form that will be used or stored; it's possible for
encodings to be used to disguise characters.  This is especially true
if the input data also specifies the encoding; many encodings leave the
commonly checked-for characters alone, but Python includes some
encodings such as `'base64'' that modify every single character.

  For example, let's say you have a content management system that
takes a Unicode filename, and you want to disallow paths with a '/'
character.  You might write this code:

    def read_file (filename, encoding):
        if '/' in filename:
            raise ValueError("'/' not allowed in filenames")
        unicode_name = filename.decode(encoding)
        f = open(unicode_name, 'r')
        # ... return contents of file ...

However, if an attacker could specify the `'base64'' encoding, they
could pass `'L2V0Yy9wYXNzd2Q='', which is the base-64 encoded form of
the string `'/etc/passwd'', to read a system file.  The above code
looks for `'/'' characters in the encoded form and misses the dangerous
character in the resulting decoded form.

11.10.3.3 References
....................

The PDF slides for Marc-André Lemburg's presentation "Writing
Unicode-aware Applications in Python" are available at
<<http://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>>
and discuss questions of character encodings as well as how to
internationalize and localize an application.

11.10.4 Revision History and Acknowledgements
---------------------------------------------

Thanks to the following people who have noted errors or offered
suggestions on this article: Nicholas Bastin, Marius Gedminas, Kent
Johnson, Ken Krugler, Marc-André Lemburg, Martin von Löwis, Chad
Whitacre.

  Version 1.0: posted August 5 2005.

  Version 1.01: posted August 7 2005.  Corrects factual and markup
errors; adds several links.

  Version 1.02: posted August 16 2005.  Corrects factual errors.

  Version 1.03: posted June 20 2010.  Notes that Python 3.x is not
covered, and that the HOWTO only covers 2.x.

11.11 HOWTO Fetch Internet Resources Using urllib2
==================================================

     Author: Michael Foord
     (http://www.voidspace.org.uk/python/index.shtml)

     Note: There is an French translation of an earlier revision of this
     HOWTO, available at urllib2 - Le Manuel manquant
     (http://www.voidspace.org.uk/python/articles/urllib2_francais.shtml).

11.11.1 Introduction
--------------------

Related Articles
................

You may also find useful the following article on fetching web resources
with Python :

   * Basic Authentication
     (http://www.voidspace.org.uk/python/articles/authentication.shtml)

          A tutorial on _Basic Authentication_, with examples in Python.

  *urllib2* is a Python (http://www.python.org) module for fetching URLs
(Uniform Resource Locators). It offers a very simple interface, in the
form of the _urlopen_ function. This is capable of fetching URLs using
a variety of different protocols. It also offers a slightly more
complex interface for handling common situations - like basic
authentication, cookies, proxies and so on. These are provided by
objects called handlers and openers.

  urllib2 supports fetching URLs for many "URL schemes" (identified by
the string before the ":" in URL - for example "ftp" is the URL scheme
of "<ftp://python.org/>") using their associated network protocols
(e.g. FTP, HTTP).  This tutorial focuses on the most common case, HTTP.

  For straightforward situations _urlopen_ is very easy to use. But as
soon as you encounter errors or non-trivial cases when opening HTTP
URLs, you will need some understanding of the HyperText Transfer
Protocol. The most comprehensive and authoritative reference to HTTP is RFC
2616 (http://tools.ietf.org/html/rfc2616.html). This is a technical
document and not intended to be easy to read. This HOWTO aims to
illustrate using _urllib2_, with enough detail about HTTP to help you
through. It is not intended to replace the urllib2 docs,
but is supplementary to them.

11.11.2 Fetching URLs
---------------------

The simplest way to use urllib2 is as follows:

    import urllib2
    response = urllib2.urlopen('http://python.org/')
    html = response.read()

Many uses of urllib2 will be that simple (note that instead of an
'http:' URL we could have used an URL starting with 'ftp:', 'file:',
etc.).  However, it's the purpose of this tutorial to explain the more
complicated cases, concentrating on HTTP.

  HTTP is based on requests and responses - the client makes requests
and servers send responses. urllib2 mirrors this with a `Request'
object which represents the HTTP request you are making. In its
simplest form you create a Request object that specifies the URL you
want to fetch. Calling `urlopen' with this Request object returns a
response object for the URL requested. This response is a file-like
object, which means you can for example call `.read()' on the response:

    import urllib2

    req = urllib2.Request('http://www.voidspace.org.uk')
    response = urllib2.urlopen(req)
    the_page = response.read()

Note that urllib2 makes use of the same Request interface to handle all
URL schemes.  For example, you can make an FTP request like so:

    req = urllib2.Request('ftp://example.com/')

In the case of HTTP, there are two extra things that Request objects
allow you to do: First, you can pass data to be sent to the server.
Second, you can pass extra information ("metadata") _about_ the data or
the about request itself, to the server - this information is sent as
HTTP "headers".  Let's look at each of these in turn.

11.11.2.1 Data
..............

Sometimes you want to send data to a URL (often the URL will refer to a
CGI (Common Gateway Interface) script (1) or other web application).
With HTTP, this is often done using what's known as a *POST* request.
This is often what your browser does when you submit a HTML form that
you filled in on the web. Not all POSTs have to come from forms: you
can use a POST to transmit arbitrary data to your own application. In
the common case of HTML forms, the data needs to be encoded in a
standard way, and then passed to the Request object as the `data'
argument. The encoding is done using a function from the `urllib'
library _not_ from `urllib2'.

    import urllib
    import urllib2

    url = 'http://www.someserver.com/cgi-bin/register.cgi'
    values = {'name' : 'Michael Foord',
              'location' : 'Northampton',
              'language' : 'Python' }

    data = urllib.urlencode(values)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    the_page = response.read()

Note that other encodings are sometimes required (e.g. for file upload
from HTML forms - see HTML Specification, Form Submission
(http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13)for more
details).

  If you do not pass the `data' argument, urllib2 uses a *GET* request.
One way in which GET and POST requests differ is that POST requests
often have "side-effects": they change the state of the system in some
way (for example by placing an order with the website for a
hundredweight of tinned spam to be delivered to your door).  Though the
HTTP standard makes it clear that POSTs are intended to _always_ cause
side-effects, and GET requests _never_ to cause side-effects, nothing
prevents a GET request from having side-effects, nor a POST requests
from having no side-effects. Data can also be passed in an HTTP GET
request by encoding it in the URL itself.

  This is done as follows:

    >>> import urllib2
    >>> import urllib
    >>> data = {}
    >>> data['name'] = 'Somebody Here'
    >>> data['location'] = 'Northampton'
    >>> data['language'] = 'Python'
    >>> url_values = urllib.urlencode(data)
    >>> print url_values
    name=Somebody+Here&language=Python&location=Northampton
    >>> url = 'http://www.example.com/example.cgi'
    >>> full_url = url + '?' + url_values
    >>> data = urllib2.open(full_url)

Notice that the full URL is created by adding a `?' to the URL,
followed by the encoded values.

11.11.2.2 Headers
.................

We'll discuss here one particular HTTP header, to illustrate how to add
headers to your HTTP request.

  Some websites (2) dislike being browsed by programs, or send
different versions to different browsers (3) . By default urllib2
identifies itself as `Python-urllib/x.y' (where `x' and `y' are the
major and minor version numbers of the Python release, e.g.
`Python-urllib/2.5'), which may confuse the site, or just plain not
work. The way a browser identifies itself is through the `User-Agent'
header (4). When you create a Request object you can pass a dictionary
of headers in. The following example makes the same request as above,
but identifies itself as a version of Internet Explorer (5).

    import urllib
    import urllib2

    url = 'http://www.someserver.com/cgi-bin/register.cgi'
    user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
    values = {'name' : 'Michael Foord',
              'location' : 'Northampton',
              'language' : 'Python' }
    headers = { 'User-Agent' : user_agent }

    data = urllib.urlencode(values)
    req = urllib2.Request(url, data, headers)
    response = urllib2.urlopen(req)
    the_page = response.read()

The response also has two useful methods. See the section on info and geturl.
which comes after we have a look at what happens
when things go wrong.

11.11.3 Handling Exceptions
---------------------------

_urlopen_ raises `URLError' when it cannot handle a response (though as
usual with Python APIs, built-in exceptions such as ValueError,
see TypeError etc. may also be raised).

  `HTTPError' is the subclass of `URLError' raised in the specific case
of HTTP URLs.

11.11.3.1 URLError
..................

Often, URLError is raised because there is no network connection (no
route to the specified server), or the specified server doesn't exist.
In this case, the exception raised will have a 'reason' attribute,
which is a tuple containing an error code and a text error message.

  e.g.

    >>> req = urllib2.Request('http://www.pretend_server.org')
    >>> try: urllib2.urlopen(req)
    >>> except URLError, e:
    >>>    print e.reason
    >>>
    (4, 'getaddrinfo failed')

11.11.3.2 HTTPError
...................

Every HTTP response from the server contains a numeric "status code".
Sometimes the status code indicates that the server is unable to fulfil
the request. The default handlers will handle some of these responses
for you (for example, if the response is a "redirection" that requests
the client fetch the document from a different URL, urllib2 will handle
that for you). For those it can't handle, urlopen will raise an
`HTTPError'. Typical errors include '404' (page not found), '403'
(request forbidden), and '401' (authentication required).

  See section 10 of RFC 2616 for a reference on all the HTTP error
codes.

  The `HTTPError' instance raised will have an integer 'code'
attribute, which corresponds to the error sent by the server.

11.11.3.3 Error Codes
.....................

Because the default handlers handle redirects (codes in the 300 range),
and codes in the 100-299 range indicate success, you will usually only
see error codes in the 400-599 range.

  `BaseHTTPServer.BaseHTTPRequestHandler.responses' is a useful
dictionary of response codes in that shows all the response codes used
by RFC 2616. The dictionary is reproduced here for convenience

    # Table mapping response codes to messages; entries have the
    # form {code: (shortmessage, longmessage)}.
    responses = {
        100: ('Continue', 'Request received, please continue'),
        101: ('Switching Protocols',
              'Switching to new protocol; obey Upgrade header'),

        200: ('OK', 'Request fulfilled, document follows'),
        201: ('Created', 'Document created, URL follows'),
        202: ('Accepted',
              'Request accepted, processing continues off-line'),
        203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
        204: ('No Content', 'Request fulfilled, nothing follows'),
        205: ('Reset Content', 'Clear input form for further input.'),
        206: ('Partial Content', 'Partial content follows.'),

        300: ('Multiple Choices',
              'Object has several resources -- see URI list'),
        301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
        302: ('Found', 'Object moved temporarily -- see URI list'),
        303: ('See Other', 'Object moved -- see Method and URL list'),
        304: ('Not Modified',
              'Document has not changed since given time'),
        305: ('Use Proxy',
              'You must use proxy specified in Location to access this '
              'resource.'),
        307: ('Temporary Redirect',
              'Object moved temporarily -- see URI list'),

        400: ('Bad Request',
              'Bad request syntax or unsupported method'),
        401: ('Unauthorized',
              'No permission -- see authorization schemes'),
        402: ('Payment Required',
              'No payment -- see charging schemes'),
        403: ('Forbidden',
              'Request forbidden -- authorization will not help'),
        404: ('Not Found', 'Nothing matches the given URI'),
        405: ('Method Not Allowed',
              'Specified method is invalid for this server.'),
        406: ('Not Acceptable', 'URI not available in preferred format.'),
        407: ('Proxy Authentication Required', 'You must authenticate with '
              'this proxy before proceeding.'),
        408: ('Request Timeout', 'Request timed out; try again later.'),
        409: ('Conflict', 'Request conflict.'),
        410: ('Gone',
              'URI no longer exists and has been permanently removed.'),
        411: ('Length Required', 'Client must specify Content-Length.'),
        412: ('Precondition Failed', 'Precondition in headers is false.'),
        413: ('Request Entity Too Large', 'Entity is too large.'),
        414: ('Request-URI Too Long', 'URI is too long.'),
        415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
        416: ('Requested Range Not Satisfiable',
              'Cannot satisfy request range.'),
        417: ('Expectation Failed',
              'Expect condition could not be satisfied.'),

        500: ('Internal Server Error', 'Server got itself in trouble'),
        501: ('Not Implemented',
              'Server does not support this operation'),
        502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
        503: ('Service Unavailable',
              'The server cannot process the request due to a high load'),
        504: ('Gateway Timeout',
              'The gateway server did not receive a timely response'),
        505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
        }

When an error is raised the server responds by returning an HTTP error
code _and_ an error page. You can use the `HTTPError' instance as a
response on the page returned. This means that as well as the code
attribute, it also has read, geturl, and info, methods.

    >>> req = urllib2.Request('http://www.python.org/fish.html')
    >>> try:
    >>>     urllib2.urlopen(req)
    >>> except HTTPError, e:
    >>>     print e.code
    >>>     print e.read()
    >>>
    404
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/html4/loose.dtd">
    <?xml-stylesheet href="./css/ht2html.css"
        type="text/css"?>
    <html><head><title>Error 404: File Not Found</title>
    ...... etc...

11.11.3.4 Wrapping it Up
........................

So if you want to be prepared for `HTTPError' _or_ `URLError' there are
two basic approaches. I prefer the second approach.

11.11.3.5 Number 1
..................

    from urllib2 import Request, urlopen, URLError, HTTPError
    req = Request(someurl)
    try:
        response = urlopen(req)
    except HTTPError, e:
        print 'The server couldn\'t fulfill the request.'
        print 'Error code: ', e.code
    except URLError, e:
        print 'We failed to reach a server.'
        print 'Reason: ', e.reason
    else:
        # everything is fine

     Note: The `except HTTPError' _must_ come first, otherwise `except
     URLError' will _also_ catch an `HTTPError'.

11.11.3.6 Number 2
..................

    from urllib2 import Request, urlopen, URLError
    req = Request(someurl)
    try:
        response = urlopen(req)
    except URLError, e:
        if hasattr(e, 'reason'):
            print 'We failed to reach a server.'
            print 'Reason: ', e.reason
        elif hasattr(e, 'code'):
            print 'The server couldn\'t fulfill the request.'
            print 'Error code: ', e.code
    else:
        # everything is fine

11.11.4 info and geturl
-----------------------

The response returned by urlopen (or the `HTTPError' instance) has two
useful methods `info()' and `geturl()'.

  *geturl* - this returns the real URL of the page fetched. This is
useful because `urlopen' (or the opener object used) may have followed a
redirect. The URL of the page fetched may not be the same as the URL
requested.

  *info* - this returns a dictionary-like object that describes the page
fetched, particularly the headers sent by the server. It is currently an
`httplib.HTTPMessage' instance.

  Typical headers include 'Content-length', 'Content-type', and so on.
See the Quick Reference to HTTP Headers
(http://www.cs.tut.fi/~jkorpela/http.html) for a useful listing of HTTP
headers with brief explanations of their meaning and use.

11.11.5 Openers and Handlers
----------------------------

When you fetch a URL you use an opener (an instance of the perhaps
confusingly-named urllib2.OpenerDirector). Normally we
have been using the default opener - via `urlopen' - but you can create
custom openers. Openers use handlers. All the "heavy lifting" is done
by the handlers. Each handler knows how to open URLs for a particular
URL scheme (http, ftp, etc.), or how to handle an aspect of URL
opening, for example HTTP redirections or HTTP cookies.

  You will want to create openers if you want to fetch URLs with
specific handlers installed, for example to get an opener that handles
cookies, or to get an opener that does not handle redirections.

  To create an opener, instantiate an `OpenerDirector', and then call
`.add_handler(some_handler_instance)' repeatedly.

  Alternatively, you can use `build_opener', which is a convenience
function for creating opener objects with a single function call.
`build_opener' adds several handlers by default, but provides a quick
way to add more and/or override the default handlers.

  Other sorts of handlers you might want to can handle proxies,
authentication, and other common but slightly specialised situations.

  `install_opener' can be used to make an `opener' object the (global)
default opener. This means that calls to `urlopen' will use the opener
you have installed.

  Opener objects have an `open' method, which can be called directly to
fetch urls in the same way as the `urlopen' function: there's no need
to call `install_opener', except as a convenience.

11.11.6 Basic Authentication
----------------------------

To illustrate creating and installing a handler we will use the
`HTTPBasicAuthHandler'. For a more detailed discussion of this subject -
including an explanation of how Basic Authentication works - see the
Basic Authentication Tutorial
(http://www.voidspace.org.uk/python/articles/authentication.shtml).

  When authentication is required, the server sends a header (as well
as the 401 error code) requesting authentication.  This specifies the
authentication scheme and a 'realm'. The header looks like :
`Www-authenticate: SCHEME realm="REALM"'.

  e.g.

    Www-authenticate: Basic realm="cPanel Users"

The client should then retry the request with the appropriate name and
password for the realm included as a header in the request. This is
'basic authentication'. In order to simplify this process we can create
an instance of `HTTPBasicAuthHandler' and an opener to use this handler.

  The `HTTPBasicAuthHandler' uses an object called a password manager
to handle the mapping of URLs and realms to passwords and usernames. If
you know what the realm is (from the authentication header sent by the
server), then you can use a `HTTPPasswordMgr'. Frequently one doesn't
care what the realm is. In that case, it is convenient to use
`HTTPPasswordMgrWithDefaultRealm'. This allows you to specify a default
username and password for a URL. This will be supplied in the absence
of you providing an alternative combination for a specific realm. We
indicate this by providing `None' as the realm argument to the
`add_password' method.

  The top-level URL is the first URL that requires authentication. URLs
"deeper" than the URL you pass to .add_password() will also match.

    # create a password manager
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

    # Add the username and password.
    # If we knew the realm, we could use it instead of None.
    top_level_url = "http://example.com/foo/"
    password_mgr.add_password(None, top_level_url, username, password)

    handler = urllib2.HTTPBasicAuthHandler(password_mgr)

    # create "opener" (OpenerDirector instance)
    opener = urllib2.build_opener(handler)

    # use the opener to fetch a URL
    opener.open(a_url)

    # Install the opener.
    # Now all calls to urllib2.urlopen use our opener.
    urllib2.install_opener(opener)

     Note: In the above example we only supplied our
     `HTTPBasicAuthHandler' to `build_opener'. By default openers have
     the handlers for normal situations - `ProxyHandler',
     `UnknownHandler', `HTTPHandler', `HTTPDefaultErrorHandler',
     `HTTPRedirectHandler', `FTPHandler', `FileHandler',
     `HTTPErrorProcessor'.

  `top_level_url' is in fact _either_ a full URL (including the 'http:'
scheme component and the hostname and optionally the port number) e.g.
"<http://example.com/>" _or_ an "authority" (i.e. the hostname,
optionally including the port number) e.g. "example.com" or
"example.com:8080" (the latter example includes a port number).  The
authority, if present, must NOT contain the "userinfo" component - for
example "joe@password:example.com" is not correct.

11.11.7 Proxies
---------------

*urllib2* will auto-detect your proxy settings and use those. This is
through the `ProxyHandler' which is part of the normal handler chain.
Normally that's a good thing, but there are occasions when it may not
be helpful (6). One way to do this is to setup our own `ProxyHandler',
with no proxies defined. This is done using similar steps to setting up
a Basic Authentication
(http://www.voidspace.org.uk/python/articles/authentication.shtml)
handler :

    >>> proxy_support = urllib2.ProxyHandler({})
    >>> opener = urllib2.build_opener(proxy_support)
    >>> urllib2.install_opener(opener)

     Note: Currently `urllib2' _does not_ support fetching of `https'
     locations through a proxy.  However, this can be enabled by
     extending urllib2 as shown in the recipe (7).

11.11.8 Sockets and Layers
--------------------------

The Python support for fetching resources from the web is layered.
urllib2 uses the httplib library, which in turn uses the socket library.

  As of Python 2.3 you can specify how long a socket should wait for a
response before timing out. This can be useful in applications which
have to fetch web pages. By default the socket module has _no timeout_
and can hang. Currently, the socket timeout is not exposed at the
httplib or urllib2 levels.  However, you can set the default timeout
globally for all sockets using

    import socket
    import urllib2

    # timeout in seconds
    timeout = 10
    socket.setdefaulttimeout(timeout)

    # this call to urllib2.urlopen now uses the default timeout
    # we have set in the socket module
    req = urllib2.Request('http://www.voidspace.org.uk')
    response = urllib2.urlopen(req)

______________________________________________________________________

11.11.9 Footnotes
-----------------

This document was reviewed and revised by John Lee.

     (1) For an introduction to the CGI protocol see Writing Web
     Applications in Python
     (http://www.pyzine.com/Issue008/Section_Articles/article_CGIOne.html).

     (2) Like Google for example. The _proper_ way to use google from a
     program is to use PyGoogle (http://pygoogle.sourceforge.net)of
     course. See Voidspace Google
     (http://www.voidspace.org.uk/python/recipebook.shtml#google)for
     some examples of using the Google API.

     (3) Browser sniffing is a very bad practise for website design -
     building sites using web standards is much more sensible.
     Unfortunately a lot of sites still send different versions to
     different browsers.

     (4) The user agent for MSIE 6 is _'Mozilla/4.0 (compatible; MSIE
     6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'_

     (5) For details of more HTTP request headers, see Quick Reference
     to HTTP Headers (http://www.cs.tut.fi/~jkorpela/http.html).

     (6) In my case I have to use a proxy to access the internet at
     work. If you attempt to fetch _localhost_ URLs through this proxy
     it blocks them. IE is set to use the proxy, which urllib2 picks up
     on. In order to test scripts with a localhost server, I have to
     prevent urllib2 from using the proxy.

     (7) urllib2 opener for SSL proxy (CONNECT method): ASPN Cookbook
     Recipe
     (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195).

11.12 HOWTO Use Python in the web
=================================

     Author: Marek Kubica

Abstract
........

     This document shows how Python fits into the web.  It presents
     some ways to integrate Python with a web server, and general
     practices useful for developing web sites.

  Programming for the Web has become a hot topic since the rise of "Web
2.0", which focuses on user-generated content on web sites.  It has
always been possible to use Python for creating web sites, but it was a
rather tedious task.  Therefore, many frameworks and helper tools have
been created to assist developers in creating faster and more robust
sites.  This HOWTO describes some of the methods used to combine Python
with a web server to create dynamic content.  It is not meant as a
complete introduction, as this topic is far too broad to be covered in
one single document.  However, a short overview of the most popular
libraries is provided.

  While this HOWTO tries to give an overview of Python in the web, it
cannot always be as up to date as desired.  Web development in Python
is rapidly moving forward, so the wiki page on Web Programming
(http://wiki.python.org/moin/WebProgramming)may be more in sync with
recent development.

11.12.1 The Low-Level View
--------------------------

When a user enters a web site, their browser makes a connection to the
site's web server (this is called the _request_).  The server looks up
the file in the file system and sends it back to the user's browser,
which displays it (this is the _response_).  This is roughly how the
underlying protocol, HTTP, works.

  Dynamic web sites are not based on files in the file system, but
rather on programs which are run by the web server when a request comes
in, and which _generate_ the content that is returned to the user.
They can do all sorts of useful things, like display the postings of a
bulletin board, show your email, configure software, or just display
the current time.  These programs can be written in any programming
language the server supports.  Since most servers support Python, it is
easy to use Python to create dynamic web sites.

  Most HTTP servers are written in C or C++, so they cannot execute
Python code directly - a bridge is needed between the server and the
program.  These bridges, or rather interfaces, define how programs
interact with the server.  There have been numerous attempts to create
the best possible interface, but there are only a few worth mentioning.

  Not every web server supports every interface.  Many web servers only
support old, now-obsolete interfaces; however, they can often be
extended using third-party modules to support newer ones.

11.12.1.1 Common Gateway Interface
..................................

This interface, most commonly referred to as "CGI", is the oldest, and
is supported by nearly every web server out of the box.  Programs using
CGI to communicate with their web server need to be started by the
server for every request.  So, every request starts a new Python
interpreter - which takes some time to start up - thus making the whole
interface only usable for low load situations.

  The upside of CGI is that it is simple - writing a Python program
which uses CGI is a matter of about three lines of code.  This
simplicity comes at a price: it does very few things to help the
developer.

  Writing CGI programs, while still possible, is no longer recommended.
With WSGI, a topic covered later in this document, it is
possible to write programs that emulate CGI, so they can be run as CGI
if no better option is available.

  The Python standard library includes some modules that are helpful for
creating plain CGI programs:

   * cgi. - Handling of user input in CGI scripts

   * cgitb. - Displays nice tracebacks when errors happen in
     CGI applications, instead of presenting a "500 Internal Server
     Error" message

  The Python wiki features a page on CGI scripts
(http://wiki.python.org/moin/CgiScripts)with some additional information
about CGI in Python.

11.12.1.2 Simple script for testing CGI
.......................................

To test whether your web server works with CGI, you can use this short
and simple CGI program:

    #!/usr/bin/env python
    # -*- coding: UTF-8 -*-

    # enable debugging
    import cgitb
    cgitb.enable()

    print "Content-Type: text/plain;charset=utf-8"
    print

    print "Hello World!"

Depending on your web server configuration, you may need to save this
code with a `.py' or `.cgi' extension.  Additionally, this file may
also need to be in a `cgi-bin' folder, for security reasons.

  You might wonder what the `cgitb' line is about.  This line makes it
possible to display a nice traceback instead of just crashing and
displaying an "Internal Server Error" in the user's browser.  This is
useful for debugging, but it might risk exposing some confidential data
to the user.  You should not use `cgitb' in production code for this
reason.  You should _always_ catch exceptions, and display proper error
pages - end-users don't like to see nondescript "Internal Server
Errors" in their browsers.

11.12.1.3 Setting up CGI on your own server
...........................................

If you don't have your own web server, this does not apply to you.  You
can check whether it works as-is, and if not you will need to talk to
the administrator of your web server. If it is a big host, you can try
filing a ticket asking for Python support.

  If you are your own administrator or want to set up CGI for testing
purposes on your own computers, you have to configure it by yourself.
There is no single way to configure CGI, as there are many web servers
with different configuration options.  Currently the most widely used
free web server is Apache HTTPd (http://httpd.apache.org/), or Apache
for short. Apache can be easily installed on nearly every system using
the system's package management tool.  lighttpd
(http://www.lighttpd.net)is another alternative and is said to have
better performance.  On many systems this server can also be installed
using the package management tool, so manually compiling the web server
may not be needed.

   * On Apache you can take a look at the Dynamic Content with CGI
     (http://httpd.apache.org/docs/2.2/howto/cgi.html)tutorial, where
     everything is described.  Most of the time it is enough just to
     set `+ExecCGI'.  The tutorial also describes the most common
     gotchas that might arise.

   * On lighttpd you need to use the CGI module
     (http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModCGI), which can
     be configured in a straightforward way.  It boils down to setting
     `cgi.assign' properly.

11.12.1.4 Common problems with CGI scripts
..........................................

Using CGI sometimes leads to small annoyances while trying to get these
scripts to run.  Sometimes a seemingly correct script does not work as
expected, the cause being some small hidden problem that's difficult to
spot.

  Some of these potential problems are:

   * The Python script is not marked as executable.  When CGI scripts
     are not executable most web servers will let the user download it,
     instead of running it and sending the output to the user.  For CGI
     scripts to run properly on Unix-like operating systems, the `+x'
     bit needs to be set.  Using `chmod a+x your_script.py' may solve
     this problem.

   * On a Unix-like system, The line endings in the program file must
     be Unix style line endings.  This is important because the web
     server checks the first line of the script (called shebang) and
     tries to run the program specified there.  It gets easily confused
     by Windows line endings (Carriage Return & Line Feed, also called
     CRLF), so you have to convert the file to Unix line endings (only
     Line Feed, LF).  This can be done automatically by uploading the
     file via FTP in text mode instead of binary mode, but the
     preferred way is just telling your editor to save the files with
     Unix line endings.  Most editors support this.

   * Your web server must be able to read the file, and you need to
     make sure the permissions are correct.  On unix-like systems, the
     server often runs as user and group `www-data', so it might be
     worth a try to change the file ownership, or making the file world
     readable by using `chmod a+r your_script.py'.

   * The web server must know that the file you're trying to access is
     a CGI script.  Check the configuration of your web server, as it
     may be configured to expect a specific file extension for CGI
     scripts.

   * On Unix-like systems, the path to the interpreter in the shebang
     (`#!/usr/bin/env python') must be correct.  This line calls
     `/usr/bin/env' to find Python, but it will fail if there is no
     `/usr/bin/env', or if Python is not in the web server's path.  If
     you know where your Python is installed, you can also use that
     full path.  The commands `whereis python' and `type -p python'
     could help you find where it is installed.  Once you know the
     path, you can change the shebang accordingly: `#!/usr/bin/python'.

   * The file must not contain a BOM (Byte Order Mark). The BOM is
     meant for determining the byte order of UTF-16 and UTF-32
     encodings, but some editors write this also into UTF-8 files.  The
     BOM interferes with the shebang line, so be sure to tell your
     editor not to write the BOM.

   * If the web server is using mod_python, `mod_python'
     may be having problems.  `mod_python' is able to handle CGI
     scripts by itself, but it can also be a source of issues.

11.12.1.5 mod_python
....................

People coming from PHP often find it hard to grasp how to use Python in
the web.  Their first thought is mostly mod_python
(http://www.modpython.org/), because they think that this is the
equivalent to `mod_php'.  Actually, there are many differences.  What
`mod_python' does is embed the interpreter into the Apache process,
thus speeding up requests by not having to start a Python interpreter
for each request.  On the other hand, it is not "Python intermixed with
HTML" in the way that PHP is often intermixed with HTML. The Python
equivalent of that is a template engine.  `mod_python' itself is much
more powerful and provides more access to Apache internals.  It can
emulate CGI, work in a "Python Server Pages" mode (similar to JSP)
which is "HTML intermingled with Python", and it has a "Publisher"
which designates one file to accept all requests and decide what to do
with them.

  `mod_python' does have some problems.  Unlike the PHP interpreter,
the Python interpreter uses caching when executing files, so changes to
a file will require the web server to be restarted.  Another problem is
the basic concept - Apache starts child processes to handle the
requests, and unfortunately every child process needs to load the whole
Python interpreter even if it does not use it.  This makes the whole
web server slower.  Another problem is that, because `mod_python' is
linked against a specific version of `libpython', it is not possible to
switch from an older version to a newer (e.g. 2.4 to 2.5) without
recompiling `mod_python'.  `mod_python' is also bound to the Apache web
server, so programs written for `mod_python' cannot easily run on other
web servers.

  These are the reasons why `mod_python' should be avoided when writing
new programs.  In some circumstances it still might be a good idea to
use `mod_python' for deployment, but WSGI makes it possible to run WSGI
programs under `mod_python' as well.

11.12.1.6 FastCGI and SCGI
..........................

FastCGI and SCGI try to solve the performance problem of CGI in another
way.  Instead of embedding the interpreter into the web server, they
create long-running background processes. There is still a module in
the web server which makes it possible for the web server to "speak"
with the background process.  As the background process is independent
of the server, it can be written in any language, including Python.
The language just needs to have a library which handles the
communication with the webserver.

  The difference between FastCGI and SCGI is very small, as SCGI is
essentially just a "simpler FastCGI".  As the web server support for
SCGI is limited, most people use FastCGI instead, which works the same
way.  Almost everything that applies to SCGI also applies to FastCGI as
well, so we'll only cover the latter.

  These days, FastCGI is never used directly.  Just like `mod_python',
it is only used for the deployment of WSGI applications.

   * FastCGI, SCGI, and Apache: Background and Future
     (http://www.vmunix.com/mark/blog/archives/2006/01/02/fastcgi-scgi-and-apache-background-and-future/)is
     a discussion on why the concept of FastCGI and SCGI is better that
     that of mod_python.

11.12.1.7 Setting up FastCGI
............................

Each web server requires a specific module.

   * Apache has both mod_fastcgi (http://www.fastcgi.com/drupal/)and
     mod_fcgid (http://fastcgi.coremail.cn/).  `mod_fastcgi' is the
     original one, but it has some licensing issues, which is why it is
     sometimes considered non-free.  `mod_fcgid' is a smaller,
     compatible alternative.  One of these modules needs to be loaded
     by Apache.

   * lighttpd ships its own FastCGI module
     (http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModFastCGI)as well
     as an SCGI module
     (http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModSCGI).

   * nginx (http://nginx.org/)also supports FastCGI
     (http://wiki.nginx.org/NginxSimplePythonFCGI).

  Once you have installed and configured the module, you can test it
with the following WSGI-application:

    #!/usr/bin/env python
    # -*- coding: UTF-8 -*-

    from cgi import escape
    import sys, os
    from flup.server.fcgi import WSGIServer

    def app(environ, start_response):
        start_response('200 OK', [('Content-Type', 'text/html')])

        yield '<h1>FastCGI Environment</h1>'
        yield '<table>'
        for k, v in sorted(environ.items()):
             yield '<tr><th>%s</th><td>%s</td></tr>' % (escape(k), escape(v))
        yield '</table>'

    WSGIServer(app).run()

This is a simple WSGI application, but you need to install flup
(http://pypi.python.org/pypi/flup/1.0)first, as flup handles the low
level FastCGI access.

  There is some documentation on setting up Django with FastCGI
(http://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/), most
of which can be reused for other WSGI-compliant frameworks and
libraries.  Only the `manage.py' part has to be changed, the example
used here can be used instead.  Django does more or less the exact same
thing.

11.12.1.8 mod_wsgi
..................

mod_wsgi (http://code.google.com/p/modwsgi/) is an attempt to get rid
of the low level gateways.  Given that FastCGI, SCGI, and mod_python
are mostly used to deploy WSGI applications, mod_wsgi was started to
directly embed WSGI applications into the Apache web server. mod_wsgi
is specifically designed to host WSGI applications.  It makes the
deployment of WSGI applications much easier than deployment using other
low level methods, which need glue code.  The downside is that mod_wsgi
is limited to the Apache web server; other servers would need their own
implementations of mod_wsgi.

  mod_wsgi supports two modes: embedded mode, in which it integrates
with the Apache process, and daemon mode, which is more FastCGI-like.
Unlike FastCGI, mod_wsgi handles the worker-processes by itself, which
makes administration easier.

11.12.2 Step back: WSGI
-----------------------

WSGI has already been mentioned several times, so it has to be something
important.  In fact it really is, and now it is time to explain it.

  The _Web Server Gateway Interface_,  or WSGI for short, is defined in PEP
333 (http://www.python.org/dev/peps/pep-0333) and is currently the best
way to do Python web programming.  While it is great for programmers
writing frameworks, a normal web developer does not need to get in
direct contact with it.  When choosing a framework for web development
it is a good idea to choose one which supports WSGI.

  The big benefit of WSGI is the unification of the application
programming interface.  When your program is compatible with WSGI -
which at the outer level means that the framework you are using has
support for WSGI - your program can be deployed via any web server
interface for which there are WSGI wrappers.  You do not need to care
about whether the application user uses mod_python or FastCGI or
mod_wsgi - with WSGI your application will work on any gateway
interface.  The Python standard library contains its own WSGI server,
see wsgiref, which is a small web server that can be used for
testing.

  A really great WSGI feature is middleware.  Middleware is a layer
around your program which can add various functionality to it.  There
is quite a bit of middleware
(http://wsgi.org/wsgi/Middleware_and_Utilities)already available.  For
example, instead of writing your own session management (HTTP is a
stateless protocol, so to associate multiple HTTP requests with a single
user your application must create and manage such state via a session),
you can just download middleware which does that, plug it in, and get
on with coding the unique parts of your application.  The same thing
with compression - there is existing middleware which handles
compressing your HTML using gzip to save on your server's bandwidth.
Authentication is another a problem easily solved using existing
middleware.

  Although WSGI may seem complex, the initial phase of learning can be
very rewarding because WSGI and the associated middleware already have
solutions to many problems that might arise while developing web sites.

11.12.2.1 WSGI Servers
......................

The code that is used to connect to various low level gateways like CGI
or mod_python is called a _WSGI server_.  One of these servers is
`flup', which supports FastCGI and SCGI, as well as AJP
(http://en.wikipedia.org/wiki/Apache_JServ_Protocol).  Some of these
servers are written in Python, as `flup' is, but there also exist
others which are written in C and can be used as drop-in replacements.

  There are many servers already available, so a Python web application
can be deployed nearly anywhere.  This is one big advantage that Python
has compared with other web technologies.

  A good overview of WSGI-related code can be found in the WSGI wiki
(http://wsgi.org/wsgi), which contains an extensive list of WSGI
servers (http://wsgi.org/wsgi/Servers) which can be used by _any_
application supporting WSGI.

  You might be interested in some WSGI-supporting modules already
contained in the standard library, namely:

   * wsgiref. - some tiny utilities and servers for WSGI

11.12.2.2 Case study: MoinMoin
..............................

What does WSGI give the web application developer?  Let's take a look at
an application that's been around for a while, which was written in
Python without using WSGI.

  One of the most widely used wiki software packages is MoinMoin
(http://moinmo.in/).  It was created in 2000, so it predates WSGI by
about three years.  Older versions needed separate code to run on CGI,
mod_python, FastCGI and standalone.

  It now includes support for WSGI.  Using WSGI, it is possible to
deploy MoinMoin on any WSGI compliant server, with no additional glue
code.  Unlike the pre-WSGI versions, this could include WSGI servers
that the authors of MoinMoin know nothing about.

11.12.3 Model-View-Controller
-----------------------------

The term _MVC_ is often encountered in statements such as "framework
_foo_ supports MVC".  MVC is more about the overall organization of
code, rather than any particular API.  Many web frameworks use this
model to help the developer bring structure to their program.  Bigger
web applications can have lots of code, so it is a good idea to have an
effective structure right from the beginning.  That way, even users of
other frameworks (or even other languages, since MVC is not
Python-specific) can easily understand the code, given that they are
already familiar with the MVC structure.

  MVC stands for three components:

   * The _model_.  This is the data that will be displayed and
     modified.  In Python frameworks, this component is often
     represented by the classes used by an object-relational mapper.

   * The _view_.  This component's job is to display the data of the
     model to the user.  Typically this component is implemented via
     templates.

   * The _controller_.  This is the layer between the user and the
     model.  The controller reacts to user actions (like opening some
     specific URL), tells the model to modify the data if necessary,
     and tells the view code what to display,

  While one might think that MVC is a complex design pattern, in fact
it is not.  It is used in Python because it has turned out to be useful
for creating clean, maintainable web sites.

     Note: While not all Python frameworks explicitly support MVC, it
     is often trivial to create a web site which uses the MVC pattern
     by separating the data logic (the model) from the user interaction
     logic (the controller) and the templates (the view).  That's why
     it is important not to write unnecessary Python code in the
     templates - it works against the MVC model and creates chaos in
     the code base, making it harder to understand and modify.

  The English Wikipedia has an article about the Model-View-Controller
pattern (http://en.wikipedia.org/wiki/Model-view-controller).  It
includes a long list of web frameworks for various programming
languages.

11.12.4 Ingredients for Websites
--------------------------------

Websites are complex constructs, so tools have been created to help web
developers make their code easier to write and more maintainable.
Tools like these exist for all web frameworks in all languages.
Developers are not forced to use these tools, and often there is no
"best" tool.  It is worth learning about the available tools because
they can greatly simplify the process of developing a web site.

  There are far more components than can be presented here.  The Python
wiki has a page about these components, called Web Components
(http://wiki.python.org/moin/WebComponents).

11.12.4.1 Templates
...................

Mixing of HTML and Python code is made possible by a few libraries.
While convenient at first, it leads to horribly unmaintainable code.
That's why templates exist.  Templates are, in the simplest case, just
HTML files with placeholders.  The HTML is sent to the user's browser
after filling in the placeholders.

  Python already includes two ways to build simple templates:

    >>> template = "<html><body><h1>Hello %s!</h1></body></html>"
    >>> print template % "Reader"
    <html><body><h1>Hello Reader!</h1></body></html>

    >>> from string import Template
    >>> template = Template("<html><body><h1>Hello ${name}</h1></body></html>")
    >>> print template.substitute(dict(name='Dinsdale'))
    <html><body><h1>Hello Dinsdale!</h1></body></html>

To generate complex HTML based on non-trivial model data, conditional
and looping constructs like Python's _for_ and _if_ are generally
needed.  _Template engines_ support templates of this complexity.

  There are a lot of template engines available for Python which can be
used with or without a framework.  Some of these define a
plain-text programming language which is easy to learn, partly because
it is limited in scope.  Others use XML, and the template output is
guaranteed to be always be valid XML.  There are many other variations.

  Some frameworks ship their own template engine or
recommend one in particular.  In the absence of a reason to use a
different template engine, using the one provided by or recommended by
the framework is a good idea.

  Popular template engines include:

        * Mako (http://www.makotemplates.org/)

        * Genshi (http://genshi.edgewall.org/)

        * Jinja (http://jinja.pocoo.org/2/)

  There are many template engines competing for attention, because it is
pretty easy to create them in Python.  The page Templating
(http://wiki.python.org/moin/Templating) in the wiki lists a big,
ever-growing number of these.  The three listed above are considered
"second generation" template engines and are a good place to start.

11.12.4.2 Data persistence
..........................

_Data persistence_, while sounding very complicated, is just about
storing data.  This data might be the text of blog entries, the
postings on a bulletin board or the text of a wiki page.  There are, of
course, a number of different ways to store information on a web server.

  Often, relational database engines like MySQL
(http://www.mysql.com/)or PostgreSQL (http://www.postgresql.org/)are
used because of their good performance when handling very large
databases consisting of millions of entries.  There is also a small
database engine called SQLite (http://www.sqlite.org/), which is
bundled with Python in the sqlite3.  module, and which uses
only one file.  It has no other dependencies.  For smaller sites SQLite
is just enough.

  Relational databases are _queried_ using a language called SQL
(http://en.wikipedia.org/wiki/SQL).  Python programmers in general do
not like SQL too much, as they prefer to work with objects.  It is
possible to save Python objects into a database using a technology
called ORM
(http://en.wikipedia.org/wiki/Object-relational_mapping)(Object
Relational Mapping).  ORM translates all object-oriented access into
SQL code under the hood, so the developer does not need to think about
it.  Most frameworks use ORMs, and it works quite well.

  A second possibility is storing data in normal, plain text files (some
times called "flat files").  This is very easy for simple sites, but
can be difficult to get right if the web site is performing many
updates to the stored data.

  A third possibility are object oriented databases (also called "object
databases").  These databases store the object data in a form that
closely parallels the way the objects are structured in memory during
program execution.  (By contrast, ORMs store the object data as rows of
data in tables and relations between those rows.)  Storing the objects
directly has the advantage that nearly all objects can be saved in a
straightforward way, unlike in relational databases where some objects
are very hard to represent.

  Frameworks often give hints on which data storage method
to choose.  It is usually a good idea to stick to the data store
recommended by the framework unless the application has special
requirements better satisfied by an alternate storage mechanism.

   * Persistence Tools
     (http://wiki.python.org/moin/PersistenceTools)lists possibilities
     on how to save data in the file system.  Some of these modules are
     part of the standard library

   * Database Programming
     (http://wiki.python.org/moin/DatabaseProgramming)helps with
     choosing a method for saving data

   * SQLAlchemy (http://www.sqlalchemy.org/), the most powerful
     OR-Mapper for Python, and Elixir (http://elixir.ematia.de/), which
     makes SQLAlchemy easier to use

   * SQLObject (http://www.sqlobject.org/), another popular OR-Mapper

   * ZODB (https://launchpad.net/zodb)and Durus
     (http://www.mems-exchange.org/software/durus/), two object oriented
     databases

11.12.5 Frameworks
------------------

The process of creating code to run web sites involves writing code to
provide various services.  The code to provide a particular service
often works the same way regardless of the complexity or purpose of the
web site in question.  Abstracting these common solutions into reusable
code produces what are called "frameworks" for web development.
Perhaps the most well-known framework for web development is Ruby on
Rails, but Python has its own frameworks.  Some of these were partly
inspired by Rails, or borrowed ideas from Rails, but many existed a
long time before Rails.

  Originally Python web frameworks tended to incorporate all of the
services needed to develop web sites as a giant, integrated set of
tools.  No two web frameworks were interoperable:  a program developed
for one could not be deployed on a different one without considerable
re-engineering work.  This led to the development of "minimalist" web
frameworks that provided just the tools to communicate between the
Python code and the http protocol, with all other services to be added
on top via separate components.  Some ad hoc standards were developed
that allowed for limited interoperability between frameworks, such as a
standard that allowed different template engines to be used
interchangeably.

  Since the advent of WSGI, the Python web framework world has been
evolving toward interoperability based on the WSGI standard.  Now many
web frameworks, whether "full stack" (providing all the tools one needs
to deploy the most complex web sites) or minimalist, or anything in
between, are built from collections of reusable components that can be
used with more than one framework.

  The majority of users will probably want to select a "full stack"
framework that has an active community.  These frameworks tend to be
well documented, and provide the easiest path to producing a fully
functional web site in minimal time.

11.12.5.1 Some notable frameworks
.................................

There are an incredible number of frameworks, so they cannot all be
covered here.  Instead we will briefly touch on some of the most
popular.

11.12.5.2 Django
................

Django (http://www.djangoproject.com/) is a framework consisting of
several tightly coupled elements which were written from scratch and
work together very well.  It includes an ORM which is quite powerful
while being simple to use, and has a great online administration
interface which makes it possible to edit the data in the database with
a browser.  The template engine is text-based and is designed to be
usable for page designers who cannot write Python.  It supports
template inheritance and filters (which work like Unix pipes).  Django
has many handy features bundled, such as creation of RSS feeds or
generic views, which make it possible to create web sites almost
without writing any Python code.

  It has a big, international community, the members of which have
created many web sites.  There are also a lot of add-on projects which
extend Django's normal functionality.  This is partly due to Django's
well written online documentation (http://docs.djangoproject.com/)and
the Django book (http://www.djangobook.com/).

     Note: Although Django is an MVC-style framework, it names the
     elements differently, which is described in the Django FAQ
     (http://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names).

11.12.5.3 TurboGears
....................

Another popular web framework for Python is TurboGears
(http://www.turbogears.org/).  TurboGears takes the approach of using
already existing components and combining them with glue code to create
a seamless experience.  TurboGears gives the user flexibility in
choosing components. For example the ORM and template engine can be
changed to use packages different from those used by default.

  The documentation can be found in the TurboGears wiki
(http://docs.turbogears.org/), where links to screencasts can be found.
TurboGears has also an active user community which can respond to most
related questions.  There is also a TurboGears book
(http://turbogearsbook.com/)published, which is a good starting point.

  The newest version of TurboGears, version 2.0, moves even further in
direction of WSGI support and a component-based architecture.
TurboGears 2 is based on the WSGI stack of another popular
component-based web framework, Pylons (http://pylonshq.com/).

11.12.5.4 Zope
..............

The Zope framework is one of the "old original" frameworks.  Its current
incarnation in Zope2 is a tightly integrated full-stack framework.  One
of its most interesting feature is its tight integration with a
powerful object database called the ZODB (https://launchpad.net/zodb)
(Zope Object Database).  Because of its highly integrated nature, Zope
wound up in a somewhat isolated ecosystem:  code written for Zope
wasn't very usable outside of Zope, and vice-versa.  To solve this
problem the Zope 3 effort was started.  Zope 3 re-engineers Zope as a
set of more cleanly isolated components.  This effort was started
before the advent of the WSGI standard, but there is WSGI support for
Zope 3 from the Repoze (http://repoze.org/)project.  Zope components
have many years of production use behind them, and the Zope 3 project
gives access to these components to the wider Python community.  There
is even a separate framework based on the Zope components: Grok
(http://grok.zope.org/).

  Zope is also the infrastructure used by the Plone
(http://plone.org/)content management system, one of the most powerful
and popular content management systems available.

11.12.5.5 Other notable frameworks
..................................

Of course these are not the only frameworks that are available.  There
are many other frameworks worth mentioning.

  Another framework that's already been mentioned is Pylons
(http://pylonshq.com/).  Pylons is much like TurboGears, but with an
even stronger emphasis on flexibility, which comes at the cost of being
more difficult to use.  Nearly every component can be exchanged, which
makes it necessary to use the documentation of every single component,
of which there are many.  Pylons builds upon Paste
(http://pythonpaste.org/), an extensive set of tools which are handy
for WSGI.

  And that's still not everything.  The most up-to-date information can
always be found in the Python wiki.

  The Python wiki contains an extensive list of web frameworks
(http://wiki.python.org/moin/WebFrameworks).

  Most frameworks also have their own mailing lists and IRC channels,
look out for these on the projects' web sites.  There is also a general
"Python in the Web" IRC channel on freenode called #python.web
(http://wiki.python.org/moin/PoundPythonWeb).

12 Python Frequently Asked Questions
************************************

     Release: 2.7

     Date: February 08, 2011

12.1 General Python FAQ
=======================

12.1.1 General Information
--------------------------

12.1.1.1 What is Python?
........................

Python is an interpreted, interactive, object-oriented programming
language.  It incorporates modules, exceptions, dynamic typing, very
high level dynamic data types, and classes.  Python combines remarkable
power with very clear syntax.  It has interfaces to many system calls
and libraries, as well as to various window systems, and is extensible
in C or C++.  It is also usable as an extension language for
applications that need a programmable interface.  Finally, Python is
portable: it runs on many Unix variants, on the Mac, and on PCs under
MS-DOS, Windows, Windows NT, and OS/2.

  To find out more, start with The Python Tutorial.  The
Beginner's Guide to Python
(http://wiki.python.org/moin/BeginnersGuide)links to other introductory
tutorials and resources for learning Python.

12.1.1.2 What is the Python Software Foundation?
................................................

The Python Software Foundation is an independent non-profit
organization that holds the copyright on Python versions 2.1 and newer.
The PSF's mission is to advance open source technology related to the
Python programming language and to publicize the use of Python.  The
PSF's home page is at <http://www.python.org/psf/>.

  Donations to the PSF are tax-exempt in the US.  If you use Python and
find it helpful, please contribute via the PSF donation page
(http://www.python.org/psf/donations/).

12.1.1.3 Are there copyright restrictions on the use of Python?
...............................................................

You can do anything you want with the source, as long as you leave the
copyrights in and display those copyrights in any documentation about
Python that you produce.  If you honor the copyright rules, it's OK to
use Python for commercial use, to sell copies of Python in source or
binary form (modified or unmodified), or to sell products that
incorporate Python in some form.  We would still like to know about all
commercial use of Python, of course.

  See the PSF license page (http://python.org/psf/license/)to find
further explanations and a link to the full text of the license.

  The Python logo is trademarked, and in certain cases permission is
required to use it.  Consult the Trademark Usage Policy
(http://www.python.org/psf/trademarks/) for more information.

12.1.1.4 Why was Python created in the first place?
...................................................

Here's a _very_ brief summary of what started it all, written by Guido
van Rossum:

     I had extensive experience with implementing an interpreted
     language in the ABC group at CWI, and from working with this group
     I had learned a lot about language design.  This is the origin of
     many Python features, including the use of indentation for
     statement grouping and the inclusion of very-high-level data types
     (although the details are all different in Python).

     I had a number of gripes about the ABC language, but also liked
     many of its features.  It was impossible to extend the ABC
     language (or its implementation) to remedy my complaints - in fact
     its lack of extensibility was one of its biggest problems.  I had
     some experience with using Modula-2+ and talked with the designers
     of Modula-3 and read the Modula-3 report.  Modula-3 is the origin
     of the syntax and semantics used for exceptions, and some other
     Python features.

     I was working in the Amoeba distributed operating system group at
     CWI.  We needed a better way to do system administration than by
     writing either C programs or Bourne shell scripts, since Amoeba
     had its own system call interface which wasn't easily accessible
     from the Bourne shell.  My experience with error handling in
     Amoeba made me acutely aware of the importance of exceptions as a
     programming language feature.

     It occurred to me that a scripting language with a syntax like ABC
     but with access to the Amoeba system calls would fill the need.  I
     realized that it would be foolish to write an Amoeba-specific
     language, so I decided that I needed a language that was generally
     extensible.

     During the 1989 Christmas holidays, I had a lot of time on my
     hand, so I decided to give it a try.  During the next year, while
     still mostly working on it in my own time, Python was used in the
     Amoeba project with increasing success, and the feedback from
     colleagues made me add many early improvements.

     In February 1991, after just over a year of development, I decided
     to post to USENET.  The rest is in the `Misc/HISTORY' file.

12.1.1.5 What is Python good for?
.................................

Python is a high-level general-purpose programming language that can be
applied to many different classes of problems.

  The language comes with a large standard library that covers areas
such as string processing (regular expressions, Unicode, calculating
differences between files), Internet protocols (HTTP, FTP, SMTP,
XML-RPC, POP, IMAP, CGI programming), software engineering (unit
testing, logging, profiling, parsing Python code), and operating system
interfaces (system calls, filesystems, TCP/IP sockets).  Look at the
table of contents for The Python Standard Library to get an
idea of what's available.  A wide variety of third-party extensions are
also available.  Consult the Python Package Index
(http://pypi.python.org/pypi)to find packages of interest to you.

12.1.1.6 How does the Python version numbering scheme work?
...........................................................

Python versions are numbered A.B.C or A.B.  A is the major version
number - it is only incremented for really major changes in the
language.  B is the minor version number, incremented for less
earth-shattering changes.  C is the micro-level - it is incremented for
each bugfix release.  See PEP 6
(http://www.python.org/dev/peps/pep-0006) for more information about
bugfix releases.

  Not all releases are bugfix releases.  In the run-up to a new major
release, a series of development releases are made, denoted as alpha,
beta, or release candidate.  Alphas are early releases in which
interfaces aren't yet finalized; it's not unexpected to see an
interface change between two alpha releases.  Betas are more stable,
preserving existing interfaces but possibly adding new modules, and
release candidates are frozen, making no changes except as needed to
fix critical bugs.

  Alpha, beta and release candidate versions have an additional suffix.
The suffix for an alpha version is "aN" for some small number N, the
suffix for a beta version is "bN" for some small number N, and the
suffix for a release candidate version is "cN" for some small number N.
In other words, all versions labeled 2.0aN precede the versions labeled
2.0bN, which precede versions labeled 2.0cN, and _those_ precede 2.0.

  You may also find version numbers with a "+" suffix, e.g. "2.2+".
These are unreleased versions, built directly from the Subversion
trunk.  In practice, after a final minor release is made, the
Subversion trunk is incremented to the next minor version, which
becomes the "a0" version, e.g. "2.4a0".

  See also the documentation for `sys.version', `sys.hexversion', and
`sys.version_info'.

12.1.1.7 How do I obtain a copy of the Python source?
.....................................................

The latest Python source distribution is always available from
python.org, at <http://www.python.org/download/>.  The latest
development sources can be obtained via anonymous Subversion at
<http://svn.python.org/projects/python/trunk>.

  The source distribution is a gzipped tar file containing the complete
C source, Sphinx-formatted documentation, Python library modules,
example programs, and several useful pieces of freely distributable
software.  The source will compile and run out of the box on most UNIX
platforms.

  Consult the Developer FAQ (http://www.python.org/dev/faq/) for more
information on getting the source code and compiling it.

12.1.1.8 How do I get documentation on Python?
..............................................

The standard documentation for the current stable version of Python is
available at <http://docs.python.org/>.  PDF, plain text, and
downloadable HTML versions are also available at
<http://docs.python.org/download.html>.

  The documentation is written in reStructuredText and processed by the
Sphinx documentation tool (http://sphinx.pocoo.org/).  The
reStructuredText source for the documentation is part of the Python
source distribution.

12.1.1.9 I've never programmed before. Is there a Python tutorial?
..................................................................

There are numerous tutorials and books available.  The standard
documentation includes The Python Tutorial.

  Consult the Beginner's Guide
(http://wiki.python.org/moin/BeginnersGuide)to find information for
beginning Python programmers, including lists of tutorials.

12.1.1.10 Is there a newsgroup or mailing list devoted to Python?
.................................................................

There is a newsgroup, `comp.lang.python', and a mailing list,
python-list (http://mail.python.org/mailman/listinfo/python-list).  The
newsgroup and mailing list are gatewayed into each other - if you can
read news it's unnecessary to subscribe to the mailing list.
`comp.lang.python' is high-traffic, receiving hundreds of postings
every day, and Usenet readers are often more able to cope with this
volume.

  Announcements of new software releases and events can be found in
comp.lang.python.announce, a low-traffic moderated list that receives
about five postings per day.  It's available as the python-announce
mailing list
(http://mail.python.org/mailman/listinfo/python-announce-list).

  More info about other mailing lists and newsgroups can be found at
<http://www.python.org/community/lists/>.

12.1.1.11 How do I get a beta test version of Python?
.....................................................

Alpha and beta releases are available from
<http://www.python.org/download/>.  All releases are announced on the
comp.lang.python and comp.lang.python.announce newsgroups and on the
Python home page at <http://www.python.org/>; an RSS feed of news is
available.

  You can also access the development version of Python through
Subversion.  See <http://www.python.org/dev/faq/> for details.

12.1.1.12 How do I submit bug reports and patches for Python?
.............................................................

To report a bug or submit a patch, please use the Roundup installation
at <http://bugs.python.org/>.

  You must have a Roundup account to report bugs; this makes it
possible for us to contact you if we have follow-up questions.  It will
also enable Roundup to send you updates as we act on your bug. If you
had previously used SourceForge to report bugs to Python, you can
obtain your Roundup password through Roundup's password reset procedure
(http://bugs.python.org/user?@template=forgotten).

  For more information on how Python is developed, consult the Python
Developer's Guide (http://python.org/dev/).

12.1.1.13 Are there any published articles about Python that I can reference?
.............................................................................

It's probably best to cite your favorite book about Python.

  The very first article about Python was written in 1991 and is now
quite outdated.

     Guido van Rossum and Jelke de Boer, "Interactively Testing Remote
     Servers Using the Python Programming Language", CWI Quarterly,
     Volume 4, Issue 4 (December 1991), Amsterdam, pp 283-303.

12.1.1.14 Are there any books on Python?
........................................

Yes, there are many, and more are being published.  See the python.org
wiki at <http://wiki.python.org/moin/PythonBooks> for a list.

  You can also search online bookstores for "Python" and filter out the
Monty Python references; or perhaps search for "Python" and "language".

12.1.1.15 Where in the world is www.python.org located?
.......................................................

It's currently in Amsterdam, graciously hosted by XS4ALL
(http://www.xs4all.nl).  Thanks to Thomas Wouters for his work in
arranging python.org's hosting.

12.1.1.16 Why is it called Python?
..................................

When he began implementing Python, Guido van Rossum was also reading the
published scripts from "Monty Python's Flying Circus"
(http://pythonline.com/), a BBC comedy series from the 1970s.  Van
Rossum thought he needed a name that was short, unique, and slightly
mysterious, so he decided to call the language Python.

12.1.1.17 Do I have to like "Monty Python's Flying Circus"?
...........................................................

No, but it helps.  :)

12.1.2 Python in the real world
-------------------------------

12.1.2.1 How stable is Python?
..............................

Very stable.  New, stable releases have been coming out roughly every 6
to 18 months since 1991, and this seems likely to continue.  Currently
there are usually around 18 months between major releases.

  The developers issue "bugfix" releases of older versions, so the
stability of existing releases gradually improves.  Bugfix releases,
indicated by a third component of the version number (e.g. 2.5.3,
2.6.2), are managed for stability; only fixes for known problems are
included in a bugfix release, and it's guaranteed that interfaces will
remain the same throughout a series of bugfix releases.

  The latest stable releases can always be found on the Python download
page (http://python.org/download/).  There are two recommended
production-ready versions at this point in time, because at the moment
there are two branches of stable releases: 2.x and 3.x.  Python 3.x may
be less useful than 2.x, since currently there is more third party
software available for Python 2 than for Python 3.  Python 2 code will
generally not run unchanged in Python 3.

12.1.2.2 How many people are using Python?
..........................................

There are probably tens of thousands of users, though it's difficult to
obtain an exact count.

  Python is available for free download, so there are no sales figures,
and it's available from many different sites and packaged with many
Linux distributions, so download statistics don't tell the whole story
either.

  The comp.lang.python newsgroup is very active, but not all Python
users post to the group or even read it.

12.1.2.3 Have any significant projects been done in Python?
...........................................................

See <http://python.org/about/success> for a list of projects that use
Python.  Consulting the proceedings for past Python conferences
(http://python.org/community/workshops/)will reveal contributions from
many different companies and organizations.

  High-profile Python projects include the Mailman mailing list manager
(http://www.list.org)and the Zope application server
(http://www.zope.org).  Several Linux distributions, most notably Red
Hat (http://www.redhat.com), have written part or all of their
installer and system administration software in Python.  Companies that
use Python internally include Google, Yahoo, and Lucasfilm Ltd.

12.1.2.4 What new developments are expected for Python in the future?
.....................................................................

See <http://www.python.org/dev/peps/> for the Python Enhancement
Proposals (PEPs). PEPs are design documents describing a suggested new
feature for Python, providing a concise technical specification and a
rationale.  Look for a PEP titled "Python X.Y Release Schedule", where
X.Y is a version that hasn't been publicly released yet.

  New development is discussed on the python-dev mailing list
(http://mail.python.org/mailman/listinfo/python-dev/).

12.1.2.5 Is it reasonable to propose incompatible changes to Python?
....................................................................

In general, no.  There are already millions of lines of Python code
around the world, so any change in the language that invalidates more
than a very small fraction of existing programs has to be frowned upon.
Even if you can provide a conversion program, there's still the problem
of updating all documentation; many books have been written about
Python, and we don't want to invalidate them all at a single stroke.

  Providing a gradual upgrade path is necessary if a feature has to be
changed.  PEP 5 (http://www.python.org/dev/peps/pep-0005) describes the
procedure followed for introducing backward-incompatible changes while
minimizing disruption for users.

12.1.2.6 Is Python Y2K (Year 2000) Compliant?
.............................................

As of August, 2003 no major problems have been reported and Y2K
compliance seems to be a non-issue.

  Python does very few date calculations and for those it does perform
relies on the C library functions.  Python generally represents times
either as seconds since 1970 or as a `(year, month, day, ...)' tuple
where the year is expressed with four digits, which makes Y2K bugs
unlikely.  So as long as your C library is okay, Python should be okay.
Of course, it's possible that a particular application written in
Python makes assumptions about 2-digit years.

  Because Python is available free of charge, there are no absolute
guarantees.  If there _are_ unforeseen problems, liability is the
user's problem rather than the developers', and there is nobody you can
sue for damages.  The Python copyright notice contains the following
disclaimer:

     4. PSF is making Python 2.3 available to Licensee on an "AS IS"
     basis.  PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
     IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
     DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR
     FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 2.3
     WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

     5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
     2.3 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS
     AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON
     2.3, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY
     THEREOF.

  The good news is that _if_ you encounter a problem, you have full
source available to track it down and fix it.  This is one advantage of
an open source programming environment.

12.1.2.7 Is Python a good language for beginning programmers?
.............................................................

Yes.

  It is still common to start students with a procedural and statically
typed language such as Pascal, C, or a subset of C++ or Java.  Students
may be better served by learning Python as their first language.
Python has a very simple and consistent syntax and a large standard
library and, most importantly, using Python in a beginning programming
course lets students concentrate on important programming skills such
as problem decomposition and data type design.  With Python, students
can be quickly introduced to basic concepts such as loops and
procedures.  They can probably even work with user-defined objects in
their very first course.

  For a student who has never programmed before, using a statically
typed language seems unnatural.  It presents additional complexity that
the student must master and slows the pace of the course.  The students
are trying to learn to think like a computer, decompose problems,
design consistent interfaces, and encapsulate data.  While learning to
use a statically typed language is important in the long term, it is
not necessarily the best topic to address in the students' first
programming course.

  Many other aspects of Python make it a good first language.  Like
Java, Python has a large standard library so that students can be
assigned programming projects very early in the course that _do_
something.  Assignments aren't restricted to the standard four-function
calculator and check balancing programs.  By using the standard
library, students can gain the satisfaction of working on realistic
applications as they learn the fundamentals of programming.  Using the
standard library also teaches students about code reuse.  Third-party
modules such as PyGame are also helpful in extending the students'
reach.

  Python's interactive interpreter enables students to test language
features while they're programming.  They can keep a window with the
interpreter running while they enter their program's source in another
window.  If they can't remember the methods for a list, they can do
something like this:

    >>> L = []
    >>> dir(L)
    ['append', 'count', 'extend', 'index', 'insert', 'pop', 'remove',
    'reverse', 'sort']
    >>> help(L.append)
    Help on built-in function append:

    append(...)
        L.append(object) -- append object to end
    >>> L.append(1)
    >>> L
    [1]

With the interpreter, documentation is never far from the student as
he's programming.

  There are also good IDEs for Python.  IDLE is a cross-platform IDE
for Python that is written in Python using Tkinter.  PythonWin is a
Windows-specific IDE.  Emacs users will be happy to know that there is
a very good Python mode for Emacs.  All of these programming
environments provide syntax highlighting, auto-indenting, and access to
the interactive interpreter while coding.  Consult
<http://www.python.org/editors/> for a full list of Python editing
environments.

  If you want to discuss Python's use in education, you may be
interested in joining the edu-sig mailing list
(http://python.org/community/sigs/current/edu-sig).

12.1.3 Upgrading Python
-----------------------

12.1.3.1 What is this bsddb185 module my application keeps complaining about?
.............................................................................

Starting with Python2.3, the distribution includes the `PyBSDDB package
<http://pybsddb.sf.net/>' as a replacement for the old bsddb module.  It
includes functions which provide backward compatibility at the API
level, but requires a newer version of the underlying Berkeley DB
(http://www.sleepycat.com)library.  Files created with the older bsddb
module can't be opened directly using the new module.

  Using your old version of Python and a pair of scripts which are part
of Python 2.3 (db2pickle.py and pickle2db.py, in the Tools/scripts
directory) you can convert your old database files to the new format.
Using your old Python version, run the db2pickle.py script to convert
it to a pickle, e.g.:

    python2.2 <pathto>/db2pickley.py database.db database.pck

Rename your database file:

    mv database.db olddatabase.db

Now convert the pickle file to a new format database:

    python <pathto>/pickle2db.py database.db database.pck

The precise commands you use will vary depending on the particulars of
your installation.  For full details about operation of these two
scripts check the doc string at the start of each one.

12.2 Programming FAQ
====================

12.2.1 General Questions
------------------------

12.2.1.1 Is there a source code level debugger with breakpoints, single-stepping, etc.?
.......................................................................................

Yes.

  The pdb module is a simple but adequate console-mode debugger for
Python. It is part of the standard Python library, and is
documented in the Library Reference Manual. You can also write
your own debugger by using the code for pdb as an example.

  The IDLE interactive development environment, which is part of the
standard Python distribution (normally available as
Tools/scripts/idle), includes a graphical debugger.  There is
documentation for the IDLE debugger at
<http://www.python.org/idle/doc/idle2.html#Debugger>.

  PythonWin is a Python IDE that includes a GUI debugger based on pdb.
The Pythonwin debugger colors breakpoints and has quite a few cool
features such as debugging non-Pythonwin programs.  Pythonwin is
available as part of the Python for Windows Extensions
(http://sourceforge.net/projects/pywin32/) project and as a part of the
ActivePython distribution (see
<http://www.activestate.com/Products/ActivePython/index.html>).

  Boa Constructor (http://boa-constructor.sourceforge.net/)is an IDE
and GUI builder that uses wxWidgets.  It offers visual frame creation
and manipulation, an object inspector, many views on the source like
object browsers, inheritance hierarchies, doc string generated html
documentation, an advanced debugger, integrated help, and Zope support.

  Eric (http://www.die-offenbachs.de/eric/index.html)is an IDE built on
PyQt and the Scintilla editing component.

  Pydb is a version of the standard Python debugger pdb, modified for
use with DDD (Data Display Debugger), a popular graphical debugger
front end.  Pydb can be found at <http://bashdb.sourceforge.net/pydb/>
and DDD can be found at <http://www.gnu.org/software/ddd>.

  There are a number of commercial Python IDEs that include graphical
debuggers.  They include:

   * Wing IDE (<http://wingware.com/>)

   * Komodo IDE (<http://www.activestate.com/Products/Komodo>)

12.2.1.2 Is there a tool to help find bugs or perform static analysis?
......................................................................

Yes.

  PyChecker is a static analysis tool that finds bugs in Python source
code and warns about code complexity and style.  You can get PyChecker
from <http://pychecker.sf.net>.

  Pylint (http://www.logilab.org/projects/pylint)is another tool that
checks if a module satisfies a coding standard, and also makes it
possible to write plug-ins to add a custom feature.  In addition to the
bug checking that PyChecker performs, Pylint offers some additional
features such as checking line length, whether variable names are
well-formed according to your coding standard, whether declared
interfaces are fully implemented, and more.
<http://www.logilab.org/card/pylint_manual> provides a full list of
Pylint's features.

12.2.1.3 How can I create a stand-alone binary from a Python script?
....................................................................

You don't need the ability to compile Python to C code if all you want
is a stand-alone program that users can download and run without having
to install the Python distribution first.  There are a number of tools
that determine the set of modules required by a program and bind these
modules together with a Python binary to produce a single executable.

  One is to use the freeze tool, which is included in the Python source
tree as `Tools/freeze'. It converts Python byte code to C arrays; a C
compiler you can embed all your modules into a new program, which is
then linked with the standard Python modules.

  It works by scanning your source recursively for import statements
(in both forms) and looking for the modules in the standard Python path
as well as in the source directory (for built-in modules).  It then
turns the bytecode for modules written in Python into C code (array
initializers that can be turned into code objects using the marshal
module) and creates a custom-made config file that only contains those
built-in modules which are actually used in the program.  It then
compiles the generated C code and links it with the rest of the Python
interpreter to form a self-contained binary which acts exactly like
your script.

  Obviously, freeze requires a C compiler.  There are several other
utilities which don't. One is Thomas Heller's py2exe (Windows only) at

     <http://www.py2exe.org/>

  Another is Christian Tismer's SQFREEZE
(http://starship.python.net/crew/pirx)which appends the byte code to a
specially-prepared Python interpreter that can find the byte code in
the executable.

  Other tools include Fredrik Lundh's Squeeze
(http://www.pythonware.com/products/python/squeeze)and Anthony
Tuininga's cx_Freeze
(http://starship.python.net/crew/atuining/cx_Freeze/index.html).

12.2.1.4 Are there coding standards or a style guide for Python programs?
.........................................................................

Yes.  The coding style required for standard library modules is
documented as PEP 8 (http://www.python.org/dev/peps/pep-0008).

12.2.1.5 My program is too slow. How do I speed it up?
......................................................

That's a tough one, in general.  There are many tricks to speed up
Python code; consider rewriting parts in C as a last resort.

  In some cases it's possible to automatically translate Python to C or
x86 assembly language, meaning that you don't have to modify your code
to gain increased speed.

  Pyrex (http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/)can
compile a slightly modified version of Python code into a C extension,
and can be used on many different platforms.

  Psyco (http://psyco.sourceforge.net)is a just-in-time compiler that
translates Python code into x86 assembly language.  If you can use it,
Psyco can provide dramatic speedups for critical functions.

  The rest of this answer will discuss various tricks for squeezing a
bit more speed out of Python code.  _Never_ apply any optimization
tricks unless you know you need them, after profiling has indicated
that a particular function is the heavily executed hot spot in the
code.  Optimizations almost always make the code less clear, and you
shouldn't pay the costs of reduced clarity (increased development time,
greater likelihood of bugs) unless the resulting performance benefit is
worth it.

  There is a page on the wiki devoted to performance tips
(http://wiki.python.org/moin/PythonSpeed/PerformanceTips).

  Guido van Rossum has written up an anecdote related to optimization at
<http://www.python.org/doc/essays/list2str.html>.

  One thing to notice is that function and (especially) method calls
are rather expensive; if you have designed a purely OO interface with
lots of tiny functions that don't do much more than get or set an
instance variable or call another method, you might consider using a
more direct way such as directly accessing instance variables.  Also
see the standard module profile which makes it possible to
find out where your program is spending most of its time (if you have
some patience - the profiling itself can slow your program down by an
order of magnitude).

  Remember that many standard optimization heuristics you may know from
other programming experience may well apply to Python.  For example it
may be faster to send output to output devices using larger writes
rather than smaller ones in order to reduce the overhead of kernel
system calls.  Thus CGI scripts that write all output in "one shot" may
be faster than those that write lots of small pieces of output.

  Also, be sure to use Python's core features where appropriate.  For
example, slicing allows programs to chop up lists and other sequence
objects in a single tick of the interpreter's mainloop using highly
optimized C implementations.  Thus to get the same effect as:

    L2 = []
    for i in range[3]:
        L2.append(L1[i])

it is much shorter and far faster to use

    L2 = list(L1[:3])  # "list" is redundant if L1 is a list.

Note that the functionally-oriented built-in functions such as
map(), zip(), and friends can be a convenient
accelerator for loops that perform a single task.  For example to pair
the elements of two lists together:

    >>> zip([1, 2, 3], [4, 5, 6])
    [(1, 4), (2, 5), (3, 6)]

or to compute a number of sines:

    >>> map(math.sin, (1, 2, 3, 4))
    [0.841470984808, 0.909297426826, 0.14112000806, -0.756802495308]

The operation completes very quickly in such cases.

  Other examples include the `join()' and `split()' methods of string objects.
For example if s1..s7 are large (10K+) strings
then `"".join([s1,s2,s3,s4,s5,s6,s7])' may be far faster than the more
obvious `s1+s2+s3+s4+s5+s6+s7', since the "summation" will compute many
subexpressions, whereas `join()' does all the copying in one pass.  For
manipulating strings, use the `replace()' and the `format()'
methods on string objects.  Use regular expressions only when
you're not dealing with constant string patterns.  You may still use
the old % operations. `string % tuple' and `string %
dictionary'.

  Be sure to use the `list.sort()' built-in method to do sorting, and
see the sorting mini-HOWTO
(http://wiki.python.org/moin/HowTo/Sorting)for examples of moderately
advanced usage.  `list.sort()' beats other techniques for sorting in
all but the most extreme circumstances.

  Another common trick is to "push loops into functions or methods."
For example suppose you have a program that runs slowly and you use the
profiler to determine that a Python function `ff()' is being called
lots of times.  If you notice that `ff()':

    def ff(x):
        ... # do something with x computing result...
        return result

tends to be called in loops like:

    list = map(ff, oldlist)

or:

    for x in sequence:
        value = ff(x)
        ... # do something with value...

then you can often eliminate function call overhead by rewriting `ff()'
to:

    def ffseq(seq):
        resultseq = []
        for x in seq:
            ... # do something with x computing result...
            resultseq.append(result)
        return resultseq

and rewrite the two examples to `list = ffseq(oldlist)' and to:

    for value in ffseq(sequence):
        ... # do something with value...

Single calls to `ff(x)' translate to `ffseq([x])[0]' with little
penalty.  Of course this technique is not always appropriate and there
are other variants which you can figure out.

  You can gain some performance by explicitly storing the results of a
function or method lookup into a local variable.  A loop like:

    for key in token:
        dict[key] = dict.get(key, 0) + 1

resolves `dict.get' every iteration.  If the method isn't going to
change, a slightly faster implementation is:

    dict_get = dict.get  # look up the method once
    for key in token:
        dict[key] = dict_get(key, 0) + 1

Default arguments can be used to determine values once, at compile time
instead of at run time.  This can only be done for functions or objects
which will not be changed during program execution, such as replacing

    def degree_sin(deg):
        return math.sin(deg * math.pi / 180.0)

with

    def degree_sin(deg, factor=math.pi/180.0, sin=math.sin):
        return sin(deg * factor)

Because this trick uses default arguments for terms which should not be
changed, it should only be used when you are not concerned with
presenting a possibly confusing API to your users.

12.2.2 Core Language
--------------------

12.2.2.1 Why am I getting an UnboundLocalError when the variable has a value?
.............................................................................

It can be a surprise to get the UnboundLocalError in previously working
code when it is modified by adding an assignment statement somewhere in
the body of a function.

  This code:

    >>> x = 10
    >>> def bar():
    ...     print x
    >>> bar()
    10

works, but this code:

    >>> x = 10
    >>> def foo():
    ...     print x
    ...     x += 1

results in an UnboundLocalError:

    >>> foo()
    Traceback (most recent call last):
      ...
    UnboundLocalError: local variable 'x' referenced before assignment

This is because when you make an assignment to a variable in a scope,
that variable becomes local to that scope and shadows any similarly
named variable in the outer scope.  Since the last statement in foo
assigns a new value to `x', the compiler recognizes it as a local
variable.  Consequently when the earlier `print x' attempts to print
the uninitialized local variable and an error results.

  In the example above you can access the outer scope variable by
declaring it global:

    >>> x = 10
    >>> def foobar():
    ...     global x
    ...     print x
    ...     x += 1
    >>> foobar()
    10

This explicit declaration is required in order to remind you that
(unlike the superficially analogous situation with class and instance
variables) you are actually modifying the value of the variable in the
outer scope:

    >>> print x
    11

12.2.2.2 What are the rules for local and global variables in Python?
.....................................................................

In Python, variables that are only referenced inside a function are
implicitly global.  If a variable is assigned a new value anywhere
within the function's body, it's assumed to be a local.  If a variable
is ever assigned a new value inside the function, the variable is
implicitly local, and you need to explicitly declare it as 'global'.

  Though a bit surprising at first, a moment's consideration explains
this.  On one hand, requiring global for assigned variables
provides a bar against unintended side-effects.  On the other hand, if
`global' was required for all global references, you'd be using
`global' all the time.  You'd have to declare as global every reference
to a built-in function or to a component of an imported module.  This
clutter would defeat the usefulness of the `global' declaration for
identifying side-effects.

12.2.2.3 How do I share global variables across modules?
........................................................

The canonical way to share information across modules within a single
program is to create a special module (often called config or cfg).
Just import the config module in all modules of your application; the
module then becomes available as a global name.  Because there is only
one instance of each module, any changes made to the module object get
reflected everywhere.  For example:

  config.py:

    x = 0   # Default value of the 'x' configuration setting

mod.py:

    import config
    config.x = 1

main.py:

    import config
    import mod
    print config.x

Note that using a module is also the basis for implementing the
Singleton design pattern, for the same reason.

12.2.2.4 What are the "best practices" for using import in a module?
....................................................................

In general, don't use `from modulename import *'.  Doing so clutters the
importer's namespace.  Some people avoid this idiom even with the few
modules that were designed to be imported in this manner.  Modules
designed in this manner include Tkinter, and
threading.

  Import modules at the top of a file.  Doing so makes it clear what
other modules your code requires and avoids questions of whether the
module name is in scope.  Using one import per line makes it easy to
add and delete module imports, but using multiple imports per line uses
less screen space.

  It's good practice if you import modules in the following order:

  1. standard library modules - e.g. `sys', `os', `getopt', `re'

  2. third-party library modules (anything installed in Python's
     site-packages directory) - e.g. mx.DateTime, ZODB, PIL.Image, etc.

  3. locally-developed modules

  Never use relative package imports.  If you're writing code that's in
the `package.sub.m1' module and want to import `package.sub.m2', do not
just write `import m2', even though it's legal.  Write `from
package.sub import m2' instead.  Relative imports can lead to a module
being initialized twice, leading to confusing bugs.  See PEP 328
(http://www.python.org/dev/peps/pep-0328) for details.

  It is sometimes necessary to move imports to a function or class to
avoid problems with circular imports.  Gordon McMillan says:

     Circular imports are fine where both modules use the "import
     <module>" form of import.  They fail when the 2nd module wants to
     grab a name out of the first ("from module import name") and the
     import is at the top level.  That's because names in the 1st are
     not yet available, because the first module is busy importing the
     2nd.

  In this case, if the second module is only used in one function, then
the import can easily be moved into that function.  By the time the
import is called, the first module will have finished initializing, and
the second module can do its import.

  It may also be necessary to move imports out of the top level of code
if some of the modules are platform-specific.  In that case, it may not
even be possible to import all of the modules at the top of the file.
In this case, importing the correct modules in the corresponding
platform-specific code is a good option.

  Only move imports into a local scope, such as inside a function
definition, if it's necessary to solve a problem such as avoiding a
circular import or are trying to reduce the initialization time of a
module.  This technique is especially helpful if many of the imports
are unnecessary depending on how the program executes.  You may also
want to move imports into a function if the modules are only ever used
in that function.  Note that loading a module the first time may be
expensive because of the one time initialization of the module, but
loading a module multiple times is virtually free, costing only a
couple of dictionary lookups.  Even if the module name has gone out of
scope, the module is probably available in sys.modules.

  If only instances of a specific class use a module, then it is
reasonable to import the module in the class's `__init__' method and
then assign the module to an instance variable so that the module is
always available (via that instance variable) during the life of the
object.  Note that to delay an import until the class is instantiated,
the import must be inside a method.  Putting the import inside the
class but outside of any method still causes the import to occur when
the module is initialized.

12.2.2.5 How can I pass optional or keyword parameters from one function to another?
....................................................................................

Collect the arguments using the `*' and `**' specifiers in the
function's parameter list; this gives you the positional arguments as a
tuple and the keyword arguments as a dictionary.  You can then pass
these arguments when calling another function by using `*' and `**':

    def f(x, *args, **kwargs):
        ...
        kwargs['width'] = '14.3c'
        ...
        g(x, *args, **kwargs)

In the unlikely case that you care about Python versions older than
2.0, use apply().:

    def f(x, *args, **kwargs):
        ...
        kwargs['width'] = '14.3c'
        ...
        apply(g, (x,)+args, kwargs)

12.2.2.6 How do I write a function with output parameters (call by reference)?
..............................................................................

Remember that arguments are passed by assignment in Python.  Since
assignment just creates references to objects, there's no alias between
an argument name in the caller and callee, and so no call-by-reference
per se.  You can achieve the desired effect in a number of ways.

  1. By returning a tuple of the results:

         def func2(a, b):
             a = 'new-value'        # a and b are local names
             b = b + 1              # assigned to new objects
             return a, b            # return new values

         x, y = 'old-value', 99
         x, y = func2(x, y)
         print x, y                 # output: new-value 100

     This is almost always the clearest solution.

  2. By using global variables.  This isn't thread-safe, and is not
     recommended.

  3. By passing a mutable (changeable in-place) object:

         def func1(a):
             a[0] = 'new-value'     # 'a' references a mutable list
             a[1] = a[1] + 1        # changes a shared object

         args = ['old-value', 99]
         func1(args)
         print args[0], args[1]     # output: new-value 100

  4. By passing in a dictionary that gets mutated:

         def func3(args):
             args['a'] = 'new-value'     # args is a mutable dictionary
             args['b'] = args['b'] + 1   # change it in-place

         args = {'a':' old-value', 'b': 99}
         func3(args)
         print args['a'], args['b']

  5. Or bundle up values in a class instance:

         class callByRef:
             def __init__(self, **args):
                 for (key, value) in args.items():
                     setattr(self, key, value)

         def func4(args):
             args.a = 'new-value'        # args is a mutable callByRef
             args.b = args.b + 1         # change object in-place

         args = callByRef(a='old-value', b=99)
         func4(args)
         print args.a, args.b

     There's almost never a good reason to get this complicated.

  Your best choice is to return a tuple containing the multiple results.

12.2.2.7 How do you make a higher order function in Python?
...........................................................

You have two choices: you can use nested scopes or you can use callable
objects.  For example, suppose you wanted to define `linear(a,b)' which
returns a function `f(x)' that computes the value `a*x+b'.  Using
nested scopes:

    def linear(a, b):
        def result(x):
            return a * x + b
        return result

Or using a callable object:

    class linear:

        def __init__(self, a, b):
            self.a, self.b = a, b

        def __call__(self, x):
            return self.a * x + self.b

In both cases,

    taxes = linear(0.3, 2)

gives a callable object where `taxes(10e6) == 0.3 * 10e6 + 2'.

  The callable object approach has the disadvantage that it is a bit
slower and results in slightly longer code.  However, note that a
collection of callables can share their signature via inheritance:

    class exponential(linear):
        # __init__ inherited
        def __call__(self, x):
            return self.a * (x ** self.b)

Object can encapsulate state for several methods:

    class counter:

        value = 0

        def set(self, x):
            self.value = x

        def up(self):
            self.value = self.value + 1

        def down(self):
            self.value = self.value - 1

    count = counter()
    inc, dec, reset = count.up, count.down, count.set

Here `inc()', `dec()' and `reset()' act like functions which share the
same counting variable.

12.2.2.8 How do I copy an object in Python?
...........................................

In general, try copy.copy() or copy.deepcopy().
for the general case.  Not all objects can be copied, but most can.

  Some objects can be copied more easily.  Dictionaries have a
copy().  method:

    newdict = olddict.copy()

Sequences can be copied by slicing:

    new_l = l[:]

12.2.2.9 How can I find the methods or attributes of an object?
...............................................................

For an instance x of a user-defined class, `dir(x)' returns an
alphabetized list of the names containing the instance attributes and
methods and attributes defined by its class.

12.2.2.10 How can my code discover the name of an object?
.........................................................

Generally speaking, it can't, because objects don't really have names.
Essentially, assignment always binds a name to a value; The same is
true of `def' and `class' statements, but in that case the value is a
callable. Consider the following code:

    class A:
        pass

    B = A

    a = B()
    b = a
    print b
    <__main__.A instance at 0x16D07CC>
    print a
    <__main__.A instance at 0x16D07CC>

Arguably the class has a name: even though it is bound to two names and
invoked through the name B the created instance is still reported as an
instance of class A.  However, it is impossible to say whether the
instance's name is a or b, since both names are bound to the same value.

  Generally speaking it should not be necessary for your code to "know
the names" of particular values. Unless you are deliberately writing
introspective programs, this is usually an indication that a change of
approach might be beneficial.

  In comp.lang.python, Fredrik Lundh once gave an excellent analogy in
answer to this question:

     The same way as you get the name of that cat you found on your
     porch: the cat (object) itself cannot tell you its name, and it
     doesn't really care - so the only way to find out what it's called
     is to ask all your neighbours (namespaces) if it's their cat
     (object)...

     ....and don't be surprised if you'll find that it's known by many
     names, or no name at all!

12.2.2.11 What's up with the comma operator's precedence?
.........................................................

Comma is not an operator in Python.  Consider this session:

    >>> "a" in "b", "a"
    (False, 'a')

Since the comma is not an operator, but a separator between expressions
the above is evaluated as if you had entered:

    >>> ("a" in "b"), "a"

not:

    >>> "a" in ("b", "a")

The same is true of the various assignment operators (`=', `+=' etc).
They are not truly operators but syntactic delimiters in assignment
statements.

12.2.2.12 Is there an equivalent of C's "?:" ternary operator?
..............................................................

Yes, this feature was added in Python 2.5. The syntax would be as
follows:

    [on_true] if [expression] else [on_false]

    x, y = 50, 25

    small = x if x < y else y

For versions previous to 2.5 the answer would be 'No'.

  In many cases you can mimic `a ? b : c' with `a and b or c', but
there's a flaw: if _b_ is zero (or empty, or `None' - anything that
tests false) then _c_ will be selected instead.  In many cases you can
prove by looking at the code that this can't happen (e.g. because _b_
is a constant or has a type that can never be false), but in general
this can be a problem.

  Tim Peters (who wishes it was Steve Majewski) suggested the following
solution: `(a and [b] or [c])[0]'.  Because `[b]' is a singleton list
it is never false, so the wrong path is never taken; then applying
`[0]' to the whole thing gets the _b_ or _c_ that you really wanted.
Ugly, but it gets you there in the rare cases where it is really
inconvenient to rewrite your code using 'if'.

  The best course is usually to write a simple `if...else' statement.
Another solution is to implement the `?:' operator as a function:

    def q(cond, on_true, on_false):
        if cond:
            if not isfunction(on_true):
                return on_true
            else:
                return on_true()
        else:
            if not isfunction(on_false):
                return on_false
            else:
                return on_false()

In most cases you'll pass b and c directly: `q(a, b, c)'.  To avoid
evaluating b or c when they shouldn't be, encapsulate them within a
lambda function, e.g.: `q(a, lambda: b, lambda: c)'.

  It has been asked _why_ Python has no if-then-else expression.  There
are several answers: many languages do just fine without one; it can
easily lead to less readable code; no sufficiently "Pythonic" syntax
has been discovered; a search of the standard library found remarkably
few places where using an if-then-else expression would make the code
more understandable.

  In 2002, PEP 308 (http://www.python.org/dev/peps/pep-0308) was
written proposing several possible syntaxes and the community was asked
to vote on the issue.  The vote was inconclusive.  Most people liked
one of the syntaxes, but also hated other syntaxes; many votes implied
that people preferred no ternary operator rather than having a syntax
they hated.

12.2.2.13 Is it possible to write obfuscated one-liners in Python?
..................................................................

Yes.  Usually this is done by nesting lambda within
lambda.  See the following three examples, due to Ulf Bartelt:

    # Primes < 1000
    print filter(None,map(lambda y:y*reduce(lambda x,y:x*y!=0,
    map(lambda x,y=y:y%x,range(2,int(pow(y,0.5)+1))),1),range(2,1000)))

    # First 10 Fibonacci numbers
    print map(lambda x,f=lambda x,f:(f(x-1,f)+f(x-2,f)) if x>1 else 1: f(x,f),
    range(10))

    # Mandelbrot set
    print (lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,
    Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM,
    Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,
    i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y
    >=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(
    64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy
    ))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24)
    #    \___ ___/  \___ ___/  |   |   |__ lines on screen
    #        V          V      |   |______ columns on screen
    #        |          |      |__________ maximum of "iterations"
    #        |          |_________________ range on y axis
    #        |____________________________ range on x axis

Don't try this at home, kids!

12.2.3 Numbers and strings
--------------------------

12.2.3.1 How do I specify hexadecimal and octal integers?
.........................................................

To specify an octal digit, precede the octal value with a zero, and
then a lower or uppercase "o".  For example, to set the variable "a" to
the octal value "10" (8 in decimal), type:

    >>> a = 0o10
    >>> a
    8

Hexadecimal is just as easy.  Simply precede the hexadecimal number
with a zero, and then a lower or uppercase "x".  Hexadecimal digits can
be specified in lower or uppercase.  For example, in the Python
interpreter:

    >>> a = 0xa5
    >>> a
    165
    >>> b = 0XB2
    >>> b
    178

12.2.3.2 Why does -22 // 10 return -3?
......................................

It's primarily driven by the desire that `i % j' have the same sign as
`j'.  If you want that, and also want:

    i == (i // j) * j + (i % j)

then integer division has to return the floor.  C also requires that
identity to hold, and then compilers that truncate `i // j' need to
make `i % j' have the same sign as `i'.

  There are few real use cases for `i % j' when `j' is negative.  When
`j' is positive, there are many, and in virtually all of them it's more
useful for `i % j' to be `>= 0'.  If the clock says 10 now, what did it
say 200 hours ago?  `-190 % 12 == 2' is useful; `-190 % 12 == -10' is a
bug waiting to bite.

     Note: On Python 2, `a / b' returns the same as `a // b' if
     `__future__.division' is not in effect.  This is also known as
     "classic" division.

12.2.3.3 How do I convert a string to a number?
...............................................

For integers, use the built-in int() type constructor, e.g.
`int('144') == 144'.  Similarly, see float() converts to
floating-point, e.g. `float('144') == 144.0'.

  By default, these interpret the number as decimal, so that
`int('0144') == 144' and `int('0x144')' raises ValueError.
`int(string, base)' takes the base to convert from as a second optional
argument, so `int('0x144', 16) == 324'.  If the base is specified as 0,
the number is interpreted using Python's rules: a leading '0' indicates
octal, and '0x' indicates a hex number.

  Do not use the built-in function eval() if all you need
is to convert strings to numbers.  eval() will be
significantly slower and it presents a security risk: someone could
pass you a Python expression that might have unwanted side effects.
For example, someone could pass `__import__('os').system("rm -rf
$HOME")' which would erase your home directory.

  eval() also has the effect of interpreting numbers as
Python expressions, so that e.g. `eval('09')' gives a syntax error
because Python regards numbers starting with '0' as octal (base 8).

12.2.3.4 How do I convert a number to a string?
...............................................

To convert, e.g., the number 144 to the string '144', use the built-in
type constructor str().  If you want a hexadecimal or octal
representation, use the built-in functions hex() or
oct().  For fancy formatting, see the Format String Syntax
section, e.g. `"{:04d}".format(144)' yields `'0144'' and
`"{:.3f}".format(1/3)' yields `'0.333''.  You may also use the % operator
on strings.  See the library reference manual for
details.

12.2.3.5 How do I modify a string in place?
...........................................

You can't, because strings are immutable.  If you need an object with
this ability, try converting the string to a list or use the array
module:

    >>> s = "Hello, world"
    >>> a = list(s)
    >>> print a
    ['H', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
    >>> a[7:] = list("there!")
    >>> ''.join(a)
    'Hello, there!'

    >>> import array
    >>> a = array.array('c', s)
    >>> print a
    array('c', 'Hello, world')
    >>> a[0] = 'y' ; print a
    array('c', 'yello world')
    >>> a.tostring()
    'yello, world'

12.2.3.6 How do I use strings to call functions/methods?
........................................................

There are various techniques.

   * The best is to use a dictionary that maps strings to functions.
     The primary advantage of this technique is that the strings do not
     need to match the names of the functions.  This is also the
     primary technique used to emulate a case construct:

         def a():
             pass

         def b():
             pass

         dispatch = {'go': a, 'stop': b}  # Note lack of parens for funcs

         dispatch[get_input()]()  # Note trailing parens to call function

   * Use the built-in function getattr().:

         import foo
         getattr(foo, 'bar')()

     Note that getattr() works on any object, including
     classes, class instances, modules, and so on.

     This is used in several places in the standard library, like this:

         class Foo:
             def do_foo(self):
                 ...

             def do_bar(self):
                 ...

         f = getattr(foo_instance, 'do_' + opname)
         f()

   * Use locals() or eval() to resolve the
     function name:

         def myFunc():
             print "hello"

         fname = "myFunc"

         f = locals()[fname]
         f()

         f = eval(fname)
         f()

     Note: Using eval() is slow and dangerous.  If you
     don't have absolute control over the contents of the string,
     someone could pass a string that resulted in an arbitrary function
     being executed.

12.2.3.7 Is there an equivalent to Perl's chomp() for removing trailing newlines from strings?
..............................................................................................

Starting with Python 2.2, you can use `S.rstrip("\r\n")' to remove all
occurrences of any line terminator from the end of the string `S'
without removing other trailing whitespace.  If the string `S'
represents more than one line, with several empty lines at the end, the
line terminators for all the blank lines will be removed:

    >>> lines = ("line 1 \r\n"
    ...          "\r\n"
    ...          "\r\n")
    >>> lines.rstrip("\n\r")
    'line 1 '

Since this is typically only desired when reading text one line at a
time, using `S.rstrip()' this way works well.

  For older versions of Python, there are two partial substitutes:

   - If you want to remove all trailing whitespace, use the `rstrip()'
     method of string objects.  This removes all trailing whitespace,
     not just a single newline.

   - Otherwise, if there is only one line in the string `S', use
     `S.splitlines()[0]'.

12.2.3.8 Is there a scanf() or sscanf() equivalent?
...................................................

Not as such.

  For simple input parsing, the easiest approach is usually to split
the line into whitespace-delimited words using the split().
method of string objects and then convert decimal strings to numeric
values using int() or float().  `split()'
supports an optional "sep" parameter which is useful if the line uses
something other than whitespace as a separator.

  For more complicated input parsing, regular expressions are more
powerful than C's `sscanf()' and better suited for the task.

12.2.3.9 What does 'UnicodeError: ASCII [decoding,encoding] error: ordinal not in range(128)' mean?
...................................................................................................

This error indicates that your Python installation can handle only
7-bit ASCII strings.  There are a couple ways to fix or work around the
problem.

  If your programs must handle data in arbitrary character set
encodings, the environment the application runs in will generally
identify the encoding of the data it is handing you.  You need to
convert the input to Unicode data using that encoding.  For example, a
program that handles email or web input will typically find character
set encoding information in Content-Type headers.  This can then be
used to properly convert input data to Unicode. Assuming the string
referred to by `value' is encoded as UTF-8:

    value = unicode(value, "utf-8")

will return a Unicode object.  If the data is not correctly encoded as
UTF-8, the above call will raise a UnicodeError exception.

  If you only want strings converted to Unicode which have non-ASCII
data, you can try converting them first assuming an ASCII encoding, and
then generate Unicode objects if that fails:

    try:
        x = unicode(value, "ascii")
    except UnicodeError:
        value = unicode(value, "utf-8")
    else:
        # value was valid ASCII data
        pass

It's possible to set a default encoding in a file called
`sitecustomize.py' that's part of the Python library.  However, this
isn't recommended because changing the Python-wide default encoding may
cause third-party extension modules to fail.

  Note that on Windows, there is an encoding known as "mbcs", which
uses an encoding specific to your current locale.  In many cases, and
particularly when working with COM, this may be an appropriate default
encoding to use.

12.2.4 Sequences (Tuples/Lists)
-------------------------------

12.2.4.1 How do I convert between tuples and lists?
...................................................

The type constructor `tuple(seq)' converts any sequence (actually, any
iterable) into a tuple with the same items in the same order.

  For example, `tuple([1, 2, 3])' yields `(1, 2, 3)' and `tuple('abc')'
yields `('a', 'b', 'c')'.  If the argument is a tuple, it does not make
a copy but returns the same object, so it is cheap to call
tuple() when you aren't sure that an object is already a tuple.

  The type constructor `list(seq)' converts any sequence or iterable
into a list with the same items in the same order.  For example,
`list((1, 2, 3))' yields `[1, 2, 3]' and `list('abc')' yields `['a',
'b', 'c']'.  If the argument is a list, it makes a copy just like
`seq[:]' would.

12.2.4.2 What's a negative index?
.................................

Python sequences are indexed with positive numbers and negative
numbers.  For positive numbers 0 is the first index 1 is the second
index and so forth.  For negative indices -1 is the last index and -2
is the penultimate (next to last) index and so forth.  Think of
`seq[-n]' as the same as `seq[len(seq)-n]'.

  Using negative indices can be very convenient.  For example `S[:-1]'
is all of the string except for its last character, which is useful for
removing the trailing newline from a string.

12.2.4.3 How do I iterate over a sequence in reverse order?
...........................................................

Use the reversed() built-in function, which is new in
Python 2.4:

    for x in reversed(sequence):
        ... # do something with x...

This won't touch your original sequence, but build a new copy with
reversed order to iterate over.

  With Python 2.3, you can use an extended slice syntax:

    for x in sequence[::-1]:
        ... # do something with x...

12.2.4.4 How do you remove duplicates from a list?
..................................................

See the Python Cookbook for a long discussion of many ways to do this:

     <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560>

  If you don't mind reordering the list, sort it and then scan from the
end of the list, deleting duplicates as you go:

    if mylist:
        mylist.sort()
        last = mylist[-1]
        for i in range(len(mylist)-2, -1, -1):
            if last == mylist[i]:
                del mylist[i]
            else:
                last = mylist[i]

If all elements of the list may be used as dictionary keys (i.e. they
are all hashable) this is often faster

    d = {}
    for x in mylist:
        d[x] = 1
    mylist = list(d.keys())

In Python 2.5 and later, the following is possible instead:

    mylist = list(set(mylist))

This converts the list into a set, thereby removing duplicates, and
then back into a list.

12.2.4.5 How do you make an array in Python?
............................................

Use a list:

    ["this", 1, "is", "an", "array"]

Lists are equivalent to C or Pascal arrays in their time complexity;
the primary difference is that a Python list can contain objects of
many different types.

  The `array' module also provides methods for creating arrays of fixed
types with compact representations, but they are slower to index than
lists.  Also note that the Numeric extensions and others define
array-like structures with various characteristics as well.

  To get Lisp-style linked lists, you can emulate cons cells using
tuples:

    lisp_list = ("like",  ("this",  ("example", None) ) )

If mutability is desired, you could use lists instead of tuples.  Here
the analogue of lisp car is `lisp_list[0]' and the analogue of cdr is
`lisp_list[1]'.  Only do this if you're sure you really need to,
because it's usually a lot slower than using Python lists.

12.2.4.6 How do I create a multidimensional list?
.................................................

You probably tried to make a multidimensional array like this:

    A = [[None] * 2] * 3

This looks correct if you print it:

    >>> A
    [[None, None], [None, None], [None, None]]

But when you assign a value, it shows up in multiple places:

    >>> A[0][0] = 5
    >>> A
    [[5, None], [5, None], [5, None]]

The reason is that replicating a list with `*' doesn't create copies,
it only creates references to the existing objects.  The `*3' creates a
list containing 3 references to the same list of length two.  Changes
to one row will show in all rows, which is almost certainly not what
you want.

  The suggested approach is to create a list of the desired length
first and then fill in each element with a newly created list:

    A = [None] * 3
    for i in range(3):
        A[i] = [None] * 2

This generates a list containing 3 different lists of length two.  You
can also use a list comprehension:

    w, h = 2, 3
    A = [[None] * w for i in range(h)]

Or, you can use an extension that provides a matrix datatype; Numeric
Python (http://numpy.scipy.org/)is the best known.

12.2.4.7 How do I apply a method to a sequence of objects?
..........................................................

Use a list comprehension:

    result = [obj.method() for obj in mylist]

More generically, you can try the following function:

    def method_map(objects, method, arguments):
        """method_map([a,b], "meth", (1,2)) gives [a.meth(1,2), b.meth(1,2)]"""
        nobjects = len(objects)
        methods = map(getattr, objects, [method]*nobjects)
        return map(apply, methods, [arguments]*nobjects)

12.2.5 Dictionaries
-------------------

12.2.5.1 How can I get a dictionary to display its keys in a consistent order?
..............................................................................

You can't.  Dictionaries store their keys in an unpredictable order, so
the display order of a dictionary's elements will be similarly
unpredictable.

  This can be frustrating if you want to save a printable version to a
file, make some changes and then compare it with some other printed
dictionary.  In this case, use the `pprint' module to pretty-print the
dictionary; the items will be presented in order sorted by the key.

  A more complicated solution is to subclass `dict' to create a
`SortedDict' class that prints itself in a predictable order.  Here's
one simpleminded implementation of such a class:

    class SortedDict(dict):
        def __repr__(self):
            keys = sorted(self.keys())
            result = ("{!r}: {!r}".format(k, self[k]) for k in keys)
            return "{{{}}}".format(", ".join(result))

        __str__ = __repr__

This will work for many common situations you might encounter, though
it's far from a perfect solution. The largest flaw is that if some
values in the dictionary are also dictionaries, their values won't be
presented in any particular order.

12.2.5.2 I want to do a complicated sort: can you do a Schwartzian Transform in Python?
.......................................................................................

The technique, attributed to Randal Schwartz of the Perl community,
sorts the elements of a list by a metric which maps each element to its
"sort value". In Python, just use the `key' argument for the `sort()'
method:

    Isorted = L[:]
    Isorted.sort(key=lambda s: int(s[10:15]))

The `key' argument is new in Python 2.4, for older versions this kind of
sorting is quite simple to do with list comprehensions.  To sort a list
of strings by their uppercase values:

    tmp1 = [(x.upper(), x) for x in L]  # Schwartzian transform
    tmp1.sort()
    Usorted = [x[1] for x in tmp1]

To sort by the integer value of a subfield extending from positions
10-15 in each string:

    tmp2 = [(int(s[10:15]), s) for s in L]  # Schwartzian transform
    tmp2.sort()
    Isorted = [x[1] for x in tmp2]

Note that Isorted may also be computed by

    def intfield(s):
        return int(s[10:15])

    def Icmp(s1, s2):
        return cmp(intfield(s1), intfield(s2))

    Isorted = L[:]
    Isorted.sort(Icmp)

but since this method calls `intfield()' many times for each element of
L, it is slower than the Schwartzian Transform.

12.2.5.3 How can I sort one list by values from another list?
.............................................................

Merge them into a single list of tuples, sort the resulting list, and
then pick out the element you want.

    >>> list1 = ["what", "I'm", "sorting", "by"]
    >>> list2 = ["something", "else", "to", "sort"]
    >>> pairs = zip(list1, list2)
    >>> pairs
    [('what', 'something'), ("I'm", 'else'), ('sorting', 'to'), ('by', 'sort')]
    >>> pairs.sort()
    >>> result = [ x[1] for x in pairs ]
    >>> result
    ['else', 'sort', 'to', 'something']

An alternative for the last step is:

    >>> result = []
    >>> for p in pairs: result.append(p[1])

If you find this more legible, you might prefer to use this instead of
the final list comprehension.  However, it is almost twice as slow for
long lists.  Why?  First, the `append()' operation has to reallocate
memory, and while it uses some tricks to avoid doing that each time, it
still has to do it occasionally, and that costs quite a bit.  Second,
the expression "result.append" requires an extra attribute lookup, and
third, there's a speed reduction from having to make all those function
calls.

12.2.6 Objects
--------------

12.2.6.1 What is a class?
.........................

A class is the particular object type created by executing a class
statement.  Class objects are used as templates to create instance
objects, which embody both the data (attributes) and code (methods)
specific to a datatype.

  A class can be based on one or more other classes, called its base
class(es). It then inherits the attributes and methods of its base
classes. This allows an object model to be successively refined by
inheritance.  You might have a generic `Mailbox' class that provides
basic accessor methods for a mailbox, and subclasses such as
`MboxMailbox', `MaildirMailbox', `OutlookMailbox' that handle various
specific mailbox formats.

12.2.6.2 What is a method?
..........................

A method is a function on some object `x' that you normally call as
`x.name(arguments...)'.  Methods are defined as functions inside the
class definition:

    class C:
        def meth (self, arg):
            return arg * 2 + self.attribute

12.2.6.3 What is self?
......................

Self is merely a conventional name for the first argument of a method.
A method defined as `meth(self, a, b, c)' should be called as
`x.meth(a, b, c)' for some instance `x' of the class in which the
definition occurs; the called method will think it is called as
`meth(x, a, b, c)'.

  See also Why must 'self' be used explicitly in method definitions and calls?.

12.2.6.4 How do I check if an object is an instance of a given class or of a subclass of it?
............................................................................................

Use the built-in function `isinstance(obj, cls)'.  You can check if an
object is an instance of any of a number of classes by providing a
tuple instead of a single class, e.g. `isinstance(obj, (class1, class2,
...))', and can also check whether an object is one of Python's
built-in types, e.g.  `isinstance(obj, str)' or `isinstance(obj, (int,
long, float, complex))'.

  Note that most programs do not use isinstance() on
user-defined classes very often.  If you are developing the classes
yourself, a more proper object-oriented style is to define methods on
the classes that encapsulate a particular behaviour, instead of
checking the object's class and doing a different thing based on what
class it is.  For example, if you have a function that does something:

    def search(obj):
        if isinstance(obj, Mailbox):
            # ... code to search a mailbox
        elif isinstance(obj, Document):
            # ... code to search a document
        elif ...

A better approach is to define a `search()' method on all the classes
and just call it:

    class Mailbox:
        def search(self):
            # ... code to search a mailbox

    class Document:
        def search(self):
            # ... code to search a document

    obj.search()

12.2.6.5 What is delegation?
............................

Delegation is an object oriented technique (also called a design
pattern).  Let's say you have an object `x' and want to change the
behaviour of just one of its methods.  You can create a new class that
provides a new implementation of the method you're interested in
changing and delegates all other methods to the corresponding method of
`x'.

  Python programmers can easily implement delegation.  For example, the
following class implements a class that behaves like a file but
converts all written data to uppercase:

    class UpperOut:

        def __init__(self, outfile):
            self._outfile = outfile

        def write(self, s):
            self._outfile.write(s.upper())

        def __getattr__(self, name):
            return getattr(self._outfile, name)

Here the `UpperOut' class redefines the `write()' method to convert the
argument string to uppercase before calling the underlying
`self.__outfile.write()' method.  All other methods are delegated to the
underlying `self.__outfile' object.  The delegation is accomplished via
the `__getattr__' method; consult the language reference.
for more information about controlling attribute access.

  Note that for more general cases delegation can get trickier. When
attributes must be set as well as retrieved, the class must define a
__setattr__().  method too, and it must do so carefully.
The basic implementation of __setattr__() is roughly
equivalent to the following:

    class X:
        ...
        def __setattr__(self, name, value):
            self.__dict__[name] = value
        ...

Most __setattr__() implementations must modify
`self.__dict__' to store local state for self without causing an
infinite recursion.

12.2.6.6 How do I call a method defined in a base class from a derived class that overrides it?
...............................................................................................

If you're using new-style classes, use the built-in super().
function:

    class Derived(Base):
        def meth (self):
            super(Derived, self).meth()

If you're using classic classes: For a class definition such as `class
Derived(Base): ...' you can call method `meth()' defined in `Base' (or
one of `Base''s base classes) as `Base.meth(self, arguments...)'.  Here,
`Base.meth' is an unbound method, so you need to provide the `self'
argument.

12.2.6.7 How can I organize my code to make it easier to change the base class?
...............................................................................

You could define an alias for the base class, assign the real base
class to it before your class definition, and use the alias throughout
your class.  Then all you have to change is the value assigned to the
alias.  Incidentally, this trick is also handy if you want to decide
dynamically (e.g. depending on availability of resources) which base
class to use.  Example:

    BaseAlias = <real base class>

    class Derived(BaseAlias):
        def meth(self):
            BaseAlias.meth(self)
            ...

12.2.6.8 How do I create static class data and static class methods?
....................................................................

Both static data and static methods (in the sense of C++ or Java) are
supported in Python.

  For static data, simply define a class attribute.  To assign a new
value to the attribute, you have to explicitly use the class name in
the assignment:

    class C:
        count = 0   # number of times C.__init__ called

        def __init__(self):
            C.count = C.count + 1

        def getcount(self):
            return C.count  # or return self.count

`c.count' also refers to `C.count' for any `c' such that `isinstance(c,
C)' holds, unless overridden by `c' itself or by some class on the
base-class search path from `c.__class__' back to `C'.

  Caution: within a method of C, an assignment like `self.count = 42'
creates a new and unrelated instance named "count" in `self''s own
dict.  Rebinding of a class-static data name must always specify the
class whether inside a method or not:

    C.count = 314

Static methods are possible since Python 2.2:

    class C:
        def static(arg1, arg2, arg3):
            # No 'self' parameter!
            ...
        static = staticmethod(static)

With Python 2.4's decorators, this can also be written as

    class C:
        @staticmethod
        def static(arg1, arg2, arg3):
            # No 'self' parameter!
            ...

However, a far more straightforward way to get the effect of a static
method is via a simple module-level function:

    def getcount():
        return C.count

If your code is structured so as to define one class (or tightly
related class hierarchy) per module, this supplies the desired
encapsulation.

12.2.6.9 How can I overload constructors (or methods) in Python?
................................................................

This answer actually applies to all methods, but the question usually
comes up first in the context of constructors.

  In C++ you'd write

    class C {
        C() { cout << "No arguments\n"; }
        C(int i) { cout << "Argument is " << i << "\n"; }
    }

In Python you have to write a single constructor that catches all cases
using default arguments.  For example:

    class C:
        def __init__(self, i=None):
            if i is None:
                print "No arguments"
            else:
                print "Argument is", i

This is not entirely equivalent, but close enough in practice.

  You could also try a variable-length argument list, e.g.

    def __init__(self, *args):
        ...

The same approach works for all method definitions.

12.2.6.10 I try to use __spam and I get an error about _SomeClassName__spam.
............................................................................

Variable names with double leading underscores are "mangled" to provide
a simple but effective way to define class private variables.  Any
identifier of the form `__spam' (at least two leading underscores, at
most one trailing underscore) is textually replaced with
`_classname__spam', where `classname' is the current class name with
any leading underscores stripped.

  This doesn't guarantee privacy: an outside user can still
deliberately access the "_classname__spam" attribute, and private
values are visible in the object's `__dict__'.  Many Python programmers
never bother to use private variable names at all.

12.2.6.11 My class defines __del__ but it is not called when I delete the object.
.................................................................................

There are several possible reasons for this.

  The del statement does not necessarily call __del__(). -
it simply decrements the object's reference count, and if this reaches
zero __del__() is called.

  If your data structures contain circular links (e.g. a tree where
each child has a parent reference and each parent has a list of
children) the reference counts will never go back to zero.  Once in a
while Python runs an algorithm to detect such cycles, but the garbage
collector might run some time after the last reference to your data
structure vanishes, so your __del__() method may be called
at an inconvenient and random time. This is inconvenient if you're
trying to reproduce a problem. Worse, the order in which object's
__del__().  methods are executed is arbitrary.  You can run
gc.collect() to force a collection, but there _are_ pathological
cases where objects will never be collected.

  Despite the cycle collector, it's still a good idea to define an
explicit `close()' method on objects to be called whenever you're done
with them.  The `close()' method can then remove attributes that refer
to subobjecs.  Don't call __del__() directly - see
__del__() should call `close()' and `close()' should make sure
that it can be called more than once for the same object.

  Another way to avoid cyclical references is to use the weakref
module, which allows you to point to objects without incrementing
their reference count.  Tree data structures, for instance, should use
weak references for their parent and sibling references (if they need
them!).

  If the object has ever been a local variable in a function that
caught an expression in an except clause, chances are that a reference
to the object still exists in that function's stack frame as contained
in the stack trace.  Normally, calling sys.exc_clear() will
take care of this by clearing the last recorded exception.

  Finally, if your __del__() method raises an exception, a
warning message is printed to sys.stderr.

12.2.6.12 How do I get a list of all instances of a given class?
................................................................

Python does not keep track of all instances of a class (or of a
built-in type).  You can program the class's constructor to keep track
of all instances by keeping a list of weak references to each instance.

12.2.7 Modules
--------------

12.2.7.1 How do I create a .pyc file?
.....................................

When a module is imported for the first time (or when the source is
more recent than the current compiled file) a `.pyc' file containing
the compiled code should be created in the same directory as the `.py'
file.

  One reason that a `.pyc' file may not be created is permissions
problems with the directory. This can happen, for example, if you
develop as one user but run as another, such as if you are testing with
a web server.  Creation of a .pyc file is automatic if you're importing
a module and Python has the ability (permissions, free space, etc...)
to write the compiled module back to the directory.

  Running Python on a top level script is not considered an import and
no `.pyc' will be created.  For example, if you have a top-level module
`abc.py' that imports another module `xyz.py', when you run abc,
`xyz.pyc' will be created since xyz is imported, but no `abc.pyc' file
will be created since `abc.py' isn't being imported.

  If you need to create abc.pyc - that is, to create a .pyc file for a
module that is not imported - you can, using the py_compile.
and compileall modules.

  The py_compile module can manually compile any module.
One way is to use the `compile()' function in that module interactively:

    >>> import py_compile
    >>> py_compile.compile('abc.py')

This will write the `.pyc' to the same location as `abc.py' (or you can
override that with the optional parameter `cfile').

  You can also automatically compile all files in a directory or
directories using the compileall module.  You can do it from
the shell prompt by running `compileall.py' and providing the path of a
directory containing Python files to compile:

    python -m compileall .

12.2.7.2 How do I find the current module name?
...............................................

A module can find out its own module name by looking at the predefined
global variable `__name__'.  If this has the value `'__main__'', the
program is running as a script.  Many modules that are usually used by
importing them also provide a command-line interface or a self-test,
and only execute this code after checking `__name__':

    def main():
        print 'Running test...'
        ...

    if __name__ == '__main__':
        main()

12.2.7.3 How can I have modules that mutually import each other?
................................................................

Suppose you have the following modules:

  foo.py:

    from bar import bar_var
    foo_var = 1

bar.py:

    from foo import foo_var
    bar_var = 2

The problem is that the interpreter will perform the following steps:

   * main imports foo

   * Empty globals for foo are created

   * foo is compiled and starts executing

   * foo imports bar

   * Empty globals for bar are created

   * bar is compiled and starts executing

   * bar imports foo (which is a no-op since there already is a module
     named foo)

   * bar.foo_var = foo.foo_var

  The last step fails, because Python isn't done with interpreting
`foo' yet and the global symbol dictionary for `foo' is still empty.

  The same thing happens when you use `import foo', and then try to
access `foo.foo_var' in global code.

  There are (at least) three possible workarounds for this problem.

  Guido van Rossum recommends avoiding all uses of `from <module>
import ...', and placing all code inside functions.  Initializations of
global variables and class variables should use constants or built-in
functions only.  This means everything from an imported module is
referenced as `<module>.<name>'.

  Jim Roskind suggests performing steps in the following order in each
module:

   * exports (globals, functions, and classes that don't need imported
     base classes)

   * `import' statements

   * active code (including globals that are initialized from imported
     values).

  van Rossum doesn't like this approach much because the imports appear
in a strange place, but it does work.

  Matthias Urlichs recommends restructuring your code so that the
recursive import is not necessary in the first place.

  These solutions are not mutually exclusive.

12.2.7.4 __import__('x.y.z') returns <module 'x'>; how do I get z?
..................................................................

Try:

    __import__('x.y.z').y.z

For more realistic situations, you may have to do something like

    m = __import__(s)
    for i in s.split(".")[1:]:
        m = getattr(m, i)

See importlib for a convenience function called
import_module().

12.2.7.5 When I edit an imported module and reimport it, the changes don't show up.  Why does this happen?
..........................................................................................................

For reasons of efficiency as well as consistency, Python only reads the
module file on the first time a module is imported.  If it didn't, in a
program consisting of many modules where each one imports the same
basic module, the basic module would be parsed and re-parsed many
times.  To force rereading of a changed module, do this:

    import modname
    reload(modname)

Warning: this technique is not 100% fool-proof.  In particular, modules
containing statements like

    from modname import some_objects

will continue to work with the old version of the imported objects.  If
the module contains class definitions, existing class instances will
_not_ be updated to use the new class definition.  This can result in
the following paradoxical behaviour:

    >>> import cls
    >>> c = cls.C()                # Create an instance of C
    >>> reload(cls)
    <module 'cls' from 'cls.pyc'>
    >>> isinstance(c, cls.C)       # isinstance is false?!?
    False

The nature of the problem is made clear if you print out the class
objects:

    >>> c.__class__
    <class cls.C at 0x7352a0>
    >>> cls.C
    <class cls.C at 0x4198d0>

12.3 Design and History FAQ
===========================

12.3.1 Why does Python use indentation for grouping of statements?
------------------------------------------------------------------

Guido van Rossum believes that using indentation for grouping is
extremely elegant and contributes a lot to the clarity of the average
Python program.  Most people learn to love this feature after a while.

  Since there are no begin/end brackets there cannot be a disagreement
between grouping perceived by the parser and the human reader.
Occasionally C programmers will encounter a fragment of code like this:

    if (x <= y)
            x++;
            y--;
    z++;

Only the `x++' statement is executed if the condition is true, but the
indentation leads you to believe otherwise.  Even experienced C
programmers will sometimes stare at it a long time wondering why `y' is
being decremented even for `x > y'.

  Because there are no begin/end brackets, Python is much less prone to
coding-style conflicts.  In C there are many different ways to place
the braces.  If you're used to reading and writing code that uses one
style, you will feel at least slightly uneasy when reading (or being
required to write) another style.

  Many coding styles place begin/end brackets on a line by themselves.
This makes programs considerably longer and wastes valuable screen
space, making it harder to get a good overview of a program.  Ideally,
a function should fit on one screen (say, 20-30 lines).  20 lines of
Python can do a lot more work than 20 lines of C.  This is not solely
due to the lack of begin/end brackets - the lack of declarations and
the high-level data types are also responsible - but the
indentation-based syntax certainly helps.

12.3.2 Why am I getting strange results with simple arithmetic operations?
--------------------------------------------------------------------------

See the next question.

12.3.3 Why are floating point calculations so inaccurate?
---------------------------------------------------------

People are often very surprised by results like this:

    >>> 1.2 - 1.0
    0.199999999999999996

and think it is a bug in Python. It's not.  This has nothing to do with
Python, but with how the underlying C platform handles floating point
numbers, and ultimately with the inaccuracies introduced when writing
down numbers as a string of a fixed number of digits.

  The internal representation of floating point numbers uses a fixed
number of binary digits to represent a decimal number.  Some decimal
numbers can't be represented exactly in binary, resulting in small
roundoff errors.

  In decimal math, there are many numbers that can't be represented
with a fixed number of decimal digits, e.g.  1/3 = 0.3333333333.......

  In base 2, 1/2 = 0.1, 1/4 = 0.01, 1/8 = 0.001, etc.  .2 equals 2/10
equals 1/5, resulting in the binary fractional number
0.001100110011001...

  Floating point numbers only have 32 or 64 bits of precision, so the
digits are cut off at some point, and the resulting number is
0.199999999999999996 in decimal, not 0.2.

  A floating point number's `repr()' function prints as many digits are
necessary to make `eval(repr(f)) == f' true for any float f.  The
`str()' function prints fewer digits and this often results in the more
sensible number that was probably intended:

    >>> 1.1 - 0.9
    0.20000000000000007
    >>> print 1.1 - 0.9
    0.2

One of the consequences of this is that it is error-prone to compare
the result of some computation to a float with `=='. Tiny inaccuracies
may mean that `==' fails.  Instead, you have to check that the
difference between the two numbers is less than a certain threshold:

    epsilon = 0.0000000000001  # Tiny allowed error
    expected_result = 0.4

    if expected_result-epsilon <= computation() <= expected_result+epsilon:
        ...

Please see the chapter on floating point arithmetic in the
Python tutorial for more information.

12.3.4 Why are Python strings immutable?
----------------------------------------

There are several advantages.

  One is performance: knowing that a string is immutable means we can
allocate space for it at creation time, and the storage requirements
are fixed and unchanging.  This is also one of the reasons for the
distinction between tuples and lists.

  Another advantage is that strings in Python are considered as
"elemental" as numbers.  No amount of activity will change the value 8
to anything else, and in Python, no amount of activity will change the
string "eight" to anything else.

12.3.5 Why must 'self' be used explicitly in method definitions and calls?
--------------------------------------------------------------------------

The idea was borrowed from Modula-3.  It turns out to be very useful,
for a variety of reasons.

  First, it's more obvious that you are using a method or instance
attribute instead of a local variable.  Reading `self.x' or
`self.meth()' makes it absolutely clear that an instance variable or
method is used even if you don't know the class definition by heart.
In C++, you can sort of tell by the lack of a local variable
declaration (assuming globals are rare or easily recognizable) - but in
Python, there are no local variable declarations, so you'd have to look
up the class definition to be sure.  Some C++ and Java coding standards
call for instance attributes to have an `m_' prefix, so this
explicitness is still useful in those languages, too.

  Second, it means that no special syntax is necessary if you want to
explicitly reference or call the method from a particular class.  In
C++, if you want to use a method from a base class which is overridden
in a derived class, you have to use the `::' operator - in Python you
can write `baseclass.methodname(self, <argument list>)'.  This is
particularly useful for __init__() methods, and in general
in cases where a derived class method wants to extend the base class
method of the same name and thus has to call the base class method
somehow.

  Finally, for instance variables it solves a syntactic problem with
assignment: since local variables in Python are (by definition!) those
variables to which a value is assigned in a function body (and that
aren't explicitly declared global), there has to be some way to tell
the interpreter that an assignment was meant to assign to an instance
variable instead of to a local variable, and it should preferably be
syntactic (for efficiency reasons).  C++ does this through
declarations, but Python doesn't have declarations and it would be a
pity having to introduce them just for this purpose.  Using the
explicit `self.var' solves this nicely.  Similarly, for using instance
variables, having to write `self.var' means that references to
unqualified names inside a method don't have to search the instance's
directories.  To put it another way, local variables and instance
variables live in two different namespaces, and you need to tell Python
which namespace to use.

12.3.6 Why can't I use an assignment in an expression?
------------------------------------------------------

Many people used to C or Perl complain that they want to use this C
idiom:

    while (line = readline(f)) {
        // do something with line
    }

where in Python you're forced to write this:

    while True:
        line = f.readline()
        if not line:
            break
        ... # do something with line

The reason for not allowing assignment in Python expressions is a
common, hard-to-find bug in those other languages, caused by this
construct:

    if (x = 0) {
        // error handling
    }
    else {
        // code that only works for nonzero x
    }

The error is a simple typo: `x = 0', which assigns 0 to the variable
`x', was written while the comparison `x == 0' is certainly what was
intended.

  Many alternatives have been proposed.  Most are hacks that save some
typing but use arbitrary or cryptic syntax or keywords, and fail the
simple criterion for language change proposals: it should intuitively
suggest the proper meaning to a human reader who has not yet been
introduced to the construct.

  An interesting phenomenon is that most experienced Python programmers
recognize the `while True' idiom and don't seem to be missing the
assignment in expression construct much; it's only newcomers who
express a strong desire to add this to the language.

  There's an alternative way of spelling this that seems attractive but
is generally less robust than the "while True" solution:

    line = f.readline()
    while line:
        ... # do something with line...
        line = f.readline()

The problem with this is that if you change your mind about exactly how
you get the next line (e.g. you want to change it into
`sys.stdin.readline()') you have to remember to change two places in
your program - the second occurrence is hidden at the bottom of the
loop.

  The best approach is to use iterators, making it possible to loop
through objects using the `for' statement.  For example, in the current
version of Python file objects support the iterator protocol, so you
can now write simply:

    for line in f:
        ... # do something with line...

12.3.7 Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?
-----------------------------------------------------------------------------------------------------------------------

The major reason is history. Functions were used for those operations
that were generic for a group of types and which were intended to work
even for objects that didn't have methods at all (e.g. tuples).  It is
also convenient to have a function that can readily be applied to an
amorphous collection of objects when you use the functional features of
Python (`map()', `apply()' et al).

  In fact, implementing `len()', `max()', `min()' as a built-in
function is actually less code than implementing them as methods for
each type.  One can quibble about individual cases but it's a part of
Python, and it's too late to make such fundamental changes now. The
functions have to remain to avoid massive code breakage.

     Note: For string operations, Python has moved from external
     functions (the `string' module) to methods.  However, `len()' is
     still a function.

12.3.8 Why is join() a string method instead of a list or tuple method?
-----------------------------------------------------------------------

Strings became much more like other standard types starting in Python
1.6, when methods were added which give the same functionality that has
always been available using the functions of the string module.  Most
of these new methods have been widely accepted, but the one which
appears to make some programmers feel uncomfortable is:

    ", ".join(['1', '2', '4', '8', '16'])

which gives the result:

    "1, 2, 4, 8, 16"

There are two common arguments against this usage.

  The first runs along the lines of: "It looks really ugly using a
method of a string literal (string constant)", to which the answer is
that it might, but a string literal is just a fixed value. If the
methods are to be allowed on names bound to strings there is no logical
reason to make them unavailable on literals.

  The second objection is typically cast as: "I am really telling a
sequence to join its members together with a string constant".  Sadly,
you aren't.  For some reason there seems to be much less difficulty
with having split() as a string method, since in that case
it is easy to see that

    "1, 2, 4, 8, 16".split(", ")

is an instruction to a string literal to return the substrings
delimited by the given separator (or, by default, arbitrary runs of
white space).  In this case a Unicode string returns a list of Unicode
strings, an ASCII string returns a list of ASCII strings, and everyone
is happy.

  join() is a string method because in using it you are
telling the separator string to iterate over a sequence of strings and
insert itself between adjacent elements.  This method can be used with
any argument which obeys the rules for sequence objects, including any
new classes you might define yourself.

  Because this is a string method it can work for Unicode strings as
well as plain ASCII strings.  If `join()' were a method of the sequence
types then the sequence types would have to decide which type of string
to return depending on the type of the separator.

  If none of these arguments persuade you, then for the moment you can
continue to use the `join()' function from the string module, which
allows you to write

    string.join(['1', '2', '4', '8', '16'], ", ")

12.3.9 How fast are exceptions?
-------------------------------

A try/except block is extremely efficient.  Actually catching an
exception is expensive.  In versions of Python prior to 2.0 it was
common to use this idiom:

    try:
        value = mydict[key]
    except KeyError:
        mydict[key] = getvalue(key)
        value = mydict[key]

This only made sense when you expected the dict to have the key almost
all the time.  If that wasn't the case, you coded it like this:

    if mydict.has_key(key):
        value = mydict[key]
    else:
        mydict[key] = getvalue(key)
        value = mydict[key]

     Note: In Python 2.0 and higher, you can code this as `value =
     mydict.setdefault(key, getvalue(key))'.

12.3.10 Why isn't there a switch or case statement in Python?
-------------------------------------------------------------

You can do this easily enough with a sequence of `if... elif... elif...
else'.  There have been some proposals for switch statement syntax, but
there is no consensus (yet) on whether and how to do range tests.  See PEP
275 (http://www.python.org/dev/peps/pep-0275) for complete details and
the current status.

  For cases where you need to choose from a very large number of
possibilities, you can create a dictionary mapping case values to
functions to call.  For example:

    def function_1(...):
        ...

    functions = {'a': function_1,
                 'b': function_2,
                 'c': self.method_1, ...}

    func = functions[value]
    func()

For calling methods on objects, you can simplify yet further by using
the getattr() built-in to retrieve methods with a
particular name:

    def visit_a(self, ...):
        ...
    ...

    def dispatch(self, value):
        method_name = 'visit_' + str(value)
        method = getattr(self, method_name)
        method()

It's suggested that you use a prefix for the method names, such as
`visit_' in this example.  Without such a prefix, if values are coming
from an untrusted source, an attacker would be able to call any method
on your object.

12.3.11 Can't you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?
----------------------------------------------------------------------------------------------------------------

Answer 1: Unfortunately, the interpreter pushes at least one C stack
frame for each Python stack frame.  Also, extensions can call back into
Python at almost random moments.  Therefore, a complete threads
implementation requires thread support for C.

  Answer 2: Fortunately, there is Stackless Python
(http://www.stackless.com), which has a completely redesigned
interpreter loop that avoids the C stack.  It's still experimental but
looks very promising.  Although it is binary compatible with standard
Python, it's still unclear whether Stackless will make it into the core
- maybe it's just too revolutionary.

12.3.12 Why can't lambda forms contain statements?
--------------------------------------------------

Python lambda forms cannot contain statements because Python's syntactic
framework can't handle statements nested inside expressions.  However,
in Python, this is not a serious problem.  Unlike lambda forms in other
languages, where they add functionality, Python lambdas are only a
shorthand notation if you're too lazy to define a function.

  Functions are already first class objects in Python, and can be
declared in a local scope.  Therefore the only advantage of using a
lambda form instead of a locally-defined function is that you don't
need to invent a name for the function - but that's just a local
variable to which the function object (which is exactly the same type
of object that a lambda form yields) is assigned!

12.3.13 Can Python be compiled to machine code, C or some other language?
-------------------------------------------------------------------------

Not easily.  Python's high level data types, dynamic typing of objects
and run-time invocation of the interpreter (using eval() or
exec)  together mean that a "compiled" Python program would
probably consist mostly of calls into the Python run-time system, even
for seemingly simple operations like `x+1'.

  Several projects described in the Python newsgroup or at past Python
conferences (http://python.org/community/workshops/)have shown that this
approach is feasible, although the speedups reached so far are only
modest (e.g. 2x).  Jython uses the same strategy for compiling to Java
bytecode.  (Jim Hugunin has demonstrated that in combination with
whole-program analysis, speedups of 1000x are feasible for small demo
programs.  See the proceedings from the 1997 Python conference
(http://python.org/workshops/1997-10/proceedings/)for more information.)

  Internally, Python source code is always translated into a bytecode
representation, and this bytecode is then executed by the Python virtual
machine.  In order to avoid the overhead of repeatedly parsing and
translating modules that rarely change, this byte code is written into
a file whose name ends in ".pyc" whenever a module is parsed.  When the
corresponding .py file is changed, it is parsed and translated again
and the .pyc file is rewritten.

  There is no performance difference once the .pyc file has been
loaded, as the bytecode read from the .pyc file is exactly the same as
the bytecode created by direct translation.  The only difference is
that loading code from a .pyc file is faster than parsing and
translating a .py file, so the presence of precompiled .pyc files
improves the start-up time of Python scripts.  If desired, the
Lib/compileall.py module can be used to create valid .pyc files for a
given set of modules.

  Note that the main script executed by Python, even if its filename
ends in .py, is not compiled to a .pyc file.  It is compiled to
bytecode, but the bytecode is not saved to a file.  Usually main
scripts are quite short, so this doesn't cost much speed.

  There are also several programs which make it easier to intermingle
Python and C code in various ways to increase performance.  See, for
example, Psyco (http://psyco.sourceforge.net/), Pyrex
(http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/), PyInline
(http://pyinline.sourceforge.net/), Py2Cmod
(http://sourceforge.net/projects/py2cmod/), and Weave
(http://www.scipy.org/Weave).

12.3.14 How does Python manage memory?
--------------------------------------

The details of Python memory management depend on the implementation.
The standard C implementation of Python uses reference counting to
detect inaccessible objects, and another mechanism to collect reference
cycles, periodically executing a cycle detection algorithm which looks
for inaccessible cycles and deletes the objects involved. The gc
module provides functions to perform a garbage collection, obtain
debugging statistics, and tune the collector's parameters.

  Jython relies on the Java runtime so the JVM's garbage collector is
used.  This difference can cause some subtle porting problems if your
Python code depends on the behavior of the reference counting
implementation.

  Sometimes objects get stuck in tracebacks temporarily and hence are
not deallocated when you might expect.  Clear the tracebacks with:

    import sys
    sys.exc_clear()
    sys.exc_traceback = sys.last_traceback = None

Tracebacks are used for reporting errors, implementing debuggers and
related things.  They contain a portion of the program state extracted
during the handling of an exception (usually the most recent exception).

  In the absence of circularities and tracebacks, Python programs do
not need to manage memory explicitly.

  Why doesn't Python use a more traditional garbage collection scheme?
For one thing, this is not a C standard feature and hence it's not
portable.  (Yes, we know about the Boehm GC library.  It has bits of
assembler code for _most_ common platforms, not for all of them, and
although it is mostly transparent, it isn't completely transparent;
patches are required to get Python to work with it.)

  Traditional GC also becomes a problem when Python is embedded into
other applications.  While in a standalone Python it's fine to replace
the standard malloc() and free() with versions provided by the GC
library, an application embedding Python may want to have its _own_
substitute for malloc() and free(), and may not want Python's.  Right
now, Python works with anything that implements malloc() and free()
properly.

  In Jython, the following code (which is fine in CPython) will
probably run out of file descriptors long before it runs out of memory:

    for file in very_long_list_of_files:
        f = open(file)
        c = f.read(1)

Using the current reference counting and destructor scheme, each new
assignment to f closes the previous file.  Using GC, this is not
guaranteed.  If you want to write code that will work with any Python
implementation, you should explicitly close the file or use the
with statement; this will work regardless of GC:

    for file in very_long_list_of_files:
        with open(file) as f:
            c = f.read(1)

12.3.15 Why isn't all memory freed when Python exits?
-----------------------------------------------------

Objects referenced from the global namespaces of Python modules are not
always deallocated when Python exits.  This may happen if there are
circular references.  There are also certain bits of memory that are
allocated by the C library that are impossible to free (e.g. a tool
like Purify will complain about these).  Python is, however, aggressive
about cleaning up memory on exit and does try to destroy every single
object.

  If you want to force Python to delete certain things on deallocation
use the atexit module to run a function that will force
those deletions.

12.3.16 Why are there separate tuple and list data types?
---------------------------------------------------------

Lists and tuples, while similar in many respects, are generally used in
fundamentally different ways.  Tuples can be thought of as being
similar to Pascal records or C structs; they're small collections of
related data which may be of different types which are operated on as a
group.  For example, a Cartesian coordinate is appropriately
represented as a tuple of two or three numbers.

  Lists, on the other hand, are more like arrays in other languages.
They tend to hold a varying number of objects all of which have the
same type and which are operated on one-by-one.  For example,
`os.listdir('.')' returns a list of strings representing the files in
the current directory.  Functions which operate on this output would
generally not break if you added another file or two to the directory.

  Tuples are immutable, meaning that once a tuple has been created, you
can't replace any of its elements with a new value.  Lists are mutable,
meaning that you can always change a list's elements.  Only immutable
elements can be used as dictionary keys, and hence only tuples and not
lists can be used as keys.

12.3.17 How are lists implemented?
----------------------------------

Python's lists are really variable-length arrays, not Lisp-style linked
lists.  The implementation uses a contiguous array of references to
other objects, and keeps a pointer to this array and the array's length
in a list head structure.

  This makes indexing a list `a[i]' an operation whose cost is
independent of the size of the list or the value of the index.

  When items are appended or inserted, the array of references is
resized.  Some cleverness is applied to improve the performance of
appending items repeatedly; when the array must be grown, some extra
space is allocated so the next few times don't require an actual resize.

12.3.18 How are dictionaries implemented?
-----------------------------------------

Python's dictionaries are implemented as resizable hash tables.
Compared to B-trees, this gives better performance for lookup (the most
common operation by far) under most circumstances, and the
implementation is simpler.

  Dictionaries work by computing a hash code for each key stored in the
dictionary using the hash() built-in function.  The hash
code varies widely depending on the key; for example, "Python" hashes
to -539294296 while "python", a string that differs by a single bit,
hashes to 1142331976.  The hash code is then used to calculate a
location in an internal array where the value will be stored.  Assuming
that you're storing keys that all have different hash values, this
means that dictionaries take constant time - O(1), in computer science
notation - to retrieve a key.  It also means that no sorted order of
the keys is maintained, and traversing the array as the `.keys()' and
`.items()' do will output the dictionary's content in some arbitrary
jumbled order.

12.3.19 Why must dictionary keys be immutable?
----------------------------------------------

The hash table implementation of dictionaries uses a hash value
calculated from the key value to find the key.  If the key were a
mutable object, its value could change, and thus its hash could also
change.  But since whoever changes the key object can't tell that it
was being used as a dictionary key, it can't move the entry around in
the dictionary.  Then, when you try to look up the same object in the
dictionary it won't be found because its hash value is different.  If
you tried to look up the old value it wouldn't be found either, because
the value of the object found in that hash bin would be different.

  If you want a dictionary indexed with a list, simply convert the list
to a tuple first; the function `tuple(L)' creates a tuple with the same
entries as the list `L'.  Tuples are immutable and can therefore be
used as dictionary keys.

  Some unacceptable solutions that have been proposed:

   - Hash lists by their address (object ID).  This doesn't work
     because if you construct a new list with the same value it won't
     be found; e.g.:

         mydict = {[1, 2]: '12'}
         print mydict[[1, 2]]

     would raise a KeyError exception because the id of the `[1, 2]'
     used in the second line differs from that in the first line.  In
     other words, dictionary keys should be compared using `==', not
     using is.

   - Make a copy when using a list as a key.  This doesn't work because
     the list, being a mutable object, could contain a reference to
     itself, and then the copying code would run into an infinite loop.

   - Allow lists as keys but tell the user not to modify them.  This
     would allow a class of hard-to-track bugs in programs when you
     forgot or modified a list by accident. It also invalidates an
     important invariant of dictionaries: every value in `d.keys()' is
     usable as a key of the dictionary.

   - Mark lists as read-only once they are used as a dictionary key.
     The problem is that it's not just the top-level object that could
     change its value; you could use a tuple containing a list as a
     key.  Entering anything as a key into a dictionary would require
     marking all objects reachable from there as read-only - and again,
     self-referential objects could cause an infinite loop.

  There is a trick to get around this if you need to, but use it at
your own risk: You can wrap a mutable structure inside a class instance
which has both a __eq__() and a __hash__().
method.  You must then make sure that the hash value for all such
wrapper objects that reside in a dictionary (or other hash based
structure), remain fixed while the object is in the dictionary (or
other structure).

    class ListWrapper:
        def __init__(self, the_list):
            self.the_list = the_list
        def __eq__(self, other):
            return self.the_list == other.the_list
        def __hash__(self):
            l = self.the_list
            result = 98767 - len(l)*555
            for i, el in enumerate(l):
                try:
                    result = result + (hash(el) % 9999999) * 1001 + i
                except Exception:
                    result = (result % 7777777) + i * 333
            return result

Note that the hash computation is complicated by the possibility that
some members of the list may be unhashable and also by the possibility
of arithmetic overflow.

  Furthermore it must always be the case that if `o1 == o2' (ie
`o1.__eq__(o2) is True') then `hash(o1) == hash(o2)' (ie,
`o1.__hash__() == o2.__hash__()'), regardless of whether the object is
in a dictionary or not.  If you fail to meet these restrictions
dictionaries and other hash based structures will misbehave.

  In the case of ListWrapper, whenever the wrapper object is in a
dictionary the wrapped list must not change to avoid anomalies.  Don't
do this unless you are prepared to think hard about the requirements
and the consequences of not meeting them correctly.  Consider yourself
warned.

12.3.20 Why doesn't list.sort() return the sorted list?
-------------------------------------------------------

In situations where performance matters, making a copy of the list just
to sort it would be wasteful. Therefore, `list.sort()' sorts the list
in place. In order to remind you of that fact, it does not return the
sorted list.  This way, you won't be fooled into accidentally
overwriting a list when you need a sorted copy but also need to keep
the unsorted version around.

  In Python 2.4 a new built-in function - see sorted(). - has
been added.  This function creates a new list from a provided iterable,
sorts it and returns it.  For example, here's how to iterate over the
keys of a dictionary in sorted order:

    for key in sorted(mydict):
        ... # do whatever with mydict[key]...

12.3.21 How do you specify and enforce an interface spec in Python?
-------------------------------------------------------------------

An interface specification for a module as provided by languages such
as C++ and Java describes the prototypes for the methods and functions
of the module.  Many feel that compile-time enforcement of interface
specifications helps in the construction of large programs.

  Python 2.6 adds an abc module that lets you define Abstract
Base Classes (ABCs).  You can then use isinstance() and
issubclass() to check whether an instance or a class
implements a particular ABC.  The collections modules
defines a set of useful ABCs such as `Iterable', `Container', and
`MutableMapping'.

  For Python, many of the advantages of interface specifications can be
obtained by an appropriate test discipline for components.  There is
also a tool, PyChecker, which can be used to find problems due to
subclassing.

  A good test suite for a module can both provide a regression test and
serve as a module interface specification and a set of examples.  Many
Python modules can be run as a script to provide a simple "self test."
Even modules which use complex external interfaces can often be tested
in isolation using trivial "stub" emulations of the external interface.
The doctest and unittest modules or third-party
test frameworks can be used to construct exhaustive test suites that
exercise every line of code in a module.

  An appropriate testing discipline can help build large complex
applications in Python as well as having interface specifications
would.  In fact, it can be better because an interface specification
cannot test certain properties of a program.  For example, the
`append()' method is expected to add new elements to the end of some
internal list; an interface specification cannot test that your
`append()' implementation will actually do this correctly, but it's
trivial to check this property in a test suite.

  Writing test suites is very helpful, and you might want to design
your code with an eye to making it easily tested.  One increasingly
popular technique, test-directed development, calls for writing parts
of the test suite first, before you write any of the actual code.  Of
course Python allows you to be sloppy and not write test cases at all.

12.3.22 Why are default values shared between objects?
------------------------------------------------------

This type of bug commonly bites neophyte programmers.  Consider this
function:

    def foo(mydict={}):  # Danger: shared reference to one dict for all calls
        ... compute something ...
        mydict[key] = value
        return mydict

The first time you call this function, `mydict' contains a single item.
The second time, `mydict' contains two items because when `foo()' begins
executing, `mydict' starts out with an item already in it.

  It is often expected that a function call creates new objects for
default values. This is not what happens. Default values are created
exactly once, when the function is defined.  If that object is changed,
like the dictionary in this example, subsequent calls to the function
will refer to this changed object.

  By definition, immutable objects such as numbers, strings, tuples,
and `None', are safe from change. Changes to mutable objects such as
dictionaries, lists, and class instances can lead to confusion.

  Because of this feature, it is good programming practice to not use
mutable objects as default values.  Instead, use `None' as the default
value and inside the function, check if the parameter is `None' and
create a new list/dictionary/whatever if it is.  For example, don't
write:

    def foo(mydict={}):
        ...

but:

    def foo(mydict=None):
        if mydict is None:
            mydict = {}  # create a new dict for local namespace

This feature can be useful.  When you have a function that's
time-consuming to compute, a common technique is to cache the
parameters and the resulting value of each call to the function, and
return the cached value if the same value is requested again.  This is
called "memoizing", and can be implemented like this:

    # Callers will never provide a third parameter for this function.
    def expensive (arg1, arg2, _cache={}):
        if (arg1, arg2) in _cache:
            return _cache[(arg1, arg2)]

        # Calculate the value
        result = ... expensive computation ...
        _cache[(arg1, arg2)] = result           # Store result in the cache
        return result

You could use a global variable containing a dictionary instead of the
default value; it's a matter of taste.

12.3.23 Why is there no goto?
-----------------------------

You can use exceptions to provide a "structured goto" that even works
across function calls.  Many feel that exceptions can conveniently
emulate all reasonable uses of the "go" or "goto" constructs of C,
Fortran, and other languages.  For example:

    class label: pass  # declare a label

    try:
         ...
         if (condition): raise label()  # goto label
         ...
    except label:  # where to goto
         pass
    ...

This doesn't allow you to jump into the middle of a loop, but that's
usually considered an abuse of goto anyway.  Use sparingly.

12.3.24 Why can't raw strings (r-strings) end with a backslash?
---------------------------------------------------------------

More precisely, they can't end with an odd number of backslashes: the
unpaired backslash at the end escapes the closing quote character,
leaving an unterminated string.

  Raw strings were designed to ease creating input for processors
(chiefly regular expression engines) that want to do their own
backslash escape processing. Such processors consider an unmatched
trailing backslash to be an error anyway, so raw strings disallow that.
In return, they allow you to pass on the string quote character by
escaping it with a backslash.  These rules work well when r-strings are
used for their intended purpose.

  If you're trying to build Windows pathnames, note that all Windows
system calls accept forward slashes too:

    f = open("/mydir/file.txt")  # works fine!

If you're trying to build a pathname for a DOS command, try e.g. one of

    dir = r"\this\is\my\dos\dir" "\\"
    dir = r"\this\is\my\dos\dir\ "[:-1]
    dir = "\\this\\is\\my\\dos\\dir\\"

12.3.25 Why doesn't Python have a "with" statement for attribute assignments?
-----------------------------------------------------------------------------

Python has a 'with' statement that wraps the execution of a block,
calling code on the entrance and exit from the block.  Some language
have a construct that looks like this:

    with obj:
        a = 1               # equivalent to obj.a = 1
        total = total + 1   # obj.total = obj.total + 1

In Python, such a construct would be ambiguous.

  Other languages, such as Object Pascal, Delphi, and C++, use static
types, so it's possible to know, in an unambiguous way, what member is
being assigned to. This is the main point of static typing - the
compiler _always_ knows the scope of every variable at compile time.

  Python uses dynamic types. It is impossible to know in advance which
attribute will be referenced at runtime. Member attributes may be added
or removed from objects on the fly. This makes it impossible to know,
from a simple reading, what attribute is being referenced: a local one,
a global one, or a member attribute?

  For instance, take the following incomplete snippet:

    def foo(a):
        with a:
            print x

The snippet assumes that "a" must have a member attribute called "x".
However, there is nothing in Python that tells the interpreter this.
What should happen if "a" is, let us say, an integer?  If there is a
global variable named "x", will it be used inside the with block?  As
you see, the dynamic nature of Python makes such choices much harder.

  The primary benefit of "with" and similar language features
(reduction of code volume) can, however, easily be achieved in Python
by assignment.  Instead of:

    function(args).mydict[index][index].a = 21
    function(args).mydict[index][index].b = 42
    function(args).mydict[index][index].c = 63

write this:

    ref = function(args).mydict[index][index]
    ref.a = 21
    ref.b = 42
    ref.c = 63

This also has the side-effect of increasing execution speed because name
bindings are resolved at run-time in Python, and the second version
only needs to perform the resolution once.

12.3.26 Why are colons required for the if/while/def/class statements?
----------------------------------------------------------------------

The colon is required primarily to enhance readability (one of the
results of the experimental ABC language).  Consider this:

    if a == b
        print a

versus

    if a == b:
        print a

Notice how the second one is slightly easier to read.  Notice further
how a colon sets off the example in this FAQ answer; it's a standard
usage in English.

  Another minor reason is that the colon makes it easier for editors
with syntax highlighting; they can look for colons to decide when
indentation needs to be increased instead of having to do a more
elaborate parsing of the program text.

12.3.27 Why does Python allow commas at the end of lists and tuples?
--------------------------------------------------------------------

Python lets you add a trailing comma at the end of lists, tuples, and
dictionaries:

    [1, 2, 3,]
    ('a', 'b', 'c',)
    d = {
        "A": [1, 5],
        "B": [6, 7],  # last trailing comma is optional but good style
    }

There are several reasons to allow this.

  When you have a literal value for a list, tuple, or dictionary spread
across multiple lines, it's easier to add more elements because you
don't have to remember to add a comma to the previous line.  The lines
can also be sorted in your editor without creating a syntax error.

  Accidentally omitting the comma can lead to errors that are hard to
diagnose.  For example:

    x = [
      "fee",
      "fie"
      "foo",
      "fum"
    ]

This list looks like it has four elements, but it actually contains
three: "fee", "fiefoo" and "fum".  Always adding the comma avoids this
source of error.

  Allowing the trailing comma may also make programmatic code
generation easier.

12.4 Library and Extension FAQ
==============================

12.4.1 General Library Questions
--------------------------------

12.4.1.1 How do I find a module or application to perform task X?
.................................................................

Check the Library Reference to see if there's a relevant
standard library module.  (Eventually you'll learn what's in the
standard library and will able to skip this step.)

  For third-party packages, search the Python Package Index
(http://pypi.python.org/pypi) or try Google (http://www.google.com)or
another Web search engine.  Searching for "Python" plus a keyword or
two for your topic of interest will usually find something helpful.

12.4.1.2 Where is the math.py (socket.py, regex.py, etc.) source file?
......................................................................

If you can't find a source file for a module it may be a built-in or
dynamically loaded module implemented in C, C++ or other compiled
language.  In this case you may not have the source file or it may be
something like mathmodule.c, somewhere in a C source directory (not on
the Python Path).

  There are (at least) three kinds of modules in Python:

  1. modules written in Python (.py);

  2. modules written in C and dynamically loaded (.dll, .pyd, .so, .sl,
     etc);

  3. modules written in C and linked with the interpreter; to get a
     list of these, type:

         import sys
         print sys.builtin_module_names

12.4.1.3 How do I make a Python script executable on Unix?
..........................................................

You need to do two things: the script file's mode must be executable
and the first line must begin with `#!' followed by the path of the
Python interpreter.

  The first is done by executing `chmod +x scriptfile' or perhaps
`chmod 755 scriptfile'.

  The second can be done in a number of ways.  The most straightforward
way is to write

    #!/usr/local/bin/python

as the very first line of your file, using the pathname for where the
Python interpreter is installed on your platform.

  If you would like the script to be independent of where the Python
interpreter lives, you can use the "env" program.  Almost all Unix
variants support the following, assuming the Python interpreter is in a
directory on the user's $PATH:

    #!/usr/bin/env python

_Don't_ do this for CGI scripts.  The $PATH variable for CGI scripts is
often very minimal, so you need to use the actual absolute pathname of
the interpreter.

  Occasionally, a user's environment is so full that the /usr/bin/env
program fails; or there's no env program at all.  In that case, you can
try the following hack (due to Alex Rezinsky):

    #! /bin/sh
    """:"
    exec python $0 ${1+"$@"}
    """

The minor disadvantage is that this defines the script's __doc__ string.
However, you can fix that by adding

    __doc__ = """...Whatever..."""

12.4.1.4 Is there a curses/termcap package for Python?
......................................................

For Unix variants: The standard Python source distribution comes with a
curses module in the `Modules/' subdirectory, though it's not compiled
by default (note that this is not available in the Windows distribution
- there is no curses module for Windows).

  The curses module supports basic curses features as well as many
additional functions from ncurses and SYSV curses such as colour,
alternative character set support, pads, and mouse support. This means
the module isn't compatible with operating systems that only have BSD
curses, but there don't seem to be any currently maintained OSes that
fall into this category.

  For Windows: use the consolelib module
(http://effbot.org/zone/console-index.htm).

12.4.1.5 Is there an equivalent to C's onexit() in Python?
..........................................................

The atexit module provides a register function that is
similar to C's onexit.

12.4.1.6 Why don't my signal handlers work?
...........................................

The most common problem is that the signal handler is declared with the
wrong argument list.  It is called as

    handler(signum, frame)

so it should be declared with two arguments:

    def handler(signum, frame):
        ...

12.4.2 Common tasks
-------------------

12.4.2.1 How do I test a Python program or component?
.....................................................

Python comes with two testing frameworks.  The doctest.
module finds examples in the docstrings for a module and runs them,
comparing the output with the expected output given in the docstring.

  The unittest module is a fancier testing framework
modelled on Java and Smalltalk testing frameworks.

  For testing, it helps to write the program so that it may be easily
tested by using good modular design.  Your program should have almost
all functionality encapsulated in either functions or class methods -
and this sometimes has the surprising and delightful effect of making
the program run faster (because local variable accesses are faster than
global accesses).  Furthermore the program should avoid depending on
mutating global variables, since this makes testing much more difficult
to do.

  The "global main logic" of your program may be as simple as

    if __name__ == "__main__":
        main_logic()

at the bottom of the main module of your program.

  Once your program is organized as a tractable collection of functions
and class behaviours you should write test functions that exercise the
behaviours.  A test suite can be associated with each module which
automates a sequence of tests.  This sounds like a lot of work, but
since Python is so terse and flexible it's surprisingly easy.  You can
make coding much more pleasant and fun by writing your test functions
in parallel with the "production code", since this makes it easy to
find bugs and even design flaws earlier.

  "Support modules" that are not intended to be the main module of a
program may include a self-test of the module.

    if __name__ == "__main__":
        self_test()

Even programs that interact with complex external interfaces may be
tested when the external interfaces are unavailable by using "fake"
interfaces implemented in Python.

12.4.2.2 How do I create documentation from doc strings?
........................................................

The pydoc module can create HTML from the doc strings in
your Python source code.  An alternative for creating API documentation
purely from docstrings is epydoc (http://epydoc.sf.net/).  Sphinx
(http://sphinx.pocoo.org) can also include docstring content.

12.4.2.3 How do I get a single keypress at a time?
..................................................

For Unix variants: There are several solutions.  It's straightforward
to do this using curses, but curses is a fairly large module to learn.
Here's a solution without curses:

    import termios, fcntl, sys, os
    fd = sys.stdin.fileno()

    oldterm = termios.tcgetattr(fd)
    newattr = termios.tcgetattr(fd)
    newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO
    termios.tcsetattr(fd, termios.TCSANOW, newattr)

    oldflags = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)

    try:
        while 1:
            try:
                c = sys.stdin.read(1)
                print "Got character", repr(c)
            except IOError: pass
    finally:
        termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)
        fcntl.fcntl(fd, fcntl.F_SETFL, oldflags)

You need the termios and the fcntl module for
any of this to work, and I've only tried it on Linux, though it should
work elsewhere.  In this code, characters are read and printed one at a
time.

  termios.tcsetattr() turns off stdin's echoing and
disables canonical mode.  `fcntl.fnctl()' is used to obtain stdin's
file descriptor flags and modify them for non-blocking mode.  Since
reading stdin when it is empty results in an IOError, this
error is caught and ignored.

12.4.3 Threads
--------------

12.4.3.1 How do I program using threads?
........................................

Be sure to use the threading module and not the
thread module.  The threading module builds
convenient abstractions on top of the low-level primitives provided by
the thread module.

  Aahz has a set of slides from his threading tutorial that are
helpful; see <http://www.pythoncraft.com/OSCON2001/>.

12.4.3.2 None of my threads seem to run: why?
.............................................

As soon as the main thread exits, all threads are killed.  Your main
thread is running too quickly, giving the threads no time to do any
work.

  A simple fix is to add a sleep to the end of the program that's long
enough for all the threads to finish:

    import threading, time

    def thread_task(name, n):
        for i in range(n): print name, i

    for i in range(10):
        T = threading.Thread(target=thread_task, args=(str(i), i))
        T.start()

    time.sleep(10) # <----------------------------!

But now (on many platforms) the threads don't run in parallel, but
appear to run sequentially, one at a time!  The reason is that the OS
thread scheduler doesn't start a new thread until the previous thread
is blocked.

  A simple fix is to add a tiny sleep to the start of the run function:

    def thread_task(name, n):
        time.sleep(0.001) # <---------------------!
        for i in range(n): print name, i

    for i in range(10):
        T = threading.Thread(target=thread_task, args=(str(i), i))
        T.start()

    time.sleep(10)

Instead of trying to guess how long a time.sleep() delay
will be enough, it's better to use some kind of semaphore mechanism.
One idea is to use the Queue module to create a queue
object, let each thread append a token to the queue when it finishes,
and let the main thread read as many tokens from the queue as there are
threads.

12.4.3.3 How do I parcel out work among a bunch of worker threads?
..................................................................

Use the Queue module to create a queue containing a list of
jobs.  The Queue class maintains a list of objects with
`.put(obj)' to add an item to the queue and `.get()' to return an item.
The class will take care of the locking necessary to ensure that each
job is handed out exactly once.

  Here's a trivial example:

    import threading, Queue, time

    # The worker thread gets jobs off the queue.  When the queue is empty, it
    # assumes there will be no more work and exits.
    # (Realistically workers will run until terminated.)
    def worker ():
        print 'Running worker'
        time.sleep(0.1)
        while True:
            try:
                arg = q.get(block=False)
            except Queue.Empty:
                print 'Worker', threading.currentThread(),
                print 'queue empty'
                break
            else:
                print 'Worker', threading.currentThread(),
                print 'running with argument', arg
                time.sleep(0.5)

    # Create queue
    q = Queue.Queue()

    # Start a pool of 5 workers
    for i in range(5):
        t = threading.Thread(target=worker, name='worker %i' % (i+1))
        t.start()

    # Begin adding work to the queue
    for i in range(50):
        q.put(i)

    # Give threads time to run
    print 'Main thread sleeping'
    time.sleep(5)

When run, this will produce the following output:

     Running worker Running worker Running worker Running worker
     Running worker Main thread sleeping Worker <Thread(worker 1,
     started)> running with argument 0 Worker <Thread(worker 2,
     started)> running with argument 1 Worker <Thread(worker 3,
     started)> running with argument 2 Worker <Thread(worker 4,
     started)> running with argument 3 Worker <Thread(worker 5,
     started)> running with argument 4 Worker <Thread(worker 1,
     started)> running with argument 5 ...

  Consult the module's documentation for more details; the `Queue' class
provides a featureful interface.

12.4.3.4 What kinds of global value mutation are thread-safe?
.............................................................

A global interpreter lock (GIL) is used internally to ensure that only
one thread runs in the Python VM at a time.  In general, Python offers
to switch among threads only between bytecode instructions; how
frequently it switches can be set via sys.setcheckinterval().
Each bytecode instruction and therefore all the C implementation
code reached from each instruction is therefore atomic from the point
of view of a Python program.

  In theory, this means an exact accounting requires an exact
understanding of the PVM bytecode implementation.  In practice, it
means that operations on shared variables of built-in data types (ints,
lists, dicts, etc) that "look atomic" really are.

  For example, the following operations are all atomic (L, L1, L2 are
lists, D, D1, D2 are dicts, x, y are objects, i, j are ints):

    L.append(x)
    L1.extend(L2)
    x = L[i]
    x = L.pop()
    L1[i:j] = L2
    L.sort()
    x = y
    x.field = y
    D[x] = y
    D1.update(D2)
    D.keys()

These aren't:

    i = i+1
    L.append(L[-1])
    L[i] = L[j]
    D[x] = D[x] + 1

Operations that replace other objects may invoke those other objects'
__del__() method when their reference count reaches zero,
and that can affect things.  This is especially true for the mass
updates to dictionaries and lists.  When in doubt, use a mutex!

12.4.3.5 Can't we get rid of the Global Interpreter Lock?
.........................................................

The Global Interpreter Lock (GIL) is often seen as a hindrance to
Python's deployment on high-end multiprocessor server machines, because
a multi-threaded Python program effectively only uses one CPU, due to
the insistence that (almost) all Python code can only run while the GIL
is held.

  Back in the days of Python 1.5, Greg Stein actually implemented a
comprehensive patch set (the "free threading" patches) that removed the
GIL and replaced it with fine-grained locking.  Unfortunately, even on
Windows (where locks are very efficient) this ran ordinary Python code
about twice as slow as the interpreter using the GIL.  On Linux the
performance loss was even worse because pthread locks aren't as
efficient.

  Since then, the idea of getting rid of the GIL has occasionally come
up but nobody has found a way to deal with the expected slowdown, and
users who don't use threads would not be happy if their code ran at
half at the speed.  Greg's free threading patch set has not been kept
up-to-date for later Python versions.

  This doesn't mean that you can't make good use of Python on multi-CPU
machines!  You just have to be creative with dividing the work up
between multiple _processes_ rather than multiple _threads_.  Judicious
use of C extensions will also help; if you use a C extension to perform
a time-consuming task, the extension can release the GIL while the
thread of execution is in the C code and allow other threads to get
some work done.

  It has been suggested that the GIL should be a per-interpreter-state
lock rather than truly global; interpreters then wouldn't be able to
share objects.  Unfortunately, this isn't likely to happen either.  It
would be a tremendous amount of work, because many object
implementations currently have global state.  For example, small
integers and short strings are cached; these caches would have to be
moved to the interpreter state.  Other object types have their own free
list; these free lists would have to be moved to the interpreter state.
And so on.

  And I doubt that it can even be done in finite time, because the same
problem exists for 3rd party extensions.  It is likely that 3rd party
extensions are being written at a faster rate than you can convert them
to store all their global state in the interpreter state.

  And finally, once you have multiple interpreters not sharing any
state, what have you gained over running each interpreter in a separate
process?

12.4.4 Input and Output
-----------------------

12.4.4.1 How do I delete a file? (And other file questions...)
..............................................................

Use `os.remove(filename)' or `os.unlink(filename)'; for documentation,
see the os module.  The two functions are identical;
`unlink()' is simply the name of the Unix system call for this function.

  To remove a directory, use os.rmdir().; use
os.mkdir() to create one.  `os.makedirs(path)' will create any
intermediate directories in `path' that don't exist.
`os.removedirs(path)' will remove intermediate directories as long as
they're empty; if you want to delete an entire directory tree and its
contents, use shutil.rmtree().

  To rename a file, use `os.rename(old_path, new_path)'.

  To truncate a file, open it using `f = open(filename, "r+")', and use
`f.truncate(offset)'; offset defaults to the current seek position.
There's also `os.ftruncate(fd, offset)' for files opened with
os.open(), where `fd' is the file descriptor (a small integer).

  The shutil module also contains a number of functions to
work on files including copyfile(), copytree(),
and rmtree().

12.4.4.2 How do I copy a file?
..............................

The shutil module contains a copyfile().
function.  Note that on MacOS 9 it doesn't copy the resource fork and
Finder info.

12.4.4.3 How do I read (or write) binary data?
..............................................

To read or write complex binary data formats, it's best to use the
struct.  module.  It allows you to take a string containing
binary data (usually numbers) and convert it to Python objects; and
vice versa.

  For example, the following code reads two 2-byte integers and one
4-byte integer in big-endian format from a file:

    import struct

    f = open(filename, "rb")  # Open in binary mode for portability
    s = f.read(8)
    x, y, z = struct.unpack(">hhl", s)

The '>' in the format string forces big-endian data; the letter 'h'
reads one "short integer" (2 bytes), and 'l' reads one "long integer"
(4 bytes) from the string.

  For data that is more regular (e.g. a homogeneous list of ints or
thefloats), you can also use the array module.

12.4.4.4 I can't seem to use os.read() on a pipe created with os.popen(); why?
..............................................................................

os.read() is a low-level function which takes a file
descriptor, a small integer representing the opened file.
os.popen() creates a high-level file object, the same type
returned by the built-in open() function.  Thus, to read n
bytes from a pipe p created with os.popen(), you need to use
`p.read(n)'.

12.4.4.5 How do I run a subprocess with pipes connected to both input and output?
.................................................................................

Use the popen2 module.  For example:

    import popen2
    fromchild, tochild = popen2.popen2("command")
    tochild.write("input\n")
    tochild.flush()
    output = fromchild.readline()

Warning: in general it is unwise to do this because you can easily
cause a deadlock where your process is blocked waiting for output from
the child while the child is blocked waiting for input from you.  This
can be caused because the parent expects the child to output more text
than it does, or it can be caused by data being stuck in stdio buffers
due to lack of flushing.  The Python parent can of course explicitly
flush the data it sends to the child before it reads any output, but if
the child is a naive C program it may have been written to never
explicitly flush its output, even if it is interactive, since flushing
is normally automatic.

  Note that a deadlock is also possible if you use `popen3()' to read
stdout and stderr. If one of the two is too large for the internal
buffer (increasing the buffer size does not help) and you `read()' the
other one first, there is a deadlock, too.

  Note on a bug in popen2: unless your program calls `wait()' or
`waitpid()', finished child processes are never removed, and eventually
calls to popen2 will fail because of a limit on the number of child
processes.  Calling os.waitpid() with the
os.WNOHANG option can prevent this; a good place to insert such
a call would be before calling `popen2' again.

  In many cases, all you really need is to run some data through a
command and get the result back.  Unless the amount of data is very
large, the easiest way to do this is to write it to a temporary file
and run the command with that temporary file as input.  The standard
module tempfile exports a `mktemp()' function to generate
unique temporary file names.

    import tempfile
    import os

    class Popen3:
        """
        This is a deadlock-safe version of popen that returns
        an object with errorlevel, out (a string) and err (a string).
        (capturestderr may not work under windows.)
        Example: print Popen3('grep spam','\n\nhere spam\n\n').out
        """
        def __init__(self,command,input=None,capturestderr=None):
            outfile=tempfile.mktemp()
            command="( %s ) > %s" % (command,outfile)
            if input:
                infile=tempfile.mktemp()
                open(infile,"w").write(input)
                command=command+" <"+infile
            if capturestderr:
                errfile=tempfile.mktemp()
                command=command+" 2>"+errfile
            self.errorlevel=os.system(command) >> 8
            self.out=open(outfile,"r").read()
            os.remove(outfile)
            if input:
                os.remove(infile)
            if capturestderr:
                self.err=open(errfile,"r").read()
                os.remove(errfile)

Note that many interactive programs (e.g. vi) don't work well with pipes
substituted for standard input and output.  You will have to use pseudo
ttys ("ptys") instead of pipes. Or you can use a Python interface to
Don Libes' "expect" library.  A Python extension that interfaces to
expect is called "expy" and available from
<http://expectpy.sourceforge.net>.  A pure Python solution that works
like expect is pexpect (http://pypi.python.org/pypi/pexpect/).

12.4.4.6 How do I access the serial (RS232) port?
.................................................

For Win32, POSIX (Linux, BSD, etc.), Jython:

     <http://pyserial.sourceforge.net>

  For Unix, see a Usenet post by Mitch Chapman:

     <http://groups.google.com/groups?selm=34A04430.CF9@ohioee.com>

12.4.4.7 Why doesn't closing sys.stdout (stdin, stderr) really close it?
........................................................................

Python file objects are a high-level layer of abstraction on top of C
streams, which in turn are a medium-level layer of abstraction on top
of (among other things) low-level C file descriptors.

  For most file objects you create in Python via the built-in `file'
constructor, `f.close()' marks the Python file object as being closed
from Python's point of view, and also arranges to close the underlying
C stream.  This also happens automatically in `f''s destructor, when
`f' becomes garbage.

  But stdin, stdout and stderr are treated specially by Python, because
of the special status also given to them by C.  Running
`sys.stdout.close()' marks the Python-level file object as being
closed, but does _not_ close the associated C stream.

  To close the underlying C stream for one of these three, you should
first be sure that's what you really want to do (e.g., you may confuse
extension modules trying to do I/O).  If it is, use os.close:

    os.close(0)   # close C's stdin stream
    os.close(1)   # close C's stdout stream
    os.close(2)   # close C's stderr stream

12.4.5 Network/Internet Programming
-----------------------------------

12.4.5.1 What WWW tools are there for Python?
.............................................

See the chapters titled Internet Protocols and Support and
Internet Data Handling in the Library Reference Manual.
Python has many modules that will help you build server-side and
client-side web systems.

  A summary of available frameworks is maintained by Paul Boddie at
<http://wiki.python.org/moin/WebProgramming> .

  Cameron Laird maintains a useful set of pages about Python web
technologies at <http://phaseit.net/claird/comp.lang.python/web_python>.

12.4.5.2 How can I mimic CGI form submission (METHOD=POST)?
...........................................................

I would like to retrieve web pages that are the result of POSTing a
form. Is there existing code that would let me do this easily?

  Yes. Here's a simple example that uses httplib:

    #!/usr/local/bin/python

    import httplib, sys, time

    ### build the query string
    qs = "First=Josephine&MI=Q&Last=Public"

    ### connect and send the server a path
    httpobj = httplib.HTTP('www.some-server.out-there', 80)
    httpobj.putrequest('POST', '/cgi-bin/some-cgi-script')
    ### now generate the rest of the HTTP headers...
    httpobj.putheader('Accept', '*/*')
    httpobj.putheader('Connection', 'Keep-Alive')
    httpobj.putheader('Content-type', 'application/x-www-form-urlencoded')
    httpobj.putheader('Content-length', '%d' % len(qs))
    httpobj.endheaders()
    httpobj.send(qs)
    ### find out what the server said in response...
    reply, msg, hdrs = httpobj.getreply()
    if reply != 200:
        sys.stdout.write(httpobj.getfile().read())

Note that in general for percent-encoded POST operations, query strings
must be quoted using urllib.quote().  For example to send
name="Guy Steele, Jr.":

    >>> from urllib import quote
    >>> x = quote("Guy Steele, Jr.")
    >>> x
    'Guy%20Steele,%20Jr.'
    >>> query_string = "name="+x
    >>> query_string
    'name=Guy%20Steele,%20Jr.'

12.4.5.3 What module should I use to help with generating HTML?
...............................................................

There are many different modules available:

   * HTMLgen is a class library of objects corresponding to all the
     HTML 3.2 markup tags. It's used when you are writing in Python and
     wish to synthesize HTML pages for generating a web or for CGI
     forms, etc.

   * DocumentTemplate and Zope Page Templates are two different systems
     that are part of Zope.

   * Quixote's PTL uses Python syntax to assemble strings of text.

  Consult the Web Programming wiki pages
(http://wiki.python.org/moin/WebProgramming)for more links.

12.4.5.4 How do I send mail from a Python script?
.................................................

Use the standard library module smtplib.

  Here's a very simple interactive mail sender that uses it.  This
method will work on any host that supports an SMTP listener.

    import sys, smtplib

    fromaddr = raw_input("From: ")
    toaddrs  = raw_input("To: ").split(',')
    print "Enter message, end with ^D:"
    msg = ''
    while True:
        line = sys.stdin.readline()
        if not line:
            break
        msg += line

    # The actual mail send
    server = smtplib.SMTP('localhost')
    server.sendmail(fromaddr, toaddrs, msg)
    server.quit()

A Unix-only alternative uses sendmail.  The location of the sendmail
program varies between systems; sometimes it is `/usr/lib/sendmail',
sometime `/usr/sbin/sendmail'.  The sendmail manual page will help you
out.  Here's some sample code:

    SENDMAIL = "/usr/sbin/sendmail" # sendmail location
    import os
    p = os.popen("%s -t -i" % SENDMAIL, "w")
    p.write("To: receiver@example.com\n")
    p.write("Subject: test\n")
    p.write("\n") # blank line separating headers from body
    p.write("Some text\n")
    p.write("some more text\n")
    sts = p.close()
    if sts != 0:
        print "Sendmail exit status", sts

12.4.5.5 How do I avoid blocking in the connect() method of a socket?
.....................................................................

The select module is commonly used to help with asynchronous I/O on
sockets.

  To prevent the TCP connect from blocking, you can set the socket to
non-blocking mode.  Then when you do the `connect()', you will either
connect immediately (unlikely) or get an exception that contains the
error number as `.errno'.  `errno.EINPROGRESS' indicates that the
connection is in progress, but hasn't finished yet.  Different OSes
will return different values, so you're going to have to check what's
returned on your system.

  You can use the `connect_ex()' method to avoid creating an exception.
It will just return the errno value.  To poll, you can call
`connect_ex()' again later - 0 or `errno.EISCONN' indicate that you're
connected - or you can pass this socket to select to check if it's
writable.

12.4.6 Databases
----------------

12.4.6.1 Are there any interfaces to database packages in Python?
.................................................................

Yes.

  Python 2.3 includes the bsddb package which provides an
interface to the BerkeleyDB library.  Interfaces to disk-based hashes
such as DBM.  and GDBM are also included with
standard Python.

  Support for most relational databases is available.  See the
DatabaseProgramming wiki page
(http://wiki.python.org/moin/DatabaseProgramming)for details.

12.4.6.2 How do you implement persistent objects in Python?
...........................................................

The pickle library module solves this in a very general way
(though you still can't store things like open files, sockets or
windows), and the shelve library module uses pickle and
(g)dbm to create persistent mappings containing arbitrary Python
objects.  For better performance, you can use the cPickle.
module.

  A more awkward way of doing things is to use pickle's little sister,
marshal.  The marshal module provides very fast ways to
store noncircular basic Python types to files and strings, and back
again.  Although marshal does not do fancy things like store instances
or handle shared references properly, it does run extremely fast.  For
example loading a half megabyte of data may take less than a third of a
second.  This often beats doing something more complex and general such
as using gdbm with pickle/shelve.

12.4.6.3 Why is cPickle so slow?
................................

The default format used by the pickle module is a slow one that results
in readable pickles.  Making it the default, but it would break backward
compatibility:

    largeString = 'z' * (100 * 1024)
    myPickle = cPickle.dumps(largeString, protocol=1)

12.4.6.4 If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come?
...................................................................................................

Databases opened for write access with the bsddb module (and often by
the anydbm module, since it will preferentially use bsddb) must
explicitly be closed using the `.close()' method of the database.  The
underlying library caches database contents which need to be converted
to on-disk form and written.

  If you have initialized a new bsddb database but not written anything
to it before the program crashes, you will often wind up with a
zero-length file and encounter an exception the next time the file is
opened.

12.4.6.5 I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data?
.....................................................................................................................................

Don't panic! Your data is probably intact. The most frequent cause for
the error is that you tried to open an earlier Berkeley DB file with a
later version of the Berkeley DB library.

  Many Linux systems now have all three versions of Berkeley DB
available.  If you are migrating from version 1 to a newer version use
db_dump185 to dump a plain text version of the database.  If you are
migrating from version 2 to version 3 use db2_dump to create a plain
text version of the database.  In either case, use db_load to create a
new native database for the latest version installed on your computer.
If you have version 3 of Berkeley DB installed, you should be able to
use db2_load to create a native version 2 database.

  You should move away from Berkeley DB version 1 files because the
hash file code contains known bugs that can corrupt your data.

12.4.7 Mathematics and Numerics
-------------------------------

12.4.7.1 How do I generate random numbers in Python?
....................................................

The standard module random implements a random number
generator.  Usage is simple:

    import random
    random.random()

This returns a random floating point number in the range [0, 1).

  There are also many other specialized generators in this module, such
as:

   * `randrange(a, b)' chooses an integer in the range [a, b).

   * `uniform(a, b)' chooses a floating point number in the range [a,
     b).

   * `normalvariate(mean, sdev)' samples the normal (Gaussian)
     distribution.

  Some higher-level functions operate on sequences directly, such as:

   * `choice(S)' chooses random element from a given sequence

   * `shuffle(L)' shuffles a list in-place, i.e. permutes it randomly

  There's also a `Random' class you can instantiate to create
independent multiple random number generators.

12.5 Extending/Embedding FAQ
============================

12.5.1 Can I create my own functions in C?
------------------------------------------

Yes, you can create built-in modules containing functions, variables,
exceptions and even new types in C.  This is explained in the document
Extending and Embedding the Python Interpreter.

  Most intermediate or advanced Python books will also cover this topic.

12.5.2 Can I create my own functions in C++?
--------------------------------------------

Yes, using the C compatibility features found in C++.  Place `extern
"C" { ... }' around the Python include files and put `extern "C"'
before each function that is going to be called by the Python
interpreter.  Global or static C++ objects with constructors are
probably not a good idea.

12.5.3 Writing C is hard; are there any alternatives?
-----------------------------------------------------

There are a number of alternatives to writing your own C extensions,
depending on what you're trying to do.

  If you need more speed, Psyco (http://psyco.sourceforge.net/)
generates x86 assembly code from Python bytecode.  You can use Psyco to
compile the most time-critical functions in your code, and gain a
significant improvement with very little effort, as long as you're
running on a machine with an x86-compatible processor.

  Pyrex (http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/) is a
compiler that accepts a slightly modified form of Python and generates
the corresponding C code.  Pyrex makes it possible to write an
extension without having to learn Python's C API.

  If you need to interface to some C or C++ library for which no Python
extension currently exists, you can try wrapping the library's data
types and functions with a tool such as SWIG (http://www.swig.org).
SIP (http://www.riverbankcomputing.co.uk/software/sip/), CXX
(http://cxx.sourceforge.net/)Boost
(http://www.boost.org/libs/python/doc/index.html), or Weave
(http://www.scipy.org/Weave) are also alternatives for wrapping C++
libraries.

12.5.4 How can I execute arbitrary Python statements from C?
------------------------------------------------------------

The highest-level function to do this is PyRun_SimpleString()
which takes a single string argument to be executed in the
context of the module `__main__' and returns 0 for success and -1 when
an exception occurred (including `SyntaxError').  If you want more
control, use PyRun_String().; see the source for
PyRun_SimpleString() in `Python/pythonrun.c'.

12.5.5 How can I evaluate an arbitrary Python expression from C?
----------------------------------------------------------------

Call the function PyRun_String() from the previous
question with the start symbol Py_eval_input.; it parses an
expression, evaluates it and returns its value.

12.5.6 How do I extract C values from a Python object?
------------------------------------------------------

That depends on the object's type.  If it's a tuple, see
PyTuple_Size().  returns its length and PyTuple_GetItem()
returns the item at a specified index.  Lists have similar
functions, `PyListSize()' and PyList_GetItem().

  For strings, see PyString_Size() returns its length and
PyString_AsString() a pointer to its value.  Note that
Python strings may contain null bytes so C's `strlen()' should not be
used.

  To test the type of an object, first make sure it isn't _NULL_, and
then use PyString_Check(), PyTuple_Check(),
PyList_Check(), etc.

  There is also a high-level API to Python objects which is provided by
the so-called 'abstract' interface - read `Include/abstract.h' for
further details.  It allows interfacing with any kind of Python
sequence using calls like PySequence_Length(),
PySequence_GetItem(), etc.)  as well as many other useful
protocols.

12.5.7 How do I use Py_BuildValue() to create a tuple of arbitrary length?
--------------------------------------------------------------------------

You can't.  Use `t = PyTuple_New(n)' instead, and fill it with objects
using `PyTuple_SetItem(t, i, o)' - note that this "eats" a reference
count of `o', so you have to Py_INCREF() it.  Lists have
similar functions `PyList_New(n)' and `PyList_SetItem(l, i, o)'.  Note
that you _must_ set all the tuple items to some value before you pass
the tuple to Python code - `PyTuple_New(n)' initializes them to NULL,
which isn't a valid Python value.

12.5.8 How do I call an object's method from C?
-----------------------------------------------

The PyObject_CallMethod() function can be used to call an
arbitrary method of an object.  The parameters are the object, the name
of the method to call, a format string like that used with
Py_BuildValue(), and the argument values:

    PyObject *
    PyObject_CallMethod(PyObject *object, char *method_name,
                        char *arg_format, ...);

This works for any object that has methods - whether built-in or
user-defined.  You are responsible for eventually Py_DECREF().'
ing the return value.

  To call, e.g., a file object's "seek" method with arguments 10, 0
(assuming the file object pointer is "f"):

    res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0);
    if (res == NULL) {
            ... an exception occurred ...
    }
    else {
            Py_DECREF(res);
    }

Note that since PyObject_CallObject(). _always_ wants a
tuple for the argument list, to call a function without arguments, pass
"()" for the format, and to call a function with one argument, surround
the argument in parentheses, e.g. "(i)".

12.5.9 How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?
-----------------------------------------------------------------------------------------------

In Python code, define an object that supports the `write()' method.
Assign this object to sys.stdout and sys.stderr.
Call print_error, or just allow the standard traceback mechanism to
work. Then, the output will go wherever your `write()' method sends it.

  The easiest way to do this is to use the StringIO class in the
standard library.

  Sample code and use for catching stdout:

    >>> class StdoutCatcher:
    ...     def __init__(self):
    ...         self.data = ''
    ...     def write(self, stuff):
    ...         self.data = self.data + stuff
    ...
    >>> import sys
    >>> sys.stdout = StdoutCatcher()
    >>> print 'foo'
    >>> print 'hello world!'
    >>> sys.stderr.write(sys.stdout.data)
    foo
    hello world!

12.5.10 How do I access a module written in Python from C?
----------------------------------------------------------

You can get a pointer to the module object as follows:

    module = PyImport_ImportModule("<modulename>");

If the module hasn't been imported yet (i.e. it is not yet present in
sys.modules), this initializes the module; otherwise it
simply returns the value of `sys.modules["<modulename>"]'.  Note that
it doesn't enter the module into any namespace - it only ensures it has
been initialized and is stored in sys.modules.

  You can then access the module's attributes (i.e. any name defined in
the module) as follows:

    attr = PyObject_GetAttrString(module, "<attrname>");

Calling PyObject_SetAttrString() to assign to variables in
the module also works.

12.5.11 How do I interface to C++ objects from Python?
------------------------------------------------------

Depending on your requirements, there are many approaches.  To do this
manually, begin by reading the "Extending and Embedding" document.
Realize that for the Python run-time system, there
isn't a whole lot of difference between C and C++ - so the strategy of
building a new Python type around a C structure (pointer) type will
also work for C++ objects.

  For C++ libraries, see Writing C is hard; are there any alternatives?.

12.5.12 I added a module using the Setup file and the make fails; why?
----------------------------------------------------------------------

Setup must end in a newline, if there is no newline there, the build
process fails.  (Fixing this requires some ugly shell script hackery,
and this bug is so minor that it doesn't seem worth the effort.)

12.5.13 How do I debug an extension?
------------------------------------

When using GDB with dynamically loaded extensions, you can't set a
breakpoint in your extension until your extension is loaded.

  In your `.gdbinit' file (or interactively), add the command:

    br _PyImport_LoadDynamicModule

Then, when you run GDB:

    $ gdb /local/bin/python
    gdb) run myscript.py
    gdb) continue # repeat until your extension is loaded
    gdb) finish   # so that your extension is loaded
    gdb) br myfunction.c:50
    gdb) continue

12.5.14 I want to compile a Python module on my Linux system, but some files are missing. Why?
----------------------------------------------------------------------------------------------

Most packaged versions of Python don't include the
`/usr/lib/python2._x_/config/' directory, which contains various files
required for compiling Python extensions.

  For Red Hat, install the python-devel RPM to get the necessary files.

  For Debian, run `apt-get install python-dev'.

12.5.15 What does "SystemError: _PyImport_FixupExtension: module yourmodule not loaded" mean?
---------------------------------------------------------------------------------------------

This means that you have created an extension module named
"yourmodule", but your module init function does not initialize with
that name.

  Every module init function will have a line similar to:

    module = Py_InitModule("yourmodule", yourmodule_functions);

If the string passed to this function is not the same name as your
extension module, the SystemError exception will be raised.

12.5.16 How do I tell "incomplete input" from "invalid input"?
--------------------------------------------------------------

Sometimes you want to emulate the Python interactive interpreter's
behavior, where it gives you a continuation prompt when the input is
incomplete (e.g. you typed the start of an "if" statement or you didn't
close your parentheses or triple string quotes), but it gives you a
syntax error message immediately when the input is invalid.

  In Python you can use the codeop module, which
approximates the parser's behavior sufficiently.  IDLE uses this, for
example.

  The easiest way to do it in C is to call
PyRun_InteractiveLoop(). (perhaps in a separate thread) and let
the Python interpreter handle the input for you. You can also set the
`PyOS_ReadlineFunctionPointer()' to point at your custom input
function. See `Modules/readline.c' and `Parser/myreadline.c' for more
hints.

  However sometimes you have to run the embedded Python interpreter in
the same thread as your rest application and you can't allow the
PyRun_InteractiveLoop() to stop while waiting for user input.
The one solution then is to call `PyParser_ParseString()' and test for
`e.error' equal to `E_EOF', which means the input is incomplete).
Here's a sample code fragment, untested, inspired by code from Alex
Farber:

    #include <Python.h>
    #include <node.h>
    #include <errcode.h>
    #include <grammar.h>
    #include <parsetok.h>
    #include <compile.h>

    int testcomplete(char *code)
      /* code should end in \n */
      /* return -1 for error, 0 for incomplete, 1 for complete */
    {
      node *n;
      perrdetail e;

      n = PyParser_ParseString(code, &_PyParser_Grammar,
                               Py_file_input, &e);
      if (n == NULL) {
        if (e.error == E_EOF)
          return 0;
        return -1;
      }

      PyNode_Free(n);
      return 1;
    }

Another solution is trying to compile the received string with
Py_CompileString(). If it compiles without errors, try to execute
the returned code object by calling PyEval_EvalCode().
Otherwise save the input for later. If the compilation fails, find out
if it's an error or just more input is required - by extracting the
message string from the exception tuple and comparing it to the string
"unexpected EOF while parsing".  Here is a complete example using the
GNU readline library (you may want to ignore *SIGINT* while calling
readline()):

    #include <stdio.h>
    #include <readline.h>

    #include <Python.h>
    #include <object.h>
    #include <compile.h>
    #include <eval.h>

    int main (int argc, char* argv[])
    {
      int i, j, done = 0;                          /* lengths of line, code */
      char ps1[] = ">>> ";
      char ps2[] = "... ";
      char *prompt = ps1;
      char *msg, *line, *code = NULL;
      PyObject *src, *glb, *loc;
      PyObject *exc, *val, *trb, *obj, *dum;

      Py_Initialize ();
      loc = PyDict_New ();
      glb = PyDict_New ();
      PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ());

      while (!done)
      {
        line = readline (prompt);

        if (NULL == line)                          /* CTRL-D pressed */
        {
          done = 1;
        }
        else
        {
          i = strlen (line);

          if (i > 0)
            add_history (line);                    /* save non-empty lines */

          if (NULL == code)                        /* nothing in code yet */
            j = 0;
          else
            j = strlen (code);

          code = realloc (code, i + j + 2);
          if (NULL == code)                        /* out of memory */
            exit (1);

          if (0 == j)                              /* code was empty, so */
            code[0] = '\0';                        /* keep strncat happy */

          strncat (code, line, i);                 /* append line to code */
          code[i + j] = '\n';                      /* append '\n' to code */
          code[i + j + 1] = '\0';

          src = Py_CompileString (code, "<stdin>", Py_single_input);

          if (NULL != src)                         /* compiled just fine - */
          {
            if (ps1  == prompt ||                  /* ">>> " or */
                '\n' == code[i + j - 1])           /* "... " and double '\n' */
            {                                               /* so execute it */
              dum = PyEval_EvalCode ((PyCodeObject *)src, glb, loc);
              Py_XDECREF (dum);
              Py_XDECREF (src);
              free (code);
              code = NULL;
              if (PyErr_Occurred ())
                PyErr_Print ();
              prompt = ps1;
            }
          }                                        /* syntax error or E_EOF? */
          else if (PyErr_ExceptionMatches (PyExc_SyntaxError))
          {
            PyErr_Fetch (&exc, &val, &trb);        /* clears exception! */

            if (PyArg_ParseTuple (val, "sO", &msg, &obj) &&
                !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */
            {
              Py_XDECREF (exc);
              Py_XDECREF (val);
              Py_XDECREF (trb);
              prompt = ps2;
            }
            else                                   /* some other syntax error */
            {
              PyErr_Restore (exc, val, trb);
              PyErr_Print ();
              free (code);
              code = NULL;
              prompt = ps1;
            }
          }
          else                                     /* some non-syntax error */
          {
            PyErr_Print ();
            free (code);
            code = NULL;
            prompt = ps1;
          }

          free (line);
        }
      }

      Py_XDECREF(glb);
      Py_XDECREF(loc);
      Py_Finalize();
      exit(0);
    }

12.5.17 How do I find undefined g++ symbols __builtin_new or __pure_virtual?
----------------------------------------------------------------------------

To dynamically load g++ extension modules, you must recompile Python,
relink it using g++ (change LINKCC in the Python Modules Makefile), and
link your extension module using g++ (e.g., `g++ -shared -o mymodule.so
mymodule.o').

12.5.18 Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?
------------------------------------------------------------------------------------------------------------------------

In Python 2.2, you can inherit from built-in classes such as int,
see list, dict, etc.

  The Boost Python Library (BPL,
<http://www.boost.org/libs/python/doc/index.html>) provides a way of
doing this from C++ (i.e. you can inherit from an extension class
written in C++ using the BPL).

12.5.19 When importing module X, why do I get "undefined symbol: PyUnicodeUCS2*"?
---------------------------------------------------------------------------------

You are using a version of Python that uses a 4-byte representation for
Unicode characters, but some C extension module you are importing was
compiled using a Python that uses a 2-byte representation for Unicode
characters (the default).

  If instead the name of the undefined symbol starts with
`PyUnicodeUCS4', the problem is the reverse: Python was built using
2-byte Unicode characters, and the extension module was compiled using
a Python with 4-byte Unicode characters.

  This can easily occur when using pre-built extension packages.
RedHat Linux 7.x, in particular, provided a "python2" binary that is
compiled with 4-byte Unicode.  This only causes the link failure if the
extension uses any of the `PyUnicode_*()' functions.  It is also a
problem if an extension uses any of the Unicode-related format
specifiers for Py_BuildValue(). (or similar) or parameter
specifications for PyArg_ParseTuple().

  You can check the size of the Unicode character a Python interpreter
is using by checking the value of sys.maxunicode:

    >>> import sys
    >>> if sys.maxunicode > 65535:
    ...     print 'UCS4 build'
    ... else:
    ...     print 'UCS2 build'

The only way to solve this problem is to use extension modules compiled
with a Python binary built using the same size for Unicode characters.

12.6 Python on Windows FAQ
==========================

12.6.1 How do I run a Python program under Windows?
---------------------------------------------------

This is not necessarily a straightforward question. If you are already
familiar with running programs from the Windows command line then
everything will seem obvious; otherwise, you might need a little more
guidance.  There are also differences between Windows 95, 98, NT, ME,
2000 and XP which can add to the confusion.

Python Development on XP
(http://www.showmedo.com/videos/series?name=pythonOzsvaldPyNewbieSeries)

  This series of screencasts aims to get you up and running with Python
on Windows XP.  The knowledge is distilled into 1.5 hours and will get
you up and running with the right Python distribution, coding in your
choice of IDE, and debugging and writing solid code with unit-tests.

  Unless you use some sort of integrated development environment, you
will end up _typing_ Windows commands into what is variously referred
to as a "DOS window" or "Command prompt window".  Usually you can
create such a window from your Start menu; under Windows 2000 the menu
selection is _Start ‣ Programs ‣ Accessories ‣ Command Prompt_.  You
should be able to recognize when you have started such a window because
you will see a Windows "command prompt", which usually looks like this:

    C:\>

The letter may be different, and there might be other things after it,
so you might just as easily see something like:

    D:\Steve\Projects\Python>

depending on how your computer has been set up and what else you have
recently done with it.  Once you have started such a window, you are
well on the way to running Python programs.

  You need to realize that your Python scripts have to be processed by
another program called the Python interpreter.  The interpreter reads
your script, compiles it into bytecodes, and then executes the
bytecodes to run your program. So, how do you arrange for the
interpreter to handle your Python?

  First, you need to make sure that your command window recognises the
word "python" as an instruction to start the interpreter.  If you have
opened a command window, you should try entering the command `python'
and hitting return.  You should then see something like:

    Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

You have started the interpreter in "interactive mode". That means you
can enter Python statements or expressions interactively and have them
executed or evaluated while you wait.  This is one of Python's
strongest features.  Check it by entering a few expressions of your
choice and seeing the results:

    >>> print "Hello"
    Hello
    >>> "Hello" * 3
    HelloHelloHello

Many people use the interactive mode as a convenient yet highly
programmable calculator.  When you want to end your interactive Python
session, hold the Ctrl key down while you enter a Z, then hit the
"Enter" key to get back to your Windows command prompt.

  You may also find that you have a Start-menu entry such as _Start ‣
Programs ‣ Python 2.2 ‣ Python (command line)_ that results in you
seeing the `>>>' prompt in a new window.  If so, the window will
disappear after you enter the Ctrl-Z character; Windows is running a
single "python" command in the window, and closes it when you terminate
the interpreter.

  If the `python' command, instead of displaying the interpreter prompt
`>>>', gives you a message like:

    'python' is not recognized as an internal or external command,
    operable program or batch file.

Adding Python to DOS Path
(http://showmedo.com/videos/video?name=960000&fromSeriesID=96)

  Python is not added to the DOS path by default.  This screencast will
walk you through the steps to add the correct entry to the `System
Path', allowing Python to be executed from the command-line by all
users.

  or:

    Bad command or filename

then you need to make sure that your computer knows where to find the
Python interpreter.  To do this you will have to modify a setting
called PATH, which is a list of directories where Windows will look for
programs.

  You should arrange for Python's installation directory to be added to
the PATH of every command window as it starts.  If you installed Python
fairly recently then the command

    dir C:\py*

will probably tell you where it is installed; the usual location is
something like `C:\Python23'.  Otherwise you will be reduced to a
search of your whole disk ... use _Tools ‣ Find_ or hit the _Search_
button and look for "python.exe".  Supposing you discover that Python is
installed in the `C:\Python23' directory (the default at the time of
writing), you should make sure that entering the command

    c:\Python23\python

starts up the interpreter as above (and don't forget you'll need a
"CTRL-Z" and an "Enter" to get out of it). Once you have verified the
directory, you need to add it to the start-up routines your computer
goes through.  For older versions of Windows the easiest way to do this
is to edit the `C:\AUTOEXEC.BAT' file. You would want to add a line
like the following to `AUTOEXEC.BAT':

    PATH C:\Python23;%PATH%

For Windows NT, 2000 and (I assume) XP, you will need to add a string
such as

    ;C:\Python23

to the current setting for the PATH environment variable, which you
will find in the properties window of "My Computer" under the
"Advanced" tab.  Note that if you have sufficient privilege you might
get a choice of installing the settings either for the Current User or
for System.  The latter is preferred if you want everybody to be able
to run Python on the machine.

  If you aren't confident doing any of these manipulations yourself,
ask for help!  At this stage you may want to reboot your system to make
absolutely sure the new setting has taken effect.  You probably won't
need to reboot for Windows NT, XP or 2000.  You can also avoid it in
earlier versions by editing the file `C:\WINDOWS\COMMAND\CMDINIT.BAT'
instead of `AUTOEXEC.BAT'.

  You should now be able to start a new command window, enter `python'
at the `C:\>' (or whatever) prompt, and see the `>>>' prompt that
indicates the Python interpreter is reading interactive commands.

  Let's suppose you have a program called `pytest.py' in directory
`C:\Steve\Projects\Python'.  A session to run that program might look
like this:

    C:\> cd \Steve\Projects\Python
    C:\Steve\Projects\Python> python pytest.py

Because you added a file name to the command to start the interpreter,
when it starts up it reads the Python script in the named file,
compiles it, executes it, and terminates, so you see another `C:\>'
prompt.  You might also have entered

    C:\> python \Steve\Projects\Python\pytest.py

if you hadn't wanted to change your current directory.

  Under NT, 2000 and XP you may well find that the installation process
has also arranged that the command `pytest.py' (or, if the file isn't
in the current directory, `C:\Steve\Projects\Python\pytest.py') will
automatically recognize the ".py" extension and run the Python
interpreter on the named file. Using this feature is fine, but _some_
versions of Windows have bugs which mean that this form isn't exactly
equivalent to using the interpreter explicitly, so be careful.

  The important things to remember are:

  1. Start Python from the Start Menu, or make sure the PATH is set
     correctly so Windows can find the Python interpreter.

         python

     should give you a '>>>' prompt from the Python interpreter. Don't
     forget the CTRL-Z and ENTER to terminate the interpreter (and, if
     you started the window from the Start Menu, make the window
     disappear).

  2. Once this works, you run programs with commands:

         python {program-file}

  3. When you know the commands to use you can build Windows shortcuts
     to run the Python interpreter on any of your scripts, naming
     particular working directories, and adding them to your menus.
     Take a look at

         python --help

     if your needs are complex.

  4. Interactive mode (where you see the `>>>' prompt) is best used for
     checking that individual statements and expressions do what you
     think they will, and for developing code by experiment.

12.6.2 How do I make Python scripts executable?
-----------------------------------------------

On Windows 2000, the standard Python installer already associates the
.py extension with a file type (Python.File) and gives that file type
an open command that runs the interpreter (`D:\Program
Files\Python\python.exe "%1" %*').  This is enough to make scripts
executable from the command prompt as 'foo.py'.  If you'd rather be
able to execute the script by simple typing 'foo' with no extension you
need to add .py to the PATHEXT environment variable.

  On Windows NT, the steps taken by the installer as described above
allow you to run a script with 'foo.py', but a longtime bug in the NT
command processor prevents you from redirecting the input or output of
any script executed in this way.  This is often important.

  The incantation for making a Python script executable under WinNT is
to give the file an extension of .cmd and add the following as the
first line:

    @setlocal enableextensions & python -x %~f0 %* & goto :EOF

12.6.3 Why does Python sometimes take so long to start?
-------------------------------------------------------

Usually Python starts very quickly on Windows, but occasionally there
are bug reports that Python suddenly begins to take a long time to
start up.  This is made even more puzzling because Python will work
fine on other Windows systems which appear to be configured identically.

  The problem may be caused by a misconfiguration of virus checking
software on the problem machine.  Some virus scanners have been known
to introduce startup overhead of two orders of magnitude when the
scanner is configured to monitor all reads from the filesystem.  Try
checking the configuration of virus scanning software on your systems
to ensure that they are indeed configured identically.  McAfee, when
configured to scan all file system read activity, is a particular
offender.

12.6.4 Where is Freeze for Windows?
-----------------------------------

"Freeze" is a program that allows you to ship a Python program as a
single stand-alone executable file.  It is _not_ a compiler; your
programs don't run any faster, but they are more easily distributable,
at least to platforms with the same OS and CPU.  Read the README file
of the freeze program for more disclaimers.

  You can use freeze on Windows, but you must download the source tree
(see <http://www.python.org/download/source>).  The freeze program is
in the `Tools\freeze' subdirectory of the source tree.

  You need the Microsoft VC++ compiler, and you probably need to build
Python.  The required project files are in the PCbuild directory.

12.6.5 Is a `*.pyd' file the same as a DLL?
-------------------------------------------

Yes, .pyd files are dll's, but there are a few differences.  If you
have a DLL named `foo.pyd', then it must have a function `initfoo()'.
You can then write Python "import foo", and Python will search for
foo.pyd (as well as foo.py, foo.pyc) and if it finds it, will attempt
to call `initfoo()' to initialize it.  You do not link your .exe with
foo.lib, as that would cause Windows to require the DLL to be present.

  Note that the search path for foo.pyd is PYTHONPATH, not the same as
the path that Windows uses to search for foo.dll.  Also, foo.pyd need
not be present to run your program, whereas if you linked your program
with a dll, the dll is required.  Of course, foo.pyd is required if you
want to say `import foo'.  In a DLL, linkage is declared in the source
code with `__declspec(dllexport)'.  In a .pyd, linkage is defined in a
list of available functions.

12.6.6 How can I embed Python into a Windows application?
---------------------------------------------------------

Embedding the Python interpreter in a Windows app can be summarized as
follows:

  1. Do _not_ build Python into your .exe file directly.  On Windows,
     Python must be a DLL to handle importing modules that are
     themselves DLL's.  (This is the first key undocumented fact.)
     Instead, link to `python_NN_.dll'; it is typically installed in
     `C:\Windows\System'.  _NN_ is the Python version, a number such as
     "23" for Python 2.3.

     You can link to Python in two different ways.  Load-time linking
     means linking against `python_NN_.lib', while run-time linking
     means linking against `python_NN_.dll'.  (General note:
     `python_NN_.lib' is the so-called "import lib" corresponding to
     `python_NN_.dll'.  It merely defines symbols for the linker.)

     Run-time linking greatly simplifies link options; everything
     happens at run time.  Your code must load `python_NN_.dll' using
     the Windows `LoadLibraryEx()' routine.  The code must also use
     access routines and data in `python_NN_.dll' (that is, Python's C
     API's) using pointers obtained by the Windows `GetProcAddress()'
     routine.  Macros can make using these pointers transparent to any
     C code that calls routines in Python's C API.

     Borland note: convert `python_NN_.lib' to OMF format using
     Coff2Omf.exe first.

  2. If you use SWIG, it is easy to create a Python "extension module"
     that will make the app's data and methods available to Python.
     SWIG will handle just about all the grungy details for you.  The
     result is C code that you link _into_ your .exe file (!)  You do
     _not_ have to create a DLL file, and this also simplifies linking.

  3. SWIG will create an init function (a C function) whose name
     depends on the name of the extension module.  For example, if the
     name of the module is leo, the init function will be called
     initleo().  If you use SWIG shadow classes, as you should, the
     init function will be called initleoc().  This initializes a
     mostly hidden helper class used by the shadow class.

     The reason you can link the C code in step 2 into your .exe file
     is that calling the initialization function is equivalent to
     importing the module into Python! (This is the second key
     undocumented fact.)

  4. In short, you can use the following code to initialize the Python
     interpreter with your extension module.

         #include "python.h"
         ...
         Py_Initialize();  // Initialize Python.
         initmyAppc();  // Initialize (import) the helper class.
         PyRun_SimpleString("import myApp") ;  // Import the shadow class.

  5. There are two problems with Python's C API which will become
     apparent if you use a compiler other than MSVC, the compiler used
     to build pythonNN.dll.

     Problem 1: The so-called "Very High Level" functions that take
     FILE * arguments will not work in a multi-compiler environment
     because each compiler's notion of a struct FILE will be different.
     From an implementation standpoint these are very _low_ level
     functions.

     Problem 2: SWIG generates the following code when generating
     wrappers to void functions:

         Py_INCREF(Py_None);
         _resultobj = Py_None;
         return _resultobj;

     Alas, Py_None is a macro that expands to a reference to a complex
     data structure called _Py_NoneStruct inside pythonNN.dll.  Again,
     this code will fail in a mult-compiler environment.  Replace such
     code by:

         return Py_BuildValue("");

     It may be possible to use SWIG's `%typemap' command to make the
     change automatically, though I have not been able to get this to
     work (I'm a complete SWIG newbie).

  6. Using a Python shell script to put up a Python interpreter window
     from inside your Windows app is not a good idea; the resulting
     window will be independent of your app's windowing system.
     Rather, you (or the wxPythonWindow class) should create a "native"
     interpreter window.  It is easy to connect that window to the
     Python interpreter.  You can redirect Python's i/o to _any_ object
     that supports read and write, so all you need is a Python object
     (defined in your extension module) that contains read() and
     write() methods.

12.6.7 How do I use Python for CGI?
-----------------------------------

On the Microsoft IIS server or on the Win95 MS Personal Web Server you
set up Python in the same way that you would set up any other scripting
engine.

  Run regedt32 and go to:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3SVC\Parameters\ScriptMap

and enter the following line (making any specific changes that your
system may need):

    .py :REG_SZ: c:\<path to python>\python.exe -u %s %s

This line will allow you to call your script with a simple reference
like: `http://yourserver/scripts/yourscript.py' provided "scripts" is an
"executable" directory for your server (which it usually is by
default).  The -u flag specifies unbuffered and binary mode
for stdin - needed when working with binary data.

  In addition, it is recommended that using ".py" may not be a good
idea for the file extensions when used in this context (you might want
to reserve `*.py' for support modules and use `*.cgi' or `*.cgp' for
"main program" scripts).

  In order to set up Internet Information Services 5 to use Python for
CGI processing, please see the following links:

     <http://www.e-coli.net/pyiis_server.html> (for Win2k Server)
     <http://www.e-coli.net/pyiis.html> (for Win2k pro)

  Configuring Apache is much simpler.  In the Apache configuration file
`httpd.conf', add the following line at the end of the file:

    ScriptInterpreterSource Registry

Then, give your Python CGI-scripts the extension .py and put them in
the cgi-bin directory.

12.6.8 How do I keep editors from inserting tabs into my Python source?
-----------------------------------------------------------------------

The FAQ does not recommend using tabs, and the Python style guide, PEP
8 (http://www.python.org/dev/peps/pep-0008), recommends 4 spaces for
distributed Python code; this is also the Emacs python-mode default.

  Under any editor, mixing tabs and spaces is a bad idea.  MSVC is no
different in this respect, and is easily configured to use spaces: Take
_Tools ‣ Options ‣ Tabs_, and for file type "Default" set "Tab size"
and "Indent size" to 4, and select the "Insert spaces" radio button.

  If you suspect mixed tabs and spaces are causing problems in leading
whitespace, run Python with the -t switch or run
`Tools/Scripts/tabnanny.py' to check a directory tree in batch mode.

12.6.9 How do I check for a keypress without blocking?
------------------------------------------------------

Use the msvcrt module.  This is a standard Windows-specific extension
module.  It defines a function `kbhit()' which checks whether a
keyboard hit is present, and `getch()' which gets one character without
echoing it.

12.6.10 How do I emulate os.kill() in Windows?
----------------------------------------------

Prior to Python 2.7 and 3.2, to terminate a process, you can use
ctypes.:

    import ctypes

    def kill(pid):
        """kill function for Win32"""
        kernel32 = ctypes.windll.kernel32
        handle = kernel32.OpenProcess(1, 0, pid)
        return (0 != kernel32.TerminateProcess(handle, 0))

In 2.7 and 3.2, see os.kill() is implemented similar to the
above function, with the additional feature of being able to send
CTRL+C and CTRL+BREAK to console subprocesses which are designed to
handle those signals. See os.kill() for further details.

12.6.11 Why does os.path.isdir() fail on NT shared directories?
---------------------------------------------------------------

The solution appears to be always append the "\" on the end of shared
drives.

    >>> import os
    >>> os.path.isdir( '\\\\rorschach\\public')
    0
    >>> os.path.isdir( '\\\\rorschach\\public\\')
    1

It helps to think of share points as being like drive letters.  Example:

    k: is not a directory
    k:\ is a directory
    k:\media is a directory
    k:\media\ is not a directory

The same rules apply if you substitute "k:" with "\conkyfoo":

    \\conky\foo  is not a directory
    \\conky\foo\ is a directory
    \\conky\foo\media is a directory
    \\conky\foo\media\ is not a directory

12.6.12 cgi.py (or other CGI programming) doesn't work sometimes on NT or win95!
--------------------------------------------------------------------------------

Be sure you have the latest python.exe, that you are using python.exe
rather than a GUI version of Python and that you have configured the
server to execute

    "...\python.exe -u ..."

for the CGI execution.  The -u. (unbuffered) option on NT
and Win95 prevents the interpreter from altering newlines in the
standard input and output.  Without it post/multipart requests will
seem to have the wrong length and binary (e.g. GIF) responses may get
garbled (resulting in broken images, PDF files, and other binary
downloads failing).

12.6.13 Why doesn't os.popen() work in PythonWin on NT?
-------------------------------------------------------

The reason that os.popen() doesn't work from within PythonWin is due to
a bug in Microsoft's C Runtime Library (CRT). The CRT assumes you have
a Win32 console attached to the process.

  You should use the win32pipe module's popen() instead which doesn't
depend on having an attached Win32 console.

  Example:

    import win32pipe
    f = win32pipe.popen('dir /c c:\\')
    print f.readlines()
    f.close()

12.6.14 Why doesn't os.popen()/win32pipe.popen() work on Win9x?
---------------------------------------------------------------

There is a bug in Win9x that prevents os.popen/win32pipe.popen* from
working. The good news is there is a way to work around this problem.
The Microsoft Knowledge Base article that you need to lookup is:
Q150956. You will find links to the knowledge base at:
<http://support.microsoft.com/>.

12.6.15 PyRun_SimpleFile() crashes on Windows but not on Unix; why?
-------------------------------------------------------------------

This is very sensitive to the compiler vendor, version and (perhaps)
even options.  If the FILE* structure in your embedding program isn't
the same as is assumed by the Python interpreter it won't work.

  The Python 1.5.* DLLs (`python15.dll') are all compiled with MS VC++
5.0 and with multithreading-DLL options (`/MD').

  If you can't change compilers or flags, try using
`Py_RunSimpleString()'.  A trick to get it to run an arbitrary file is
to construct a call to execfile() with the name of your
file as argument.

  Also note that you can not mix-and-match Debug and Release versions.
If you wish to use the Debug Multithreaded DLL, then your module _must_
have an "_d" appended to the base name.

12.6.16 Importing _tkinter fails on Windows 95/98: why?
-------------------------------------------------------

Sometimes, the import of _tkinter fails on Windows 95 or 98,
complaining with a message like the following:

    ImportError: DLL load failed: One of the library files needed
    to run this application cannot be found.

It could be that you haven't installed Tcl/Tk, but if you did install
Tcl/Tk, and the Wish application works correctly, the problem may be
that its installer didn't manage to edit the autoexec.bat file
correctly.  It tries to add a statement that changes the PATH
environment variable to include the Tcl/Tk 'bin' subdirectory, but
sometimes this edit doesn't quite work.  Opening it with notepad
usually reveals what the problem is.

  (One additional hint, noted by David Szafranski: you can't use long
filenames here; e.g. use `C:\PROGRA~1\Tcl\bin' instead of `C:\Program
Files\Tcl\bin'.)

12.6.17 How do I extract the downloaded documentation on Windows?
-----------------------------------------------------------------

Sometimes, when you download the documentation package to a Windows
machine using a web browser, the file extension of the saved file ends
up being .EXE.  This is a mistake; the extension should be .TGZ.

  Simply rename the downloaded file to have the .TGZ extension, and
WinZip will be able to handle it.  (If your copy of WinZip doesn't, get
a newer one from <http://www.winzip.com>.)

12.6.18 Missing cw3215mt.dll (or missing cw3215.dll)
----------------------------------------------------

Sometimes, when using Tkinter on Windows, you get an error that
cw3215mt.dll or cw3215.dll is missing.

  Cause: you have an old Tcl/Tk DLL built with cygwin in your path
(probably `C:\Windows').  You must use the Tcl/Tk DLLs from the
standard Tcl/Tk installation (Python 1.5.2 comes with one).

12.6.19 Warning about CTL3D32 version from installer
----------------------------------------------------

The Python installer issues a warning like this:

    This version uses CTL3D32.DLL which is not the correct version.
    This version is used for windows NT applications only.

Tim Peters:

     This is a Microsoft DLL, and a notorious source of problems.  The
     message means what it says: you have the wrong version of this DLL
     for your operating system.  The Python installation did not cause
     this - something else you installed previous to this overwrote the
     DLL that came with your OS (probably older shareware of some sort,
     but there's no way to tell now).  If you search for "CTL3D32"
     using any search engine (AltaVista, for example), you'll find
     hundreds and hundreds of web pages complaining about the same
     problem with all sorts of installation programs.  They'll point
     you to ways to get the correct version reinstalled on your system
     (since Python doesn't cause this, we can't fix it).

  David A Burton has written a little program to fix this.  Go to
<http://www.burtonsys.com/downloads.html> and click on "ctl3dfix.zip".

12.7 Graphic User Interface FAQ
===============================

12.7.1 What platform-independent GUI toolkits exist for Python?
---------------------------------------------------------------

Depending on what platform(s) you are aiming at, there are several.

12.7.1.1 Tkinter
................

Standard builds of Python include an object-oriented interface to the
Tcl/Tk widget set, called Tkinter.  This is probably the easiest to
install and use.  For more info about Tk, including pointers to the
source, see the Tcl/Tk home page at <http://www.tcl.tk>.  Tcl/Tk is
fully portable to the MacOS, Windows, and Unix platforms.

12.7.1.2 wxWidgets
..................

wxWidgets (<http://www.wxwidgets.org>) is a free, portable GUI class
library written in C++ that provides a native look and feel on a number
of platforms, with Windows, MacOS X, GTK, X11, all listed as current
stable targets.  Language bindings are available for a number of
languages including Python, Perl, Ruby, etc.

  wxPython (<http://www.wxpython.org>) is the Python binding for
wxwidgets.  While it often lags slightly behind the official wxWidgets
releases, it also offers a number of features via pure Python
extensions that are not available in other language bindings.  There is
an active wxPython user and developer community.

  Both wxWidgets and wxPython are free, open source, software with
permissive licences that allow their use in commercial products as well
as in freeware or shareware.

12.7.1.3 Qt
...........

There are bindings available for the Qt toolkit (PyQt
(http://www.riverbankcomputing.co.uk/software/pyqt/)) and for KDE
(PyKDE (http://www.riverbankcomputing.co.uk/software/pykde/intro)).  If
you're writing open source software, you don't need to pay for PyQt,
but if you want to write proprietary applications, you must buy a PyQt
license from Riverbank Computing
(http://www.riverbankcomputing.co.uk)and (up to Qt 4.4; Qt 4.5 upwards
is licensed under the LGPL license) a Qt license from Trolltech
(http://www.trolltech.com).

12.7.1.4 Gtk+
.............

PyGtk bindings for the Gtk+ toolkit (http://www.gtk.org)have been
implemented by James Henstridge; see <<http://www.pygtk.org>>.

12.7.1.5 FLTK
.............

Python bindings for the FLTK toolkit (http://www.fltk.org), a simple yet
powerful and mature cross-platform windowing system, are available from
the PyFLTK project (http://pyfltk.sourceforge.net).

12.7.1.6 FOX
............

A wrapper for the FOX toolkit (http://www.fox-toolkit.org/)called FXpy
(http://fxpy.sourceforge.net/)is available.  FOX supports both Unix
variants and Windows.

12.7.1.7 OpenGL
...............

For OpenGL bindings, see PyOpenGL (http://pyopengl.sourceforge.net).

12.7.2 What platform-specific GUI toolkits exist for Python?
------------------------------------------------------------

The Mac port (http://python.org/download/mac)by Jack Jansen has a rich
and ever-growing set of modules that support the native Mac toolbox
calls.  The port supports MacOS X's Carbon libraries.

  By installing the PyObjc Objective-C bridge
(http://pyobjc.sourceforge.net), Python programs can use MacOS X's
Cocoa libraries. See the documentation that comes with the Mac port.

  Pythonwin by Mark Hammond includes an interface to the
Microsoft Foundation Classes and a Python programming environment
that's written mostly in Python using the MFC classes.

12.7.3 Tkinter questions
------------------------

12.7.3.1 How do I freeze Tkinter applications?
..............................................

Freeze is a tool to create stand-alone applications.  When freezing
Tkinter applications, the applications will not be truly stand-alone,
as the application will still need the Tcl and Tk libraries.

  One solution is to ship the application with the Tcl and Tk
libraries, and point to them at run-time using the `TCL_LIBRARY' and `TK_LIBRARY'
environment variables.

  To get truly stand-alone applications, the Tcl scripts that form the
library have to be integrated into the application as well. One tool
supporting that is SAM (stand-alone modules), which is part of the Tix
distribution (<http://tix.sourceforge.net/>).

  Build Tix with SAM enabled, perform the appropriate call to
`Tclsam_init()', etc. inside Python's `Modules/tkappinit.c', and link
with libtclsam and libtksam (you might include the Tix libraries as
well).

12.7.3.2 Can I have Tk events handled while waiting for I/O?
............................................................

Yes, and you don't even need threads!  But you'll have to restructure
your I/O code a bit.  Tk has the equivalent of Xt's `XtAddInput()'
call, which allows you to register a callback function which will be
called from the Tk mainloop when I/O is possible on a file descriptor.
Here's what you need:

    from Tkinter import tkinter
    tkinter.createfilehandler(file, mask, callback)

The file may be a Python file or socket object (actually, anything with
a fileno() method), or an integer file descriptor.  The mask is one of
the constants tkinter.READABLE or tkinter.WRITABLE.  The callback is
called as follows:

    callback(file, mask)

You must unregister the callback when you're done, using

    tkinter.deletefilehandler(file)

Note: since you don't know _how many bytes_ are available for reading,
you can't use the Python file object's read or readline methods, since
these will insist on reading a predefined number of bytes.  For
sockets, the `recv()' or `recvfrom()' methods will work fine; for other
files, use `os.read(file.fileno(), maxbytecount)'.

12.7.3.3 I can't get key bindings to work in Tkinter: why?
..........................................................

An often-heard complaint is that event handlers bound to events with the
`bind()' method don't get handled even when the appropriate key is
pressed.

  The most common cause is that the widget to which the binding applies
doesn't have "keyboard focus".  Check out the Tk documentation for the
focus command.  Usually a widget is given the keyboard focus by
clicking in it (but not for labels; see the takefocus option).

12.8 "Why is Python Installed on my Computer?" FAQ
==================================================

12.8.1 What is Python?
----------------------

Python is a programming language.  It's used for many different
applications.  It's used in some high schools and colleges as an
introductory programming language because Python is easy to learn, but
it's also used by professional software developers at places such as
Google, NASA, and Lucasfilm Ltd.

  If you wish to learn more about Python, start with the Beginner's
Guide to Python (http://wiki.python.org/moin/BeginnersGuide).

12.8.2 Why is Python installed on my machine?
---------------------------------------------

If you find Python installed on your system but don't remember
installing it, there are several possible ways it could have gotten
there.

   * Perhaps another user on the computer wanted to learn programming
     and installed it; you'll have to figure out who's been using the
     machine and might have installed it.

   * A third-party application installed on the machine might have been
     written in Python and included a Python installation.  For a home
     computer, the most common such application is PySol
     (http://pysolfc.sourceforge.net/), a solitaire game that includes
     over 1000 different games and variations.

   * Some Windows machines also have Python installed.  At this writing
     we're aware of computers from Hewlett-Packard and Compaq that
     include Python.  Apparently some of HP/Compaq's administrative
     tools are written in Python.

   * All Apple computers running Mac OS X have Python installed; it's
     included in the base installation.

12.8.3 Can I delete Python?
---------------------------

That depends on where Python came from.

  If someone installed it deliberately, you can remove it without
hurting anything.  On Windows, use the Add/Remove Programs icon in the
Control Panel.

  If Python was installed by a third-party application, you can also
remove it, but that application will no longer work.  You should use
that application's uninstaller rather than removing Python directly.

  If Python came with your operating system, removing it is not
recommended.  If you remove it, whatever tools were written in Python
will no longer run, and some of them might be important to you.
Reinstalling the whole system would then be required to fix things
again.

13 Glossary
***********

`>>>'
     The default Python prompt of the interactive shell.  Often seen
     for code examples which can be executed interactively in the
     interpreter.

`...'
     The default Python prompt of the interactive shell when entering
     code for an indented code block or within a pair of matching left
     and right delimiters (parentheses, square brackets or curly
     braces).

2to3
     A tool that tries to convert Python 2.x code to Python 3.x code by
     handling most of the incompatibilities which can be detected by
     parsing the source and traversing the parse tree.

     2to3 is available in the standard library as lib2to3.; a
     standalone entry point is provided as `Tools/scripts/2to3'.  See
     2to3 - Automated Python 2 to 3 code translation.

abstract base class
     Abstract Base Classes (abbreviated ABCs) complement
     duck-typing by providing a way to define interfaces when
     other techniques like hasattr() would be clumsy.
     Python comes with many built-in ABCs for data structures (in the
     collections module), numbers (in the numbers
     module), and streams (in the io module). You can
     create your own ABC with the abc module.

argument
     A value passed to a function or method, assigned to a named local
     variable in the function body.  A function or method may have both
     positional arguments and keyword arguments in its definition.
     Positional and keyword arguments may be variable-length: `*'
     accepts or passes (if in the function definition or call) several
     positional arguments in a list, while `**' does the same for
     keyword arguments in a dictionary.

     Any expression may be used within the argument list, and the
     evaluated value is passed to the local variable.

attribute
     A value associated with an object which is referenced by name using
     dotted expressions.  For example, if an object _o_ has an attribute
     _a_ it would be referenced as _o.a_.

BDFL
     Benevolent Dictator For Life, a.k.a. Guido van Rossum
     (http://www.python.org/~guido/), Python's creator.

bytecode
     Python source code is compiled into bytecode, the internal
     representation of a Python program in the interpreter.  The
     bytecode is also cached in `.pyc' and `.pyo' files so that
     executing the same file is faster the second time (recompilation
     from source to bytecode can be avoided).  This "intermediate
     language" is said to run on a virtual machine.  that
     executes the machine code corresponding to each bytecode.

     A list of bytecode instructions can be found in the documentation
     for the dis module.

class
     A template for creating user-defined objects. Class definitions
     normally contain method definitions which operate on instances of
     the class.

classic class
     Any class which does not inherit from object.  See
     new-style class.  Classic classes will be removed in
     Python 3.0.

coercion
     The implicit conversion of an instance of one type to another
     during an operation which involves two arguments of the same type.
     For example, `int(3.15)' converts the floating point number to the
     integer `3', but in `3+4.5', each argument is of a different type
     (one int, one float), and both must be converted to the same type
     before they can be added or it will raise a `TypeError'.  Coercion
     between two operands can be performed with the `coerce' built-in
     function; thus, `3+4.5' is equivalent to calling
     `operator.add(*coerce(3, 4.5))' and results in `operator.add(3.0,
     4.5)'.  Without coercion, all arguments of even compatible types
     would have to be normalized to the same value by the programmer,
     e.g., `float(3)+4.5' rather than just `3+4.5'.

complex number
     An extension of the familiar real number system in which all
     numbers are expressed as a sum of a real part and an imaginary
     part.  Imaginary numbers are real multiples of the imaginary unit
     (the square root of `-1'), often written `i' in mathematics or `j'
     in engineering.  Python has built-in support for complex numbers,
     which are written with this latter notation; the imaginary part is
     written with a `j' suffix, e.g., `3+1j'.  To get access to complex
     equivalents of the math module, use cmath.
     Use of complex numbers is a fairly advanced mathematical feature.
     If you're not aware of a need for them, it's almost certain you
     can safely ignore them.

context manager
     An object which controls the environment seen in a with.
     statement by defining __enter__() and
     __exit__() methods.  See PEP 343
     (http://www.python.org/dev/peps/pep-0343).

CPython
     The canonical implementation of the Python programming language.
     The term "CPython" is used in contexts when necessary to
     distinguish this implementation from others such as Jython or
     IronPython.

decorator
     A function returning another function, usually applied as a
     function transformation using the `@wrapper' syntax.  Common
     examples for decorators are classmethod() and
     staticmethod().

     The decorator syntax is merely syntactic sugar, the following two
     function definitions are semantically equivalent:

         def f(...):
             ...
         f = staticmethod(f)

         @staticmethod
         def f(...):
             ...

     See the documentation for function definition for more
     about decorators.

descriptor
     Any _new-style_ object which defines the methods __get__(),
     see __set__(), or __delete__().  When a
     class attribute is a descriptor, its special binding behavior is
     triggered upon attribute lookup.  Normally, using _a.b_ to get,
     set or delete an attribute looks up the object named _b_ in the
     class dictionary for _a_, but if _b_ is a descriptor, the
     respective descriptor method gets called.  Understanding
     descriptors is a key to a deep understanding of Python because
     they are the basis for many features including functions, methods,
     properties, class methods, static methods, and reference to super
     classes.

     For more information about descriptors' methods, see
     Implementing Descriptors.

dictionary
     An associative array, where arbitrary keys are mapped to values.
     The keys can be any object with __hash__() function
     and __eq__().  methods. Called a hash in Perl.

docstring
     A string literal which appears as the first expression in a class,
     function or module.  While ignored when the suite is executed, it
     is recognized by the compiler and put into the `__doc__' attribute
     of the enclosing class, function or module.  Since it is available
     via introspection, it is the canonical place for documentation of
     the object.

duck-typing
     A programming style which does not look at an object's type to
     determine if it has the right interface; instead, the method or
     attribute is simply called or used ("If it looks like a duck and
     quacks like a duck, it must be a duck.")  By emphasizing
     interfaces rather than specific types, well-designed code improves
     its flexibility by allowing polymorphic substitution.  Duck-typing
     avoids tests using type() or isinstance().
     (Note, however, that duck-typing can be complemented with
     abstract base class.es.)  Instead, it typically employs
     hasattr() tests or EAFP programming.

EAFP
     Easier to ask for forgiveness than permission.  This common Python
     coding style assumes the existence of valid keys or attributes and
     catches exceptions if the assumption proves false.  This clean and
     fast style is characterized by the presence of many try
     and except.  statements.  The technique contrasts
     with the LBYL style common to many other languages
     such as C.

expression
     A piece of syntax which can be evaluated to some value.  In other
     words, an expression is an accumulation of expression elements
     like literals, names, attribute access, operators or function
     calls which all return a value.  In contrast to many other
     languages, not all language constructs are expressions.  There are
     also statement.s which cannot be used as expressions,
     such as print or if.  Assignments are also
     statements, not expressions.

extension module
     A module written in C or C++, using Python's C API to interact
     with the core and with user code.

finder
     An object that tries to find the loader for a module.
     It must implement a method named `find_module()'. See PEP 302
     (http://www.python.org/dev/peps/pep-0302) for details.

floor division
     Mathematical division that rounds down to nearest integer.  The
     floor division operator is `//'.  For example, the expression `11
     // 4' evaluates to `2' in contrast to the `2.75' returned by float
     true division.  Note that `(-11) // 4' is `-3' because that is
     `-2.75' rounded _downward_. See PEP 238
     (http://www.python.org/dev/peps/pep-0238).

function
     A series of statements which returns some value to a caller. It
     can also be passed zero or more arguments which may be used in the
     execution of the body. See also argument and
     method.

__future__
     A pseudo-module which programmers can use to enable new language
     features which are not compatible with the current interpreter.
     For example, the expression `11/4' currently evaluates to `2'. If
     the module in which it is executed had enabled _true division_ by
     executing:

         from __future__ import division

     the expression `11/4' would evaluate to `2.75'.  By importing the
     __future__ module and evaluating its variables, you can
     see when a new feature was first added to the language and when it
     will become the default:

         >>> import __future__
         >>> __future__.division
         _Feature((2, 2, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), 8192)

garbage collection
     The process of freeing memory when it is not used anymore.  Python
     performs garbage collection via reference counting and a cyclic
     garbage collector that is able to detect and break reference
     cycles.

generator
     A function which returns an iterator.  It looks like a normal
     function except that it contains yield statements for
     producing a series a values usable in a for-loop or that can be
     retrieved one at a time with the next() function. Each
     yield temporarily suspends processing, remembering the
     location execution state (including local variables and pending
     try-statements).  When the generator resumes, it picks-up where it
     left-off (in contrast to functions which start fresh on every
     invocation).

generator expression
     An expression that returns an iterator.  It looks like a normal
     expression followed by a for expression defining a
     loop variable, range, and an optional if expression.
     The combined expression generates values for an enclosing function:

         >>> sum(i*i for i in range(10))         # sum of squares 0, 1, 4, ... 81
         285

GIL
     See global interpreter lock.

global interpreter lock
     The lock used by Python threads to assure that only one thread
     executes in the CPython. See virtual machine.
     at a time.  This simplifies the CPython implementation by assuring
     that no two processes can access the same memory at the same time.
     Locking the entire interpreter makes it easier for the interpreter
     to be multi-threaded, at the expense of much of the parallelism
     afforded by multi-processor machines.  Efforts have been made in
     the past to create a "free-threaded" interpreter (one which locks
     shared data at a much finer granularity), but so far none have
     been successful because performance suffered in the common
     single-processor case.

hashable
     An object is _hashable_ if it has a hash value which never changes
     during its lifetime (it needs a __hash__() method),
     and can be compared to other objects (it needs an __eq__()
     or __cmp__() method).  Hashable objects which
     compare equal must have the same hash value.

     Hashability makes an object usable as a dictionary key and a set
     member, because these data structures use the hash value
     internally.

     All of Python's immutable built-in objects are hashable, while no
     mutable containers (such as lists or dictionaries) are.  Objects
     which are instances of user-defined classes are hashable by
     default; they all compare unequal, and their hash value is their
     id().

IDLE
     An Integrated Development Environment for Python.  IDLE is a basic
     editor and interpreter environment which ships with the standard
     distribution of Python.

immutable
     An object with a fixed value.  Immutable objects include numbers,
     strings and tuples.  Such an object cannot be altered.  A new
     object has to be created if a different value has to be stored.
     They play an important role in places where a constant hash value
     is needed, for example as a key in a dictionary.

integer division
     Mathematical division discarding any remainder.  For example, the
     expression `11/4' currently evaluates to `2' in contrast to the
     `2.75' returned by float division.  Also called _floor division_.
     When dividing two integers the outcome will always be another
     integer (having the floor function applied to it). However, if one
     of the operands is another numeric type (such as a float),
     the result will be coerced (see coercion) to a
     common type.  For example, an integer divided by a float will
     result in a float value, possibly with a decimal fraction.
     Integer division can be forced by using the `//' operator instead
     of the `/' operator.  See also __future__.

importer
     An object that both finds and loads a module; both a finder
     and loader object.

interactive
     Python has an interactive interpreter which means you can enter
     statements and expressions at the interpreter prompt, immediately
     execute them and see their results.  Just launch `python' with no
     arguments (possibly by selecting it from your computer's main
     menu). It is a very powerful way to test out new ideas or inspect
     modules and packages (remember `help(x)').

interpreted
     Python is an interpreted language, as opposed to a compiled one,
     though the distinction can be blurry because of the presence of the
     bytecode compiler.  This means that source files can be run
     directly without explicitly creating an executable which is then
     run.  Interpreted languages typically have a shorter
     development/debug cycle than compiled ones, though their programs
     generally also run more slowly.  See also interactive.

iterable
     A container object capable of returning its members one at a time.
     Examples of iterables include all sequence types (such as
     list, str, and tuple) and some
     non-sequence types like dict and file and
     objects of any classes you define with an __iter__().
     or __getitem__() method.  Iterables can be used in a
     for loop and in many other places where a sequence is
     needed (see zip(), map(), ...).  When an iterable
     object is passed as an argument to the built-in function
     iter(), it returns an iterator for the object.  This iterator
     is good for one pass over the set of values.  When using
     iterables, it is usually not necessary to call iter().
     or deal with iterator objects yourself.  The `for' statement does
     that automatically for you, creating a temporary unnamed variable
     to hold the iterator for the duration of the loop.  See also
     iterator, sequence, and generator.

iterator
     An object representing a stream of data.  Repeated calls to the
     iterator's next() method return successive items in
     the stream.  When no more data are available a
     StopIteration exception is raised instead.  At this point,
     the iterator object is exhausted and any further calls to its
     next() method just raise StopIteration.
     again.  Iterators are required to have an __iter__().
     method that returns the iterator object itself so every iterator
     is also iterable and may be used in most places where other
     iterables are accepted.  One notable exception is code which
     attempts multiple iteration passes.  A container object (such as a
     list) produces a fresh new iterator each time you pass
     it to the iter() function or use it in a for
     loop.  Attempting this with an iterator will just return the
     same exhausted iterator object used in the previous iteration
     pass, making it appear like an empty container.

     More information can be found in Iterator Types.

key function
     A key function or collation function is a callable that returns a
     value used for sorting or ordering.  For example, see
     locale.strxfrm() is used to produce a sort key that is
     aware of locale specific sort conventions.

     A number of tools in Python accept key functions to control how
     elements are ordered or grouped.  They include min(),
     max(), sorted(), `list.sort()', see
     heapq.nsmallest(), heapq.nlargest(), and
     itertools.groupby().

     There are several ways to create a key function.  For example. the
     str.lower() method can serve as a key function for
     case insensitive sorts.  Alternatively, an ad-hoc key function can
     be built from a lambda expression such as `lambda r:
     (r[0], r[2])'.  Also, the operator module provides
     three key function constuctors: attrgetter(),
     itemgetter(), and methodcaller().  See the
     Sorting HOW TO for examples of how to create and use key
     functions.

keyword argument
     Arguments which are preceded with a `variable_name=' in the call.
     The variable name designates the local name in the function to
     which the value is assigned.  `**' is used to accept or pass a
     dictionary of keyword arguments.  See argument.

key function
     A key function or collation function is a callable that returns a
     value used for sorting or ordering.  For example, see
     locale.strxfrm() is used to produce a sort key that is
     aware of locale specific sort conventions.

     A number of tools in Python accept key functions to control how
     elements are ordered or grouped.  They include min(),
     max(), sorted(), `list.sort()', see
     heapq.nsmallest(), heapq.nlargest(), and
     itertools.groupby().

     There are several ways to create a key function.  For example. the
     str.lower() method can serve as a key function for
     case insensitive sorts.  Alternatively, an ad-hoc key function can
     be built from a lambda expression such as `lambda r:
     (r[0], r[2])'.  Also, the operator module provides
     three key function constuctors: attrgetter(),
     itemgetter(), and methodcaller().  See the
     Sorting HOW TO for examples of how to create and use key
     functions.

lambda
     An anonymous inline function consisting of a single
     expression.  which is evaluated when the function is called.
     The syntax to create a lambda function is `lambda [arguments]:
     expression'

LBYL
     Look before you leap.  This coding style explicitly tests for
     pre-conditions before making calls or lookups.  This style
     contrasts with the EAFP approach and is characterized
     by the presence of many if statements.

list
     A built-in Python sequence.  Despite its name it is
     more akin to an array in other languages than to a linked list
     since access to elements are O(1).

list comprehension
     A compact way to process all or part of the elements in a sequence
     and return a list with the results.  `result = ["0x%02x" % x for x
     in range(256) if x % 2 == 0]' generates a list of strings
     containing even hex numbers (0x..) in the range from 0 to 255. The
     if.  clause is optional.  If omitted, all elements in
     `range(256)' are processed.

loader
     An object that loads a module. It must define a method named
     `load_module()'. A loader is typically returned by a finder.
     See PEP 302 (http://www.python.org/dev/peps/pep-0302) for
     details.

mapping
     A container object (such as dict) which supports
     arbitrary key lookups using the special method
     __getitem__().  Mappings also support __len__(),
     __iter__(), and __contains__().

metaclass
     The class of a class.  Class definitions create a class name, a
     class dictionary, and a list of base classes.  The metaclass is
     responsible for taking those three arguments and creating the
     class.  Most object oriented programming languages provide a
     default implementation.  What makes Python special is that it is
     possible to create custom metaclasses.  Most users never need this
     tool, but when the need arises, metaclasses can provide powerful,
     elegant solutions.  They have been used for logging attribute
     access, adding thread-safety, tracking object creation,
     implementing singletons, and many other tasks.

     More information can be found in Customizing class creation.

method
     A function which is defined inside a class body.  If called as an
     attribute of an instance of that class, the method will get the
     instance object as its first argument. (which is
     usually called `self').  See function and
     nested scope.

mutable
     Mutable objects can change their value but keep their id().
     See also immutable.

named tuple
     Any tuple-like class whose indexable elements are also accessible
     using named attributes (for example, see time.localtime().
     returns a tuple-like object where the _year_ is accessible either
     with an index such as `t[0]' or with a named attribute like
     `t.tm_year').

     A named tuple can be a built-in type such as
     time.struct_time, or it can be created with a regular class
     definition.  A full featured named tuple can also be created with
     the factory function collections.namedtuple().  The
     latter approach automatically provides extra features such as a
     self-documenting representation like `Employee(name='jones',
     title='programmer')'.

namespace
     The place where a variable is stored.  Namespaces are implemented
     as dictionaries.  There are the local, global and built-in
     namespaces as well as nested namespaces in objects (in methods).
     Namespaces support modularity by preventing naming conflicts.  For
     instance, the functions `__builtin__.open()' and os.open()
     are distinguished by their namespaces.  Namespaces also aid
     readability and maintainability by making it clear which module
     implements a function.  For instance, writing random.seed()
     or itertools.izip() makes it clear that those
     functions are implemented by the random and
     itertools.  modules, respectively.

nested scope
     The ability to refer to a variable in an enclosing definition.  For
     instance, a function defined inside another function can refer to
     variables in the outer function.  Note that nested scopes work
     only for reference and not for assignment which will always write
     to the innermost scope.  In contrast, local variables both read
     and write in the innermost scope.  Likewise, global variables read
     and write to the global namespace.

new-style class
     Any class which inherits from object.  This includes
     all built-in types like list and dict.
     Only new-style classes can use Python's newer, versatile features
     like __slots__, descriptors, properties, and
     __getattribute__().

     More information can be found in New-style and classic classes.

object
     Any data with state (attributes or value) and defined behavior
     (methods).  Also the ultimate base class of any new-style class.

positional argument
     The arguments assigned to local names inside a function or method,
     determined by the order in which they were given in the call.  `*'
     is used to either accept multiple positional arguments (when in the
     definition), or pass several arguments as a list to a function.
     See argument.

Python 3000
     Nickname for the next major Python version, 3.0 (coined long ago
     when the release of version 3 was something in the distant
     future.)  This is also abbreviated "Py3k".

Pythonic
     An idea or piece of code which closely follows the most common
     idioms of the Python language, rather than implementing code using
     concepts common to other languages.  For example, a common idiom
     in Python is to loop over all elements of an iterable using a
     for.  statement.  Many other languages don't have this
     type of construct, so people unfamiliar with Python sometimes use
     a numerical counter instead:

         for i in range(len(food)):
             print food[i]

     As opposed to the cleaner, Pythonic method:

         for piece in food:
             print piece

reference count
     The number of references to an object.  When the reference count
     of an object drops to zero, it is deallocated.  Reference counting
     is generally not visible to Python code, but it is a key element
     of the CPython implementation.  The sys.
     module defines a `getrefcount()' function that programmers can
     call to return the reference count for a particular object.

__slots__
     A declaration inside a new-style class that saves
     memory by pre-declaring space for instance attributes and
     eliminating instance dictionaries.  Though popular, the technique
     is somewhat tricky to get right and is best reserved for rare
     cases where there are large numbers of instances in a
     memory-critical application.

sequence
     An iterable which supports efficient element access
     using integer indices via the __getitem__() special
     method and defines a len() method that returns the
     length of the sequence.  Some built-in sequence types are
     list, str, tuple, and unicode.
     Note that dict also supports __getitem__()
     and __len__(), but is considered a mapping rather
     than a sequence because the lookups use arbitrary immutable
     keys rather than integers.

slice
     An object usually containing a portion of a sequence.
     A slice is created using the subscript notation, `[]' with colons
     between numbers when several are given, such as in
     `variable_name[1:3:5]'.  The bracket (subscript) notation uses
     slice objects internally (or in older versions, see
     __getslice__() and __setslice__()).

special method
     A method that is called implicitly by Python to execute a certain
     operation on a type, such as addition.  Such methods have names
     starting and ending with double underscores.  Special methods are
     documented in Special method names.

statement
     A statement is part of a suite (a "block" of code).  A statement
     is either an expression or a one of several
     constructs with a keyword, such as if, while
     or print.

triple-quoted string
     A string which is bound by three instances of either a quotation
     mark (") or an apostrophe (').  While they don't provide any
     functionality not available with single-quoted strings, they are
     useful for a number of reasons.  They allow you to include
     unescaped single and double quotes within a string and they can
     span multiple lines without the use of the continuation character,
     making them especially useful when writing docstrings.

type
     The type of a Python object determines what kind of object it is;
     every object has a type.  An object's type is accessible as its
     `__class__' attribute or can be retrieved with `type(obj)'.

view
     The objects returned from dict.viewkeys(),
     dict.viewvalues(), and dict.viewitems() are
     called dictionary views.  They are lazy sequences that will see
     changes in the underlying dictionary.  To force the dictionary
     view to become a full list use `list(dictview)'.  See
     Dictionary view objects.

virtual machine
     A computer defined entirely in software.  Python's virtual machine
     executes the bytecode emitted by the bytecode compiler.

Zen of Python
     Listing of Python design principles and philosophies that are
     helpful in understanding and using the language.  The listing can
     be found by typing "`import this'" at the interactive prompt.

14 About these documents
************************

These documents are generated from reStructuredText
(http://docutils.sf.net/rst.html) sources by Sphinx
(http://sphinx.pocoo.org/), a document processor specifically written
for the Python documentation.

  Development of the documentation and its toolchain takes place on the
<docs@python.org> mailing list.  We're always looking for volunteers
wanting to help with the docs, so feel free to send a mail there!

  Many thanks go to:

   * Fred L. Drake, Jr., the creator of the original Python
     documentation toolset and writer of much of the content;

   * the Docutils (http://docutils.sf.net/) project for creating
     reStructuredText and the Docutils suite;

   * Fredrik Lundh for his Alternative Python Reference
     (http://effbot.org/zone/pyref.htm)project from which Sphinx got
     many good ideas.

  See Reporting Bugs for information how to report bugs in
this documentation, or Python itself.

14.1 Contributors to the Python Documentation
=============================================

This section lists people who have contributed in some way to the Python
documentation.  It is probably not complete - if you feel that you or
anyone else should be on this list, please let us know (send email to
<docs@python.org>), and we'll be glad to correct the problem.

   * Aahz

   * Michael Abbott

   * Steve Alexander

   * Jim Ahlstrom

   * Fred Allen

   * A. Amoroso

   * Pehr Anderson

   * Oliver Andrich

   * Heidi Annexstad

   * Jesús Cea Avión

   * Manuel Balsera

   * Daniel Barclay

   * Chris Barker

   * Don Bashford

   * Anthony Baxter

   * Alexander Belopolsky

   * Bennett Benson

   * Jonathan Black

   * Robin Boerdijk

   * Michal Bozon

   * Aaron Brancotti

   * Georg Brandl

   * Keith Briggs

   * Ian Bruntlett

   * Lee Busby

   * Lorenzo M. Catucci

   * Carl Cerecke

   * Mauro Cicognini

   * Gilles Civario

   * Mike Clarkson

   * Steve Clift

   * Dave Cole

   * Matthew Cowles

   * Jeremy Craven

   * Andrew Dalke

   * Ben Darnell

   * L. Peter Deutsch

   * Robert Donohue

   * Fred L. Drake, Jr.

   * Josip Dzolonga

   * Jeff Epler

   * Michael Ernst

   * Blame Andy Eskilsson

   * Carey Evans

   * Martijn Faassen

   * Carl Feynman

   * Dan Finnie

   * Hernán Martínez Foffani

   * Stefan Franke

   * Jim Fulton

   * Peter Funk

   * Lele Gaifax

   * Matthew Gallagher

   * Gabriel Genellina

   * Ben Gertzfield

   * Nadim Ghaznavi

   * Jonathan Giddy

   * Shelley Gooch

   * Nathaniel Gray

   * Grant Griffin

   * Thomas Guettler

   * Anders Hammarquist

   * Mark Hammond

   * Harald Hanche-Olsen

   * Manus Hand

   * Gerhard Häring

   * Travis B. Hartwell

   * Tim Hatch

   * Janko Hauser

   * Thomas Heller

   * Bernhard Herzog

   * Magnus L. Hetland

   * Konrad Hinsen

   * Stefan Hoffmeister

   * Albert Hofkamp

   * Gregor Hoffleit

   * Steve Holden

   * Thomas Holenstein

   * Gerrit Holl

   * Rob Hooft

   * Brian Hooper

   * Randall Hopper

   * Michael Hudson

   * Eric Huss

   * Jeremy Hylton

   * Roger Irwin

   * Jack Jansen

   * Philip H. Jensen

   * Pedro Diaz Jimenez

   * Kent Johnson

   * Lucas de Jonge

   * Andreas Jung

   * Robert Kern

   * Jim Kerr

   * Jan Kim

   * Greg Kochanski

   * Guido Kollerie

   * Peter A. Koren

   * Daniel Kozan

   * Andrew M. Kuchling

   * Dave Kuhlman

   * Erno Kuusela

   * Thomas Lamb

   * Detlef Lannert

   * Piers Lauder

   * Glyph Lefkowitz

   * Robert Lehmann

   * Marc-André Lemburg

   * Ross Light

   * Ulf A. Lindgren

   * Everett Lipman

   * Mirko Liss

   * Martin von Löwis

   * Fredrik Lundh

   * Jeff MacDonald

   * John Machin

   * Andrew MacIntyre

   * Vladimir Marangozov

   * Vincent Marchetti

   * Laura Matson

   * Daniel May

   * Rebecca McCreary

   * Doug Mennella

   * Paolo Milani

   * Skip Montanaro

   * Paul Moore

   * Ross Moore

   * Sjoerd Mullender

   * Dale Nagata

   * Ng Pheng Siong

   * Koray Oner

   * Tomas Oppelstrup

   * Denis S. Otkidach

   * Zooko O'Whielacronx

   * Shriphani Palakodety

   * William Park

   * Joonas Paalasmaa

   * Harri Pasanen

   * Bo Peng

   * Tim Peters

   * Benjamin Peterson

   * Christopher Petrilli

   * Justin D. Pettit

   * Chris Phoenix

   * François Pinard

   * Paul Prescod

   * Eric S. Raymond

   * Edward K. Ream

   * Terry J. Reedy

   * Sean Reifschneider

   * Bernhard Reiter

   * Armin Rigo

   * Wes Rishel

   * Armin Ronacher

   * Jim Roskind

   * Guido van Rossum

   * Donald Wallace Rouse II

   * Mark Russell

   * Nick Russo

   * Chris Ryland

   * Constantina S.

   * Hugh Sasse

   * Bob Savage

   * Scott Schram

   * Neil Schemenauer

   * Barry Scott

   * Joakim Sernbrant

   * Justin Sheehy

   * Charlie Shepherd

   * Michael Simcich

   * Ionel Simionescu

   * Michael Sloan

   * Gregory P. Smith

   * Roy Smith

   * Clay Spence

   * Nicholas Spies

   * Tage Stabell-Kulo

   * Frank Stajano

   * Anthony Starks

   * Greg Stein

   * Peter Stoehr

   * Mark Summerfield

   * Reuben Sumner

   * Kalle Svensson

   * Jim Tittsler

   * David Turner

   * Ville Vainio

   * Martijn Vries

   * Charles G. Waldman

   * Greg Ward

   * Barry Warsaw

   * Corran Webster

   * Glyn Webster

   * Bob Weiner

   * Eddy Welbourne

   * Jeff Wheeler

   * Mats Wichmann

   * Gerry Wiener

   * Timothy Wild

   * Paul Winkler

   * Collin Winter

   * Blake Winton

   * Dan Wolfe

   * Steven Work

   * Thomas Wouters

   * Ka-Ping Yee

   * Rory Yorke

   * Moshe Zadka

   * Milan Zamazal

   * Cheng Zhang

  It is only with the input and contributions of the Python community
that Python has such wonderful documentation - Thank You!

15 Reporting Bugs
*****************

Python is a mature programming language which has established a
reputation for stability.  In order to maintain this reputation, the
developers would like to know of any deficiencies you find in Python.

15.1 Documentation bugs
=======================

If you find a bug in this documentation or would like to propose an
improvement, please send an e-mail to <docs@python.org> describing the
bug and where you found it.  If you have a suggestion how to fix it,
include that as well.

  <docs@python.org> is a mailing list run by volunteers; your request
will be noticed, even if it takes a while to be processed.

  Of course, if you want a more persistent record of your issue, you
can use the issue tracker for documentation bugs as well.

15.2 Using the Python issue tracker
===================================

Bug reports for Python itself should be submitted via the Python Bug
Tracker (<http://bugs.python.org/>).  The bug tracker offers a Web form
which allows pertinent information to be entered and submitted to the
developers.

  The first step in filing a report is to determine whether the problem
has already been reported.  The advantage in doing so, aside from
saving the developers time, is that you learn what has been done to fix
it; it may be that the problem has already been fixed for the next
release, or additional information is needed (in which case you are
welcome to provide it if you can!).  To do this, search the bug
database using the search box on the top of the page.

  If the problem you're reporting is not already in the bug tracker, go
back to the Python Bug Tracker and log in.  If you don't already have a
tracker account, select the "Register" link or, if you use OpenID, one
of the OpenID provider logos in the sidebar.  It is not possible to
submit a bug report anonymously.

  Being now logged in, you can submit a bug.  Select the "Create New"
link in the sidebar to open the bug reporting form.

  The submission form has a number of fields.  For the "Title" field,
enter a _very_ short description of the problem; less than ten words is
good.  In the "Type" field, select the type of your problem; also
select the "Component" and "Versions" to which the bug relates.

  In the "Comment" field, describe the problem in detail, including
what you expected to happen and what did happen.  Be sure to include
whether any extension modules were involved, and what hardware and
software platform you were using (including version information as
appropriate).

  Each bug report will be assigned to a developer who will determine
what needs to be done to correct the problem.  You will receive an
update each time action is taken on the bug.  See
<http://www.python.org/dev/workflow/> for a detailed description of the
issue workflow.

How to Report Bugs Effectively (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
     Article which goes into some detail about how to create a useful
     bug report.  This describes what kind of information is useful and
     why it is useful.

Bug Writing Guidelines (http://developer.mozilla.org/en/docs/Bug_writing_guidelines)
     Information about writing a good bug report.  Some of this is
     specific to the Mozilla project, but describes general good
     practices.

16 Copyright
************

Python and this documentation is:

  Copyright © 2001-2010 Python Software Foundation. All rights reserved.

  Copyright © 2000 BeOpen.com. All rights reserved.

  Copyright © 1995-2000 Corporation for National Research Initiatives.
All rights reserved.

  Copyright © 1991-1995 Stichting Mathematisch Centrum. All rights
reserved.

______________________________________________________________________

See History and License for complete license and
permissions information.

17 History and License
**********************

17.1 History of the software
============================

Python was created in the early 1990s by Guido van Rossum at Stichting
Mathematisch Centrum (CWI, see <http://www.cwi.nl/>) in the Netherlands
as a successor of a language called ABC.  Guido remains Python's
principal author, although it includes many contributions from others.

  In 1995, Guido continued his work on Python at the Corporation for
National Research Initiatives (CNRI, see
<http://www.cnri.reston.va.us/>) in Reston, Virginia where he released
several versions of the software.

  In May 2000, Guido and the Python core development team moved to
BeOpen.com to form the BeOpen PythonLabs team.  In October of the same
year, the PythonLabs team moved to Digital Creations (now Zope
Corporation; see <http://www.zope.com/>).  In 2001, the Python Software
Foundation (PSF, see <http://www.python.org/psf/>) was formed, a
non-profit organization created specifically to own Python-related
Intellectual Property.  Zope Corporation is a sponsoring member of the
PSF.

  All Python releases are Open Source (see <http://www.opensource.org/>
for the Open Source Definition). Historically, most, but not all,
Python releases have also been GPL-compatible; the table below
summarizes the various releases.

Release              Derived from       Year            Owner            GPL compatible?
-----------------------------------------------------------------------------------------------
0.9.0 thru 1.2       n/a                1991-1995       CWI              yes
1.3 thru 1.5.2       1.2                1995-1999       CNRI             yes
1.6                  1.5.2              2000            CNRI             no
2.0                  1.6                2000            BeOpen.com       no
1.6.1                1.6                2001            CNRI             no
2.1                  2.0+1.6.1          2001            PSF              no
2.0.1                2.0+1.6.1          2001            PSF              yes
2.1.1                2.1+2.0.1          2001            PSF              yes
2.2                  2.1.1              2001            PSF              yes
2.1.2                2.1.1              2002            PSF              yes
2.1.3                2.1.2              2002            PSF              yes
2.2.1                2.2                2002            PSF              yes
2.2.2                2.2.1              2002            PSF              yes
2.2.3                2.2.2              2002-2003       PSF              yes
2.3                  2.2.2              2002-2003       PSF              yes
2.3.1                2.3                2002-2003       PSF              yes
2.3.2                2.3.1              2003            PSF              yes
2.3.3                2.3.2              2003            PSF              yes
2.3.4                2.3.3              2004            PSF              yes
2.3.5                2.3.4              2005            PSF              yes
2.4                  2.3                2004            PSF              yes
2.4.1                2.4                2005            PSF              yes
2.4.2                2.4.1              2005            PSF              yes
2.4.3                2.4.2              2006            PSF              yes
2.4.4                2.4.3              2006            PSF              yes
2.5                  2.4                2006            PSF              yes
2.5.1                2.5                2007            PSF              yes
2.5.2                2.5.1              2008            PSF              yes
2.5.3                2.5.2              2008            PSF              yes
2.6                  2.5                2008            PSF              yes
2.6.1                2.6                2008            PSF              yes
2.6.2                2.6.1              2009            PSF              yes
2.6.3                2.6.2              2009            PSF              yes
2.6.4                2.6.3              2010            PSF              yes
2.7                  2.6                2010            PSF              yes

     Note: GPL-compatible doesn't mean that we're distributing Python
     under the GPL.  All Python licenses, unlike the GPL, let you
     distribute a modified version without making your changes open
     source. The GPL-compatible licenses make it possible to combine
     Python with other software that is released under the GPL; the
     others don't.

  Thanks to the many outside volunteers who have worked under Guido's
direction to make these releases possible.

17.2 Terms and conditions for accessing or otherwise using Python
=================================================================

                PSF LICENSE AGREEMENT FOR PYTHON 2.7.1

  1. This LICENSE AGREEMENT is between the Python Software Foundation
     ("PSF"), and the Individual or Organization ("Licensee") accessing
     and otherwise using Python 2.7.1 software in source or binary form
     and its associated documentation.

  2. Subject to the terms and conditions of this License Agreement, PSF
     hereby grants Licensee a nonexclusive, royalty-free, world-wide
     license to reproduce, analyze, test, perform and/or display
     publicly, prepare derivative works, distribute, and otherwise use
     Python 2.7.1 alone or in any derivative version, provided,
     however, that PSF's License Agreement and PSF's notice of
     copyright, i.e., "Copyright © 2001-2010 Python Software
     Foundation; All Rights Reserved" are retained in Python 2.7.1
     alone or in any derivative version prepared by Licensee.

  3. In the event Licensee prepares a derivative work that is based on
     or incorporates Python 2.7.1 or any part thereof, and wants to
     make the derivative work available to others as provided herein,
     then Licensee hereby agrees to include in any such work a brief
     summary of the changes made to Python 2.7.1.

  4. PSF is making Python 2.7.1 available to Licensee on an "AS IS"
     basis.  PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
     IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
     DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR
     FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 2.7.1
     WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

  5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
     2.7.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR
     LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING
     PYTHON 2.7.1, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE
     POSSIBILITY THEREOF.

  6. This License Agreement will automatically terminate upon a
     material breach of its terms and conditions.

  7. Nothing in this License Agreement shall be deemed to create any
     relationship of agency, partnership, or joint venture between PSF
     and Licensee.  This License Agreement does not grant permission to
     use PSF trademarks or trade name in a trademark sense to endorse
     or promote products or services of Licensee, or any third party.

  8. By copying, installing or otherwise using Python 2.7.1, Licensee
     agrees to be bound by the terms and conditions of this License
     Agreement.

              BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0
         BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1

  1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an
     office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the
     Individual or Organization ("Licensee") accessing and otherwise
     using this software in source or binary form and its associated
     documentation ("the Software").

  2. Subject to the terms and conditions of this BeOpen Python License
     Agreement, BeOpen hereby grants Licensee a non-exclusive,
     royalty-free, world-wide license to reproduce, analyze, test,
     perform and/or display publicly, prepare derivative works,
     distribute, and otherwise use the Software alone or in any
     derivative version, provided, however, that the BeOpen Python
     License is retained in the Software, alone or in any derivative
     version prepared by Licensee.

  3. BeOpen is making the Software available to Licensee on an "AS IS"
     basis.  BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
     IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO
     AND DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR
     FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE
     WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

  4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE
     SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR
     LOSS AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE,
     OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY
     THEREOF.

  5. This License Agreement will automatically terminate upon a
     material breach of its terms and conditions.

  6. This License Agreement shall be governed by and interpreted in all
     respects by the law of the State of California, excluding conflict
     of law provisions.  Nothing in this License Agreement shall be
     deemed to create any relationship of agency, partnership, or joint
     venture between BeOpen and Licensee.  This License Agreement does
     not grant permission to use BeOpen trademarks or trade names in a
     trademark sense to endorse or promote products or services of
     Licensee, or any third party.  As an exception, the "BeOpen
     Python" logos available at <http://www.pythonlabs.com/logos.html>
     may be used according to the permissions granted on that web page.

  7. By copying, installing or otherwise using the software, Licensee
     agrees to be bound by the terms and conditions of this License
     Agreement.

                CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1

  1. This LICENSE AGREEMENT is between the Corporation for National
     Research Initiatives, having an office at 1895 Preston White
     Drive, Reston, VA 20191 ("CNRI"), and the Individual or
     Organization ("Licensee") accessing and otherwise using Python
     1.6.1 software in source or binary form and its associated
     documentation.

  2. Subject to the terms and conditions of this License Agreement,
     CNRI hereby grants Licensee a nonexclusive, royalty-free,
     world-wide license to reproduce, analyze, test, perform and/or
     display publicly, prepare derivative works, distribute, and
     otherwise use Python 1.6.1 alone or in any derivative version,
     provided, however, that CNRI's License Agreement and CNRI's notice
     of copyright, i.e., "Copyright © 1995-2001 Corporation for
     National Research Initiatives; All Rights Reserved" are retained
     in Python 1.6.1 alone or in any derivative version prepared by
     Licensee.  Alternately, in lieu of CNRI's License Agreement,
     Licensee may substitute the following text (omitting the quotes):
     "Python 1.6.1 is made available subject to the terms and
     conditions in CNRI's License Agreement.  This Agreement together
     with Python 1.6.1 may be located on the Internet using the
     following unique, persistent identifier (known as a handle):
     1895.22/1013.  This Agreement may also be obtained from a proxy
     server on the Internet using the following URL:
     <http://hdl.handle.net/1895.22/1013>."

  3. In the event Licensee prepares a derivative work that is based on
     or incorporates Python 1.6.1 or any part thereof, and wants to
     make the derivative work available to others as provided herein,
     then Licensee hereby agrees to include in any such work a brief
     summary of the changes made to Python 1.6.1.

  4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS"
     basis.  CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
     IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND
     DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR
     FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1
     WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

  5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
     1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR
     LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING
     PYTHON 1.6.1, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE
     POSSIBILITY THEREOF.

  6. This License Agreement will automatically terminate upon a
     material breach of its terms and conditions.

  7. This License Agreement shall be governed by the federal
     intellectual property law of the United States, including without
     limitation the federal copyright law, and, to the extent such U.S.
     federal law does not apply, by the law of the Commonwealth of
     Virginia, excluding Virginia's conflict of law provisions.
     Notwithstanding the foregoing, with regard to derivative works
     based on Python 1.6.1 that incorporate non-separable material that
     was previously distributed under the GNU General Public License
     (GPL), the law of the Commonwealth of Virginia shall govern this
     License Agreement only as to issues arising under or with respect
     to Paragraphs 4, 5, and 7 of this License Agreement.  Nothing in
     this License Agreement shall be deemed to create any relationship
     of agency, partnership, or joint venture between CNRI and
     Licensee.  This License Agreement does not grant permission to use
     CNRI trademarks or trade name in a trademark sense to endorse or
     promote products or services of Licensee, or any third party.

  8. By clicking on the "ACCEPT" button where indicated, or by copying,
     installing or otherwise using Python 1.6.1, Licensee agrees to be
     bound by the terms and conditions of this License Agreement.

                                ACCEPT
          CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2
  Copyright © 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The
Netherlands.  All rights reserved.

  Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted,
provided that the above copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation, and that the name of Stichting Mathematisch
Centrum or CWI not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior permission.

  STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD
TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE
FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

17.3 Licenses and Acknowledgements for Incorporated Software
============================================================

This section is an incomplete, but growing list of licenses and
acknowledgements for third-party software incorporated in the Python
distribution.

17.3.1 Mersenne Twister
-----------------------

The `_random' module includes code based on a download from
<http://www.math.keio.ac.jp/> matumoto/MT2002/emt19937ar.html. The
following are the verbatim comments from the original code:

    A C-program for MT19937, with initialization improved 2002/1/26.
    Coded by Takuji Nishimura and Makoto Matsumoto.

    Before using, initialize the state by using init_genrand(seed)
    or init_by_array(init_key, key_length).

    Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura,
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:

     1. Redistributions of source code must retain the above copyright
        notice, this list of conditions and the following disclaimer.

     2. Redistributions in binary form must reproduce the above copyright
        notice, this list of conditions and the following disclaimer in the
        documentation and/or other materials provided with the distribution.

     3. The names of its contributors may not be used to endorse or promote
        products derived from this software without specific prior written
        permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
    CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
    EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
    PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
    PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
    LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
    SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

    Any feedback is very welcome.
    http://www.math.keio.ac.jp/matumoto/emt.html
    email: matumoto@math.keio.ac.jp

17.3.2 Sockets
--------------

The socket module uses the functions, `getaddrinfo()', and
`getnameinfo()', which are coded in separate source files from the WIDE
Project, <http://www.wide.ad.jp/>.

    Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project.
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.
    3. Neither the name of the project nor the names of its contributors
       may be used to endorse or promote products derived from this software
       without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND
    GAI_ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE
    FOR GAI_ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON GAI_ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN GAI_ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.

17.3.3 Floating point exception control
---------------------------------------

The source for the fpectl module includes the following
notice:

      ---------------------------------------------------------------------
     /                       Copyright (c) 1996.                           \
    |          The Regents of the University of California.                 |
    |                        All rights reserved.                           |
    |                                                                       |
    |   Permission to use, copy, modify, and distribute this software for   |
    |   any purpose without fee is hereby granted, provided that this en-   |
    |   tire notice is included in all copies of any software which is or   |
    |   includes  a  copy  or  modification  of  this software and in all   |
    |   copies of the supporting documentation for such software.           |
    |                                                                       |
    |   This  work was produced at the University of California, Lawrence   |
    |   Livermore National Laboratory under  contract  no.  W-7405-ENG-48   |
    |   between  the  U.S.  Department  of  Energy and The Regents of the   |
    |   University of California for the operation of UC LLNL.              |
    |                                                                       |
    |                              DISCLAIMER                               |
    |                                                                       |
    |   This  software was prepared as an account of work sponsored by an   |
    |   agency of the United States Government. Neither the United States   |
    |   Government  nor the University of California nor any of their em-   |
    |   ployees, makes any warranty, express or implied, or  assumes  any   |
    |   liability  or  responsibility  for the accuracy, completeness, or   |
    |   usefulness of any information,  apparatus,  product,  or  process   |
    |   disclosed,   or  represents  that  its  use  would  not  infringe   |
    |   privately-owned rights. Reference herein to any specific  commer-   |
    |   cial  products,  process,  or  service  by trade name, trademark,   |
    |   manufacturer, or otherwise, does not  necessarily  constitute  or   |
    |   imply  its endorsement, recommendation, or favoring by the United   |
    |   States Government or the University of California. The views  and   |
    |   opinions  of authors expressed herein do not necessarily state or   |
    |   reflect those of the United States Government or  the  University   |
    |   of  California,  and shall not be used for advertising or product   |
     \  endorsement purposes.                                              /
      ---------------------------------------------------------------------

17.3.4 MD5 message digest algorithm
-----------------------------------

The source code for the md5 module contains the following
notice:

    Copyright (C) 1999, 2002 Aladdin Enterprises.  All rights reserved.

    This software is provided 'as-is', without any express or implied
    warranty.  In no event will the authors be held liable for any damages
    arising from the use of this software.

    Permission is granted to anyone to use this software for any purpose,
    including commercial applications, and to alter it and redistribute it
    freely, subject to the following restrictions:

    1. The origin of this software must not be misrepresented; you must not
       claim that you wrote the original software. If you use this software
       in a product, an acknowledgment in the product documentation would be
       appreciated but is not required.
    2. Altered source versions must be plainly marked as such, and must not be
       misrepresented as being the original software.
    3. This notice may not be removed or altered from any source distribution.

    L. Peter Deutsch
    ghost@aladdin.com

    Independent implementation of MD5 (RFC 1321).

    This code implements the MD5 Algorithm defined in RFC 1321, whose
    text is available at
          http://www.ietf.org/rfc/rfc1321.txt
    The code is derived from the text of the RFC, including the test suite
    (section A.5) but excluding the rest of Appendix A.  It does not include
    any code or documentation that is identified in the RFC as being
    copyrighted.

    The original and principal author of md5.h is L. Peter Deutsch
    <ghost@aladdin.com>.  Other authors are noted in the change history
    that follows (in reverse chronological order):

    2002-04-13 lpd Removed support for non-ANSI compilers; removed
          references to Ghostscript; clarified derivation from RFC 1321;
          now handles byte order either statically or dynamically.
    1999-11-04 lpd Edited comments slightly for automatic TOC extraction.
    1999-10-18 lpd Fixed typo in header comment (ansi2knr rather than md5);
          added conditionalization for C++ compilation from Martin
          Purschke <purschke@bnl.gov>.
    1999-05-03 lpd Original version.

17.3.5 Asynchronous socket services
-----------------------------------

The asynchat and asyncore modules contain the
following notice:

    Copyright 1996 by Sam Rushing

                            All Rights Reserved

    Permission to use, copy, modify, and distribute this software and
    its documentation for any purpose and without fee is hereby
    granted, provided that the above copyright notice appear in all
    copies and that both that copyright notice and this permission
    notice appear in supporting documentation, and that the name of Sam
    Rushing not be used in advertising or publicity pertaining to
    distribution of the software without specific, written prior
    permission.

    SAM RUSHING DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
    INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN
    NO EVENT SHALL SAM RUSHING BE LIABLE FOR ANY SPECIAL, INDIRECT OR
    CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
    OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
    NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

17.3.6 Cookie management
------------------------

The Cookie module contains the following notice:

    Copyright 2000 by Timothy O'Malley <timo@alum.mit.edu>

                   All Rights Reserved

    Permission to use, copy, modify, and distribute this software
    and its documentation for any purpose and without fee is hereby
    granted, provided that the above copyright notice appear in all
    copies and that both that copyright notice and this permission
    notice appear in supporting documentation, and that the name of
    Timothy O'Malley  not be used in advertising or publicity
    pertaining to distribution of the software without specific, written
    prior permission.

    Timothy O'Malley DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
    SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
    AND FITNESS, IN NO EVENT SHALL Timothy O'Malley BE LIABLE FOR
    ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
    WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
    ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
    PERFORMANCE OF THIS SOFTWARE.

17.3.7 Profiling
----------------

The profile and pstats modules contain the
following notice:

    Copyright 1994, by InfoSeek Corporation, all rights reserved.
    Written by James Roskind

    Permission to use, copy, modify, and distribute this Python software
    and its associated documentation for any purpose (subject to the
    restriction in the following sentence) without fee is hereby granted,
    provided that the above copyright notice appears in all copies, and
    that both that copyright notice and this permission notice appear in
    supporting documentation, and that the name of InfoSeek not be used in
    advertising or publicity pertaining to distribution of the software
    without specific, written prior permission.  This permission is
    explicitly restricted to the copying and modification of the software
    to remain in Python, compiled Python, or other languages (such as C)
    wherein the modified or derived code is exclusively imported into a
    Python module.

    INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
    SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
    FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY
    SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
    RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
    CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

17.3.8 Execution tracing
------------------------

The trace module contains the following notice:

    portions copyright 2001, Autonomous Zones Industries, Inc., all rights...
    err...  reserved and offered to the public under the terms of the
    Python 2.2 license.
    Author: Zooko O'Whielacronx
    http://zooko.com/
    mailto:zooko@zooko.com

    Copyright 2000, Mojam Media, Inc., all rights reserved.
    Author: Skip Montanaro

    Copyright 1999, Bioreason, Inc., all rights reserved.
    Author: Andrew Dalke

    Copyright 1995-1997, Automatrix, Inc., all rights reserved.
    Author: Skip Montanaro

    Copyright 1991-1995, Stichting Mathematisch Centrum, all rights reserved.

    Permission to use, copy, modify, and distribute this Python software and
    its associated documentation for any purpose without fee is hereby
    granted, provided that the above copyright notice appears in all copies,
    and that both that copyright notice and this permission notice appear in
    supporting documentation, and that the name of neither Automatrix,
    Bioreason or Mojam Media be used in advertising or publicity pertaining to
    distribution of the software without specific, written prior permission.

17.3.9 UUencode and UUdecode functions
--------------------------------------

The uu module contains the following notice:

    Copyright 1994 by Lance Ellinghouse
    Cathedral City, California Republic, United States of America.
                           All Rights Reserved
    Permission to use, copy, modify, and distribute this software and its
    documentation for any purpose and without fee is hereby granted,
    provided that the above copyright notice appear in all copies and that
    both that copyright notice and this permission notice appear in
    supporting documentation, and that the name of Lance Ellinghouse
    not be used in advertising or publicity pertaining to distribution
    of the software without specific, written prior permission.
    LANCE ELLINGHOUSE DISCLAIMS ALL WARRANTIES WITH REGARD TO
    THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
    FITNESS, IN NO EVENT SHALL LANCE ELLINGHOUSE CENTRUM BE LIABLE
    FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
    ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
    OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

    Modified by Jack Jansen, CWI, July 1995:
    - Use binascii module to do the actual line-by-line conversion
      between ascii and binary. This results in a 1000-fold speedup. The C
      version is still 5 times faster, though.
    - Arguments more compliant with Python standard

17.3.10 XML Remote Procedure Calls
----------------------------------

The xmlrpclib module contains the following notice:

        The XML-RPC client interface is

    Copyright (c) 1999-2002 by Secret Labs AB
    Copyright (c) 1999-2002 by Fredrik Lundh

    By obtaining, using, and/or copying this software and/or its
    associated documentation, you agree that you have read, understood,
    and will comply with the following terms and conditions:

    Permission to use, copy, modify, and distribute this software and
    its associated documentation for any purpose and without fee is
    hereby granted, provided that the above copyright notice appears in
    all copies, and that both that copyright notice and this permission
    notice appear in supporting documentation, and that the name of
    Secret Labs AB or the author not be used in advertising or publicity
    pertaining to distribution of the software without specific, written
    prior permission.

    SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD
    TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT-
    ABILITY AND FITNESS.  IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR
    BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY
    DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
    WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
    ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
    OF THIS SOFTWARE.

17.3.11 test_epoll
------------------

The `test_epoll' contains the following notice:

    Copyright (c) 2001-2006 Twisted Matrix Laboratories.

    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the
    "Software"), to deal in the Software without restriction, including
    without limitation the rights to use, copy, modify, merge, publish,
    distribute, sublicense, and/or sell copies of the Software, and to
    permit persons to whom the Software is furnished to do so, subject to
    the following conditions:

    The above copyright notice and this permission notice shall be
    included in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
    MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
    NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
    LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
    WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

17.3.12 Select kqueue
---------------------

The select and contains the following notice for the kqueue
interface:

    Copyright (c) 2000 Doug White, 2006 James Knight, 2007 Christian Heimes
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.

17.3.13 strtod and dtoa
-----------------------

The file `Python/dtoa.c', which supplies C functions dtoa and strtod
for conversion of C doubles to and from strings, is derived from the
file of the same name by David M. Gay, currently available from
<http://www.netlib.org/fp/>.  The original file, as retrieved on March
16, 2009, contains the following copyright and licensing notice:

    /****************************************************************
     *
     * The author of this software is David M. Gay.
     *
     * Copyright (c) 1991, 2000, 2001 by Lucent Technologies.
     *
     * Permission to use, copy, modify, and distribute this software for any
     * purpose without fee is hereby granted, provided that this entire notice
     * is included in all copies of any software which is or includes a copy
     * or modification of this software and in all copies of the supporting
     * documentation for such software.
     *
     * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED
     * WARRANTY.  IN PARTICULAR, NEITHER THE AUTHOR NOR LUCENT MAKES ANY
     * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY
     * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE.
     *
     ***************************************************************/

17.3.14 OpenSSL
---------------

The modules hashlib, posix, ssl,
crypt use the OpenSSL library for added performance if made
available by the operating system. Additionally, the Windows installers
for Python include a copy of the OpenSSL libraries, so we include a
copy of the OpenSSL license here:

     LICENSE ISSUES
     ==============

     The OpenSSL toolkit stays under a dual license, i.e. both the conditions of
     the OpenSSL License and the original SSLeay license apply to the toolkit.
     See below for the actual license texts. Actually both licenses are BSD-style
     Open Source licenses. In case of any license issues related to OpenSSL
     please contact openssl-core@openssl.org.

     OpenSSL License
     ---------------

       /* ====================================================================
        * Copyright (c) 1998-2008 The OpenSSL Project.  All rights reserved.
        *
        * Redistribution and use in source and binary forms, with or without
        * modification, are permitted provided that the following conditions
        * are met:
        *
        * 1. Redistributions of source code must retain the above copyright
        *    notice, this list of conditions and the following disclaimer.
        *
        * 2. Redistributions in binary form must reproduce the above copyright
        *    notice, this list of conditions and the following disclaimer in
        *    the documentation and/or other materials provided with the
        *    distribution.
        *
        * 3. All advertising materials mentioning features or use of this
        *    software must display the following acknowledgment:
        *    "This product includes software developed by the OpenSSL Project
        *    for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
        *
        * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
        *    endorse or promote products derived from this software without
        *    prior written permission. For written permission, please contact
        *    openssl-core@openssl.org.
        *
        * 5. Products derived from this software may not be called "OpenSSL"
        *    nor may "OpenSSL" appear in their names without prior written
        *    permission of the OpenSSL Project.
        *
        * 6. Redistributions of any form whatsoever must retain the following
        *    acknowledgment:
        *    "This product includes software developed by the OpenSSL Project
        *    for use in the OpenSSL Toolkit (http://www.openssl.org/)"
        *
        * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
        * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
        * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
        * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
        * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
        * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
        * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
        * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
        * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
        * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
        * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
        * OF THE POSSIBILITY OF SUCH DAMAGE.
        * ====================================================================
        *
        * This product includes cryptographic software written by Eric Young
        * (eay@cryptsoft.com).  This product includes software written by Tim
        * Hudson (tjh@cryptsoft.com).
        *
        */

    Original SSLeay License
    -----------------------

       /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
        * All rights reserved.
        *
        * This package is an SSL implementation written
        * by Eric Young (eay@cryptsoft.com).
        * The implementation was written so as to conform with Netscapes SSL.
        *
        * This library is free for commercial and non-commercial use as long as
        * the following conditions are aheared to.  The following conditions
        * apply to all code found in this distribution, be it the RC4, RSA,
        * lhash, DES, etc., code; not just the SSL code.  The SSL documentation
        * included with this distribution is covered by the same copyright terms
        * except that the holder is Tim Hudson (tjh@cryptsoft.com).
        *
        * Copyright remains Eric Young's, and as such any Copyright notices in
        * the code are not to be removed.
        * If this package is used in a product, Eric Young should be given attribution
        * as the author of the parts of the library used.
        * This can be in the form of a textual message at program startup or
        * in documentation (online or textual) provided with the package.
        *
        * Redistribution and use in source and binary forms, with or without
        * modification, are permitted provided that the following conditions
        * are met:
        * 1. Redistributions of source code must retain the copyright
        *    notice, this list of conditions and the following disclaimer.
        * 2. Redistributions in binary form must reproduce the above copyright
        *    notice, this list of conditions and the following disclaimer in the
        *    documentation and/or other materials provided with the distribution.
        * 3. All advertising materials mentioning features or use of this software
        *    must display the following acknowledgement:
        *    "This product includes cryptographic software written by
        *     Eric Young (eay@cryptsoft.com)"
        *    The word 'cryptographic' can be left out if the rouines from the library
        *    being used are not cryptographic related :-).
        * 4. If you include any Windows specific code (or a derivative thereof) from
        *    the apps directory (application code) you must include an acknowledgement:
        *    "This product includes software written by Tim Hudson (tjh@cryptsoft.com)"
        *
        * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
        * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
        * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
        * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
        * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
        * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
        * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
        * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
        * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
        * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
        * SUCH DAMAGE.
        *
        * The licence and distribution terms for any publically available version or
        * derivative of this code cannot be changed.  i.e. this code cannot simply be
        * copied and put under another distribution licence
        * [including the GNU Public Licence.]
        */

17.3.15 expat
-------------

The `pyexpat' extension is built using an included copy of the expat
sources unless the build is configured `--with-system-expat':

    Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd
                                   and Clark Cooper

    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the
    "Software"), to deal in the Software without restriction, including
    without limitation the rights to use, copy, modify, merge, publish,
    distribute, sublicense, and/or sell copies of the Software, and to
    permit persons to whom the Software is furnished to do so, subject to
    the following conditions:

    The above copyright notice and this permission notice shall be included
    in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
    MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
    IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
    TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
    SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

17.3.16 libffi
--------------

The `_ctypes' extension is built using an included copy of the libffi
sources unless the build is configured `--with-system-libffi':

    Copyright (c) 1996-2008  Red Hat, Inc and others.

    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the
    ``Software''), to deal in the Software without restriction, including
    without limitation the rights to use, copy, modify, merge, publish,
    distribute, sublicense, and/or sell copies of the Software, and to
    permit persons to whom the Software is furnished to do so, subject to
    the following conditions:

    The above copyright notice and this permission notice shall be included
    in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,
    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
    MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
    NONINFRINGEMENT.  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
    HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
    WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    DEALINGS IN THE SOFTWARE.

17.3.17 zlib
------------

The zlib extension is built using an included copy of the
zlib sources unless the zlib version found on the system is too old to
be used for the build:

    Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler

    This software is provided 'as-is', without any express or implied
    warranty.  In no event will the authors be held liable for any damages
    arising from the use of this software.

    Permission is granted to anyone to use this software for any purpose,
    including commercial applications, and to alter it and redistribute it
    freely, subject to the following restrictions:

    1. The origin of this software must not be misrepresented; you must not
       claim that you wrote the original software. If you use this software
       in a product, an acknowledgment in the product documentation would be
       appreciated but is not required.

    2. Altered source versions must be plainly marked as such, and must not be
       misrepresented as being the original software.

    3. This notice may not be removed or altered from any source distribution.

    Jean-loup Gailly        Mark Adler
    jloup@gzip.org          madler@alumni.caltech.edu