Doug Hellmann

svnbackup 1.5

2008-07-07T08:23:00.001-04:00

Matthew McDonald contributed a patch to let svnbackup work on repositories with spaces in the name. I also added a --compress option so users can choose between gzip and bzip2 (the default is still bzip2).

The new release is available from the download page at the Google Code project site.

Thanks, Matthew!

PyMOTW: xmlrpclib

2008-07-06T17:06:00.001-04:00

As a follow-up to last week's article on SimpleXMLRPCServer, this week covers the client-side library xmlrpclib.

Module: xmlrpclib
Purpose: Client-side library for XML-RPC communication.
Python Version: 2.2 and later

Description:

Last week we looked at the library for creating an XML-RPC server. The xmlrpclib module lets you communicate from Python with any XML-RPC server written in any language. All of the examples below use the server defined in xmlrpclib_server.py, available in the source distribution and repeated here for reference:

Connecting to a Server:

The simplest way to connect a client to a server is to instantiate a ServerProxy object, giving it the URI of the server. For example, the demo server runs on port 9000 of localhost:

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000')
print 'Ping:', server.ping()

In this case, the ping() method of the service takes no arguments and returns a single boolean value.


$ python xmlrpclib_ServerProxy.py
Ping: True

Other options are available to support alternate transport. Both HTTP and HTTPS are supported out of the box, as are basic authentication. You would only need to provide a transport class if your communication channel was not one of the supported types. It would be an interesting exercise, for example, to implement XML-RPC over SMTP. Not terribly useful, but interesting.

The verbose option gives you debugging information useful for working out where communication errors might be happening.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000', verbose=True)
print 'Ping:', server.ping()


$ python xmlrpclib_ServerProxy_verbose.py
Ping: connect: (localhost, 9000)
connect fail: ('localhost', 9000)
connect: (localhost, 9000)
connect fail: ('localhost', 9000)
connect: (localhost, 9000)
send: 'POST /RPC2 HTTP/1.0\r\nHost: localhost:9000\r\nUser-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: text/xml\r\nContent-Length: 98\r\n\r\n'
send: "<?xml version='1.0'?>\n<methodCall>\n<methodName>ping</methodName>\n<params>\n</params>\n</methodCall>\n"
reply: 'HTTP/1.0 200 OK\r\n'
header: Server: BaseHTTP/0.3 Python/2.5.1
header: Date: Sun, 06 Jul 2008 19:56:13 GMT
header: Content-type: text/xml
header: Content-length: 129
body: "<?xml version='1.0'?>\n<methodResponse>\n<params>\n<param>\n<value><boolean>1</boolean></value>\n</param>\n</params>\n</methodResponse>\n"
True

You can change the default encoding from UTF-8 if you need to use an alternate system.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000', encoding='ISO-8859-1')
print 'Ping:', server.ping()

The server should automatically detect the correct encoding.


$ python xmlrpclib_ServerProxy_encoding.py
Ping: True

The allow_none option controls whether Python's None value is automatically translated to a nil value or if it causes an error.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000', allow_none=True)
print 'Allowed:', server.show_type(None)

server = xmlrpclib.ServerProxy('http://localhost:9000', allow_none=False)
print 'Not allowed:', server.show_type(None)

Note that the error is raised locally if the client does not allow None, but can also be raised from within the server if it is not configured to allow None.


$ python xmlrpclib_ServerProxy_allow_none.py
Allowed: ['None', "<type 'NoneType'>", None]
Not allowed:
Traceback (most recent call last):
  File "/Users/dhellmann/Documents/PyMOTW/in_progress/xmlrpclib/xmlrpclib_ServerProxy_allow_none.py", line 17, in <module>
    print 'Not allowed:', server.show_type(None)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1431, in __request
    allow_none=self.__allow_none)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1080, in dumps
    data = m.dumps(params)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 623, in dumps
    dump(v, write)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 635, in __dump
    f(self, value, write)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 639, in dump_nil
    raise TypeError, "cannot marshal None unless allow_none is enabled"
TypeError: cannot marshal None unless allow_none is enabled

The use_datetime option lets you pass datetime.datetime and related objects in to the proxy or receive them from the server. If use_datetime is False, the internal DateTime class is used to represent dates instead.

Data Types:

The XML-RPC protocol recognizes a limited set of common data types. The types can be passed as arguments or return values and combined to create more complex data structures.

import xmlrpclib
import datetime

server = xmlrpclib.ServerProxy('http://localhost:9000')

for t, v in [ ('boolean', True), 
              ('integer', 1),
              ('floating-point number', 2.5),
              ('string', 'some text'), 
              ('datetime', datetime.datetime.now()),
              ('array', ['a', 'list']),
              ('array', ('a', 'tuple')),
              ('structure', {'a':'dictionary'}),
            ]:
    print '%-22s:' % t, server.show_type(v)

The simple types:


$ python xmlrpclib_types.py
boolean               : ['True', "<type 'bool'>", True]
integer               : ['1', "<type 'int'>", 1]
floating-point number : ['2.5', "<type 'float'>", 2.5]
string                : ['some text', "<type 'str'>", 'some text']
datetime              : ['20080706T16:22:49', "<type 'instance'>", <DateTime '20080706T16:22:49' at a5d030>]
array                 : ["['a', 'list']", "<type 'list'>", ['a', 'list']]
array                 : ["['a', 'tuple']", "<type 'list'>", ['a', 'tuple']]
structure             : ["{'a': 'dictionary'}", "<type 'dict'>", {'a': 'dictionary'}]

And of course, they can be nested to create values of arbitrary complexity:

import xmlrpclib
import datetime
import pprint

server = xmlrpclib.ServerProxy('http://localhost:9000')

data = { 'boolean':True, 
         'integer': 1,
         'floating-point number': 2.5,
         'string': 'some text',
         'datetime': datetime.datetime.now(),
         'array': ['a', 'list'],
         'array': ('a', 'tuple'),
         'structure': {'a':'dictionary'},
         }
arg = []
for i in range(3):
    d = {}
    d.update(data)
    d['integer'] = i
    arg.append(d)

print 'Before:'
pprint.pprint(arg)

print
print 'After:'
pprint.pprint(server.show_type(arg)[-1])


$ python xmlrpclib_types_nested.py
Before:
[{'array': ('a', 'tuple'),
  'boolean': True,
  'datetime': datetime.datetime(2008, 7, 6, 16, 24, 52, 348849),
  'floating-point number': 2.5,
  'integer': 0,
  'string': 'some text',
  'structure': {'a': 'dictionary'}},
 {'array': ('a', 'tuple'),
  'boolean': True,
  'datetime': datetime.datetime(2008, 7, 6, 16, 24, 52, 348849),
  'floating-point number': 2.5,
  'integer': 1,
  'string': 'some text',
  'structure': {'a': 'dictionary'}},
 {'array': ('a', 'tuple'),
  'boolean': True,
  'datetime': datetime.datetime(2008, 7, 6, 16, 24, 52, 348849),
  'floating-point number': 2.5,
  'integer': 2,
  'string': 'some text',
  'structure': {'a': 'dictionary'}}]

After:
[{'array': ['a', 'tuple'],
  'boolean': True,
  'datetime': <DateTime '20080706T16:24:52' at a5be18>,
  'floating-point number': 2.5,
  'integer': 0,
  'string': 'some text',
  'structure': {'a': 'dictionary'}},
 {'array': ['a', 'tuple'],
  'boolean': True,
  'datetime': <DateTime '20080706T16:24:52' at a5bf30>,
  'floating-point number': 2.5,
  'integer': 1,
  'string': 'some text',
  'structure': {'a': 'dictionary'}},
 {'array': ['a', 'tuple'],
  'boolean': True,
  'datetime': <DateTime '20080706T16:24:52' at a5bf80>,
  'floating-point number': 2.5,
  'integer': 2,
  'string': 'some text',
  'structure': {'a': 'dictionary'}}]

Passing Objects:

Instances of Python classes are treated as structures and passed as a dictionary, with the attributes of the object as values in the dictionary.

import xmlrpclib

class MyObj:
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __repr__(self):
        return 'MyObj(%s, %s)' % (repr(self.a), repr(self.b))

server = xmlrpclib.ServerProxy('http://localhost:9000')

o = MyObj(1, 'b goes here')
print 'o=', o
print server.show_type(o)

o2 = MyObj(2, o)
print 'o2=', o2
print server.show_type(o2)

Round-tripping the value gives a dictionary on the client, since there is nothing encoded in the values to tell the server (or client) that it should be instantiated as part of a class.


$ python xmlrpclib_types_object.py
o= MyObj(1, 'b goes here')
["{'a': 1, 'b': 'b goes here'}", "<type 'dict'>", {'a': 1, 'b': 'b goes here'}]
o2= MyObj(2, MyObj(1, 'b goes here'))
["{'a': 2, 'b': {'a': 1, 'b': 'b goes here'}}", "<type 'dict'>", {'a': 2, 'b': {'a': 1, 'b': 'b goes here'}}]

Binary Data:

All values passed to the server are encoded and escaped automatically. However, some data types may contain characters that are not valid XML. For example, binary image data may include byte values in the ASCII control range 0 to 31. If you need to pass binary data, it is best to use the Binary class to encode it for transport.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000')

s = 'This is a string with control characters' + '\0'
print 'Local string:', s

data = xmlrpclib.Binary(s)
print 'As binary:', server.send_back_binary(data)

print 'As string:', server.show_type(s)

If we pass the string containing a NULL byte to show_type(), an exception is raised in the XML parser:


$ python xmlrpclib_Binary.py
Local string: This is a string with control characters
As binary: This is a string with control characters
As string:
Traceback (most recent call last):
  File "/Users/dhellmann/Documents/PyMOTW/in_progress/xmlrpclib/xmlrpclib_Binary.py", line 21, in <module>
    print 'As string:', server.show_type(s)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1437, in __request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1201, in request
    return self._parse_response(h.getfile(), sock)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
    return u.close()
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 787, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<class 'xml.parsers.expat.ExpatError'>:not well-formed (invalid token): line 6, column 55">

Binary data can also be used to send objects using pickles. The normal security issues related to sending what amounts to executable code over the wire apply here (i.e., don't do this unless you're sure your communication channel is secure).

import xmlrpclib
import cPickle as pickle

class MyObj:
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __repr__(self):
        return 'MyObj(%s, %s)' % (repr(self.a), repr(self.b))

server = xmlrpclib.ServerProxy('http://localhost:9000')

o = MyObj(1, 'b goes here')
print 'Local:', o, id(o)

print 'As object:', server.show_type(o)

p = pickle.dumps(o)
b = xmlrpclib.Binary(p)
r = server.send_back_binary(b)

o2 = pickle.loads(r.data)
print 'From pickle:', o2, id(o2)

Remember, the data attribute of the Binary instance contains the pickled version of the object, so it has to be unpickled before it can be used. That results in a different object (with a new id value).


$ python xmlrpclib_Binary_pickle.py
Local: MyObj(1, 'b goes here') 9620936
As object: ["{'a': 1, 'b': 'b goes here'}", "<type 'dict'>", {'a': 1, 'b': 'b goes here'}]
From pickle: MyObj(1, 'b goes here') 11049200

Exception Handling:

Since the XML-RPC server might be written in any language, exception classes cannot be transmitted directly. Instead, exceptions raised in the server are converted to Fault objects and raised as exceptions locally.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000')
try:
    server.raises_exception('A message')
except Exception, err:
    print 'Fault code:', err.faultCode
    print 'Message   :', err.faultString


$ python xmlrpclib_exception.py
Fault code: 1
Message   : <type 'exceptions.RuntimeError'>:A message

MultiCall:

Multicall is an extension to the XML-RPC protocol to allow more than one call to be sent at the same time, with the responses collected and returned to the caller. The MultiCall class was added to xmlrpclib in Python 2.4. To use a MultiCall instance, invoke the methods on it as with a ServerProxy, then call the object with no arguments. The result is an iterator with the results.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000')

multicall = xmlrpclib.MultiCall(server)
multicall.ping()
multicall.show_type(1)
multicall.show_type('string')

for i, r in enumerate(multicall()):
    print i, r


$ python xmlrpclib_MultiCall.py
0 True
1 ['1', "<type 'int'>", 1]
2 ['string', "<type 'str'>", 'string']

If one of the calls causes a Fault or otherwise raises an exception, the exception is raised when the result is produced from the iterator and no more results are available.

import xmlrpclib

server = xmlrpclib.ServerProxy('http://localhost:9000')

multicall = xmlrpclib.MultiCall(server)
multicall.ping()
multicall.show_type(1)
multicall.raises_exception('Next to last call stops execution')
multicall.show_type('string')

for i, r in enumerate(multicall()):
    print i, r


$ python xmlrpclib_MultiCall_exception.py
0 True
1 ['1', "<type 'int'>", 1]
Traceback (most recent call last):
  File "/Users/dhellmann/Documents/PyMOTW/in_progress/xmlrpclib/xmlrpclib_MultiCall_exception.py", line 21, in <module>
    for i, r in enumerate(multicall()):
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 949, in __getitem__
    raise Fault(item['faultCode'], item['faultString'])
xmlrpclib.Fault: <Fault 1: "<type 'exceptions.RuntimeError'>:Next to last call stops execution">

References:

PyMOTW: SimpleXMLRPCServer
XML-RPC How To
XML-RPC Extensions
Python Module of the Week Home
Download Sample Code

Technorati Tags:
python, PyMOTW

Automated Testing with unittest and Proctor

2008-07-01T08:31:00.001-04:00

Automated testing is an important part of Agile development methodologies, and the practice is seeing increasing adoption even in environments where other Agile tools are not used. This article discusses testing techniques for you to use with the open source tool Proctor. By using Proctor, you will not only manage your automated test suite more effectively, but you will also obtain better results in the process.

This article was originally published in Python Magazine in March of 2008.

Read more

PyMOTW: SimpleXMLRPCServer

2008-06-29T12:28:00.001-04:00

The SimpleXMLRPCServer module makes it easy to use remote procedure calls in Python that work with many other languages, too.

Module: SimpleXMLRPCServer
Purpose: Implements an XML-RPC server.
Python Version: 2.2 and later

Description:

The SimpleXMLRPCServer module contains classes for creating your own cross-platform, language-independent server using the XML-RPC protocol. Client libraries exist for many other languages, making XML-RPC an easy choice for building RPC-style services.

Note: All of the examples provided here include a client module as well to interact with the demonstration server. If you want to download the code and run the examples, you will want to use 2 separate shell windows, one for the server and one for the client.

A Simple Server:

This simple server example exposes a single function that takes the name of a directory and returns the contents (obviously a security issue, but useful for demonstration purposes). The first step is to create the SimpleXMLRPCServer instance and tell it where to listen for incoming requests ('localhost' port 9000 in this case). Then we define a function to be part of the service, and register the function so the server knows how to call it. The final step is to put the server into an infinite loop receiving and responding to requests.

from SimpleXMLRPCServer import SimpleXMLRPCServer
import logging
import os

# Set up logging
logging.basicConfig(level=logging.DEBUG)

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

# Expose a function
def list_contents(dir_name):
    logging.debug('list_contents(%s)', dir_name)
    return os.listdir(dir_name)
server.register_function(list_contents)

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

The server can be accessed at the URL http://localhost:9000 using xmlrpclib. This client code illustrates how to call the list_contents() service from Python.

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print proxy.list_contents('/tmp')

Notice that we simply connect the ServerProxy to the server using its base URL, and then call methods directly on the proxy. Each method invoked on the proxy is translated into a request to the server. The arguments are formatted using XML, and then POSTed to the server. The server unpacks the XML and figures out what function to call based on the method name invoked from the client. The arguments are passed to the function, and the return value is translated back to XML to be returned to the client.

Starting the server gives:

$ python SimpleXMLRPCServer_function.py 
Use Control-C to exit

Running the client in a second window shows the contents of my /tmp directory:

$ python SimpleXMLRPCServer_function_client.py 
['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 
'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 'emacs527',
'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 'launchd-242.T5UzTy', 
'var_backups']

and after the request is finished you'll see log output in the server window:

$ python SimpleXMLRPCServer_function.py 
Use Control-C to exit
DEBUG:root:list_contents(/tmp)
localhost - - [29/Jun/2008 09:32:07] "POST /RPC2 HTTP/1.0" 200 -

The first line of output is from the logging.debug() call inside list_contents(). The second line is from the server logging the request because I set logRequests=True.

Alternate Names:

Sometimes the function names you use inside your modules or libraries are not the names you want to use in your external API. You might need to load a platform-specific implementation, build the service API dynamically based on a configuration file, or replace real functions with stubs for testing. If you want to register a function with an alternate name, pass the name as the second argument to register_function(), like this:

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os

server = SimpleXMLRPCServer(('localhost', 9000))

# Expose a function with an alternate name
def list_contents(dir_name):
    return os.listdir(dir_name)
server.register_function(list_contents, 'dir')

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

The client should now use the name 'dir()' instead of 'list_contents()':

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print 'dir():', proxy.dir('/tmp')
print 'list_contents():', proxy.list_contents('/tmp')

Calling list_contents() results in an error, since the server no longer has a handler registered by that name.


$ python SimpleXMLRPCServer_alternate_name_client.py
dir(): ['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 'emacs527', 'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 'launchd-242.T5UzTy', 'temp_textmate.V6YKzm', 'var_backups']
list_contents():
Traceback (most recent call last):
  File "/Users/dhellmann/Documents/PyMOTW/in_progress/SimpleXMLRPCServer/SimpleXMLRPCServer_alternate_name_client.py", line 15, in <module>
    print 'list_contents():', proxy.list_contents('/tmp')
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1437, in __request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1201, in request
    return self._parse_response(h.getfile(), sock)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
    return u.close()
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 787, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: '<type \'exceptions.Exception\'>:method "list_contents" is not supported'>

Dotted Names:

Individual functions can be registered with names that are not normally legal for Python identifiers. For example, you can include '.' in your names to separate the namespace in the service. This example extends our "directory" service to add "create" and "remove" calls. All of the functions are registered using the prefix "dir." so that the same server can provide other services using a different prefix. One other difference in this example is that some of the functions return None, so we have to tell the server to translate the None values to a nil value (see XML-RPC Extensions).

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os

server = SimpleXMLRPCServer(('localhost', 9000), allow_none=True)

server.register_function(os.listdir, 'dir.list')
server.register_function(os.mkdir, 'dir.create')
server.register_function(os.rmdir, 'dir.remove')

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

To call the service functions in the client, simply refer to them with the dotted name, like so:

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print 'BEFORE       :', 'EXAMPLE' in proxy.dir.list('/tmp')
print 'CREATE       :', proxy.dir.create('/tmp/EXAMPLE')
print 'SHOULD EXIST :', 'EXAMPLE' in proxy.dir.list('/tmp')
print 'REMOVE       :', proxy.dir.remove('/tmp/EXAMPLE')
print 'AFTER        :', 'EXAMPLE' in proxy.dir.list('/tmp')

and (assuming you don't have a /tmp/EXAMPLE on your system) the output for the sample client script looks like:


$ python SimpleXMLRPCServer_dotted_name_client.py
BEFORE       : False
CREATE       : None
SHOULD EXIST : True
REMOVE       : None
AFTER        : False

Arbitrary Names:

A less useful, but potentially interesting feature is the ability to register functions with names that are otherwise invalid attribute names. This example service registers a function with the name "multiply args".

from SimpleXMLRPCServer import SimpleXMLRPCServer

server = SimpleXMLRPCServer(('localhost', 9000))

def my_function(a, b):
    return a * b
server.register_function(my_function, 'multiply args')

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

Since the registered name contains a space, we can't use dot notation to access it directly from the proxy. We can, however, use getattr().

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print getattr(proxy, 'multiply args')(5, 5)

This example is provide not necessarily because it is a good idea, but because you may encounter existing services with arbitrary names and need to be able to call them.


$ python SimpleXMLRPCServer_arbitrary_name_client.py
25

Exposing Methods of Objects:

We've talked about establishing APIs using good naming conventions. Another way to do that is to use instances of classes and expose their methods. We can recreate the first example using an instance with a single method.

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

class DirectoryService:
    def list(self, dir_name):
        return os.listdir(dir_name)

server.register_instance(DirectoryService())

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

A client can call the method directly:

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print proxy.list('/tmp')

and receive output like:


$ python SimpleXMLRPCServer_instance_client.py
['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 
'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 
'emacs527', 'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 
'launchd-242.T5UzTy', 'temp_textmate.XNiIdy', 'var_backups']

We've lost the 'dir.' prefix for the service, though, so let's define a class to let us set up a service tree that can be invoked from clients.

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

class ServiceRoot:
    pass

class DirectoryService:
    def list(self, dir_name):
        return os.listdir(dir_name)

root = ServiceRoot()
root.dir = DirectoryService()

server.register_instance(root, allow_dotted_names=True)

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

By registering the instance of ServiceRoot with allow_dotted_names=True, we give the server permission to walk the tree of objects when a request comes in to find the named method using getattr().

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print proxy.dir.list('/tmp')


$ python SimpleXMLRPCServer_instance_dotted_names_client.py
['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 'emacs527', 'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 'launchd-242.T5UzTy', 'temp_textmate.adghkQ', 'var_backups']

Dispatching Calls Yourself:

By default register_instance() finds all callable attributes of the instance with names not starting with '_' and registers them with their name. If you want to be more careful about the exposed methods, you could provide your own dispatching logic. For example:

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

def expose(f):
    "Decorator to set exposed flag on a function."
    f.exposed = True
    return f

def is_exposed(f):
    "Test whether another function should be publicly exposed."
    return getattr(f, 'exposed', False)

class MyService:
    PREFIX = 'prefix'

    def _dispatch(self, method, params):
        # Remove our prefix from the method name
        if not method.startswith(self.PREFIX + '.'):
            raise Exception('method "%s" is not supported' % method)
        
        method_name = method.partition('.')[2]
        func = getattr(self, method_name)            
        if not is_exposed(func):
            raise Exception('method "%s" is not supported' % method)
        
        return func(*params)

    @expose
    def public(self):
        return 'This is public'
        
    def private(self):
        return 'This is private'

server.register_instance(MyService())

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

The public() method of MyService is marked as exposed to the XML-RPC service while private() is not. The _dispatch() method is invoked when the client tries to access a function that is part of MyService. It first enforces the use of a prefix ("prefix." in this case, but you can use anything, of course). Then it requires the function to have an attribute called "exposed" with a true value. The exposed flag is set on a function using a decorator for convenience.

Here are a few sample client calls:

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print 'public():', proxy.prefix.public()
try:
    print 'private():', proxy.prefix.private()
except Exception, err:
    print 'ERROR:', err
try:
    print 'public() without prefix:', proxy.public()
except Exception, err:
    print 'ERROR:', err

and the resulting output, with the expected error messages trapped and reported:


$ python SimpleXMLRPCServer_instance_with_prefix_client.py
public(): This is public
private(): ERROR: <Fault 1: '<type \'exceptions.Exception\'>:method "prefix.private" is not supported'>
public() without prefix: ERROR: <Fault 1: '<type \'exceptions.Exception\'>:method "public" is not supported'>

There are several other ways to override the dispatching mechanism, including subclassing directly from SimpleXMLRPCServer. Check out the docstrings in the module for more details.

Introspection API:

As with many network services, it is possible to query an XML-RPC server to ask it what methods it supports and learn how to use them. SimpleXMLRPCServer includes a set of public methods for performing this introspection. By default they are turned off, but can be enabled with register_introspection_functions(). You can add explicit support for system.listMethods() and system.methodHelp() by defining _listMethods() and _methodHelp() on your service class. For example,

from SimpleXMLRPCServer import SimpleXMLRPCServer, list_public_methods
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)
server.register_introspection_functions()

class DirectoryService:
    
    def _listMethods(self):
        return list_public_methods(self)

    def _methodHelp(self, method):
        f = getattr(self, method)
        return inspect.getdoc(f)
    
    def list(self, dir_name):
        """list(dir_name) => [<filenames>]
        
        Returns a list containing the contents of the named directory.
        """
        return os.listdir(dir_name)

server.register_instance(DirectoryService())

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

In this case, the convenience function list_public_methods() scans an instance to return the names of callable attributes that do not start with '_'. You can, of course, redefine _listMethods() to apply whatever rules you like. Similarly, for this basic example _methodHelp() returns the docstring of the function, but could be written to build a help string from any information you like.

This client queries the server and reports on all of the publicly callable methods.

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
for method_name in proxy.system.listMethods():
    print '=' * 60
    print method_name
    print '-' * 60
    print proxy.system.methodHelp(method_name)
    print

Notice that the system methods are included in the results.


$ python SimpleXMLRPCServer_introspection_client.py
============================================================
list
------------------------------------------------------------
list(dir_name) => [<filenames>]

Returns a list containing the contents of the named directory.

============================================================
system.listMethods
------------------------------------------------------------
system.listMethods() => ['add', 'subtract', 'multiple']

Returns a list of the methods supported by the server.

============================================================
system.methodHelp
------------------------------------------------------------
system.methodHelp('add') => "Adds two integers together"

Returns a string containing documentation for the specified method.

============================================================
system.methodSignature
------------------------------------------------------------
system.methodSignature('add') => [double, int, int]

Returns a list describing the signature of the method. In the
above example, the add method takes two integers as arguments
and returns a double result.

This server does NOT support system.methodSignature.

References:

XML-RPC How To
XML-RPC Extensions
Python Module of the Week Home
Download Sample Code

Technorati Tags:
python, PyMOTW

PyWorks 2008: Call for papers

2008-06-26T07:24:00.001-04:00

MTA, publishers of Python Magazine, are pleased to announce the first PyWorks conference to be held in Atlanta, GA on November 12-14, 2008.

We are currently looking for speakers and tutorial instructors on a wide range of topics. The deadline for proposals is July 25, so check out the site for details about how to submit yours today!

Python Magazine for June 2008

2008-06-26T07:17:00.001-04:00

The June issue is available for download now.

Grig Gheorghiu's cover story on Pybots talks about the fantastic automation system that has been put in place to make sure new releases of Python software are as robust and stable as possible.

In the second part of his PyGame series, Terry Hancock adds some interactivity to the simple game he started building in the May issue. This tutorial is shaping up to be a great introduction for new programmers.

JC Cruz returns this month with another article on combining Python and OS X. This time, he shows us how to create a Cocoa application with PyObjC and Xcode 3.0. This is a topic I've been wanting to dig into myself for some time, so I was glad to have a chance to read his article.

And rounding out the features this month, Massimo Di Pierro introduces web2py, a relative newcomer on the web framework front. While originally designed primarily as a teaching tool, Massimo explains why that includes many features that make it attractive for rapid application development situations, too.

Our columns this month are all about the future:

Mark Mruss takes us quite literally into the __future__ by talking about how to take advantage of some features of Python 3.0 that are already available now in Python 2.5.

Jesse Noller talks with Adam Olsen about the "safe thread" project, a set of patches for the C interpreter that eliminate the Global Interpreter Lock and that may eventually be included in the Python core.

Steve Holden responds to Tim Bray's post Multi-Inflection-Point Alert by fixing his attention on a few technology trends and looking for portents in their convergence.

And finally, this issue also marks the transition as I take over from Brian Jones as Editor in Chief. I want to thank Brian for all he has done to teach me about the position and magazines in general. I'm looking forward to working with the excellent team of reviewers and editors we have put together as we continue to shape the magazine to be something we can all be proud to be a part of.

I'll give more details about our road ahead in another post, but for the time being go download the June issue and enjoy!

svnbackup news

2008-06-22T12:55:00.001-04:00

I've received several patches for svnbackup.sh recently, and an offer to host an RPM version of the code. I decided it probably made sense to go ahead and open the project up to contributors, so I created a project on code.google.com.

Unfortunately, the name svnbackup was already taken, and svnbackup.sh wasn't a valid project name. I'm linking from my site to the new project, though, so hopefully the minor name change won't be too confusing.

Since I had the project on my local svn repository, and not in its own repo, importing the history into google's svn repository was something of a pain. I'm glad there are only a few files in the source tree.

There's a new release today, as well. Version 1.4 includes patches from Arjen Van Drie to fix unary
operator problems and an empty history file bug. See the downloads page for a tarball.