As I started using xlwt, I found myself wanting some more convenient methods for dumping tabular data into a worksheet, especially when all the data can be treated as strings.  Tabular data in this context is an iterable of iterables, such as a list of tuples.

Here’s what I’ve got so far:

"""Excel utilities.

'Tabular data' in this context is an iterable of iterables.
"""

import xlwt

def to_workbook(tabular_data, workbook=None, sheetname=None):
    """
    Returns the Excel workbook (creating a new workbook
    if necessary) with the tabular data written to a worksheet
    with the name passed in the 'sheetname' parameter (or a
    default value if sheetname is None or empty).
    """
    wb = workbook or xlwt.Workbook()
    ws = wb.add_sheet(sheetname or 'Data')
    to_worksheet(tabular_data, ws)
    return wb

def to_worksheet(tabular_data, worksheet):
    """
    Writes the tabular data to the worksheet (returns None).
    Thanks to John Machin for the tip on using enumerate().
    """
    for row_index, row_data in enumerate(tabular_data):
        worksheet_row = worksheet.row(row_index)
        for col_index, col_data in enumerate(row_data):
            worksheet_row.write(col_index, col_data)

In a Django context, then, you have a very straightforward way turning a query into an Excel file using the values_list() QuerySet method, e.g.:

wb = to_workbook(MyModel.objects.values_list())

Since values_list() outputs the attribute values for each object in the same order in which they’re defined in the model class, you can insert a row of headings to your table:

table = MyModel.objects.values_list()
headings = [f.name for f in MyModel._meta.fields]
table.insert(0, headings)
wb = to_workbook(table)

I found myself creating a number of simple custom context processors which simply return custom settings that I have added to my Django settings module (I actually keep these custom settings in my_settings.py and import then into settings.py, just to keep them separate).  I decided it was a good idea to add exception handling to these functions so that I would get a useful error message if I tried to use a particular context processor without implementing its required setting(s).  So, after some refactoring, I came up with this:

from django.conf import settings
from django.core.exceptions import ImproperlyConfigured

class SettingsContextProcessor(object):
    """
    Class for creating simple context processors that
    return one or more Django settings.
    """

    def __init__(self, *setting_names):
        self.setting_names = setting_names

    def __call__(self, request):
        extra_context = {}
        for sn in self.setting_names:
            try:
                extra_context[sn] = getattr(settings, sn)
            except AttributeError, e:
                raise ImproperlyConfigured('Missing required setting: %s' % sn)
        return extra_context

Now I can create custom settings-based context processors like this:

google_analytics = SettingsContextProcessor('GOOGLE_ANALYTICS_PROFILE_ID')
jquery = SettingsContextProcessor('JQUERY_VERSION')
jqueryui = SettingsContextProcessor('JQUERYUI_VERSION')
static_media = SettingsContextProcessor('STATIC_MEDIA_URL')
yui = SettingsContextProcessor('YUI_VERSION')

OK, this doesn’t really disable it, but it does make the “browser uploader” the default …

<Directory /path/to/wp/wp-admin>
  <Files media-new.php>
    # Force browser uploader instead of Flash
    RewriteCond %{QUERY_STRING} !=flash=0
    RewriteRule ^/(.*) /$1?flash=0
  </Files>
</Directory>

(WordPress 2.8.4)

It should clear from this blog that I’m a big fan of Django.  I use it for as much of my work as possible.  Recently a couple of other developers in my shop have entered the Django arena, which, as a general proposition, is a good thing — in that we’re using the framework more widely.  But as a result, one of Django’s few weaknesses has been made more painfully obvious, namely, the fragility of Django projects.

The problem is that one import error in any installed app’s models, any referenced URLconf, view module, or other module imported by one of those breaks the entire project immediately and horribly.  This makes the installation of new apps and updates of installed apps inherently risky to the entire site, which IMO is a *bad thing*.  Now, I haven’t delved into the guts of Django’s initialization process to see what, if anything, could be done about this, but on a conceptual level it seems that the project as a whole should have some way to recover from a bad app or module, unless it’s related to the core functionality of the project (like a middleware or context processor module).

Is that unreasonable?

Or, developing Django data applications without a database.

In recent months I have been working intensively on a user interface to a data store that lives behind a web services API.  While this might not be considered a natural fit for Django, the smart de-coupling of URLs, views, and templates from data models means that the former retain their value even without the latter.  And hey, it’s all just Python, right?  Django models are just one way to handle data in your application, albeit a very convenient and powerful one when dealing with an RDBMS.

Before working on this project, I had already developed two other Django apps based on data sources accessed via HTTP, both of which were read-only, which of course made things quite a bit simpler.  The first of these was an XML-RPC interface (built with django_xmlrpc, Python’s standard xmlrpclib module and python-ldap) to an LDAP directory.  I manage user and group information for a number of staff applications, including Django itself, for which it is very useful to draw upon a central source of personnel data.  The Django-based service provides convenient methods for common operations while hiding the complexities of LDAP connections and search queries.

The second app dealt with requests to web services of a library catalog which return content in a custom XML format.  Since only HTTP GET requests were required, I used Python’s standard urllib and urllib2 modules for the request/response handling, and lxml.etree for the XML parsing and XSLT application (see also my previous post on the virtues of lxml).

The current project, as opposed to the previous two, involves both read and write actions on the backend data store.  Also, because the web services API implements a REST architecture, the “client” code I was tasked with has to support a wider range of HTTP request methods (not just GET and POST) and responses.  Finally, the client code stack includes a full-blown user interface (the ultimate purpose of the app) for managing the backend data.  In my next post, I’ll talk about how I broke down the problem.

So, I wanted one of those horizontal multi-selects with “available” and “chosen” boxes to associate users with groups in the Django auth admin UI.  Turns out it all I had to do was import the UserAdmin class from django.contrib.auth.admin and override the filter_horizontal attribute:

from django.contrib.auth.admin import UserAdmin

UserAdmin.filter_horizontal = ('user_permissions', 'groups')

Since importing the UserAdmin class runs the admin module from django.contrib.auth, the UserAdmin and GroupAdmin classes are registered for the admin site.  All I have to do then is import my custom admin module in my URLconf instead of the one from django.contrib.auth to make sure my customizations are applied in my admin site.

On balance I’m satisfied with Vista x64, but I’ve run into a few challenges and at least one casualty.

The casualty was my Palm Tungsten E2, which can’t connect to Vista64 with the USB cable.  According to Palm, you can connect with Bluetooth, but I don’t have Bluetooth on the computer.  Fortunately I don’t use the Palm much anymore and I synced mainly to back it up.

As for the challenges, it seems that some applications, notably OpenOffice.org 3, have to be installed using XP-SP2 compatibility mode.  Also, some installation software doesn’t trigger privilege elevation which may be required to write to certain registry keys or system folders, so the “Run as Administrator” and “CMD prompt here as administrator” Elevation PowerToys are practically necessary.  Sun’s Java RE (which I probably wouldn’t install except that it’s a dependency of OOo) has an annoying bug in it’s control panel such that you can’t change the settings, so you can’t disable automatic updating.  The workaround is to run bin/javacpl.exe as administrator using the Elevation PowerToy.  Using 32-bit Windows help files (.hlp) requires a special download (this affects all Vista versions) as Microsoft is no longer updating the older help program and so doesn’t distribute it with the OS.  Only two of the four buttons on my Kensington Expert Trackball Mouse work, and Vista thinks it’s a regular mouse — but it wasn’t working fully under XP-SP3 either.  Don’t bother to install the MouseWorks software, although some folks claim they’ve gotten it work (or at least the OS to use the driver).  I would guess at this point Kensington isn’t going to release a Vista-compatible driver for the older devices.

I can live with two buttons.

Working with XML isn’t my favorite task. Sure, I get why it’s useful for data transfer, but I’d rather deal with an interface that hides the gory details … meaning the raw XML.  Of course, sometimes that isn’t possible.  The Python standard library modules as of version 2.4 only offers the DOM and SAX APIs, which, while useful, just don’t offer enough power and flexibility to do ad hoc XML processing without significant pain, at least for the casual user. Python 2.5 adds the ElementTree API (xml.etree.ElementTree), but we’re still missing full XPath and XSLT support.

In steps lxml, a library that really makes everything else obsolete. Prior to lxml there was 4Suite, but its latest release was posted in December 2006 and the project appears to be dead. Fortunately, lxml runs on Python 2.3 or later (I’m stuck on 2.4 for the time being).  The only hiccup, depending on your environment, can be getting the C dependencies installed – libxml and libxslt. But if you’re going to do some serious work with XML, it’s worth the effort to get lxml installed.  You won’t want to go back.

DOM

>>> xmlsrc = '<root><element>text</element></root>'
>>> from xml.dom.minidom import parseString
>>> parseString(xmlsrc).getElementsByTagName('element')[0].firstChild.nodeValue
u'text'

Yikes!

lxml

>>> xml = '<root><element>text</element></root>'
>>> from lxml import etree
>>> etree.fromstring(xml).findtext('//element')
'text'

Ahh … better.

I discovered “accidentally” recently that the Django model delete() method not only deletes the model instance, but all its related objects — at least those which are related to the original object via a ForeignKey field.  (The source code is so labyrinthine that I gave up trying to determine exactly what it does.)  This is not necessarily a bad thing; many times it’s exactly what you want to preserve the “referential integrity” of your data.  For instance, if you have some kind of “collection” object and a number of “item” objects which are related to it in a one-to-many relationship, it’s reasonable to expect that when the collection is deleted, the related items are also deleted.

Now, in the Django admin app, when you click the button to delete an item you get a helpful warning informing you of all the other objects you will delete if you proceed.  But what if you’re using the API directly?  No warning there, it just whacks the whole lot.  At first I thought maybe this was an ORM thing, but it’s not.  In fact it has nothing to do with the backend database: Django manually fetches all the related objects and deletes them.  For example, in my case the backend is MySQL 5.0 and MyISAM tables.  In MySQL 5.0 only InnoDB tables support foreign key constraints; MyISAM tables will parse the syntax but do nothing with it.  In any case, the constraints that Django generates do not include an ON DELETE clause, so MySQL 5.0 would use “RESTRICT” as the default value, meaning that the database will not allow the deletion of a row from the “parent” table if the primary key value exists in the referenced foreign key of a row in the related table.  I suppose that’s good as far as it goes, but you shouldn’t be messing with the database directly, right?  Anyway, that’s not relevant because we’re not talking about raw SQL commands, but the Django model API methods.

So, there’s an outstanding proposal to add keyword arguments to the ForeignKey model field to control how DELETE and UPDATE on the parent model affect the related model.  The status of the proposal is unclear and there hasn’t been a lot of serious discussion on the mailing lists AFAICT.  I would guess that since you can override the delete() method on a per-model basis, and presumably other issues are more pressing, that the core developers don’t want to worry about this right now.  They may be right, but I sure had an unpleasant surprise when I discovered this behavior the wrong way.

I have an organizational directory database with Persons and OrgUnits.  Persons are automatically added and removed based on information retrieved an another data source (LDAP).  An OrgUnit (e.g., a department) can have a “head”, which is a person:

head = models.ForeignKey(Person, blank=True, null=True, related_name='head_of')

Now, of course, if a Person who happened to be the head of a department left the organization I wouldn’t want the department to be deleted, right?  In this case, as it turns out, it was worse than that.  Since the OrgUnit structure is hierarchical, the OrgUnit model has a one-to-many relationship with itself:

parent = models.ForeignKey('self', null=True, blank=True, related_name='children')

Now what happens if the head of a top-level OrgUnit is deleted?  The OrgUnit of which he/she was head is deleted, and every “descendant” OrgUnit under that one!  So, I had to override the delete() methods on both the Person and OrgUnit models.

Person:

def delete(self):
"""
Override default model method so an OrgUnit is not deleted
when its head is deleted.
"""
self.head_of.clear()
super(Person, self).delete()

OrgUnit:

def delete(self):
"""
Override default model method so that OrgUnit children are
not deleted when the parent OrgUnit is deleted.
"""
self.children.clear()
super(OrgUnit, self).delete()

On balance, Django’s default behavior is probably the right thing to do — as long as you’re aware of it!

I’ve long had the nagging feeling that I’m not writing enough tests for my code. It’s kind of like doing the dishes, not exactly a task I look forward to.  The fun of programming is solving the problems, not writing code that tests whether your solutions actually work! 

What I love about Python’s doctest is that it fits so naturally into the flow of coding: you write the function signature, then write the docstring saying what it’s supposed to do, right?  So then, right there, you write one or more simple tests to demonstrate the basic functionality you expect — before you’ve written a line of code in the body of the function. You write the function, save the file and then — boom! — run the tests.  Immediate feedback.  And I start to feel like my approach to programming is changing, because I can see that writing tests is not just about due diligence or whatever, but it’s a technique that can actually help me write code.

And — you can paste in commands from the Python interactive interpreter unchanged.  Yet another reason why Python feels like it’s designed with the needs of programmers, human beings, in mind.  It’s trying to make your life easier, simpler, and more productive.  While Python isn’t perfect (no language is), those things alone make it rate pretty high in my book.