Django Development with Djblets: Data Grids

It’s been a while since my last post in this series, but this one’s a good one.

A common task in many web applications is to display a grid of data, such as rows from a database. Think the Inbox from GMail, the model lists from the Django administration interface, or the Dashboard from Review Board. It’s not too hard to write something that displays a grid by doing something like:

{% for user in users %}
{% endfor %}
Username First Name Last Name
{{user.username}} {{user.first_name}} {{user.last_name}}

This works fine, so long as you don’t want anything fancy, like sortable columns, reorderable columns, or the ability to let users specify which columns they want to see. This requires something a bit more complex.


We wrote a nifty little set of classes for making data grids easy. Let’s take the above example and convert it over.

# myapp/
from django.contrib.auth.models import User
from djblets.datagrid.grids import Column, DataGrid

class UserDataGrid(DataGrid):
    username = Column("Username", sortable=True)
    first_name = Column("First Name", sortable=True)
    last_name = Column("Last Name", sortable=True)

    def __init__(self, request):
        DataGrid.__init__(self, request, User.objects.filter(is_active=True), "Users")
        self.default_sort = ['username']
        self.default_columns = ['username', 'first_name', 'last_name']
# myapp/
from myapp.datagrids import UserDataGrid

def user_list(request, template_name='myapp/datagrid.html'):
    return UserDataGrid(request).render_to_response(template_name)

Now while this may look a bit more verbose, it offers many benefits in return. First off, users will be able to reorder the columns how they like and choose which columns to see. You could extend the above DataGrid to add some new column but leave it out of default_columns so that only users who really care about it would see it.

Custom columns

Let’s take this a step further. Say we want our users datagrid to optionally show staff members with a special badge. This might be useful to some users, but not all, so we’ll leave it out by default. We can implement this by creating a custom column with image data:

class StaffBadgeColumn(Column):
    def __init__(self, *args, **kwargs):
        Column.__init__(self, *args, **kwargs)

        # These define what will appear in the column list menu.
        self.image_url = settings.MEDIA_URL + "images/staff_badge.png"
        self.image_width = 16
        self.image_height = 16
        self.image_alt = "Staff"

        # Give the entry in the columns list menu a name.
        self.detailed_label = "Staff"

        # Take up as little space as possible.
        self.shrink = True

    def render_data(self, user):
        if user.is_staff:
            return '%s' % 
                (self.image_url, self.image_width, self.image_height, self.image_alt)

        return ""

class UserDataGrid(DataGrid):
    staff_badge = StaffBadgeColumn()

This will add a new entry to the datagrid’s column customization menu showing the staff badge with a label saying “Staff.” If users enable this column, their datagrid will update to show an staff icon for any users who are marked as staff.

Custom columns can render data in a couple of different ways.

The first is to bind the column to a field in the model. By default, a Column instance added to a DataGrid will use its own name (such as “first_name” above) as a lookup in the object. If you want to use a custom field, set the db_field attribute on the column. This field can span relationships as well. For example:

class UserDataGrid(DataGrid):
    # Uses the field name as the lookup.
    username = Column("Username")

    # Spans relationships, looking up the age in the profile.
    age = Column("profile__age")

The second is the way shown above, by overriding the render_data function. This takes an object, which is the object the datagrid represents, and outputs HTML. The logic in this can be quite complicated, depending on what’s needed.

Linking to objects

A datagrid is pretty worthless if you can’t link entries for the object to a URL. We have a couple of ways to do this.

To link a column on a datagrid to a URL, set the link parameter on the DataGrid to True. By default, the URL used will be the result of calling get_absolute_url() on the object represented by the datagrid, but you can override this:

# myapp/

class UserDataGrid(DataGrid):
    username = Column("Username", sortable=True, link=True)
    def link_to_object(user, value):
        return "/users/%s/" % user


link_to_object takes an object and a value. The object is the object being represented by the datagrid, and the value is the rendered data in the cell. The result is a valid path or URL. In the above example, we’re explicitly defining a path, but we could use Django’s reverse function.

Usually the value is ignored, but it can be useful. Imagine that your Column represents a field that is a ForeignKey of an object. The contents of the cell is the string version of that object (implemented by the __str__ or __unicode__ function), but it’s just an object and you want to link to it. This is where value comes in handy, as instead of being HTML output, it’s actually an object you can link to.

If you intend to link a particular column to value.get_absolute_url(), we provide a handy utility function called link_to_value which exists as a static method on the DataGrid. You can pass it (or any function) to the Column’s constructor using the link_func parameter.

class UserDataGrid(DataGrid):
    myobj = Column(link=True, link_func=link_to_value)

Custom columns can hard-code their own link function by explicitly setting these values in the constructor.

Saving column settings

If a user customizes his column settings by adding/removing columns or rearranging them, he probably expects to see his customizations the next time he visits the page. DataGrids can automatically save these settings in the user’s profile, if the application supports it. Handling this is as simple as adding some fields to the app’s Profile model and setting a couple options in the DataGrid. For example:

# myapp/

class Profile(models.Model):
    user = models.ForeignKey(User, unique=True)

    # Sort settings. One per datagrid type.
    sort_user_columns = models.CharField(max_length=256, blank=True)

    # Column settings. One per datagrid type.
    user_columns = models.CharField(max_length=256, blank=True)
# myapp/

class UserDataGrid(DataGrid):
    def __init__(self, request):
        self.profile_sort_field = "sort_user_columns"
        self.profile_columns_field = "user_columns"

The DataGrid code will handle all the loading and saving of customizations in the Profile. Easy, isn’t it?

Requirements and compatibility

Right now, datagrids require both the Yahoo! UI Library and either yui-ext (which is hard to find now) or its replacement, ExtJS.

Review Board, which Djblets was mainly developed for, still relies on yui-ext instead of ExtJS, so datagrids today by default are built with that in mind. However, the JavaScript portion of datagrids should work with ExtJS today. The default template for rendering the datagrids (located in djblets/datagrids/templates/datagrids/) assumes both YUI and yui-ext are present in the media path, but sites using datagrids are encouraged to provide their own template to reflect the actual locations and libraries used.

We would like to work with more JavaScript libraries, so if anyone wants to provide an implementation for their favorite library or help to dumb down our implementation to work with any and all, we’d welcome patches.

See it in action

We make extensive use of datagrids on Review Board. You can get a sense of it and play around by looking at the users list and the dashboard (which requires an account).

That’s it for now!

There’s more to datagrids than what I’ve covered here, but this should give you a good understanding of how they work. Later on I’d like to do a more in-depth article on how datagrids work and some more advanced use cases for them.

For now, take a look at the source code and Review Board’s datagrids for some real-world examples.

Pipe into your netlife: Comic feeds

Penny Arcade and Control-Alt-Del are great comics. I love to read them and do so when I’m not feeling really lazy. See, I pretty much live inside my Google Calendar, GMail, Remember the Milk, Google Reader, and Netvibes tabs, for the most part. I’m pretty lazy when it comes to some forms of content, and if a comic doesn’t show up in Google Reader, I’ll typically ignore it until someone points out a particularly good strip to me.

While playing around with Yahoo! Pipes, I realized I could finally do something about this. I began to play around with a couple of pipes to read in the RSS feeds, look for all comic entries, and change the content. The trick was to copy the location the feed item was pointing to (which would contain the actual comic image within the page) to the description, and then apply a regular expression to the description to turn it into an <img> tag pointing to the image itself.

I got lucky. To my knowledge, there is currently no way to fetch content from any arbitrary HTML page and do something with a piece of that page. I suspect their Fetch Data module might let me, but I haven’t managed to get it to work just yet. I was able to pull this off since the comic image was stored with a predictable path based on the date of the comic, and the page being linked to also contained the date. A regular expression was all that was needed to parse out the date and rebuild the path.

Anyway. the end result is that I now have inline comics in my Penny Arcade and Control-Alt-Del RSS feeds! You can add them to your RSS reader below, or take a look at how they were made.

  • [Pipe] [RSS] Penny Arcade News and Inline Comics
  • [Pipe] [RSS] Penny Arcade Inline Comics
  • [Pipe] [RSS] Control-Alt-Del Inline Comics

Pipe into your netlife

Yahoo is doing a lot of interesting things these days. While Google gains a lot of the attention when it comes to search and web applications, Yahoo should not be ignored. and Flickr are of course two widely popular services, but they have a few useful utilities floating around their developer site.

There’s one tool in particular that I found tonight that has already proven useful. Pipes. Pipes allows a user to quickly put together a simple set of pipeline filters for turning various forms of data into an RSS feed and accompanying JSON file.

Such forms of data include user input (validated as a date, geographical location, number, text or URL) and web-based input (Flickr pictures, RSS2/Atom feeds, JSON/XML data, Google Base listings, Yahoo! Local searches, and Yahoo! search results). This data can be fed through several layers of pipes (including back into another Flickr pipeline and such as query input). The pipes can transform the data, walk through each feed item and modify or extract data, combine data together, sort, remove duplicates, apply regexes, translate languages, and so on.

This can be pretty powerful. While still a young project, many users have already published pipes, myself included. With the increase in API-enabled web services, I can only expect this to become more powerful, with work. It’s just a little tricky coming up with actual useful applications.

So I played around a bit and started to experiment with what could be done. I ended up with a couple of simple, but very useful pipes. One thing I have wanted for the longest time was a way to see feeds from several Planets in one listing (for Netvibes, since space is precious), without having to deal with duplicate entries. Pipes made this all too simple.

Unique Planets Pipe

I’m feeding several feeds into a Unique operator, saying to filter based on the title. I then output that. That’s all it takes. You can see the results and even add the RSS feed.

I then took this one step further and decided to write a quick pipe for searching through the planets. Now, pipes are reusable, so I was able to incorporate the Unique Planets pipe into this. This was fed into a Filter, using a couple of text inputs (for a text string and a name) as parameters to the filter. The screenshot below will clarify this. The result is the ability to quickly search four planets by name or content.

Planet Search Pipe

You can play with the results. Go ahead, give it a try.

Pipes can be published for other people to use, or they can be used privately. Private pipes are great when you want to deal with data that can’t easily be queried, such as your Twitter feed or your own Flickr feed.

Pipes are also quite useful when you have a small web application that needs to deal with several other feeds, filtering results or combining data from multiple sources. Sure, you could write this all yourself, but it’s far easier to change and maintain a Pipe than a whole bunch of code.

If you want to play with pipes, I recommend just jumping in and playing. Also take a look at some other people’s pipes, and you may want to browse the tutorials. For some starter ideas, try making a pipe that searches your local area for sales or singles or something using Google Base and Craigslist, or one that searches all your favorite blogs for a certain keyword, or maybe something that keeps track of your friends’ blogs and Flickr posts.

Now, pipelines are hardly a new concept. Several programs offer them, including some development environments that rely solely on pipelines for development in order to quickly produce simple programs. What makes Yahoo!’s Pipes interesting is that they make it very easy for almost anybody to quickly build a pipe to modify or search all kinds of data on the web that people actually use. This makes them more immediately useful to many people, and of course Yahoo makes it dead simple to start out.

What would be useful in the future, aside from adding native support for more services, would be to output data in other formats or somehow easily lay out information onto a page from one or more feeds. The project seems pretty young though, so I’m sure in time, this will mature into a much more useful project, both to developers and (certain) end users.