Django Development with Djblets: Dynamic Site Configuration

Django’s a powerful toolkit and offers many things to ease the creation of websites and web applications. Features such as the automatically built Administration UI are just awesome and save developers from having to reinvent the wheel on every project. It does fall short in a few areas, however, and one of them is site configuration.

Now, Django’s settings support isn’t bad. You get a nice little settings.py file to put settings in for your site/webapp, and since it’s just a Python module, your settings can really hold whatever data you want and can be well documented with comments. That’s all great, but when you want to actually change a setting, you must SSH in, edit the file, save it, and restart the server.

This is fine for small websites with a few visitors, or sites with almost no custom settings, but for large sites it can be a problem. Bringing down the site however briefly could interrupt users, generate errors, causing worries during a checkout process or when editing a post on a site. Doing anything more complex and preventing downtime really means rolling your own thing.

So we rolled our own thing. Review Board has operated with the standard Django settings module since day 1, but it’s become obvious over time that it wasn’t good enough for us. So we wrote the dynamic Site Configuration app for Djblets.

djblets.siteconfig

siteconfig is a relatively small Django app that stores setting data in the database, along with project versioning data in case you need it to handle migrations of some sort down the road. There’s a lot of useful things in this app, so I’ll summarize what it provides before I jump into details:

  • Saving/Retrieving settings.

    Setting values can be simple strings, integers, booleans, or something more complex like an array of dictionaries of key/value pairs.

  • Default values for settings.

    These default values could be provided by your application directory or could be based off the contents in your settings.py file.

  • Syncing to/from settings.py.

    Just because you’re using djblets.siteconfig doesn’t mean you can’t still support settings.py. Settings can be pulled from your old settings and saved back there when modified.

  • Compatibility with standard Django settings.

    Django offers a lot of useful settings that you may want to customize. We do the hard work of mapping these and keeping them in sync so you can customize them dynamically.

  • Auto-generated settings pages.

    Much like the existing admin UI, we offer an easy way to provide settings pages for your application. Go ahead and stick these in the admin UI if you like.

Getting Started

To start out, you’ll want to add some code in your app that creates a SiteConfiguration model, perhaps as a post_syncdb management hook. This is project-specific, since you’ll be storing version information in here and linking it with your existing Site. Here’s one way to do it. We’ll use this file as a template in later examples:

# myapp/siteconfig.py
# NOTE: Import this file in your urls.py or some place before
#       any code relying on settings is imported.
from django.contrib.sites.models import Site

from djblets.siteconfig.models import SiteConfiguration


def load_site_config():
    """Sets up the SiteConfiguration, provides defaults and syncs settings."""
    try:
        siteconfig = SiteConfiguration.objects.get_current()
    except SiteConfiguration.DoesNotExist:
        # Either warn or just create the thing. Depends on your app
        siteconfig = SiteConfiguration(site=Site.objects.get_current(),
                                       version="1.0")

    # Code will go here for settings work in later examples.


load_site_config()

Saving/Retrieving Settings

Settings exist in a SiteConfiguration model inside a JSONField. Anything that Django’s native JSON serializer/deserializer code can handle, you can store. Usually you’ll want to store primitive values (strings, booleans, integers, etc.), but you’re free to do something more complex.

from djblets.siteconfig.models import SiteConfiguration


siteconfig = SiteConfiguration.objects.get_current()
siteconfig.set("mystring", "foobar")
siteconfig.set("mybool", True)
siteconfig.set("myarray", [1, 2, 3, 4])

mybool = siteconfig.get("mybool")

It’s pretty straightforward. The set/get functions are just simple accessors for the SiteConfiguration.settings JSON field, but you’ll want to use them because they’ll handle the registered defaults for settings, which I will get to in a moment.

Since this is just a Djblets JSONField, which is a glorified dictionary, you can do a lot of things, such as iterate through the keys and values:

for key, value in siteconfig.settings:
    print "%s: %s" % (key, value)

And so on. Let’s go a step further.

Default Values

Say you’re starting fresh and just created your SiteConfiguration instance, or you’ve introduced a new setting into an existing site. The setting won’t be in the saved settings, so what value is returned when you do a get call? Well, that depends on your setup a bit.

In the first case, with a fresh new setting:

>>> print siteconfig.get("mynewsetting")
None

Any setting without a default value and not in the saved settings will return None. Good ol’ reliable None.. If you want to make sure you return something sane at this specific callpoint, you can specify a default value in the call to get, like so:

>>> print siteconfig.get("mynewsetting", default=123)
123

This can be really useful at times, but it’s a pain to have to do this in every call and keep the values in sync. So we provide a third option: Registered default values.

# myapp/siteconfig.py


defaults = {
    'mystring':     'Foobar',
    'mybool':       False,
    'mynewsetting': 123,
}


def load_site_config():
    ...

    if not siteconfig.get_defaults():
        siteconfig.add_defaults(defaults)

That’s all it takes. Now calling get with just the setting name will return the default value, if the setting hasn’t been saved in the database.

Down the road we’re going to have support for automatic querying of apps to grab their settings, giving third party apps an easier way to integrate into codebases using siteconfig.

Syncing Settings with settings.py

Django’s settings.py is and will continue to be an important place for some settings to go. If you’re developing a reusable application for other projects, you may not want to fully ditch settings.py. Or maybe you’re using a third party application that uses settings.py and want to make the settings dynamic.

siteconfig was designed from the beginning to handle syncing settings in both places. Now, when I say syncing, I don’t mean we write out to settings.py, since that would require enforcing certain permissions on the file. What we do instead is to load in the values from settings.py if not set in the siteconfig settings, and to write out to the settings object, an in-memory version of settings.py.

Let’s revisit our example above, but in this case we want to be able to dynamically control Django’s EMAIL_HOST setting.

# myapp/siteconfig.py
from django.conf import settings

from djblets.siteconfig.django_settings import apply_django_settings, 
                                               generate_defaults


settings_map = {
    # siteconfig key    settings.py key
    'mail_host':        'EMAIL_HOST',
}

defaults = {
    ...
}
defaults.update(generate_defaults(settings_map))


def load_site_config():
    ...

    apply_django_settings(siteconfig, settings_map)

What we did here was to generate a mapping table of our custom settings to Django settings.py settings. We can then generate a set of defaults using the generate_defaults function. This goes through the Django settings keys, pulls out the values if set in settings.py, and returns a dictionary similar to the one we wrote above. This guarantees that our default values will be what you have configured in settings.py, handling the first part of our synchronization.

The second part is handled through use of the apply_django_settings function. This will take a siteconfig and a settings map and write out all values that have actually changed. So if “mail_host” above is never modified, apply_django_settings won’t write it out to the database, allowing the default value to change without having to do anything too fancy.

Compatibility with Django’s Settings

The above example dealt with the settings.EMAIL_HOST setting, but in reality we won’t have to cover this at all. There’s a few other goodies in the django_settings module that handle all the Django settings for you.

First off is get_django_settings_map. You’ll rarely need to call this directly, but it returns a settings map for all the Django settings sites are likely to want to change.

Then there’s get_django_defaults, which you’ll want to either merge into your existing defaults table or add directly. We don’t do it for you because you may very well not care about these settings.

The third is one you just saw, apply_django_settings. In the above example, we passed in our settings map, but if you don’t specify one it will use get_django_settings_map.

Let’s take a look at how this works.

# myapp/siteconfig.py

from djblets.siteconfig.django_settings import apply_django_settings, 
                                               get_django_defaults


def load_site_config():
    ...

    if not siteconfig.get_defaults():
        siteconfig.add_defaults(defaults)
        siteconfig.add_defaults(get_django_defaults())

    apply_django_settings(siteconfig, settings_map)
    apply_django_settings(siteconfig)

This is all it takes.

If you take a look at django_settings.py, you’ll notice that we don’t reuse the existing Django settings names and instead provide our own. This is an attempt to cleanly namespace the settings used. You’ll also notice that there’s several settings maps and defaults functions. This is so you can be more picky if you really want to.

Auto-generated Settings Pages

Oh, this is where it gets fun.

Dynamic settings are all well and good, but if you can’t set them through your browser, why bother?

We wanted to make sure that not only could these be modified through a browser, but that it was dead simple to put up pages for changing settings. All you have to do is add a URL mapping and one form per page.

The form doesn’t even need much more than fields.

Let’s start out with some code. It should be pretty self-explanatory.

# urls.py
from myapp.forms import GeneralSettingsForm

urlpatterns = patterns('',
    (r'^admin/settings/general/$', 'djblets.siteconfig.views.site_settings',
     {'form_class': GeneralSettingsForm}),
    (r'^admin/(.*)', admin.site.root),
)
# myapp/forms.py
from django import forms
from djblets.siteconfig.forms import SiteSettingsForm


class GeneralSettingsForm(SiteSettingsForm):
    mystring = forms.CharField(
        label="My String",
        required=True)

    mybool = forms.BooleanField(
        label="My Boolean",
        required=False)

    mail_host = forms.CharField(
        label="Mail Server",
        required=False)


    class Meta:
        title = "General Settings"

That’s it. Now just go to /admin/settings/general/ and you’ll see your brand new settings form. It will auto-load existing or default settings and save modified settings. Like any standard form, it will handle help text, field validation and error generation.

Example Settings Page

Proxy Fields

For a lot of sites, this will be sufficient. For more complicated settings, we have a few more things we can do.

Let’s take the array above. Django’s forms doesn’t have any concept of array values, but we can fake it with custom load and save methods and a proxy field.

# myapp/forms.py
import re

class GeneralSettingsForm(SiteSettingsForm):
    ...

    my_array = forms.CharField(
        label="My Array",
        required=False,
        help_text="A comma-separated list of anything!")

    def load(self):
        self.fields['my_array'].initial = 
            ', '.join(self.siteconfig.get('myarray'))

        super(GeneralSettingsForm, self).load()

    def save(self):
        self.siteconfig.set('myarray',
            re.split(r',s*', self.cleaned_data['my_array']))

        super(GeneralSettingsForm, self).save()

    class Meta:
        save_blacklist = ('my_array',)

What we’re essentially doing here is to come up with a new field not in the settings database (notice “my_array” versus “myarray”) and to use this as a proxy for the real setting. We then handle the serialization/deserialization in the save and load methods.

When doing this, it’s very important to add the proxy field to the save_blacklist tuple in the Meta class so that the form doesn’t save your proxy field in the database!

Fieldsets

Much like Django’s own administration UI, the SiteSettingsForm allows for placing fields into custom fieldsets, allowing settings to be grouped together under a title and optional description. This is done by filling out the fieldsets variable in the Meta class, like so:

class GeneralSettingsForm(SiteSettingsForm):
    class Meta:
        fieldsets = (
            {
                'title':   "General",
                'classes': ('wide',),
                'fields':  ('mystring', 'mybool', 'my_array',),
            },
            {
                'title':       "E-Mail",
                'description': "Some description of e-mail settings.",
                'classes':     ('wide',),
                'fields':      ('mail_host',),
            },
        )

This will result in two groups of settings, each with a title, and one with a description. The description can span multiple paragraphs by inserting newlines (“n“). All fields in the dictionary except for 'fields' are optional.

Re-applying Django settings

When you modify a field that corresponds to a setting in settings.py, you want to make sure you re-apply the setting or you could break something. This is pretty simple. Again override save as follows:

from myapp.siteconfig import load_site_config


class GeneralSettingsForm(SiteSettingsForm):
    def save(self):
        super(GeneralSettingsForm, self).save()
        load_site_config()

This will re-synchronize the settings, making sure everything is correct. Note that down the road, this will likely be automatic.

There’s one last thing to show you…

Disabled Fields

Sometimes it’s handy to disable fields. Maybe a particular setting requires a third party module that isn’t installed. The SiteSettingsForm has two special dictionaries, disabled_fields and disabled_reasons, that handle this. Just override your load function again.

class GeneralSettingsForm(SiteSettingsForm):
    def load(self):
        if not myarray_supported():
            self.disabled_fields['my_array'] = True
            self.disabled_reasons['my_array'] = 
                "This cannot be modified because we don't support arrays today."

        super(GeneralSettingsForm, self).load()

The resulting fields in the HTML will be disabled, and the error message will be displayed below the field.

Depending on whether or not you have a custom administration UI, you may want to tweak the CSS file or completely override it. Note that this all assumes that the djblets/media/ directory exists in your MEDIA_URL as the “djblets” directory. If you have something custom, you can always override siteconfig/settings.html and pass the new template to the view along with your settings form.

More Coming Soon!

There’s some new stuff in the works for the siteconfig app. Auto-detection of defaults and settings maps is the big one. I also have a few more useful tricks for doing advanced things with settings pages that I’ll demonstrate.

If you’re a user of Djblets, let us know, and feel free to blog about it! I’ll link periodically to other blogs talking about Djblets and projects using Djblets.

And as always, suggestions, constructive criticism and patches are always welcome 🙂

Django Development with Djblets: Data Grids

It’s been a while since my last post in this series, but this one’s a good one.

A common task in many web applications is to display a grid of data, such as rows from a database. Think the Inbox from GMail, the model lists from the Django administration interface, or the Dashboard from Review Board. It’s not too hard to write something that displays a grid by doing something like:


{% for user in users %}
 
{% endfor %}
Username First Name Last Name
{{user.username}} {{user.first_name}} {{user.last_name}}

This works fine, so long as you don’t want anything fancy, like sortable columns, reorderable columns, or the ability to let users specify which columns they want to see. This requires something a bit more complex.

djblets.datagrids

We wrote a nifty little set of classes for making data grids easy. Let’s take the above example and convert it over.

# myapp/datagrids.py
from django.contrib.auth.models import User
from djblets.datagrid.grids import Column, DataGrid

class UserDataGrid(DataGrid):
    username = Column("Username", sortable=True)
    first_name = Column("First Name", sortable=True)
    last_name = Column("Last Name", sortable=True)

    def __init__(self, request):
        DataGrid.__init__(self, request, User.objects.filter(is_active=True), "Users")
        self.default_sort = ['username']
        self.default_columns = ['username', 'first_name', 'last_name']
# myapp/views.py
from myapp.datagrids import UserDataGrid

def user_list(request, template_name='myapp/datagrid.html'):
    return UserDataGrid(request).render_to_response(template_name)

Now while this may look a bit more verbose, it offers many benefits in return. First off, users will be able to reorder the columns how they like and choose which columns to see. You could extend the above DataGrid to add some new column but leave it out of default_columns so that only users who really care about it would see it.

Custom columns

Let’s take this a step further. Say we want our users datagrid to optionally show staff members with a special badge. This might be useful to some users, but not all, so we’ll leave it out by default. We can implement this by creating a custom column with image data:

class StaffBadgeColumn(Column):
    def __init__(self, *args, **kwargs):
        Column.__init__(self, *args, **kwargs)

        # These define what will appear in the column list menu.
        self.image_url = settings.MEDIA_URL + "images/staff_badge.png"
        self.image_width = 16
        self.image_height = 16
        self.image_alt = "Staff"

        # Give the entry in the columns list menu a name.
        self.detailed_label = "Staff"

        # Take up as little space as possible.
        self.shrink = True

    def render_data(self, user):
        if user.is_staff:
            return '%s' % 
                (self.image_url, self.image_width, self.image_height, self.image_alt)

        return ""

class UserDataGrid(DataGrid):
    ...
    staff_badge = StaffBadgeColumn()
    ...

This will add a new entry to the datagrid’s column customization menu showing the staff badge with a label saying “Staff.” If users enable this column, their datagrid will update to show an staff icon for any users who are marked as staff.

Custom columns can render data in a couple of different ways.

The first is to bind the column to a field in the model. By default, a Column instance added to a DataGrid will use its own name (such as “first_name” above) as a lookup in the object. If you want to use a custom field, set the db_field attribute on the column. This field can span relationships as well. For example:

class UserDataGrid(DataGrid):
    ...
    # Uses the field name as the lookup.
    username = Column("Username")

    # Spans relationships, looking up the age in the profile.
    age = Column("profile__age")

The second is the way shown above, by overriding the render_data function. This takes an object, which is the object the datagrid represents, and outputs HTML. The logic in this can be quite complicated, depending on what’s needed.

Linking to objects

A datagrid is pretty worthless if you can’t link entries for the object to a URL. We have a couple of ways to do this.

To link a column on a datagrid to a URL, set the link parameter on the DataGrid to True. By default, the URL used will be the result of calling get_absolute_url() on the object represented by the datagrid, but you can override this:

# myapp/datagrids.py

class UserDataGrid(DataGrid):
    username = Column("Username", sortable=True, link=True)
    ...
    @staticmethod
    def link_to_object(user, value):
        return "/users/%s/" % user

Then

link_to_object takes an object and a value. The object is the object being represented by the datagrid, and the value is the rendered data in the cell. The result is a valid path or URL. In the above example, we’re explicitly defining a path, but we could use Django’s reverse function.

Usually the value is ignored, but it can be useful. Imagine that your Column represents a field that is a ForeignKey of an object. The contents of the cell is the string version of that object (implemented by the __str__ or __unicode__ function), but it’s just an object and you want to link to it. This is where value comes in handy, as instead of being HTML output, it’s actually an object you can link to.

If you intend to link a particular column to value.get_absolute_url(), we provide a handy utility function called link_to_value which exists as a static method on the DataGrid. You can pass it (or any function) to the Column’s constructor using the link_func parameter.

class UserDataGrid(DataGrid):
    ...
    myobj = Column(link=True, link_func=link_to_value)

Custom columns can hard-code their own link function by explicitly setting these values in the constructor.

Saving column settings

If a user customizes his column settings by adding/removing columns or rearranging them, he probably expects to see his customizations the next time he visits the page. DataGrids can automatically save these settings in the user’s profile, if the application supports it. Handling this is as simple as adding some fields to the app’s Profile model and setting a couple options in the DataGrid. For example:

# myapp/models.py

class Profile(models.Model):
    user = models.ForeignKey(User, unique=True)

    # Sort settings. One per datagrid type.
    sort_user_columns = models.CharField(max_length=256, blank=True)

    # Column settings. One per datagrid type.
    user_columns = models.CharField(max_length=256, blank=True)
# myapp/datagrids.py

class UserDataGrid(DataGrid):
    ...
    def __init__(self, request):
        self.profile_sort_field = "sort_user_columns"
        self.profile_columns_field = "user_columns"

The DataGrid code will handle all the loading and saving of customizations in the Profile. Easy, isn’t it?

Requirements and compatibility

Right now, datagrids require both the Yahoo! UI Library and either yui-ext (which is hard to find now) or its replacement, ExtJS.

Review Board, which Djblets was mainly developed for, still relies on yui-ext instead of ExtJS, so datagrids today by default are built with that in mind. However, the JavaScript portion of datagrids should work with ExtJS today. The default template for rendering the datagrids (located in djblets/datagrids/templates/datagrids/) assumes both YUI and yui-ext are present in the media path, but sites using datagrids are encouraged to provide their own template to reflect the actual locations and libraries used.

We would like to work with more JavaScript libraries, so if anyone wants to provide an implementation for their favorite library or help to dumb down our implementation to work with any and all, we’d welcome patches.

See it in action

We make extensive use of datagrids on Review Board. You can get a sense of it and play around by looking at the users list and the dashboard (which requires an account).

That’s it for now!

There’s more to datagrids than what I’ve covered here, but this should give you a good understanding of how they work. Later on I’d like to do a more in-depth article on how datagrids work and some more advanced use cases for them.

For now, take a look at the source code and Review Board’s datagrids for some real-world examples.

Django Tips: PIL, ImageField and Unit Tests

I recently spent a few days trying to track down a bug in Review Board with our unit tests. Basically, unit tests involving uploading a file to a models.ImageField would fail with a validation error specifying that the file wasn’t an image. This worked fine when actually using the application, just not during unit tests.

The following code would cause this problem. (Note that this isn’t actual code, just a demonstration.)

# myapp/models.py
from django.db import models

class MyModel(models.Model):
    image = models.ImageField()
# myapp/forms.py
from django import forms
from myapp.models import MyModel

class UploadImageForm(forms.Form):
    image_path = forms.ImageField(required=True)

    def create(self, file):
        mymodel = MyModel()
        mymodel.save_image_file(file.name, file, save=True)
        return mymodel
# myapp/views.py
from myapp.forms import UploadImageForm

def upload(request):
    form = UploadImageForm(request.POST, request.FILES)

    if not form.is_valid():
        # Return some error
        pass

    # Return success
# myapp/tests.py
from django.test import TestCase

class MyTests(TestCase):
    def testImageUpload(self):
        f = open("testdata/image.png", "r")
        resp = self.client.post("/upload/", {
            'image_path': f,
        })
        f.close()

After a good many hours going through the Django code, expecting there to be a problem with file handles becoming invalidated or the read position not being reset to the beginning of the file, I finally decided to look at PIL, the Python Imaging Library. It turns out that at this point, due to some weirdness with unit tests, PIL hadn’t populated its known list of file formats. All it knew was BMP, and so my perfectly fine PNG file didn’t work.

The solution? Be sure to import PIL and call init() before the unit tests begin. If you have a custom runner (as we do), then putting it in here is appropriate. Just add the following:

from PIL import Image
Image.init()

Hopefully this helps someone else with the problems I’ve dealt with the past few days.

Django Development with Djblets: Unrooting your URLs

Typically, Django sites are designed with the assumption that they’ll have a domain or subdomain to themselves. Often times this is fine, but if you’re developing a web application designed for redistribution, sometimes you can’t make that assumption.

During development of Review Board, many of our users wanted the ability to install Review Board into a subdirectory of one of their domains, rather than a subdomain.

There’s a few rules that are important when making your site relocatable:

  • Always use MEDIA_URL when referring to your media directory, so that people can customize where they put their media files.
  • Don’t hard-code URLs in templates. Use the {% url %} tag or get_absolute_url() when possible.

These solve some of the issues, but doesn’t address relocating a Django site to a subdirectory.

Djblets fills in this gap by providing a special URL handler designed to act as a prefix for all your projects’ URLs. To make use of this, you need to modify your settings.py file as follows:

settings.py

TEMPLATE_CONTEXT_PROCESSORS = (
    ...
    'djblets.util.context_processors.siteRoot',
)

SITE_ROOT_URLCONF = 'yourproject.urls'
ROOT_URLCONF = 'djblets.util.rooturl'

SITE_ROOT = '/'

SITE_ROOT specifies where the site is located. By default, this should be “/”, but this can be changed to point to any path. For example, “/myproject/”. Note that it should always have a leading and a trailing slash.

The custom template context processor (djblets.util.context_processors.siteRoot) will make SITE_ROOT accessible to templates.

SITE_ROOT should be used in templates when you need to refer to URLs that aren’t designed to respect SITE_ROOT (such as User.get_absolute_url). Your own custom applications should always respect SITE_ROOT whenever providing a URL.

ROOT_URLCONF is typically what you would set to point to your project’s URL. However, in this case, you’ll be pointing it to djblets.util.rooturl. This in turn will forward all URLs to your project’s handler, defined in SITE_ROOT_URLCONF.

This is all you need to have a fully relocatable Django site!

To sum up:

  1. Add djblets.util.context_processors.siteRoot to your TEMPLATE_CONTEXT_PROCESSORS.
  2. Set SITE_ROOT_URLCONF to your project’s URL handler.
  3. Set ROOT_URLCONF to ‘djblets.util.rooturl’
  4. Prefix any URLs with SITE_ROOT in your templates, unless the URL would already take SITE_ROOT into account.

This is functionality that will hopefully make its way into Django at some point. For now, you have an easy way of unrooting your Django project.

Django Development with Djblets: Custom Tag Helpers

I’m planning to cover all of what Django can do, but for now, let’s start simple with something most Django developers spend way too much time creating: Custom tags.

Django’s nice enough to provide a @register.simple_tag decorator for creating very basic tags that don’t take a context but do take parameters. This is great, but what if you want more? Many Django applications use the same boilerplate time and time again to create their tags, but we make it much easier.

Introducing @basictag and @blocktag.

@basictag

Ever wanted to use Django’s @register.simple_tag but needed access to the context? I’ve found far too many cases where this would be useful, but Django doesn’t make this easy. Your tag code would end up looking like this:

class MyTagNode(template.Node):
    def __init__(self, arg1, arg2):
        self.arg1 = arg1
        self.arg2 = arg2

    def render(self, context):
        arg1 = Variable(self.arg1).resolve(context)
        arg2 = Variable(self.arg2).resolve(context)

        return context['user']

@register.tag
def mytag(parser, token):
    bits = token.split_contents()
    return MyTagNode(bits[1], bits[2])

Do this a few times and it’s bound to drive you nuts. How about this instead?

from djblets.util.decorators import basictag

@register.tag
@basictag(takes_context=True)
def mytag(context, arg1, arg2):
    return context['user']

Far less code and increased readability. Hooray!

@blocktag

@blocktag aims to do the same thing @basictag does but for block tags. A block tag is a tag that contains nested content, like @spaceless or @for. This usually requires even more boilerplate than the above code fragment, except with the added complexity of having to grab the contents of the block.

We’ve condensed it down to this:

from djblets.util.decorators import blocktag

@register.tag
@blocktag
def blinkblock(context, nodelist, arg1, arg2):
    return "<blink>%s</blink>" % nodelist.render(context)

If you’ve built block tags in the past, you’ll appreciate how simple that was.

Django Development with Djblets

Django is an awesome development platform for web applications. With such features as database abstraction, template/view/url separation, built-in authentication with interchangeable backends, it’s made web development much more enjoyable.

We use Django in Review Board with much success. Over time, as we’ve come to develop new features, we realized that much of our codebase was useful outside of Review Board and, bit by bit, moved pieces into a library we call Djblets.

What does Djblets do?

A bit of everything, really. Any time we have useful functionality that isn’t tied to Review Board, we put it here.

Djblet’s feature list currently consists of:

  • Authentication improvements, making it easy to register and login in one step, seamlessly, handle password recovery, and more.
  • Flexible datagrids for displaying data in a paginated list with user-specific column customization, ordering and sorting.
  • Decorators to drastically simplify creation of simple and block template tags.
  • Caching functions for calling a function and caching the result if the data isn’t already in the cache, and a special URL pattern matcher that prevents caching of any contained URLs.
  • Unit testing utility classes.

And of course more little things here and there.

Downloading Djblets

Djblets is not a released app, but it’s pretty stable and well tested. You can check out a copy from our SVN repository, or automatically include it in your own repository through an svn:externals entry.

Djblets is licensed under the MIT license, making it usable in most projects.

Using Djblets

Over time I’ll be writing articles on using the many features of Djblets. See the other posts in the series, or dig around the Djblets source code.

Review Board: The Past 5 Months

It’s been about 5 months since I last gave a Review Board status update. Way too long, given how much has changed. So once again, let’s start off with a few stats.

  • Total bugs open: 25
  • Total bugs fixed: 186
  • Feature requests open: 43
  • Companies known to be using Review Board: at least 23

Review Board has matured in recent months and has a very nice feature set. More and more we’re seeing and hearing about companies using it in one or more teams. I was even lucky enough to talk about it in the official Django Book.

Development shows no sign of stalling. Our feature request list is a mile long, and our personal TODO lists are longer still. We’ve implemented so many new features and fixed so many bugs that I can’t even list them all, but let’s take a look at the highlights.

New Top-Level Features

  • iPhone support. Basic read-only iPhone support was added. It’s more of a proof of concept and to make sure our codebase handles different UIs on top of it, but if your Review Board server is accessible from your iPhone, point Safari to /iphone/ for some fun.
  • Status Reports. Basic support for status reports have been added. While we don’t have a fleshed out UI in place, the /reports/ URL will give you simple reports showing which review requests you’ve reviewed, and other bits of information. This can even be presented in Wiki format!

Revision Control Systems Integration

  • Mercurial support. One of our contributors has written support for doing review requests against Mercurial repositories. This supports local and remote repositories.
  • Git support. Basic Git support was written as well. It only works with local Git repositories (as it has to have access to .git directories).

Diff Viewer

  • Improved diff parser. We no longer require third party tools in order to parse diff files. We can do it ourselves faster and with greater flexibility. This has given us some speed advantages, reduced the hacks needed, and improved diff compatibility.
  • Interdiffs. Review Board can now display the differences between two revisions of a diff. This makes it much easier to review several iterations of large changes spanning many files.
  • Fixed diff line numbers. Line numbers in the diff viewer used to be artificial. They were essentially table row numbers. Now line numbers on the left-hand side of the diff viewer represent the actual line numbers in the original file, and line numbers on the right represent the new file.
  • “Review” link. Added a “Review” link on the diff viewer and screenshot page for bringing up the Review dialog. Previously users had to click a link on the diff viewer regardless of whether they were leaving a comment on the diff.

Reviews

  • Show commented screenshots on reviews. Portions of a screenshot that the user has commented on will appear in the review body, much like diff fragments do. This greatly improves the review process when it comes to screenshots.
  • Updated diffs create a new draft. Newly updated diffs used to instantly appear and send out an e-mail, which was annoying if you realized you needed to change and re-upload the diff again. Now updating the diff just creates a draft, if one doesn’t already exist. The diff won’t show up or spam users until you’re ready for it to.
  • Auto-completion for reviewers. The reviewer lists now have support for auto-completion of group names and usernames. As you’re typing, a list of choices based on the current text will appear. Navigating with the arrow keys or hitting Tab will auto-complete the selected entry.
  • Default reviewers. Administrators can now specify default reviewers for file paths (as defined by a regular expression). This allows certain groups to “own” files or paths and to be included on the reviewers list any time a diff touching those is uploaded.
  • Improved page banners. The draft banner at the top of the review request page has been improved and is now more clear. No longer do you have to save a draft and then publish it. It’s now one single button on the banner. We also added “discarded” and “submitted” banners.
  • Alpha-numeric bug numbers. Not all bug trackers use numeric-only bug identifiers. We now support alpha-numeric bug numbers.

Dashboard and Review Request Lists

  • Starred review requests and groups. Users can now “star” a review request they wish to keep track of in their dashboard. They’ll be placed on the CC list and see changes in the dashboard. Users can also star a group in order to add it to their “Watched Groups” list in the dashboard.
  • Toggle display of submitted review requests. The “All Review Requests” page can now filter out submitted review requests via a toggleable “Show/Hide Review Requests” link.
  • Customizable columns. The various lists pages and the dashboard now support customizable columns. There are non-default columns that can be added to the view to show extra data, and existing columns can be removed. If you prefer all dates to be relative or absolute, just add the right columns. Furthermore, columns can be reordered simply by dragging them into the desired order.
  • New dashboard column types:
    • New Updates. This column shows a speech bubble icon when new discussions have taken place on a review request since the user last visited it.
    • Ship It. This column makes it easy to see if anybody has marked the review request as “Ship It!”
    • Absolute/relative timestamps. Users wishing to see only relative or absolute timestamps in the dashboard can add the Last Updated/Posted Absolute or Relative timestamp columns.
    • Number of Reviews. Sometimes it’s handy to see how many reviews have been made to a review request. This column provides that number.
    • Starred. Allows users to star/unstar a review request or group. This is like adding yourself to a CC list.

post-review

  • Upload diffs from a revision range. --revision-range has been added to allow for uploading diffs from a range of revisions on the server. This is currently only implemented for SVN.
  • Specify a default summary. --summary has been added to provide a default summary for the review request.
  • Open a browser after uploading. --open has been added to open a browser to the new review request.

Distribution/Installation

  • make install. It’s now trivial to create a Review Board tarball or to install it on your system. We integrate with autoconf/automake to generate the Makefiles and sample/default configuration files. This brings us a giant step closer to putting out releases.

What’s Next?

We have several things in the works. A couple of the major highlights would be a search interface and support for 3rd party extensions to Review Board. Stay tuned!