Representing Relationships in Django Templates Without Writing Extra Code (RelatedManager and ManyRelatedManager)

I’m writing an application that deals with some slightly complex relationships. There are several offices, and each office has several workers. Workers can have multiple projects, and each project can have multiple workers. In addition, each project can serve multiple clients.

Here’s what that’d look like in a Django models.py file:

class Office(models.Model):
   office_code = models.CharField(max_length=24, blank=True)
   street_num = models.CharField(max_length=24)
   street_name = models.CharField(max_length=64)
   bldg_no = models.CharField(max_length=12, blank=True)
   suite = models.CharField(max_length=12, blank=True)
   city = models.CharField(max_length=100)
   state = USStateField() # grandiose assumption
   zipcode = models.CharField(max_length=10)
   main_phone = PhoneNumberField()
   site_mgr = models.ForeignKey(User, unique=True)

class Worker(models.Model):
   user = models.ForeignKey(User, unique=True)
   extension = models.CharField(max_length=8)
   office = models.ForeignKey(Office)

class Client(models.Model):
   fname = models.CharField(max_length=64)
   lname = models.CharField(max_length=64)
   street_num = models.CharField(max_length=24, blank=True)
   street_name = models.CharField(max_length=128, blank=True)
   apt_no = models.CharField(max_length=24, blank=True)
   city = models.CharField(max_length=128, blank=True)
   state = USStateField(blank=True)
   zipcode = models.CharField(max_length=10, blank=True)

class Project(models.Model):
   date_started = models.DateTimeField()
   worker = models.ManyToManyField(Worker)
   office = models.ForeignKey(Office)
   client = models.ManyToManyField(Client)

While writing the template for my worker detail page, I decided that I didn’t want to just list the projects for that worker, but I also wanted to list the clients for each project. I ran into a bit of an issue at first in doing this. I tried something like this:

{% block content %}
   <h2>Projects for {{object.user.first_name}} {{object.user.last_name}}</h2>
   <ul>
   {% for project in object.projects %}
      <li><a href="{{project.get_absolute_url}}">{{project.id}} (Opened: {{project.date_started.date}})</a>
      <ul>
         {% for obj in project.get_clients %}
            <li>{{obj.lname}}, {{obj.fname}}</li>
         {% endfor %}
      </ul>
   <h2>Clients for project {{project.id}}</h2>
{% endfor %}</ul>
{% endblock %}

Looking back at the models, you’ll note that there’s no “projects” attribute of the Worker class. There’s also no “get_clients” method for the Project class. After reading some forum and blog posts, I got the idea to add these to my models manually. It seems a lot of people solve similar issues this way, and I don’t believe it’s necessary, which is why I’m posting this. What I added to my models looked something like this:

###
### in Worker model
###
def projects(self):
   return self.project_set.filter(worker = self.pk)

###
### in Project model
###
def get_clients(self):
   return self.client_set.all()

Adding these to the models actually does solve the problem, but it’s reinventing the wheel. Perhaps at some point in history Django’s ORM didn’t have the functionality it does now, but these days Django takes care of accessing the objects of related entities for you through the use of the RelatedManager (for one-to-one, foreign key relationships) and the ManyRelatedManager (for many-to-many relationships).

When you create a ForeignKey field or ManyToMany field in a Django model, Django’s ORM becomes aware of the relationship, and implements lots of shortcuts to help you in managing/exploiting it.

After reading some online documentation (see the “Related Objects” area, for one), I was able to get all of the data I wanted into my template without adding a single character of code to my models, which is what I had hoped for. Here’s the right way to do this, unless I’m mistaken (please let me know the best way if I am):

{% block content %}
<h2>Projects for {{object.user.first_name}} {{object.user.last_name}}</h2>
<ul>
   {% for project in object.project_set.all %}
      <li><a href="{{project.get_absolute_url}}">{{project.id}} (Opened: {{project.date_started.date}})</a>
      <ul>
         <h2>Clients for project {{project.id}}</h2>
         {% for obj in project.client.all %}
            <li>{{obj.lname}}, {{obj.fname}}</li>
         {% endfor %}
      </ul>
   {% endfor %}
</ul>
{% endblock %}

I’ve replaced “object.projects” with “object.project_set.all”. Note that, in a Django template, unless you specify otherwise, the name of a single object passed to a template is “object”, so in this case, “object” is a “Worker” object. The Worker model makes no mention at all of projects, and yet I’m able to easily grab data about project objects. This is because Django’s ORM actually gives you access to related object data from either side of a relationship. Since the Project model has a ManyToMany field referencing Worker, you can access project data from the worker object by using “object._set.all”, where is replaced with the lower-cased name of the model that points to “object”. Hence, “object.project_set.all”.

Now, in the second case, I’ve replaced “project.get_clients” with “project.client.all”. The Project model directly contains a field named “client” that is a ForeignKey to the Client model. When this condition exists, Django will happily traverse the relationship for you by just referencing the model’s field directly! The “all” method is a standard method of any Manager object I’m aware of, and it’s inherited by the RelatedManager and ManyRelatedManager objects.

One interesting thing I found, too, was that there’s no mention of “RelatedManager” or “ManyRelatedManager” in the online Django documentation. This is highly unusual. In my experience, Django’s documentation blows away the docs for just about any project in existence. Did I miss something?

Lessons Learned While Creating a Generic Taxonomy App for Django

So, when I first picked up a guitar, the first song I sat down to learn, by ear, was Stairway to Heaven, not “Twinkle, Twinkle, Little Star”. So goes my experience with Django :)

The Background

I was humming along on my recreation of LinuxLaboratory.org. I got a simple blog in place in just a couple of days, a code-sharing app in place a few days later (if that), and a very simple CMS I threw together using flatpages. A good bit of the base code I used came from the 2nd edition of “Practical Django Projects”, but I soon veered off in other directions, and started analyzing the work I’d already done a bit more closely.

One of the things that was glaringly obvious to me was that my method of classifying content was a little schizophrenic. I had three separate apps to represent different types of content, which is great, but each separate app had its own “Category” model. Yuck. On top of that, I was using django-tagging to enable tagging in addition to the categorization each app supported.

The Problem

So… for one type of classification (Categories), it’s built into the specific application, and for the other (tagging), it’s not built in, but it’s pretty tightly coupled. There are a few fundamental drawbacks to this approach:

First, you have to make a pretty big commitment to these things. The easiest way to implement them in your app is to add support for them at the outset, because adding them in later is going to be a bit of a headache. Categories aren’t quite so bad — I implemented them the way the book does, which is with a ManyToMany field. In Django, when you create a ManyToMany field in a model, there’s no corresponding field in that model’s table in the database. Instead, Django creates a lookup table for you, which is nice, because it means you *could* add categories at a later time without *too* much trouble. Tags use the django-tagging app, which implements tags as a multi-valued field in the database table representing the model that will use the tags. So adding this in later is a little bit more of a hassle.

The second issue is that this approach doesn’t treat classification in a consistent manner. One is in the app, the other is a separate app, one is a field, the other is a model, one affects the model’s table, the other doesn’t, etc. One place where this inconsistency becomes obvious is in your templates, where you’re likely to want to give users the ability to browse by category, or browse by tag. Browsing by category across all the different content types is going to be pretty tough if they all have their own implementation. Tags are a little easier, but it’s still a little cumbersome.

The third issue is specific to Categories, and has to do with maintenance: if I come up with some fantastic idea for the Category implementation (like, I dunno, subcategories?), I have to implement it separately in all of the apps that are using categories. No Bueno™.

The Dream

Wouldn’t it be nice if you could just say “give me a list of the taxonomy types used to classify this piece of content, whatever it is, whichever app it comes from, and also a list of the taxonomy terms involved”? Wouldn’t it be nice if you could just add support for categories and tags using a single app that doesn’t add anything to your existing tables? Wouldn’t it be nice to be able to come up with your own taxonomies and, perhaps, hierarchical taxonomies and more complex relationships? Wouldn’t it be nice to be able to say “here’s a category name, show me all of the content associated with it, sorted by content type” and then change your mind and say “no wait, show it to me sorted by title, with a content type indicator over here”, and then change your mind again and say “er, how about showing the taxonomy type, then the term, then all of the content objects under that type:term pairing”?

The Solution

I think it would, so I started creating this beast that I just call “taxonomy”. Right now it’s pretty simplistic, and it’ll likely change slightly based on some things I’ve learned, but surprisingly, I think my first shot at it is really darn close! I’ve stopped being surprised at how quickly I’m able to prototype in Django: getting this together, including the creation of the models, getting it into the admin interface, and getting it linked with any random content type from any app that wants to use it within the admin interface (to add taxonomies and labels to a piece of content in the ‘edit’ interface) took probably 4 hours, including time to read documentation and fall down a few times.

The admin interface for taxonomy lets you create a taxonomy, so if none of your apps currently don’t support the notion of a “Category”, you can go create a taxonomy called “Category”. Once that’s there, you can create a “taxonomy term”, where you’d select the “type” for this term (your new Category), and then a term. So if your term was “Django”, then you would have just created a category called “Django” that could be used by any other app/model in your project. The same, of course, would go for tags, and whatever other classification devices you want.

There’s support for parent-child relationships at the taxonomy term level (so you can have subcategories, or even subtags if you want, etc. I guess you could even categorize tags, and tag categories! They’re coming to take me away!!!). I haven’t given much thought to having hierarchical relationships for the taxonomies themselves. That would be a little overboard, no? I’m interested to hear realistic use cases for that :)

Once you’ve created a taxonomy and a term, the next thing to do is figure out how to associate your actual content to it. So, if our taxonomy is “Category” and the category name (the “term”) is “Django”, the way I’ve implemented it is that you’d go into the edit interface for the article, and a form for associating it with your category appears. This was created using a GenericInlineModelAdmin, which was a gem of a find in the documentation. Inlines let you easily create a form to update a piece of content using concepts and attributes from other models, and even other applications. If you don’t know much about Django, this sounds like a big mess, but in reality, it’s fairly elegant.

I’ve done some testing to see that I can pull things out of the database and associate things properly in the presentation layer, but I’d like to work on making it smoother before I go releasing code or anything like that…. which reminds me that I *did* look and ask around about an app that maybe already did this and came up dry. If anyone sees this and says “why not just use x”, let me know, because it’s not really a goal to write code for the sake of coding. I actually thought this was an interesting feature and couldn’t find it.

Lessons Learned #1: URLConf is a Choice, Not a Requirement

First, I learned that it’s completely possible to create an application that doesn’t have a URLConf at all. Currently, taxonomies actually work in testing, and there’s no URLConf. There actually *will* be one when I figure out how I want the data to be used on my own site, and how to enable users to do whatever they want with it as well. One thought, for example, is that it would be really awesome to be able to go to “/categories/django” and have my app somehow “just know” that “categories”, when singularized, is “category”, which is a taxonomy. From there, the taxonomy app takes over, and magic happens. I have faith that I can make this happen without having the word “taxonomy” in the url. We’ll see.

Anyway, the point is that you don’t have to have a URLconf, and that hadn’t really occurred to me. For the record, django-tagging also doesn’t have a URLconf.

Lessons Learned #2: ContentTypes Let Your Models Be All-Knowing

The second thing I learned was that, using the ContentTypes framework within Django, it’s possible to create a model that will deal with data, and relationships to data, in a dynamic way, such that you don’t have to know what type of data your models will be working with at the time you create them.

For example, my taxonomy app can be used with my blog’s “Entry” model, my code-sharing app’s “Snippet” model, and my CMSs “Page” model. If I pass the app to you, you can use it for your news site’s “Story” model, your ad network’s “Ad” model, and your Twitter clone’s “Tweet” model. No problem. This is in the docs, but here’s what I’ve done:

class Taxonomy(models.Model):
 """A facility for creating custom content classification types"""
 type = models.CharField(max_length=50, unique=True)

class TaxonomyTerm(models.Model):
 """Terms are associated with a specific Taxonomy, and should be generically usable with any contenttype"""
 type = models.ForeignKey(Taxonomy)
 term = models.CharField(max_length=50)
 parent = models.ForeignKey('self', null=True,blank=True)

class TaxonomyMap(models.Model):
 """Mappings between content and any taxonomy types/terms used to classify it"""
 term        = models.ForeignKey(TaxonomyTerm, db_index=True)
 type        = models.ForeignKey(Taxonomy, db_index=True)
 content_type = models.ForeignKey(ContentType, verbose_name='content type', db_index=True)
 object_id      = models.PositiveIntegerField(db_index=True)   
 object         = generic.GenericForeignKey('content_type', 'object_id')

Note that I’ve removed some stuff from the model defs — what you see here are just the fields, which are the relevant bit for what I’m explaining.

The TaxonomyMap model (a model is a class definition, by the way) has foreign keys to map to a human readable ‘term’ and ‘type’ in the other models. TaxonomyMap is just to store mappings between content objects and taxonomies (lower-level details of this might change to make it cleaner/more efficient – I know it’s not perfect). So, how does my app know that I’m storing a mapping to an “Entry” from my blog app? How does it get the id for that Entry? What’s going on?

Well, Django stores a list of every content type used by Django and any installed apps, and I’ve made a foreign key to ContentType so I can access the content type of the object that’s being dealt with and get its ID. I also have a “GenericForeignKey” field, which essentially creates a “dynamic” foreign key to the table that represents the object that’s being dealt with, so if I’m dealing with an “Entry” object from “monk” (which is the name of my blog app), then the foreign key will point to “monk_entry”, which is the table that stores my blog entries. When you create a taxonomy, and a term, and associate them to a piece of content, the resulting rows in the affected tables look like this:

mysql> select * from taxonomy_taxonomy;
+----+--------------+
| id | type         |
+----+--------------+
|  1 | TestCategory |
+----+--------------+
1 row in set (0.01 sec)

mysql> select * from taxonomy_taxonomyterm;
+----+---------+----------+-----------+
| id | type_id | term     | parent_id |
+----+---------+----------+-----------+
|  1 |       1 | TestTerm |      NULL |
+----+---------+----------+-----------+
1 row in set (0.00 sec)

mysql> select * from taxonomy_taxonomymap;
+----+---------+---------+-----------------+-----------+
| id | term_id | type_id | content_type_id | object_id |
+----+---------+---------+-----------------+-----------+
|  1 |       1 |       1 |              10 |         2 |
+----+---------+---------+-----------------+-----------+
1 row in set (0.00 sec)

Note that the table for the model that’s using the taxonomy app is untouched. Only taxonomy tables are used.

Seeing this, you might think that it’d be hard to put a form in the admin interface for arbitrary content types to classify them with taxonomies. Not so — which brings me to more lessons I learned.

Lessons Learned #3: Collecting Data About a Model Without Extending the Model and Creating Database Badness

If you have a model (we’ll use “Entry” again), and it has a core set of attributes, but you want to associate data with instances of this model not represented in the model definition (like, say, a taxonomy, for an arbitrary example), you can add a form to the admin interface for that model in about 5 minutes. This rocks, for those who didn’t know, because the alternative would either involve really ugly code, or really ugly data (you’d have to store the taxonomies in the table for the model, creating either tons of duplicate data, or multi-valued fields… and you’d still have duplicate data).

Typically, it seems that the normal use case for this is to relate models in the admin interface that are part of the same application and are explicitly related through a direct foreign key reference. This might even be enforced in the case of “InlineModelAdmin” objects, but I haven’t dealt with those personally. However, while reading about “GenericInlineModelAdmin” objects, it occurred to me that it shouldn’t matter that the related items are from different apps. I tried it, and it worked. Here’s what I did:

from django.contrib import admin
from django.contrib.contenttypes import generic
from monk.models import Entry
from taxonomy.models import TaxonomyMap

class TaxonomyMapInline(generic.GenericTabularInline):
   model = TaxonomyMap

class EntryAdmin(admin.ModelAdmin):
   prepopulated_fields = { 'slug': ['title'] }
   inlines = [ TaxonomyMapInline, ]

admin.site.register(Entry, EntryAdmin)

Again, I’ve edited out the irrelevant bits. The above comes from my blog app’s admin.py file. What I did was created an “inline” called “TaxonomyMapInline”, and then associated that inline with the “EntryAdmin” ModelAdmin object using ModelAdmin’s ‘inlines’ attribute, which takes a Python list, which means you can keep adding more inlines all day long if you like.

The result is that, when I go to edit a blog entry, there’s now a form at the bottom that lets the user select a taxonomy type and term (i.e. “Category” “Django”), and associate it with the post. When I added the inline to the admin.py file, it was a test to see what would happen. Since TaxonomyMap doesn’t hold anything but numeric IDs, I assumed I would have to go back and manually map the IDs to human readable values. Not true. Apparently, if the field being presented in the admin form maps to a ForeignKey field, Django automagically does the lookup for you and presents the human-readable text! And, when you save, it converts everything back to numeric IDs before going to the database, so everything “just works”. So the work I thought I’d be doing myself was already done for me!

Using a robots.txt File With Django and Apache (on Webfaction)

I’ve developed in a few different environments, including multi-tier ones with middle tier Java app servers and stuff, but it always seemed pretty straightforward to serve something directly from disk. And in the case of PHP, everything is served from disk. There’s no middleware to speak of, so you can throw a robots.txt file in place and it “just works”. With Django, it’s slightly different because of two things:

  1. Django shouldn’t be serving static content (and therefore makes it a little inconvenient though not impossible to do so).
  2. Django works kinda like an application server that expects to receive URLs, and expects there to be some configuration in place telling it how to deal with that URL.

If you have Django serving static content, you’re wasting resources, so I’m not covering that here. My web host is webfaction, and they give you access to the configuration of your own Apache instance in addition to your Django installation’s configuration (in fact, I’m just running an svn checkout of django-trunk), so this gives you a lot of flexibility in how you deal with static files like CSS, images, or a robots.txt file. To handle robots.txt on my “staging” version of my site, I added the following lines to my apache httpd.conf file:

LoadModule alias_module modules/mod_alias.so
<Location "/robots.txt">
 SetHandler None
</Location>
alias /robots.txt /home/myusername/webapps/mywsgiapp/htdocs/robots.txt

If you don’t add mod_alias, you’ll get an error saying that the keyword “alias” is a misspelling or is not supported by Apache. I use “<Location>” here instead of “<File>” or “<Directory>” because I’m applying the rule only to incoming requests for “/robots.txt” explicitly, and it isn’t likely that I’ll have more than one way of reaching that file, since I’m not aware of engines that look for robots.txt in some other way. <Directory> applies rules to an entire directory and its subdirectories, and <File> applies rules to a file on disk so the rules will apply even if there’s more than one URL that maps to the file.

Django Settings in Dev and Production: Why the hoops?

So, I’ve taken a break from active development on my project to take a step back and really get a good development workflow together. I’ve been fighting with various components of my development workflow, and in the end decided to compromise: I won’t have something that looks exactly like production, but I’ll have something that works and is easy to use. I’ll make up for it by having a staging environment on the same host as the production deployment, which will catch any differences between dev and production that result from my non-identical setup on my laptop.

In getting things going, one thing I ran into immediately was that things on my dev box are different from production: database credentials, paths to media, etc., and since I’m not using Apache and a reverse proxy some of the paths in settings.py will also be different. So what to do?

Turns out there’s an entire page on the Django wiki detailing the ways in which people keep their dev and production settings from trampling each other. There are also various blog posts where people have come up with interesting ways to make things work properly, usually by taking advantage of the fact that your app’s settings are just python code. As such, you can perform any valid Python wizardry you want to make the right things happen.

I don’t think there’s anything necessarily wrong with doing all of that stuff, but what I’m wondering is this: why not just tell your version control system of choice to *ignore* your settings.py file? This way, you can have a settings.py file on your dev box that works perfectly for your dev environment, and a separate one in production that works perfectly for that environment. Never the twain shall meet.

If you’re using one of the methods described on the Django site or any of the blog posts, what are you getting out of it that couldn’t be accomplished by ignoring settings.py? It just seems like it’s simpler and cleaner to do it that way, but I assume there’s some benefit to jumping through these hoops that I’m missing.

Input is hereby solicited. Please tweet/link/post this wherever, ‘cos I’d like some opinions on the matter.

Django App Design: Extend or Build From Scratch?

Django has proven itself (to me, anyway) as a great tool for the job of creating very robust web applications with lightning speed. In just a week or two, I’ve created a site that encompasses a blog with an akismet-powered comment moderation system, a code sharing section complete with highlighted code segments (and line numbers), and I have a way to create static HTML pages and map them to whatever URL I want. Behind all of this is an administrative area where I can add/edit/delete any type of content hosted on the site, add users, put them in groups, and even create new sites, and map content to the different sites. It’s pretty powerful.

One thing I really like about Django is that it lets you be sort of a commitment-phobe, and actually rewards you for it. If you’re not sure which direction you want to go in with a given feature, just make it as generic as possible. Not sure which URL you want some content to show up at? No problem — edit a line in the project’s URLConf, and it’s somewhere else now. Need to redo that sidebar menu to pull in content from a completely new app? No problem. Edit a template with a couple of new tags, delete some others, and if you’ve set up inheritance properly, that’s it. Done. Decided you want users to be logged in to see some content? Multiple ways to do that – edit the view, edit the template… easy, easy, easy.

And did I say powerful?

However, with great power, of course, comes great responsibility. One thing I don’t want to do is create some monstrosity of code with tightly-coupled entities, because that takes away my ability to be non-commital and make on-the-fly changes with the greatest of ease. As it turns out, being non-commital takes a bit of thought, and a bit of work.

So here’s the deal…

I have a blog app which I think is extremely similar to the blog app in Practical Django Projects, 2nd Edition. It has an Entry object which has some rather nice features that I think are generically useful regardless of whether this “Entry” happens to be a blog post, an article in a CMS, a code snippet, or whatever. And this is the root of my design dilemma.

I’d like to devote a section of my site to republished articles I’ve written over the years, and new ones I plan to write going forward. Having used Django’s built-in flatpages app for static HTML content, I can already see that it’s really not up to doing double-duty as a CMS. Specifically, it doesn’t have the notion of status, so I can’t mark something “Draft” and come back to it later. It also requires me to put in a URL to create a static mapping between a URL and a flatpage object. Also, flatpage objects don’t have attributes I’d like to have like “Original Publication Date”, “Original Publication URL”, “Last Updated”, etc. There’s no notion of categories, tags, or anything like that, either, which I think could be useful in presenting the list of available articles.

But my blog app has a good bit of that functionality, and my Entry object has a good number of the attributes I want. So where to from here?

Well, I haven’t looked deeply into this, but I know I could subclass Entry, make the one or two changes I’d need, and go that route. Unfortunately, doing this basically means that the CMS isn’t really “standalone”. It depends on my blog app.

The other option is to build the features I want *into* the blog app itself. I haven’t looked deeply into this either, but I don’t think it would be all that difficult to create a way for me to specify at the time I create an “Entry” that this piece of content is a “Blog Post” or an “Article”. However, at that point I’m basically turning my standalone blog app into something that is not just a blog app. If I ever built another site and only wanted a blog, I couldn’t just move my blog app into place and have it. Or… I could, but I’d have all this “Article” cruft hanging around in my admin interface.

This is all not to mention that I’d be breaking a tradition of keeping things small and simple, both to ease debugging, and to ease future development of the small, simple app. Everything becomes more complex. Maybe I’m answering my own question…

What to do?

I’m unsure which way to take it, to be honest. My thinking right now is that I’ll just copy my blog app’s models.py file to my new app, make my changes there in the new app, and ‘syncdb’. Sure, it’s redundant… but it’s only redundant *today*. If I decide later on to add a feature to the blog that makes absolutely no sense for the CMS (or vice versa), I win. If I decide to create a new site that only needs a CMS *or* a blog, I also win.

I’m always open to suggestions and bits of wisdom from folks who’ve been down this road before. If you see big gaping flaws in my logic, let me know. Otherwise, I’m gonna create some short-term redundancy in exchange for some long-term flexibility.

Create a Tagging Index Page with django-tagging

For those not following along, I’ve been recreating LinuxLaboratory.org using Django. It’s my first Django project that you could call “real work”. I’ve been using the Django documentation, various blogs, and the 2nd edition of “Practical Django Projects”, which has given me a lot of ideas, and a lot of actual code. Some of it worked, some of it didn’t, some of it didn’t do things the way I wanted, and some of it was left as an exercise to the user. This was the case with some of the django-tagging-related stuff, which has been broken on the site for a while.

I finally got tired of tagging not working properly on LinuxLaboratory.org, so I started diving into the code and found that one of the things I wanted to do was actually pretty darn easy. In the process, I thought of something else I’ll probably implement later as well. Not *all* of my problems are solved, but I’m on my way!

So, Linuxlaboratory is made up of three different sections: a blog, which is its own app, a “Code” area which is another separate app, and a content management system (flatpages right now) that will handle storing republished articles when I get around to importing them all.

I enabled tagging on everything. I’m not solidly in one camp or the other on the whole “Tagging everything is bad” debate. Rather than theorize, I decided to give it a go and see how it does. My guess is that once I add a search box it will rarely actually be used, but what do I know?

The problem this presented me with was trying to figure out a way to present the user with one big monstrosity of an “index lookup” page, which would list all of the tags, and for each tag, list links to anything of any content type that was tagged with it. I understand that this could become unwieldy at some point, but if I need to I suppose I can pretty easily paginate, or present alphabet links, or perhaps both!

Though I understood the potential for future disaster, it still bothered me that I couldn’t find a quick answer to the question, so here it is for those wanting to do something similar with django-tagging. For reference, my tag index page is here.

Template Tags in django-tagging

I had actually started creating a custom template tag, and was looking at the Django docs, which stated “a lot of apps supply custom template tags”. Duh! I cd’d into the tagging directory, and there was a directory called “templatetags”. The tagging_tags.py file inside was pretty well documented, and the tag I was about to write myself is called ‘tagged_objects’. Here’s the docstring for that tag:

“””
Retrieves a list of instances of a given model which are tagged with
a given “Tag“ and stores them in a context variable.

Usage::

{% tagged_objects [tag] in [model] as [varname] %}

The model is specified in “[appname].[modelname]“ format.

The tag must be an instance of a “Tag“, not the name of a tag.

Example::

{% tagged_objects comedy_tag in tv.Show as comedies %}

“””

Perfect. I already had a tag_list.html template (which, if memory serves, is one of the things left as an exercise to the user in Practical Django Projects), and it listed the tags in use on the site, but instead of linking off to a ‘tag_detail’ page for each tag, I envisioned something more interesting. I’m not there yet, but this index page is step one.

Putting Together a Template

What I needed to do was simply {% load tagging_tags %}, and then call the {% tagged_objects %} tag with the proper arguments, which consist of a tag *object* (not a tag name), the model you want to grab instances of, and a variable name you want to store the instance in. Here’s the content block from my tag_list.html:

{% block content %}
{% for tag in object_list %}
<div id="entry">
   <p>{{ tag.name }}</p>
   <ul>   {% load tagging_tags %}
      {% tagged_objects tag in  monk.Entry as taggedentries %}
      {% for entry in taggedentries %}
         <li><a href="{{ entry.get_absolute_url }}">{{ entry.title }}</a></li>
      {% endfor %}
      {% tagged_objects tag in ray.Snippet as taggedsnippets %}
      {% for snippet in taggedsnippets %}
         <li><a href="{{ snippet.get_absolute_url }}">{{snippet.title}}</a></li>
      {% endfor %}
   </ul>
</div>
{% endfor %}
{% endblock %}

So, the view I’m using supplies Tag objects in a variable called object_list. For each Tag object, I spit out the name of the tag. Underneath that on the page, for each tag, there’s an unordered list. The list items are the Entries from my “monk” application, and my Snippets from my “ray” application. I hope reading this template along with the bit above from the docstring for the template tag helps someone out. And check out the other tags in tagging_tags.py as well!

Rome Wasn’t Built in a Day

Of course, there’s still an issue with my particular implementation. Tagging was originally implemented specifically for the entries in the “/weblog/” part of my site. However, now that they’ve been applied to things in the “/snippets/” part of my site, this page doesn’t *really* belong in either one. However, if you go to the page, you’ll see that the “Blog” tab in the navigation bar is still highlighted. I’ll figure out what to do with that at some point. Until then, enjoy, and if you have any input or wisdom to share, please leave comments! Also, you should follow me on twitter!


	

How To Get Firefox to Recognize Your Site’s RSS Feed

I just implemented an RSS feed in one of my sites the other day, and realized that visiting my site in Firefox did *not* result in the little blue feed icon showing up in the address bar area.

Turns out it’s not so automagic. You need to do stuff to make it happen. To get firefox to show that blue feed icon, your pages need to have something like this in the source somewhere (this is for my site — edit for your own):

<link rel="alternate" type="application/rss+xml"
    title="LinuxLaboratory Blog" href="http://feeds.feedburner.com/linlab">

You’ll probably want to put this bit in some file that gets included on every page, so no matter what page of your site someone visits, they have an opportunity to subscribe to your feed. I’m using Django for LinuxLaboratory, and I put it into the base template that gets included in all of the pages (well, not every single page, because I use more than one Django app to run my site, but I’ll eventually get it on all pages when I consolidate/organize my templates).

Enjoy!

The Neverending Search for “Free” Wi-Fi

So, I’m a freelancer. I work a lot on remote machines as a system administrator, a troubleshooter of LAMP stacks and web applications, etc. I also do a little bit of web development (but not design. I’m a horrible designer). I work from home a lot. I used to work outside of the home a lot, but what I found is that “free” wireless access has so many downsides that it’s just easier to stay home. I live in the Princeton, NJ area, and have attempted to get free wireless access at Barnes & Noble, Borders, Panera, Starbucks, and a few local businesses. Here’s what I found:

Panera Bread

Yes, the wireless access is free, but it kicks you off for TWO HOURS during the lunch rush. What makes this truly horrible is that there isn’t (as far as I know) an option to *pay* for your wireless access and bypass this limitation. The odd thing is that it seems to backfire on them: if I were able to browse my RSS feeds while I ate a nice Panera lunch, I’d probably stick around. As it stands, if I go there at all, I leave at lunch time and either go home or somewhere else. I’ll eat breakfast there because they don’t turn off wireless at that time.

Turning off wireless is just not acceptable for someone who needs it to be on pretty much all the time. Clearly, Panera isn’t catering to people who are going to hang around there all day, but maybe they should: if they didn’t turn off wifi, I’d spend more than double what I spend there in a given day. I get a coffee and maybe a pastry in the morning, but if wifi stayed on, or I had the option of paying for it, I’d add to that a Frontega Chicken sandwich, maybe a bread bowl of soup in the winter, and at least two lemonades.

But now… I go somewhere else.

Barnes & Noble

Barnes & Noble recently announced that they now have free wifi. The problems with going to BN for this are many. First, going free increases demand for free wifi, which of course increases the demand for power outlets. There are surprisingly few at the location near me. The cafe area in particular hasn’t got even one single power outlet.

But power availability isn’t the worst of it. The worst part is that AT&T runs the wifi access, and as soon as I saw that, I knew something was going to be completely wrong, and I was right: AT&T drops your DHCP lease every 2 hours. EVERY TWO HOURS. There’s no warning dialog either that pops up to say “hey, we’re gonna drop you in 10 minutes”. Things just disappear. Then you have to visit the registration page again and click a checkbox and a button to be reconnected.

Probably ok for a casual email checker, but not for anyone looking to hang out for a while and do “real work”.

Starbucks

Ugh. Forget it. AT&T runs this one as well, and when I asked at my local store how to get on, they asked about my Starbucks card. I have one of those black cards that they call a “Gold Card”. Whatever. The numbers are worn off of it, and I only use it as a discount card — it’s not registered. So it needs to be registered, and then I have to WAIT 48 HOURS, and then I’m entitled to 2 hours free wifi per day. But to register, I have to go through some procedure, and they had to find a way to retrieve the last 4 numbers on my card, because they put the numbers in the area that gets swiped (bright), and they’d rubbed off.

I considered getting one of the new mini cards, which has numbers embedded underneath the plastic, but it was recommended that I stick with only one card or the other. There was seemingly no valid reason for this. I didn’t understand the recommendation, but whatever.

The alternative is to pay for it on the spot, which I might’ve done, but the wifi was down when I tried to connect.

Anyway, this all seems rather messy, doesn’t it? Between my iPhone, Barnes & Noble, and Starbucks, AT&T is making nothing so clear as the fact that they don’t want my money.

Borders

I’m actually writing this post from a Border’s bookstore. The wi-fi here IS NOT free. Know what that means? Well, it means I have to pay for it of course, but it also means there’s almost nobody here. In a cafe area that probably seats 60 or more, at 10:15AM, there are 4 people here, and I’m the only one with a laptop.

Wi-fi here is $8 for a day pass, which isn’t horrifically bad. What *is* pretty bad is that almost all of the chairs here are made of 100% hard wood with no padding of any kind. What is HORRIBLY HORRIBLY bad here is the food. If it’s advertised as edible, DON’T EAT IT. I mean bad. There aren’t English words to describe the badness. It’s No Bueno™. The selection of lunch-worthy food is super small, too. And bad. Did I say the food is bad? It is.

So I pay $8, I get access for 24 hours, and I can leave and walk across the parking lot for lunch, come back, and sign right back in. Not bad. If I had my lap desk with me, I could even sit in one of the well-padded armchairs. I feel a little guilty spending almost no money here, but I’ve *tried* to spend money on food and drinks, and I’ve really just been horribly disappointed. The only thing I’ll ingest here is the coffee. The saving grace for my conscience is that I’m paying for the wi-fi, so I don’t feel the need to spend money on stuff I might not otherwise be interested in.

The Locals Win It

Two local businesses stand out in terms of their free wifi offering. A local person that it turns out I actually know opened up a Camille’s Cafe, and there’s a local coffee shop in Hopewell that I am slowly starting to adore.

Camille’s is closer to my house, but it has, for the entire place, something like two power supplies, and they’re not placed very conveniently. However, the wifi is Really, Truly Free, and that’s good. The food is also good, and you can get healthy stuff there, so I don’t have to buy something deep-fried or made of 85% refined sugar to justify my being there sucking up their wifi.

The local coffee shop is perhaps my favorite place. The wifi is Really, Truly Free, and I would call the power situation “adequate”. The coffee and the food are both really good, and you can also get healthy stuff there. The only problem that exists at this place is parking, but usually I can get around that without too much trouble.

My Django Project Update: RSS Feed, “Home” Link, and more.

In continuing the rebuild of LinuxLaboratory.org using Django, I’m happy to say that things have moved fairly smoothly. I’m using a good mix at this point of stuff from the 2nd edition of “Practical Django Projects”, the Django documentation, blog posts, and docs from other apps I’m making use of.

RSS

I said in one of my previous posts that I’d wait until I burned my feed before giving out the link, and I just did that, so if you want to subscribe to the LinuxLaboratory Blog feed, here’s the link to do that. Right now there’s just one feed for all of the blog entries, but since I post almost all of my really geeky stuff here, the LLO Blog will be mostly site updates like new articles, code, or features being added. The LLO Blog isn’t something that’s intended to get tons of traffic or have tons of posts all the time. The meat of the site will be the content management system which houses articles, and the “Snippets” area which will house scripts and hacks and stuff.

The “Home” Link in Django

I’m not sure why, but it took me a little time to figure out how to link to the base site in a Django template. I had some URL routing set up such that, well… here’s what I have:

In the main project’s urls.py:

(r'^$', include('monk.urls.entries'))

I named my blog app “monk”, after Thelonius Monk. There’s actually a reason I picked his name for a blog app, but it’s not important right now (though, for a chuckle, I picked his last name because his first name breaks a long-standing “8 character” tradition in UNIX).

Anyway, in the corresponding URLConf in monk, I have:

(r'^$', 'archive_index', entry_latest_dict, 'monk_entry_archive_index')

And then in one of my base templates I had this (which is perfectly valid code):

<a href="{% url 'monk_entry_archive_index'}">LinuxLaboratory.org</a>

The ‘{% url %}’ tag can take a URLConf name as an argument, and it’ll do a reverse lookup to get the URL, which is nice, except that this would always land people at “http://linuxlaboratory.org/weblog”, and I wanted them to just go to the base URL for the site. The canonical home page. The root URL. Whatever you want to call it.

There are multiple ways to link back to the base domain from within a template, but I’m not sure if there’s a canonical, “Django-sanctioned” method. You can just make an href pointing to “/”, you can hard-code the whole URL, and I found that doing “href={{ settings.SITE_ID }}” also worked just fine. I tried that last one after discovering that the base URL for the site isn’t in settings.py, and reading that SITE_ID was used by some applications to help them figure out their own URL routing. SITE_ID is a numeric value that represents, according to the Django docs, “the current site in the django_site database table”.

That’s a little confusing, but if you just have a look at the table, it starts to become clear how this could work:

mysql> select * from django_site; 
+----+---------------------+---------------------+
| id | domain              | name                |
+----+---------------------+---------------------+
|  1 | linuxlaboratory.org | linuxlaboratory.org | 
+----+---------------------+---------------------+

It seems logical that using {{ settings.SITE_ID }} in a template could cause the right things to happen, but I haven’t gone diving into the source code I’d need to to prove that it does.

What’s the canonical way of doing this?

Up Next…

So, I have what I think will be a decent setup for code sharing (complete with highlighted syntax), a solid foundation for a blog app, and I’m working on the content management system. I’m using TinyMCE in the admin interface to edit blog posts as well as the CMS content. I’ve got very very basic CSS in place. The basics are here. Now what?

Well, first I need to get my ducks in a row. This includes:

  • Stabilizing a proper development and deployment workflow. There’s a rather nice setup over here, and the 2nd edition of Practical Django Projects has also been enlightening in this regard.
  • Cleaning up my templates. I created the Blog app first, and so of course now I have like 3 separate apps, and I don’t really want them all to have a different look and feel, so I need to abstract some bits and perhaps create a “/templates/common/” directory that will be referenced by all of the apps in the project…? How do you do it?
  • I’d like to get some high-level navigation horizontally across the top instead of having those links in the sidebar. I don’t want fancy popout menus — just a very simple bar where the nav links basically just represent the functions of the different apps: Blog, Articles (CMS), Code, and maybe an About link or something.
  • And more!!

After that stuff is out of the way I’ll start thinking about some new features:

  • I’d like to be able to include images (maybe multiple images) in various types of content. In fact, perhaps all kinds of content. Screenshots for the code snippets, stock images for CMS and blog content, etc. I’m a little intimidated by this because I know my web host (webfaction) limits the amount of memory I can use at any given time. I guess I can just manually scp images to my static media location and link them into the content, but it doesn’t seem ideal. Ideally I can upload them in the same interface where I edit the content, and maybe have the img src tag associated with the instance of the model I’m editing in the database. Does something do this already?
  • I’d really like to have a wordpress-style “Stats” page. Actually, I’d like to have a much better stats page than the wordpress one, but that’d be a start. The stats page I’d actually *like* to have is best described by Marty Allchin here (2 years ago. Anyone know of an app that is aiming for anything close to that?)
  • RSS feeds for the code snippets section (this should be simple)
  • On-the-fly PDF generation for downloading the PDF version of… whatever. A code snippet, an article… Haven’t even investigated this yet.
  • I really, really, really, really want to turn my geek conference calendar into a google maps mashup using GeoDjango. This is another bit I’m slightly intimidated by, because I didn’t realize GeoDjango was built-in these days, and I started out down this road using MySQL as my database for whatever reason (instead of PostgreSQL, which I actually like better and, as it turns out, has way more mature GIS functionality).
  • At some point, I’d like to ensure that the old URLs to content that was on the old site will actually still work, and land people on the same content in the new site. The URL layout isn’t actually horrifically different, so hopefully I can get that in place without *too* much fuss. I know Django has a redirection app built-in, but I’m not sure if this is the right way to do it, or if I should just use Apache rewrite rules. Anyone compared the two?

So those are the big goals. Some are simple, others less so, but I hope to complete all of this within the next… well, before the new baby is born, which is going to be some time in the first half of September. Wish me luck (on both), and please share your tips on how I might accomplish any of the above goals (I’ve heard all the tips I can handle about kids, thanks).

Django, Pygments, Templates, Code Sharing, and Design

Welcome to the latest update!

I spent a total of about an hour on the “Social Code Sharing” application in the 2nd edition of “Practical Django Projects”, and I’m not completely finished with it, but I’ve got syntax highlighting and the basics covered. I can add, edit, and delete snippets, list snippets, and show a particular snippet, complete with syntax highlighting. In an hour. That’s pretty ok, considering I spent a lot of that time debugging what I would call, at best, an unexplained inconsistency in the book that caused me to run into a series of errors before finally figuring out the right way to do it (it’ll all be in the book review – stay tuned for that).

I don’t have links or user contribution set up yet. I only have a quick example up right now, which you can see here, which brings me to the design question: I want my navigation sidebar on the right, but I really don’t want the code to overlap it in any way. I’m kinda wondering if there’s an elegant way to fix that. I considered moving the sidebar to the left, but didn’t want to do that. I considered just making the main content area bigger, but then I’m just guessing, and praying that it fixes the problem for all future code snippets. What’s there now is just a CSS-based solution that creates a scrollable area for the code if it’s too long. Maybe this will suffice (though I have some CSSing to do to refine what’s currently there — the entire content area scrolls right now!)

Since there’s no example template for dealing with code highlighting in the book, and the downloadable code only goes through the first app in the book (the CMS app), here’s the template I’m using for the snippet_detail.hml file:

{% block title %}{{block.super}} | Random hacks{% endblock %}
{% block extrahead %}
<link rel="stylesheet" type="text/css" href="/static_media/llo_main/css/pygments.css" />
{% endblock %}
{% block content %}

 <ul>
 <li>{{ object.title }}</li>
 <li>Published: {{ object.pub_date }} Updated: {{ object.updated_date }}</li>
 <li>Language: {{ object.language }}</li>
 <li>Author: {{ object.author }}</li>
{% load markup %}
 <li>Description: {{ object.description_html|markdown }}</li>
 </ul>

<p>
{{ object.highlighted_code|markdown }}
{% endblock %}

Your CSS path will be different, in all likelihood, and there are some touches missing, like use of a variable to make the title bar dynamic and stuff, but right now I’m shooting for function — the form will come later.

For the snippet_list.html file, I found that what’s in the book just doesn’t work at all. It uses pagination using Django’s built-in pagination capabilities, but the small subset of a template that’s provided references a {{ page }} variable, which does nothing. The Paginator class appears to make individual pages available to templates in a variable you’d reference as {{ page_obj }}, not {{ page }}. Referencing {{ page_obj }} by itself (without referencing any attribute of it) results in output that says something like “<Page 1 of 1>”.

It’s not necessarily pretty to have those “< >” around the edges of the output, and it’s not necessary to use this method of getting the page number and the page count. The page number is available as an attribute of the page class, and the total number of pages is available from the associated paginator object’s ‘num_pages’ method, which you can grab via {{ page_obj.paginator.num_pages }}. You can see my snippets_list.html output here, and here’s my template (at time of writing – note that my site is a work in progress and can change without warning):

{% block content %}

<p>Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}</p>
<p>
{% if page_obj.has_previous %}
<a href="?page={{page_obj.previous_page_number}}">Previous page</a>
{% endif %}
{% if page_obj.has_next_page %}
<a href="?page={{ page_obj.next_page_number }}">Next page</a>
{% endif %}</p>

{% for snippet in object_list %}
 <ul>
 <li><a href="{{snippet.get_absolute_url}}">Title: {{ snippet.title }}</a></li>
 <li>Published: {{ snippet.pub_date }} Updated: {{ snippet.updated_date }}</li>
 <li>Language: {{ snippet.language }}</li>
 <li>Author: {{ snippet.author }}</li>
 <li>Description: {{ snippet.description_html }}</li>
 </ul>
{% endfor %}

{% endblock %}

In the interest of full disclosure, I should also say that, at time of writing, I haven’t added enough snippets to fully test the previous and next page links. I’ll get there, I’m sure — especially when I get user contributions in place, which will happen when I feel a little more confident that this is going to happen in a secure manner. Having been to PHP land, it’s a little unnatural for me to trust the code that comes with a framework to do things properly, which my sysadmin brain always parses as “securely, and in a functionally correct manner”.

Please share tips in the comments! That’s the update for now — stay tuned for more as the recreation of LinuxLaboratory.org continues to unfold.