[UPDATE] It got updated due to some internal changes, see here.
A lot of times I have to serialize data into the JSON format, which by itself is actually no problem at all. I just found that django not really provides a perfectly working interface for that. Especially when you are working with django query sets, that need to be serialized. In this article I just want to show the problems I encountered and why I have written a tiny custom function to solve the task in a very simple way (you can find the alternative JSON serializer here).
An every day tasks
Using AJAX and all the hyped technologies nobody gets around not using JSON anymore. It is a very lightweight and very well readable notation, JSON integrates perfectly with JavaScript and is very quick transporting simple data. I really am very happy to have such an easy interface that connects my back-end services with the front-end.
Since django is reducing my code by so much I am often happy to write very short functions that provide data for a view. An entire view function that looks kind of easy as the following is not rare:
def search_users(request, search_string):
users = User.objects.filter(name__startswith=search_string)
return util.json_response(users)
You can see that the JSON serializer (called from inside util.json_response) also has to be able to handle a queryset, the content of the variable users, properly. A QuerySet represents a collection of objects from your database. (from django doc)
How does django do that?
The documentation has a chapter about serialization, including JSON serialization. And it also tells there that this API might change in the future, so I am just using the latest which is from around version 0.96.
That is how the documentation suggests to do the serialization.
from django.core import serializers
serializers.serialize("json", queryset)
Looks quite simple. And that is what we all love about django, simple and effective code :-).
Django implements various serializers, therefore this functionality is abstracted out into various modules. I tried to enhance the current implementation to handle the problems I found (see below). Somehow I had the feeling that the current solution is very complicated for the simply task, that the integration of JSON serialization into django actually is. After playing around for a while I decided to write a small function that does the JSON integration. And it also should solve the problems I found, which I want to describe now.
Adding custom properties to querysets
I caught myself quite often adding some custom properties to a queryset and pass those data on to be serialized. Like the following:
def search_users(request, search_string):
now = datetime.now()
users = User.objects.filter(name__startswith=search_string)
for u in users:
u.logged_in_for = now - u.last_login
return util.json_response(users)
Unfortunately it looks like the django serializer only walks through the fields defined in the model and serializes them. So adding the extra field “logged_in_for” which contains the time for which the user is logged in now, would not be serialized using the django serializer. So that was one important task I needed the alternative serializer to fulfill.
Of course you could write a function that you tell which properties shall be encoded and build a new dictionary of it and so on, but why, if I could have it for free.
Problems with certain types
It seems that the django serializer has problems serializing certain types. A FloatField seems to be one.
I have a field in a model with: models.FloatField(max_digits=20, decimal_places=10) trying to serialize that brings the following result:
>>> serializers.serialize("json",result)
...
File "/Users/cain/programming/django/trunk/django/core/serializers/json.py", line 51, in default
return super(DateTimeAwareJSONEncoder, self).default(o)
File "/Users/cain/programming/django/trunk/django/utils/simplejson/encoder.py", line 258, in default
raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("43.3251776800") is not JSON serializable
But that seems an easy task to do. Why does that not work. Actually I didn’t try to dive into the django serialization to find out, I just converted those fields to strings. That was an ugly unnecessary hack!
Foreign keys cause problems
I had a model with a couple of ForeignKeys and m2m fields, and for some reason the django serializer had no fun with them, it just threw out this:
>>> s = Subchapter.objects.all()[:10]
>>> serializers.serialize("json",s)
Traceback (most recent call last):
File "", line 1, in ?
File "/Users/cain/programming/django/trunk/django/core/serializers/__init__.py", line 55, in serialize
s.serialize(queryset, **options)
File "/Users/cain/programming/django/trunk/django/core/serializers/base.py", line 44, in serialize
self.handle_fk_field(obj, field)
File "/Users/cain/programming/django/trunk/django/core/serializers/python.py", line 38, in handle_fk_field
related = getattr(obj, field.name)
File "/Users/cain/programming/django/trunk/django/db/models/fields/related.py", line 165, in __get__
raise self.field.rel.to.DoesNotExist
DoesNotExist
Problem number three.
The returned structure is a bit to verbose, imho
The structure of the JSON object django’s serialization returned seems unnecessarily complex to me. It doesn’t just spit out the data, but also some meta information about the model. It wraps the actual data into an object with the name “fields”. Well, that is no real bug, but it was simply in the way for me and I just wanted a flat structure.
Alternative JSON serializer
I saw that for my purposes this django serializer is a dead-end, so I set down and wrote an alternative serializer (see the code here). And that was very simple! If the task would not have been a very easy one, I would definitely have sat down and fixed the existing code. But sometimes there is this feeling that tells me that there is a better way, especially when a current solution seems too complex.
Actually this serializer is already a couple weeks old, but since I use it every day and I always think “others might have the same problem” I finally decided to at least write about it, so people can decide if it is worth giving it a try.
If there is even interest in integrating this serializer into django directly, I would be happy to help. If not it will definitely stay in my toolkit until a better solution comes my way.
cain@home:~> ./manage runserver