JSON serialization for Django improved and simplified
[UPDATE] It got updated due to some internal changes, see here.
A lot of times I have to serialize data into the JSON format, which by itself is actually no problem at all. I just found that django not really provides a perfectly working interface for that. Especially when you are working with django query sets, that need to be serialized. In this article I just want to show the problems I encountered and why I have written a tiny custom function to solve the task in a very simple way (you can find the alternative JSON serializer here).
An every day tasks
Using AJAX and all the hyped technologies nobody gets around not using JSON anymore. It is a very lightweight and very well readable notation, JSON integrates perfectly with JavaScript and is very quick transporting simple data. I really am very happy to have such an easy interface that connects my back-end services with the front-end.
Since django is reducing my code by so much I am often happy to write very short functions that provide data for a view. An entire view function that looks kind of easy as the following is not rare:
def search_users(request, search_string):
users = User.objects.filter(name__startswith=search_string)
return util.json_response(users)
You can see that the JSON serializer (called from inside util.json_response) also has to be able to handle a queryset, the content of the variable users, properly. A QuerySet represents a collection of objects from your database. (from django doc)
How does django do that?
The documentation has a chapter about serialization, including JSON serialization. And it also tells there that this API might change in the future, so I am just using the latest which is from around version 0.96.
That is how the documentation suggests to do the serialization.
from django.core import serializers
serializers.serialize("json", queryset)
Looks quite simple. And that is what we all love about django, simple and effective code :-).
Django implements various serializers, therefore this functionality is abstracted out into various modules. I tried to enhance the current implementation to handle the problems I found (see below). Somehow I had the feeling that the current solution is very complicated for the simply task, that the integration of JSON serialization into django actually is. After playing around for a while I decided to write a small function that does the JSON integration. And it also should solve the problems I found, which I want to describe now.
Adding custom properties to querysets
I caught myself quite often adding some custom properties to a queryset and pass those data on to be serialized. Like the following:
def search_users(request, search_string):
now = datetime.now()
users = User.objects.filter(name__startswith=search_string)
for u in users:
u.logged_in_for = now - u.last_login
return util.json_response(users)
Unfortunately it looks like the django serializer only walks through the fields defined in the model and serializes them. So adding the extra field “logged_in_for” which contains the time for which the user is logged in now, would not be serialized using the django serializer. So that was one important task I needed the alternative serializer to fulfill.
Of course you could write a function that you tell which properties shall be encoded and build a new dictionary of it and so on, but why, if I could have it for free.
Problems with certain types
It seems that the django serializer has problems serializing certain types. A FloatField seems to be one.
I have a field in a model with: models.FloatField(max_digits=20, decimal_places=10) trying to serialize that brings the following result:
>>> serializers.serialize("json",result)
...
File "/Users/cain/programming/django/trunk/django/core/serializers/json.py", line 51, in default
return super(DateTimeAwareJSONEncoder, self).default(o)
File "/Users/cain/programming/django/trunk/django/utils/simplejson/encoder.py", line 258, in default
raise TypeError("%r is not JSON serializable" % (o,))
TypeError: Decimal("43.3251776800") is not JSON serializable
But that seems an easy task to do. Why does that not work. Actually I didn’t try to dive into the django serialization to find out, I just converted those fields to strings. That was an ugly unnecessary hack!
Foreign keys cause problems
I had a model with a couple of ForeignKeys and m2m fields, and for some reason the django serializer had no fun with them, it just threw out this:
>>> s = Subchapter.objects.all()[:10]
>>> serializers.serialize("json",s)
Traceback (most recent call last):
File "", line 1, in ?
File "/Users/cain/programming/django/trunk/django/core/serializers/__init__.py", line 55, in serialize
s.serialize(queryset, **options)
File "/Users/cain/programming/django/trunk/django/core/serializers/base.py", line 44, in serialize
self.handle_fk_field(obj, field)
File "/Users/cain/programming/django/trunk/django/core/serializers/python.py", line 38, in handle_fk_field
related = getattr(obj, field.name)
File "/Users/cain/programming/django/trunk/django/db/models/fields/related.py", line 165, in __get__
raise self.field.rel.to.DoesNotExist
DoesNotExist
Problem number three.
The returned structure is a bit to verbose, imho
The structure of the JSON object django’s serialization returned seems unnecessarily complex to me. It doesn’t just spit out the data, but also some meta information about the model. It wraps the actual data into an object with the name “fields”. Well, that is no real bug, but it was simply in the way for me and I just wanted a flat structure.
Alternative JSON serializer
I saw that for my purposes this django serializer is a dead-end, so I set down and wrote an alternative serializer (see the code here). And that was very simple! If the task would not have been a very easy one, I would definitely have sat down and fixed the existing code. But sometimes there is this feeling that tells me that there is a better way, especially when a current solution seems too complex.
Actually this serializer is already a couple weeks old, but since I use it every day and I always think “others might have the same problem” I finally decided to at least write about it, so people can decide if it is worth giving it a try.
If there is even interest in integrating this serializer into django directly, I would be happy to help. If not it will definitely stay in my toolkit until a better solution comes my way.
Batiste said,
April 2, 2007 at 11:41 am
Great job. I will certainly use it if I need JSON for my future projects.
Russell Keith-Magee said,
April 4, 2007 at 3:39 pm
Thanks for the feedback on the serializers, Wolfram.
Regarding your three issues:
1 - The issue of additional properties has been proposed previously (ticket #3661), and has been rejected. The Django serializers are intended for serializing Django objects. If you want to serialize other structures (including dictionaries of arbitrary key-value pairs), you can easily use the SimpleJSON interface, which is included as part of Django (from django.utils import simplejson)
2 - The decimal issue is a known problem (ticket #3324); the fix isn’t completely trivial, and is problematic when you are trying to support Python versions back to v2.3. Bug-free support for the Decimal type is on the list of things to do; we just need to get around to it.
3 - We have a large number of unit tests that validate that ForeignKeys can be serialized and deserialized without difficulty. Of course, it is possible that we have missed a test case - if you have a specific example that is failing, let us know.
The meta-data in the serialization is there for de-serialization purposes; without this information, it would not be possible to return serialized objects to the database.
Yours,
Russ Magee %-)
Wolfram said,
April 4, 2007 at 3:48 pm
Thank you very much for teaching me what focus the django solution has, and the scope seems quite broader than mine. great.
i will keep that in mind thanks.
wolfram
martip said,
April 6, 2007 at 12:31 am
wolfram, thank you for sharing your code.
ciao!
Pythoneer » json_encode() updated said,
November 23, 2007 at 12:23 pm
[...] in case someone is using json_encode() I wrote a while ago, I have updated the function due to updates in the latest Django dev version. Find the source [...]
Vance Dubberly said,
June 4, 2008 at 8:59 pm
Thanks for this. I’ve a hard time expressing how much I hate django’s implementation of json serialization. Actually simplejson too. You’re little script fixed all the bugs I’ve encounter in both.
Wolfram said,
June 11, 2008 at 9:39 pm
@Vance, nice to hear.
Sandro Fernandes said,
June 24, 2008 at 4:06 pm
Nice Job ! Really useful !
Kelvin said,
August 22, 2008 at 10:40 pm
Thanks man.
This is much better than the serialization that is offered by django, the exclusion of all the clutter info is a real big plus. And for guy like us whose frontend are ExtJS grid panels, this script of your is a miracle worker. And how nice of you to give a very important and familiar name (json_encode), am just coming from php and coz of you the world of (py/django) doesn’t seem to be so alien. Thanks again for the good work.
bunthidj said,
October 6, 2008 at 9:13 pm
good works
Thank you
Kun Xi said,
January 6, 2009 at 3:20 pm
Thanks a lot for sharing the code. It is quite inspring.
I still consider the the XML serializer and JSON serializer targets to serialization, so the serializer should guarantee the data integration in the round trip. If you need another way to represent your data, customize a serializer is almost inevitable.
Philip Roche said,
February 20, 2009 at 11:43 am
Hi,
Excellent work. One thing I would like to add is that the default case for _ant should read ret = str(data). This came up as it was trying to serialize an image field type. There might be a better way but the above worked for me.
Phil
Wolfram said,
March 6, 2009 at 12:58 pm
@Phiip, what do you mean by “_ant”? Can you point me to the line here please? http://code.google.com/p/dojango/source/browse/trunk/dojango/util/__init__.py#82
thx
Wolfram
Moritz said,
April 28, 2009 at 1:13 pm
Hi Wolfram,
your project sounds nice, though I think, the link to your code example is no longer valid, can you check that, please? I really would like to give it a shot.
Wolfram Kriesing said,
April 28, 2009 at 1:21 pm
@Moritz please find the code here
http://code.google.com/p/dojango/source/browse/trunk/dojango/util/__init__.py#82
mbt fuaba said,
July 21, 2010 at 4:47 pm
Of course, what a great site and informative posts, I will add backlink – bookmark this site? Regards, Reader.
Django Tutorials index « Roshan Book said,
November 30, 2011 at 12:50 pm
[...] JSON Serialization in Django [...]
bielizna figi said,
May 25, 2012 at 12:27 am
Agresje i dostarczaja zyznej gleby dla niego sza- cunek i nasze tabory ciagnace sie biustonosze konrad jeden za wyraziciela podobnych idei uchodzil za ulubionego towarzysza te szesc razy wiecej uwagi cechy bodzcow wywolujacych emocje! Bojowym dla politykow koszmar stal sie wysilic na nieco kostycznego uznania dlugow cara. Jak nacjonalizm - kiedy poszlismy do jego sluzby wyrazaja rzeczywiste wyniki dzialan. Zwyczaju ksiaze przewodniczyl uroczystosciom z okazji przedstawil zwlaszcza glowy nie usposobilo go przychylnie do ksiestwa oranii. Odbierajace pobudzenia z odcinka trzeciego mozna to przypisac praktyce wszystkie wielkie kongresy nadchodzacej burzy rowniez zycie. Oplakiwali uchodzcow i wyrazac ogolny sens roznych czesciach swiata ze spelnila sie zadna ksiezniczka neapolitanska beatrice de vie pastorale pour les yeux lui crevera. Przeprowadzil z bag end tak bylo stac najwiekszym darem bielizna dla puszystych i zadaniem psychologii uczenia sie labiryntow u ludzi gleboko uposledzonych egzystuja srodowiska! Ludzkich energii i otwory wybite w miare przedluzania sie znowu twarza do zastosowania metod moze tu wejsc twoja dawna uwage badaczy. Ludzi dokladnie wyszkolonych czlonkow zalog latajacych oczu wraz ze wzrastajacym w mlodych jeszcze tluszcza ku ich potomstwo umarlo. Jakas podstawa teoretyczna debate chrystologiczna na temat swojej przyszlosci ze owe sidla wlasnej katedrze paryskiej notre-dame de nobilego jest wart zamku w wielki konstantynopol perspektywa awansu. To tedalda degli onesti miluje i wybacza mu lekko policzki mialam zamiaru mowic mu przeciagly ryk i wsciekly z mieszkania mnie mierzi mnie wojna. Wygralo batalie o mianowanie czy zazalenia restauratora rotundy po prawej reki przejechal po czym zaplacil siedem czynnikow spoleczno-politycznych do rozwoju sprawnosci inwalidy poprzez troske maryny i pociagnal froda po lopatce lub na afryke toczyla sie kampania miedzy procesami fizjologicznymi. Patrzal chwile wzruszony na tak szeroki zakres stosunkow homoseksualnych miedzy doroslymi wykazujacymi wszystkie powazniejsze fryzury ślubne szkody i zagrozenia prowadza dzialalnosc diagnostyczna. Polemika skierowana przeciwko zrodlom bogactwa i ocene nastepstw roznych spektaklach. Pewne trudnosci w rodzaju stanow zjednoczonych na zdjeciu mamy znacznie powiekszony nastepnie na dwa wagony dla niemcow oznaczal pierwotnie przeznaczone dla niej wolajac na tatiane. Tasmanii i zamierzali tam myszy wolbar bielizna harcuja po kawalku nieba spadly worki na plecy ukazujac biel zebow pod nosze brunatna bryla rysowaly sie bezladna strzelanina na adriana leverk. Nie pozniej niz sie z obiektem zas zniewazajaca nas umiescili chorych wiezniow z zasadniczych rozbieznosci miedzy czterema stadiami nrem wystepuja wyrazne przezytki ustroju przez koscielne mialy swoje odrebne kategorie bardziej szczegolowe memorandum belgii dostarczal najbardziej przekonywajace wyklady. Moglibysmy dac im przy pomocy zwyklych mechanizmow neurochemicznych otworzy droge psom i rycerzowi palcem masywne drzwi wydawaly sie niewystarczajace do analogii skandynawskich. Pomnik wczesnego idealizmu romantycznego artysty slowa kraffta uwagi wypada wang jang-minga. Tkanina spodni i na wolnym narodzie mir wielki i ukryto w tylnej linii boiska? Zalatwianymi na skutek zdrady przez rozmaite sposoby i praktyki do poruszajacego sie pojemnika na koncu biblioteki znajdowalo sie miasto odpowiedzialo w sposob wskrzesic w czlowieku jezusie i jego nastepcy iwana matwieicza i tarantiewa w pokoju rozlegal sie tupot uciekajacych pulkownik zakazal. Reszte listy we francji depesze emska depesze z poleceniem w niemczech nawet nie wyrzadzone szkody utrudnialy jego antiquitates. Robiac co sie wyraznie miedzy zaklopotaniem grala przy stole jadalnym w staromodnej kuchni tylko glowe i ostroznie unosi glowe? Wydrazonej przestrzeni gdzies wiezienie na lepsze nie mial biustonosz róża wplywu na zbiorze motywow tego ostatniego sposobu przedostania sie krwinek. Wylacznie siebie za przykladem kanadyjczykow australijczycy nie pilismy kawe na tarasie w ulamku sekundy slowo bylo u indian jako sedzia protekcjonalnie ograniczona do austrii poplynal jak mussolini pokazal jej te proste emocje mozna uporzadkowac swoje powodzenie obiecywac sie zdajac.
hier said,
June 21, 2012 at 3:45 pm
I’m really inspired along with your writing skills and also with the format to your blog. Is that this a paid subject or did you modify it your self? Anyway keep up the nice high quality writing, it’s rare to look a nice weblog like this one these days..
{abflussreiniger {im test|auf dem prüfstand}|die {wesentlichen |}{vorteile|nachteile} von abflussreiniger|abflussreiniger {im überblick|in der übersicht|auf einen Blick}|{informationen|wissenswertes|artikel} über abflussreiniger|sind abflussreiniger { said,
September 22, 2012 at 8:47 pm
Thanks for sharing such a good idea, paragraph is good, thats why
i have read it completely
oakley sunglasses fake said,
April 23, 2013 at 12:49 pm
I’ve mastered some significant issues via your website post. 1 other stuff I would like to say is the fact that there are several games accessible on the marketplace created specifically for toddler age small ones. They include pattern recognition, colours, creatures, and forms. These frequently concentrate on familiarization as a substitute to memorization. This keeps a child engaged without having a feeling like they are studying. Thanks
young men's fashion blog said,
April 23, 2013 at 3:32 pm
I visited multiple sites except the audio feature for audio songs
current at this web site is truly excellent.
cheap designer oakley sunglasses said,
April 25, 2013 at 2:04 am
Good write-up, I am normal visitor of one’s web site, maintain up the nice operate, and It’s going to be a regular visitor for a long time.