Pythoneer

I was fighting four days now, with a threading problem, which are known to be hard to track. But I finally found it and learned that I actually had made a beginner’s mistake.

What happened?
From the front end I trigger via AJAX a view that again starts a thread that does some import work, that might take quite a while. This enables the user to keep going and have the import run without interrupting him/her. Every once in a while an asynchronous call checks on the state of the import.
And here lies the problem: while the thread is running and busy like a bee adding data in the DB the asynchronous call to check on the state also tries to run a query and that causes the following exception:

...
 File "/Users/cain/programming/django/trunk/django/db/backends/mysql/base.py",
line 42, in execute
   return self.cursor.execute(sql, params)
 File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/MySQLdb/cursors.py",
line 137, in execute
   self.errorhandler(self, exc, value)
 File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/MySQLdb/connections.py",
line 33, in defaulterrorhandler
   raise errorclass, errorvalue
ReferenceError: weakly-referenced object no longer exists

That really made me mad, as you can imagine. The problem was that I didn’t really know where to start debugging and seraching. Well, debugging was hard anyway, since it was multi threaded and I didn’t yet spend the time to get my WingIDE to run Django so I could debug it, but that’s another problem.

The problem and it’s solution
So after trying and a lot of thinking I found out that it had something to do with the connection. Finally I found out that the connection object was shared between the threads in the MySQLdb/cursors.py, so it seemed not to be thread-safe. The problem only occured when args was passed to the execute() method, so I dove deeper this way. And found out that the connection.literal() call was actually causing the problem. My guess now is that the connection got reseted by the main thread and was not available anymore, so that the exception above was thrown.
I found it. But too late, at this moment I thought I should upgrade MySQLdb, and I did. (I am working with the Django trunk, so I could not be more up to date on this front.) And after this upgrade the problem was solved. How could I forget to try and upgrade first? I got an excuse, I thought the problem was in Django, not in MySQLdb.
Now I only needed to see if the problem I had found was also the one that got fixed. Since I had learned some stuff about weakrefs and threading, etc. these days I more or less knew what to look for. And there it was.

from weakref import proxy
self.connection = proxy(connection)

That part in the __init__() method of BaseCursor was the bit that obviously solved my problem. I didn’t verify it, but I am pretty sure.

The upgrade
When it occured to me that I should upgrade MySQLdb, I checked my current version number of course.

>>> import MySQLdb
>>> MySQLdb.version_info
(1, 2, 0, 'final', 1)

Note: That is the version that has the threading problem!!!

So I upgraded to MySQLdb 1.2.1_p2 the version that fixes the problem above.

But I didn’t say it was easy to “just” upgrade:

>>> import MySQLdb
Traceback (most recent call last):
  File "", line 1, in ?
  File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/MySQLdb/__init__.py", line 34, in ?
    from sets import ImmutableSet
ImportError: cannot import name ImmutableSet

Grrrr …
But the problem and it’s solution were found quickly, even on a Django site http://code.djangoproject.com/wiki/InstallationPitfalls.
After removing the sets.py(c) it worked just fine.

>>> import MySQLdb
>>> MySQLdb.version_info
(1, 2, 1, 'final', 2)

Good luck threading away …

Decorators is another one of those features that is easy to use and also easy to implement (once you know how). I proved to myself again, that it is very much easier in Python than one can imagine and that I am thinking too difficult again.
The must read article about decorators, that explains it all in depth and the different use cases is the article Python 2.4 Decorators by Phillip Eby.
Finally I am sure not only about how to use them :-).

I really had thought I knew a lot of the programming terminology, methodolgy and stuff. That was about two years ago, when I had reached the point that PHP, the language I had used the last years and knew very well, had become boring and caused those kind of feelings. When starting with Python back then I saw that I had taken the blue pill before, it’s time to take the red one.

Currying was one of those things that crossed my way and I first thought it was some dirty hack thingy. But a lot more of those things crossed my way and I was glad to be back in the space of things I didn’t know yet but things that seemed everyone knew, but me. Finally a lot of challenges. And then I came across curry in Dojo too. Now my interest was finally raised and I had to learn in depth what currying is.

There is a Wikipedia article about Currying, but I feel more attracted by the article by Svend Tofte Curried JavaScript functions. And I guess there are a lot more.

So let me read and explore the real world.

My article Django’s template syntax, enhanced ifchanged contained a couple of errors in my thinking.
Read the rest of this entry »

This article generally discusses Django’s template syntax and in detail offers a patch for the existing tag ifchanged. The patch allows to pass a parameter to ifchanged, which is then evaluated and checked if it has changed, instead of the tag’s content. If it has changed the content is being printed, as usual. I.e. you can do
{% ifchanged variable %}print if variable's value has changed{% ifchanged %}
You can download the patch here, the Django-ticket is here. To know more about the details, read on.
Read the rest of this entry »

Quite a long list of all kinds of testing tools for Python, and not only for unit testing. That looks very interesting, I will need to spend some time on that.
Found via jotsite.com.

How often was I hitting TAB to get completion for so many things. When I discovered in the Django docs that it has bash completion for all the Django managment commands I just had to activate them right that seconds. Great! Thats another little step to a better working environment.

* Type django-admin.py.
* Press [TAB] to see all available options.
* Type sql, then [TAB], to see all available options whose names start with sql.

You can find the file you need to install inside “extras/django_bash_completion”. It contains the following instructions how to get it working in your bash:

# Installing
# ==========
#
# To install this, point to this file from your .bash_profile, like so:
#
# . ~/path/to/django_bash_completion
#
# Do the same in your .bashrc if .bashrc doesn’t invoke .bash_profile.
#
# Settings will take effect the next time you log in.

Lately I am doing a lot of Python again. Especially Django and I have to say I am starting to be productive with it. I was first fighting with the support of DB-views and other features MySQL 5 has finally bundled, but a couple fixes got me around. More about that in a later article.

While writing a couple of tests (btw still using testOOB) I needed to do some date and time calculation. I just wanted to update a timestamp in the DB and compare the delta. While I was thinking about how to do it, it popped to my mind, that I am probably thinking too complicated. So let’s try the obvious one.

>>> from datetime import datetime
>>> now = datetime.now()
>>> now - 1
Traceback (most recent call last):
File "< stdin>“, line 1, in ?
TypeError: unsupported operand type(s) for -: ‘datetime.datetime’ and ‘int’

Read the rest of this entry »

After working with Python for a while and enjoying the pleasure of having a shell where you can test your code before you actually programm it I came back to PHP and missed the shell big time. Especially when you don’t exactly know how to use special objects or functions. Now the pain of not having such a shell is over, Jan wrote the real shell for PHP. Thanks a lot. This will make testing and just playing with PHP so much easier. It provides of course auto loading of classes.

Install
Download the actual PHP_Shell class here, and rename the file to “.php”. Download the php-shell script, that you want to start later, from the command line here, rename it to “*.php” too.
To make it callable from the command line using “php-shell” you might want remove the “.php” from the later file too and put either both in one directory that is globally reachable (that is in your PATH). The other opportunity, when you don’t want the PHP_Shell file in you bin-directory, would be to put the PHP_Shell class in the include_path, but then you have to adjust the php-shell script to not include the PHP_Shell from “./PHP_Shell”, but as “PHP_Shell”.

Have fun …

This page is devoted to various tips and tricks that help improve the performance of your Python programs. Note: You should always test these tips with your application and the version of Python you intend to use and not just blindly accept that one method is faster than another.

read more | digg story

My PHP5 and PDO (PHP Data Objects) experience is very young (to be exact: one day), but I have already learned an important lesson, which is rather a mysql lesson, than PDO, I guess.

PDO default is unbuffered query
I had the simple code:

$dbh = new PDO(’mysql:host=localhost;dbname=test’, ‘root’, ”);
$dbh->prepare(’SELECT * FROM test WHERE test_id=?’);
$dbh->execute(array(1));
$dbh->fetchColumn(0);
$dbh->prepare(’SELECT * FROM test1 WHERE test1_id=?’);
$dbh->execute(array(2));

It always gave me a “Call to a member function execute() on a non-object” in the line of the last execute(). I was really speechless. I even tried to create a new PDO object every time, that seg-faulted on me. It wasn’t the right thing to do anyway, so that was ok.
With Johann’s help we found out that the unbuffered query is the problem. I didn’t retreive the complete result set (I only do a fetchColumn()), so there is still an open result set on the server. And a new execute() is awry. Calling a fetchAll() after the fetchColumn() helps, but is not what you want to do. The solution is to use buffered queries, create the new PDO object with the attribute PDO::MYSQL_ATTR_USE_BUFFERED_QUERY set to true, like so:

$dbh = new PDO(’mysql:host=localhost;dbname=test’, ‘root’, ”
,array(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => true));

Ilia already mentioned this problem last year. Anyway it still seems to be a point that needs to be stated clearer in the docs. I am happy I learned something. That’s not bad for one day PHP5 and PDO.

Why not Python?
If you followed this blog and are wondering how come that I am doing PHP again, well it is the simplest to get running. I would love to do it in Python, but getting set and started simply takes much more time, and I just want to get a prototype out the door. I am also much more familiar using PHP and my latest experience with the EXIF stuff for Python doesn’t make me very optimistic, that Python is easy enough to use for all the web tasks I expect to do.
I am sure that if I did about a month of this stuff in Python I will have understood it all too, but I don’t have this time now, unfortunately. But at least I have the plan to redo it all in Python (btw. I have many plans).

Actually I thought to do it in PHP, but for security issues and also for improving my Python skills I decided to do it in Python. The first problem already was finding a module that can read out the EXIF data of images. After trying pexif I was a bit disappointed, it just couldn’t read my pic’s data. There were some tags it couldn’t handle, so it just stopped with an exception. Then I was going to try jpeg a simple module, there I read in the docs that for doing this and that you should use http://pyexif.sourceforge.net - ok, now I know what I need. Actually it was clear that the module would be named pyexif, but for some reason I was not searching for it. So I downloaded it and thanks to sourceforge this didn’t take less than about half an hour. After unpacking and skimming through the files I was astonished that there was no setup.py. Ok, back to their project page and read a bit again. Oh great, the last comment in big red letters (from 2002) said that they hadn’t done anything on the library for a long time, f***. And I should go and check out “Gene Cash’s EXIF.py library”, which I finally found here. To judge by the comments this one has also been last updated in 2004. Oh man, I think I am going back to PHP, there will surely be up-to-date libraries. But I will give it just another chance, let’s see if there are going to be any interesting hints and/or comments to this entry, hopefully about new modules I didn’t find. Disappointed …

I am so glad that Guido van Rossum is asking:
Please Teach me Web Frameworks for Python! So I only need to follow the discussions and get all the insight I need, I hope. Ok, one question might still be open: Which JavaScript Framework shall I use?
Some of the web frameworks integrate basic AJAX libraries, but not really everything that is needed if you really want to dive in. And then the question will come up if the JS libs are compatible or if they can be made compatible. I had heard that prototype does override some standard prototypes, which bothers other libs. I will keep you updated once I have decided and why!

I really loved to learn the Python shell, for those who don’t know what it is, here it is:
I just had a simple issue, I was looking for the path where testoob is installed. Oops, I have a local copy of it in my projects folder. So lets remove the project project-path from the sys.path (which is the default import path Python uses) and reload() the module. Way cool.

It’s just sooo handy trying things out on the Python shell, if you didn’t use it yet, start doing that!

Ori just released a new verison of TestOOB, congrats. I had actually planned to contribute another tiny feature, but didn’t get around to implement it yet, but I will get into gears now.

Great tool …

A little while ago I was discussing with some friends why dynamic languages (I mean those that allow for rapid prototyping, such as Python) are not used more often on embedded devices. And one argument was that those languages need dynamic memory management, which sounds reasonable. So it would take quite something to run Python on some embedded device and build all the necessary memory management etc. around it. But hardware is getting cheaper and even faster, Flash RAM could replace a hard disk and you could be up an running in a low budget area. Just throw Damn Small Linux on it and that’s it.

Ok, there are some embedded devices that are just too small or have too time critical applications where you can’t do that. But DSL just refreshes this question in my head, since I am currently working with embedded devices again. I think the good old C will stick around, of course, but dynamic languages will more and more spread where C is still used. The prices just have to drop and the devices just need to get smaller. Or not?

Just imagine which impact that would have on the development cycles, the stability and testability of this kind of software. When you are programming you actually don’t want to manage the memory, you want to solve a problem. And this fact needs to be more focused on when programming. And languages like C just don’t allow that.

There is hope …

Update:
Bertrand just commented about Nokia putting Python on their Series 60.

Actually I am not that deep in yet, I just started to look at PyObjC in order to write a QuickSilver (btw a must have) plugin and so I got a bit deeper into Cocoa GUI building using the Interface builder. And I have to say I am impressed. I never made a GUI that easily. I would not have written this here when Jannis had not reported about this Swing-animation. Watch it, it really is fun and btw. try the PyObjC tutorial - it’s cool!

It is a really old article from Eric Raymond, but still soooo true. Especially for my friend Daniel, who still swears on C. Get over it!
It feels really good when there is someone who can write out and put things in words, what I only have as a feeling. Great article …

… accepting the debugging overhead of buffer overruns, pointer-aliasing problems, malloc/free memory leaks and all the other associated ills is just crazy on today’s machines. Far better to trade a few cycles and a few kilobytes of memory for the overhead of a scripting language’s memory manager and economize on far more valuable human time.

and this one:

Python’s use of whitespace stopped feeling unnatural after about twenty minutes. I just indented code, pretty much as I would have done in a C program anyway, and it worked.

for i in range(len(my_list)):
   if some_condition:
       del my_list[i]

the range(len()) doesn’t get calculated again, so i get an IndexError for sure, the closer it gets to the end. Because in the beginning there might were 5 elements in my_list but when I delete one and at the end I want to access my_list[4] I get the IndexError, because that’s now my_list[3], or course. Looking at the code didn’t make it obvious, because “I knew it was correct” and “it is just too simple to contain an error”. I am not human, I am a programmer who knows!

That’s why I should be debugging again, I saw that right away, just put the watch on the my_list and the first del made me see the problem. That’s why I should get used to debug right away!

That made me think about my supposedly development in developing.
In 1986 or 1987 I started with Basic, I don’t think I was debugging that. There was not even a debugger for it, or was there? For my KC87 and KC 85/3 there was surely no debugger. Then, I don’t remember the year, I switched to Turbo Pascal (cooles pic). That had debugging of course. And that was just easy as changing dipers (is for me now)! F4, F5, F6 and all those key combinations for Step Over, Step Into, etc. Then in 1990 *thinking* … yes, when I was in 11th grade. I started to use C, that was strange. All those curly braces and this strange i++ construct. Hoooh, that scared me at first. Hey man I was programming graphic stuff in Pascal, I even knew how to use the mouse and all that, and now you come to me with curly braces and that stuff. But a debugger was just standard at this time, of course! Later I was doing a lot with Delphi, Assembler, C++ and some others I guess. And I was always debugging my code. Well I remember that I didn’t when I was programming multi-threaded stuff in Delphi for my diploma, of course.

So what the heck was I doing when I accepted to stop debugging when I started programming PHP (actually already with Perl, that I was doing little before)? I think it was the fascinating world of the web, that has opened to me and made me forget about the comforts of debugging and other programming tools, that I had used until then.

But fortunately there are also debuggers for PHP now and thanks to the WingIDE there is a great debugger for Python, with all the “step into”, “step over” and watch list things, even the same key combinations as I remember them from Pascal. Great. So I will start to creep out of my black whole and debug again.

I finally found some challenge again. I am learning again, I am reading source code that really is nicely written and well structured. I have the feeling that got somehow lost in my PHP days. The TestOOB source makes use of various design patterns, where it makes sense and is thread-safe. Somehow it seems that the language Python attracts the people that really understand those concepts and the level of programming is much higher than in most PHP projects. That is a challenge and it will take me a while to make it up to understand it all and to feel comfortable at this level, I am very excited about the upcoming time. That is the salt in the soup of a programmers life. Yeah!
Read the rest of this entry »