Multithreading with MySQLdb and weakrefs
I was fighting four days now, with a threading problem, which are known to be hard to track. But I finally found it and learned that I actually had made a beginner’s mistake.
What happened?
From the front end I trigger via AJAX a view that again starts a thread that does some import work, that might take quite a while. This enables the user to keep going and have the import run without interrupting him/her. Every once in a while an asynchronous call checks on the state of the import.
And here lies the problem: while the thread is running and busy like a bee adding data in the DB the asynchronous call to check on the state also tries to run a query and that causes the following exception:
...
File "/Users/cain/programming/django/trunk/django/db/backends/mysql/base.py",
line 42, in execute
return self.cursor.execute(sql, params)
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/MySQLdb/cursors.py",
line 137, in execute
self.errorhandler(self, exc, value)
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/MySQLdb/connections.py",
line 33, in defaulterrorhandler
raise errorclass, errorvalue
ReferenceError: weakly-referenced object no longer exists
That really made me mad, as you can imagine. The problem was that I didn’t really know where to start debugging and seraching. Well, debugging was hard anyway, since it was multi threaded and I didn’t yet spend the time to get my WingIDE to run Django so I could debug it, but that’s another problem.
The problem and it’s solution
So after trying and a lot of thinking I found out that it had something to do with the connection. Finally I found out that the connection object was shared between the threads in the MySQLdb/cursors.py, so it seemed not to be thread-safe. The problem only occured when args was passed to the execute() method, so I dove deeper this way. And found out that the connection.literal() call was actually causing the problem. My guess now is that the connection got reseted by the main thread and was not available anymore, so that the exception above was thrown.
I found it. But too late, at this moment I thought I should upgrade MySQLdb, and I did. (I am working with the Django trunk, so I could not be more up to date on this front.) And after this upgrade the problem was solved. How could I forget to try and upgrade first? I got an excuse, I thought the problem was in Django, not in MySQLdb.
Now I only needed to see if the problem I had found was also the one that got fixed. Since I had learned some stuff about weakrefs and threading, etc. these days I more or less knew what to look for. And there it was.
from weakref import proxy
self.connection = proxy(connection)
That part in the __init__() method of BaseCursor was the bit that obviously solved my problem. I didn’t verify it, but I am pretty sure.
The upgrade
When it occured to me that I should upgrade MySQLdb, I checked my current version number of course.
>>> import MySQLdb
>>> MySQLdb.version_info
(1, 2, 0, 'final', 1)
Note: That is the version that has the threading problem!!!
So I upgraded to MySQLdb 1.2.1_p2 the version that fixes the problem above.
But I didn’t say it was easy to “just” upgrade:
>>> import MySQLdb
Traceback (most recent call last):
File "", line 1, in ?
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/MySQLdb/__init__.py", line 34, in ?
from sets import ImmutableSet
ImportError: cannot import name ImmutableSet
Grrrr …
But the problem and it’s solution were found quickly, even on a Django site 
http://code.djangoproject.com/wiki/InstallationPitfalls.
After removing the sets.py(c) it worked just fine.
>>> import MySQLdb
>>> MySQLdb.version_info
(1, 2, 1, 'final', 2)
Good luck threading away …
Rene Dudfield said,
December 5, 2006 at 5:11 am
Yah for more bugs being fixed in infrastructure code. The more people doing website work - no matter with which frame work they use. More usage, means more real world testing.
Slowly python people can be more safe with the libraries they rely on.
Leworan said,
May 20, 2011 at 6:01 am
They need to recognize if you have any of these conditions:
* eye or illusion problems, including a rare inherited purpose bug called retinitis pigmentosa
* pity virus, angina, far up or improper blood pressure, a curriculum vitae of heart fight, or other heart problems
* kidney malady
* liver illness
* stroke
* an peculiar or allergic reprisal to sildenafil, other medicines, foods, dyes, or preservatives
buy cialis online now
Eloy Jaremka said,
December 28, 2011 at 10:36 am
Pretty section of content.I just stumbled upon your blog and in accession capital to assert that I get in fact enjoyed account your blog posts.Any way I’ll be subscribing to your feeds and even I achievement you access consistently fast
Sammy Colborn said,
January 5, 2012 at 9:31 am
If you’re still on the fence: grab your favorite earphones, head down to a Best Buy and ask to plug them into a Zune then an iPod and see which one sounds better to you, and which interface makes you smile more. Then you’ll know which is right for you.
hermes handbags outlet said,
January 18, 2012 at 5:09 am
Just discovered this site through Yahoo, just what a method to perk up my month!
carving watermelons baby carriage said,
February 10, 2012 at 10:49 am
I’m not that much of a internet reader to be honest but your blogs really nice, keep it up! Ill go ahead and bookmark your website to come back in the future. Cheers
CottMecehep said,
February 15, 2012 at 12:22 pm
Shifting from Australia to United Standing, temperature drop to zero, specially in frozen winter, snow heavily north face jackets the garments for kids with the similar time.When the winter is coming across the corner, keeping ourselves warmer
plastic surgeon seo said,
March 6, 2012 at 9:07 pm
I leave a response whenever I like a article on a blog or I have something to valuable to contribute to the discussion. Usually it is caused by the fire communicated in the post I browsed. And on this article this blog name. I was excited enough to drop a thought
I do have 2 questions for you if it’s okay. Could it be just me or do a few of the remarks appear as if they are left by brain dead visitors? 
And, if you are writing at additional online social sites, I would like to follow anything fresh you have to post. Could you list every one of your social sites like your linkedin profile, Facebook page or twitter feed?
Susanemyh said,
June 28, 2012 at 11:04 pm
I pointed out there were three flavours of ACT!, nicely the 3rd is ACT! for Net. Inside UK this can be at this time bundled in the field no cost of cost once you acquire ACT! Premium. Fundamentally it truly dre studio beats is ACT! Premium with the further bits wanted to publish the database to an IIS world-wide-web server which you have to host on your own (or use a single with the paid back-for ACT! internet hosting providers).
cheap I want to buy beats dre headphones at http://www.verybeatsbydre.com beats by dre studio beats dre
Low frequency is still Studio focus - performing pretty good in low frequency. Not like Tour, Studio low frequency is thicker and simpler, no exaggerating descending but just meeting the needs of music like Hip-Hop. High frequency is one of the aspects that we have criticized on Tour - it is too sharp and not pleasant at all.
You should raise your Aces preflop most of the time, especially in online play. You wouldn’t be making a mistake if you raised them EVERY time. If you’re playing NL, raise at least 3 to 4 times the big blind. If you tend to be somebody who wants to listen to songs, you might have already heard concerning the Dr Dre headphones, or most generally known as monster headphones. Who would have although that Dr Dre would be making and delivering a revolutionary pair of headphones? Allow us to identify it, nowadays, majority of the youngster own an iPod or mp3, an iPod or mp3or has some thing inside their handbag to make them listen to songs each time and also anywhere. If iPods and other mp3, mp4 and mp5 players have earphones included within the package deal, then what makes these monsters beats earphones definitely diverse from them.
Luis Comito said,
September 4, 2012 at 1:19 pm
I do consider all of the ideas you’ve offered in your post. They’re really convincing and can definitely work. Still, the posts are very brief for beginners. May just you please extend them a little from subsequent time? Thank you for the post.
http://gredatechnologies.com/bungoma/profile/PhoebeYor said,
January 16, 2013 at 12:39 am
Some of the companies that offer printable coupons online include Aurelio’s pizza, California pizza chicken, Bucks pizza, Breadaeux pizza, chuck E. Cook the chicken first in a little olive oil - cut the chicken into little cubes. Once the consistency is there, I knead for about a minute.
apply for a credit card said,
January 30, 2013 at 8:35 pm
Hello! This post could not be written any better! Reading this post
reminds me of my old room mate! He always kept chatting
about this. I will forward this page to him.
Pretty sure he will have a good read. Thanks for
sharing!
http://home.leshantv.net/space said,
March 22, 2013 at 11:00 am
Shoes Under is a simple space-saving solution to shoe-cluttered closets and floors. And that’s the acumen why Christian Louboutin charge no advertisement. Particularly in the summer season they are ideal footwear options for both men and women. No matter how much a woman may discredit her figure, Christian Louboutin boots have a way of enhancing it to look lovely.. Quarter round molding is typically ” x ” with a ” radius profile, but you may be able to find different sized quarter molding. http://home.leshantv.net/space.php?uid=53311&do=blog&id=32030 http://home.leshantv.net/space.php?uid=53311&do=blog&id=32030
affiliate marketing said,
April 19, 2013 at 3:16 pm
After looking over a number of the blog articles on your website, I seriously
like your way of writing a blog. I book marked
it to my bookmark site list and will be checking back in the near future.
Please check out my website as well and let me know what you think.
http://www.atlascopair.com/Shownews.asp?id=3604 said,
May 6, 2013 at 1:48 am
I’m gone to convey my little brother, that he should also pay a visit this web site on regular basis to take updated from newest news.
buy best cheapest garden and outdoor online said,
May 8, 2013 at 2:23 am
My programmer is trying to convince me to move to .net from PHP.
I have always disliked the idea because of the expenses.
But he’s tryiong none the less. I’ve been using WordPress on a variety of websites
for about a year and am concerned about switching to another platform.
I have heard great things about blogengine.
net. Is there a way I can transfer all my wordpress posts into it?
Any kind of help would be really appreciated!
kinobo said,
May 20, 2013 at 11:58 am
Dead indited subject background , value it for information .
infobrokering said,
May 20, 2013 at 3:14 pm
Congratulations for the author of this blog! Great job, man. You’re a specialist. You should be example for other site owners and bloggers.
baltimore custom sticker said,
May 21, 2013 at 1:21 am
Sorry for my bad English.Excellent goods from you, man. I’ve understand your stuff previous to and you are just too magnificent. I really like what you have acquired here, certainly like what you are stating and the way in which you say it. You make it enjoyable and you still care for to keep it smart. I can not wait to read far more from you. This is really a great website.
Jerrell Blakeman said,
May 22, 2013 at 1:13 am
Hello, i feel that i saw you visited my blog thus i got here to “go back the prefer”.I’m attempting to to find things to improve my web site!I guess its adequate to use some of your ideas!!
carnivor said,
May 22, 2013 at 10:25 pm
Hi there would you mind letting me know which hosting company you’re using? I’ve loaded your blog in 3 completely different browsers and I must say this blog loads a lot quicker then most. Can you recommend a good hosting provider at a honest price? Thank you, I appreciate it!