Tag: python

IPython Craziness on Mac OSX Leopard

For those who are unfamiliar with IPython: it’s a Python shell replacement that adds many features for developers. It’s beyond the scope of this post to go into greater detail, but if you want to learn more you can visit the IPython website.

IPython has been an enormous aid to me in developing with Django, but there was one issue I was having that nearly drove me nuts. Running on Mac OSX Leopard, I found that occasionally, as I was typing a long bit of Python, the screen would go all (for lack of a better word) wonky and part of what I was typing would go into another part of the terminal screen away from the actual command line. Other times, as I would cycle through previously typed commands, the same thing would occur, obscuring or removing entirely the possibility of editing certain lines I had typed in. The end result: I ended up having to waste much time retyping the same or nearly the same commands over again, or worse, an entire block if I was in the middle of writing a method.

Well, today, I finally got sick of it and started searching for a fix. It was difficult to search for since it’s hard to even explain what happens, but I did find a fix at IPython’s site, of all places. According to the site:

Many of the problems users have experienced in IPython on OS X leopard are due to Apple’s use of the BSD editline library instead of the GNU readline library in Leopard’s system python. Ludwig Schwardt’s has created an egg that statically links agains the GNU readline library.

To get the egg, simply run the following from the command-line:

sudo easy_install -f http://ipython.scipy.org/dist/ readline

The egg can also be downloaded directly from IPython’s site.

  • Del.icio.us
  • Digg
  • Facebook
  • Google Bookmark
  • StumbleUpon
  • Technorati

A Django Snippet to Refresh Your Database

Django doesn’t have anything remotely close to migrations in Rails. It does have the syncdb management command, but that’s only good for initial table creation. If you need to add or remove a model attribute at a later point, you’re pretty much stuck with manual alter table commands to your database.

I’ve found that I have to do this entirely too often in development, and it becomes increasingly frustrating with frequency. As a result, I’ve written a little python script to take care of all the grunt work for me.

Disclaimer

Now, before I post the code, let me make clear that although I’ve used this a number of times already, I cannot guarantee that it’s 100% bug-free. It should definitely only be used on a development database, if for no other reason than should the catastrophic happen, you’ve only lost dummy data and not real user accounts and such. Also, I don’t claim to be a python expert. In fact, part of the reason I’m posting this is that hopefully a few python experts might stumble by and critique my code. Still, all that said, I think it still has benefit for the community, and hopefully it will be a help to some of you.

Code

#!/usr/bin/env python

print “Setting Up Environment… “,
try:
import settings # Assumed to be in the same directory.
except ImportError:
import sys
sys.stderr.write(”Error: Can’t find the file ’settings.py’ in the directory containing %r.\n(If the file settings.py does indeed exist, it’s causing an ImportError somehow.)\n” % __file__)
sys.exit(1)

from django.core.management import setup_environ, call_command
setup_environ(settings)
print “Done”

import sys

print “Dumping Data… “,
sys.stdout = open(’dumped_data.json’, ‘w’)
call_command(’dumpdata’, format=’json’, indent=4)
sys.stdout.close()
sys.stdout = sys.__stdout__
print “Done”

print “Deleting Tables… “,
import StringIO
from django.db.models import get_apps
app_labels = [app.__name__.split('.')[-2] for app in get_apps()]
sys.stdout = buffer = StringIO.StringIO()
call_command(’sqlclear’, *app_labels)
sys.stdout = sys.__stdout__

queries = buffer.getvalue().split(’;')[1:-2]

from django.db import connection
cursor = connection.cursor()
for query in queries:
cursor.execute(query.strip())
print “Done”

print “Synching Database… ”
#disable the “create a super user” question
from django.contrib.auth.management import create_superuser
from django.contrib.auth import models as auth_app
from django.db.models import signals
from django.dispatch import dispatcher
dispatcher.disconnect(create_superuser, sender=auth_app, signal=signals.post_syncdb)

call_command(’syncdb’)
print “Done”

print “Loading Back Data… ”
call_command(’loaddata’, ‘dumped_data.json’)
import os
os.remove(’dumped_data.json’)
print “Done”

What Does it Do?

First, the script loads in your project’s settings.py and sets up the Django environment. If you’ve ever looked at the code behind manage.py, you’ll notice I basically ripped that part off. Second, the script uses Django’s dumpdata management command to save all the current data in the database in a JSON object.

Third, it gets a list of all the currently install apps, and uses Django’s sqlclear management command to get the database specific delete statements. Essentially, it’s just emptying out the database. I debated about simply dropping the database and creating a new one, but I wanted this script to be as database agnostic as possible.

Next, it runs Django’s syncdb management command to recreate the database based on the current state of your models. However, before is does that, it disables the signal responsible for the “create a superuser” prompt that Django’s Auth module gives. Finally, it reloads the data we dumped previously back into the database.

At this point, you should have an exact duplicate of your old database, only with the changes you made to your models implemented.

How to Use it?

Save that code in an file in the root of your project (same level as settings.py). I called mine ‘refresh_db.py’. If you’re on a *nix based platform you’ll want to give the script execute priveleges, as well (chmod 755). Then, just run it from the command line, whenever you see fit.

  • Del.icio.us
  • Digg
  • Facebook
  • Google Bookmark
  • StumbleUpon
  • Technorati

Signals in Django: Stuff That’s Not Documented (Well)

I’ve just spent the last few hours learning how to use signals in Django. After many, many searches on Google and much trial and error, I think I finally have a grasp on these silly things, and since I’m an all around nice guy, I’m going to spare those lucky few that happen upon this post the same hell.

Before I start, I want to go ahead and give credit to those who provided some of the crucial pieces to the puzzle during my quest.

Okay, now let’s get started.

Creating Custom Signals

In the application I’m working on, I needed to send an email whenever a user reset his/her password, a pretty common use case. Unfortunately, none of Django’s built-in signals fit the bill. The User model gets saved when the password is reset (allowing the use of the post_save signal), but it also gets saved in a lot of other scenarios. The password reset is a special case and needed to be handled as such.

Turns out that it’s actually not that difficult to set this up. If you haven’t already, create a file named signals.py in directory of the application you’re working on. Then in that file, add the following:

# myapp/signals.py

password_reset = object()

The name `password_reset` is inconsequential; use whatever name best conveys the action that causes the signal to be sent.

Next, set up a listener for that signal. The following code can technically go just about anywhere, as long as it gets executed before the signal is sent. I put it in models.py for ease.

# myapp/models.py

from django.dispatch import dispatcher
from myproject.myapp import signals as custom_signals

dispatcher.connect(send_password_reset_email, signal=custom_signals.password_reset)

`send_password_reset_email` is the function that will be called when the signal is received. Obviously, we’ll need to set that up. Back to signals.py:

# myapp/signals.py

from django.conf import settings
from django.core.mail import send_mail
from django.contrib.sites.models import Site
from django.template.loader import render_to_string

def send_password_reset_email(sender, user, new_pass, signal, *args, **kwargs):
current_site = Site.objects.get_current()
subject = “Password Reset on %(site_name)s” % { ’site_name’: current_site.name }
message = render_to_string(
‘account/password_reset_email.txt’,
{ ‘username’: user.username,
‘new_pass’: new_pass,
‘current_site’: current_site }
)
send_mail(subject, message, settings.DEFAULT_FROM_EMAIL, [user.email])

Exactly how this code works is left as an exercise to the reader. I provided it merely to be comprehensive. What the function that gets called when the signal is received does will be specific to your purposes.

However, there are a few points worth mentioning. When you use Django’s built-in signals, your function definition will almost invariably look like the following:

def my_function(sender, instance, signal, *args, **kwargs):

Notice that my definition didn’t include `instance` and had `user` and `new_pass` arguments instead. You can pass whatever arguments you like when you send the signal (we’ll get to this in a second). The only requirement is that the function that gets called can handle them. Django simply chose to use an argument named `instance`, nothing more, nothing less.

Finally, at the exact point where you want function associated with the signal executed (in my case, right after a new password is generated and the User instance is saved) insert the following:

dispatcher.send(signal=custom_signals.password_reset, user=self.user, new_pass=new_pass)

The only required part is obviously the signal you want to send. Everything after is simply data you’d like to pass along. Remember that the function definition in signals.py must accept all the arguments you choose to pass in.

Also don’t forget to add the following to your imports in the file where you call dispatcher.send():

from django.dispatch import dispatcher
from myproject.myapp import signals as custom_signals

And we’re done. Easy as pie, once you know what you’re doing.

Signaling Just When an Object is Created

Conspicuously missing from Django’s built-in signals is one for the creation of an object. We have pre_save and post_save signals, but both of those work whether the object is being created or just being updated.

Again, finding a solution to this little issue was spurred by my own needs. I wanted to send a welcome email when a user first registers, another common use case. Obviously, I didn’t want the same welcome email sent everytime the user updates their details so post_save was out. Or was it?

After a fair amount of digging, I found the solution squirreled away in Django’s model tests. Apparently, Django automagically passes in a `created` flag if the object was created, so all that’s required is to test for that flag before you whatever you plan on doing (sending the welcome email in my case).

def send_welcome_email(sender, instance, signal, *args, **kwargs):
if ‘created’ in kwargs:
if kwargs['created']:
# Send email

First, we test that a `created` argument was passed in. If it was, we verify that it’s value is True.

Finally, set up a listener as usual. Again, where you put it matters not as long as it gets executed before the signal gets sent; models.py is a good place.

# myapp/models.py

from django.dispatch import dispatcher
from django.db.models import signals

dispatcher.connect(send_welcome_email, signal=signals.post_save, sender=UserProfile)

The `sender` argument limits the signal to being sent only for that particular model. I chose to send it upon the creation of UserProfile, the profile module associated with User in my app. If you left this part out, our send_welcome_email function would be called everytime any model in your application was saved, which would obviously not be desirable.

And, just like that, you get code that will only execute when the model is first created.

Handling Signals Asynchronously

Both of the above examples send emails. It normally doesn’t take much time to send an email, but if the server load is heavy, it could take longer than normal. Ideally, anytime you do anything like this, you want it to be done asynchronously so that users don’t have to wait for the processing to finish before they can move on to something else.

Django’s signals provide half the functionality by decoupling the code for sending the email from the view. However, by default, Django’s signals are synchronous; the signal gets sent and the application waits for its successful completion before moving on. Thankfully, Python has the answer in its threading module.

A thread, extremely simplified, can be thought of as a branch of a running program (Django in this case). It becomes its own entity, able to run independently of its parent process. Extremely simplified, again, you could think of it as a little mini-Django tasked with a very specific and finite purpose. Once it completes its function, it goes away. This purchases us the ability to let something run, while still continuing on in our application in general.

While it sounds rather complicated, it’s actually relatively easy to set up. The following is the actual code I’m using for the welcome email discussed earlier:

import threading

class WelcomeEmailThread(threading.Thread):
def __init__(self, instance):
self.instance = instance
threading.Thread.__init__(self)

def run (self):
# The actual code we want to run, i.e. sending the email

def send_welcome_email(sender, instance, signal, *args, **kwargs):
if ‘created’ in kwargs:
if kwargs['created']:
WelcomeEmailThread(instance).start()

First, we create a new class which subclasses threading.Thread. The name of the class is inconsequential; just pick something descriptive. In this class, we have two functions defined: `__init__` and `run`.

The `__init__` function is provided to allow us to pass in the instance we received from the signal. We store the instance as an attribute so it can be retrieved later, and then we call the `__init__` method on threading.Thread. This is necessary because we have overloaded (replaced) the `__init__` function inherited from threading.Thread, but we did not replace all of its functionality as well. Therefore, it still needs to complete its normal initialization procedures.

The `run` function is the heart of the class. This is where the email sending will now occur.

Finally, in the `send_welcome_email` function, which previously housed the code for sending the email, we now start the thread instead. The `if` statements are just specifying that this code should only run if the object is being created instead of updated (see previous topic).

That’s all that’s required. Now, when the signal is received it will simply spawn off a thread to send the email and return processing back to the rest of our application. Not bad at all.

Wrap Up

That’s all I’ve got for now, but I think it covers the three most confusing areas of Django’s signals. Happy Django’ing.

  • Del.icio.us
  • Digg
  • Facebook
  • Google Bookmark
  • StumbleUpon
  • Technorati

Rails is a Ghetto?

Just noticed Zed Shaw’s soon to be infamous rant on Rails and the Rails community. As a little background, Zed Shaw is of course the creator of Mongrel, the web server that has become the staple deployment apparatus for Rails.

Zed’s rant is long and boisterous; typically, I view such ranting with a healthy dose of skepticism. However, some of what he says in the post makes a little too much sense to shrug off as just malevolent lashing-out.

I jumped on the Rails bandwagon mid-last-year, and to me, it became a salvation of sorts. I experienced a programming environment that simultaneously gave me power to create and joy in development. Often those two things are toss ups, where one need either be limited or lulled into a near comatose state.

However, as I’ve worked with Rails more, I’ve realized that while it may have charted new frontier (being one of the first enterprise-level web frameworks for the common man), it is not the end all be all that many see it as.

Lately, I’ve begun development on a project at work that was slated to be developed in Django. This meant that I had to take a break from Rails and learn a new framework, a new language at that since I was not familiar with Python previously. Coming from Rails, I had a slight bias to Django from the start. For the most part, the two kids play nice, but every so often someone would post on the virtues of one of the two over the other. Django, in particular, seemed a bit of an upstart to me, with users and even the creators themselves often making grandiose claims of its superiority.

Initially, I found myself recoiling from many of Django’s “ways”. URLconfs made me want to kick someone’s ass. It was not until I got to creating templates in Django that it began to earn some respect in my eyes, but the clencher was the first time I tried to implement some AJAX.

Back when I was working on the TipDish project with the group from Startup Weekend Houston, I remember the hell I went through trying to implement AJAX in Rails. For some reason, I had a real mental with it. I knew how to do what I was trying to do outside of Rails, but I just couldn’t seem to get it working the Rails way inside Rails. I devoted two full nights of scouring the web and lurking in IRC before I found the answers I needed.

Django, I was sure, would be worse, knowing that documentation is more sparse for Django than Rails. I anticipated an even more hellish encounter with this framework, and at first, I seemed to receive it. I searched online for information to little avail. I found many references to Dojo and its imminent inclusion in Django. I looked at Dojo and cried. (As an aside in case I happen to the catch the notice of anyone on the Django development team: *please*, if you’re going to package a javascript framework, let it be anything but Dojo.)

Then, a miraculous light of entered my mind, complete with angels strumming on harps. I realized I was free. I could pick whatever javascript framework I wanted, or none at all. I could do it anyway I liked. I was not restricted into using built in functions or special code. And that’s when Django suddenly took on a whole new light for me. Here was something that did just what it was supposed to do: take care of all the cruft, so I don’t have to. After that, it get’s out of the way and let’s you do your thing.

Rails, for all it’s beauty, is controlling. It wants you to do things its way. Indeed, many times, I felt as if I was learning a new language just from the framework. Rails is training wheels. Even those that use it and swear by it seem to rewrite most of it as their apps grow, leaving very little Rails left to Rails. As Zed points out in his rant, the name itself admits to this; you’re supposed to stay on the “rails”.

Now, all this is not to say that I’m rejecting Rails now. I believe it will stay in my arsenal for quite some while, and I also believe it does have a lot of merit to itself. What I am saying is that the honeymoon period is over and I’m no longer star-struck in love with Rails. I’m also saying that the drastic shift of Zed Shaw against the Rails community (while probably not to be unexpected from Zed), should still be seen as an omen of sorts. There does seem to be a little too much “headiness” in the Rails community, where some people seem to feel they are just too damn smart to coexist with the rest of us. It’s not representative of the entire community, but it’s there, and it should be dealt with.

That said, check out Zed’s rant. If nothing else, it makes an entertaining read. And, if you haven’t already, give Django a look. You can never have too many tools in your arsenal as a web developer and its got some real power.

  • Del.icio.us
  • Digg
  • Facebook
  • Google Bookmark
  • StumbleUpon
  • Technorati