bitkickers

Using Access-Control-Allow-Origin to make cross domain POST requests from javsacript

2012-01-27T14:42:00.000-05:00

Making ajax calls from javascript, even without a framework like jQuery, is pretty trivial. However, once you try to make the same request cross-domain, it gets hard fast. This is due to the security model all modern browsers use, known as the same origin policy. This policy makes sense in a lot of ways, but it's also somewhat broken and antiquated on the web today.

Why might you want to make cross domain requests? Just take a look at some examples:

Google Analytics sends metrics from your page to their central servers.
HousingMaps, a mashup of craigslist and GoogleMaps.
The Read Later Instapaper bookmarklet.

For cross domain ajax requests, the same origin policy is "broken" in the sense that there have been work-arounds available for a years. First there were proxy pages, where you wrap a remote URL in a page being served by the site the user is actually on. The proxy requires that you be in control of the page the user is on. Then there was a dual IFRAME hack, but that requires that you be in control of both ends.

Recently, JSONP has emerged as a standard to allow cross domain requests. It's something of a hack that works by returning an executable javascript function from a script tag request. It also exposes you to potential javascript injection vulnerabilities if you are not in control of the remote host. On the plus side, it's provided seamlessly as a feature in recent jQuery versions.

Luckily, there is a true standard emerging that's not built on top of a hack. In 2004, the W3C started work on a draft called Cross-Origin Resource Sharing (CORS). In 2009, Mozilla was the first browser to implement support for CORS. Currently Chrome also supports it. It allows a HTTP server to set some new headers that tell the browser to modify its same origin policy.

For example, the following Django code with allow an ajax request from another domain to your server.

def myview(_request):
    response = HttpResponse(json.dumps({"key": "value", "key2": "value"}))
    response["Access-Control-Allow-Origin"] = "*"
    response["Access-Control-Allow-Methods"] = "POST, GET, OPTIONS"
    response["Access-Control-Max-Age"] = "1000"
    response["Access-Control-Allow-Headers"] = "*"
    return response

In typical fashion, Microsoft went ahead and implemented their own incompatible version of the same thing, which they call XDR. If you're masochistic, you can try to use it. Otherwise, you can wait until IE10 when they will support CORS, too.

Python class attributes are evaluated on declaration

2012-01-20T13:31:00.000-05:00

In Python, class attributes are evaluated and put into memory when the class is defined (or imported). For example, if you run the following code in an interactive interpreter, it will print out "Something __init__() called":

class Something:
    def __init__(self):
        print "Something __init__() called"

class UsesSomething:
    field = Something()

This surprised me, even though I've been coding almost entirely in Python for two years. I expected the print statement to execute if I instantiated a UsesSomething object, but I did not expect it to execute by simply declaring the class definition. In practice, the declaration is likely to be when you import the module containing the code.

This came up in a real world scenario when my Django app suddenly began taking a lot longer to start up. On a hunch, I started removing imports from urls.py until I found the one that was causing the slowness. Then I started removing code in that module, until I traced the problem to something like the following:

class MyForm(ModelForm):
    class Meta:
        model = MyModel
    field1 = MyChoiceField()

class MyChoiceField(ChoiceField):
    def __init__(self, choices=(), required=True, widget=None, label=None,
             initial=None, help_text=None, *args, **kwargs):
    super(ChoiceField, self).__init__(required, widget, label, initial,
                                      help_text, *args, **kwargs)     
    self.choices = [(m.id, m.name) for m in ReallyLargeTableModel.objects.all()]

For me, this was a pretty standard pattern. I had a custom field that was pulling its set of valid choices from the database. As the full ReallyLargeTableModel data set got larger and larger, it eventually got larger than the maximum size our caching back-end would store, and thus the query started getting run every time the run time started up.

In this case, it's not hard to work around. Simply removing the list comprehension will allow Django to hold off on running the query until it actually requests the choices property (ie, when the field is displayed to the user). Alternatively, you could pass the data in from the MyForm constructor, which I gather is more idiomatic anyway.

Solutions are besides the point. In this case, the important take away is to be aware that class attributes will be evaluated on import, NOT when they are instantiated. So you want to avoid heavy weight operations or external dependencies in class attribute definitions.

Django reset database connection

2012-01-06T14:15:00.002-05:00

Django handles database connections transparently in almost all cases. It will start a new connection when your request starts up, and commit it at the end of the request lifetime. Other times you need to dive in further and do your own granular transaction management. But for the most part, it's fully automatic.

However, sometimes your use case may require that you close the current database connection and open a new one. While this is possible in Django, it's not well documented.

Why would you want to do this? I my case, I was writing an automation test framework. Some of the automation tests make database calls through the Django ORM to setup records, clean up after the test, etc. Each test is executed in the same process space, via a thread pool. We found that if one of the early tests threw an unrecoverable database error, such as an IntegrityError due to violating a unique constraint, the database connection would be aborted. Subsequent tests that tried to use the database would raise a DatabaseError:

Traceback (most recent call last):
  File /home/user/project/app/test.py, line 73, in tearDown
    MyModel.objects.all()
  File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 444, in delete
    collector.collect(del_query)
  File /usr/local/lib/python2.6/dist-packages/django/db/models/deletion.py, line 146, in collect
    reverse_dependency=reverse_dependency)
  File /usr/local/lib/python2.6/dist-packages/django/db/models/deletion.py, line 91, in add
    if not objs:
  File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 113, in __nonzero__
    iter(self).next()
  File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 107, in _result_iter
    self._fill_cache()
  File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 772, in _fill_cache
    self._result_cache.append(self._iter.next())
  File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 273, in iterator
    for row in compiler.results_iter():
  File /usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py, line 680, in results_iter
    for rows in self.execute_sql(MULTI):
  File /usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py, line 735, in execute_sql
    cursor.execute(sql, params)
  File /usr/local/lib/python2.6/dist-packages/django/db/backends/postgresql_psycopg2/base.py, line 44, in execute
    return self.cursor.execute(query, args)
DatabaseError: server closed the connection unexpectedly
 This probably means the server terminated abnormally
 before or while processing the request.

It turns out that it's relatively easy to reset the database connection. We just called the following function at the start of every test. Django is smart enough to re-initialize the connection the next time it's used, assuming that it's disconnected properly.

def reset_database_connection():
    from django import db
    db.close_connection()

nginx + SSL reverse proxy tutorial

2011-12-21T16:40:00.000-05:00

If you have an existing HTTP application that you want to enable SSL for, nginx provides a convenient reverse proxy mode. In this mode, nginx is just responsible for providing SSL encryption; the actual HTTP content is still being served from your existing web server or load balancer.

First, you need to install nginx. On Ubuntu 11.10, you can install it directly from the Ubuntu repositories.

sudo apt-get install nginx

On older Ubuntu distributions, you will need to add a custom PPA.

sudo -s
nginx=stable # use nginx=development for latest development version
echo "deb http://ppa.launchpad.net/nginx/$nginx/ubuntu lucid main" > /etc/apt/sources.list.d/nginx-$nginx-lucid.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
apt-get update 
apt-get install nginx

Next, you need to generate a Certificate Signing Request and a secret key file. I created mine directly inside a new directory that we will point nginx to.

sudo mkdir /usr/local/nginx
sudo mkdir /usr/local/nginx/conf
cd /usr/local/nginx/conf
sudo openssl genrsa -des3 -out server.key 2048
sudo openssl req -new -key server.key -out server.csr

You will be asked a bunch of questions about the SSL certificate. This is semi-official information, so you want it to be accurate. The certificate authority may actually call/email people to verify some of the information. Technically, the only important field is the Common Name, which is the domain name you want the certificate for. You should look up some SSL creation tutorials to make sure you have the correct values.

In my case, I wanted a wildcard certificate, which can be used for all subdomains on the domain in question, so I entered *.example.com for the common name.

Now you get out your credit card. You need to purchase a certificate based on this .csr file. I chose Digicert. A wildcard certificate cost about $1300 for three years.

They will ask you for the csr file. You can also paste in the text of the file, which looks something like this.

The certificate authority vendor will produce a .pem file, which has both your new certificate and the vendor's certificate bundled together. Once you get the certificate via email, you can proceed to setup a reverse proxy in nginx. First, remove the passphrase from the .key file. Otherwise, nginx will prompt for it interactively every time the service starts.

sudo cp server.key server.key.org
sudo openssl rsa -in server.key.org -out server.key

Then you have to configure nginx to use the .pem file from the certificate authority, as well as your secret .key file. There is an excellent wiki for other configurations that nginx supports.

server {

    listen 443 default_server;
    server_name www.example.com;

    ssl on;
    ssl_certificate /usr/local/nginx/conf/server.pem;
    ssl_certificate_key /usr/local/nginx/conf/server.key;
    ssl_session_cache shared:SSL:10m;

    location / {

        proxy_pass http://localhost:8000; # my existing apache instance
        proxy_set_header Host $host;

        # re-write redirects to http as to https, example: /home
        proxy_redirect http:// https://;
    }
}

Note the proxy_redirect directive. I noticed that when my application performed a redirect internally, it was using the global URL, ie http://www.example.com. To keep the user on the SSL version w/o having to make my application SSL aware, or change to protocol relative redirects, I simply told nginx to re-write the redirect on the fly. Since I am only asking nginx to handle SSL traffic, it can safely use https for all redirects, given that I'm not redirecting to other sites.

That's it. You just need to start the server with "service nginx start". You can test that it's working in a browser, or "openssl s_client -connect localhost:443" for a more verbose test.

Migrate your Python app to Facebook OAuth 2.0

2011-12-16T11:28:00.000-05:00

Facebook recently started enforcing OAuth 2.0 on app developers. Supposedly this was slated for Oct 1, 2011, but in practice it seems to have not started until Dec 15. If, like me, you didn't realize the change was coming until it was already broken, then you might be scrambling for a fix. Here are the changes I needed to make in our Django app using the Facebook python sdk and javascript sdk.

On the javascript side, there are some new FB.init options. Here is our old code:

window.fbAsyncInit = function() {
 FB.init({appId: window.FACEBOOK_TOKEN, status: true, cookie: true, xfbml: true});
 FB.Canvas.setAutoResize();
};

And here is the new code:

window.fbAsyncInit = function() {
    FB.init({
      appId      : window.FACEBOOK_TOKEN,
      channelUrl : window.SITE_URL + "/static/channel.html",
      status     : true,
      cookie     : true,
      oauth      : true,
      xfbml      : false
    });
    FB.Canvas.setAutoResize();
  };

Notice the "oauth" setting, and the "channelUrl" setting. channelUrl references a static file on your server, and is used to work-around cross domain scripting issues. It just needs to have the following static content in it:

<script src="//connect.facebook.net/en_US/all.js"></script>

Besides the init method, the FB.login changed the name on the permissions attribute from "perms" to "scope":

FB.login(function(response) {
    /* ... do stuff here */
}, { scope: "publish_stream,offline_access" }); /* used to be "perms"

The more troublesome change was on the Python SDK side. Previously, when the javascript sdk authenticated a user, it would add a "_fbs_APP_ID" cookie to the response, which you would use get_user_from_cookie() from the python sdk to parse. With the OAuth 2.0 change, they moved to a new _fbr_APP_ID cookie (note the "r"), with a new format. As of Dec 15, the official python sdk has not been updated to support this new format.

Luckily, I was able to find a community produced version.

#!/usr/bin/env python
#
# Copyright 2010 Facebook
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

"""Python client library for the Facebook Platform.

This client library is designed to support the Graph API and the official
Facebook JavaScript SDK, which is the canonical way to implement
Facebook authentication. Read more about the Graph API at
http://developers.facebook.com/docs/api. You can download the Facebook
JavaScript SDK at http://github.com/facebook/connect-js/.

If your application is using Google AppEngine's webapp framework, your
usage of this module might look like this:

    user = facebook.get_user_from_cookie(self.request.cookies, key, secret)
    if user:
        graph = facebook.GraphAPI(user["access_token"])
        profile = graph.get_object("me")
        friends = graph.get_connections("me", "friends")

"""

import cgi
import hashlib
import time
import urllib

# Find a JSON parser
try:
    import json
    _parse_json = lambda s: json.loads(s)
except ImportError:
    try:
        import simplejson
        _parse_json = lambda s: simplejson.loads(s)
    except ImportError:
        # For Google AppEngine
        from django.utils import simplejson
        _parse_json = lambda s: simplejson.loads(s)


class GraphAPI(object):
    """A client for the Facebook Graph API.

    See http://developers.facebook.com/docs/api for complete documentation
    for the API.

    The Graph API is made up of the objects in Facebook (e.g., people, pages,
    events, photos) and the connections between them (e.g., friends,
    photo tags, and event RSVPs). This client provides access to those
    primitive types in a generic way. For example, given an OAuth access
    token, this will fetch the profile of the active user and the list
    of the user's friends:

       graph = facebook.GraphAPI(access_token)
       user = graph.get_object("me")
       friends = graph.get_connections(user["id"], "friends")

    You can see a list of all of the objects and connections supported
    by the API at http://developers.facebook.com/docs/reference/api/.

    You can obtain an access token via OAuth or by using the Facebook
    JavaScript SDK. See http://developers.facebook.com/docs/authentication/
    for details.

    If you are using the JavaScript SDK, you can use the
    get_user_from_cookie() method below to get the OAuth access token
    for the active user from the cookie saved by the SDK.
    """
    def __init__(self, access_token=None):
        self.access_token = access_token

    def get_object(self, id, **args):
        """Fetchs the given object from the graph."""
        return self.request(id, args)

    def get_objects(self, ids, **args):
        """Fetchs all of the given object from the graph.

        We return a map from ID to object. If any of the IDs are invalid,
        we raise an exception.
        """
        args["ids"] = ",".join(ids)
        return self.request("", args)

    def get_connections(self, id, connection_name, **args):
        """Fetchs the connections for given object."""
        return self.request(id + "/" + connection_name, args)

    def put_object(self, parent_object, connection_name, **data):
        """Writes the given object to the graph, connected to the given parent.

        For example,

            graph.put_object("me", "feed", message="Hello, world")

        writes "Hello, world" to the active user's wall. Likewise, this
        will comment on a the first post of the active user's feed:

            feed = graph.get_connections("me", "feed")
            post = feed["data"][0]
            graph.put_object(post["id"], "comments", message="First!")

        See http://developers.facebook.com/docs/api#publishing for all of
        the supported writeable objects.

        Most write operations require extended permissions. For example,
        publishing wall posts requires the "publish_stream" permission. See
        http://developers.facebook.com/docs/authentication/ for details about
        extended permissions.
        """
        assert self.access_token, "Write operations require an access token"
        return self.request(parent_object + "/" + connection_name, post_args=data)

    def put_wall_post(self, message, attachment={}, profile_id="me"):
        """Writes a wall post to the given profile's wall.

        We default to writing to the authenticated user's wall if no
        profile_id is specified.

        attachment adds a structured attachment to the status message being
        posted to the Wall. It should be a dictionary of the form:

            {"name": "Link name"
             "link": "http://www.example.com/",
             "caption": "{*actor*} posted a new review",
             "description": "This is a longer description of the attachment",
             "picture": "http://www.example.com/thumbnail.jpg"}

        """
        return self.put_object(profile_id, "feed", message=message, **attachment)

    def put_comment(self, object_id, message):
        """Writes the given comment on the given post."""
        return self.put_object(object_id, "comments", message=message)

    def put_like(self, object_id):
        """Likes the given post."""
        return self.put_object(object_id, "likes")

    def delete_object(self, id):
        """Deletes the object with the given ID from the graph."""
        self.request(id, post_args={"method": "delete"})

    def request(self, path, args=None, post_args=None):
        """Fetches the given path in the Graph API.

        We translate args to a valid query string. If post_args is given,
        we send a POST request to the given path with the given arguments.
        """
        if not args: args = {}
        if self.access_token:
            if post_args is not None:
                post_args["access_token"] = self.access_token
            else:
                args["access_token"] = self.access_token
        post_data = None if post_args is None else urllib.urlencode(post_args)
        file = urllib.urlopen("https://graph.facebook.com/" + path + "?" +
                              urllib.urlencode(args), post_data)
        try:
            response = _parse_json(file.read())
        finally:
            file.close()
        if response.get("error"):
            raise GraphAPIError(response["error"]["type"],
                                response["error"]["message"])
        return response


class GraphAPIError(Exception):
    def __init__(self, type, message):
        Exception.__init__(self, message)
        self.type = type


##### NEXT TWO FUNCTIONS PULLED FROM https://github.com/jgorset/facepy/blob/master/facepy/signed_request.py

import base64
import hmac


def urlsafe_b64decode(str):
    """Perform Base 64 decoding for strings with missing padding."""

    l = len(str)
    pl = l % 4
    return base64.urlsafe_b64decode(str.ljust(l+pl, "="))


def parse_signed_request(signed_request, secret):
    """
    Parse signed_request given by Facebook (usually via POST),
    decrypt with app secret.

    Arguments:
    signed_request -- Facebook's signed request given through POST
    secret -- Application's app_secret required to decrpyt signed_request
    """

    if "." in signed_request:
        esig, payload = signed_request.split(".")
    else:
        return {}

    sig = urlsafe_b64decode(str(esig))
    data = _parse_json(urlsafe_b64decode(str(payload)))

    if not isinstance(data, dict):
        raise SignedRequestError("Pyload is not a json string!")
        return {}

    if data["algorithm"].upper() == "HMAC-SHA256":
        if hmac.new(secret, payload, hashlib.sha256).digest() == sig:
            return data

    else:
        raise SignedRequestError("Not HMAC-SHA256 encrypted!")

    return {}



def get_user_from_cookie(cookies, app_id, app_secret):
    """Parses the cookie set by the official Facebook JavaScript SDK.

    cookies should be a dictionary-like object mapping cookie names to
    cookie values.

    If the user is logged in via Facebook, we return a dictionary with the
    keys "uid" and "access_token". The former is the user's Facebook ID,
    and the latter can be used to make authenticated requests to the Graph API.
    If the user is not logged in, we return None.

    Download the official Facebook JavaScript SDK at
    http://github.com/facebook/connect-js/. Read more about Facebook
    authentication at http://developers.facebook.com/docs/authentication/.
    """

    cookie = cookies.get("fbsr_" + app_id, "")
    if not cookie:
        return None

    response = parse_signed_request(cookie, app_secret)
    if not response:
        return None

    args = dict(
        code = response['code'],
        client_id = app_id,
        client_secret = app_secret,
        redirect_uri = '',
    )

    file = urllib.urlopen("https://graph.facebook.com/oauth/access_token?" + urllib.urlencode(args))
    try:
        token_response = file.read()
    finally:
        file.close()

    access_token = cgi.parse_qs(token_response)["access_token"][-1]

    return dict(
        uid = response["user_id"],
        access_token = access_token,
    )

jQuery hashtag input control

2011-12-09T14:12:00.001-05:00

This jQuery snippet will turn an INPUT element into a hashtag input control. For example, if the user type "jobs business networking", it will translate it into "#jobs #business #networking" as they are typing it. It will also string commas and whitespace.

$(document).ready(function() {

 $("input.hastags").bind("keyup", function () {

  /* replaces text as you type with hashtags */

  var input = $(this);
  var value = input.val();
  var ends_with_space = (value.substr(-1) == " ");

  var hashed_value = "";
  var parts = value.split(" ");
  for (var i = 0; i < parts.length; i++) {
   var part = parts[i];
   if (part.indexOf("#") != 0) {
    part = "#" + part;
   }
   if (part != "#") {
    if (hashed_value == "") {
     hashed_value = part;
    } else {
     hashed_value += " " + part;
    }
   }
  }
  if (ends_with_space) {
   hashed_value = hashed_value + " ";
  }
  input.val(hashed_value.replace(",", ""));
 });

});

Ubuntu 11.10 Lock Screen - Suspend Monitors

2011-12-02T10:14:00.001-05:00

Whenever I get up from my desk, I reflexively lock my workstation. Ever since upgrading to Ubuntu 11.10, this has failed to put my monitors into suspend mode. In addition, it also continues to display some parts of the Unity interface, most critically the title of the currently active window.

I assume that it's related to using the proprietary ATI drivers, possibly also with multiple monitors. I've actually filed a bug for it, and I'm not the only one. In the meantime, there is a work-around, at least for the power saving component. Here is the bash script:

#!/bin/bash
gnome-screensaver-command -l
sleep 3
xset dpms force off

Save this as a file, mark it as executable (chmod +x), and they you can map a custom keyboard shortcut to it.

Open the keyboard shortcut configuration ("keyboard" from the dash)
Go to "Custom Shortcuts"
Add a shortcut called "Lock Screen + Sleep Monitor", and point it to the bash script.

On my machine, I've mapped it to Control + ALT + L. It doesn't solve the security issue, but at least you're not burning out your monitors.

Python/Django: disk based caching decorator

2011-11-23T14:22:00.001-05:00

One of the great things about Django's caching framework is that you can cache complex objects. Say you have a list of dictionaries representing favorite movies from Netflix. You can jam that sucker right into the cache as is. No need to normalize it into a relational database schema, unless you actually need to deal with it relationaly.

The most popular Django cache back-end is memcached. If you do such caching a lot, you will eventually run into a limit on the maximum size of those objects, which is 1MB. Now, you could just up the configured limit. But you need to worry about potentially kicking other items out of the cache early as you increase the average cached object's size.

In the case of Netflix movies, maybe the speed of the cache isn't important. If you're just caching them to save an API hit, and you only need the data occasionally, maybe you can get away to caching to disk. This will keep the hit rate on your regular cache high.

Here is a quick a dirty decorator that takes any object that pickle can serialize, and writes it to a file. Subsequent calls to the decorator just bring backed the cached object for a configurable duration of time.

def cache_disk(seconds = 900, cache_folder="/tmp"):
    def doCache(f):
        def inner_function(*args, **kwargs):

            # calculate a cache key based on the decorated method signature
            key = sha1(str(f.__module__) + str(f.__name__) + str(args) + str(kwargs)).hexdigest()
            filepath = os.path.join(cache_folder, key)

            # verify that the cached object exists and is less than $seconds old
            if os.path.exists(filepath):
                modified = os.path.getmtime(filepath)
                age_seconds = time.time() - modified
                if age_seconds < seconds:
                    return pickle.load(open(filepath, "rb"))

            # call the decorated function...
            result = f(*args, **kwargs)

            # ... and save the cached object for next time
            pickle.dump(result, open(filepath, "wb"))

            return result
        return inner_function
    return doCache

You can then wrap any function in this decorator to cache the results to disk.

@cache_disk(seconds = 900, cache_folder="/tmp"):
def get_netflix_favorites(account_id):
   ... do somthing really expensive
   return {
      "account_id": account_id,
      "data": {
           ... more stuff here
      }
   }

Running a JAR as a service (Linux/upstart)

2011-11-18T11:53:00.001-05:00

Running a java JAR as a daemon in Linux is fairly easy, but it took me some digging to figure out how. This is using the new upstart init script functionality, which is included in recent distributions of Ubuntu.

Instead of creating a script in /etc/init.d, as with System-V init, you create a .conf file in /etc/init. The syntax is much simpler, and upstart takes care of PID files and killing the process for you. You don't have to mark the file as executable.

This particular example is for a java JAR by SauceLabs for their SauceConnect service. All you have to do is create the following as /etc/init/sauceconnect.conf, where "sauceconnect" is the name of the service you will be invoking with service start/stop.

description "SauceLabs SauceConnect Service"
author "Chase Seibert"

start on runlevel [3]
stop on shutdown

expect fork

script   
    cd /home/chase/bullhorn/tools/sauceconnect
    java -jar /home/chase/bullhorn/tools/sauceconnect/Sauce-Connect.jar USERNAME PASSWORD >/var/log/sauceconnect.log 2>&1
    emit sauceconnect_running
end script

That's it. Now you can issue the "service sauceconnect start" and "service sauceconnect stop" commands. You can tail /var/log/sauceconnect.log to verify that it's working.

Backing up gmail to Dropbox using getmail on Linux

2011-11-10T13:09:00.001-05:00

What if gmail went away tomorrow, or at least deleted all your mail? It may not be as far fetched as you think. It's always prudent to have a backup.

On Linux, you can use the excellent getmail program to backup any POP3 or IMAP account. For gmail in particular, the first step is to enable IMAP access.

Installing and configuring getmail is simple.

sudo apt-get install getmail

# decide where you want you mail to live, and create an empty file there
touch ~/Dropbox/backup/gmail.mbox

mkdir ~/.getmail
vim ~/.getmail/getmailrc

# add the following
[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
username = YOURGMAILADDRESS
password = YOUGMAILPASSWORD
mailboxes = ("[Gmail]/All Mail",)

[destination]
type = Mboxrd
path = ~/Dropbox/backup/gmail.mbox

[options]
read_all = False # just get the new messages on subsequent runs
message_log_syslog = True

You can test whether your configuration working by running getmail manually from the command line. The initial download can take a while to begin if you have thousands of messages. The actual download will likewise take a long time for large mailboxes. While it's in progress, you will get a series of log messages on the console (and in syslog).

getmail

... several minutes later ...

SimpleIMAPSSLRetriever:chase.seibert@gmail.com@imap.gmail.com:993:
  msg      1/105702 (325 bytes) delivered
  msg      2/105702 (335 bytes) delivered
  msg      3/105702 (363 bytes) delivered
  msg      4/105702 (180479 bytes) delivered
  msg      5/105702 (311 bytes) delivered
  msg      6/105702 (1057 bytes) delivered
  msg      7/105702 (818 bytes) delivered
  msg      8/105702 (331 bytes) delivered
  msg      9/105702 (321 bytes) delivered
  msg     10/105702 (882 bytes) delivered
  msg     11/105702 (1201 bytes) delivered
  msg     12/105702 (1614 bytes) delivered
  msg     13/105702 (359 bytes) delivered
  msg     14/105702 (359 bytes) delivered
  ...

I choose to configure getmail to use a single mbox file, but you can also use one file per message. In addition to this large file, getmail will keep a smaller index file in ~/.getmail.

If you want, you can setup a cron job to update your backup nightly.

crontab -e

# add the following line, which will run the backup at 1AM every day
* 1 * * * /usr/bin/getmail >/dev/null 2>&1

Nagios check for Varnish Backends

2011-10-28T14:56:00.002-04:00

We recently starting using Varnish to cache un-authenticated requests to our web farm. It even has this great feature called grace mode, where it will keep serving a cached version of a page if the back-end server goes down. But we still want to be alerted that the back-end is down.

I wrote a Nagios check to do just that. It's written in Python, and using the varnishadm command-line tool that ships with Varnish.

#!/usr/bin/python
# save as /usr/lib/nagios/plugins/check_varnish_backends.py

from optparse import OptionParser
import subprocess

def getOptions():
    arguments = OptionParser()
    arguments.add_option("--host", dest="host", help="Host varnishadm is running on", type="string", default="localhost")
    arguments.add_option("--port", dest="port", help="varnishadm port", type="string", default="6082")
    arguments.add_option("--secret", dest="secret", help="varnishadm secret file", type="string", default="/etc/varnish/secret")
    arguments.add_option("--command", dest="command", help="varnishadm backend health command", type="string", default="debug.health")
    return arguments.parse_args()[0]

def run(command, exit_on_fail=True):
    # don't use check_output in order to supportPython 2.6
    process = subprocess.Popen(command.split(" "), stdout=subprocess.PIPE)
    output, unused_err = process.communicate()
    _retcode = process.poll()
    return output

if __name__ == '__main__':

    options = getOptions()
    varnishadm_raw = run("varnishadm -T %(host)s:%(port)s -S %(secret)s %(command)s" % options.__dict__)

    lines = varnishadm_raw.split("\n")
    backends_sick, backends_healthy = [], []
    for line in lines:
        if line.startswith("Backend"):
            if line.endswith("Sick"):
                backends_sick.append(line)
            else:
                backends_healthy.append(line)

    if not backends_healthy and not backends_sick:
        print "There are NO backends"
        exit(2)

    if backends_sick:
        print "".join(backends_sick)
        exit(2)

    print "All %s backends are healthy" % len(backends_healthy)

Because varnishadm uses a shared secret file, I decided to have the checks run on the actual Varnish hosts, using Nagios NRPE.

# add the following config line on the varnish hosts
vim /etc/nagios/nrpe.cfg
command[check_varnish_backends]=/usr/lib/nagios/plugins/check_varnish_backends.py

service nagios-nrpe-server restart

# and configure the Nagios server to look for that check
vim /etc/nagios3/conf.d/services_nagios2.cfg
define service {
        use                             generic-service
        hostgroup_name                  proxy-servers
        service_description             check_varnish_backends
        check_command                   check_nrpe_1arg!check_varnish_backends
}

service nagios3 restart

Note: depending on your setup, you may need to use chmod to give the nagios user access to ready the shared secret file (/etc/varnish/secret) on the Varnish servers.

Reduce javascript UI code with Django forms

2011-10-21T15:43:00.002-04:00

Typically, I try to implement a dynamic UI feature without javascript on the first pass. Then, I layer in a little javascript goodness to make it more responsive. Whenever I stray from this practice, I inevitably end up re-discovering why I started doing this in the first place. At some point, I become horrified at the amount of javascript I have written, and disheartened at how long it's taking to modify. In short, things have gotten complicated.

Everything should be made as simple as possible, but no simpler. - Albert Einstein

Here is an example from just the other day. The requirements specified a form behavior that we didn't have already on the site. On a list of job records, there should be a close job link. Clicking the link reveals a new form that must be filled out successfully before the job is actually closed. When the form is successfully posted, hide the form and grey out the row. Here is the workflow:

Clicking close job actually just reveals the real form.

Attempting to post that form may throw validation exceptions.

When the form is successfully posted, it goes away and the row gets grayed out.

Of course, this is not an overly complicated example, but there are a few moving pieces. Nevertheless, I ended up with almost 100 lines of javascript code, even with the validation logic for the inner form being server side.

$(document).ready(function() {

 // hitting the initial close link, GETs the form from the back-end and shows it
 $(".jobs-record-list a.toggle-job-close").click(function (){

  var link = $(this);
  var record = link.parents(".record");
  var form_box = record.find(".job-close-form-box");
  var job_link = record.find(".title a");

  $.ajax({
   url: link.attr("href") + "?ajax",
   type: "GET",
   success: function (data) {
    form_box.html(data);
    bind_job_close_form(form_box);
   },
   error: function (jqXHR, textStatus, errorThrown) {
    // fallback to closing the job from the job overview page
    window.location = form.attr(job_link.attr("href"));
   }
  });

  link.remove();
  return false;
 });

 // process the inner form submit to actually close the job, then put the open link back on the page
 function bind_job_close_form(form_box) {

  var form = form_box.find("form");
  var parentRow = form.parents(".record");

  form.submit(function () {

   $.ajax({
    url: form.attr("action") + "?ajax",
    type: "POST",
    data: form.serialize(),
    success: function (data) {
     form_box.hide();
     parentRow.find(".content").toggleClass("record-disabled");
     parentRow.find(".block").toggleClass("record-disabled");
     var link = form.find("a");
     link.text(link.text() == "close job" ? "open job" : "close job");
    },
    error: function (jqXHR, textStatus, errorThrown) {
     form_box.html(jqXHR.response);
     bind_job_close_form(form_box); // rebind event handler
    }
   });

   return false;
  });
 }

        // if the page is refresh after the inner form is shown, re-show it
 $("form.toggle-job-open-closed").submit(function () {
  var form = $(this);
  var questions = form.find(".questions");
  if (questions.is(":hidden")) {
   questions.show();
   return false;
  }
  return true;
 });

 // re-opening a job, POST to the server then gray out the row
 $("a.toggle-job-open").submit(function () {

  var form = $(this);
  var parentRow = form.parents(".record");

  $.ajax({
   url: form.attr("action") + "?ajax",
   data: form.serialize(),
   type: form.attr("method"),
   success: function (data) {
    parentRow.find(".content").toggleClass("record-disabled");
    parentRow.find(".block").toggleClass("record-disabled");    
   }
  });

  return false;
 });
});

At this stage, my form is rather simple, but I have a view that's trying to do a lot; processing both the Ajax and non-javascript versions of requests this page. I'm also doing some craziness to pass the current form around in the session scope. This is so that if the user refreshes, the form is still in the same state. Here is just the form:

class OpenCloseJobForm(Html5Form):

    candidate = forms.ChoiceField(required=True)

    def __init__(self, *args, **kwargs):

        kwargs = helpers.copy_to_self(self, "job", kwargs)
        super(OpenCloseJobForm, self).__init__(*args, **kwargs)

        self.fields["candidate"].label = "Who did you hire for this job?"
        self.fields["candidate"].required = False
        self.fields["candidate"].choices = [("", "-------------")]
        candidates = Candidate.objects.filter(job=self.job)
        self.fields["candidate"].choices += [("Candidates", [(c.id, c) for c in candidates] + [("other", "Someone else" if candidates else "Candidate not in %s" % settings.SITE_NAME)])]
        self.fields["candidate"].choices += [("none", "No hire made")]

    def clean_candidate(self):
        candidate = self.cleaned_data.get("candidate")
        if not candidate:
            raise forms.ValidationError("This field is required.")
        return candidate

    def save(self):

        choice = self.data.get("candidate")

        if not choice:
            return None

        Hire.objects.filter(job=self.job).delete()

        if choice == "none":
            return None

        candidate_id = choice
        candidate = None if choice == "other" else Candidate.objects.get(id=candidate_id)

        hire = Hire(job=self.job, candidate=candidate)
        hire.save()
        return hire

At this point, I decide to refactor to move as much of the UI logic as possible into the form itself, and reduce the amount of javsacript. I end up with a widget and a form that implements the show/hide inner form behavior. That part grows from 40 lines to 90 lines.

class ToggleLinkWidget(widgets.HiddenInput):

    button_value = None

    def __init__(self, attrs=None, check_test=bool, button_value="toggle"):
        super(ToggleLinkWidget, self).__init__(attrs)
        self.button_value = button_value

    def render(self, name, value, attrs=None):
        html = super(ToggleLinkWidget, self).render(name, value, attrs)
        link = ""
        if not value:
            button_value = self.button_value
            return "<input class='button-link' type='submit' name='%(name)s' value='%(button_value)s'>" % locals()
        return html

class OpenCloseJobForm(Html5Form):

    job_id = fields.CharField(required=True, widget=widgets.HiddenInput())
    expanded = fields.BooleanField(required=False, initial=False, widget=ToggleLinkWidget(button_value="close"))
    candidate = fields.ChoiceField(required=True)

    def __init__(self, *args, **kwargs):

        super(OpenCloseJobForm, self).__init__(*args, **kwargs)
        self.job = Job.objects.get(id=self.data.get("job_id"))

        self.fields["candidate"].label = "Who did you hire for this job?"
        self.fields["candidate"].required = False
        self.fields["candidate"].choices = [("", "-------------")]
        candidates = Candidate.objects.filter(job=self.job)
        self.fields["candidate"].choices += [("Candidates", [(c.id, c) for c in candidates] + [("other", "Someone else" if candidates else "Candidate not in %s" % settings.SITE_NAME)])]
        self.fields["candidate"].choices += [("none", "No hire made")]

        # the close button is NOT toggled on; hide thother fields
        if not self.is_expanded():
            del self.fields["candidate"]

        # don't show validation errors when first expanding the form
        if not self.data.get("inner-submit"):
            self._errors = {}

    def is_valid(self):
        if self.data.get("inner-submit"):
            return super(OpenCloseJobForm, self).is_valid()
        return False

    def is_expanded(self):
        return self.data and self.data.get("expanded") == "close"

    def as_ul(self):
        html = super(OpenCloseJobForm, self).as_ul()
        if self.is_expanded():
            button = "<input type='submit' name='inner-submit' value='Close Job'>"
            return mark_safe("<div class='expanded-box'>%(html)s %(button)s</div>" % locals())
        return html

    def clean_candidate(self):
        candidate = self.cleaned_data.get("candidate")
        if not candidate:
            raise forms.ValidationError("This field is required.")
        return candidate

    def save(self, request):

        self.job.is_open = not self.job.is_open

        if self.job.is_open:
            Hire.objects.filter(job=self.job).delete()
            messages.success(request, "%s was set to open and is publicly available." % self.job)
        else:
            messages.success(request, "%s was closed and is no longer available publicly." % self.job)

        self.job.save()

        choice = self.data.get("candidate")

        if not choice:
            return None

        Hire.objects.filter(job=self.job).delete()

        if choice == "none":
            return None

        candidate_id = choice
        candidate = None if choice == "other" else Candidate.objects.get(id=candidate_id)

        hire = Hire(job=self.job, candidate=candidate)
        hire.save()
        return hire

But look at the javascript, which has shrunk from 90 lines to 40.

$(document).ready(function() {

 function submit_handler() {

  var button = $(this);
  var record = button.parents(".record");
  var form = record.find(".job-prompt-hire-form");
  var form_box = form.parents(".form-box");
  var job_link = record.find(".title a");

  var form_data = form.serialize();
  // also record which input button was pressed
  if (button.is(".button-link")) {
   form_data = form_data += "&expanded=close";
  } else {
   form_data = form_data += "&inner-submit=Close Job";
  }

  $.ajax({
   url: form.attr("action") + "?ajax",
   data: form_data,
   type: form.attr("method"),
   success: function (data) {
    form_box.html(data);
    if (!data) {
     record.find(".content").toggleClass("record-disabled");
     record.find(".block").toggleClass("record-disabled");
    }
   },
   error: function (jqXHR, textStatus, errorThrown) {
    // fallback to closing the job from the job overview page
    window.location = form.attr(job_link.attr("href"));
   }
  });

  return false;
 }

 // jQuery does not support live submit handling in IE
 // also, need to record actual pressed element
 $(".jobs-record-list .job-prompt-hire-form .button-link").live("click", submit_handler);
 $(".jobs-record-list .job-prompt-hire-form input[type=submit]").live("click", submit_handler);

});

Interestingly, the total line count has remained around the same. This suggests that for my style, the code density of python and jQuery is about the same. Personally, I find the python code to be more maintainable. In particular, I very much like that I'm minimizing the number of times jQuery is changing the DOM, and rebinding events. All of the state transitions are happening in the form. As a side effect, the show/hide behavior is totally functional without javascript enabled.

Profile and reduce memory use in Django with .iterator()

2011-10-12T10:12:00.000-04:00

For the most part, objects allocated by Django are short-lived, and are eligible for garbage collection when the request ends. In our case, we also have some long running jobs using celery. One in particular, a job to create a several hundred megabyte XML file, was consistently using all the RAM on the machine.

This wasn't too surprising because we were initially using Django templating to create the file, which keeps the entire response in memory while it's still being composed. But even after we moved to a SAX parser, which is specifically designed for running with little memory by streaming the file, we were still running out of memory occasionally.

We decided it was time to stop guessing, and profile the memory usage. Never having done this before, I did some research and came up with this excellent guide to using pdb for memory profiling.

To get started, I needed something I could run from the command line, outside of the celery task framework. Using the Django Command framework, I was easily able to compose a job that could run via manage.py.

# in file myapp/management/commands/xmlmemtest.py
from django.core.management.base import BaseCommand
from myapp.helpers.scheduler import write_job_board_feed
import uuid

class Command(BaseCommand):    
    def handle(self, *args, **options):
        write_job_board_feed("simplyhired", "", "justjobs", nocache=uuid.uuid4())

# can be run via: ./manage.py xmlmemtest

With that, I was set to launch this process using pdb, the Python debugger.

chase@chase:~$ pdb ./manage.py xmlmemtest 
-> from django.core.management import execute_manager
(Pdb) r

#... wait for a while until memory is getting high (will be much slower than usual) ...

# pause execution
<Ctrl+C>

# evoke the garbage collector manually to make sure you're only seeing referenced objects
(Pdb) import gc
(Pdb) gc.collect()
58
(Pdb) gc.collect()
0

# show the top items in memory
(Pdb) import objgraph
(Pdb) objgraph.show_most_common_types(limit=5)
Job                        184791
builtin_function_or_method 57542
tuple                      55478
list                       14900
dict                       8631

I was excepting to see some SAX parser objects at the top of the list. Instead, most of the memory was tied up in Job objects, which are a Django model in my application. Looking at the function, jobs were first referenced in the following code block.

    for job in jobs:
        feed.write_entry(2, job)

Playing around a bit, I tried the following, which immediately solved the memory issue.

    for _job in jobs:
        job = Job.objects.get(id=_job.id)
        feed.write_entry(2, job)

Basically, doing a separate query for each job, and making sure it goes out of scope after we're done using it. In retrospect, this was pretty obvious. So obvious, that Django even provides a handy idiom called .iterator() to do just this.

Evaluates the QuerySet (by performing the query) and returns an iterator (see PEP 234) over the results. A QuerySet typically caches its results internally so that repeated evaluations do not result in additional queries. In contrast, iterator() will read results directly, without doing any caching at the QuerySet level (internally, the default iterator calls iterator() and caches the return value). For a QuerySet which returns a large number of objects that you only need to access once, this can results in better performance and a significant reduction in memory.

Django Performance: Latency Kills

2011-10-07T15:07:00.002-04:00

Do you know how many database queries your Django pages are making? If you're like me, you probably don't know exactly. That's the beauty of the ORM; it abstracts database access to such a level that you rarely have to think about it. But do yourself a favor, and invest a few minutes in getting django-debug-toolbar up and running. Among other things, it will tell you exactly what queries and running on any given page.

Here is some example debug output for a simple record list in my current application.

Note that the page made 70 queries. The first three, shown on the left, averaged just under one millisecond each. In fact, if I scroll down, I can see that no single query is taking over two milliseconds. In fact, all 70 of them take only 50 milliseconds. Database time is dwarfed by the total CPU time, although with the debug toolbar enabled, CPU time tends to be much higher than normal.

Where are all these queries coming from? Some are done automatically by the framework, such as the session lookup or model permissions checks. Some are done as soon as you access a property on a model, such as request.user. But most of them you're doing yourself, via statements such as Model.objects.filter(...), etc.

They pile up fast. In my experience, it's not unusual for relatively simple pages to get into the 50 to 100 query range. I have some report pages that are doing over 1,000 queries in under 300 milliseconds!

Compared to some other frameworks, especially those without an ORM, this may seem excessive. I remember writing web pages in JSP, PHP and ColdFusion with far fewer queries. Maybe 10 or 15 on a complicated page. However, those were written by hand. That's the key. When you're writing SQL by hand, the developer has a natural tendency to write fewer, but larger, queries. In Java, every query you write is wrapped in several lines of boilerplate code; hardly something you want to do too often.

On balance, I think we're better off with more, smaller, queries. It's certainly better for database performance. I remember writing literally 1000+ line queries in ColdFusion that would take hundreds of milliseconds to run (if you were lucky), maybe even locking the table for writes all the while. Sure, Django may be running 70 queries, but they are all lightning quick. They are also easier to maintain. I think we can all agree that hand-written SQL is harder to maintain (and no fun, anyway).

But the hidden killer here is server to server latency. We're used to thinking about latency between clients and servers. Over the internet, seeing a latency over 100 millisecond is not unusual. In fact, even at the speed of light, your packet cannot even theoretically get from the east coast of the united states to the west coast in under 40 milliseconds. Factor in sub-optimal routing, and you're easily in the hundreds of milliseconds.

But inside your hosting environment, server to server latency should be much lower. Amazon EC2 hovers around .5 milliseconds. Yes, half of one millisecond. My experience on Slicehost/Rackspace has been similar.

However, what would happen if that latency spiked to "just" 10 milliseconds? Your 100 query page just jumped from 50 milliseconds of latency overhead to one full second. The 1,000 query page? That just went from half a second to 10 full seconds! That kind of slowness can obliterate any other performance work you're doing.

The good news is that sub-millisecond latency is totally standard. However, you would be smart to setup some kind of server to server latency monitoring. We use Nagios with the built-in PING plug-in, set to alert us if it jumps over 10 milliseconds consistently. This can lead to some entertaining conversations with tech support. "Honestly, we do consider 18 msec ping times to be within standard operating parameters."

If anyone tries to tell you that, feel free to send them to this blog post.

Using pgpool2 to timeout idle Postgres connections from Django

2011-09-30T13:24:00.000-04:00

Django has a well known issue with "<IDLE> in transaction" queries piling up. I have found that simply "<IDLE>" queries also have a tendency to hang out. This can cause problems for a high traffic site, as you can run out of open connections slots on the database side.

To see if you're affected, you can run a database query to see the status of open connections.

>select count(*), current_query from pg_stat_activity group by current_query;
 count |                                current_query                                 
-------+------------------------------------------------------------------------------
    31 | <IDLE> in transaction
    54 | <IDLE>
     1 | select count(*), current_query from pg_stat_activity group by current_query;
(3 rows)

>SHOW max_connections;
125

In this case, you can see that 86 out of a possible 125 connections are open, and there is only one "real" query running. After a while, you might start seeing the OperationalError: FATAL: sorry, too many clients already error from Postgres, by way of Django.

Why are these connections hanging out? Maybe you're opening a transaction and not closing it. In my case, I have no idea. But I don't really care either, I just want them closed after say 90 seconds. Surprisingly, there is no configuration option in Django or Postgres for this. Instead, you have to rely on a Postgres add-on like pgpool2.

Here are my notes for installing pgpool2 on Ubuntu, fixing a bug in the Ubuntu 11.04 init script and configuring a 90 second connection timeout. In this example, I'm installing pgpool2 on the Postgres database server itself, and having pgpool2 take over post 5432, moving Postgres itself to 5431 (to make it a seamless transition as far as Django config is concerned).

apt-get install pgpool2 

vim /etc/init.d/pgpool2 
PIDFILE=/var/run/pgpool/pgpool.pid 

vim /etc/pgpool.conf 
port = 5432 
connection_life_time = 90 
client_idle_limit = 90 
backend_hostname0 = 'localhost' 
backend_port0 = 5431 
backend_weight0 = 1 

vim /etc/postgresql/8.4/main/postgresql.conf 
port = 5431 

/etc/init.d/postgresql-8.4 stop 
/etc/init.d/postgresql-8.4 start 
service pgpool2 restart

Varnish caching for unauthenticated Django views

2011-09-23T13:14:00.001-04:00

Varnish is an HTTP accelerator designed for content-heavy dynamic web sites. In contrast to other HTTP accelerators, such as Squid, which began life as a client-side cache, or Apache, which is primarily an origin server, Varnish was designed from the ground up as an HTTP accelerator. -- Wikipedia

Varnish is pretty awesome. Typical usage would be to stick a caching layer in front of a slow CMS like Drupal, and tell it cache all page requests for an hour. You can serve thousands of hits to the homepage over the course of that hour for just one back-end call. Not only are you taking a lot of load off the backend, but the actual responses will be lightning quick. By default, Varnish doesn't cache pages with cookies set, though.

A lot of people running Varnish are not aware of one simple fact. If the client sends you a cookie Varnish will not issue a cached page but instead fetch a new page from the backend servers. Varnish takes a cookie as a good indication that the content is specially adapted and will, with the default configuration, not cache it. So, having a cookie that is not handled correctly will completely kill your performance. -- Varnish Blog

This makes complete sense. Most web frameworks use cookie based sessions. If you didn't look at the cookies to determine whether to cache, you would be sticking the private dynamic content for the first user to come along into the cache, and then serving it up for other users. If Facebook did that, you would login and see another users feed.

To use Varnish with Django, you just have to be careful about when you're using the request.session scope. If you put anything in there, even if the user is not logged in, the user will get a sessionid cookie, and Varnish will stop caching. Similarly, if you show the user a form with CSRF protection enabled (which it is by default), they will get a csrftoken cookie.

In general, those are the only two cookie Django will set, and either of those constitutes a page that should not be cached. In addition, you may have other cookies, such as Google Analytics. One solution is to strip these out in vcl_recv(), before they get passed to the back-end. Note: this does not mean that the client will no longer have access to them. As far as the client is concerned, the cookie is still set on your site. You're just not passing to to the webservers.

sub vcl_recv {

  # unless sessionid/csrftoken is in the request, don't pass ANY cookies (referral_source, utm, etc)
  if (req.request == "GET" && (req.url ~ "^/static" || (req.http.cookie !~ "sessionid" && req.http.cookie !~ "csrftoken"))) {
    remove req.http.Cookie;
  }

  # normalize accept-encoding to account for different browsers
  # see: https://www.varnish-cache.org/trac/wiki/VCLExampleNormalizeAcceptEncoding
  if (req.http.Accept-Encoding) {
    if (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unkown algorithm
      remove req.http.Accept-Encoding;
    }
  }

}

sub vcl_fetch {

  # static files always cached
  if (req.url ~ "^/static") {
       unset beresp.http.set-cookie;
       return (deliver);
  }

  # pass through for anything with a session/csrftoken set
  if (beresp.http.set-cookie ~ "sessionid" || beresp.http.set-cookie ~ "csrftoken") {
    return (pass);
  } else {
    return (deliver);
  }

}

This config is also accounting for differences in Accept-Encoding between various browsers and versions. Otherwise, you will end up with one copy of each page per unique Accept-Encoding value. Also, I'm making sure to always cache anything under /static, which is where all my media files are located.

Django/Haystack: latitude/longitude radius search w/ SOLR

2011-09-16T14:04:00.001-04:00

Haystack is a great indexed search framework for Django. Getting started is easy, and it includes many data types and facets out of the box. However, one data type it does not do natively is location based search. Specifically, I wanted to do radius searching based on latitude and longitude. GIS search is coming, but it could be a little while:

There aren't words to express how badly I want to incorporate geospatial search. However, I'm waiting on Solr 1.5's official implementation, as well as seeing if the Xapian folks land their GIS branch. I haven't pursued the third-party options because it makes the setup more complex and there's no guarantees on compatibility, which increase my support headaches. - Daniel, Haystack Google Group

In the meantime, you can use one of several SOLR extensions, or you can just do it yourself. Here is a very basic, but functional, implementation.

First, I introduce a custom index field "GeoPointField" on my SearchIndex model, mapped to some existing fields in my Django ORM that store latitude and longitude as float values.

class GeoPointField(indexes.CharField):

    def __init__(self, **kwargs):
        kwargs["default"] = "000000000000000"
        super(GeoPointField, self).__init__(**kwargs)

    def convert(self, value):
        return geo2solr(value)

class JobIndex(indexes.SearchIndex):

    text = indexes.CharField(document=True, use_template=True, template_name="search/index/job_text.txt")
    latitude = GeoPointField(model_attr="location__latitude", null=True)
    longitude = GeoPointField(model_attr="location__longitude", null=True)

    def index_queryset(self):
        return Job.objects.all()

site.register(Job, JobIndex)

What's the deal with that default of "000000000000000"? Because everything is a string in SOLR, I have decided to encode lat/long as a string between "000000000000000" and "999999999999999". The actual "geo2solr" algorithm for the mapping can be arbitrary, as long as the outputs maintain relative comparability when compared as strings. i.e., a > b implies that geo2solr(a) > geo2solr(b).

def geo2solr(lat_or_long):
    """ Converts a floating point latitude or longitude to a string for the SOLR index.
    The string representations need to be str-comparable to each other. Ie,
    04235863500 < 05000000000

    Negative values are handled by adding 180 to everything (Longitude is +/- 180)

    >>> geo2solr(42.35863500)
    '000022235863500'

    >>> geo2solr(-42.35863500)
    '000013764136500'

    >>> geo2solr('42.35863500')
    '000022235863500'

    >>> geo2solr('-179.35863500')
    '000000064136500'

    >>> geo2solr(0)
    "000018000000000"

    >>> geo2solr(None)

    >>> geo2solr("foobar")

    >>> geo2solr(50000)

    """
    try:
        value = float(lat_or_long)
        if -180 <= abs(value) <= 180:
            return str(int((value + 180) * 100000000)).rjust(15, "0")
    except TypeError:
        pass
    except ValueError:
        pass
    return None

To do the actual radius search part requires a way to calculate the radius range in my geo-coded units, and a SearchForm/FacetedSearchForm to perform that search.

def geo_radius(lat, long, miles=50):
    """ Given a latitude and longitude (as floats), returns two offsets, one
    for latitude and one for longitude, that defines a radius (in miles)
    around the location.

    Uses Haversine formula: http://en.wikipedia.org/wiki/Haversine_formula
    http://blog.fedecarg.com/2009/02/08/geo-proximity-search-the-haversine-equation/

    Note: long is not actually used. This is correct. Longitude distance is also
    based on latitude in Haversine.

    >>> geo_radius(53.754842, -2.708077)
    (0.7246376811594203, 1.2256204746052668)

    """
    return Decimal(miles / 69.0), Decimal(miles / abs(math.cos(math.radians(lat)) * 69.0))

class LocationRadiusSearchForm(FacetedSearchForm):

    q = forms.CharField(required=False)
    location = ChoiceField(required=False)

    def search(self):
        sqs = super(LocationRadiusSearchForm, self).search()
        try:
            location_name = self.cleaned_data.get("location")
            location = Location.objects.get(name=location_name)
            if location:
                lat_offset, long_offset = geo_radius(location.latitude, location.longitude)
                sqs = sqs.filter_and(
                    latitude__range=[
                        geo2solr(location.latitude - lat_offset),
                        geo2solr(location.latitude + lat_offset)],
                    longitude__range=[
                        geo2solr(location.longitude - long_offset),
                        geo2solr(location.longitude + long_offset)])
        except Exception:
            pass

        return sqs

That's it! This will generated SOLR queries like "longitude:[000005666355690 TO 000005849715509] AND latitude:[000021705248731 TO 000021850176268]"

Celery: blacklists and custom formatting for exception emails

2011-09-09T16:59:00.000-04:00

Celery 2.3 has a few high level knobs to turn with regards to exception emails. You can whitelist exceptions by type. You can change the recipients, email servers to use, etc. But there are many powerful things that you couldn't do, until now.

Starting with the next release, you will be able to define a custom error mail handler at the task level. From there, you can do lots of new things. You could blacklist exceptions per-task. You can change the subject of body formatting of the errors. You can hijack the email and submit the content to github as a bug, if you want.

The mechanism for all this is the ErrorMail class, which is defined in utils.mail. Every task has a Task.ErrorMail property which designates a handler for its exception emails. By setting that property to a new class that inherits from ErrorMail, you can over-ride some or all of its methods, such as should_send(), format_subject(), format_body() and send().

class ErrorMail(object):

    # pep8.py borks on a inline signature separator and
    # says "trailing whitespace" ;)
    EMAIL_SIGNATURE_SEP = "-- "

    #: Format string used to generate error email subjects.
    subject = """\
        [celery@%(hostname)s] Error: Task %(name)s (%(id)s): %(exc)s
    """

    #: Format string used to generate error email content.
    body = """
Task %%(name)s with id %%(id)s raised exception:\n%%(exc)r


Task was called with args: %%(args)s kwargs: %%(kwargs)s.

The contents of the full traceback was:

%%(traceback)s

%(EMAIL_SIGNATURE_SEP)s
Just to let you know,
celeryd at %%(hostname)s.
""" % {"EMAIL_SIGNATURE_SEP": EMAIL_SIGNATURE_SEP}

    error_whitelist = None

    def __init__(self, task, **kwargs):
        #subject=None, body=None, error_whitelist=None
        self.task = task
        self.email_subject = kwargs.get("subject", self.subject)
        self.email_body = kwargs.get("body", self.body)
        self.error_whitelist = getattr(task, "error_whitelist")

    def should_send(self, context, exc):
        allow_classes = tuple(map(get_symbol_by_name,  self.error_whitelist))
        return not self.error_whitelist or isinstance(exc, allow_classes)

    def format_subject(self, context):
        return self.subject.strip() % context

    def format_body(self, context):
        return self.body.strip() % context

    def send(self, context, exc, fail_silently=True):
        if self.should_send(context, exc):
            self.task.app.mail_admins(self.format_subject(context),
                                      self.format_body(context),
                                      fail_silently=fail_silently)

Here is one example that formats the subject of the email with just the task name, and doesn't send the email at all for a certain set of exceptions.

class NaiveAuthenticateServer(Task):
    ErrorMail = MyErrorMail
    ...

class MyErrorMail(ErrorMail):

    def format_subject(self, context):
        # options: hostname, id, name, exc, traceback, args, kwargs
        return "[celery] %(name)s" % context

    def should_send(self, context, exc):
        return not isinstance(exc, (ZeroDivisionError, TypeError))

Django: sticky URL query parameters per view

2011-09-02T15:58:00.000-04:00

On pages that contain filter controls, such as search results pages, it's common for the filter selections to be put into the URL using query parameters. This makes the search results bookmark-able. A common requirement on such pages is to have the application remember the particular filter parameters the user set the last time they viewed the page. They should be able to browse away from the page, come back, and pick up where they left off.

In Django, it's reasonable to store those filter settings in the user's session, which will persist as long as they are logged in. You could easily code this up inside a particular view. But if this is something you do a lot, it makes more sense to extract the logic into a view decorator. Here is a generic decorator you can apply to any view to have a subset of the query parameters "stick" between renders.

from django.http import HttpResponseRedirect
import urlparse
import urllib

def remember_last_query_params(url_name, query_params):

    """Stores the specified list of query params from the last time this user
    looked at this URL (by url_name). Stores the last values in the session.
    If the view is subsequently rendered w/o specifying ANY of the query params,
    it will redirect to the same URL with the last query params added to the URL.

    url_name is a unique identifier key for this view or view type if you want
    to group multiple views together in terms of shared history

    Example:

    @remember_last_query_params("jobs", ["category", "location"])
    def myview(request):
        pass

    """

    def is_query_params_specified(request, query_params):
        """ Are any of the query parameters we are interested in on this request URL?"""
        for current_param in request.GET:
            if current_param in query_params:
                return True
        return False

    def params_from_last_time(request, key_prefix, query_params):
        """ Gets a dictionary of JUST the params from the last render with values """
        params = {}
        for query_param in query_params:
            last_value = request.session.get(key_prefix + query_param)
            if last_value:
                params[query_param] = last_value
        return params

    def update_url(url, params):
        """ update an existing URL with or without paramters to include new parameters
        from http://stackoverflow.com/questions/2506379/add-params-to-given-url-in-python
        """
        if not params:
            return url
        if not url: # handle None
            url = ""
        url_parts = list(urlparse.urlparse(url))
        # http://docs.python.org/library/urlparse.html#urlparse.urlparse, part 4 == params
        query = dict(urlparse.parse_qsl(url_parts[4]))
        query.update(params)
        url_parts[4] = urllib.urlencode(query)
        return urlparse.urlunparse(url_parts)

    def do_decorator(view_func):

        def decorator(*args, **kwargs):

            request = args[0]

            key_prefix =  url_name + "_"

            if is_query_params_specified(request, query_params):
                for query_param in query_params:
                    request.session[key_prefix + query_param] = request.GET.get(query_param)

            else:
                last_params = params_from_last_time(request, key_prefix, query_params)
                if last_params and last_params != {}:
                    current_url = "%s?%s" % (request.META.get("PATH_INFO"), request.META.get("QUERY_STRING"))
                    new_url = update_url(current_url, last_params)
                    return HttpResponseRedirect(new_url)

            return view_func(*args, **kwargs)

        return decorator

    return do_decorator

Django: exclude some views from middleware

2011-08-31T13:43:00.006-04:00

In my Django applications, I tend to use custom middleware extensively for common tasks. I have middleware that logs page runtime, middleware that sets context that most views will end up needing anyway, and middleware that copies the HTTP_REFERRER header from an entry page into the session scope for use later in the session.

At some point, I inadvertently created a middleware class invalidated the browser cache for certain views. Typically, just wrapping a view in @cache_control(max_age=3600) is enough to have the browser cache that view for an hour. But if you do something innocuous like evaluate request.user.is_authenticated() in a middleware class, then Django will set the Vary: Cookie header, invalidating the cache.

In my case, what I really wanted was a decorator that I could attach to a view that would skip my custom middleware, like an exclude list. Of course, you could just attach your middleware explicitly to each view that needs it, but that's needless code repetition if a middleware should wrap almost all views. You could also change each of your middleware classes to exclude particular views by URL, but you might end up having to alter many different middleware classes with that logic.

As another option, you can use the following decorator/middleware pair to short-circuit the middleware execution of any view, for any middleware defined in your settings file AFTER this one.

""" Allows short-curcuiting of ALL remaining middleware by attaching the
@shortcircuitmiddleware decorator as the TOP LEVEL decorator of a view.

Example settings.py:

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',

    # THIS MIDDLEWARE
    'myapp.middleware.shortcircuit.ShortCircuitMiddleware',

    # SOME OTHER MIDDLE WARE YOU WANT TO SKIP SOMETIMES
    'myapp.middleware.package.MostOfTheTimeMiddleware',

    # MORE MIDDLEWARE YOU WANT TO SKIP SOMETIMES HERE
)

Example view to exclude from MostOfTheTimeMiddleware (and any subsequent):

@shortcircuitmiddleware
def myview(request):
    ...

"""

def shortcircuitmiddleware(f):
    """ view decorator, the sole purpose to is 'rename' the function
    '_shortcircuitmiddleware' """
    def _shortcircuitmiddleware(*args, **kwargs):
        return f(*args, **kwargs)
    return _shortcircuitmiddleware

class ShortCircuitMiddleware(object):
    """ Middleware; looks for a view function named '_shortcircuitmiddleware'
    and short-circuits. Relies on the fact that if you return an HttpResponse
    from a view, it will short-circuit other middleware, see:
    https://docs.djangoproject.com/en/dev/topics/http/middleware/#process-request
     """
    def process_view(self, request, view_func, view_args, view_kwargs):
        if view_func.func_name == "_shortcircuitmiddleware":
            return view_func(request, *view_args, **view_kwargs)
        return None

Easier source control merges by trimming trailing whitespace

2011-08-19T13:08:00.005-04:00

If you do frequent branch merges in any source control system, you have likely encountered merge conflicts. When two branches have legitimate differences, most often you have to resolve those differences by hand. But all too often, when resolving diffs, you end up looking at a lot of "phantom diffs", too. The most common phantom diff is the trailing whitespace difference.

Just like standardization of space/tab between developers, deciding as a team to not commit changes with trailing whitespace can make your merging easier. You could to this with a git hook, or you could just configure your editor to do it.

In Eclipse, this requires a plug-in called AnyEdit. Once the plug-in is installed, enabled this option is under Preferences -> General -> Editors -> AnyEdit Tools.

But what about all the code you already have? Ideally you would get all the developers on a unified branch, perhaps after a sprint ends, and update the whole codebase at once to remove trailing spaces. In Linux, you run the following from the root of your code directory. Note the extension list in the first part.

echo "*.py,*.js,*.css,*.html" |xargs -d, -I ext find . -type f -name ext -print0 |xargs -0 sed -i .bak -e "s/[[:space:]]*$//"

django-celery: blacklist errors by class

2011-08-12T13:46:00.008-04:00

Celery has a built-in mechanism for sending untrapped errors to admins via email. This is great for trouble-shooting. Sending the emails directly to developers is also a good way to make sure they actually get fixed. I know 1,000 emails in my inbox motivates me!

However, you may not want to receive all errors. For example, our celery instance routinely throws TimeLimitExceeded exceptions that can be safely ignored, at least on some tasks. TimeLimitExceeded is tough, too, because you cannot catch it; it's handled by the celery framework.

While django-celery provides a mechanism to supply a exception whitelist, that may not work for you. If you're like me, you don't know what errors you want to receive emails for; by definition you do not have the foresight to trap them ahead of time! In that case, what you really need is an exception BLACKLIST. While I have opened a ticket for that, in the meantime you can use the following decorator around your task as a solution.

import sys
import traceback
from django.core.mail import mail_admins
from django.utils.text import truncate_words
from django.template.defaultfilters import slugify
from celery.exceptions import SoftTimeLimitExceeded
from celery import task as _task

class email_errors(object):

    """ Wraps a celery task in a try/catch with custom error types, only
        sending admin alert emails on non-blacklisted exceptions.

        Can be attached as a decorator to a task, such as :

        @email_errors(error_blacklist=[MaybeEncodingError, ])
        @task
        def some_task():
            pass
    """
    
    default_error_blacklist = [SoftTimeLimitExceeded, ]
    
    def __init__(self, error_blacklist=[]):
        self.error_blacklist = error_blacklist
        self.error_blacklist.extend(self.default_error_blacklist)       
        
    def __call__(self, task):
        task.unsynchronized_run = task.run
        @wraps(task.unsynchronized_run)
        def wrapper(*args, **kwargs):        
        try:
            task.unsynchronized_run(*args, **kwargs)
        except Exception, e:
            if type(e) in self.error_blacklist:
                stacktrace = get_stacktrace(e)
                print "ERROR: task=%s\n%s" % (task.__name__, stacktrace)
            else:
                mail_exception(e, prefix="[celery]")
        task.run = wrapper
        return task

def get_stacktrace(error):
    return "".join(traceback.format_exception(type(error), error, sys.exc_traceback))
    

def mail_exception(error, prefix=None):
    """ Mails an exception w/ stacktrace to ADMINS, can pass a prefix like '[celery]' """
    try:
        stacktrace = get_stacktrace(error)
        print stacktrace             
        for subject in _error_subjects(error):
            try:
                if prefix:
                    subject = "%s %s" % (prefix, subject)
                mail_admins(subject, "%s\n%s" % (error, stacktrace))
                break
            except:
                pass
    except Exception, e:
        print e

def _error_subjects(error):
    """ returns a list of potential subjects for an error email, some may fail """
    error_str = str(error)
    return [truncate_words(error_str, 5), slugify(truncate_words(error_str, 5)), "mail_exception error"]

In this scenario, you are sending errors emails yourself. You will want to disable CELERY_SEND_TASK_ERROR_EMAILS. By disabling that setting, you will also not get the TimeLimitExceeded emails.

Django: white-label styling with URL prefixes

2011-08-05T17:05:00.006-04:00

Many websites have branding or "white label" functionality where the site can have custom styles per customer. A common use case with commercial products is to allow the customer to use their own logo and colors in place of the defaults, usually for an extra fee.

In web apps, customers will often want to customize either the subdomain of their branded version, or perhaps use a CNAME one on of their existing domains. Alternatively, you can also differentiate brandings based on a URL prefix, such as Google apps does with www.google.com/a/customer_name. The later method is the focus of this blog post.

In Django, you can take advantage of the template inheritance and URL mapping features built into the framework to do most of the heavy lifting for you. For styling, you can easily image inserting a new branding template between your existing pages and your base template.

You can switch between your regular base template and the branding template by simply using a variable for the extends declaration.

{% extends request.BASE_TEMPLATE %}

In that template, you can override any styling you need to. But what about the links in those templates? Plus, how do we set request.BASE_TEMPLATE w/o a bunch of duplicate code everywhere? Well, you can do that at the same time you're providing the URL mapping to your branding prefix.

# url handler for ANYthing under /sites/, slug is the per-customer branding prefix
url(r'^sites/(?P<slug>([^//]+))/.*$', dispatch),

def dispatch(request, slug):
     
      # {% url %} will now render urls w/ this branding prefix
      company_url_prefix = "/sites/%s" % slug    
      old_prefix = urlresolvers.get_script_prefix()
      urlresolvers.set_script_prefix(company_url_prefix)

      try:

     request.BASE_TEMPLATE = "branded.html"
     # you can also set any state you need to do the actual branding here
      
     rest_of_url = request.META.get("PATH_INFO").replace(company_url_prefix, "")
     view, args, kwargs = urlresolvers.resolve(rest_of_url)
     return view(request, **kwargs)
      finally:
            # set_script_prefix is thread specific, so you need to be careful to reset it
            urlresolvers.set_script_prefix(old_prefix)

The trick here is set_script_prefix, a little known Django utility method that will prefix any template call to the {% url %} tag with a custom string. So, if your page normally renders a link as /job/44, it will now render as /sites/XXX/job/44, effectively keeping the user in the same branding experience.

Python: calculate lighter/darker RGB colors

2011-07-29T13:56:00.003-04:00

Many times color palettes have lighter and darker variations of the same color. This may be used to convey relative importance, or for something as simple as a gradient. Usually the designer will specify both colors. However, if you have a site that needs to allow user configurable styling, you may not want to ask the user for two variations of the same color.

Here is some Python code to take a single color in RGB, and output an artitrarily lighter or darker variation of the same color. You could wrap this in a filter and use it right in your Django templates.

def color_variant(hex_color, brightness_offset=1):
    """ takes a color like #87c95f and produces a lighter or darker variant """
    if len(hex_color) != 7:
        raise Exception("Passed %s into color_variant(), needs to be in #87c95f format." % hex_color)
    rgb_hex = [hex_color[x:x+2] for x in [1, 3, 5]]
    new_rgb_int = [int(hex_value, 16) + brightness_offset for hex_value in rgb_hex]
    new_rgb_int = [min([255, max([0, i])]) for i in new_rgb_int] # make sure new values are between 0 and 255
    # hex() produces "0x88", we want just "88"
    return "#" + "".join([hex(i)[2:] for i in new_rgb_int])

Checking RabbitMQ queue size/age with Nagios

2011-07-01T16:11:00.004-04:00

For months we were using RabbitMQ (with celery) with no real insight into what was going on inside the queue. Recently, we deployed the mangement plug-in, which has a nifty web UI:

From there, is seemed like a logical thing to add a Nagios check for. Here is some Python code to do just that.

#!/usr/bin/python

from optparse import OptionParser
import urllib2
import json
import pprint
import time
import datetime

def getOptions():
    arguments = OptionParser()
    arguments.add_option("--host", dest="host", help="Host rabbitmq is running on", type="string", default="localhost")
    arguments.add_option("--queue", dest="queue", help="Name of the queue in inspect", type="string", default="celery")
    arguments.add_option("--username", dest="username", help="RabbitMQ API username", type="string", default="rabbitmq")
    arguments.add_option("--password", dest="password", help="RabbitMQ API password", type="string", default="rabbitmq")
    arguments.add_option("--port", dest="port", help="RabbitMQ API port", type="string", default="55672")
    
    arguments.add_option("--warning-queue-size", dest="warn_queue", help="Size of the queue to alert as warning", type="int", default=10000)
    arguments.add_option("--critical-queue-size", dest="crit_queue", help="Size of the queue to alert as critical", type="int", default=20000)

    arguments.add_option("--warning-seconds", dest="warn_seconds", help="Last event processes in seconds ago to alert as warning", type="int", default=3600)
    arguments.add_option("--critical-seconds", dest="crit_seconds", help="Last event processes in seconds ago to alert as critical", type="int", default=14400)
        
    return arguments.parse_args()[0]

if __name__ == '__main__':
    
    options = getOptions()    
    
    url = "http://%s:%s/api/queues/reach/%s" % (options.host, options.port, options.queue)
    
    # handle HTTP Auth
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
    top_level_url = url
    password_mgr.add_password(None, top_level_url, options.username, options.password)
    handler = urllib2.HTTPBasicAuthHandler(password_mgr)
    opener = urllib2.build_opener(handler)    
    
    response = None
    try:
        request = opener.open(url)
        response = request.read()
        request.close()
    except urllib2.HTTPError, e:
        print "Error code %s hitting %s" % (e.code, url)
        exit(1)
        
    data = json.loads(response)
    
    #pp = pprint.PrettyPrinter(indent=4)
    #pp.pprint(data)
    
    num_messages = data.get("messages")
    if num_messages > options.crit_queue or num_messages > options.warn_queue:
        print "%s messages in %s queue" % (num_messages, options.queue)
        exit(1 if num_messages > options.crit_queue else 2) 
    
    message_stats = data.get("message_stats")
    deliver_details = message_stats.get("deliver_details")
    rate = deliver_details.get("rate")
    
    #1309542487 
    #1309548601517    
    last_event = deliver_details.get("last_event") / 1000
    last_event_time = time.ctime(last_event)
    
    #diff = abs(last_event_time - datetime.datetime.today())
    #last_event_time_diff_seconds = diff.seconds + diff.days * 86400
    last_event_time_diff_seconds = abs(last_event - int(time.time()))
    if last_event_time_diff_seconds > options.crit_seconds or last_event_time_diff_seconds > options.warn_seconds:
        print "%s seconds since last event consumed on %s" % (last_event_time_diff_seconds, options.queue)
        exit(1 if last_event_time_diff_seconds > options.crit_seconds else 2)
    
    print "Last event consumed: %s" % last_event_time

You can run it manually:

check_rabbitmq --host sched1 --warning-queue-size 100 --critical-queue-size 400 --warning-seconds 600 --critical-seconds 2400

Or, you can configure Nagios to run it:

# vim /etc/nagios3/conf.d/services_nagios2.cfg
define command {
        command_name    check_rabbitmq
        command_line    /usr/lib/nagios/plugins/check_rabbitmq --host $HOSTADDRESS$ --queue $ARG1$ --warning-queue-size $ARG2$ --critical-queue-size $ARG3$ --warning-seconds $ARG4$ --critical-seconds $ARG5$
}

define service {
        host_name                       sched1
        service_description             QUEUE
        check_command                   check_rabbitmq!celery!100!400!600!2400
        contact_groups                  pager
        use                             generic-service
        notification_interval           0 ; set > 0 if you want to be renotified
}

Creating large XML files in Python with saxutils

2011-06-24T14:54:00.003-04:00

In the past, when I needed to produce an XML file problematically, I typically turned to templating engines. Something like the following example in Django templates:

<?xml version="1.0" encoding="utf-8"?>
<jobs>
    {% for job in jobs %}
        {% with job.user|get_profile as profile %}
      <job>
          <title>{{ job.title|xml_escape }}</title>
          <job-board-name>{{ 'JOB_FEED_SITE_NAME'|setting }}</job-board-name>
          <job-board-url>{{ 'SITE_URL_NO_SLASH'|setting }}</job-board-url>
          <job-code>job{{ job.id }}</job-code>
          <detail-url>{{ 'SITE_URL_NO_SLASH'|setting}}{% url job job.id job.slug %}{{ 'simplyhired.com'|campaign_tracking }}</detail-url>

...

You get the idea. Once you start including free text data from real users, you pretty quickly run into issues with invalid XML. Typically, you need some combination of escaping (like saxutils.escape) and stripping (for control character that are illegal in XML). For whatever reason, it's pretty common for the XML parsers themselves not to give you the tools you need to produce valid XML from imperfect data. You can counter this with a XML cleaning library such as 4Suite, or write your own.

This solution is inelegant, but it works. However, if you need to produce very large XML files, you will run into memory constraints. Unless your templating engine has the ability to stream the file contents to disk, which Django's doesn't, you'll be storing all that XML in memory before it can be written. This can lead to hours of XML composing followed by an out of memory exception, with nothing writing to disk.

Using an XML parser to actually compose XML is much more robust, even if the code is somewhat less straight forward. You could use a DOM parser, but that wouldn't solve your memory issue; the whole XML document must still be kept in memory before it can be written. Instead, you can use a SAX parser, which is explicitly designed for streaming.

Python ships with saxutils, which is very capable, if a bit verbose to use as an API. Here is a quick abstraction for simple nested tags without attributes, which is a common pattern.

from xml.sax.saxutils import XMLGenerator
from xml.sax.xmlreader import AttributesNSImpl
from website.helpers import xml_helper
import types
from django.utils.encoding import smart_str, force_unicode
from xml.sax import saxutils   
    
def strip_illegal_xml_characters(input):
    
    if input:
            
        import re
        
        # unicode invalid characters
        RE_XML_ILLEGAL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])' + \
                         u'|' + \
                         u'([%s-%s][^%s-%s])|([^%s-%s][%s-%s])|([%s-%s]$)|(^[%s-%s])' % \
                          (unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),
                           unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),
                           unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),
                           )
        input = re.sub(RE_XML_ILLEGAL, "", input)
                        
        # ascii control characters
        input = re.sub(r"[\x01-\x1F\x7F]", "", input)
            
    return input

def escape_xml(value):
    return strip_illegal_xml_characters(force_unicode(smart_str(value))) 

class SimpleSaxWriter():
    
    def __init__(self, output, encoding, top_level_tag=u"jobs"):
        logger = XMLGenerator(output, encoding)
        logger.startDocument()
        attrs = AttributesNSImpl({}, {})
        logger.startElementNS((None, top_level_tag), top_level_tag, attrs)        
        self._logger = logger
        self.top_level_tag = top_level_tag
        
    def start_tag(self, name):                
        attrs = AttributesNSImpl({}, {})
        self._logger.startElementNS((None, name), name, attrs)

    def end_tag(self, name):                
        self._logger.endElementNS((None, name), name)
        
    def simple_tag(self, name, contents):
        
        if contents:
            
            if not isinstance(contents, types.UnicodeType):
                raise TypeError("XML character data must be passed in as a unicode object")
                    
            self.start_tag(name)
            # saxutils will let invalid XML though, left to it's own devices
            self._logger.characters(escape_xml(contents))                                    
            self.end_tag(name)
                
    def write_entry(self, level, msg):
        raise NotImplementedError()

    def close(self):
        self._logger.endElementNS((None, self.top_level_tag), self.top_level_tag)
        self._logger.endDocument()
        return

Given that framework, an implementing class might look like:

class JobWriter(SimpleSaxWriter):        
    
    def write_entry(self, level, job):
        
        from website.helpers.tags import label
        from website import helpers
        
        self.start_tag("job")
        
        self.simple_tag("title", job.title)
        self.simple_tag("job-board-name", unicode(settings.JOB_FEED_SITE_NAME))

 ...
        
        self.start_tag("description")        
        self.simple_tag("summary", job.description_plaintext)
        self.end_tag("description")

 ...

To actually write to disk, you need to pass in a file. Notice that I'm using mode "wb", because saxutils will produce a byte stream, and that I'm NOT passing an encoding, even though I'm dealing with UTF-8 data. That's because Python will guess wrong in that case, and try to output ASCII.

    
    file = codecs.open("test.xml", mode="wb")
    feed = JobWriter(file, "UTF-8")
    for job in jobs: # just a list of objects from Django's orm, could be anything
        feed.write_entry(2, job)
    feed.close()
    file.close()

Django CSRF verification failed in IE IFRAME

2011-06-17T11:37:00.015-04:00

I ran into an interesting issue with the Django's CSRF (cross site request forgery) protection this week. Some users were reporting seeing the dreaded "CSRF verification failed. Request aborted." error trying to submit a particular form. Unable to reproduce the issue myself, we finally learned that this was happening when coming from LinkedIn.

Digging deeper, it turns out only to happen in IE (6/7/8/9), and only in an IFRAME, like the one LinkedIn is using for their framebar. After some research, it was revealed that IE6 added a security feature to block all third party cookies by default, which includes any pages in an IFRAME on a different domain than the top level parent. There are a few options for work-arounds.

Work around #1: Framebusting

You can use javascript to "break out" of the IFRAME, essentially replacing the top level window with yourself. There are various techniques, but I like the simple:

if(top != self) top.location.replace(location);

However, this won't work if you want to allow IFRAMEs in somecases, say from your own domain(s). The code could quickly become brittle if you were to hard-code a whitelist of development/demo/qa/prod domains, or weak if you hard-code a blacklist. Also, it's fairly easy to intentionally break a frame buster. In the end, if you get into an arms race with a framer, you will lose.

Work around #2: x-frame-options

Newer browsers actually have a mechanism to dis-allow framing your site all together, via the x-frame-options HTTP header. This is great, but only works on newish browsers, and is somewhat coarsely grained. You can only make an exception for same domain IFRAMEs, not arbitrary domains. Also, the user experience kind of blows; it looks like an error to the end-user.

Work around #3: P3P

You can send a HTTP header to tell the browser to allow third-party cookies in this instance. This uses the P3P standard. However, it should be noted that you're essentially making legally binding claims about how you handle user data. For example, the smallest change you can make that will notify IE to allow third-party by default is:

vim /etc/apache2/sites-available/default

# add the following
Header append P3P "CP=\"CAO PSA OUR\""

ln -s /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled/headers.load
service apache2 restart

Here is a quick rundown of what legal claims you're making in this case.

CAO - Identified Contact Information and Other Identified Data: access is given to identified online and physical contact information as well as to certain other identified data.
PSA - Information may be used to create or build a record of a particular individual or computer that is tied to a pseudonymous identifier, without tying identified data.
OUR - Ourselves and/or entities acting as our agents or entities for whom we are acting as an agent (can access the data).

Work around #4: Custom CSRF error page

This isn't really a work-around, and should probably be done anyway. But Django gives you the ability to define a custom error page for CSRF validation errors, where could could tell the user what the problem is, and maybe have them to a manual frame break.

django-celery with postgres: "connection already closed" errors

2011-06-03T16:57:00.004-04:00

I ran into a mysterious, transient exception this week using django-celery.

DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
    before or while processing the request.

At other times, I was seeing a slightly different error message.

InterfaceError: connection already closed

After some searching, I found resolved issue #149 in the django-celery maintainer's github issues's list. It turns out that it's specifically a postgres issue, and only happens when the number of workers (the "c" parameter) is greater than one. Note: it defaults to the number of cores on your system.

A new issue #46 has been opened, so this will likely get fixed. In the meantime, you can monkey patch the code to fix it yourself.

sudo vim /usr/local/lib/python2.6/dist-packages/djcelery/loaders.py

    # inside DjangoLoader
    def on_task_init(self, task_id, task):
        self.close_database()

Stripping control characters in Python

2011-05-20T13:28:00.003-04:00

Want to remove ASCII control characters from a string? Have Unicode control characters that can't be encoded into XML? Here is how to strip them with regex in Python.

def strip_control_characters(input):
    
    if input:
            
        import re
        
        # unicode invalid characters
        RE_XML_ILLEGAL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])' + \
                         u'|' + \
                         u'([%s-%s][^%s-%s])|([^%s-%s][%s-%s])|([%s-%s]$)|(^[%s-%s])' % \
                          (unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),
                           unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),
                           unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),
                           )
        input = re.sub(RE_XML_ILLEGAL, "", input)
                        
        # ascii control characters
        input = re.sub(r"[\x01-\x1F\x7F]", "", input)
            
    return input

Unicode section adapted from Max Harper.

Ubuntu Unity: I want my KeePassX panel back!

2011-05-13T22:17:00.002-04:00

Big props to Ubuntu for taking a risk and trying to innovate in the desktop experience arena. Unity is pretty cool, but pretty raw, too. For example, some apps don't yet support the new panel. Even worse, some apps can get stuck in a "headless" mode where you can't access any of their windows due to not having a panel icon.

Here is how to fix that. For a specific app, just add it to the whitelist. Hopefully these apps will be updated soon, making this a temporary fix.

>gsettings get com.canonical.Unity.Panel systray-whitelist
['JavaEmbeddedFrame', 'Mumble', 'Wine', 'Skype', 'hp-systray', 'scp-dbus-service']

# added keepassx
>gsettings set com.canonical.Unity.Panel systray-whitelist "['JavaEmbeddedFrame', 'Mumble', 'Wine', 'Skype', 'hp-systray', 'scp-dbus-service', 'keepassx']"

Selenium: Execute arbitrary javascript (even jQuery)

2011-05-06T20:14:00.004-04:00

We're very happy with our new-ish Selenium + Saucelabs setup for web site automation testing. But as easy as Selenium makes it to compose tests, you will inevitably be frustrated trying to do some of the more complicated test scenarios.

For example, the Selenium IDE has not been sufficient for us to automatically generate code to look inside a javascript based rich text editor control. Similarly, we could not figure out how to uncheck ALL checkboxes in a certain form when you don't know their IDs/names up front.

As a web developer, when I run into issues like this with Selenium, I find myself wishing that I could just use jQuery. It's such a familiar tool, that I often think of how I would implement something with jQuery, and then attempt to translate that into the Selenium API.

Maybe in the future, Selenium will let you just code in javascript. What could be more natural? For now, you can use a work-around like the following python helper.

# adapted from http://pivotallabs.com/users/patn/blog/articles/717-run-javascript-in-selenium-tests-easily-
def javascript(sel, script):
    return sel.get_eval("""
    (function() {
            with(this) {
              %(script)s
            }
          }).call(selenium.browserbot.getUserWindow());
    """ % locals())

There is a tricky bit here. When you call get_eval(), your javascript is executed in the context of the test runner, not the actual test window. Hence wrapping your javascript in an anonymous function, and executing it in the context of the browserbot window via the call() method.

Another subtly is using getUserWindow() instead of getCurrentWindow(). This is so you can use the jQuery library already loaded by your app. Due to SEL-558, getCurrentWindow() no longer gives you access to dynamically defined document properties, like "jQuery" or "$". Thanks to Christopher Schmidt.

Assuming you already have jQuery loaded in your application, un-checking all the checkboxes on the page is as simple as:

class MyTest(SeleniumTestCase):
   def runTest(self):               
      javascript(self.selenium, "$('input[type=checkbox]').attr('checked', false);")

You can also use a jQuery selector to find a particular element and then return it. Any string returned from javascript() will get passed back to the python test runner code.

cron job to mail Splunk license violation alerts

2011-04-26T16:44:00.003-04:00

Have a problem staying under the Splunk free edition cap of 500MB/day? Splunk is no help. It silently logs violations, only taking action finally by shutting down your searches all together. Here is a quick cron job to email you license alerts as they happen.

touch ~/splunk-license-alert
chmod +x ~/splunk-license-alert
vim ~/splunk-license-alert

#!/bin/bash
cp ~/.splunk-licence-violations ~/.splunk-licence-violations.last
/opt/splunk/bin/splunk show license |grep violation > ~/.splunk-licence-violations
if diff ~/.splunk-licence-violations ~/.splunk-licence-violations.last >/dev/null ; then
   echo "No new license violations."
else
   cat ~/.splunk-licence-violations |mail -s "Splunk license violation" -t "user@example.com"
fi

You can run this a couple times to prime the temp files. Then, you should start seeing "No new license violations." Complete the install by setting up cron:

>crontab -e

# run every morning at 1AM; Splunk tallies license violations at 12 midnight
00 1 * * * ~/splunk-license-alert >/dev/null 2>&1

Measure navigation timing with javascript

2011-04-22T15:17:00.002-04:00

Historically, site administrators would measure their site performance by timing individual page renders. If your site is serving pages in less than 500ms, it must be pretty fast, right? The full picture is significantly more complicated.

With the advent of decent browser-based timing information, developers and admins realized just how much of the picture they were previously missing. Now, with sites like yotta, you can get this information without even sullying your own browser.

Notice that the "signup" HTML itself came back in just 158 milliseconds. This is the number you would see in typical apache log analyzer tools. But according to the browser based timing, the user did not see anything on the page for over 900ms. Even then, they could not fully interact with the page for over 2 seconds.

Instead of getting this information from expensive third party monitoring services, what if you could harvest it from your actual visitors? Enter the W3C Navigation Timing spec. It turns out browser vendors are way ahead of us.

User latency is an important quality benchmark for Web Applications. While JavaScript-based mechanisms can provide comprehensive instrumentation for user latency measurements within an application, in many cases, they are unable to provide a complete end-to-end latency picture.

In turns out that you can access this information right now, at least in Chrome latest and IE 9.0. It's exposed as a simple javascript object.

So how do you log this information on the server side? There are many ways, but recently we solved this problem on one of our sites by sticking the values into a cookie. Then you can examine this cookie server side on a subsequent page request. From there, we decided to log to a file and create pretty graphs with Splunk.

function store_client_performance_data() {

    var t = performance.timing;

    // Before the load event fires, the loadEventEnd value will be zero
    if (t.loadEventEnd > 0) {

        var earliestTime = t.navigationStart;

        // Redirects from a different origin cause navigationStart to
        // be zeroed out.  See NavigationTiming spec:
        //
        // https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
        //
        if (earliestTime == 0) {
            earliestTime = t.fetchStart;
        }

        fetchStartTime = t.fetchStart - earliestTime;
        responseEndTime = t.responseEnd - earliestTime;       
        loadEventEndTime = t.loadEventEnd - earliestTime;

        jQuery.cookie("clientSidePerformance",
                      fetchStartTime + "|" +
                      responseEndTime + "|" +
                      loadEventEndTime,
                      {"path": "/" });

    } else {
        setTimeout("store_client_performance_data()", 1000);
    }
}

if (Object.prototype.hasOwnProperty.call(window, "performance")) {
    setTimeout("store_client_performance_data()", 1000);
}

The only tricky part here is that you can't be sure when the page will stop loading. Instead, we run this function every second until loadEventEnd is non-zero.

Django: show me the sql!

2011-04-07T12:25:00.003-04:00

This week, I noticed that there was a select query in our production system without a where clause, and it was running a lot. Tracking it down, I found the piece of code that was running the query, but I couldn't be sure that my fix was actually changing the SQL to what I wanted.

In Django, it's pretty easy to ask the framework for the SQL in question. You can use the Debug toolbar, but in my case, this wasn't something that was running in a view. Instead, I used the interactive shell.

 
./manage.py shell
from django.db import connection
from myapp.models import MyModel
MyModel.objects.all()
>[<MyModel: 1>, <MyModel: 2>, <MyModel: 3>, '...(remaining elements truncated)...']
print connection.queries
>[{'time': '0.002', 'sql': 'SELECT "myapp_mymodel"."id", "myapp_mymodel"."field1", "myapp_mymodel"."field1" FROM "myapp_mymodel"'}]

In my case, I was calling .exists() before getting the first element in the result set. Of course, the exists was returning all the items in the database! Instead, I just used a try/catch around MyModel.objects.all()[0].

Python: Convert your FBML Facebook app to IFRAME

2011-04-01T15:53:00.007-04:00

Recently, users started reporting that our Facebook app was returning a blank screen. Tracking it down, it turned out that it was specifically when the user was in the new "Use Facebook as a Page" mode. Some other developers have reported a similar issue. Given that FBML apps were recently deprecated, I figured it was time to migration to IFRAMEs.

With our recent launch of Requests and the support for iframe on Pages Tabs, we are now ready to move forward with our previously announced plans to deprecate FBML and FBJS as a primary technology for building apps on Facebook. On March 11, 2011, you will no longer be able to create new FBML apps and Pages will no longer be able to add the Static FBML app. While all existing apps on Pages using FBML or the Static FBML app will continue to work, we strongly recommend that these apps transition to iframes as soon as possible. - Facebook Developers Blog

The first big different is the authentication model. Instead of passing the facebook page id directly, they are now passing a signed_request field. I was able to track down some open source code, and then modify it myself to only optionally take a secret, and remain backwards compatible with the FBML fb_sig_page_id parameter in Django.

import base64
import hashlib
import hmac
import simplejson as json

# from http://sunilarora.org/parsing-signedrequest-parameter-in-python-bas

def base64_url_decode(inp):
    padding_factor = (4 - len(inp) % 4) % 4
    inp += "="*padding_factor 
    return base64.b64decode(unicode(inp).translate(dict(zip(map(ord, u'-_'), u'+/'))))

def parse_signed_request(signed_request, secret=None):

    l = signed_request.split('.', 2)
    encoded_sig = l[0]
    payload = l[1]

    sig = base64_url_decode(encoded_sig)
    data = json.loads(base64_url_decode(payload))

    if data.get('algorithm').upper() != 'HMAC-SHA256':
        print "Facebook: Unknown algorithm"
        return None
    else:
        if secret:
            expected_sig = hmac.new(secret, msg=payload, digestmod=hashlib.sha256).digest()
    
    if secret and sig != expected_sig:
        return None
    
    return data

def get_facebook_page_id(request):
    facebook_page_id = request.REQUEST.get("fb_sig_page_id")
    if not facebook_page_id:
        signed_request = request.REQUEST.get("signed_request")
        if signed_request and "." in signed_request:
            data = parse_signed_request(signed_request)
            #{'user_id': 'XXX', 'algorithm': 'HMAC-SHA256', 'expires': 0, 
            # 'oauth_token': 'XXX', 
            # 'user': {'locale': 'en_US', 'country': 'us', 'age': {'min': 21}}, 'issued_at': 1301682530, 'page': 
            # {'admin': True, 'liked': False, 'id': 'XXX'}}
            page = data.get("page")            
            if page:
                facebook_page_id = page.get("id")
                
    return facebook_page_id

The only change to my existing view code was to call get_facebook_page_id(request) instead of just request.REQUEST.get("fb_sig_page_id").

The second big change was that you must style your apps yourself if you want it to look like the rest of Facebook in terms of fonts and colors. FBML apps were pretty locked down, which sucked in a lot of ways, but at least it automatically looked like the rest of Facebook. In any case, not such a big deal:

/* put a facebook-app class on your BODY element, or a box div around all your content */
.facebook-app { 
    font-size: 11px;
    color: #333;
    font-family: 'lucida grande', tahoma, verdana, arial;
} 
.facebook-app a { color: #3B5998; text-decoration: none; }

I didn't have to change my HTML at all; it still omits the html/body tags just like FBML apps are required to do, but at least it's backwards compatible. In fact, I can switch my app between FBML and IFRAME modes and not see much difference.

Django: send emails for doctest failures

2011-03-25T13:06:00.003-04:00

If you're using Django's unit/doc testing functionality, you would typically be running those tests manually on your local machine. This week, I decided that I wanted to run these nightly from our integration server, and send failures to the dev team.

The easiest way I could think of to do this was to write a little python script. You could run this script nightly using cron, or you could even run it on every commit as a post-build hook in your continuous integration tool.

#!/usr/bin/python
from optparse import OptionParser
import smtplib
from email.mime.text import MIMEText
import settings
import commands

def getOptions():
    arguments = OptionParser()
    arguments.add_option("--email", action="store_true", dest="email_failures", help="Send email failure reports")
    return arguments.parse_args()[0]

def send_email(body):
    message = MIMEText(body)
    message["Subject"] = settings.MAIL_SUBJECT_UNITTESTS
    message["From"] = settings.MAIL_FROM
    message["To"] = ", ".join(settings.MAIL_TO)
    smtp = smtplib.SMTP(host=settings.SMTP_HOST)
    smtp.sendmail(settings.MAIL_FROM, settings.MAIL_TO, message.as_string())
    smtp.quit()
                        
if __name__ == '__main__':        
    options = getOptions()
    # STDIN (0), STDOUT (1) and STDERR (2)
    # just output stderr, which has the failures
    exit_code, output = commands.getstatusoutput("../manage.py test --noinput YOURAPPNAME 1>/dev/null")
    # exit_code == 256 (ERROR), 0 (SUCCESS)
    if exit_code != 0 and options.email_failures:
        send_email(output)

Ubuntu + Cisco VPN from the command-line

2011-03-18T12:26:00.004-04:00

While the vpn client feature in the Ubuntu UI is very good, sometimes you need to VPN purely from the command-line. For example, if you need to VPN from a headless server into the network.

Say your config in Ubuntu looks like as follows. Setting up the vpnc command line version is fairly straight forward.

>apt-get install vpnc
>vim /etc/vpnc.conf

# add the following to /etc/vpnc.conf
IPSec gateway bhofc.bullhorn.com
IPSec ID MYCOMPANY
IPSec secret ***
Xauth username username
Xauth password ***
NAT Traversal Mode cisco-udp

# exit vim
>vpnc-connect
VPNC started in background (pid: 12490)...

# test, 192.168.1.7 is on your VPN network
>ping 192.168.1.7

If you have iptables setup to only allow white listed outbound traffic, you may be blocking necessary ports. Here is the config to open the required ports.

>iptables-save > /etc/iptables.conf
>vim /etc/iptables.conf

# add the following rules
-A OUTPUT -p tcp -m tcp --dport 500 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 500 -j ACCEPT
-A OUTPUT -p esp -j ACCEPT 

# exit vim
>iptables-restore </etc/iptables.conf

Facebook: Authenticate w/o the JavaScript SDK (Python)

2011-03-04T15:05:00.007-05:00

If you're using the Facebook JavaScript SDK to implement Facebook connect, your site may be more fragile than you think. Last weekend, I ran into an issue where an unrelated JavaScript error (the dreaded comma of death) caused IE7 users to fail trying to connect to Facebook. To make the site more robust, I decided to implement a non-JavaScript fallback.

Normally with Facebook connect, the login links have event handlers attached to them via JavaScript, such as this jQuery code.

$("#facebook-login-button").click(function () {
 FB.login(function(response) {  
  if (response.session) {
   window.location = "/facebook/authenticated";
  } 
 }, { perms: "email" });
 return false;
});

Normally that URL has a null href, like a pound symbol. We are still going to attach the Facebook handler, but we're also going to supply a fall back href.

connect to Facebook

Generating the correct Facebook URL to redirect the user to, and handling the "code" handed back by Facebook is done in Python/Django:

import urllib
import json

@register.filter()
def facebook_oauth_url(request, perms=None, next=None):
    return get_oauth_url(request, perms, next)

# generates the URL to put into the facebook connect login link as a non-js fallback
def get_oauth_url(request, perms=None, next=None): 
    
    client_id = settings.FACEBOOK_TOKEN
    
    # there are various kinds of users that needs to authenticate via Facebook
    # each needs to be taken back to a different page, and possibly use different perms
    request.session["facebook_next"] = next
    if perms == None:
        perms = "email,publish_stream"
        
    redirect_uri = reverse("facebook_login", prefix=settings.SITE_URL)
    return "https://graph.facebook.com/oauth/authorize?client_id=%(client_id)s&scope=%(perms)s&redirect_uri=%(redirect_uri)s" % locals()

# handles the "facebook_login" request; back from Facebook's side
def login(request):
     
    code = request.REQUEST.get("code")
    if code:
        
        access_token_url = "https://graph.facebook.com/oauth/access_token?"
        
        args = {}
        args["client_id"] = settings.FACEBOOK_TOKEN
        args["client_secret"] = settings.FACEBOOK_SECRET
 # facebook_login is THIS url, need to pass something in or the token will not be generated
        args["redirect_uri"] = reverse("facebook_login", prefix=settings.SITE_URL)
        args["code"] = code
        
        url = access_token_url + urlencode(args)
        file = urllib.urlopen(url)
        
        try:
            contents = file.read()
            
            if file.getcode() == 200:
                access_token = cgi.parse_qs(contents).get("access_token")[-1]                
                return HttpResponse("Got OAuth token: " + access_token)    
                
            else:                                
                response = _parse_json(contents)
                print "Facebook error code: " + response.get("error")
                    
        except ValueError, e:
            print "Error connecting to Facebook. %s" % e
        finally:
            file.close()

That's it! Once you have the access token, you can make your API calls. I use the Facebook GraphAPI SDK for Python.

HAProxy Quickstart w/ full example config file

2011-02-26T10:23:00.005-05:00

I recently installed HAProxy as a web server load balancer. While their documentation is great, I found it lacking a complete example of a working configuration file. For reference, here is the config file I ended up with along with comments.

# /etc/haproxy/haproxy.cfg, version 1.4

global
   maxconn 4096
   user haproxy
   group haproxy
   daemon

defaults
   log   global
   mode   http
   # logs which servers requests go to, plus current connections and a whole lot of other stuff 
   option   httplog
   option   dontlognull
   retries   3
   option redispatch
   maxconn   2000
   contimeout   5000
   clitimeout   50000
   srvtimeout   50000
   log        127.0.0.1       local0
   # use rsyslog rules to forword to a centralized server  
   log        127.0.0.1       local7 debug
   # check webservers for health, taking them out of the queue as necessary 
   option httpchk

# this load balancer servers both www.site.com and static.site.com, but those two URLS have  
# different servers on the backend (app servers versus statis media apache instances)  
# also, I want to server www.site.com/static/* from the later farm  

frontend http

   bind 0.0.0.0:80

   # important, see comment from Willy Tarreau bellow
   option http-server-close

   # NAT static host names and static paths in other hostnames to static.bullhornreach.com
   acl host_static hdr_beg(host) -i static
   acl url_static  path_beg    /static
   use_backend static if host_static
   use_backend static if url_static

    default_backend www

backend www
   balance roundrobin
   server www1 www1 check port 80
   server www2 www2 check port 80
   server www3 www3 check port 80
   # provide a maintenance page functionality, only used when all other servers are down
   server load1 localhost:8080 backup

backend static
   # for static media, connections are cheap, plus the client is very likely to request multiple files  
   # so, keep the connection open (KeepAlive is the default)  
   balance roundrobin
   server media1 media1 check port 80
   server media2 media2 check port 80

listen stats :1936
   mode http
   stats enable
   stats scope http
   stats scope www
   stats scope static
   stats scope static_httpclose
   stats realm Haproxy\ Statistics
   stats uri /
   stats auth haproxy:YOURPASSWORDHERE

Edit: made changes suggested by Willy Tarreau.

Django fix_IE_for_vary "bug"

2011-02-18T14:13:00.003-05:00

I was briefly perplexed by an IE only bug this week. An Ajax call was throwing the following 500 exception, but only in IE.

Traceback (most recent call last):

File "/home/chase/bullhorn/branches/talladega/virtualenv/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 280, in run
self.result = application(self.environ, self.start_response)

File "/home/chase/bullhorn/branches/talladega/virtualenv/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 674, in _call_
return self.application(environ, start_response)

File "/home/chase/bullhorn/branches/talladega/virtualenv/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 246, in _call_
response = self.apply_response_fixes(request, response)

File "/home/chase/bullhorn/branches/talladega/virtualenv/lib/python2.6/site-packages/django/core/handlers/base.py", line 195, in apply_response_fixes
response = func(request, response)

File "/home/chase/bullhorn/branches/talladega/virtualenv/lib/python2.6/site-packages/django/http/utils.py", line 77, in fix_IE_for_vary
if response['Content-Type'].split(';')[0] not in safe_mime_types:

AttributeError: 'int' object has no attribute 'split'

Originally, I thought this looked like a bug in Django, however unlikely that is. But after walking through the code with a debugger, it was clear that the "Content-Type" reponse header was not "text/plain", or some such. Instead it was the integer 200. I finally traced it back to my own code.

return HttpResponse("job saved", 200) # should be status=200

The problem is that the second positional argument to HttpResponse is content type, but I was just assuming it was the status code in this case. Easy fix.

cat to syslog

2011-02-11T15:14:00.002-05:00

Ever wanted to submit the results of regular command-line tools to syslog? Logger lets you do that. Here is an example of logging CPU load and vmstat into to a centralized syslog server.

cat /proc/loadavg |logger -p local7.info -t "load-cpu"
vmstat |tail -n +3 |logger -p local7.info -t "load-vmstat"

Quickstart: Log shipping with Postgres 8.4

2011-02-03T09:47:00.018-05:00

The official Postgres documentation covers log shipping (aka WAL Archiving, aka warm standby) in some depth. But it's still quite a lot to absorb. The basic idea is to give you a hot spare to fail over to if your main database server crashes. Here is a step by step guide on how to set this up on the command line in Ubuntu (10.04). I assume you have two identical servers (vm clone?), a master and a slave.

First, you need to setup a directory on the slave which will hold the WAL archives (logs). Unless otherwise notes, all commands are run as root.

# on the slave, must be outside main b/c we are going to blow that away
mkdir /var/lib/postgresql/8.4/pg_wal
chown postgres:postgres /var/lib/postgresql/8.4/pg_wal

We're going to use ssh/scp to copy WAL files from the master to the slave(10.177.1.247), so you need to exchange ssh keys for the postgres users on each end.

# on master, share postgres credentials with slave
sudo su postgres
ssh-keygen -t rsa # press enter to all prompts (don't set explicit password)
ssh postgres@10.177.1.247 mkdir -p ~/.ssh
cat ~/.ssh/id_rsa.pub | ssh postgres@10.177.1.247 'cat >> ~/.ssh/authorized_keys'

# test the ssh connection
ssh postgres@10.177.1.247 # should not prompt for a password
exit # exit new ssh session
exit # exit sudo as postgres, you're now root again

# setup postgres WAL archiving to copy to slave
vim /etc/postgresql/8.4/main/postgresql.conf

# make the following edits
archive_mode = on
archive_command = 'scp %p postgres@10.177.1.247:/var/lib/postgresql/8.4/pg_wal/%f </dev/null'
archive_timeout = 600 # 10 minutes

# now, outside of vim, restart postgres
/etc/init.d/postgresql restart

If you wait 10 minutes, you should see a new file in /var/lib/postgresql/8.4/pg_wal on the slave. Assuming that's working, you're ready to create the initial snapshot.

I was confused by the documentation on this point. Basically, you want to issue a pg_start_backup SQL command on the master, then copy the entire /var/lib/postgresql/8.4/main directory to the slave. The pg_start_backup ensures that the physical files will be in a consistent state. You can perform this copy while master is running.

# on master
sudo -u postgres psql mydbname
SELECT pg_start_backup('initial');
# may take a few minutes, wait then control-D to exit

# zip up the main data directory, and copy it to the slave
cd /var/lib/postgresql/8.4
zip -r main.zip main
scp main.zip postgres@10.177.1.247:/var/lib/postgresql/8.4

sudo -u postgres psql mydbname
SELECT pg_stop_backup();

Now the initial snapshot is on the slave, but we need to actually restore those files as the new primary files on the slave. At the same time, we can restart the slave in recovery mode, using pg_standby. Recovery is enabled by creating a .conf file right in main.

# on the SLAVE
apt-get install postgresql-contrib-8.4 # installs the pg_standby tool
/etc/init.d/postgresql-8.4 stop
cd /var/lib/postgresql/8.4
mv main main.bak
unzip main.zip
rm -f main/pg_xlog/* # these are from master, and are not needed

# create the recovery config file
vim /var/lib/postgresql/8.4/main/recovery.conf

# add the following, and save
restore_command = '/usr/lib/postgresql/8.4/bin/pg_standby -d -t /tmp/pgsql.trigger.5442 /var/lib/postgresql/8.4/pg_wal %f %p %r 2>>/var/log/postgresql/standby.log'
recovery_end_command = 'rm -f /tmp/pgsql.trigger.5442'

# make sure the new main (as well as recovery.conf) have the correct ownership
chown postgres:postgres main -R

# restart, and keep and eye on the new standby.log file to see if it's working
/etc/init.d/postgresql-8.4 restart
tail -f /var/log/postgresql/standby.log -n 100

You should see log lines like the following.

Trigger file   : /tmp/pgsql.trigger.5442
Waiting for WAL file : 0000000100000006000000AE.00000020.backup
WAL file path  : /var/lib/postgresql/8.4/pg_wal/0000000100000006000000AE.00000020.backup
Restoring to  : pg_xlog/RECOVERYHISTORY
Sleep interval  : 5 seconds
Max wait interval : 0 forever
Command for restore : cp "/var/lib/postgresql/8.4/pg_wal/0000000100000006000000AE.00000020.backup" "pg_xlog/RECOVERYHISTORY"
Keep archive history : 000000000000000000000000 and later
running restore  : OK
Trigger file   : /tmp/pgsql.trigger.5442
Waiting for WAL file : 0000000100000006000000AE
WAL file path  : /var/lib/postgresql/8.4/pg_wal/0000000100000006000000AE
Restoring to  : pg_xlog/RECOVERYXLOG
Sleep interval  : 5 seconds
Max wait interval : 0 forever
Command for restore : cp "/var/lib/postgresql/8.4/pg_wal/0000000100000006000000AE" "pg_xlog/RECOVERYXLOG"
Keep archive history : 000000000000000000000000 and later
running restore  : OK

Trigger file   : /tmp/pgsql.trigger.5442
Waiting for WAL file : 0000000100000006000000AF
WAL file path  : /var/lib/postgresql/8.4/pg_wal/0000000100000006000000AF
Restoring to  : pg_xlog/RECOVERYXLOG
Sleep interval  : 5 seconds
Max wait interval : 0 forever
Command for restore : cp "/var/lib/postgresql/8.4/pg_wal/0000000100000006000000AF" "pg_xlog/RECOVERYXLOG"
Keep archive history : 0000000100000006000000AE and later
WAL file not present yet. Checking for trigger file...
WAL file not present yet. Checking for trigger file...
WAL file not present yet. Checking for trigger file...

Sanitize HTML with Beautiful Soup

2011-01-28T13:45:00.003-05:00

If you have a website that displays user-generated HTML (emails, rich text entry, etc), you likely want to scrub that HTML before you display it. At the very least, you want to provide a reasonable protection against XSS. But maybe you also want to prevent the HTML from breaking your page layout. Either way, Beautiful Soup is a good tool for the job.

Here is some Python code that uses Beautiful Soup to clean HTML of any tags and attributes not explicitly white listed. The CSS property scrubbing is blacklist based, which sucks and should be redone with a real CSS parsing library. If you want a really diesel solution to this issue, I suggest you look at AntiSamy for Java or .NET.

As a bonus, there is also a method for converting html to plaintext, while making a best effort to preserve the structural whitespace.

from BeautifulSoup import BeautifulSoup, Comment
import re, htmlentitydefs
from HTMLParser import HTMLParseError
from datetime import datetime
import subprocess
import os
import urllib2

def safe_html(html):
    
    if not html:
        return None

    # remove these tags, complete with contents.
    blacklist = ["script", "style" ]
    
    whitelist = [
        "div", "span", "p", "br", "pre",
        "table", "tbody", "thead", "tr", "td", "a",
        "blockquote",
        "ul", "li", "ol", 
        "b", "em", "i", "strong", "u", "font"                 
        ]

    try:
        # BeautifulSoup is catching out-of-order and unclosed tags, so markup
        # can't leak out of comments and break the rest of the page.            
        soup = BeautifulSoup(html)        
    except HTMLParseError, e:
        # special handling?
        raise e

    # now strip HTML we don't like.
    for tag in soup.findAll():
        if tag.name.lower() in blacklist:
            # blacklisted tags are removed in their entirety
            tag.extract()
        elif tag.name.lower() in whitelist:
            # tag is allowed. Make sure all the attributes are allowed.
            tag.attrs = [(a[0], safe_css(a[0], a[1])) for a in tag.attrs if _attr_name_whitelisted(a[0])]
        else:
            # not a whitelisted tag. I'd like to remove it from the tree
            # and replace it with its children. But that's hard. It's much
            # easier to just replace it with an empty span tag.
            tag.name = "span"
            tag.attrs = []

    # scripts can be executed from comments in some cases
    comments = soup.findAll(text=lambda text:isinstance(text, Comment))
    for comment in comments:
        comment.extract()

    safe_html = unicode(soup)
    
    if safe_html == ", -":
        return None   
    
    return safe_html
    
def _attr_name_whitelisted(attr_name):
    return attr_name.lower() in ["href", "style", "color", "size", "bgcolor", "border"]
     
def safe_css(attr, css):
    if attr == "style":
        return re.sub("(width|height):[^;]+;", "", css)
    return css

def plaintext(input):
    """Converts HTML to plaintext, preserving whitespace."""
    
    # from http://effbot.org/zone/re-sub.htm#unescape-html
    def _unescape(text):
        def fixup(m):
            text = m.group(0)
            if text[:2] == "&#":
                # character reference
                try:
                    if text[:3] == "&#x":
                        return unichr(int(text[3:-1], 16))
                    else:
                        return unichr(int(text[2:-1]))
                except ValueError:
                    pass
            else:
                # named entity
                try:
                    text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
                except KeyError:
                    pass
            return text # leave as is
        return re.sub("&#?\w+;", fixup, text)
        
    input = safe_html(input) # basic sanitation first
    text = "".join(BeautifulSoup("<body>%s</body>" % input).body(text=True))
    text = text.replace("xml version='1.0' encoding='%SOUP-ENCODING%'", "") # strip BS meta-data
    return _unescape(text)

Note: This method is neither fast, nor particularly bullet proof. You definitely want to cache the results so you're not performing this transformation while the user is waiting. Some documents will throw a HTMLParseError if Beautiful Soup cannot parse them. You can choose whether you want to do the secure thing and not show it at all, or if you want to risk it and show them the original HTML.

Youtube: Detecting X-Frame-Options/framebreaking in Python

2011-01-21T13:07:00.005-05:00

My current project involves URL shortening. Like many URL shorteners, we wanted to get into frame bars. These are alternately called banners, share bars and "eyebrows". It's basically just putting some small content on the top of the screen, and then wrapping the "real" content in an IFRAME that takes the rest of the screen. Maybe you've seen when LinkedIn does this.

Now, I'm not crazy about the ethics of framing other peoples' content. I'm not even convinced that it would not be ruled illegal some day. In any case, lot's of people are doing it, and we decided we needed to be doing it too. In my case, we're just showing you who shared it, we're not doing ads or anything.

Understandably, many sites don't want you to frame their content. This has lead to a kind of arms race, where content sites use javascript to try to break out of the frame (aka framebreaking), and then the wrapper sites try to break the framebreaker.

This eventually lead to full-fledged support at the browser level for framebreaking, in the form of the X-Frame-Options header. Essentially, a site can publish this HTTP header on all their content, and most modern browsers will display an error/warning if the content is put into a frame (or, at least a frame on another domain). In Firefox, this looks like the following.

This is actually a pretty cool development. However, in order to provide a good user experience when you're frame barring (is that an oxymoron?), you ideally would detect ahead of time whether the site you're framing has this X-Frame-Option set, and just redirect to the content in that case.

Here is some Python code that does just that. It basically just makes an HTTP call to the content server to server.

def is_framing_allowed(url)
   request = urllib2.Request(url)
   request.add_header("Referer", "http://example.com") # you should put your real URL here
   opener = urllib2.urlopen(request)
   # returns True if ANY x-frame-options header is present
   # the only options at present are DENY and SAMEORIGIN, either of which means you can't frame
   return "x-frame-options" in opener.headers.dict

Notice that I'm setting a Referer here. Youtube in particular requires that this be set before it will serve a x-frame-options header:

curl "http://www.youtube.com/watch?v=kdoCBJe0F3o" -D headers.txt >/dev/null 2>&1; cat headers.txt
HTTP/1.1 200 OK
Date: Fri, 21 Jan 2011 18:35:39 GMT
Server: wiseguy/0.6.7
X-Content-Type-Options: nosniff
Set-Cookie: use_hitbox=72c46ff6cbcdb7c5585c36411b6b334edAEAAAAw; path=/; domain=.youtube.com
Set-Cookie: VISITOR_INFO1_LIVE=nMjwYxkZq0Q; path=/; domain=.youtube.com; expires=Sun, 18-Sep-2011 18:35:39 GMT
Set-Cookie: recently_watched_video_id_list=7291701d6005203af93186bd50b882b5WwEAAABzCwAAAGtkb0NCSmUwRjNv; path=/; domain=.youtube.com
Set-Cookie: GEO=1fe4b343c2bea46a564b0990ff037b32cwsAAAAzVVPRyoNiTTnR+w==; path=/; domain=.youtube.com
Expires: Tue, 27 Apr 1971 19:44:06 EST
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked

curl "http://www.youtube.com/watch?v=kdoCBJe0F3o" -e "http://example.com" -D headers.txt >/dev/null 2>&1; cat headers.txt
HTTP/1.1 200 OK
Date: Fri, 21 Jan 2011 18:35:55 GMT
...
X-Frame-Options: SAMEORIGIN

Backup your Amazon order history with Python

2011-01-15T12:19:00.008-05:00

Ever wanted to download your Amazon order history? Maybe you want to get it into a spreadsheet, or just keep it around in case Amazon decides to delete this information. Here is some Python code to screen scrape your account pages.

import sys
from optparse import OptionParser  
import datetime
import mechanize
from BeautifulSoup import BeautifulSoup
import pprint

"""
Usage: python amazon.py --username=foo --password=bar --firstyear=2004
"""

def getOptions():
   arguments = OptionParser()
   arguments.add_options(["--username", "--password", "--firstyear"])
   return arguments.parse_args()[0]

def _text(node):
      return "".join([unicode(s) for s in node.contents]).strip()

def _parse_orders(html):
      soup = BeautifulSoup(html)
      orders = []
      for order in soup.findAll("div", {"class": "order"}):
            date = order.find("h2")
            for item in order.findAll("li", {"class": "item "}):
                  title = item.find("span", {"class": "item-title"})
                  link = item.find("a")
                  image = item.find("img")
                  orders.append({
                        "date": _text(date),
                        "title": _text(title),
                        "link": link["href"],
                        "image": image["src"]
                  })     
      return orders

if __name__ == '__main__': 

      options = getOptions()

      br = mechanize.Browser()
      br.set_handle_robots(False)
      br.addheaders = [("User-agent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")]

      sign_in = br.open("http://www.amazon.com/gp/flex/sign-out.html")

      br.select_form(name="sign-in")
      br["email"] = options.username
      br["password"] = options.password
      logged_in = br.submit()

      error_str = "The e-mail address and password you entered do not match any accounts on record."
      if error_str in logged_in.read():
            print error_str
            sys.exit(1)

      orders = []
      for year in range(int(options.firstyear),  datetime.datetime.now().year):
            orders_html = br.open("https://www.amazon.com/gp/css/history/orders/view.html?orderFilter=year-%s&startAtIndex=1000" % year)
            new_orders = _parse_orders(orders_html.read())
            if new_orders:
                  orders.append(new_orders)

      if len(orders) == 0:
            print "No orders found."
            sys.exit(1)

      pp = pprint.PrettyPrinter(indent=4)
      pp.pprint(orders)

# usage example, your username and password are your Amazon login. The firstyear is the year of your oldest order.
python amazon.py --username=foo --password=bar --firstyear=2004

# example output (json)
    [   {   'date': u'November 26, 2007',
            'image': u'https://images-na.ssl-images-amazon.com/images/I/517SDCCC3KL._SX100_.jpg',
            'link': u'http://www.amazon.com/gp/product/0847827852/ref=oss_product',
            'title': u'Chip Kidd: Book One: Work: 1986-2006 (Chip Kidd)'},
        {   'date': u'November 23, 2007',
            'image': u'https://images-na.ssl-images-amazon.com/images/I/118MVgvXzoL._SX100_.jpg',
            'link': u'http://www.amazon.com/gp/product/B000OUP9NE/ref=oss_product',
            'title': u'Canon Black Ink Cartridge &#45; PGI5'},
        {   'date': u'November 23, 2007',
            'image': u'https://images-na.ssl-images-amazon.com/images/I/41sW%2B3x4VJL._SY100_.jpg',
            'link': u'http://www.amazon.com/gp/product/B000BUWNH2/ref=oss_product',
            'title': u'Canon CLI-8 4-Color Multipack Ink Tanks'},
        {   'date': u'November 22, 2007',
            'image': u'https://images-na.ssl-images-amazon.com/images/I/51YT5W1SEeL._SY100_.jpg',
            'link': u'http://www.amazon.com/gp/product/B0002F5E0E/ref=oss_product',
            'title': u'Meinl Kenny Aronoff Steel Bell Series Cowbell,  8 Inches'},

        ...

Please, drop me a comment if you have any bug fixes or interesting use cases for this code.

Getting arbitrary log files into Rsyslog

2011-01-07T09:27:00.004-05:00

Ideally, the applications that you're working with already supports the syslog standard. Failing that, it's often necessary to suck a regular text log file into rsyslog. Doing so is fairly easy, but I had a harder time than I suspected finding the documentation.

The key is the optional rsyslog module imfile. It's not enabled by default, so you will need to enable it.

vim /etc/rsyslog.conf

# add the following
$ModLoad imfile

You will also need to configure each log file you want to poll. If you want, you can place these settings in separate files under /etc/rsyslog.d. In this case, my application is called "celery".

vim /etc/rsyslog.d/celery.conf

# add the following
$InputFileName /var/log/django/celery.log
$InputFileTag celery
$InputFileStateFile celery-file1
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

# exit vim, and restart rsyslog
service rsyslog restart

This will poll celery.log every 10 seconds, and echo the contents into syslog. You can then forward them to remote machines as necessary.

Splunk/Rsyslog/Apache/Ubuntu Quickstart

2010-12-31T10:10:00.009-05:00

I've been using Splunk for several years. It's a great tool for log file visualization and analytics. One thing I use it for is to generate a graph of average milliseconds per request on an apache instance. Up until recently, I was running this directly on the production web server.

Lately, it became clear that we needed to move this to a dedicated machine. For one thing, we were about to add a second apache server. Also, I have noticed that running this report hammers the server pretty hard, especially if you're doing field extraction on the fly. In any case, I decided it was time to look into syslog.

Syslog is a standard for logging program messages. It allows separation of the software that generates messages from the system that stores them and the software that reports and analyzes them. It also provides devices which would otherwise be unable to communicate a means to notify administrators of problems or performance. - Wikipedia

In practice, the specific feature of syslog I wanted to leverage was the forwarding of log events from the production machine(s) to the Splunk server. In Splunk, the setup is just enabling TCP data inputs.

Port 5011 is a commonly unused port I chose at random. In my case, I also needed to open this port in iptables (the firewall) on the Splunk server.

iptables-save > /etc/iptables.conf
vim /etc/iptables.conf

# add the following (allow the 10.177 subnet to open TCP port 5011 to the Splunk server)
-A INPUT -s 10.177.0.0/16 -p tcp -m tcp --dport 5011 -j ACCEPT

/sbin/iptables-restore < /etc/iptables.conf

On a current Ubuntu installation, syslog service is provided by rsyslog. The first step is to configure rsyslog to forward log events to the Splunk server. In my case, the Splunk server is at 10.177.0.204.

vim /etc/rsyslog.d/50-default.conf

# add the following, which defines a common format, and then forwards ALL syslog events
$template SyslFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::space$
*.* @@10.177.0.204:5011;SyslFormat

service rsyslog restart

Again, I also needed to open port 5011 (outbound this time) in iptables.

-A OUTPUT -p tcp -m tcp --dport 5011 -j ACCEPT

At this point, you should start seeing syslog events in Splunk. An easy way to test it is to restart rsyslog. You should see the log events for shutting down/starting the rsyslog process. If not, you may want to try the following trouble-shooting steps. In my case, I initially ran into trouble because I was assuming the traffic was being sent over UDP, but in rsyslog it's TCP by default.

# tail the local syslog file, make sure events are being generated
tail -f /var/log/syslog

# try to open a port from the apache server to the splunk server, it's successful unless you see connection refused
telnet 10.177.0.204 5011

# try to open port 5011 on the Splunk server itself
telnet localhost 5011

The next step is to get Apache to log to syslog. In my case, the following was sufficient to log anything that Django prints to the console.

vim /etc/apache2/sites-available/default

# add the following, logs most events in syslog
ErrorLog syslog:local7

service apache2 restart

Now you should be getting Apache + other syslog events into Splunk. But they all show up with a source type of "tcp:5011" by default. What's happening is that Splunk is seeing them all as the same virtual log file. Luckily, there is an easy configuration to break them out into separate source types.

vim /opt/splunk/etc/apps/search/default/props.conf

# add the following to act on anything from tcp:5011
[source::tcp:5011]
TRANSFORMS-yummy = setSourceApache, setSourcePostfix

vim /opt/splunk/etc/apps/search/default/transforms.conf

# add the following to define setSourceApache and setSourcePostfix
[setSourceApache]
DEST_KEY = MetaData:Sourcetype
# any log line that contains apache2 
REGEX = apache2
FORMAT = sourcetype::apache

[setSourcePostfix]
DEST_KEY = MetaData:Sourcetype
REGEX = postfix
FORMAT = sourcetype::postfix

/opt/splunk/bin/splunk restart

Finally, while you're trouble-shooting all of this, you will likely be generating junk log events. If you find yourself wanting to start over, you can aways bring up a set of results in Splunk, and just pipe the search to "|delete" to blow them away.

Python: Convert a Word/PDF document to html

2010-12-17T10:51:00.007-05:00

My current Django project deals with resume files in Word/PDF format. In order to show a web preview of the file, it's necessary to translate these files to plain HTML. While this was sometimes a pain in the past, I've recently found that it's relatively easy with standard Linux tools.

AbiWord is a general purpose word processor for Linux. It has pretty good support for Word files, as well as many other formats such as PDF, RTF, etc. Usually it's invoked as a GUI app, just like Microsoft Word. However, being a Linux app, there is also good command-line support.

One of the things you can do from the command line is convert files from one format to another. Here is a quick example:

# print the HTML translation of a DOC file to the console
abiword -t output.html resume.doc; cat output.html

It's also relatively simple to invoke this from Python, using the standard libraries.

import subprocess
import os
import uuid

def document_to_html(file_path):
    tmp = "/tmp"
    guid = str(uuid.uuid1())
    # convert the file, using a temporary file w/ a random name
    command = "abiword -t %(tmp)s/%(guid)s.html %(file_path)s; cat %(tmp)s/%(guid)s.html" % locals()    
    p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, cwd=os.path.join(settings.PROJECT_DIR, "website/templates"))
    error = p.stderr.readlines()    
    if error:
        raise Exception("".join(error))
    html = p.stdout.readlines()
    return "".join(html)

AbiWord produces fairly clean HTML. If you want to scrub it even more, I would suggest something like BeautifulSoup.

Per user timezones in Django

2010-12-10T14:44:00.006-05:00

A common requirement for sites that deal with dates (most of them), is to display date and time values in the individual user's timezones. Luckily, there are a few good ready-made solutions for Python/Django. They don't solve the problem completely seamlessly. You will still need to decide when to invoke the datetime translation, but most of the heavy lifting will be done for you.

Before I go any further, it's worth noting that there is a work-around for some subset of sites. Instead of collecting and showing absolute dates and times, you may be able to get away with relative dates and times. For example, if you just need to display the time a record was created, you could say "13 minutes ago" instead of "Dec 10, 2010 at 8:55pm". In fact, here is a handy decorator to that effect:

from datetime import datetime
from django.template.defaultfilters import timesince

@register.filter
def ago(date_time):
    diff = abs(date_time - datetime.today())
    if diff.days <= 0:
        span = timesince(date_time)
        span = span.split(",")[0] # just the most significant digit
        if span == "0 minutes":
            return "seconds ago"        
        return "%s ago" % span 
    return date(date_time)

Assuming you do need to support absolute times, you would start by downloading django-timezones, which itself relies on pytz. You can get both via pip:

sudo pip install django-timezones
sudo pip install pytz

However, there is a bug in the latest stable release of django-timezones which prevents you from using their model and form definitions. If you do, you will get a "Value XXX is not a valid choice" validation error trying to submit the form. You can either install the trunk release, where the bug is fixed, or use the following work-around models definition:

from timezones.forms import PRETTY_TIMEZONE_CHOICES

class UserProfile(models.Model):
   ...
   timezone = models.CharField(max_length=255, choices=PRETTY_TIMEZONE_CHOICES, blank=True, null=True, )
   ...

With this definition, you can use a standard ModelForm, without overriding the field or widget of timezone. Once you have your form rendering in HTML, you can use the following jQuery snippet to default the timezone selection to the correct choice for that user, using getTimezoneOffset. It will only fire if there is no timezone selected already.

var selects = $("select#id_timezone");
if (selects.length > 0 && selects.val() == "") {  
 var offset_minutes = new Date().getTimezoneOffset();
 var offset = 100 * offset_minutes / 60;
 var default_value = _first_timezone_match(selects, offset);  
 selects.val(default_value);
}

function _first_timezone_match(selects, offset) {
 var match = "";
 selects.find("option").each(function() {
  // ex: "(GMT-0500) America/New_York"
  if ($(this).text().indexOf(offset) > 0) {
   match = $(this).val();
  }
 });
 return match;
}

Finally, here are some helper methods to translate to and from the server timezone and the user's timezone.

import datetime
import settings
import pytz

# need to translate to a non-naive timezone, even if timezone == settings.TIME_ZONE, so we can compare two dates
def to_user_timezone(date, profile):
    timezone = profile.timezone if profile.timezone else settings.TIME_ZONE               
    return date.replace(tzinfo=pytz.timezone(settings.TIME_ZONE)).astimezone(pytz.timezone(timezone))
    
def to_system_timezone(date, profile):
    timezone = profile.timezone if profile.timezone else settings.TIME_ZONE               
    return date.replace(tzinfo=pytz.timezone(timezone)).astimezone(pytz.timezone(settings.TIME_ZONE))
    
def now_timezone():   
    return datetime.datetime.now().replace(tzinfo=pytz.timezone(settings.TIME_ZONE)).astimezone(pytz.timezone(settings.TIME_ZONE))

Note: I'm careful to always return a datetime object w/ the timezone information. This is because otherwise, you could get a "can't compare offset-naive and offset-aware datetimes" error when you compare two datetime objects.

Render styled HTML emails consistently with Litmus

2010-12-03T13:00:00.006-05:00

Getting your HTML email to look the same on all mail clients is hard. Most clients only support a small subset of HTML. Outlook 2010 actually uses Word to display HTML, and we know how much that sucks. But it's more endemic than that; there are simply no standards to code to.

When marketing wants an email to look exactly the same on all clients, tell them they need to use plaintext. When they say they need HTML, tell them that it's impossible. When they insist, tell them that you can get it to look 90% the same across clients, but it will take 40 hours. The first hour will be spent composing the email. For the next 39 hours you will be sending it to gmail, testing, tweaking, sending it to yahoo, testing, tweaking, sending it to Outlook... etc.

Enter Litmus. For $49 a month you can send them a copy of the email, and they will render screenshots of what it looks like on the 25 email clients that actually matter.

You can zoom in on exactly what it looks like on a specific client, with or without images enabled, in plaintext or html mode. Litmus can show you the final rendered HTML code, as well as the entire MIME message. You can mark test cases as "failed", and then evaluate them again on subsequent revisions.

Having done this a couple of times recently, I came up with some rules of thumb.

Assume images won't load. Use alt/title attributes.
Many clients will not render style tags. Pretend it's 1995 and use inline styles.
Relative, absolute and float positioning will not work. Use tables.
Yahoo: add "margin-bottom: 1em;" to your paragraph tags.
Outlook: some links will break when they wrap in MIME. If you can, keep them under 80 characters, and put them on the start of a line.
Include a plaintext section. Some users prefer it, and some mobiles devices only show plaintext.

Django {% url %} tag; it can do both absolute/relative urls!

2010-11-19T10:04:00.007-05:00

Most often, when you're using Django's url tag, you want relative links, not absolute links.

Examples of relative links:

/user/103
/user/103/edit

Examples of absolute links:

http://example.com/user/103
https://example.com/user/103/edit

With relative links, you preserve the domain same and protocol (HTTP or HTTPS) of the current user session, with zero work on your part. This is the default behavior of Django's {% url %} tag. Note: confusingly, they refer to "absolute" urls incorrectly in their documentation.

However, in some cases you need to use absolute URLs. For example, in html emails. When the users is in their email client, they don't have the context of what domain/protocol the link is for.

In the past, I would typically change the code of those templates to look something like this:

 <!-- you could also pull the protocol/domain from a setting or from Django's sites metadata -->
 Please <a href="http://example.com{% url user user.id %}">click here<a>.

But what if you're using the same template for website content and email content? You could just output absolute links in both cases. You could put logic into the templates to only output the domain if a certain view scope variable is set. You could also just create separate templates for email/site content. None of these solutions is ideal.

Instead, you could use Django's set_script_prefix. I can't find it in the official documentation, but this handy little method sets a prefix for the current thread context, which will be used on all subsequent url resolutions.

In my case, I just added the following one liner to my mail sending routine before the email content template is rendered.

   set_script_prefix(settings.SITE_URL) # ie, http://example.com

Preventing duplicate form submissions with jQuery

2010-11-05T12:37:00.003-04:00

A common problem for a new site to have is for their users to occasionally submit the same form two or more times. For example, maybe they were adding a new record, and were not sure if the application registered their submit button click, so they try again. This problem can be exacerbated by a slow website, which gives the users a larger window in which to come to this conclusion.

Typically, this is not a problem for edit forms. For the most past, committing the same edits twice to a single record is fine; the record ends up in the same final state either way. Of course, some edits have side effects, which could certainly be an issue. However, one case that almost always fails out of the box would be a user adding a record twice. Typically, this ends up with two new records in the system.

One way to prevent these duplicate submissions is to use a unique token for each add/edit form. The server would render this token into a hidden field on the form, and then check for that token being present on the commit. But this can be difficult to implement on an existing code base with many different forms, unless your framework has some kind of middleware layer that you can leverage to apply to all form render/submissions.

A more typical solution is to use Javascript. Of course, this does not guarantee that the user cannot submit the form twice; all they would need to do is disable Javascript, or hit an untrapped error on your code. But it does address the core issue, which is one of user experience. It's by far a better user experience to give a visual indication that the form is being processed.

Here is a simple approach that you can include in your base jQuery code, and which will provide consistent functionality across all the existing forms on your site. It requires jQuery, and was tested in version 1.4.2.

 $("form").submit(function () {
  var form = $(this);
  form.find("input[type=submit]").attr("disabled", "disabled")
  form.find("input[type=submit]").attr("value", "Processing...");
  // prevent the user from hitting ENTER on a field to submit again
  form.find("input").attr("readonly", true);
  document.body.style.cursor = "wait";
 });

The key trick here is NOT just disabling the form. That's a common approach, but it has one major issue. Disabled forms can't actually be submitted. A common hack is to disable the form after a certain timeout, but that's inviting a race condition on a slow machine. The readonly/disable the button approach is more robust.

At the same time, you may want to change the style of the disabled form submit button, not just the text.

input[type=button][disabled], input[type=submit][disabled] { background-color: #BBB; border-color: #444; }

One issue to be aware of with this solution is that if the the user uses the back button to navigate to a previous form, the submit button will still be disabled. That may or may not be desirable. It's definitely not desirable, for example, if the form submission page threw an error and the user is legitimately trying to submit again. I'm interested in some thoughts on a solution for this case.

Django: uniform image resizing

2010-10-29T13:50:00.005-04:00

Sometimes, you want all the thumbnails you're creating to be the same dimensions. For example, your layout may require a certain width so that all the items line up in a grid. In my case, I wanted them to be exactly 96x96 pixels.

The tool of choice in Django is PIL, which is a PITA to install, but which works well once it's installed. Just be sure to check that JPEG works, it's commonly flubbed by the installer.

Another common requirement is to retain the aspect ratio of the original image. For example, given the above input image, we would not want to produce this:

For that, PIL's built-in thumbnail method will do nicely. However, that would not get us a perfect 96x96 square. Unless the original image is also a square, it will instead produce a thumbnail where only the larger dimension it 96 pixels.

To compensate, we need to add literal white space to fill out the remaining pixels. In PIL, you can accomplish this with the paste method.

import Image
import traceback
import sys

def resize_aspect_ratio(path, maxSize, method=Image.ANTIALIAS):
    """ Resizing an image to a static side, in place """

    try:
        
        # shrink, but maintain aspect ratio
        image = Image.open(path)   
        image.thumbnail(maxSize)
        
        # fill out to exactly maxSize (square image)        
        background = Image.new("RGBA", maxSize, (255, 255, 255, 0)) # use white for empty space
        background.paste(image, ((maxSize[0] - image.size[0]) / 2, (maxSize[1] - image.size[1]) / 2))
    
        background.save(path)
    
    except Exception, e:
        
        print "Error resizing image: %s" % e

per subdomain robots.txt in Apache

2010-10-15T09:25:00.006-04:00

When I develop a website, I tend to go subdomain crazy. If the site is crazy.net, I probably configure www.crazy.net, admin.crazy.net, static.crazy.net, youare.crazy.net, etc. Some allow concurrent logins as different users such as yourself, an admin account and maybe as a particular live user. Others are to keep your static/media resources cookie free, as well as allow parallel sockets for html/media content.

In my typical use case, all of these subdomains point to the same Apache instance. So you can go to a particular relative URL at any of the subdomains, and you will get the same page. But I certainly don't want Google to index all those subdomains; I want a single canonical domain for the site. I'm hardly an expert in Apache configuration, so it took me an hour to track down the solution to this problem.

Essentially, I wanted www.crazy.net to serve up a permissive robots.txt file, as well as a sitemap.

# robots.txt @ http://www.crazy.net/robots.txt
User-agent: *
Sitemap: http://www.crazy.net/sitemap.txt

However, all subdomains should return a different, restrictive robots.txt for the same URL.

# norobots.txt @ http://admin.crazy.net/robots.txt, http://static.crazy.net/robots.txt, etc
User-agent: *
Disallow: /

Here is a sample of my /etc/apache2/sites-available/default config file that made this happen.

ServerAdmin admin@example.com

ExpiresActive On
ExpiresByType text/css "access plus 12 years"
ExpiresByType application/javascript "access plus 12 years"
ExpiresByType image/png "access plus 12 years"
ExpiresByType image/gif "access plus 12 years"
ExpiresByType image/jpeg "access plus 12 years"
FileETag none

...

<VirtualHost *:80>
 alias /robots.txt /home/apache/website/norobots.txt
</VirtualHost>

<VirtualHost *:80>
 ServerName www.crazy.net
 alias /robots.txt /home/apache/website/robots.txt
 alias /sitemap.txt /home/apache/website/sitemap.txt
</VirtualHost>

Prior to this, all my directives were in a single VirtualHost. The key revelation on my part was that they don't need to be. Rather, Apache configs support inheritance from a global scope. So my VirtualHosts end up just being what's different between www.crazy.net and any other ServerName.

Pulling Google Analytics data into Django

2010-10-08T15:01:00.008-04:00

Website analytics tools are expected to do a lot. They need to allow you generate reports on metrics like sessions, pages/time per session, bounce rates, referrals... etc. Not only that, but they need to be able to segment those reports by logged in versus anonymous users, china versus the US, etc...

As a developer, the more you play with existing analytics tools, the more you appreciate not having to implement all this crap yourself. Just recently I was trying to track hits from social networks. My application would blast a tweet out to twitter, and then try to determine how many hits were coming back.

Determining raw hits is easy. But I had no idea how many automated bots would hit the URLS I was posting. I had nearly 1,2000 hits in just 24 hours to the same post! Here is a short list of bot user agents collected in just one day.

LinkedInBot
TwitterJobSearch.com
PycURL
labs.topsy.com
postrank.com
voyager/1.0
JS-Kit URL Resolver
Twitterbot
mxbot
urllib
ysearch
Twingly
TweetmemeBot
OneRiot
Googlebot
inagist.com
Jakarta Commons
facebookexternalhit
NjuiceBot
Yahoo! Slurp

Needless to say, tying to determine "real" hits by user agent was going to be both arduous and error prone.

Enter Google Analytics, the gold standard of web analytics packages. My marketing department had already requesting we include it into the application anyway.

It works via JavaScript, which is perfect for thwarting bots. The bots in question don't appear to execute JavaScript when they scrape a page, or if they do, Google Analytics is going above and beyond to filter them out.

<script type="text/javascript">

 var _gaq = _gaq || [];
 _gaq.push(['_setAccount', 'UA-XXXXXXX-X']);

 // custom variables
 _gaq.push(['_setCustomVar', 1, 'Current User', 'joe.example', 1]);
 _gaq.push(['_setCustomVar', 2, 'Current User ID', '75', 1]);                               
 _gaq.push(['_setCustomVar', 3, 'Record Owner ID', '101', 3]);
 _gaq.push(['_setCustomVar', 4, 'Record Type', 'Job', 3]);
 _gaq.push(['_setCustomVar', 5, 'Record ID', '87', 3]);
 _gaq.push(['_trackPageview']);

 (function() {
  var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
  ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
  var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
 })();

</script>

As you can seen, as long as I was including Google Analytics, I decided to go all out and leverage their custom variables. Basically, that allows you to slice and dice all their reports by your own custom segments. In this case, I'm allowing our marketing team to report on metrics by user and page type.

But I also needed to include some metrics right in the product itself. To my delight, it's not only possible to pull down analytics data via the Google Analytics API, it's also surprisingly easy thanks to an official Python library. Not only that, but the API is extremely flexible.

from django.conf import settings

import gdata.analytics.client
from gdata.sample_util import CLIENT_LOGIN, SettingsUtil

gdata_client = gdata.analytics.client.AnalyticsClient(
    source=settings.GOOGLE_ANALYTICS_APP_NAME
    )

def _login():
    
    settings_util = SettingsUtil(prefs={
        "email": settings.GOOGLE_ANALYTICS_USER_EMAIL,
        "password": settings.GOOGLE_ANALYTICS_USER_PASS,
    })
    settings_util.authorize_client(
        gdata_client,
        service=gdata_client.auth_service,
        auth_type=CLIENT_LOGIN,
        source=settings.GOOGLE_ANALYTICS_APP_NAME, 
        scopes=['https://www.google.com/analytics/feeds/']
        )
        
def get_views(year, week):
    
    _login()        
    data_query = gdata.analytics.client.DataFeedQuery({
        'ids': settings.GOOGLE_ANALYTICS_TABLE_ID,
        'start-date': '2010-10-01',
        'end-date': '2100-01-01',
        'dimensions': 'ga:customVarValue3,ga:customVarValue4,ga:week',
        'metrics': 'ga:pageviews',
        'filters': 'ga:customVarValue4==Job,ga:customVarValue4==Profile;ga:week==%s;ga:year==%s' % (week, year),
        'max-results': "10000"
        })
        
    return gdata_client.GetDataFeed(data_query)

Here, I'm getting page view data for October, and creating a pivot table right in the API results by record owner and record type (either Job or Profile). I'm also grouping the results by week. The API will return up to 10,000 results, but I could easily break it into smaller chunks if necessary.

<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns='http://www.w3.org/2005/Atom' xmlns:dxp='http://schemas.google.com/analytics/2009' xmlns:openSearch='http://a9.com/-/spec/opensearch/1.1/' xmlns:gd='http://schemas.google.com/g/2005' gd:etag='W/&quot;D04HQn4_fCp7I2A9Wx5VFUs.&quot;' gd:kind='analytics#data'>
 <id>http://www.google.com/analytics/feeds/data?ids=ga:35391211&amp;dimensions=ga:customVarValue3,ga:customVarValue4,ga:week&amp;metrics=ga:visits&amp;filters=ga:customVarValue4%3D%3DJob,ga:customVarValue4%3D%3DProfile&amp;start-date=2010-10-01&amp;end-date=2010-10-31</id>
 <updated>2010-10-08T12:25:33.044-07:00</updated>
 <title>Google Analytics Data for Profile XXXXXXXX</title>
 <link rel='self' type='application/atom+xml' href='https://www.google.com/analytics/feeds/data?max-results=5&amp;sort=-ga%3Avisits&amp;end-date=2010-10-31&amp;start-date=2010-10-01&amp;metrics=ga%3Avisits&amp;ids=ga%3A35391211&amp;dimensions=ga%3AcustomVarValue3%2Cga%3AcustomVarValue4%2Cga%3Aweek&amp;filters=ga%3AcustomVarValue4%3D%3DJob%2Cga%3AcustomVarValue4%3D%3DProfile'/>
 <link rel='next' type='application/atom+xml' href='https://www.google.com/analytics/feeds/data?start-index=6&amp;max-results=5&amp;sort=-ga%3Avisits&amp;end-date=2010-10-31&amp;start-date=2010-10-01&amp;metrics=ga%3Avisits&amp;ids=ga%3A35391211&amp;dimensions=ga%3AcustomVarValue3%2Cga%3AcustomVarValue4%2Cga%3Aweek&amp;filters=ga%3AcustomVarValue4%3D%3DJob%2Cga%3AcustomVarValue4%3D%3DProfile'/>
 <author>
  <name>Google Analytics</name>
 </author>
 <generator version='1.0'>Google Analytics</generator>
 <openSearch:totalResults>11</openSearch:totalResults>
 <openSearch:startIndex>1</openSearch:startIndex>
 <openSearch:itemsPerPage>5</openSearch:itemsPerPage>
 <dxp:aggregates>
  <dxp:metric confidenceInterval='0.0' name='ga:visits' type='integer' value='35'/>
 </dxp:aggregates>
 <dxp:dataSource>
  <dxp:property name='ga:profileId' value='XXXXXXX'/>
  <dxp:property name='ga:webPropertyId' value='UA-XXXXXXXXX'/>
  <dxp:property name='ga:accountName' value='Talladega Alpha'/>
  <dxp:tableId>ga:XXXXXXX</dxp:tableId>
  <dxp:tableName>reachrecruiters.com</dxp:tableName>
 </dxp:dataSource>
 <dxp:endDate>2010-10-31</dxp:endDate>
 <dxp:startDate>2010-10-01</dxp:startDate>
 <entry gd:etag='W/&quot;CUUEQX47eSp7I2A9Wx5bFEU.&quot;' gd:kind='analytics#datarow'>
  <id>http://www.google.com/analytics/feeds/data?ids=ga:35391211&amp;ga:customVarValue3=137&amp;ga:customVarValue4=Job&amp;ga:week=41&amp;filters=ga:customVarValue4%3D%3DJob,ga:customVarValue4%3D%3DProfile&amp;start-date=2010-10-01&amp;end-date=2010-10-31</id>
  <updated>2010-10-30T17:00:00.001-07:00</updated>
  <title>ga:customVarValue3=137 | ga:customVarValue4=Job | ga:week=41</title>
  <link rel='alternate' type='text/html' href='http://www.google.com/analytics'/>
  <dxp:dimension name='ga:customVarValue3' value='137'/>
  <dxp:dimension name='ga:customVarValue4' value='Job'/>
  <dxp:dimension name='ga:week' value='41'/>
  <dxp:metric confidenceInterval='0.0' name='ga:visits' type='integer' value='15'/>
 </entry>
 <entry gd:etag='W/&quot;CUUEQX47eSp7I2A9Wx5bFEU.&quot;' gd:kind='analytics#datarow'>
  <id>http://www.google.com/analytics/feeds/data?ids=ga:35391211&amp;ga:customVarValue3=138&amp;ga:customVarValue4=Job&amp;ga:week=41&amp;filters=ga:customVarValue4%3D%3DJob,ga:customVarValue4%3D%3DProfile&amp;start-date=2010-10-01&amp;end-date=2010-10-31</id>
  <updated>2010-10-30T17:00:00.001-07:00</updated>
  <title>ga:customVarValue3=138 | ga:customVarValue4=Job | ga:week=41</title>
  <link rel='alternate' type='text/html' href='http://www.google.com/analytics'/>
  <dxp:dimension name='ga:customVarValue3' value='138'/>
  <dxp:dimension name='ga:customVarValue4' value='Job'/>
  <dxp:dimension name='ga:week' value='41'/>
  <dxp:metric confidenceInterval='0.0' name='ga:visits' type='integer' value='7'/>
 </entry>
 <entry gd:etag='W/&quot;CUUEQX47eSp7I2A9Wx5bFEU.&quot;' gd:kind='analytics#datarow'>
  <id>http://www.google.com/analytics/feeds/data?ids=ga:35391211&amp;ga:customVarValue3=138&amp;ga:customVarValue4=Profile&amp;ga:week=41&amp;filters=ga:customVarValue4%3D%3DJob,ga:customVarValue4%3D%3DProfile&amp;start-date=2010-10-01&amp;end-date=2010-10-31</id>
  <updated>2010-10-30T17:00:00.001-07:00</updated>
  <title>ga:customVarValue3=138 | ga:customVarValue4=Profile | ga:week=41</title>
  <link rel='alternate' type='text/html' href='http://www.google.com/analytics'/>
  <dxp:dimension name='ga:customVarValue3' value='138'/>
  <dxp:dimension name='ga:customVarValue4' value='Profile'/>
  <dxp:dimension name='ga:week' value='41'/>
  <dxp:metric confidenceInterval='0.0' name='ga:visits' type='integer' value='3'/>
 </entry>
 <entry gd:etag='W/&quot;CUUEQX47eSp7I2A9Wx5bFEU.&quot;' gd:kind='analytics#datarow'>
  <id>http://www.google.com/analytics/feeds/data?ids=ga:35391211&amp;ga:customVarValue3=119&amp;ga:customVarValue4=Job&amp;ga:week=41&amp;filters=ga:customVarValue4%3D%3DJob,ga:customVarValue4%3D%3DProfile&amp;start-date=2010-10-01&amp;end-date=2010-10-31</id>
  <updated>2010-10-30T17:00:00.001-07:00</updated>
  <title>ga:customVarValue3=119 | ga:customVarValue4=Job | ga:week=41</title>
  <link rel='alternate' type='text/html' href='http://www.google.com/analytics'/>
  <dxp:dimension name='ga:customVarValue3' value='119'/>
  <dxp:dimension name='ga:customVarValue4' value='Job'/>
  <dxp:dimension name='ga:week' value='41'/>
  <dxp:metric confidenceInterval='0.0' name='ga:visits' type='integer' value='2'/>
 </entry>
 <entry gd:etag='W/&quot;CUUEQX47eSp7I2A9Wx5bFEU.&quot;' gd:kind='analytics#datarow'>
  <id>http://www.google.com/analytics/feeds/data?ids=ga:35391211&amp;ga:customVarValue3=44&amp;ga:customVarValue4=Job&amp;ga:week=41&amp;filters=ga:customVarValue4%3D%3DJob,ga:customVarValue4%3D%3DProfile&amp;start-date=2010-10-01&amp;end-date=2010-10-31</id>
  <updated>2010-10-30T17:00:00.001-07:00</updated>
  <title>ga:customVarValue3=44 | ga:customVarValue4=Job | ga:week=41</title>
  <link rel='alternate' type='text/html' href='http://www.google.com/analytics'/>
  <dxp:dimension name='ga:customVarValue3' value='44'/>
  <dxp:dimension name='ga:customVarValue4' value='Job'/>
  <dxp:dimension name='ga:week' value='41'/>
  <dxp:metric confidenceInterval='0.0' name='ga:visits' type='integer' value='2'/>
 </entry>
</feed>

The gdata library parses this for you into native nested dictionaries. From there, it's trivial to cache it somewhere like a database for future reporting.

Check if a block is defined in Django

2010-10-01T11:59:00.008-04:00

Very often when styling a webpage, you want to put content in a box. For example, say you have a standard box for contextual help content.

   <div class="help">
      <!--- actual html contents here --->
   </div>
   <style>
      .help { 
         padding: 1em;
         border: 1px dashed black;
      }
   </style>

A sensible way to make this re-usable in Django would be to use blocks.

   <div class="help">
      {% block help %}{% endblock %}
   </div>

But what happens if no help block is defined? With this implementation, you would still get the empty help DIV, complete with a border and padding. Of course, you could render the div inside the block, but then you're repeating yourself every time you define one.

One work-around is via css:empty.

   <style>
      .help:empty {
         disply: none;
      }
   </style>

However, that approach will not work in all browsers/versions. Also, it's muddying the separation between content and style.

Ideally, you would be able to omit the div at the base template level if the block is not defined. I wondered if there was an way to check if a block is defined. While hunting around, I found the next best thing. The following code (from kcarnold) will capture the contents of a block to a variable. You can then conditionally render it only if it exists and is not empty.

from django import template

register = template.Library()

@register.tag(name='captureas')
def do_captureas(parser, token):
    try:
        tag_name, args = token.contents.split(None, 1)
    except ValueError:
        raise template.TemplateSyntaxError("'captureas' node requires a variable name.")
    nodelist = parser.parse(('endcaptureas',))
    parser.delete_first_token()
    return CaptureasNode(nodelist, args)

class CaptureasNode(template.Node):
    def __init__(self, nodelist, varname):
        self.nodelist = nodelist
        self.varname = varname

    def render(self, context):
        output = self.nodelist.render(context)
        context[self.varname] = output
        return ''

 {% captureas help_content %}{% spaceless %}{% block help %}{% endblock %}{% endspaceless %}{% endcaptureas %}

 {% if help_content %}
    <div id="help">
        {{ help_content }}
    </div>
 {% endif %}

Django actually may add this functionality natively in the future.

In-context FORM help using jQuery

2010-09-24T12:11:00.005-04:00

The Apple store checkout form redesign introduced me to the concept of in-context error messages.

Basically, the new context bubble element is being used to tell the user more about this field, and potentially answer questions the user may have about how to fill out the field correctly.

I wanted to get the same effect on a site I'm working on, but for all fields, not just fields that have failed validation. Basically, I liked showing more information while the user is in the context of editing a field, and I liked the visual metaphor.

Here is what my version looks like.

I initially tried to position the bubble using just CSS, but this proved impossible. Instead, I used jQuery to get the position of the parent element, and also to compensate for the scrollbar offset. Here is the full code for the example.

<form id="myform">
 <label>Password:</label>
 <input name="password" value="">
</form>

<div id="contextual-help">
    <div class="message-box">
        <div class="close" title="Close contextual help">
            x
        </div>
        <div class="message">
        </div>
    </div>
    <div class="triangle"></div>
</div>

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>

<style>
 #myform { margin: 200px; }
 #contextual-help { position: absolute; display: none; }
 #contextual-help .message-box { 
     border-radius: 20px; -moz-border-radius: 20px; -webkit-border-radius: 20px; 
     background-color: #FFFE9D; border: 1px solid #A9A809;  padding: 1em;    
     width: 400px;
     font-size: 120%;
 }
 #contextual-help .triangle {
  background: url("contextual-help-triangle.png") no-repeat;
  margin-top: -1px;
  margin-left: 40px;
  width: 31px;
  height: 18px;
 }
 #contextual-help .close {
     cursor: pointer;
     float: right; 
     margin-top: -.7em; 
     margin-right: -.5em;
     color: #A9A809;
     font-size: 120%;
     font-weight: bold;
 }
</style>

<script>

 $(document).ready(function() {
 
  $("#contextual-help .close").click(function () {
   close_context_help();
  });
 
  bind_help($("#myform input[name=password]"),
   "Must be more than 5 characters."
   ); 
 });

 function close_context_help() {
  $("#contextual-help").hide();
  $("#contextual-help").css({top: '', left: ''}); // fix for "drifting" on IE/Chrome  
 }
 
 function bind_help(input_element, message) {

 
  input_element
   .focus(function () {
    var contextualHelp = $("#contextual-help");
    contextualHelp.find(".message").html(message);   
    var inputOffset = $(this).offset(); // top, left
    var scrollTop = $(window).scrollTop(); // how much should we offset if the user has scrolled down the page?
    contextualHelp.offset({ 
     top: inputOffset.top + scrollTop - contextualHelp.height() - 2, 
     //left: (inputOffset.left + .5 * $(this).width()) - .5 * contextualHelp.width()
     left: inputOffset.left + 20
    });   
    contextualHelp.show();
   })
   .blur(function () {
    close_context_help();
   })
   .keyup(function (event) { // keydown does not work in Firefox, keypress does not work in Chrome
    // escape
    if (event.keyCode == 27) {
     close_context_help(); 
    }
   })
   .keypress(function (event) {
    // escape
    if (event.keyCode == 27) {
     close_context_help(); 
    }
   })  
  ;
 }
</script>

Finally, a like to contextual-help-triangle.png to complete the styling.

Show username in Django 500 error emails

2010-09-17T13:09:00.002-04:00

By default, Django will send an email to the site admins when a 500 error occurs. These emails contain all kinds of great information such as stacktrace, HTTP headers and cookie data. Unfortunately, it does not show which user the error occurred for. This can make errors harder than necessary to reproduce.

Actually, there IS enough data in the email to tie back to a user. Specifically, the email includes the sessionID. You can lookup a username by sessionID, as long as the session is still active. But why should you have to?

Instead, here is a simple hack to include the username in the cookies, which in turn are displayed in the error email.

@login_required
def home(request):                    
    # other view code here
    response = render_to_response("home.html", RequestContext(request, locals()))
    response.set_cookie("username", request.user.username)
    return response

Ideally, you would do this just after the user logs in. Here, I did it on the home page for a logged in user, which is the first page they are redirected to post login.

Django log form ValidationErrors

2010-09-10T13:29:00.002-04:00

Forms are the primary way your users communicate with your website. Hopefully, the users are always entering data in the correct format, and the form always accepts their submissions. In the real world, users often encounter form validation errors.

In some ways, this is a good thing. At least you're getting valid data, in the end. However, too many validation errors can lead to form abandonment, ie the user may give up. When that happens, don't you want to know?

In Django, there is a relatively easy way to log any form validation errors that happen. Simply over-ride the is_valid() function on your forms.

class MyForm(Form):             
    def is_valid(self):
        is_valid = super(MyForm, self).is_valid(self)
        if not is_valid:
            for field in self.errors.keys():
                print "ValidationError: %s[%s] <- \"%s\" %s" % (
                    type(self),
                    field,
                    self.data[field],
                    self.errors[field].as_text()
                )
        return is_valid

This will produce the following output in your logs, which you can then grep and even email to yourself on a regular basis if you so desire.

ValidationError: <class 'website.views.profiles.Step1Form'>[password] <- "" * This field is required.
ValidationError: <class 'website.views.profiles.Step1Form'>[email] <- "aaa" * Enter a valid e-mail address.

Django HTML5 input placeholders

2010-09-03T10:05:00.008-04:00

HTML5 has a bunch of nifty progressive enhancements to forms, one of which is placeholder text. Here is an example, along with the required code.

<form style="float: right; margin: 1.75em;">
   <input size="38" placeholder="Your browser supports placeholder text">
</form>

Placeholder text is displayed inside the input field as long as the field is empty and not focused. As soon you click on (or tab to) the input field, the placeholder text disappears. - Mark Pilgrim

I wanted to leverage this feature using Django's built-in form generation. Ie, I wanted to be able to call form.as_table, and have placeholder values pulled either from the model field's help_text, or from the form definition.

My first problem was that help_text currently just dumps out into the form body. I needed a way to disable, or at least hide, those elements. There is an open feature request to wrap those elements in a class, so that they can be easily styled/hidden. While that feature will make it into Django 1.3, here is a solution in the meantime.

def wrap_helptext_as_table(self):
    return self._html_output(
        u'<tr><th>%(label)s</th><td>%(errors)s%(field)s%(help_text)s</td></tr>', 
        u'<tr><td colspan="2">%s</td></tr>', 
        '</td></tr>', 
        u'<div class="help_text">%s</div>', 
        False,
        )

class Html5Form(Form):   
          
    def as_table(self):
        return wrap_helptext_as_table(self)
     
class Html5ModelForm(ModelForm):
            
    def as_table(self):
        return wrap_helptext_as_table(self)    

# .help_text { display: none; } will now hide help_text elements

These two subclasses, one for regular forms and one for model forms, comprised the smallest change to the base classes that I could come up with to wrap help_text in a div with a css class. On to inserting the HTML5 placeholder attribute.

class Html5ModelForm(ModelForm):
            
    def as_table(self):
        return wrap_helptext_as_table(self)        
        
    def __init__(self, data=None, files=None, auto_id='id_%s', prefix=None,
                 initial=None, error_class=ErrorList, label_suffix=':',
                 empty_permitted=False, instance=None):
                             
        super(Html5ModelForm, self).__init__(data, files, auto_id, prefix, 
            initial, error_class, label_suffix, 
            empty_permitted, instance)
            
        # create an HTML5 placeholder attribute based on the field help_text 
        for field_name in self.fields:
            field = self.fields.get(field_name)
            if field:
                if type(field.widget) == TextInput:
                    field.widget.attrs["placeholder"] = field.help_text

That's a complete solution for model forms. For regular forms, all you have to do is the following in the form definition.

class MyForm(Form):
    name = forms.CharField(widget=forms.TextInput({ "placeholder": "Joe Recruiter" }))

Chained Select Progressive Enhancement

2010-08-18T11:21:00.008-04:00

Perhaps you have encountered a chained select control in your travels. The canonical example would be on Monster or CareerBuilder, picking your area of expertise. Maybe you first select "Technology", then the second menu contains only technology skills, and you select "Web Developer", etc. It most useful when there are a large number of sub-categories, and you don't want to show all the irrelevant options to the user.

This control is not one of the standard web form controls, perhaps because it's really a composite of two or more select controls. Composites are harder to style, and in this case don't even map cleanly to FORM variables. How do you define two forms fields in a parent/child one to many relationship, semantically? You have to resort to hacks like field name prefixes.

Any chained select implementation is going to involve javascript. But every implementation I could find also requires javascript. In other words, if the user has javascript disabled, or there is a bug in your javascript, they will not be able to fill out the form.

What follows is the beginnings of a chained select progressive enhancement, aka a unobstrusive javascript version. It starts with a nested HTML list. That's the baseline version.

As I mentioned, I decided on an arbitrary prefix for the input values. In this case, the format is categoryID:subcategoryID. You could potentially just post the subcategoryID of course, if your back-end could pull the category automatically.

 <ul class="chained-select chained-select-style">
  <li class="category">
   <label>Customer Service/Technical Support</label>
   <ul class="subcategory">
    <li><input type="radio" name="category" value="1:1">All Customer Service/Technical Support</li>
   </ul>
  </li>
  <li class="category">
   <label>Engineering (Non-IT)</label>
   <ul class="subcategory">
    <li><input type="radio" name="category" value="2:1">Architectural</li>
    <li><input type="radio" name="category" value="2:2">Computer Aided Design</li>
    <li><input type="radio" name="category" value="2:3">Construction</li>   
    <li><input type="radio" name="category" value="2:4">Civil Engineering</li>
    <li><input type="radio" name="category" value="2:5">Chemical Engineering</li>
    <li><input type="radio" name="category" value="2:6">Computer Aided Design (CAD)</li>
    <li><input type="radio" name="category" value="2:7">Construction</li>
    <li><input type="radio" name="category" value="2:8">Environmental Engineering</li>   
    <li><input type="radio" name="category" value="2:9">Civil Engineering</li>
    <li><input type="radio" name="category" value="2:10">Project/Program Management</li>
   </ul>
  </li>
  <li class="category">
   <label>Information Technology</label>
   <ul class="subcategory">
    <li><input type="radio" name="category" value="3:1">Business Systems Analysis</li>
    <li><input type="radio" name="category" value="3:2">ERP Applications</li>
    <li><input type="radio" name="category" value="3:3">Network Operations</li>   
    <li><input type="radio" name="category" value="3:4">Project/Program Management</li>
    <li><input type="radio" name="category" value="3:5">Security</li>
    <li><input type="radio" name="category" value="3:7">Software Development</li>
    <li><input type="radio" name="category" value="3:8">Quality Assurance</li>
    <li><input type="radio" name="category" value="3:9">Technical Writing/Documentation</li>   
    <li><input type="radio" name="category" value="3:10">Product Management</li>
    <li><input type="radio" name="category" value="3:11">Web Design</li>
   </ul>
  </li>
 </ul>

Even the baseline should look decent, however. Here is my basic styling. It also includes some classes which will only be set later by the javascript.

<style>
   #demo { width: 900px; }

   .chained-select-style li { list-style: none; float: left; }
   .chained-select-style li ul li { display: inline; }

   .chained-select-style li.category { font-weight: bold; }
   .chained-select-style .subcategory {font-weight: normal; }
   .chained-select-style li ul li { padding: 5px; margin: 5px; margin-right: 10px; border: 1px solid #B0B0B0; background-color: #F0F0F0; }
   .chained-select-style li ul li.selected { background-color: #F0F0B0; }

</style>

Now, it looks more like this.

The progressive enhancement uses jQuery, and basically just does the following.

Toggles a "selected" class when an individual item is checked off.
Hides the category labels and creates a select control with their values.
Hides the subcategory lists.
Attaches a behaviour to the new select control to show the relevant subcategory list when the select value changes.

<script>
 $(document).ready(function () {

  function select_changed() {
   if ($(this).is(":checked")) {
    if ($(this).is("[type=radio]"))
     $(this).parents(".chained-select-style").find("li").removeClass("selected");
    $(this).parent().addClass("selected");
   }
   else $(this).parent().removeClass("selected");   
  }

  var items = $(".chained-select-style input");
  items.change(select_changed);
  $.each(items.filter(":checked"), select_changed);

  function chained_select(elements) {
   $.each(elements, function () {
    var top = $("<select class='chained-select-top'>");
    $.each($(this).find(".category label"), function () {
     // escape the contents
     top.append($("<option></option>").text($(this).text()));
    });
    $(this).before(top);
   });
   elements.find(".category label").add(elements.find(".subcategory")).hide();   
   function chained_select_change() {
    var label_text = $(this).val();
    $.each(elements.find(".category label"), function () {
     if ($(this).text() == label_text) {
      $(this).next().show(); 
     } else {
      $(this).next().hide();
     }
    });
   }
   var items = $(".chained-select-top");
   items.change(chained_select_change);
   $.each(items, chained_select_change);
  }
  chained_select($(".chained-select"));
 });
</script>

What you end up with is the following. You could certainly take this one step further and convert the subcategory list into the traditional second select control. I just like the way this looks better.

Redirect console output to a Django HttpResponse

2010-08-06T14:29:00.003-04:00

Say you have some Python code that prints to the console, and you want to re-use that code in a Django view, but output the print statements as HTML in the HttpResponse. Perhaps, like me, you have a batch job that you're calling from ./manage.py script that you would also like to call from an HTTP request and see the results as text/HTML. Here is a decorator that does just that.

import sys
from django.http import HttpResponse

def print_http_response(f):
    """ Wraps a python function that prints to the console, and 
    returns those results as a HttpResponse (HTML)"""
    
    class WritableObject:
        def __init__(self):
            self.content = []
        def write(self, string):
            self.content.append(string)     
    
    def new_f(*args, **kwargs):
        printed = WritableObject()
        sys.stdout = printed
        f(*args, **kwargs)
        sys.stdout = sys.__stdout__    
        return HttpResponse(['<BR>' if c == '\n' else c for c in printed.content ])         
    return new_f

If you attached it like so to a view...

@print_http_response
def my_view(request):
   print "some output here"
   for i in [1, 2, 3]:
      print i

It would output the following HTML:

some output here<BR>
1<BR>
2<BR>
3<BR>

Blank Canvas brings @require support to Chrome

2010-07-30T10:16:00.003-04:00

Quick shout-out to the Blank Canvas Script Handler, a relatively unknown Google Chrome extension that brings @require support to userscripts in Chrome.

That just leaves NoScript, and I might actually switch to Chrome full-time.

Django zip files (create dynamic in-memory archives with Python's zipfile)

2010-07-23T14:08:00.005-04:00

Usually, Django should not be used to serve static files.

The reasoning here is that standard Web servers, such as Apache, lighttpd and Cherokee, are much more fine-tuned at serving static files than a Web application framework. - Django documentation

However, dynamic files are a different story. Say you want to wrap more than one file being generated on the fly from a form POST. In my case, I wanted to let the user download a zip archive of a few generated reports.

Using Python's zipfile it's easy to create the archive in-memory, without writing to disk. Then, serving that file as a zip content-type is easy in Django.


from StringIO import StringIO
from zipfile import ZipFile
from django.http import HttpResponse

def download(request, company_id):     
    
    in_memory = StringIO()
    zip = ZipFile(in_memory, "a")
        
    zip.writestr("file1.txt", "some text contents")
    zip.writestr("file2.csv", "csv,data,here")
    
    # fix for Linux zip files read in Windows
    for file in zip.filelist:
        file.create_system = 0    
        
    zip.close()

    response = HttpResponse(mimetype="application/zip")
    response["Content-Disposition"] = "attachment; filename=two_files.zip"
    
    in_memory.seek(0)    
    response.write(in_memory.read())
    
    return response

Free search string tokenization in Python

2010-07-16T15:51:00.004-04:00

Want to do some simple lex parsing in Python? Using shlex, you may be able to get something that meets your requirements almost for free. Here is an example I used recently to parse a search string. The requirements were that tokens could be separated by spaces or commas, and double-quotes denotes a single token.

import shlex 

def _tokens(query):
    return shlex.split(str(query))

Examples:


>>> _tokens("java, perl, c++")
['java,', 'perl,', 'c++']

>>> _tokens("java perl c++")
['java', 'perl', 'c++']

>>> _tokens("java perl c++ \"Phil's Staffing\"")
['java', 'perl', 'c++', "Phil's Staffing"]

Django/Celery Quickstart (or, how I learned to stop using cron and love celery)

2010-07-09T12:24:00.012-04:00

Websites often need tasks that run periodically, behind the scenes. Examples include sending email reminders, aggregating denormalized data and permanently deleting archived records. Very often the simplest solution is to setup a cron job to hit a URL on the site that performs the task.

Cron has the advantage of simplicity, but it's not not ideal for the job. You have to take steps to ensure that regular users of the site cannot hit those URLs directly. It also forces you to manage an external configuration. What if you forget to perform the configuration on the qa or production servers? It would be safer and easier if the configuration was in the code for the site.

For Django sites, celery seems to be the solution of choice. Celery is really focused on being a distributed task queue, but it can also be a great scheduler. Their documentation is excellent, but I found that they lack a quickstart guide for getting started with Django and celery, just for replacing cron.

Note: Celery typically runs with RabbitMQ as the back-end. For just task scheduling, this may be overkill. This guide starts out using ghettoq, which is backed by the database Django is already using.

Install django-celery, ghettoq

easy_install django-celery
easy_install ghettoq

Edit settings.py, and add the celery config info


CARROT_BACKEND = "ghettoq.taproot.Database"

INSTALLED_APPS = (
    ...
    'djcelery',
    'ghettoq',
)

Add the new tables to the Django database
```
./manage.py syncdb
```

Create a file, tasks.py in your project (same level as models.py)

from celery.task.schedules import crontab
from celery.decorators import periodic_task

# this will run every minute, see http://celeryproject.org/docs/reference/celery.task.schedules.html#celery.task.schedules.crontab
@periodic_task(run_every=crontab(hour="*", minute="*", day_of_week="*"))
def test():    
    print "firing test task"

Start the celery daemon in "beat" mode, which is required for scheduling
```
sudo ./manage.py celeryd -v 2 -B -s celery -E -l INFO
```

At this point, you should see your celery tasks in the console output, and you should see the task firing every minute.


[2010-07-09 12:57:36,520: WARNING/MainProcess] celery@chase v2.0.0 is starting.
[2010-07-09 12:57:36,520: WARNING/MainProcess] /usr/local/lib/python2.6/dist-packages/celery-2.0.0-py2.6.egg/celery/bin/celeryd.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in a production environment!
  warnings.warn("Using settings.DEBUG leads to a memory leak, "
[2010-07-09 12:57:36,525: WARNING/MainProcess] Configuration ->
    . broker -> ghettoq.taproot.Database://None@None/
    . queues ->
        . celery -> exchange:celery (direct) binding:celery
    . concurrency -> 2
    . loader -> djcelery.loaders.DjangoLoader
    . logfile -> [stderr]@INFO
    . events -> ON
    . beat -> ON
    . tasks ->
 . myproject.tasks.test
[2010-07-09 12:57:36,553: INFO/PoolWorker-3] child process calling self.run()
[2010-07-09 12:57:36,555: INFO/PoolWorker-2] child process calling self.run()
[2010-07-09 12:57:36,560: WARNING/MainProcess] celery@chase has started.
[2010-07-09 12:57:36,564: INFO/_Process-1] child process calling self.run()
[2010-07-09 12:57:36,564: INFO/_Process-1] ClockService: Starting...
[2010-07-09 12:58:37,841: INFO/MainProcess] Got task from broker: search.tasks.test[6dfafd0f-f92d-41ac-96de-63888f6c1f38]
[2010-07-09 12:58:37,864: WARNING/PoolWorker-3] firing test task
[2010-07-09 12:58:37,871: INFO/MainProcess] Task search.tasks.test[6dfafd0f-f92d-41ac-96de-63888f6c1f38] processed: None
[2010-07-09 12:59:39,242: INFO/MainProcess] Got task from broker: search.tasks.test[d7bd0592-51c1-4560-8d39-19b25a1ec43b]
[2010-07-09 12:59:39,375: WARNING/PoolWorker-3] firing test task
[2010-07-09 12:59:39,379: INFO/MainProcess] Task search.tasks.test[d7bd0592-51c1-4560-8d39-19b25a1ec43b] processed: None

If you want, you can upgrade to RabbitMQ. Just make sure to update your setting.py, as well.

You may also want to run celeryd as a service.

better unobstrusive javascript element hiding

2010-06-22T17:22:00.004-04:00

Many web pages have elements that are only displayed at the users request. Common examples are Facebook's chat tab, Gmail's "more" folders link and Yahoo's dashboard toggles. Typically, these elements are already loaded onto the page, and simply hidden by style sheets. Then when you click on a certain control, a piece of JavaScript un-hides the element.

But what about users who don't have JavaScript enabled? Ideally, you would want to allow them to access the content anyway. Sure, the experience is better with JavaScript, but that's no reason to break it for everyone else. This concept of graceful degradation is known as unobstrusive Javascript.

A typical solution would be to leave the element visible initially via the style sheet, but immediately hide if when the page loads with JavaScript. That way, users with JavaScript can dynamically un-hide it, and users without JavaScript will simply always see it.

The problem with this approach is that it can lead to a flicker effect when the page is first loaded. When JavaScript hides the content while the page is still loading, it may have already been displayed to the user. The user sees the page reconfigure itself, and may interpret this as a problem with the site. Luckily, there is a a better way.

A better way to hide this content dynamically is to output a style sheet dynamically from the HTML HEAD element.

<html xmlns="http://www.w3.org/1999/xhtml">
 <head>
  <title>My Page Title</title>
  <script>
      // check for WC3 standard DOM compliance
      if (document.getElementById) {
          document.write("<style type=\"text/css\">.hideme { display: none; }</style>");
      }
  </script>
 </head>

This will hide any element with the "hideme" class. Because it's executed inside the HEAD tag, it will process before the page itself loads. No flickering. If the user does not have JavaScript, then they will simply see the element.

Add a button to Django admin to login as a user (without the password)

2010-06-09T13:02:00.003-04:00

Django correctly stores user passwords as md5 hashes by default. This is great for security; there is zero chance that a password could be exposed via flaw in the site, attack, disgruntled employee, whatever. But what if you had a use case where you wanted to login as user without a password?

The use case I have in mind is allowing admin users to login as a user via the Django admin application. This could be very useful for reproducing bugs or verifying what a particular user is seeing. Without knowing the user's password, the only way for an admin to login as them would be to reset their password, login, do their bussiness, and then email the user the new password. Hardly ideal.

Adding a button to the user page in admin is easy. The user model is in the auth application, so all you have to do is create a file called admin/auth/change_form.html in your templates directory. There, you can extend the base change_form.html for the User model. Note: root around in the /usr/lib/pymodules/python2.6/django/contrib/admin/templates directory for an idea of what files you can extend.

{% extends "admin/change_form.html" %}

{% block object-tools %}
{% if change %}{% if not is_popup %}
  <ul class="object-tools">
    <li><a href="history/" class="historylink">History</a></li>
    <li><a href="/login/user/{{ object_id }}?hash={{ 'user'|hash:object_id }}">Login</a></li>
  </ul>
{% endif %}{% endif %}
{% endblock %}

In this case, the relative URL for the login would be the /login/user/$id. If you made the URL absolute, you could provide an alternate domain name, which would allow you a separate cookie, so you could be logged in as different users in both admin and the application at the same time.

What's that hash parameter? It's just a security feature to make sure an attacker cannot access this URL without knowing the secret key. The filter definition looks like this:

from django import template

register = template.Library()  

@register.filter()
def hash(type, id):
    hash = hashlib.md5()
    hash.update("%s:%s:%s" % (type, id, settings.ADMIN_HASH_SECRET))
    return hash.hexdigest().upper()

The URL is routed in typical fashion via urls.py.

    url(r'^login/user/(?P<user_id>[\d_]+)$', admin.login_as_user),

Finally, here is the view that implements the login securely.

from django.conf import settings
from django.http import HttpResponseRedirect
from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.contrib.auth import login, authenticate

# the same filter that I called in the template
from search.helpers.tags import logic

def login_as_user(request, user_id):

    # security check; don't let unauthorised people login
    request_hash = request.REQUEST.get("hash", "")
    if request_hash != logic.hash("user", user_id):
        raise Exception("invalid hash value")
    
    user = User.objects.get(id=user_id)
    
    # ADMIN_HASH_SECRET is set in settings.py, can be any secret string 
    user = authenticate(username=user.username, password=settings.ADMIN_HASH_SECRET)
    login(request, user)    
    
    return HttpResponseRedirect(reverse("home"))

The Django login() method does the work of logging the user in against whatever backed you have configured, just as if they logged in manually. However, authenticate() is necessary, and by default requires that the actual user's password be passed in. As mentioned previously, this is a big problem because we don't know the user's password; it's stored as a one-way hash.

It turns out to be not such a big problem after all, as Django provides an easy mechanism to extend the authentication module. First, you define your authenticator.

from django.conf import settings
from django.contrib.auth.models import User

class LoginAsUserBackend:
    """
    Allows admins to login as a user without knowing the password.     
    Will authenticate any username, given the password of settings.ADMIN_HASH_SECRET
    """
    
    def authenticate(self, username=None, password=None):
        if settings.ADMIN_HASH_SECRET != "" and password == settings.ADMIN_HASH_SECRET:
            try:
                return User.objects.get(username=username)
            except:
                pass
        return None

    def get_user(self, user_id):
        try:
            return User.objects.get(pk=user_id)
        except:
            return None

Django authenticators are called in serial; so my version will be called first, and then if that fails the base Django authenticator will have a go. In my case, I'm allowing any user to login with the secret stored in the settings file. My reasoning is that if they know that secret, they would be able to exploit my new login mechanism anyway.

Then, you just add your new authenticator into the mix in settings.py.

...
AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
    'my_application.path.to.my.authenticator.LoginAsUserBackend'
    )
...

Parsing invalid xml with Beautiful Soup

2010-06-04T12:16:00.003-04:00

Sometimes, bad xml happens to good people. In my case, I was getting a text stream back from a web-service call that proported to be xml, but was actually not well-formed. It had ampersands inside a tag.

 <?xml version="1.0" encoding='UTF-8'?>
 <resume>
   <title>Developer & Manager</title>
    ...
 </resume>

This lead to the following error parsing with python's minidom:

Traceback (most recent call last):
  ...
    response = minidom.parseString(xml)
  File "/usr/lib/python2.6/xml/dom/minidom.py", line 1928, in parseString
    return expatbuilder.parseString(string)
  File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 940, in parseString
    return builder.parseString(string)
  File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
ExpatError: not well-formed (invalid token): line 369, column 2025

Virtually all XML parses will rightly balk at this input, because it's not valid. You could easily work-around this issue by replacing all ampersands with the html-entity &, but the real issue is that the web-service is obviously not using an XML parser to create the document. It's likely creating the document by hand, which means that further cases of invalid XML are quite likely.

A more robust solution is to use a lenient parser like Beautiful Soup, which is actually an HTML parser. Even though it doesn't know anything about XML, it's enough for basic parsing. Beautiful Soup will allow ampersands (which are valid in HTML anyway), unclosed tags, bad encodings or virtually anything else. It's designed to make a best effort no matter what. It's also easy to use.

from BeautifulSoup import BeautifulSoup

response = BeautifulSoup(xml)
print response.find("title").string

Django: ManyToManyField on ModelForm as checkbox widget

2010-05-20T16:10:00.002-04:00

By default, Django will use a mutli-select widget for rendering a ManyToManyField on a ModelForm. Switching it out for checkboxes is simple.


from django.db import models
from django.forms.models import ModelForm
from django.forms.widgets import CheckboxSelectMultiple

class Company(models.Model):  
    industries = models.ManyToManyField(Industry, blank=True, null=True)

class CompanyForm(ModelForm):
    
    class Meta:
        model = Company
        fields = ("industries")
             
    def __init__(self, *args, **kwargs):
        
        super(CompanyForm, self).__init__(*args, **kwargs)
        
        self.fields["industries"].widget = CheckboxSelectMultiple()
        self.fields["industries"].queryset = Industry.objects.all()

Reuse Django's filter_horizontal admin widget

2010-05-14T13:44:00.005-04:00

The HTML select multiple control sucks mightily. Though it's a standard form widget, regular users seem to have usability problems with it. Specifically, it's easy to forget to control-click to add a new item, and you end up removing anything previously added. Also, it's hard to find options in large unsorted lists.

There are many projects out there that can transform a multiselect into a more advanced widget. Typically they operate via JavaScript, replacing the vanilla control on the fly client-side. The server-side receives the same form variables. This has the advantage of being a drop-in replacement, and of degrading nicely if the user does not have JavaScript enabled.

Django has a particularly nice one built into it's admin interface. It turns a multi-select such as...

into this...

This comes complete with add/remove all, and client-side search capability. Because this is all JavaScript, it's easy to remove from Django admin and integrate into any page. I found a decent guide online, but I think we can do it in fewer steps.

Download django-admin.multiselect.js, and include in your page.
Download django-admin-widgets.css and include in your page.

Then, all you have to do is initialise it for any multi-selects you have.

 jQuery.each($("select[multiple]"), function () {
  // "Locations" can be any label you want
  SelectFilter.init(this.id, "Locations", 0, "/media/");
 });

automatic image resize in Django for thumbnails

2010-05-07T16:29:00.004-04:00

Want to resize images uploaded to Django on the fly? It's actually very simple. This solution requires the Python Imaging Library (PIL).


import Image

class Company(models.Model):

    logo = models.ImageField(upload_to="static/images/logos")

    def save(self, force_insert=False, force_update=False):
        
        super(Company, self).save(force_insert, force_update)

        if self.id is not None:
            previous = Company.objects.get(id=self.id)
            if self.logo and self.logo != previous.logo:
                image = Image.open(self.logo.path)
                image = image.resize((96, 96), Image.ANTIALIAS)
                image.save(self.logo.path)

Updated: Django 1.2 requires that you save the file FIRST, otherwise the path will not be correct.

Django ManyToMany error: "Cannot resolve keyword XXX into a field"

2010-04-30T09:59:00.004-04:00

This week I was doing some refactoring, and started getting the following exception in Django's admin site. This was under Django 1.1.1, and Python 2.6.4.

Traceback:
File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py" in get_response
  92.                 response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/sites.py" in root
  490.                 return self.model_page(request, *url.split('/', 2))
File "/usr/lib/pymodules/python2.6/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/sites.py" in model_page
  509.         return admin_obj(request, rest_of_url)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/options.py" in __call__
  1098.             return self.change_view(request, unquote(url))
File "/usr/lib/pymodules/python2.6/django/db/transaction.py" in _commit_on_success
  240.                 res = func(*args, **kw)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/options.py" in change_view
  840.             form = ModelForm(instance=obj)
File "/home/chase/bullhorn/branches/powerfill/django/powerfill/search/admin.py" in __init__
  113.         super(CompanyAdminForm, self).__init__(*args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/forms/models.py" in __init__
  222.             object_data = model_to_dict(instance, opts.fields, opts.exclude)
File "/usr/lib/pymodules/python2.6/django/forms/models.py" in model_to_dict
  140.                 data[f.name] = [obj.pk for obj in f.value_from_object(instance)]
File "/usr/lib/pymodules/python2.6/django/db/models/fields/related.py" in value_from_object
  964.         return getattr(obj, self.attname).all()
File "/usr/lib/pymodules/python2.6/django/db/models/manager.py" in all
  105.         return self.get_query_set()
File "/usr/lib/pymodules/python2.6/django/db/models/fields/related.py" in get_query_set
  424.             return superclass.get_query_set(self)._next_is_sticky().filter(**(self.core_filters))
File "/usr/lib/pymodules/python2.6/django/db/models/query.py" in filter
  498.         return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/db/models/query.py" in _filter_or_exclude
  516.             clone.query.add_q(Q(*args, **kwargs))
File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py" in add_q
  1675.                             can_reuse=used_aliases)
File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py" in add_filter
  1569.                     negate=negate, process_extras=process_extras)
File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py" in setup_joins
  1737.                             "Choices are: %s" % (name, ", ".join(names)))

Exception Type: FieldError at /admin/search/company/2273/
Exception Value: Cannot resolve keyword 'company' into field.

Having made quite a number of changes before noticing this, it took some time to track down. It turned out that it started happening when I moved an import. Eventually, I found Django Ticket #1796. While that ticket is marked as "Fixed", it does not actually appear to be fixed in all cases.

The problem is deep in the Django stack, and involves class loading at the Python level as well. It's in a piece of the Django code that is only executed for ManyToMany relationships. Read the ticket if you're interested in the details.

In my case, the ManyToMany relationship in question was a field on a Company model which referenced a Django User model:

class Company(models.Model): 

    ...

    connection_users = models.ManyToManyField(
        User,
        symmetrical=False,
        blank=True,
        null=True
        )

I also have a UserProfile model which extends the base User model:

class UserProfile(models.Model):              
    user = models.ForeignKey(User, unique=True)
    company = models.ForeignKey(Company)
    phone = PhoneNumberField()

Essentially, the django.contrib.auth.models.User model is necessarily loaded first, then my related UserProfile model is loaded before the Company model due to Python class loading behavior. When Company finally loads, the Django bug kicks in and it confuses the company field on UserProfile with a symmetric version of the ManyToMany relationship connection_users on Company.

After much fiddling, the fix was to force the class loading order to load Company before UserProfile. A good spot to do this is at the top of the models module.

from company import Company, UserProfile

Slicehost: Six month review

2010-04-23T15:09:00.003-04:00

Six months ago, I opened an account with Slicehost for a work project. They provide a VPS service, which is essentially just a virtual machine in the cloud. In our case, we are hosting a website with a traditional LAMP stack (but where the P is Python).

We decided on a $250/mo "slice", which gets you 4GB or RAM on any 64-bit Linux distro you want, as well as more bandwidth than you can shake a stick at. They offer some nice web-based tools to clone VMs, manage DNS and reboot servers. But that's about it. Everything else you configure yourself, including firewalls, databases, webservers, etc.

That's just fine with me. All I ever want from my ISP is to be dumb pipe. I don't want your custom software on my PC (wait, what do you mean you don't "support" Linux?). Please don't throttle my connection, etc. Likewise, all I want from a VPS is a dumb terminal.

Reliability has been awesome. Speed is great. Bandwidth seems solid. So far, the only killer feature I've used is the emergency console. Having been locked out of the machine by a bad firewall config change, we were able to recover access without Slicehost support using their awesome Ajax based root login.

Any bad news? Nope, not so far. But be warned: slices are like potato chips. We started thinking we would only need one, but that "clone" button is just too damn easy! All it takes is one click to spin off another instance, already configured and ready to go. So far, we spun off a demo site AND a skunkworks box.

click on a link (follow) via jQuery/JavaScript

2010-04-09T11:41:00.010-04:00

You might reasonably assume that it would be possible to fire a click event on an "A" element (link) via JavaScript, at which point the browser would fire any click event handlers currently attached, or otherwise just follow the link. But you would be wrong, at least in the case of Firefox.

The click method is intended to be used with INPUT elements of type button, checkbox, radio, reset or submit. Gecko does not implement the click method on other elements that might be expected to respond to mouse–clicks such as links (A elements), nor will it necessarily fire the click event of other elements. - Mozila Documentation

Why is that? It seems that Firefox is following the standard quite strictly, whereas IE has no problems automating link clicks. There may also be a sandbox security argument here, but personally I don't buy that as long as we are allowing JavaScript to fire FORM POSTS.

Regardless, a cross browser solution is needed. At first blush, just setting window.location directly seems to work.

   window.location = $("a#my-link").attr("href");

This naive solution has serious drawbacks; IE will not post a HTTP Referrer header, and the back button may skip pages navigated to this way. However, it may be an acceptable solution depending on your application.

If you have control over the HTML itself, a better solution is to just use FORMs instead. It's trivial to submit a form via jQuery. It's even possible to make the button look somewhat like a HTML link.


<!-- was <a href="/my-link-url">Click me</a> -->
<form id="my-form" action="/my-link-url" action="get">
   <input class="button-link" type="submit" value="Click me" />
</form>

<script>
   // simulate a user clicking the "link"
   $(".button-link").click();

   // you could also just submit the form directly, depending on what's easier
   $("form#my-form").submit();
</script>

<style>
   .button-link {
      background-color:white;
      border:0;
      color:blue;
      text-decoration:underline;
      font-size:1em;
      font-family:inherit;
      cursor:pointer;
   }
</style>

Ubuntu keyboard shortcut cheatsheet

2010-04-02T11:03:00.004-04:00

Everyone assumes that using keyboard shortcuts is more efficient than using the mouse. But by how much? One 2005 study found that keyboard shortcuts are approximately twice as fast.

So why don't more people use them? It's apparently not an experience issue. A separate study found that keyboard shortcut usage does not tend to increase over time, but rather correlates to social experiences. People learn to use shortcuts from watching others.

With that in mind, here is a cheat sheet of my personal most commonly used keyboard shortcuts. I'm not making any claims to exhaustiveness; in fact I intentionally left out commonly used but also commonly known shortcuts like "Alt + Tab".

I'm also not claiming that these are generally applicable. I'm a developer, so I included Eclipse, etc. Others are Linux specific, which automatically rules out 99% of users. Others require customisation to enable.

Download the PDF or the original OpenOffice document.

Java hack to deal with Windows file paths in Linux

2010-03-24T14:56:00.004-04:00

Say you have a large legacy ColdFusion app on top of Java that runs on Windows. File access is done both via java.io.File and by CFFILE (which in turn also uses java.io.File), but not centralised in any way into a single file access library. Further, say you have file paths both hard-coded in the code, and also in a database.

In other words, the file paths themselves cannot change. They could be either local or remote Windows file paths, such as "c:\temp\file.txt" or "\\server\share\file.txt ".

How would you run this application on Linux without changing the code?

First, I tried running the application in WINE. This worked surprisingly well. WINE actually has awesome path conversion, including UNC support.

Don't want to run WINE? I have another solution. It's actually possible to implement your own version of java.io.File, and have the classloader pick up your version when the JVM starts. The tricky part is that since your class needs to actually be named java.io.File, you can't just extend Sun's java.io.File.

However, you can download the code for java.io.File from the JDK, and make modifications as needed.


 // added to each constructor
 pathname = translate(pathname);

 ...

 private static Pattern regexDriveLetter = Pattern.compile("^[a-zA-Z]:");
 private static Pattern regexSlashes = Pattern.compile("\\\\");
 private static Pattern regexUnc = Pattern.compile("^//");
 
 public static String translate(String pathname) {
  
  if (pathname == null) {
      return null;
  }
  
  pathname = regexDriveLetter.matcher(pathname).replaceAll("");
  pathname = regexSlashes.matcher(pathname).replaceAll("/");
  pathname = regexUnc.matcher(pathname).replaceAll("/mnt/");
  return pathname;
 }

The regular expression rules convert "c:\temp\text.txt" to "/temp/text.txt", and "\\fileserver\userfiles\test.txt" to "/mnt/fileserver/userfiles/test.txt", but of course you can customize them to do anything you like.

To load your java.io.File when the JVM starts, just compile it into a JAR, and add "-Xbootclasspath/p:javafilehack.jar" to your java command-line. Xbootclasspath/p simply places your JAR at the head of the line when the classpath is processed.

I'm not saying I recommend this approach. If you can modify the legacy code to support Linux file paths, that would be far better. If that's not an option, I would say that the WINE approach is more maintainable. With the Java hack, you would need to merge your changes separately into each version of the JVM you want to run, due to the serialVersionUID if nothing else.

I'm working on a compromise solution using AoP. I'll keep you posted.

How to not get caught in spam filters

2010-03-19T11:42:00.003-04:00

Reliably sending email without getting caught in spam filters is a full-time job, for someone. Surely not for an end-user, but for every end-user email, there is an administrator somewhere who has to deal with daily occurrences of some user message not getting through because it got stuck in a spam filter on the other end.

At the enterprise level, this could easily be several people's full-time jobs. Spam filtering is constantly evolving. This is partly due to new spam filtering initiatives that require administrators to configure something new, such as SPF or DKIM. A few years ago, SPF didn't exist. Now, anyone who sends lots of email virtually has to implement it. It's also partly due to other administrators; sometimes you just have to get on the phone with the recipient's admin to figure out what's going wrong.

This guide is not for those enterprise admins. It's for the hapless developers pressed into Postfix config duty for a small start-up, or for the first time admin just getting into outbound mail. What follows is a quick and dirty guide to making sure 99% of your email is delivered.

Make sure you're not on a DNS blacklist (aka RBL: Reverse Blacklist)

By far the most frequently used type of spam filter is the DNS blacklist. There are hundreds of free services out there that keep records of IP addresses they think send a lot of spam. Virtually every spam filtering product on the market comes pre-configured to look at a few of these every time they get a new connection. It's fast due to extremely low over-head (DNS scales, baby), and relatively accurate.

You will need to know what IP you're sending from. You can check many blacklists at once via various different sites.

If you are on a blacklist, you might be wondering how to get off it, and how you got on in the first place. Unfortunately, there is no single answer. Each blacklist has its own criteria for who it lists, and has its own process for removal. Indeed, many lists don't allow removal at all. It's the wild-west out there. If you find yourself unable to be removed from a popular blacklist, you may have no choice but to buy another IP address. Just make sure it's clean first!

Some people think blacklists are the devil. If you have ever found yourself at the mercy of a popular, but totally non-responsive blacklist, you might agree. But in general, the problem is that some administrators outright block email that matches a single blacklist. If you're an inbound admin, don't do that! You want to weigh many factors, and multiple blacklists, before you decide to reject a message. Regardless, they are a reality of the modern Internet you need to just deal with.

Make sure you're not an open relay

If you want to STAY off blacklists, you at the very least need to make sure you're not an open relay. Basically, you should not accept and definitely not send out any mail that's not destined for a domain you actually own. Testing can be done via telnet, or via a web-based tool.

Reverse DNS (aka PTR records)

Another very common check is whether your IP address is named, or unnamed. The idea here is that dynamic IPs, such as those given to home users by their ISP, generally don't need to have names associated with them. A lot of spam these days comes from zombied home machines.

This is a simple DNS fix. You just need to create an PTR record for that IP address. You can check if your PTR is setup correctly with the following command.

dig -x MY_IP_ADDRESS

MX Records, postmaster, root & abuse

While the standards RFCs don't require you to receive mail just because you're sending mail, in reality many anti-spam systems are biased against message from a domain that does not also accept mail. You don't have to send and receive from the same server(s), but if you're sending mail from @example.com, it's a good idea to make sure some real human somewhere is getting any messages sent to postmaster@example.com, root@example.com and abuse@example.com.

Postmaster IS strictly required by the RFC. Root is a legacy version of postmaster. Abuse is a relatively new "standard" that many administrators would try first to resolve a spam issue.

Inbound email is a whole other subject. But the basic gist is that you need an MX record for example.com, and it needs to point to a server that can accept mail for example.com. If you don't have an existing inbound server, or don't want to run your own, many hosted alternatives exists.

You should explicitly test postmaster, root & abuse manually via your regular email client to make sure they actually work.

HELO, I'm your mail server

Mail servers communicate via a protocol called SMTP. It's actually a plain-text protocol, which you can easily emulate via telnet. The very first line of a SMTP handshake is the "HELO" command, where the sending server identifies itself. A typical example would be "HELO example.com", meaning, "Hi, I'm the mail server for example.com".

Many spammers set this to a bogus value, or try to use the recipient's host name or IP address, which is nonsensical. In any case, the correct thing to do is for you to set it to your domain.

How you set this will vary by mailserver. In Postfix, it's the myhostname parameter in /etc/postfix/main.cf. Checking it is easy; just send a message through the server, and look at the headers on the remote end. Your hostname will show up on the first "Received" header line:

Received: by example.com (Postfix, from userid 0)
 id A72979E4144; Thu, 18 Mar 2010 23:00:01 -0400 (EDT)

SPF/DKIM

SPF and DKIM are newer standards that are slowly gaining popularity. The basic idea is that your DNS records can encode a list of rules about what IP addresses are allowed to send mail for your domain. It's a whitelist, versus a blacklist. Typically, you can ignore these unless you're sending a large volume of mail.

Monitoring

That just about covers anti-anti-spam 101. As mentioned, this will likely be an ongoing effort, and you need to keep on top of how it's going. Ideally, there would be an administrator who would be alerted if emails are bouncing due to spam filters. For postfix, I would recommend pflogsumm.

apt-get install pflogsumm

sudo crontab -e

...
# every work-day at 11pm
00 23 * * mon-fri cat /var/log/mail.log |/usr/sbin/pflogsumm -d today |mail -s "daily mail log" postmaster@example.com

Apache fix for IE "webpage has expired" on back button

2010-03-04T11:50:00.005-05:00

When you refresh a webpage that was the result of a HTTP POST, your browser asks you if you want to reload.

The reason for this dialog is to stop the user from inadvertently submitting the same blog post/email message/payment details multiple times. That behaviour is standardised across all browsers. What's not standardised is what happens when you use the "Back" button to navigate to a previous POST page. In most browsers, you get the page from your cache. In IE, you get the dreaded "webpage has expired":

The intention here is good. This is a fool proof mechanism for keeping the user from submitting that page again. But what if your site can actually handle that? Why break the back button this way? A good counter example is a search engine. The search string may be sent via HTTP POST, but you would still like the user to be able to hit "back", change the search string, and hit submit again. In IE, you need a work-around for this.

The most common work-around is to redirect the POST to a GET on the server side, aka the PRG Pattern. This adds some back-end code, but generally solves the problem. However, it can get complicated if you are doing server-side validation, as you generally want to show the form again with validation exceptions in that case. So then you have to persist the temporary state of the FORM in either the session state or the database... messy.

Another solution is to use GETs in the first place. This is ok for some things, like Google searches, but does not work if you need to pass a lot of form data between pages. Also, it ignores the principle of idempotence, which for HTTP means that you should not use GETs for state-changes.

The work-around I have been using recently is to not send a Vary header for IE only.

The Vary field value indicates the set of request-header fields that fully determines, while the response is fresh, whether a cache is permitted to use the response to reply to a subsequent request without revalidation.

In Apache, this is a simple configuration change in httpd.conf:

BrowserMatch MSIE force-no-vary

The practical effect is that when you go "back" to a POST, IE simply gets the page from the history cache. No request at all goes to the server side. I would be interested in hearing about potential down-sides of this solution, other than the fact that the user can submit the form twice. In my case, this is handled already by the CSRF token.

useful decorators for Django views

2010-02-26T10:59:00.005-05:00

Python implements a light version of aspect oriented programming via decorators. I find myself using them often, specifically for Django views. Here are a few of my favourites.

This first one enforces that requests coming into the view are coming from localhost. This can be useful if you are creating REST APIs that are for internal use only. Of course, you should also hide these views via Apache for added security.

"""When added to a view, requires that the request came from localhost.
Used as an added security measure to make sure that remote users cannot
call REST APIs designed to be called from the Java back-end. """

def localhost(f):
    """Requires that a view be invoked from localhost only."""
    
    def new_f(*args, **kwargs):
        request = args[0]
        if request.META["REMOTE_ADDR"] != "127.0.0.1":
            raise Exception("This URL is only invokable by localhost.")  
        return f(*args, **kwargs)                
    return new_f

This next one sends the view back to the referrer. This can be useful to prevent the using from refreshing a page with a FORM POST.

"""
A view decorator that sends the user back to the last page after the view
logic executes. Useful for actions that don't change the page the user is 
looking at. One example is clicking on a link to mark a message as read.
The messages view is refreshed, with that item marked as read.
"""

from django.http import HttpResponseRedirect

def back_to_referrer(f):
    
    def new_f(*args, **kwargs):        
        f(*args, **kwargs)
        request = args[0]
        return HttpResponseRedirect(request.META["HTTP_REFERER"])
            
    return new_f

The last one wraps a view in a try/catch that will redirect the user to a custom error page if the exception is a sub-class of a "UserException". I use this custom exception type to denote any error that the user themselves can resolve.

def usererror(f):
    
    def new_f(*args, **kwargs):
        
        try:
            return f(*args, **kwargs)
        except (
            UserException
            ), (error):
    
            return render_to_response("usererror.html", {
                "message": str(error)
                })  
            
    return new_f

class UserException(Exception):
    """Python does not support interfaces, but that's what this is."""     
    state = None
    
class NoEmailAddressError(UserException):        
    def __str__(self):
        return "You don't have an email address filled out on your profile."

duplicate values with jQuery.cookies (set the path!)

2010-02-10T14:10:00.005-05:00

A program should follow the "Law of Least Astonishment". What is this law? It is simply that the program should always respond to the user in the way that astonishes him least. - Geoffrey James, The Tao of Programming

I recently ran into an astonishing behaviour in an API I was using, namely the official jQuery cookie plug-in. But first, a little background on cookies.

Now, I've been writing web-apps that use cookies for about 15 years. But I had no idea that cookies could be associated with specific paths. Quirksmode (as usual) gives the low-down:

The path gives you the chance to specify a directory where the cookie is active. So if you want the cookie to be only sent to pages in the directory cgi-bin, set the path to /cgi-bin. Usually the path is set to /, which means the cookie is valid throughout the entire domain.

The fact that it defaults to "/" is why I would venture to guess that relatively few developers are aware of this option. After all, in 99% of the cases, cookies are used to persist a session, which you want to be valid at the domain or sub-domain level, but invariant bellow that.

The jQuery cookie plug-in, however, inexplicably defaults the path to the actual path of the URL you're at. This forces you to set default the path to "/" manually, a fact that the top-level "documentation", which only exists in the form of comments in the code, exacerbates by showing the initial example without the path:

 * @example $.cookie('the_cookie', 'the_value');
 * @desc Set the value of a cookie

Of course, you would probably spend 20 minutes or so tearing you're hair out about why cookies were not persisting properly. Finally, you would use Firebug to inspect the actual request/response pairs, and realise that there are multiple values of "the_cookie" being passed for each request. In fact, what virtually everyone using this API will want to do is the following:

 $.cookie('the_cookie', 'the_value', { path: "/" });

Too cute: Django schema migration tools

2010-02-05T13:48:00.000-05:00

Like Ruby on Rails, Django uses a model API to abstract the database layer. A command-line script creates schema for new object definitions. Unlike Ruby on Rails, Django did not ship with a way to modify existing schema as your models change.

Many projects have sprung up trying to fill this void. After watching a video on a presentation by some of the early contenders, I choose django-evolution for my current project, mainly because the presenter was also a Django core developer, and came off as more of a beard developer than the others.

Getting started was very easy. For the most part, migrations were smooth. But occasionally, I would hit a bug in the evolution framework. Sometimes it would be an unsupported operation, like changing a column from and int to a float. Sometimes it would be a bug related to my particular database (MySQL). Sometimes I had no idea what was wrong. You would just get a stack-trace, and have to track it down from there.

var/lib/python-support/python2.6/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
  from sets import ImmutableSet
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/var/lib/python-support/python2.6/django/core/management/__init__.py", line 340, in execute_manager
    utility.execute()
  File "/var/lib/python-support/python2.6/django/core/management/__init__.py", line 295, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/python-support/python2.6/django/core/management/base.py", line 192, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/var/lib/python-support/python2.6/django/core/management/base.py", line 219, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.6/dist-packages/django_evolution/management/commands/evolve.py", line 87, in handle
    hinted_evolution = diff.evolution()
  File "/usr/local/lib/python2.6/dist-packages/django_evolution/diff.py", line 197, in evolution
    changed_attrs[prop] = current_field_sig.get(prop, ATTRIBUTE_DEFAULTS[prop])
KeyError: 'field_type'

After spending a couple hours tracking down an similar issue that was actually blocking a QA deploy, I noticed that the django-evolution mainline had only had two check-ins in the entire year of 2009. That was when I realised that I picked a winner without looking at which project was actually being actively supported.

I briefly looked at both South and dmigrations. But then I thought about the solution we use on non-Django code-bases; manual scripts. Sure, they are a little more work and they were somewhat inelegant. But they can do absolutely any data migration, and I don't remember one ever blocking a QA deploy for more than a few minutes.

That's the core of my issue with these solutions. They are trying to be "cute", and provide a gee-wiz solution to a problem that is more of an annoyance than anything. But the most critical part of a database migration scheme is reliability. If there is even a small chance that it screws up a deploy or (god forbid) actually messes up data, then it's just not worth the time savings.

Since moving back to sequential, numbered SQL files, we had no problems deploying schema changes. Of course, we are missing out on true database independence, but I figure we can always use syncdb to generate the entire schema from scratch if we ever do change databases.

scripting multiple gnome terminal tabs

2010-01-26T14:04:00.003-05:00

Sometimes it feels like I'm looking at logs all day. Certainly there are times when this is more true than others. If you find yourself in one of those times, you might benefit from being able to call all your typical log files up at once.

On my system, I do this by opening a gnome terminal, and having one tab for each file doing a tail command. Here is a BASH script that launches such a terminal window, opens and even names the tabs for you with convenient titles.

#!/bin/bash

# space-delimited list of colon-delimited tab name, log file pairs
LOGS="jboss:~/jboss/server/bullhorn/log/server.log solr:/var/log/solr.log"

args=""
for log in $LOGS
do
 tab_name=${log%:*}
 log_file=${log#*:}
 args=" $args --tab -e \"bash -c \\\"printf '\033]2;%s\007' '$tab_name'; tail -f $log_file\\\"\""
done

eval "gnome-terminal --hide-menubar --maximize $args"

MySQL drop column if exists

2010-01-15T13:52:00.011-05:00

Idempotence describes the property of operations in mathematics and computer science that means that multiple applications of the operation do not change the result. - Wikipedia

When updating a database schema, it's very useful to make your SQL scripts idempotent. I.e. you should be able to run the script more than once, no run should error out, and the end result should be the same as when you ran it the first time.

On the face of it, this seems absurd. Why might you want to code idempotent schema updates? Say your initial version of the schema update got it slightly wrong. You actually want a column to be varchar(255), not a varchar(50). You could add yet another schema update to fix this, or if your script was idempotent, you could simply modify the original script and run it again.

The most common operation you will want to do is to drop a table or column, but only if it exists. MySQL has a built-in modifier for this.

DROP TABLE IF EXISTS candidate;
CREATE TABLE candidate...

For some reason, the same facility does not exist in MySQL for dropping a column if it exists. But you can fake it, at least in MySQL 5 or later, by querying the database meta-data do see if the column exists, and drop it if it does. However, because you need an IF statement, it will need to be a stored procedure.

drop procedure if exists schema_change;

delimiter ';;'
create procedure schema_change() begin

 /* delete columns if they exist */
 if exists (select * from information_schema.columns where table_name = 'table1' and column_name = 'column1') then
  alter table table1 drop column `column1`;
 end if;
 if exists (select * from information_schema.columns where table_name = 'table1' and column_name = 'column2') then
  alter table table1 drop column `column2`;
 end if;
 
 /* add columns */
 alter table table1 add column `column1` varchar(255) NULL;
 alter table table1 add column `column2` varchar(255) NULL;
  
end;;

delimiter ';'
call schema_change();

drop procedure if exists schema_change;

Django forms date field format

2010-01-12T13:33:00.004-05:00

Django defaults dates in forms to the format "2010-01-15". But changing the default format is fairly straight-forward, though there appears to be some confusion about how.

class MyForm(Form):
    
    # the default format is %Y-%m-%d
    date_available = forms.DateField(
        widget=forms.widgets.DateInput(format="%m/%d/%Y"))

Finding open VNC hosts with nmap

2009-12-29T09:37:00.003-05:00

Say you have a machine using DHCP that you only boot up once in a while. It's headless, so you need to remote into it. How can you find out what IP it has? Here is quick command to scan for open VNC ports on the 172.27.1.* sub-net.

nmap -p 5900 172.27.1.0/24 --open |grep Interesting

Python smtplib long header folding using tabs instead of spaces

2009-12-15T14:08:00.005-05:00

Unless you have done a lot of work with generating and parsing emails, you are probably not intimately familiar with the details of the MIME format. Here is an example of a MIME encoded email:

From: user1@example.com
To: user2@example.com
Subject: Christmas party!
Mime-Version: 1.0
Content-Type: multipart/mixed; 
 boundary="----=_Part_20502_1870283373.1261059406023"
Date: 17 Dec 2009 09:16:46 -0500

------=_Part_20502_1870283373.1261059406023
Content-Type: multipart/alternative; 
 boundary="----=_Part_20503_1134508872.1261059406023"
...

Typically, all lines in MIME wrap at 80 characters. For headers, the standard is to break before the last whitespace under the character limit. This is defined in RFC 2822, section 2.2.3. So, if you have a long subject, it might look like:

...
Subject: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec quis 
 est dolor, a faucibus nisi. Curabitur et ipsum ut arcu commodo feugiat. 
...

I ran into bug 1974 in Python where the standard smptlib example was throwing an error on long subject lines.

    from email.MIMEText import MIMEText
    
    msg = MIMEText(body, "html")
    msg["From"] = "user1@example.com"
    msg["To"] = "user2@example.com"             
    msg["subject"] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec quis est dolor, a faucibus nisi. Curabitur et ipsum ut arcu commodo feugiat."

    ...

When looking at the resulting email is some clients (Outlook, Thunderbird), the subject like was concatenating the words "quis" and "est" into "quisest", instead of "quis est". It turns out that smtplib is inserting a tab character in between them, which some clients ignore.

The solution was simple enough. Just set header fields as Header objects, instead of strings:

    from email.MIMEText import MIMEText
    from email.Header import Header
    import smtplib

    msg = MIMEText(body, "html")
    msg["From"] = "powerfill@powerfill.com"
    msg["To"] = to_addr
    msg["subject"] = Header(subject)    

    smtp = smtplib.SMTP(settings.EMAIL_HOST, settings.EMAIL_PORT)        
    smtp.sendmail(msg["From"], [msg["To"]], msg.as_string())
    smtp.quit()

Edit: Don't wrap the from/to addresses in Header object. I ran into a "TypeError: unhashable instance" deploying this code to Python 2.6.

Protocol Relative URLs

2009-12-08T20:26:00.005-05:00

If you follow YSlow rules, then you are likely separating HTTP requests for static resources (images, CSS, javascript) onto a second domain. Maybe you even have a nice script to translate your includes when you deploy to production. Then one day, you enable your site for SSL.

Most likely, your static domain resources are being referenced as absolute, global, non-SSL URLs. Ie, http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js. In most browsers, there is no problem with mixing an HTTPS page with HTTP includes. But with IE, Microsoft has made the dubious decision to warn users that "this page contains both secure and nonsecure items".

Personally, I don't think this makes much sense. Whether my static resources are being served by SSL or not does not effect the security of the page, and it definitely slows it down.

The most obvious solution to include these resources with HTTPS when appropriate is to change the includes to relative (or domain relative) URLs. Ie, /ajax/libs/jquery/1/jquery.min.js. But then YSlow rightly complains that this is not optimized; you're sending cookies for each static include, and the browser can't load the includes in parallel with your main page.

Another solution is to change the includes dynamically based on whether the main page is HTTPS or not. Fine for HTML, but what about the CSS file? That's usually static. You could dynamically generate it on the fly (performance problem), or generate two versions on deploy and include the HTTP or HTTPS one as appropriate. Neither option is appealing.

Now, I have been writing HTML for about 15 years. But until I did a little research for this issue, I had never heard of protocol relative URLs. It turns out that the browser can load a resource from a URL pointing to another domain, and decide to use either HTTP or HTTPS on the fly depending on the current page. Fantastic!

.body {
    background-image: url("//static.powerfill.com/images/woot.gif");
}

Allowing periods (and other characters) in Django's user model

2009-12-04T15:56:00.005-05:00

By default, Django's authentication middleware only allows user names with 30 characters or fewer, containing only letters, digits and underscores. What if you want to allow other user names? It's actually a very small change to admin.py:

username_regex_field = forms.RegexField (
        label = "Username", 
        max_length = 30,
        regex = r"^[\w'\.\-@]+$",
        help_text = "Required. 30 characters or fewer. Letters, apostrophes, periods, hyphens and at signs.",
        error_message = "This value must contain only letters, apostrophes, periods, hyphens and at signs."
        )

class UserCreationForm(UserCreationForm):
    username = username_regex_field
    
class UserChangeForm(UserChangeForm):
    username = username_regex_field
    
class UserProfileAdmin(UserAdmin):
    form = UserChangeForm
    add_form = UserCreationForm

All I'm doing here is defining a new RegexField, and then over-writing the default create and change user forms for Django auth.

Letting ssh remember your passwords

2009-11-25T12:35:00.008-05:00

If you're like me, you ssh into the same small set of machines over and over. Maybe sometimes you run commands that sit on top of ssh, like scp for file copies. Wouldn't it be nice if you could have ssh remember your passwords? This is especially useful for automation.

It seems like this has been built into ssh forever, but I just discovered it. So, excuse me if this is child's play for the grey beards out there.

Create your public/private key pair. Hit enter at any prompts; you don't need a password.

          ssh-keygen -t rsa

You now have a local key in /home/$user/.ssh/id_rsa.pub

Copy your key to the remote machine (server1), and append it to /home/$user/.ssh/authorized_keys

          ssh $user@server1 mkdir -p ~/.ssh
          cat ~/.ssh/id_rsa.pub | ssh $user@server1 'cat >> ~/.ssh/authorized_keys'

That's it! You should now be able to ssh and scp to the remote machine without supplying a password.

Security note: It's important to note that ssh is not remembering a password, it's using a pre-exchanged public/private key. Regardless, you want to think twice about doing this in a production environment, where access to one terminal would result in accesses to many other terminals.

Edit: Most distributions will have a ssh-copy-id command, which makes the "cat" above obsolete.

Changing image links in CSS to use a static domain with Django compress

2009-11-20T11:05:00.006-05:00

Django compress is a great little utility for hitting some of the YSlow website performance high notes. It can combine and minify css/js files on the fly, when settings.DEBUG is set to False. This allows you to keep the files separate in development for debugging.

I also wanted to use django compress to serve js, css and image resources from a static domain as per YSlow's Cookie-Free and Domain Split rules.

When the browser makes a request for a static image and sends cookies together with the request, the server doesn't have any use for those cookies. So they only create network traffic for no good reason. You should make sure static components are requested with cookie-free requests. Create a subdomain and host all your static components there.

If your domain is www.example.org, you can host your static components on static.example.org. However, if you've already set cookies on the top-level domain example.org as opposed to www.example.org, then all the requests to static.example.org will include those cookies. In this case, you can buy a whole new domain, host your static components there, and keep this domain cookie-free. Yahoo! uses yimg.com, YouTube uses ytimg.com, Amazon uses images-amazon.com and so on.

Another benefit of hosting static components on a cookie-free domain is that some proxies might refuse to cache the components that are requested with cookies. On a related note, if you wonder if you should use example.org or www.example.org for your home page, consider the cookie impact. Omitting www leaves you no choice but to write cookies to *.example.org, so for performance reasons it's best to use the www subdomain and write the cookies to that subdomain.

Splitting components allows you to maximize parallel downloads. Make sure you're using not more than 2-4 domains because of the DNS lookup penalty. For example, you can host your HTML and dynamic content on www.example.org and split static components between static1.example.org and static2.example.org

Django compress can do this for css and js resources using MEDIA_URL. But out of the box, it doesn't help with image links inside a css file. But it's easily extensible. Here is a quick filter to replace image links (or anything, really), in your css. Again, this would only fire in production.

"""Django compress filter to replace image src URLs in a CSS file with a static 
domain. See: http://developer.yahoo.com/performance/rules.html#cookie_free 

Example configuration in settings:

COMPRESS_CSS_FILTERS = ('compress.filters.csstidy.CSSTidyFilter', 
    'search.helpers.static_domain_css.StaticDomainCSSFilter')
    
COMPRESS_STATICDOMAIN_SETTINGS = [
    ("/static/images/", "http://static.powerfill.com/static/images/"),
    ("../images/", "http://static.powerfill.com/static/images/"),
    ] 

"""
from django.conf import settings
from compress.filter_base import FilterBase

COMPRESS_STATICDOMAIN_SETTINGS = getattr(
    settings, 'COMPRESS_STATICDOMAIN_SETTINGS', {})

class StaticDomainCSSFilter(FilterBase):
    def filter_css(self, css):
        for old, new in COMPRESS_STATICDOMAIN_SETTINGS:      
            css = css.replace(old, new)
        return css

YSlow 80/20

2009-11-06T15:50:00.005-05:00

I spent some time this week optimising a new site, specifically looking at some of the YSlow recommendations.

For a smaller, low traffic site, some of the recommendations are over-kill. I'm not paying for a CDN. But I did find some of the tidbits valuable, specifically around what Apache does not do by default.

First of all, Apache2 out of the box has gzip enabled, but CSS and Javascript files are not in the list the zippable MIME types. Adding them is easy. Just edit /etc/apache2/mods-available/deflate.conf:

<IfModule mod_deflate.c>
      AddOutputFilterByType DEFLATE text/html text/plain text/xml application/x-javascript text/javascript text/css application/javascript
</IfModule>

The larger win was the Expires HTTP header, which lets the browser know that it shouldn't even attempt to see if a CSS, Javascript or image has been updated until such and such a date. By default, Apache2 deploys with Expires disabled.

Yahoo! recommends setting this to a "far future" value, like 10 years from now, and then managing this by changing the URLs for CSS, Javascript or images every time they actually change on the server. Implementing that is a PITA because you need a tool that can auto-rewrite the image URLs in the CSS, and the script URLs in your HTML.

This may make sense for a large site, where you are concerned with minimising bandwidth. But on a smaller site like mine, I'm more concerned with minimising the number of HTTP connections when the user is rapidlly clicking between pages. I don't care if they have to hit the server again per resource every once in a while, just as long as in the average case they are only loading one HTML page per request.

I picked an Expire window of just 12 hours. This is long enough so that during a single session, the user will only request each resource once. But it's short enough that if I push an actual change at night, everyone will get it in the morning.

To make the actual change, you need to enable the expire_mod, and then set the actual Expires value per MIME type.

>cd /etc/apache2
>sudo ln -s ../mods-available/expires.load mods-enabled/expires.load

# edit /etc/apache2/sites-available/default
ExpiresActive On
ExpiresByType text/css "access plus 12 hours"
ExpiresByType application/javascript "access plus 12 hours"
ExpiresByType image/png "access plus 12 hours"
ExpiresByType image/gif "access plus 12 hours"

Easier Development with Unobtrusive JavaScript, Part III

2009-10-29T12:07:00.009-04:00

Part I defined what I mean by Unobtrusive JavaScript. Part II showed some specific examples. Now I want to talk a little more about why you might want to do all this.

We have already discussed how a small subset of users don't have JavaScript, or don't have it enabled. There are also users with disabilities, and SEO issues. However, none of these are likely to sell your development manager. Time for some new arguments.

Unobtrusive JavaScript means writing as little JavaScript as possible.

If you've implemented the behaviour on the back-end, all you need is a little sprinkling of JavaScript to make it dynamic. Now, why is that net benefit?

JavaScript is harder to write, debug and refactor than back-end code.

JavaScript tooling is fairly weak. Debugging tools are browser specific. Going on the theory that less code is more maintainable than more code, and that JavaScript is in general harder to maintain than say Java, Ruby or Python, it stands to reason that less JavaScript means more maintainable code.

JavaScript is harder to test than back-end code.

Likewise, unit and functional testing tools for JavaScript are weaker than their back-end counterparts, and browser specific to boot. The vast majority of the time, this results in zero automated testing coverage for JavaScript code.

JavaScript is harder to make performant than back-end code.

This is the kicker. Typically, JavaScript is thought of as a way to improve the perceived performance of a site, by reducing page refreshes. However, it is very possible to find yourself in a position where the Ajax update is actually slower than a page refresh, especially in Internet Explorer.

Often times, you may not realise this is the case until late in the development cycle. Perhaps you have not been testing on other browsers, and you find out during acceptance testing. Perhaps you waiting until the last minute to roll in new design html/css from the UI folks, and it makes the DOM insertions a lot slower. Perhaps it's the result of new content from a new feature request.

In any case, you realise that Ajax is slower. What do you do? If you used Unobtrusive JavaScript, you can selectively disable JavaScript for certain pages or components. You can even decide whether to use the JavaScript or refresh version per-browser. You can do this will almost no code change.

Without Unobtrusive JavaScript, you're looking at either living with the problem, or re-writing some code. Maybe a lot of code, if it's really JavaScript heavy.

Quick and dirty Ajax on any FORM

2009-10-23T15:49:00.010-04:00

I have been trying to write as much unobtrusive javascript as possibly lately. Mainly this means getting everything working without any javascript in the picture to start, and then layering in dynamic elements without breaking anything for javascript-disabled users.

This week, I had a pretty simply page with a single FORM element and a bunch of INPUT elements, many of which were submit buttons with various name/value pairs. Why so many buttons? Well, it's a semantically valid way to capture a single-click user action.

The user wants to go to the next page? That's a button. Want to remove a keyword from this list? That's a button. Want to expand a section? You guessing it, that's a button.

Why not use a link for this? In a simple case that's fine. If you wanted to go a whole other page, that's a link. But usually when I'm writing a web-application (as opposed to a web-site) there is a whole lot of state to pass around. Sure, the "next page" operation is the new piece of state, but what about the current page/last page state? I could wrap those into each link on the server-side, but that gets messy quickly. You're also limiting yourself to the size of the state by using a GET for that request.

Now, buttons are kind ugly. Luckily, there is a cross-browser way to display them as images using just CSS:

INPUT.ImageButton {
 border: 0px;
 padding: 0 0 0 16px !important; /* Hide text - IE */
 text-indent: -999em; /* Hide text - FF */
 height: 10px;
 width: 10px;
 margin-bottom: 3px;
 cursor: pointer;
 display: inline;
 font-size: 8px;
}

INPUT.ImageButton.Add {
 background: transparent url(/media/img/admin/icon_addlink.gif) no-repeat center;
}

...

<input type="submit" class="ImageButton Add" name="next_page" value="true"/>

What does this have to do with Ajax? Well, it makes Ajax a breeze. Here is a generic jQuery snippet for turning any page where the state is entirely FORM-based into an Ajax request.

$(document).ready(function() {
 bind(); 
})

function bind() {
 var form = $("#results-form");
 if (form) {
  
  $("input[type=submit]").click(function(e) {

   // disable the form buttons so the user can't submit twice   
   $("input[type=submit]").click(function(){ return false; });

   var form = $("#results-form");
   var button = this;
   var url = "ajax=true&" + button.name + "=" + button.value + "&";
   
   document.body.style.cursor = 'wait';
   
   $.post(form[0].action, url + form.serialize(), function (data) {    

    document.body.style.cursor = 'auto';
    
    $("#contentDiv").html(data);
        
    // re-bind new html for next Ajax call
    bind();    
   });
   
   return false;
  });
 }
}

All you need to do is change your back-end to look for the ajax parameter, and only render the contents of contentDiv in that case.

Unit testing Django with doctest

2009-10-14T12:29:00.007-04:00

There are two main ways to write tests in Django; doctests and unit tests. Units tests will be familiar to you if you're coming from Java. You basically write new Python code to setup and execute your tests. Doctests are a combination of documentation and unit testing. You actually write executable tests in your comments.

Why is this is a good idea? Well, for one thing, they make sure your comments are accurate. If you change how a function behaves, your doctests will let you know if you forget to update the comments. But I mainly appreciate them because of their brevity.

def _lucene_location(job):
    """Formats a Lucene query for searching by city/state
    
    Values must be double-quoted in case there are spaces
    >>> _lucene_location(Job(city="Fort Worth", state="TX"))
    ' +JOBCITY:"Fort Worth" +JOBSTATE:"TX"'
    
    Need to escape quotes in the city/state values
    >>> _lucene_location(Job(city='"Bad" City', state="MA"))
    ' +JOBCITY:"%22Bad%22 City" +JOBSTATE:"MA"'        
    
    """
    return "".join([
        ' +%s:"%s"' % (field_solr[attr], urllib.quote(getattr(job, attr), " "))
        for attr in  ["city", "state"]
        ])

Here we have a function that takes a Job model object and formats a Lucene search string for city and state. The primary edge case, which we are testing for, is that the city name can contain a space, so it needs to be in double-quotes. But what if the name itself has a double-quote character? Then it needs to be encoded.

These conditions are covered by doctests, where ">>>" denotes a statement to be run in a Python interactive session, and where the following line is the expected result. I can't think of a way to express these tests with less code.

Doctests have some downsides. They are not as flexible as unit tests. If your tests require a complicated setup procedure, that's probably better done as a unit test. The other downside I encounted was documentation. The Django website simply tells you to place your doctests in a file called tests.py. But of course, most developers will want to execute doctests that are in-line with functions in other .py files.

The solution is easy, but was not readily apparent to me from the documentation. Simply put this in your tests.py file:

from search.models import Job
from search.placement import _lucene_location, _lucene_query, _fuzzy_titles

__test__ = {
    "_lucene_location": _lucene_location,
    "_lucene_query": _lucene_query,
    "_fuzzy_titles": _fuzzy_titles
}

Here, I am importing anything my doctests might need, such as the Job model. Otherwise, I would have to clutter my doctests themselves with those imports. This would be a good place to do some setup common to the doctests, as well. __test__ is an undocumented dictionary that maps test name to callable object.

To run the tests, just execute the following command-line:

>python manage.py test
Creating test database...
...
Creating table search_job
...
Installing json fixture 'initial_data' from '../powerfill/search/fixtures'.
Installed 48 object(s) from 1 fixture(s)
...................
----------------------------------------------------------------------
Ran 19 tests in 2.285s

OK

As opposed to vanilla Python doctests, Django is actually spinning up a clean database, and populating your default fixture data. This is a great way to avoid having to mock models. Your continuous integration server could even run these tests without having a real database, by using sql lite.

Django - Externalize settings.py

2009-10-09T14:50:00.004-04:00

In Django, settings.py is where you store configuration such as database connection strings, directory paths for static and media resources, as well as caching settings. Obviously, these settings will be different between various developers, not to mention qa and production environments.

A typical solution is to branch settings.py, and then copy in the appropriate version after a deploy. This adds an extra moving piece that you may forget. Another solution is to put logic into settings.py to branch by host name, but this is a fairly brittle approach.

I have my settings.py file in an external directory. That way, it doesn't get checked into source control, or over-written by a deploy. I put this in the settings.py file under my Django project, along with any default settings:

import sys
import os.path

def _load_settings(path):    
    print "Loading configuration from %s" % (path)
    if os.path.exists(path):
        settings = {}
        # execfile can't modify globals directly, so we will load them manually
        execfile(path, globals(), settings)
        for setting in settings:
            globals()[setting] = settings[setting]
     
 _load_settings("/usr/local/conf/local_settings.py")

Note: This is very dangerous if you can't trust local_settings.py.

Python/Django: First Impressions

2009-10-02T15:29:00.018-04:00

I've started a brand-new codebase at work. With new code comes the possibility of a new language/platorm. What we really needed was speed; we don't know exactly what we want to build; we are going to have to iterate over many ideas quickly.

We had a long list of requirements. We wanted a simple, clean language. We needed to integrate with SOLR, but SOLR integrated with anything easily. We wanted fine-grain control of HTML output, so WSIWYG UI framewords like JSF and GWT were out. We wanted a popular platform, so there would be a good community to field questions.

Most importantly, we didn't want to limit our options down the road. Being potentially able to deploy on J2EE was huge. Ideally we would be able to import Java JARs, as well.

Both Ruby on Rails and Python/Django met our requirements. Django came out slightly ahead by having better documentation. Too much RoR info is still woefully out of date, or trapped in youtube videos.

Here are some quick thoughts on my first week programming Python/Django for an actual work project:

Python itself is as nice a language as advertised.
- Hardly a single character is wasted in the source. You can do a lot in ten lines of Python, even if you're not calling out to libraries.
- lists, dictionaries, tuples, map() really help reduce your code.
```
def id_value_long(objs):
   result = []
   for obj in objs:
       result.append((obj.id, str(obj))
   return result

# this is equivalent to id_value_long()
def id_value(objs):
   return map(lambda obj: (obj.id, str(obj)), objs) 
```
- You can make your code object oriented, or not. Mix and match. Note: private members are only by convention; the door is wide open to get at those.
- You have modern language features like aspect oriented programming (via decorators) and lambdas, implemented with clean syntax. They don't feel like in-elegant tack-ons.
- The standard library includes almost everything you could possibly want: reflection, regular expressions, dom parsing, web-services and unit testing.
Django lets you do more with less code (sense a pattern here?)
- Real templating is worlds better than ColdFusion/JSP/PHP includes.
- You can hand-code your HTML for greater control, or use generic views for common cases.
```
class SearchForm(forms.Form):
   job = forms.ChoiceField(label="Source Job")
   query = forms.ChoiceField(label="Query Options")           
  
   def __init__(self, jobs, queries, *args, **kwargs):
       super(SearchForm, self).__init__(*args, **kwargs)   
       # ChoiceField needs id/value pairs, not full model objects
       self.fields["job"].choices = id_value(jobs)
       self.fields["query"].choices = id_value(queries)

...

<FORM action="{% url results_link %}" method="get">
 {{ form.as_p }}   
 <INPUT type="submit" value="Search"/>
</FORM>
```
- Just like RoR, you can auto-generate views from models. But in Django, these are by default relegated to an "admin" sub-site, which makes sense.
- Unlike RoR, Django doesn't manage scheme updates for you. Unless you want to blow away all your data every time, you need to write schema update scripts by hand. Fixtures can mitigate this in development, but not in production.
- Django doesn't do MVC, it does "MTV" (model, view, template). In my opinion, it's really MuTV, because the url mapper plays a controller-like role.

Ruby on Rails: anyType, soap4r and handsoap to the rescue

2009-09-25T13:30:00.005-04:00

Perl makes the easy things easy and the hard things possible. - Larry Wall

This sentiment has been applied to many different languages, frameworks and systems. It not surprising; it's the underlying goal of most software. Ruby on Rails is a little different. It makes the easy things extremely easy. But god help you if you want to get off the train in between stops.

After playing with rails a few months ago, I wanted to upgrade my toy project by getting some data from a SOAP web-service. Google pointed me to soap4r, which seems to be the common solution. However, I quickly ran into problems.

It seems that the WSDL I was consuming made judicious use of the anyType primitive, which Java uses to expose an Object parameter in a method. However, the remote method end-points assumed you were going to tell it at runtime what the type of the object you're passing is. In practice, it was an integer.

<?xml version="1.0" encoding="utf-8" ?>
<env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
  <env:Body>
    <n1:find xmlns:n1="http://apiservice.bullhorn.com/">
      <session>
        <client>rO0ABXNyACpjb20uYnVsbGhvcm4uZGF0YXNlcnZpY2UuYXBpLkFwaURhdGFDbGllbnQAAAAAAAAA
AQIACEoACmxhc3RBY2Nlc3NJAA5zdXBlckNsdXN0ZXJJZEwADWNvcnBvcmF0aW9uSWR0ABNMamF2
YS9sYW5nL0ludGVnZXI7TAAGZGJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7TAAMbWFzdGVyVXNl
cklkcQB+AAFMAA5wcml2YXRlTGFiZWxJZHEAfgABTAAGdXNlcklkcQB+AAFMAAp1c2VyVHlwZUlk
cQB+AAF4cAAAASPoSwbkAAAAAHNyABFqYXZhLmxhbmcuSW50ZWdlchLioKT3gYc4AgABSQAFdmFs
dWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAARmdAAJQlVMTEhPUk4xc3EAfgAE
AE2pkXNxAH4ABAAABmZzcQB+AAQAAADpc3EAfgAEAAAP3A==</client>
        <corporationId>1126</corporationId>
        <userId>233</userId>
      </session>
      <entityName>JobOrder</entityName>
      <id xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:int">67</id>
    </n1:find>
  </env:Body>
</env:Envelope>

The soap4r generated classes output a packet without the namespaces on the id tag, which resulted in the following exception.

java.lang.IllegalArgumentException: Provided id of the wrong type for class com.bullhorn.entity.job.JobOrder. 
Expected: class java.lang.Integer, got class org.apache.xerces.dom.ElementNSImpl
 at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:196)
 at com.bullhorn.dataservice.jpa.BhEntityManagerImpl.find(BhEntityManagerImpl.java:62)
 at com.bullhorn.dataservice.serviceImpl.BaseService.find(BaseService.java:29)
 ...
Caused by: org.hibernate.TypeMismatchException: Provided id of the wrong type for class com.bullhorn.entity.job.JobOrder. 
Expected: class java.lang.Integer, got class org.apache.xerces.dom.ElementNSImpl
 at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:109)
 at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
 at org.hibernate.impl.SessionImpl.get(SessionImpl.java:842)
 at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
 at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:182)
 ... 48 more

You could rightly say that this was a Java problem, or at least a problem with the API of this web-service. It's a good illustration of why you should try to use primitives only for web-service parameters. However, in this case I could not control the web-service, so I needed to solve this in Ruby.

I briefly tried to download the WSDL and muck with it. SoapUI, by the way, has the ability to download a multi-part (think 50 parts) WSDL and save all the pieces locally. But I soon exhausted my expertise in hand-editing WSDL. Besides, ideally this needed to be done at runtime to support types other than integers.

Having "gone off the rails", I turned to a library called handsoap. Their philosophy is that you often need a higher level of control over the SOAP packets themselves.

...soap4r has problems. It's incomplete and buggy. If you try to use it for any real-world services, you quickly run into compatibility issues... Handsoap tries to do better by taking a minimalistic approach. Instead of a full abstraction layer, it is more like a toolbox with which you can write SOAP bindings. -troelskn

As promised, I did have to do the SOAP binding myself. But I also got the opportunity to do anything I wanted to the XML DOM object, including set these pesky required namespaces.

require 'handsoap'

Handsoap.http_driver = :httpclient
Handsoap::Service.logger = $stdout

class ApiService < Handsoap::Service
  endpoint API_SERVICE_ENDPOINT

  def on_create_document(doc)
    # register namespaces for the request
    doc.alias 'tns', 'http://apiservice.bullhorn.com/'
  end

  def on_response_document(doc)
    # register namespaces for the response
    doc.add_namespace 'ns', 'http://apiservice.bullhorn.com/'
  end

  def start_session!(state)
    soap_action = ''
    response = invoke('tns:startSession', soap_action) do |message|
      message.add "username", state[:username]
      message.add "password", state[:password]
      message.add "apiKey", state[:apiKey]
    end
    node = response/"//return"
    { :client => (node/"//client").to_s, :corporationId => (node/"//corporationId").to_s , :userId => (node/"//userId").to_s}
  end
  
  def find(state)
    soap_action = ''
    response = invoke('tns:find', soap_action) do |message|

      session = state[:session]

      message.add "session" do |i|
        i.add "client", session[:client]
        i.add "corporationId", session[:corporationId]
        i.add "userId", session[:userId]
      end
      
      message.add "entityName", state[:entityName]

      # ids are set to "anyType" in the WSDL, and our end-point enforces that these namespaces be in the post
      message.add "id", state[:id] do |i|
        i.set_attr("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance")
        i.set_attr("xmlns:xs", "http://www.w3.org/2001/XMLSchema")
        i.set_attr("xsi:type", "xs:int")
      end

    end
  end
  
end

PS - Installing handsoap was a pita. I could not get the curb gem, a ruby binding for the Linux curl utility, installed. So this example uses httpclient, instead.

Name resolution failure caching in SMB

2009-09-18T16:42:00.000-04:00

If you have a production file server failing, you have serious problems. Client requests are failing. But they may not be failing quickly. If your product is a website, and the web-servers make a lot of file requests, one of the fallouts is that requests will start to pile up.

What's happening? The Windows web-servers are actually making NetBIOS broadcasts, asking your network "hey, is anyone fileserver1?". If fileserver1 is actually down, then no response is coming. But the web-server waits for a handful of seconds before it's sure. That whole time, the browser request is hanging out waiting for the web-server to respond.

I ran some tests using ColdFusion, and I noticed that if you request a file on a known-down or non-existent server, the initial request takes 15 seconds to timeout. Subsequent requests fail quickly for the next 10 seconds or so. Then there is another 15 second timeout, and the pattern repeats. So there is actually a caching mechanism for unavailable servers.

I was trying to find a way to configure both the maximum amount of time a request to a non-existent server can take (the 15 seconds), as well as how long the fact that the server is down is cached (the 10 seconds). But I couldn't find any knobs to turn in Windows.

Instead, I got a capture from Wireshark showing Netbios naming service packets:

    No.     Time        Source                Destination           Protocol Info
         90 2.184614    172.27.8.7            172.27.8.255          NBNS     Name query NB CHASE-IE<20>
         97 2.920946    172.27.8.7            172.27.8.255          NBNS     Name query NB CHASE-IE<20>
        106 3.671325    172.27.8.7            172.27.8.255          NBNS     Name query NB CHASE-IE<20>
        136 12.936379   172.27.8.7            10.0.2.15             NBNS     Name query NBSTAT *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00>
        140 14.436181   172.27.8.7            10.0.2.15             NBNS     Name query NBSTAT *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00>
        142 15.936134   172.27.8.7            10.0.2.15             NBNS     Name query NBSTAT *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00>

You can see the 15 seconds the initial request is taking. It looks like it does a UDP broadcast to the whole subnet (172.27.8.255). It doesn't get an answer, and then somehow gets the right IP (10.0.2.15), perhaps via DNS. Then it spends a few seconds timing out to that server.

Finally, I was able to reduce the initial waiting period from 15 seconds to 2 seconds by putting the server into lmhosts.

Edit c:\windows\system32\drivers\etc\lmhosts (not .sam, the sample file)
Add the line "10.0.2.15 chase-ie #PRE"
Run "nbtstat -R" to reload the Netbios naming cache
Run "nbtstat -r" to check that the name is cached

Numeric Range Searches in Lucene

2009-09-11T14:00:00.013-04:00

One of the reasons Lucene is awesomely fast is that it treats all data as strings. This is a perfect optimisation for most searching tasks. However, it does mean that other data types need to be converted to strings before they are indexed or searched for. When dealing with range searches on numbers, special care needs to be taken to get it right.

Say you want to index latitude and longitude coordinates, so that you can quickly search for all records in physical proximity to any location. These coordinates are typically expressed as decimal values, such as 53.1345. They can also be negative.

The naive approach would be to convert these to strings such as "53.1345". You would need to come up with a tokenizer that didn't split based on the decimal point (period character). Then your Lucene search string would look like "longitude:53.1345". This works if you are trying to find any records with this exact longitude, but it breaks down when doing a ranged search.

Say you want to search for all longitudes between 4 and 6. You Lucene search string would look like "longitude:[4 to 6]". Critically, this would actually match a record with value "53.1345", because Lucene can only compare lexicographically. In short, the first character in "53.1345" is '5', and because that's between the '4' and '6', the whole string "53.1345" is between "4" and "6".

The suggested solution is to pad all numeric values to a fixed width. So, 53.1345 becomes "000000000000053.13450000". Notice that we can't encode ALL possible decimal values in a fixed width; you have to decide how many digits of precision you will need, and what the max value you will need to support is.

So that you don't have to write a decimal-ignoring tokenizer, it's also easier to replace the '.' character with a alphanumeric, such as 'd'. So your serialized double now looks like "000000000000053d13450000".

What about negative numbers? If you just add a negative sign to the beginning, you are no longer fixed width. There are many solutions, but I would advocate using "p" to denote positive numbers, and "n" to denote negatives. So 53.1345 becomes "p000000000000053d13450000", and -1.32 becomes "n000000000000001d31000000". These are comparable because "p" happens to be greater than "n", alphabetically. You could use any number of other special characters, but sticking to something that no rational tokenizer would ever split on makes things easier.

More importantly with negative values, comparison breaks because while -1.33 is numerically less than -1.32, "n000000000000001d33000000" is lexicographically greater than "n000000000000001d32000000".

The solution is to "invert" the magnitude of negative numbers. You do this by subtracting each digit in the number from 9. In other words, subtracting your value from 999999999999999.99999999, aka 99.99. So, "n000000000000001d32000000" becomes "n999999999999998d67999999".

Here is some Java code to convert doubles to the p/n format:

package com.bullhorn.common.lucene;

import java.text.DecimalFormat;
import java.text.NumberFormat;

/**
 * Format decimals as strings for Lucene
 * Basic format is: p00000000000051d50300000 = 51.503
 * This is done so the values are lexicographically comparable
 * Negative numbers need to be "inverted" in magnitude, so they
 * can be compared to positive values. Negatives are also prepended
 * with "n". For example: n99999999999999d86799999 = -0.132
 * 
 * This way, -0.132 > -0.133 b/c n99999999999999d86_7_99999 > n99999999999999d86_6_99999
 * also, 0.132 > -0.133 b/c p00000000000000d13200000 > n99999999999999d86699999 (b/c p > n)
 * 
 * See: http://wiki.apache.org/lucene-java/SearchNumericalFields#head-19a4340c637fad4601e9ef9db2f7a41dbf2d7518 
 * 
 * @author chase
 */
public class DecimalFormatter {
 
 public static final NumberFormat NUMBER_FORMATTER = new DecimalFormat("00000000000000.00000000");

 public static String formatDouble(double x) {
  
     String latStr = NUMBER_FORMATTER.format(x);
     if (latStr.startsWith("-")) {
         latStr = invertNegativeDouble("n".concat(latStr.substring(1)));
     } else {
         latStr = "p".concat(latStr);
     }
     return latStr.replaceAll("\\.", "d");
 }
 
 public static String invertNegativeDouble(String negDbl) {
  
  String value = "";
  for (int i = 0; i < negDbl.length(); i++) { 
   char digit = negDbl.charAt(i);
   if (digit >= '0' && digit <= '9')
    value += String.valueOf(('9' - digit));
   else value += digit;
  }
  
  return value;
 } 

}