Friday, July 9, 2010

Django/Celery Quickstart (or, how I learned to stop using cron and love celery)

16 comments:

  1. Thanks for your article pointing that! I'll certainly consider using celery instead of cron on my next project.

    ReplyDelete
  2. Thanks for the nice write up. It would have saved me a few hours if it had been posted a week earlier. Then again I used RabbitMQ but I have another project where ghettoq might be useful so that is a useful option that I did not know about.

    Here is a RabbotMQ tip: Note that you need to process a given queue at least once before events sent to that queue will show up in the standard rabbit commands that list queues, the number of queued events and such.

    ReplyDelete
  3. Awesome article. I think this is just the kick I needed to get started playing with celery.

    I really appreciated the way this was written - starting with the simplest thing that could work and then prompting to read on and expand on your own.

    I look forward to reading more!

    ReplyDelete
  4. What versions of djcelery and ghettoq were you using? There have been updates recently, and since I can't seem to get your simple example to work, I wonder if something broke.

    ReplyDelete
  5. I am using Django 1.1.1, djcelery 2.0.0 and ghettoq-0.1.2

    ReplyDelete
  6. Thanks - there was a problem with Django 1.2.1 + djcelery 2.0.1 + ghettoq 0.4.0 that was fixed with ghettoq 0.4.1 today.

    ReplyDelete
  7. Cool — this is exactly what I was looking for. Thanks!

    ReplyDelete
  8. Thanks a lot for your good article. Would you please guide me how I can setup django-celery on a production server? I want to know how celeryd can be started automatically without any need to use manage.py manually.

    ReplyDelete
  9. @rozita

    See this init script here:

    http://github.com/ask/celery/blob/master/contrib/generic-init.d/celeryd

    Here's my /etc/default/celeryd configuration file, which sets up Celery for a Django project in /opt/Opal/release:
    http://pastie.org/1216111

    ReplyDelete
  10. I was just looking for a simple way to use Celery. Their documentation frankly *sucks* ("First Steps with Celery" entices you with how easy it is without showing you how to set it up), unless you care to learn everything about it from the get-go. So even though I intend to use celery for queuing tasks rather than periodic ones (for now, anyway), your post should be very useful (assuming no snags), thank you for the short start-to-finish steps. I'm glad I don't have to bother with RabbitMQ for now.

    ReplyDelete
  11. At this command
    1. sudo ./manage.py celeryd -v 2 -B -s celery -E -l INFO

    You must add "-I tasks" to load tasks.py file ...

    ReplyDelete
  12. Huh... good catch Kiuz. You can also add the following to your settings.py:

    CELERY_IMPORTS = ("tasks", )

    ReplyDelete
  13. This blog, whilst insightful, is sadly out of date.

    ghettoq is deprecated and no longer compatible with djcelery. kombu is used instead.

    ReplyDelete
  14. Good catch Andrew. I just updated the post.

    ReplyDelete
  15. Thank you for the up-to-date instructions, and for making celery simple.

    For a lightweight task queue with cron support that is easily integrated to Django, you also have two projects by Charles Leifer: the django-utils queue, which has no hard dependencies, and Huey, which currently requires Redis.

    To keep configuration minimal, django-supervisor makes it easy and convenient to spawn the queue engine along with your choice of runserver or wsgi container.

    ReplyDelete
  16. Thanks for the instructions. A suggestion: " This guide starts out using kombu, which is backed by the database Django is already using." makes it seem like kombu is backed by the Django database. My understanding is that Kombu is implemented with a variety of transports, one of which is Django. Found it a bit confusing as a messaging newbie.

    ReplyDelete