Play Nicely - Celery, RabbitMQ & ELB Idle Timeout

14 Mar 2016
Python AWS Celery

Timeout

Please Note:
This post appears to be over 6 months old.

Celery is a sophisticated tool for distributing workloads. However, if you've ever used Celery behind Amazon's Elastic Load Balancer (ELB), you may have run into problems with timeouts.

When RabbitMQ and Celery are behind an Elastic Load Balancer, problems occur when ELB sends TCP connections to your EC2 instances. In a normal request/response cycle, the ELB keeps the connection open for a short time afterwards (default is 60 seconds). Known as the 'idle timeout', under certain circumstances, Celery won't be notified that the connection is closed.

The good news is AWS allows us to modify the idle timeout value up to 17 minutes. The bad news, this will NOT solve the problem for obvious reasons. To help address this issue of messages being lost in transit, RabbitMQ added the extension 'Publisher Confirms' to AMQP. Although currently undocumented, Celery supports this extension when using pyamqp by using the following setting in the broker transport option:

BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}

Note: By enabling this option, your application will take a small performance hit as each task waits for confirmation.

That's all Folks!

Thanks for reading. Let's keep in Touch:
Follow me on GitHub or Twitter @glynjackson


Glyn Jackson is a Python Nerd, consultant and developer.


Find out more