Limit celery concurrency workers to limit memory usage

Limit the number of concurrent workers for celery message processing.

We are using sentry for error logging and monitoring, and recently my sentry server moved from one shared server to another and with that it started crashing because of memory usage way over limit (shared server has limit of 1024MB and my celery worker was taking 2759MB).

When I saw the log, I was surprised looking at number of worker processes -

user - 66MB - 0:06:45 - 21356 - [celeryd: celery@ servername:Worker:MainProcess] -active- (celery worker -B)
user - 68MB - 0:06:41 - 21657 - gunicorn: worker [Sentry]
user - 69MB - 0:06:40 - 21658 - gunicorn: worker [Sentry]
user - 61MB - 0:06:40 - 21659 - [celery beat]
user - 61MB - 0:06:40 - 21660 - [celeryd: celery@servername:Worker-2]
user - 61MB - 0:06:40 - 21661 - [celeryd: celery@servername:Worker-3]
user - 61MB - 0:06:40 - 21662 - [celeryd: celery@servername:Worker-4]
user - 61MB - 0:06:40 - 21663 - [celeryd: celery@servername:Worker-5]
user - 61MB - 0:06:40 - 21664 - [celeryd: celery@servername:Worker-6]
user - 59MB - 0:06:40 - 21665 - [celeryd: celery@servername:Worker-7]
user - 59MB - 0:06:40 - 21666 - [celeryd: celery@servername:Worker-8]
user - 61MB - 0:06:40 - 21667 - [celeryd: celery@servername:Worker-9]
user - 59MB - 0:06:40 - 21668 - [celeryd: celery@servername:Worker-10]
user - 61MB - 0:06:40 - 21669 - [celeryd: celery@servername:Worker-11]
user - 61MB - 0:06:40 - 21670 - [celeryd: celery@servername:Worker-12]
user - 59MB - 0:06:40 - 21671 - [celeryd: celery@servername:Worker-13]
user - 61MB - 0:06:40 - 21672 - [celeryd: celery@servername:Worker-14]
user - 59MB - 0:06:40 - 21673 - [celeryd: celery@servername:Worker-15]
user - 59MB - 0:06:40 - 21675 - [celeryd: celery@servername:Worker-16]
user - 61MB - 0:06:40 - 21677 - [celeryd: celery@servername:Worker-17]
user - 61MB - 0:06:40 - 21678 - [celeryd: celery@servername:Worker-18]
.....
user - 61MB - 0:06:40 - 21678 - [celeryd: celery@servername:Worker-44]


Looking at above, obviously it was going to kill the process as the usage was going much higher than allowed.

It was happening because of default concurrent process setting for celery. As per the documentation -

-c, --concurrency

Number of child processes processing the queue. The default is the number of CPUs available on your system.

In this case probably the number of CPU of the server was much higher. So the solution was to set the restriction by providing concurrency you want. 

> celery worker -B --concurrency=4

It resolves the issue. You can validate number of celery process by - 

> ps -ef | grep celeryd

Hope it helps!