python - Capture Heroku SIGTERM in Celery workers to shutdown worker gracefully -

i've done ton of research on this, , i'm surprised haven't found answer yet anywhere.

i'm running large application on heroku, , have celery tasks run long time processing, , @ end of task save result. every time redeploy on heroku, sends sigterm (and eventually, sigkill) , kills running worker. i'm trying find way worker instance shut down gracefully , re-queue processing later can save required result instead of losing queued task.

i cannot find way works have worker listen sigterm properly. closest i've gotten, works when running python manage.py celeryd directly not when emulating heroku using foreman, following:

@app.task(bind=true, max_retries=1) def slow(self, x):     try:         x in range(100):             print 'x: ' + unicode(x)             time.sleep(10)     except exceptions.maxretriesexceedederror:         logger.error('whoa')     except (exceptions.workershutdown, exceptions.workerterminate) exc:         logger.error(u'retrying, ' + unicode(exc))         raise self.retry(exc=exc, countdown=10)     except (keyboardinterrupt, systemexit) exc:         print 'retrying'         raise self.retry(exc=exc, countdown=10)     else:         return x     finally:         logger.info('task ended!') 

when start celery task running within foreman , hit ctrl+c, following happens:

^csigint received 22:20:59 system   | sending sigterm processes 22:20:59 web.1    | exited code 0 22:21:04 system   | sending sigkill processes killed: 9 

so it's clear none of celery exceptions, nor keyboardinterrupt or systemexit exceptions i've seen in other posts, catch sigterm , shut down worker.

what right way this?

celery unfortunately not designed clean shutdown. ever. mean it. celery workers respond sigterm if task incomplete, worker processes wait finish task , exit. in case, can send sigkill if workers don't shut down in reasonable time there loss of information in case i.e. may not know jobs remained incomplete.