icp

History

Julian M. Kunkel 7a0f783aa3 Uses docker to run.		2018-05-06 13:11:47 +01:00
..
README-daemon.md	Initial copy	2018-05-05 23:19:22 +01:00
README.md	Initial copy	2018-05-05 23:19:22 +01:00
celery.py	Initial copy	2018-05-05 23:19:22 +01:00
celeryconfig.py	Initial copy	2018-05-05 23:19:22 +01:00
tasks.py	Uses docker to run.	2018-05-06 13:11:47 +01:00
tasks_debug.py	Initial copy	2018-05-05 23:19:22 +01:00

README.md

Execution runners based on Celery

Celery is a feature rich task queue implemented in Python and provides easy means to scale and distribute workloads across also large numbers of workers.

Requirements

pip3 install celery
dnf install rabbitmq-server

How to Run

# 1) make sure rabbitmq or other "message broker" is running
service rabbitmq-server start

# 2) startup a celery worker
celery worker --loglevel=info

cd ..
celery -A runner worker --loglevel=info


# if CELERY_IMPORTS is not set in celeryconfig.py, tasks to import need to be specified
celery -A runner.tasks worker --loglevel=info

Testing the worker

Manual testing a dummy task (e.g. adding two numbers):

# in terminal
python3

# in python3 console
from runner.tasks_debug import add
add.delay(4,7) # out: <AsyncResult: e76d2895-e38e-47e1-985e-71e507384548>

r = add.delay(2,3)
r.status # out: 'SUCCESS'
r.result # out: 5

Resources and References

Getting Started http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html

Celery running workers as daemon http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#daemonizing