58 lines
1.2 KiB
Markdown
58 lines
1.2 KiB
Markdown
|
# Execution runners based on Celery
|
||
|
|
|
||
|
|
Celery is a feature rich task queue implemented in Python and provides easy
|
||
|
|
means to scale and distribute workloads across also large numbers of workers.
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
|
||
|
|
pip3 install celery
|
||
|
|
dnf install rabbitmq-server
|
||
|
|
|
||
|
|
## How to Run
|
||
|
|
|
||
|
|
|
||
|
|
# 1) make sure rabbitmq or other "message broker" is running
|
||
|
|
service rabbitmq-server start
|
||
|
|
|
||
|
|
# 2) startup a celery worker
|
||
|
|
celery worker --loglevel=info
|
||
|
|
|
||
|
|
cd ..
|
||
|
|
celery -A runner worker --loglevel=info
|
||
|
|
|
||
|
|
|
||
|
|
# if CELERY_IMPORTS is not set in celeryconfig.py, tasks to import need to be specified
|
||
|
|
celery -A runner.tasks worker --loglevel=info
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Testing the worker
|
||
|
|
|
||
|
|
Manual testing a dummy task (e.g. adding two numbers):
|
||
|
|
|
||
|
|
# in terminal
|
||
|
|
python3
|
||
|
|
|
||
|
|
# in python3 console
|
||
|
|
from runner.tasks_debug import add
|
||
|
|
add.delay(4,7) # out: <AsyncResult: e76d2895-e38e-47e1-985e-71e507384548>
|
||
|
|
|
||
|
|
r = add.delay(2,3)
|
||
|
|
r.status # out: 'SUCCESS'
|
||
|
|
r.result # out: 5
|
||
|
|
|
||
|
|
## Resources and References
|
||
|
|
|
||
|
|
|
||
|
|
Getting Started
|
||
|
|
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html
|
||
|
|
|
||
|
|
Celery running workers as daemon
|
||
|
|
http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#daemonizing
|
||
|
|
|
||
|
|
|