Error handling while using chain of groups in celery
up vote
0
down vote
favorite
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
asked Nov 17 at 14:26
DUDE_MXP
45449
add a comment |
up vote
0
down vote
favorite
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
asked Nov 17 at 14:26
DUDE_MXP
45449
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 at 16:00
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
asked Nov 17 at 14:26
DUDE_MXP
45449
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
django celery celery-task celerybeat djcelery
asked Nov 17 at 14:26
DUDE_MXP
45449
asked Nov 17 at 14:26
DUDE_MXP
45449
edited Nov 17 at 14:40
asked Nov 17 at 14:26
DUDE_MXP
45449
asked Nov 17 at 14:26
DUDE_MXP
45449
asked Nov 17 at 14:26
DUDE_MXP
45449
45449
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 at 16:00
add a comment |
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 at 16:00
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 at 15:53
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 at 16:00
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 at 16:00
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 at 16:03
schillingt
4,88211620
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 at 16:03
schillingt
4,88211620
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
add a comment |
up vote
0
down vote
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 at 16:03
schillingt
4,88211620
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
add a comment |
up vote
0
down vote
up vote
0
down vote
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 at 16:03
schillingt
4,88211620
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 at 16:03
schillingt
4,88211620
answered Nov 17 at 16:03
schillingt
4,88211620
answered Nov 17 at 16:03
schillingt
4,88211620
answered Nov 17 at 16:03
schillingt
4,88211620
4,88211620
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
add a comment |
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 at 16:14
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53352122%2ferror-handling-while-using-chain-of-groups-in-celery%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 at 16:00