Best approach for multiple Django projects and async emails
I'm using Django 2 with Python 3.5 in a Ubuntu server.
I have multiple Django projects, this projects are using the same virutalenv, because they have the same source code.
For making the email sending async, I'm using post-office app for Django, this requires to run a crontab every minute to send the queued emails in database.
But this approach is getting hard to maintain with a great number of Django projects in the same server, so every minute I have a great CPU demand spike.
The emails are being handled by AWS SES.
I'm thinking in use Celery with RabbitMQ, but this is getting confusing for me.
Will I have multiple Celery and one RabbitMQ instances?
Can this be done without interference?
Are there a way to use just one async task process, an alternative for Celery and RebbitMQ?
python django celery ses
asked Nov 22 '18 at 4:48
GuiGui
333114
add a comment |
I'm using Django 2 with Python 3.5 in a Ubuntu server.
I have multiple Django projects, this projects are using the same virutalenv, because they have the same source code.
For making the email sending async, I'm using post-office app for Django, this requires to run a crontab every minute to send the queued emails in database.
But this approach is getting hard to maintain with a great number of Django projects in the same server, so every minute I have a great CPU demand spike.
The emails are being handled by AWS SES.
I'm thinking in use Celery with RabbitMQ, but this is getting confusing for me.
Will I have multiple Celery and one RabbitMQ instances?
Can this be done without interference?
Are there a way to use just one async task process, an alternative for Celery and RebbitMQ?
python django celery ses
asked Nov 22 '18 at 4:48
GuiGui
333114
What do you mean by instances? processes or servers?
– 2ps
Nov 22 '18 at 5:29
Are multiple Django projects running under apache with vhosts
– Gui
Nov 22 '18 at 21:29
add a comment |
I'm using Django 2 with Python 3.5 in a Ubuntu server.
I have multiple Django projects, this projects are using the same virutalenv, because they have the same source code.
For making the email sending async, I'm using post-office app for Django, this requires to run a crontab every minute to send the queued emails in database.
But this approach is getting hard to maintain with a great number of Django projects in the same server, so every minute I have a great CPU demand spike.
The emails are being handled by AWS SES.
I'm thinking in use Celery with RabbitMQ, but this is getting confusing for me.
Will I have multiple Celery and one RabbitMQ instances?
Can this be done without interference?
Are there a way to use just one async task process, an alternative for Celery and RebbitMQ?
python django celery ses
asked Nov 22 '18 at 4:48
GuiGui
333114
I'm using Django 2 with Python 3.5 in a Ubuntu server.
I have multiple Django projects, this projects are using the same virutalenv, because they have the same source code.
For making the email sending async, I'm using post-office app for Django, this requires to run a crontab every minute to send the queued emails in database.
But this approach is getting hard to maintain with a great number of Django projects in the same server, so every minute I have a great CPU demand spike.
The emails are being handled by AWS SES.
I'm thinking in use Celery with RabbitMQ, but this is getting confusing for me.
Will I have multiple Celery and one RabbitMQ instances?
Can this be done without interference?
Are there a way to use just one async task process, an alternative for Celery and RebbitMQ?
python django celery ses
python django celery ses
asked Nov 22 '18 at 4:48
GuiGui
333114
asked Nov 22 '18 at 4:48
GuiGui
333114
asked Nov 22 '18 at 4:48
GuiGui
333114
asked Nov 22 '18 at 4:48
GuiGui
333114
asked Nov 22 '18 at 4:48
GuiGui
333114
333114
What do you mean by instances? processes or servers?
– 2ps
Nov 22 '18 at 5:29
Are multiple Django projects running under apache with vhosts
– Gui
Nov 22 '18 at 21:29
add a comment |
What do you mean by instances? processes or servers?
– 2ps
Nov 22 '18 at 5:29
Are multiple Django projects running under apache with vhosts
– Gui
Nov 22 '18 at 21:29
What do you mean by instances? processes or servers?
– 2ps
Nov 22 '18 at 5:29
What do you mean by instances? processes or servers?
– 2ps
Nov 22 '18 at 5:29
Are multiple Django projects running under apache with vhosts
– Gui
Nov 22 '18 at 21:29
Are multiple Django projects running under apache with vhosts
– Gui
Nov 22 '18 at 21:29
add a comment |
1 Answer
1
active
oldest
votes
Assuming instances means ec2 instances, you can have as many (or as few) celery instances and as many (or few) rabbitmq instances as you like (rabbit mq can be clustered). All you have to do is make sure all of your producers (your django webservers) and all of your consumers (your celery workers) point to the same rabbit mq url as a broker.
Note, since you're using AWS, you can just as easily set up celery using Amazon's SQS service or by using AWS_hosted redis via elasticache.
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53424057%2fbest-approach-for-multiple-django-projects-and-async-emails%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Assuming instances means ec2 instances, you can have as many (or as few) celery instances and as many (or few) rabbitmq instances as you like (rabbit mq can be clustered). All you have to do is make sure all of your producers (your django webservers) and all of your consumers (your celery workers) point to the same rabbit mq url as a broker.
Note, since you're using AWS, you can just as easily set up celery using Amazon's SQS service or by using AWS_hosted redis via elasticache.
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
add a comment |
Assuming instances means ec2 instances, you can have as many (or as few) celery instances and as many (or few) rabbitmq instances as you like (rabbit mq can be clustered). All you have to do is make sure all of your producers (your django webservers) and all of your consumers (your celery workers) point to the same rabbit mq url as a broker.
Note, since you're using AWS, you can just as easily set up celery using Amazon's SQS service or by using AWS_hosted redis via elasticache.
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
add a comment |
Assuming instances means ec2 instances, you can have as many (or as few) celery instances and as many (or few) rabbitmq instances as you like (rabbit mq can be clustered). All you have to do is make sure all of your producers (your django webservers) and all of your consumers (your celery workers) point to the same rabbit mq url as a broker.
Note, since you're using AWS, you can just as easily set up celery using Amazon's SQS service or by using AWS_hosted redis via elasticache.
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
Assuming instances means ec2 instances, you can have as many (or as few) celery instances and as many (or few) rabbitmq instances as you like (rabbit mq can be clustered). All you have to do is make sure all of your producers (your django webservers) and all of your consumers (your celery workers) point to the same rabbit mq url as a broker.
Note, since you're using AWS, you can just as easily set up celery using Amazon's SQS service or by using AWS_hosted redis via elasticache.
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
answered Nov 22 '18 at 5:33
2ps2ps
7,4922930
7,4922930
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53424057%2fbest-approach-for-multiple-django-projects-and-async-emails%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What do you mean by instances? processes or servers?
– 2ps
Nov 22 '18 at 5:29
Are multiple Django projects running under apache with vhosts
– Gui
Nov 22 '18 at 21:29