Rundeck - any command execution fails when running on 5.8k nodes
I'm running a rundeck server to delegate a simple script to 5.8k other linux servers.
The very simple script is bellow
!/bin/bash
A=$(hostname)
echo $A
When i run the same job with a smaller number of targets (4089 nodes)
the comands work fine
I tried looking at my service.log page and its not incrementing anything
Any ideas on how to be able to run on all the 5.8k nodes? And where should i look for errors?
rundeck
asked Nov 20 at 13:24
Henrique Capozzi
31
add a comment |
I'm running a rundeck server to delegate a simple script to 5.8k other linux servers.
The very simple script is bellow
!/bin/bash
A=$(hostname)
echo $A
When i run the same job with a smaller number of targets (4089 nodes)
the comands work fine
I tried looking at my service.log page and its not incrementing anything
Any ideas on how to be able to run on all the 5.8k nodes? And where should i look for errors?
rundeck
asked Nov 20 at 13:24
Henrique Capozzi
31
add a comment |
I'm running a rundeck server to delegate a simple script to 5.8k other linux servers.
The very simple script is bellow
!/bin/bash
A=$(hostname)
echo $A
When i run the same job with a smaller number of targets (4089 nodes)
the comands work fine
I tried looking at my service.log page and its not incrementing anything
Any ideas on how to be able to run on all the 5.8k nodes? And where should i look for errors?
rundeck
asked Nov 20 at 13:24
Henrique Capozzi
31
I'm running a rundeck server to delegate a simple script to 5.8k other linux servers.
The very simple script is bellow
!/bin/bash
A=$(hostname)
echo $A
When i run the same job with a smaller number of targets (4089 nodes)
the comands work fine
I tried looking at my service.log page and its not incrementing anything
Any ideas on how to be able to run on all the 5.8k nodes? And where should i look for errors?
rundeck
rundeck
asked Nov 20 at 13:24
Henrique Capozzi
31
asked Nov 20 at 13:24
Henrique Capozzi
31
asked Nov 20 at 13:24
Henrique Capozzi
31
asked Nov 20 at 13:24
Henrique Capozzi
31
asked Nov 20 at 13:24
Henrique Capozzi
31
31
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Rundeck does not have limits to nodes, certainly depends on how many executions you want to run, how much ram, how many processors and disk space.
Maybe you need to increase the Java heap size:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#java-heap-size
And how to adapt this to your SSH plugin:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#built-in-ssh-plugins
answered Nov 20 at 18:20
ReinerRDECK
262
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393996%2frundeck-any-command-execution-fails-when-running-on-5-8k-nodes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Rundeck does not have limits to nodes, certainly depends on how many executions you want to run, how much ram, how many processors and disk space.
Maybe you need to increase the Java heap size:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#java-heap-size
And how to adapt this to your SSH plugin:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#built-in-ssh-plugins
answered Nov 20 at 18:20
ReinerRDECK
262
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
add a comment |
Rundeck does not have limits to nodes, certainly depends on how many executions you want to run, how much ram, how many processors and disk space.
Maybe you need to increase the Java heap size:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#java-heap-size
And how to adapt this to your SSH plugin:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#built-in-ssh-plugins
answered Nov 20 at 18:20
ReinerRDECK
262
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
add a comment |
Rundeck does not have limits to nodes, certainly depends on how many executions you want to run, how much ram, how many processors and disk space.
Maybe you need to increase the Java heap size:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#java-heap-size
And how to adapt this to your SSH plugin:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#built-in-ssh-plugins
answered Nov 20 at 18:20
ReinerRDECK
262
Rundeck does not have limits to nodes, certainly depends on how many executions you want to run, how much ram, how many processors and disk space.
Maybe you need to increase the Java heap size:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#java-heap-size
And how to adapt this to your SSH plugin:
https://rundeck.org/docs/administration/maintenance/tuning-rundeck.html#built-in-ssh-plugins
answered Nov 20 at 18:20
ReinerRDECK
262
answered Nov 20 at 18:20
ReinerRDECK
262
answered Nov 20 at 18:20
ReinerRDECK
262
answered Nov 20 at 18:20
ReinerRDECK
262
262
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
add a comment |
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
Thanks for the tip, i don't recall seen any erors on the services.log but it'll try checking the JVM heap size
– Henrique Capozzi
Nov 24 at 16:55
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
No changes on increasing the JMV heap size....I get a "Null error" but I did not found any errors on the /var/log/rundeck/* logs, even after changing the runceck log level to DEBUG... ReinerRDECK any ideias on where to go to debug it?
– Henrique Capozzi
Nov 29 at 13:05
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
And about SSH Plugins? Check in documentation: "We chose the rpm -q command which checks against the rpm database to see if a particular package was installed. For 1000 nodes we saw an average execution of 52 seconds. A 4000 node cluster took roughly 3.5 minutes, and 8000 node cluster about 7 minutes. The main limitation appears to be memory of the JVM instance relative to the number of concurrent requests. We tuned the max memory to be 12GB with a 1000 Concurrent Dispatch Threads to 1GB of Memory. GC appears to behave well during the runs given the “bursty” nature of them."
– ReinerRDECK
Nov 29 at 15:09
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393996%2frundeck-any-command-execution-fails-when-running-on-5-8k-nodes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
