(Linux AMI) EMR Cluster version of `free` command
I'm trying to allocate memory to spark when I start up pyspark from an AWS Linux AMI - ie, pass precise arguments to the driver-memory and executor-memory parameters when I call pyspark.
I understand I can look here at the Instance Types page for the off-the-shelf answer to How much memory do I have on each of this EMR cluster's instances? but I'd like to know if I can do this ("live") from the aws cli itself, so I can see how much I'm actually working with after my bootstrapped spin-up process.
I am also aware of the free command in Linux, but that appears to give me the free memory of only the master node into which I am ssh'd. I am also aware of the YARN UI and Hadoop APIs, but in this case I'd like to check from the command line, if possible.
amazon-web-services apache-spark pyspark yarn aws-cli
edited Nov 20 at 19:23
tk421
3,37131426
asked Nov 20 at 17:37
d8aninja
85831335
add a comment |
I'm trying to allocate memory to spark when I start up pyspark from an AWS Linux AMI - ie, pass precise arguments to the driver-memory and executor-memory parameters when I call pyspark.
I understand I can look here at the Instance Types page for the off-the-shelf answer to How much memory do I have on each of this EMR cluster's instances? but I'd like to know if I can do this ("live") from the aws cli itself, so I can see how much I'm actually working with after my bootstrapped spin-up process.
I am also aware of the free command in Linux, but that appears to give me the free memory of only the master node into which I am ssh'd. I am also aware of the YARN UI and Hadoop APIs, but in this case I'd like to check from the command line, if possible.
amazon-web-services apache-spark pyspark yarn aws-cli
edited Nov 20 at 19:23
tk421
3,37131426
asked Nov 20 at 17:37
d8aninja
85831335
add a comment |
I'm trying to allocate memory to spark when I start up pyspark from an AWS Linux AMI - ie, pass precise arguments to the driver-memory and executor-memory parameters when I call pyspark.
I understand I can look here at the Instance Types page for the off-the-shelf answer to How much memory do I have on each of this EMR cluster's instances? but I'd like to know if I can do this ("live") from the aws cli itself, so I can see how much I'm actually working with after my bootstrapped spin-up process.
I am also aware of the free command in Linux, but that appears to give me the free memory of only the master node into which I am ssh'd. I am also aware of the YARN UI and Hadoop APIs, but in this case I'd like to check from the command line, if possible.
amazon-web-services apache-spark pyspark yarn aws-cli
edited Nov 20 at 19:23
tk421
3,37131426
asked Nov 20 at 17:37
d8aninja
85831335
I'm trying to allocate memory to spark when I start up pyspark from an AWS Linux AMI - ie, pass precise arguments to the driver-memory and executor-memory parameters when I call pyspark.
I understand I can look here at the Instance Types page for the off-the-shelf answer to How much memory do I have on each of this EMR cluster's instances? but I'd like to know if I can do this ("live") from the aws cli itself, so I can see how much I'm actually working with after my bootstrapped spin-up process.
I am also aware of the free command in Linux, but that appears to give me the free memory of only the master node into which I am ssh'd. I am also aware of the YARN UI and Hadoop APIs, but in this case I'd like to check from the command line, if possible.
amazon-web-services apache-spark pyspark yarn aws-cli
amazon-web-services apache-spark pyspark yarn aws-cli
edited Nov 20 at 19:23
tk421
3,37131426
asked Nov 20 at 17:37
d8aninja
85831335
edited Nov 20 at 19:23
tk421
3,37131426
asked Nov 20 at 17:37
d8aninja
85831335
edited Nov 20 at 19:23
tk421
3,37131426
edited Nov 20 at 19:23
tk421
3,37131426
edited Nov 20 at 19:23
tk421
3,37131426
3,37131426
asked Nov 20 at 17:37
d8aninja
85831335
asked Nov 20 at 17:37
d8aninja
85831335
asked Nov 20 at 17:37
d8aninja
85831335
85831335
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The information you are looking for about the cluster memory usage will be available in the yarn manager.
Yarn top
Yarn Manger Provides the information about the nodes and memory used and memory available.
Keeps the track of your all applications.
The following link provides more information about yarn
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
answered Nov 20 at 18:26
Achyuth
1,421818
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
try yarn top command
– Achyuth
Nov 20 at 18:59
Nice! Closest I've seen. Seems to give the sum of allVCoresandMem(GB), so it would need to be divided by the number of number ofNodeManager(s)if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.
– d8aninja
Nov 20 at 19:10
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398542%2flinux-ami-emr-cluster-version-of-free-command%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The information you are looking for about the cluster memory usage will be available in the yarn manager.
Yarn top
Yarn Manger Provides the information about the nodes and memory used and memory available.
Keeps the track of your all applications.
The following link provides more information about yarn
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
answered Nov 20 at 18:26
Achyuth
1,421818
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
try yarn top command
– Achyuth
Nov 20 at 18:59
Nice! Closest I've seen. Seems to give the sum of allVCoresandMem(GB), so it would need to be divided by the number of number ofNodeManager(s)if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.
– d8aninja
Nov 20 at 19:10
add a comment |
The information you are looking for about the cluster memory usage will be available in the yarn manager.
Yarn top
Yarn Manger Provides the information about the nodes and memory used and memory available.
Keeps the track of your all applications.
The following link provides more information about yarn
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
answered Nov 20 at 18:26
Achyuth
1,421818
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
try yarn top command
– Achyuth
Nov 20 at 18:59
Nice! Closest I've seen. Seems to give the sum of allVCoresandMem(GB), so it would need to be divided by the number of number ofNodeManager(s)if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.
– d8aninja
Nov 20 at 19:10
add a comment |
The information you are looking for about the cluster memory usage will be available in the yarn manager.
Yarn top
Yarn Manger Provides the information about the nodes and memory used and memory available.
Keeps the track of your all applications.
The following link provides more information about yarn
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
answered Nov 20 at 18:26
Achyuth
1,421818
The information you are looking for about the cluster memory usage will be available in the yarn manager.
Yarn top
Yarn Manger Provides the information about the nodes and memory used and memory available.
Keeps the track of your all applications.
The following link provides more information about yarn
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
answered Nov 20 at 18:26
Achyuth
1,421818
edited Nov 20 at 19:12
answered Nov 20 at 18:26
Achyuth
1,421818
answered Nov 20 at 18:26
Achyuth
1,421818
answered Nov 20 at 18:26
Achyuth
1,421818
1,421818
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
try yarn top command
– Achyuth
Nov 20 at 18:59
Nice! Closest I've seen. Seems to give the sum of allVCoresandMem(GB), so it would need to be divided by the number of number ofNodeManager(s)if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.
– d8aninja
Nov 20 at 19:10
add a comment |
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
try yarn top command
– Achyuth
Nov 20 at 18:59
Nice! Closest I've seen. Seems to give the sum of allVCoresandMem(GB), so it would need to be divided by the number of number ofNodeManager(s)if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.
– d8aninja
Nov 20 at 19:10
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
Yes, valid answer. I'm also aware that the Hadoop API itself could do this programmatically / if I didn't have access to the YARN UI. But I'd like to see if there s a way to do this right from the command line.
– d8aninja
Nov 20 at 18:41
try yarn top command
– Achyuth
Nov 20 at 18:59
try yarn top command
– Achyuth
Nov 20 at 18:59
Nice! Closest I've seen. Seems to give the sum of all
VCores and Mem(GB), so it would need to be divided by the number of number of NodeManager(s) if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.– d8aninja
Nov 20 at 19:10
Nice! Closest I've seen. Seems to give the sum of all
VCores and Mem(GB), so it would need to be divided by the number of number of NodeManager(s) if I'm reading this correctly, so wont be exactly / per cluster readout. But for now this is top response.– d8aninja
Nov 20 at 19:10
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398542%2flinux-ami-emr-cluster-version-of-free-command%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
