Cannot submit jobs to pbs within R when the R source itself is invoked by a job submitted to pbs
Say I have the following documents in a directory,
> dir()
# [1] "master.job" "master.R" "P1.job" "P2.job" "P1.R" "P2.R"
master.job is submitted to pbs to soruce master.R, P1.job to source P1.R, P2.job to source P2.R etc... The source master.R creates new Px.job's and Px.R's from time to time, and submit the new job created using the command (in R) system("qsub Px.job"). master.R acts as a top-layer monitoring process and determines when to create and submit new jobs.
When connected to the cluster using SSH, I can simply invoke R in the console, then use the command:
> source("master.R")
And the process runs without problem. But this way, I will have to leave my own PC turned on, because once I disconnect from the server, the process will be killed. Therefore, I need to submit the job master.job to pbs for my purpose. However, when I did it this way, I encountered the following error when arriving at the step system("qsub Px.job"):
socket_connect_unix failed: 15137
qsub: cannot connect to server (null) (errno=15137) could not connect to trqauthd
Any suggestion? It is clear that I can submit a job through R console (either system("qsub Px.job") or source("master.R")) , but then why doesn't it work to submit the same job (Px.job) through another R process (master.R), when it is invoked by a submitted job (master.job)?
I am quite sure somebody will ask why I need to do something like that.
At the beginning I only had one job to submit: P.job, which sources P.R. P.R is a source code that repeatedly invokes another proprietary program X. The program X has a bug, that is not solved in the current version, where it freezes forever at some points. When it is freezed, P.R won't be able to detect it (I cannot make any change to program X either). Therefore, I need another separate, independent process submitted to pbs that can (1) kill (with qdel) the submitted job P.job (and thus P.R) when disk I/O at certain directories is absent for a specific amount of time, (2) check the progress, and (3) resubmit another P.job (which sources another P.R) according to the progress.
r unix pbs qsub
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
add a comment |
Say I have the following documents in a directory,
> dir()
# [1] "master.job" "master.R" "P1.job" "P2.job" "P1.R" "P2.R"
master.job is submitted to pbs to soruce master.R, P1.job to source P1.R, P2.job to source P2.R etc... The source master.R creates new Px.job's and Px.R's from time to time, and submit the new job created using the command (in R) system("qsub Px.job"). master.R acts as a top-layer monitoring process and determines when to create and submit new jobs.
When connected to the cluster using SSH, I can simply invoke R in the console, then use the command:
> source("master.R")
And the process runs without problem. But this way, I will have to leave my own PC turned on, because once I disconnect from the server, the process will be killed. Therefore, I need to submit the job master.job to pbs for my purpose. However, when I did it this way, I encountered the following error when arriving at the step system("qsub Px.job"):
socket_connect_unix failed: 15137
qsub: cannot connect to server (null) (errno=15137) could not connect to trqauthd
Any suggestion? It is clear that I can submit a job through R console (either system("qsub Px.job") or source("master.R")) , but then why doesn't it work to submit the same job (Px.job) through another R process (master.R), when it is invoked by a submitted job (master.job)?
I am quite sure somebody will ask why I need to do something like that.
At the beginning I only had one job to submit: P.job, which sources P.R. P.R is a source code that repeatedly invokes another proprietary program X. The program X has a bug, that is not solved in the current version, where it freezes forever at some points. When it is freezed, P.R won't be able to detect it (I cannot make any change to program X either). Therefore, I need another separate, independent process submitted to pbs that can (1) kill (with qdel) the submitted job P.job (and thus P.R) when disk I/O at certain directories is absent for a specific amount of time, (2) check the progress, and (3) resubmit another P.job (which sources another P.R) according to the progress.
r unix pbs qsub
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
add a comment |
Say I have the following documents in a directory,
> dir()
# [1] "master.job" "master.R" "P1.job" "P2.job" "P1.R" "P2.R"
master.job is submitted to pbs to soruce master.R, P1.job to source P1.R, P2.job to source P2.R etc... The source master.R creates new Px.job's and Px.R's from time to time, and submit the new job created using the command (in R) system("qsub Px.job"). master.R acts as a top-layer monitoring process and determines when to create and submit new jobs.
When connected to the cluster using SSH, I can simply invoke R in the console, then use the command:
> source("master.R")
And the process runs without problem. But this way, I will have to leave my own PC turned on, because once I disconnect from the server, the process will be killed. Therefore, I need to submit the job master.job to pbs for my purpose. However, when I did it this way, I encountered the following error when arriving at the step system("qsub Px.job"):
socket_connect_unix failed: 15137
qsub: cannot connect to server (null) (errno=15137) could not connect to trqauthd
Any suggestion? It is clear that I can submit a job through R console (either system("qsub Px.job") or source("master.R")) , but then why doesn't it work to submit the same job (Px.job) through another R process (master.R), when it is invoked by a submitted job (master.job)?
I am quite sure somebody will ask why I need to do something like that.
At the beginning I only had one job to submit: P.job, which sources P.R. P.R is a source code that repeatedly invokes another proprietary program X. The program X has a bug, that is not solved in the current version, where it freezes forever at some points. When it is freezed, P.R won't be able to detect it (I cannot make any change to program X either). Therefore, I need another separate, independent process submitted to pbs that can (1) kill (with qdel) the submitted job P.job (and thus P.R) when disk I/O at certain directories is absent for a specific amount of time, (2) check the progress, and (3) resubmit another P.job (which sources another P.R) according to the progress.
r unix pbs qsub
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
Say I have the following documents in a directory,
> dir()
# [1] "master.job" "master.R" "P1.job" "P2.job" "P1.R" "P2.R"
master.job is submitted to pbs to soruce master.R, P1.job to source P1.R, P2.job to source P2.R etc... The source master.R creates new Px.job's and Px.R's from time to time, and submit the new job created using the command (in R) system("qsub Px.job"). master.R acts as a top-layer monitoring process and determines when to create and submit new jobs.
When connected to the cluster using SSH, I can simply invoke R in the console, then use the command:
> source("master.R")
And the process runs without problem. But this way, I will have to leave my own PC turned on, because once I disconnect from the server, the process will be killed. Therefore, I need to submit the job master.job to pbs for my purpose. However, when I did it this way, I encountered the following error when arriving at the step system("qsub Px.job"):
socket_connect_unix failed: 15137
qsub: cannot connect to server (null) (errno=15137) could not connect to trqauthd
Any suggestion? It is clear that I can submit a job through R console (either system("qsub Px.job") or source("master.R")) , but then why doesn't it work to submit the same job (Px.job) through another R process (master.R), when it is invoked by a submitted job (master.job)?
I am quite sure somebody will ask why I need to do something like that.
At the beginning I only had one job to submit: P.job, which sources P.R. P.R is a source code that repeatedly invokes another proprietary program X. The program X has a bug, that is not solved in the current version, where it freezes forever at some points. When it is freezed, P.R won't be able to detect it (I cannot make any change to program X either). Therefore, I need another separate, independent process submitted to pbs that can (1) kill (with qdel) the submitted job P.job (and thus P.R) when disk I/O at certain directories is absent for a specific amount of time, (2) check the progress, and (3) resubmit another P.job (which sources another P.R) according to the progress.
r unix pbs qsub
r unix pbs qsub
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
asked Nov 26 '18 at 11:31
Matthew HuiMatthew Hui
16210
16210
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53480182%2fcannot-submit-jobs-to-pbs-within-r-when-the-r-source-itself-is-invoked-by-a-job%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53480182%2fcannot-submit-jobs-to-pbs-within-r-when-the-r-source-itself-is-invoked-by-a-job%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
