My storm topology fails after running for 10 days
My storm topology fails after running for 10 days , when I deploy the same topology (same JAR) with new name it runs well till date, so my question is that what are the new resources got allocated for newly deployed Storm topology including zookeeper memory. If I redeployed that topology with OLD name its fails again in few hours.
I have not done any changes before deploying it with new topology name.
Is that Storm topology consume any memory space on worker node after running for longer period which i need to take care of?
apache apache-storm
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
add a comment |
My storm topology fails after running for 10 days , when I deploy the same topology (same JAR) with new name it runs well till date, so my question is that what are the new resources got allocated for newly deployed Storm topology including zookeeper memory. If I redeployed that topology with OLD name its fails again in few hours.
I have not done any changes before deploying it with new topology name.
Is that Storm topology consume any memory space on worker node after running for longer period which i need to take care of?
apache apache-storm
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
Please describe what you mean when you say that your topology "fails". Do you get error logs, does it crash with OOME or what do you mean? Also you should consider listing the components used by your topology (spouts, bolts, is it Trident?) as well as the Storm version.
– Stig Rohde Døssing
Nov 26 '18 at 18:37
what I mean by fail is that after analyzing Storm logs I can see one port stops processing the tuples, that results in huge failures at Spout.It just print the logmetrics info for hours.No sign of tuple processing in the logs
– Milind Pardeshi
Nov 27 '18 at 2:58
2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system memory/heap.initBytes [230686720] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system _receive.sojourn_time [9952.0] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system __receive.overflow
– Milind Pardeshi
Nov 27 '18 at 3:00
Its not trident topology , it contain one spout and eight bolts
– Milind Pardeshi
Nov 27 '18 at 3:02
add a comment |
My storm topology fails after running for 10 days , when I deploy the same topology (same JAR) with new name it runs well till date, so my question is that what are the new resources got allocated for newly deployed Storm topology including zookeeper memory. If I redeployed that topology with OLD name its fails again in few hours.
I have not done any changes before deploying it with new topology name.
Is that Storm topology consume any memory space on worker node after running for longer period which i need to take care of?
apache apache-storm
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
My storm topology fails after running for 10 days , when I deploy the same topology (same JAR) with new name it runs well till date, so my question is that what are the new resources got allocated for newly deployed Storm topology including zookeeper memory. If I redeployed that topology with OLD name its fails again in few hours.
I have not done any changes before deploying it with new topology name.
Is that Storm topology consume any memory space on worker node after running for longer period which i need to take care of?
apache apache-storm
apache apache-storm
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
asked Nov 26 '18 at 12:30
Milind PardeshiMilind Pardeshi
2117
2117
Please describe what you mean when you say that your topology "fails". Do you get error logs, does it crash with OOME or what do you mean? Also you should consider listing the components used by your topology (spouts, bolts, is it Trident?) as well as the Storm version.
– Stig Rohde Døssing
Nov 26 '18 at 18:37
what I mean by fail is that after analyzing Storm logs I can see one port stops processing the tuples, that results in huge failures at Spout.It just print the logmetrics info for hours.No sign of tuple processing in the logs
– Milind Pardeshi
Nov 27 '18 at 2:58
2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system memory/heap.initBytes [230686720] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system _receive.sojourn_time [9952.0] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system __receive.overflow
– Milind Pardeshi
Nov 27 '18 at 3:00
Its not trident topology , it contain one spout and eight bolts
– Milind Pardeshi
Nov 27 '18 at 3:02
add a comment |
Please describe what you mean when you say that your topology "fails". Do you get error logs, does it crash with OOME or what do you mean? Also you should consider listing the components used by your topology (spouts, bolts, is it Trident?) as well as the Storm version.
– Stig Rohde Døssing
Nov 26 '18 at 18:37
what I mean by fail is that after analyzing Storm logs I can see one port stops processing the tuples, that results in huge failures at Spout.It just print the logmetrics info for hours.No sign of tuple processing in the logs
– Milind Pardeshi
Nov 27 '18 at 2:58
2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system memory/heap.initBytes [230686720] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system _receive.sojourn_time [9952.0] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system __receive.overflow
– Milind Pardeshi
Nov 27 '18 at 3:00
Its not trident topology , it contain one spout and eight bolts
– Milind Pardeshi
Nov 27 '18 at 3:02
Please describe what you mean when you say that your topology "fails". Do you get error logs, does it crash with OOME or what do you mean? Also you should consider listing the components used by your topology (spouts, bolts, is it Trident?) as well as the Storm version.
– Stig Rohde Døssing
Nov 26 '18 at 18:37
Please describe what you mean when you say that your topology "fails". Do you get error logs, does it crash with OOME or what do you mean? Also you should consider listing the components used by your topology (spouts, bolts, is it Trident?) as well as the Storm version.
– Stig Rohde Døssing
Nov 26 '18 at 18:37
what I mean by fail is that after analyzing Storm logs I can see one port stops processing the tuples, that results in huge failures at Spout.It just print the logmetrics info for hours.No sign of tuple processing in the logs
– Milind Pardeshi
Nov 27 '18 at 2:58
what I mean by fail is that after analyzing Storm logs I can see one port stops processing the tuples, that results in huge failures at Spout.It just print the logmetrics info for hours.No sign of tuple processing in the logs
– Milind Pardeshi
Nov 27 '18 at 2:58
2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system memory/heap.initBytes [230686720] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system _receive.sojourn_time [9952.0] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system __receive.overflow
– Milind Pardeshi
Nov 27 '18 at 3:00
2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system memory/heap.initBytes [230686720] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system _receive.sojourn_time [9952.0] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system __receive.overflow
– Milind Pardeshi
Nov 27 '18 at 3:00
Its not trident topology , it contain one spout and eight bolts
– Milind Pardeshi
Nov 27 '18 at 3:02
Its not trident topology , it contain one spout and eight bolts
– Milind Pardeshi
Nov 27 '18 at 3:02
add a comment |
1 Answer
1
active
oldest
votes
I'm familiar with at least one bug in Storm pre-1.0.0 that can cause workers to hang. If you aren't on the latest Storm version, try upgrading.
Other than that, your best bet for debugging this is to use jstack or kill -3 on the worker JVM to figure out what your hanging worker is doing. You may also want to enable debug logging if it doesn't harm your performance too much. You do this by doing config.setDebug(true); when setting up the topology.
Once you know why the worker isn't processing tuples you can try posting the stack trace here, maybe there's an issue in Storm.
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53481165%2fmy-storm-topology-fails-after-running-for-10-days%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I'm familiar with at least one bug in Storm pre-1.0.0 that can cause workers to hang. If you aren't on the latest Storm version, try upgrading.
Other than that, your best bet for debugging this is to use jstack or kill -3 on the worker JVM to figure out what your hanging worker is doing. You may also want to enable debug logging if it doesn't harm your performance too much. You do this by doing config.setDebug(true); when setting up the topology.
Once you know why the worker isn't processing tuples you can try posting the stack trace here, maybe there's an issue in Storm.
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
add a comment |
I'm familiar with at least one bug in Storm pre-1.0.0 that can cause workers to hang. If you aren't on the latest Storm version, try upgrading.
Other than that, your best bet for debugging this is to use jstack or kill -3 on the worker JVM to figure out what your hanging worker is doing. You may also want to enable debug logging if it doesn't harm your performance too much. You do this by doing config.setDebug(true); when setting up the topology.
Once you know why the worker isn't processing tuples you can try posting the stack trace here, maybe there's an issue in Storm.
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
add a comment |
I'm familiar with at least one bug in Storm pre-1.0.0 that can cause workers to hang. If you aren't on the latest Storm version, try upgrading.
Other than that, your best bet for debugging this is to use jstack or kill -3 on the worker JVM to figure out what your hanging worker is doing. You may also want to enable debug logging if it doesn't harm your performance too much. You do this by doing config.setDebug(true); when setting up the topology.
Once you know why the worker isn't processing tuples you can try posting the stack trace here, maybe there's an issue in Storm.
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
I'm familiar with at least one bug in Storm pre-1.0.0 that can cause workers to hang. If you aren't on the latest Storm version, try upgrading.
Other than that, your best bet for debugging this is to use jstack or kill -3 on the worker JVM to figure out what your hanging worker is doing. You may also want to enable debug logging if it doesn't harm your performance too much. You do this by doing config.setDebug(true); when setting up the topology.
Once you know why the worker isn't processing tuples you can try posting the stack trace here, maybe there's an issue in Storm.
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
answered Nov 27 '18 at 18:18
Stig Rohde DøssingStig Rohde Døssing
1,891235
1,891235
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53481165%2fmy-storm-topology-fails-after-running-for-10-days%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Please describe what you mean when you say that your topology "fails". Do you get error logs, does it crash with OOME or what do you mean? Also you should consider listing the components used by your topology (spouts, bolts, is it Trident?) as well as the Storm version.
– Stig Rohde Døssing
Nov 26 '18 at 18:37
what I mean by fail is that after analyzing Storm logs I can see one port stops processing the tuples, that results in huge failures at Spout.It just print the logmetrics info for hours.No sign of tuple processing in the logs
– Milind Pardeshi
Nov 27 '18 at 2:58
2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system memory/heap.initBytes [230686720] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system _receive.sojourn_time [9952.0] 2018-11-10 04:17:26.540 o.a.s.m.LoggingMetricsConsumer [INFO] 1541823446 wn0-stormp.t3rhv0qy3pnuzkwi2ieuucmbpf.ax.internal.cloudapp.net:6700 -1:__system __receive.overflow
– Milind Pardeshi
Nov 27 '18 at 3:00
Its not trident topology , it contain one spout and eight bolts
– Milind Pardeshi
Nov 27 '18 at 3:02