Apache Storm：Will the late tuple be ack or fail?
0
I have the following information in my log.
Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?
Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?
Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
Update:
My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink
Storm Version：1.2.1
CountBolt：
builder.setBolt("count_bolt", new CountBlot()
.withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS)) //3s
.withLag (new Duration(time_windowlag, TimeUnit.SECONDS)) // 1s
.withTimestampField("timestamp")
, parallelism)
.fieldsGrouping("split_bolt", new Fields("word"));
apache-storm
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
add a comment |
0
I have the following information in my log.
Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?
Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?
Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
Update:
My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink
Storm Version：1.2.1
CountBolt：
builder.setBolt("count_bolt", new CountBlot()
.withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS)) //3s
.withLag (new Duration(time_windowlag, TimeUnit.SECONDS)) // 1s
.withTimestampField("timestamp")
, parallelism)
.fieldsGrouping("split_bolt", new Fields("word"));
apache-storm
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
By looking at the source code, it is found that the late tuple will be acked.if (lateTupleStream != null) { windowedOutputCollector.emit(lateTupleStream, input, new Values(input)); } else { LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts); } windowedOutputCollector.ack(input);Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.
– Cheng Jiang
Nov 26 '18 at 7:19
You should provide some more details, like the topology you are ran, the storm version...
– Jacky
Nov 26 '18 at 10:21
add a comment |
0
0
0
I have the following information in my log.
Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?
Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?
Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
Update:
My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink
Storm Version：1.2.1
CountBolt：
builder.setBolt("count_bolt", new CountBlot()
.withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS)) //3s
.withLag (new Duration(time_windowlag, TimeUnit.SECONDS)) // 1s
.withTimestampField("timestamp")
, parallelism)
.fieldsGrouping("split_bolt", new Fields("word"));
apache-storm
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
I have the following information in my log.
Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?
Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?
Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.
2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.
Update:
My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink
Storm Version：1.2.1
CountBolt：
builder.setBolt("count_bolt", new CountBlot()
.withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS)) //3s
.withLag (new Duration(time_windowlag, TimeUnit.SECONDS)) // 1s
.withTimestampField("timestamp")
, parallelism)
.fieldsGrouping("split_bolt", new Fields("word"));
apache-storm
apache-storm
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
edited Nov 26 '18 at 12:13
Cheng Jiang
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
asked Nov 26 '18 at 1:55
Cheng JiangCheng Jiang
486
486
By looking at the source code, it is found that the late tuple will be acked.if (lateTupleStream != null) { windowedOutputCollector.emit(lateTupleStream, input, new Values(input)); } else { LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts); } windowedOutputCollector.ack(input);Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.
– Cheng Jiang
Nov 26 '18 at 7:19
You should provide some more details, like the topology you are ran, the storm version...
– Jacky
Nov 26 '18 at 10:21
add a comment |
By looking at the source code, it is found that the late tuple will be acked.if (lateTupleStream != null) { windowedOutputCollector.emit(lateTupleStream, input, new Values(input)); } else { LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts); } windowedOutputCollector.ack(input);Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.
– Cheng Jiang
Nov 26 '18 at 7:19
You should provide some more details, like the topology you are ran, the storm version...
– Jacky
Nov 26 '18 at 10:21
By looking at the source code, it is found that the late tuple will be acked.
if (lateTupleStream != null) { windowedOutputCollector.emit(lateTupleStream, input, new Values(input)); } else { LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts); } windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.– Cheng Jiang
Nov 26 '18 at 7:19
By looking at the source code, it is found that the late tuple will be acked.
if (lateTupleStream != null) { windowedOutputCollector.emit(lateTupleStream, input, new Values(input)); } else { LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts); } windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.– Cheng Jiang
Nov 26 '18 at 7:19
You should provide some more details, like the topology you are ran, the storm version...
– Jacky
Nov 26 '18 at 10:21
You should provide some more details, like the topology you are ran, the storm version...
– Jacky
Nov 26 '18 at 10:21
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473846%2fapache-storm-will-the-late-tuple-be-ack-or-fail%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473846%2fapache-storm-will-the-late-tuple-be-ack-or-fail%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
By looking at the source code, it is found that the late tuple will be acked.
if (lateTupleStream != null) { windowedOutputCollector.emit(lateTupleStream, input, new Values(input)); } else { LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts); } windowedOutputCollector.ack(input);Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.– Cheng Jiang
Nov 26 '18 at 7:19
You should provide some more details, like the topology you are ran, the storm version...
– Jacky
Nov 26 '18 at 10:21