Apache Storm：Will the late tuple be ack or fail?












0















I have the following information in my log.



Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?



Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?



Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?



2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.


Update:



My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink



Storm Version：1.2.1



CountBolt：



 builder.setBolt("count_bolt", new CountBlot()

                .withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS))   //3s

                .withLag (new Duration(time_windowlag, TimeUnit.SECONDS))  // 1s

                .withTimestampField("timestamp")

                , parallelism)

                .fieldsGrouping("split_bolt", new Fields("word"));





apache-storm







share|improve this question

























  • By looking at the source code, it is found that the late tuple will be acked. if (lateTupleStream != null) {                      windowedOutputCollector.emit(lateTupleStream, input, new Values(input));                  } else {                      LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts);                  }                  windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.

    – Cheng Jiang
    Nov 26 '18 at 7:19













  • You should provide some more details, like the topology you are ran, the storm version...

    – Jacky
    Nov 26 '18 at 10:21
















0















I have the following information in my log.



Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?



Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?



Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?



2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.


Update:



My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink



Storm Version：1.2.1



CountBolt：



 builder.setBolt("count_bolt", new CountBlot()

                .withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS))   //3s

                .withLag (new Duration(time_windowlag, TimeUnit.SECONDS))  // 1s

                .withTimestampField("timestamp")

                , parallelism)

                .fieldsGrouping("split_bolt", new Fields("word"));





apache-storm







share|improve this question

























  • By looking at the source code, it is found that the late tuple will be acked. if (lateTupleStream != null) {                      windowedOutputCollector.emit(lateTupleStream, input, new Values(input));                  } else {                      LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts);                  }                  windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.

    – Cheng Jiang
    Nov 26 '18 at 7:19













  • You should provide some more details, like the topology you are ran, the storm version...

    – Jacky
    Nov 26 '18 at 10:21














0












0








0








I have the following information in my log.



Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?



Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?



Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?



2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.


Update:



My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink



Storm Version：1.2.1



CountBolt：



 builder.setBolt("count_bolt", new CountBlot()

                .withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS))   //3s

                .withLag (new Duration(time_windowlag, TimeUnit.SECONDS))  // 1s

                .withTimestampField("timestamp")

                , parallelism)

                .fieldsGrouping("split_bolt", new Fields("word"));





apache-storm







share|improve this question
















I have the following information in my log.



Q1: Does this cause spout to receive failed messages? Causes the tuple tree to which this tuple belongs to fail?



Q2： Why are these late tuples late for a few minutes, but the complete latency shown in the UI is still a few hundred milliseconds?



Q3： There is no late tupe when the input rate is 10 tuples/s, but it will appear in large numbers when it is increased to 100 tuples/s, so it will not work properly. What should I do to solve this problem?



2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=free, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=full, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=almost, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-9-count_bolt-executor[53 53] [INFO] Received a late tuple {word=whatsoever., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=for, timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=others., timestamp=1543194524005} with ts 1543194524005. This will not be processed.

2018-11-26 09:23:55.056 o.a.s.t.WindowedBoltExecutor Thread-11-count_bolt-executor[40 40] [INFO] Received a late tuple {word=project, timestamp=1543194524005} with ts 1543194524005. This will not be processed.


Update:



My topology：KafkaSpout--->SpilitBolt--->EventTimeWindowCountBolt--->KafkaSink



Storm Version：1.2.1



CountBolt：



 builder.setBolt("count_bolt", new CountBlot()

                .withTumblingWindow(new Duration(time_windowsize, TimeUnit.SECONDS))   //3s

                .withLag (new Duration(time_windowlag, TimeUnit.SECONDS))  // 1s

                .withTimestampField("timestamp")

                , parallelism)

                .fieldsGrouping("split_bolt", new Fields("word"));





apache-storm




apache-storm






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 26 '18 at 12:13







Cheng Jiang

















asked Nov 26 '18 at 1:55









Cheng JiangCheng Jiang

486




486













  • By looking at the source code, it is found that the late tuple will be acked. if (lateTupleStream != null) {                      windowedOutputCollector.emit(lateTupleStream, input, new Values(input));                  } else {                      LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts);                  }                  windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.

    – Cheng Jiang
    Nov 26 '18 at 7:19













  • You should provide some more details, like the topology you are ran, the storm version...

    – Jacky
    Nov 26 '18 at 10:21



















  • By looking at the source code, it is found that the late tuple will be acked. if (lateTupleStream != null) {                      windowedOutputCollector.emit(lateTupleStream, input, new Values(input));                  } else {                      LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts);                  }                  windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.

    – Cheng Jiang
    Nov 26 '18 at 7:19













  • You should provide some more details, like the topology you are ran, the storm version...

    – Jacky
    Nov 26 '18 at 10:21

















By looking at the source code, it is found that the late tuple will be acked. if (lateTupleStream != null) {                      windowedOutputCollector.emit(lateTupleStream, input, new Values(input));                  } else {                      LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts);                  }                  windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.

– Cheng Jiang
Nov 26 '18 at 7:19







By looking at the source code, it is found that the late tuple will be acked. if (lateTupleStream != null) {                      windowedOutputCollector.emit(lateTupleStream, input, new Values(input));                  } else {                      LOG.info("Received a late tuple {} with ts {}. This will not be processed.", input, ts);                  }                  windowedOutputCollector.ack(input); Why is there a lot of failed messages in my UI spout? There are only a lot of late tuple messages in the spout log.

– Cheng Jiang
Nov 26 '18 at 7:19















You should provide some more details, like the topology you are ran, the storm version...

– Jacky
Nov 26 '18 at 10:21





You should provide some more details, like the topology you are ran, the storm version...

– Jacky
Nov 26 '18 at 10:21












0






active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473846%2fapache-storm-will-the-late-tuple-be-ack-or-fail%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473846%2fapache-storm-will-the-late-tuple-be-ack-or-fail%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Costa Masnaga

Image
.mw-parser-output .avviso .mbox-text-div>div,.mw-parser-output .avviso .mbox-text-full-div>div{font-size:90%}.mw-parser-output .avviso .mbox-image div{width:52px}.mw-parser-output .avviso .mbox-text-full-div .hide-when-compact{display:block} Questa voce o sezione sull'argomento Lombardia non cita le fonti necessarie o quelle presenti sono insufficienti . Puoi migliorare questa voce aggiungendo citazioni da fonti attendibili secondo le linee guida sull'uso delle fonti. Segui i suggerimenti del progetto di riferimento. Costa Masnaga comune Localizzazione Stato   Italia Regione Lombardia Provincia Lecco Amministrazione Sindaco Sabina Panzeri (Al centro il cittadino) dall'08/06/2009 Territorio Coordinate 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Coordinate: 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Altitudine 318 m s.l.m...
Read more

Fotorealismo

Image
Guerrino Boatto, Venice's St. Barnaba Church and canal Il fotorealismo è un genere di pittura basato sull'uso di una o più fotografie dalle quali l'artista prende le informazioni necessarie da utilizzare poi nel processo della creazione dell'opera, in genere ad olio o ad acrilico, al fine di avere un aspetto finale il più simile possibile alla fotografia. Indice 1 Storia 2 Lista dei fotorealisti 3 Note 4 Voci correlate 5 Altri progetti Storia | Il fotorealismo è un movimento che nasce negli Stati Uniti d'America alla fine degli anni sessanta [1] sulla scia della Pop Art [2] [3] [4] , e insieme condividono il carattere reazionario verso la travolgente ascesa dei media, che alla metà del ventesimo secolo si trasforma in un fenomeno di massa talmente imponente da minacciare il valore dell'immagine nell'arte. [5] [6] Ne consegue così che, colto il gesto d'inizio che fu della Pop Art, i fotorealisti aprono ad un ...
Read more

Sidney Franklin

Image
Questa voce sull'argomento registi statunitensi è solo un abbozzo . Contribuisci a migliorarla secondo le convenzioni di Wikipedia. Sidney Franklin, foto di Carl Van Vechten Oscar alla memoria Irving G. Thalberg 1943 Sidney Franklin , all'anagrafe Sidney Arnold Franklin , (San Francisco, 21 marzo 1893 – Santa Monica, 18 maggio 1972), è stato un regista, produttore cinematografico e attore statunitense. Il 20 giugno 1963, si sposò con l'attrice australiana Enid Bennett. Il matrimonio durò fino alla morte dell'attrice, il 14 maggio 1969 [1] Indice 1 Filmografia 1.1 Regista 1.2 Aiuto regia (parziale) 1.3 Produttore 1.4 Attore (parziale) 2 Note 3 Altri progetti 4 Collegamenti esterni Filmografia | Regista | The Baby , co-regia Chester M. Franklin (1915) The Rivals , co-regia Chester M. Franklin (1915) Little Dick's First Case , co-regia Chester M. Franklin (1915) Her Filmland Hero , co-...
Read more