Does collect return a list snapshot if run on a parallel stream?
up vote
2
down vote
favorite
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
asked Nov 19 at 10:52
reikje
1,11711325
add a comment |
up vote
2
down vote
favorite
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
asked Nov 19 at 10:52
reikje
1,11711325
Does the returned list definitely contain an element?.last()will throwNoSuchElementExceptionif it is empty
– Eamon Scullion
Nov 19 at 10:57
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
Nov 19 at 11:00
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying tocollect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.
– nullpointer
Nov 19 at 12:43
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, isgenerator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?
– Holger
Nov 19 at 13:22
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
Nov 19 at 15:43
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
asked Nov 19 at 10:52
reikje
1,11711325
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
kotlin java-8 java-stream
asked Nov 19 at 10:52
reikje
1,11711325
asked Nov 19 at 10:52
reikje
1,11711325
asked Nov 19 at 10:52
reikje
1,11711325
asked Nov 19 at 10:52
reikje
1,11711325
asked Nov 19 at 10:52
reikje
1,11711325
1,11711325
Does the returned list definitely contain an element?.last()will throwNoSuchElementExceptionif it is empty
– Eamon Scullion
Nov 19 at 10:57
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
Nov 19 at 11:00
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying tocollect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.
– nullpointer
Nov 19 at 12:43
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, isgenerator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?
– Holger
Nov 19 at 13:22
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
Nov 19 at 15:43
add a comment |
Does the returned list definitely contain an element?.last()will throwNoSuchElementExceptionif it is empty
– Eamon Scullion
Nov 19 at 10:57
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
Nov 19 at 11:00
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying tocollect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.
– nullpointer
Nov 19 at 12:43
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, isgenerator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?
– Holger
Nov 19 at 13:22
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
Nov 19 at 15:43
Does the returned list definitely contain an element?
.last() will throw NoSuchElementException if it is empty– Eamon Scullion
Nov 19 at 10:57
Does the returned list definitely contain an element?
.last() will throw NoSuchElementException if it is empty– Eamon Scullion
Nov 19 at 10:57
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
Nov 19 at 11:00
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
Nov 19 at 11:00
2
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying to
collect(Collectors.toList()) as Player list and break the part of fetching the last into two steps. Something like this in java : List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1); .. note that this shouldn't work either on Circle CI for your code.– nullpointer
Nov 19 at 12:43
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying to
collect(Collectors.toList()) as Player list and break the part of fetching the last into two steps. Something like this in java : List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1); .. note that this shouldn't work either on Circle CI for your code.– nullpointer
Nov 19 at 12:43
3
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, is
generator.generateNames(50). If this method returns one of the standard List implementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?– Holger
Nov 19 at 13:22
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, is
generator.generateNames(50). If this method returns one of the standard List implementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?– Holger
Nov 19 at 13:22
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
Nov 19 at 15:43
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
Nov 19 at 15:43
add a comment |
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
Nov 19 at 16:10
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
Nov 19 at 16:10
add a comment |
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
Nov 19 at 16:10
add a comment |
up vote
3
down vote
accepted
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
edited Nov 19 at 15:43
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
answered Nov 19 at 15:29
Jayson Minard
36.2k14104170
36.2k14104170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
Nov 19 at 16:10
add a comment |
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
Nov 19 at 16:10
1
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with a
NoSuchElementException. That got me to the wrong track. But basically you answered my question: when collect(Collectors.toList()) returns, the List is always fully populated - also when using parallel streams.– reikje
Nov 19 at 16:10
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with a
NoSuchElementException. That got me to the wrong track. But basically you answered my question: when collect(Collectors.toList()) returns, the List is always fully populated - also when using parallel streams.– reikje
Nov 19 at 16:10
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53373038%2fdoes-collect-return-a-list-snapshot-if-run-on-a-parallel-stream%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Does the returned list definitely contain an element?
.last()will throwNoSuchElementExceptionif it is empty– Eamon Scullion
Nov 19 at 10:57
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
Nov 19 at 11:00
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying to
collect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.– nullpointer
Nov 19 at 12:43
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, is
generator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?– Holger
Nov 19 at 13:22
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
Nov 19 at 15:43