Spark max amount of event time windows
up vote
0
down vote
favorite
Is there a limitation for Spark Event Time Streaming on the number of windows you can hold in parallel?
In general I would assume this depends on the available memory per executor and the amount of data you are using within the window, but there may be some metadata overhead in the master for holding references about the existence of all windows.
Does anyone know if there are some articles or official statements about that?
Lets assume I would like to create several million windows holding just small data like 120 rows. Would this cause a giant metadata overhead which might interfere with some limitations?
apache-spark bigdata spark-streaming iot
asked 2 days ago
AlexL
4071417
add a comment |
up vote
0
down vote
favorite
Is there a limitation for Spark Event Time Streaming on the number of windows you can hold in parallel?
In general I would assume this depends on the available memory per executor and the amount of data you are using within the window, but there may be some metadata overhead in the master for holding references about the existence of all windows.
Does anyone know if there are some articles or official statements about that?
Lets assume I would like to create several million windows holding just small data like 120 rows. Would this cause a giant metadata overhead which might interfere with some limitations?
apache-spark bigdata spark-streaming iot
asked 2 days ago
AlexL
4071417
It should be easy to check - just create a dummy job and see how it behaves. In general I strongly suspect it won't work - very large number of jobs is not something that Spark is good at. It also raises a question - what use case would justify such thing?
– user10465355
2 days ago
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Is there a limitation for Spark Event Time Streaming on the number of windows you can hold in parallel?
In general I would assume this depends on the available memory per executor and the amount of data you are using within the window, but there may be some metadata overhead in the master for holding references about the existence of all windows.
Does anyone know if there are some articles or official statements about that?
Lets assume I would like to create several million windows holding just small data like 120 rows. Would this cause a giant metadata overhead which might interfere with some limitations?
apache-spark bigdata spark-streaming iot
asked 2 days ago
AlexL
4071417
Is there a limitation for Spark Event Time Streaming on the number of windows you can hold in parallel?
In general I would assume this depends on the available memory per executor and the amount of data you are using within the window, but there may be some metadata overhead in the master for holding references about the existence of all windows.
Does anyone know if there are some articles or official statements about that?
Lets assume I would like to create several million windows holding just small data like 120 rows. Would this cause a giant metadata overhead which might interfere with some limitations?
apache-spark bigdata spark-streaming iot
apache-spark bigdata spark-streaming iot
asked 2 days ago
AlexL
4071417
asked 2 days ago
AlexL
4071417
asked 2 days ago
AlexL
4071417
asked 2 days ago
AlexL
4071417
asked 2 days ago
AlexL
4071417
4071417
It should be easy to check - just create a dummy job and see how it behaves. In general I strongly suspect it won't work - very large number of jobs is not something that Spark is good at. It also raises a question - what use case would justify such thing?
– user10465355
2 days ago
add a comment |
It should be easy to check - just create a dummy job and see how it behaves. In general I strongly suspect it won't work - very large number of jobs is not something that Spark is good at. It also raises a question - what use case would justify such thing?
– user10465355
2 days ago
It should be easy to check - just create a dummy job and see how it behaves. In general I strongly suspect it won't work - very large number of jobs is not something that Spark is good at. It also raises a question - what use case would justify such thing?
– user10465355
2 days ago
It should be easy to check - just create a dummy job and see how it behaves. In general I strongly suspect it won't work - very large number of jobs is not something that Spark is good at. It also raises a question - what use case would justify such thing?
– user10465355
2 days ago
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53350244%2fspark-max-amount-of-event-time-windows%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
It should be easy to check - just create a dummy job and see how it behaves. In general I strongly suspect it won't work - very large number of jobs is not something that Spark is good at. It also raises a question - what use case would justify such thing?
– user10465355
2 days ago