Fetching data from QueryDatabaseTable in NiFi
I have a Database in MySQL that has almost 2 million row count and 30 columns. My flow is QueryDatabaseTable -> convertrecord -> PutElasticSearchHttpRecord. My QueryDatabaseTable has following configs (please see the screenshots):
My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)
java.arg.2=-Xms512m
java.arg.3=-Xmx512m
I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?
What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!
mysql apache apache-nifi
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
asked Nov 21 '18 at 19:47
shradsshrads
1358
add a comment |
I have a Database in MySQL that has almost 2 million row count and 30 columns. My flow is QueryDatabaseTable -> convertrecord -> PutElasticSearchHttpRecord. My QueryDatabaseTable has following configs (please see the screenshots):
My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)
java.arg.2=-Xms512m
java.arg.3=-Xmx512m
I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?
What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!
mysql apache apache-nifi
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
asked Nov 21 '18 at 19:47
shradsshrads
1358
Thanks for updating the question, after updating jvm heap space to8GBdid you restarted theNiFi?, after restart then only nifi will run with8GB JVM memory.
– Shu
Nov 21 '18 at 21:22
Did the issue got resolved?.
– Shu
Nov 27 '18 at 15:06
1
@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 '18 at 20:04
add a comment |
I have a Database in MySQL that has almost 2 million row count and 30 columns. My flow is QueryDatabaseTable -> convertrecord -> PutElasticSearchHttpRecord. My QueryDatabaseTable has following configs (please see the screenshots):
My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)
java.arg.2=-Xms512m
java.arg.3=-Xmx512m
I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?
What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!
mysql apache apache-nifi
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
asked Nov 21 '18 at 19:47
shradsshrads
1358
I have a Database in MySQL that has almost 2 million row count and 30 columns. My flow is QueryDatabaseTable -> convertrecord -> PutElasticSearchHttpRecord. My QueryDatabaseTable has following configs (please see the screenshots):
My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)
java.arg.2=-Xms512m
java.arg.3=-Xmx512m
I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?
What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!
mysql apache apache-nifi
mysql apache apache-nifi
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
asked Nov 21 '18 at 19:47
shradsshrads
1358
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
asked Nov 21 '18 at 19:47
shradsshrads
1358
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
edited Nov 25 '18 at 21:41
marc_s
573k12811071255
573k12811071255
asked Nov 21 '18 at 19:47
shradsshrads
1358
asked Nov 21 '18 at 19:47
shradsshrads
1358
asked Nov 21 '18 at 19:47
shradsshrads
1358
1358
Thanks for updating the question, after updating jvm heap space to8GBdid you restarted theNiFi?, after restart then only nifi will run with8GB JVM memory.
– Shu
Nov 21 '18 at 21:22
Did the issue got resolved?.
– Shu
Nov 27 '18 at 15:06
1
@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 '18 at 20:04
add a comment |
Thanks for updating the question, after updating jvm heap space to8GBdid you restarted theNiFi?, after restart then only nifi will run with8GB JVM memory.
– Shu
Nov 21 '18 at 21:22
Did the issue got resolved?.
– Shu
Nov 27 '18 at 15:06
1
@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 '18 at 20:04
Thanks for updating the question, after updating jvm heap space to
8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.– Shu
Nov 21 '18 at 21:22
Thanks for updating the question, after updating jvm heap space to
8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.– Shu
Nov 21 '18 at 21:22
Did the issue got resolved?.
– Shu
Nov 27 '18 at 15:06
Did the issue got resolved?.
– Shu
Nov 27 '18 at 15:06
1
1
@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 '18 at 20:04
@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 '18 at 20:04
add a comment |
1 Answer
1
active
oldest
votes
As discussed in above comments.
NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.
To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file
As per your requirement change the configs as shown below i have increased the heap space to 8GB
# JVM memory settings
java.arg.2=-Xms8912m
java.arg.3=-Xmx8912m
And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.
For more details regards to JVM outofmemory issues refer to this link.
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53419517%2ffetching-data-from-querydatabasetable-in-nifi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
As discussed in above comments.
NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.
To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file
As per your requirement change the configs as shown below i have increased the heap space to 8GB
# JVM memory settings
java.arg.2=-Xms8912m
java.arg.3=-Xmx8912m
And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.
For more details regards to JVM outofmemory issues refer to this link.
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
add a comment |
As discussed in above comments.
NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.
To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file
As per your requirement change the configs as shown below i have increased the heap space to 8GB
# JVM memory settings
java.arg.2=-Xms8912m
java.arg.3=-Xmx8912m
And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.
For more details regards to JVM outofmemory issues refer to this link.
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
add a comment |
As discussed in above comments.
NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.
To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file
As per your requirement change the configs as shown below i have increased the heap space to 8GB
# JVM memory settings
java.arg.2=-Xms8912m
java.arg.3=-Xmx8912m
And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.
For more details regards to JVM outofmemory issues refer to this link.
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
As discussed in above comments.
NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.
To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file
As per your requirement change the configs as shown below i have increased the heap space to 8GB
# JVM memory settings
java.arg.2=-Xms8912m
java.arg.3=-Xmx8912m
And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.
For more details regards to JVM outofmemory issues refer to this link.
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
answered Nov 28 '18 at 0:09
ShuShu
4,6712418
4,6712418
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53419517%2ffetching-data-from-querydatabasetable-in-nifi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Thanks for updating the question, after updating jvm heap space to
8GBdid you restarted theNiFi?, after restart then only nifi will run with8GB JVM memory.– Shu
Nov 21 '18 at 21:22
Did the issue got resolved?.
– Shu
Nov 27 '18 at 15:06
1
@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 '18 at 20:04