SpringBoot (Kotlin) - running a data migration in one time as a batch job
So, I have a specific problem and I can't find the Spring best practice for it.
I have a Spring instance in front of a Postgres DB. I have a domain entity that I'm adding some fields to (using flyway).
Essentially, imagine I have the following object:
Book
last_text_update // 2018-11-25 07:00:00
last_writer // PUBLISHER
is_finished // true
contract_closed_timestamp // NULL
From this, we do weird calculations like
"If last_writer == PUBLISHER && is_finished == true, return IS_SELLING_IN_STORES"
As you can imagine, this is ugly, and I'm refactoring it to the following:
Book
processedStatus: PUBLISHED/EDITING/PROPOSAL
workStatus: AWAITING_EDITOR_FEEDBACK/AWAITING_CHANGES/FINISHED
etc.etc.
So, I've prepared the SQL migrations that have added the necessary status columns. The way I see it, my options are to either:
1) Figure out all of the column transitions in raw SQL and run that migration on the production server like any other migration. i.e. where timestamp = blah and other timestamp = blah, set status as follows
2) Run a one-off batch job that cycles through each object and updates it.
It seems like it would be easier to do in Kotlin code via batch job vs. doing it in SQL, but what is the consensus on this? Is there a recommended to do a one-off batch job (i.e. ssh into an instance and manually run it, start it with HTTP over the API, etc.), or do I just suck it up and use SQL?
spring postgresql spring-boot kotlin database-migration
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
add a comment |
So, I have a specific problem and I can't find the Spring best practice for it.
I have a Spring instance in front of a Postgres DB. I have a domain entity that I'm adding some fields to (using flyway).
Essentially, imagine I have the following object:
Book
last_text_update // 2018-11-25 07:00:00
last_writer // PUBLISHER
is_finished // true
contract_closed_timestamp // NULL
From this, we do weird calculations like
"If last_writer == PUBLISHER && is_finished == true, return IS_SELLING_IN_STORES"
As you can imagine, this is ugly, and I'm refactoring it to the following:
Book
processedStatus: PUBLISHED/EDITING/PROPOSAL
workStatus: AWAITING_EDITOR_FEEDBACK/AWAITING_CHANGES/FINISHED
etc.etc.
So, I've prepared the SQL migrations that have added the necessary status columns. The way I see it, my options are to either:
1) Figure out all of the column transitions in raw SQL and run that migration on the production server like any other migration. i.e. where timestamp = blah and other timestamp = blah, set status as follows
2) Run a one-off batch job that cycles through each object and updates it.
It seems like it would be easier to do in Kotlin code via batch job vs. doing it in SQL, but what is the consensus on this? Is there a recommended to do a one-off batch job (i.e. ssh into an instance and manually run it, start it with HTTP over the API, etc.), or do I just suck it up and use SQL?
spring postgresql spring-boot kotlin database-migration
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
1 seems more reasonable, given 1. you are already using Flyway and 2. It seems ugly to me to pollute the entity model to allow it to do the migration work. e.g. you will need the entity model to behave given an "incomplete" persistence form, and you need to keep the obsolete fields, and etc. Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification in entity model (e.g. sending out events etc), I would rather migrate using SQL
– Adrian Shum
Nov 26 '18 at 6:04
@AdrianShum Thank you, makes perfect sense. Can you convert that to an answer so I can mark it as accepted?
– JapanRob
Nov 27 '18 at 12:37
add a comment |
So, I have a specific problem and I can't find the Spring best practice for it.
I have a Spring instance in front of a Postgres DB. I have a domain entity that I'm adding some fields to (using flyway).
Essentially, imagine I have the following object:
Book
last_text_update // 2018-11-25 07:00:00
last_writer // PUBLISHER
is_finished // true
contract_closed_timestamp // NULL
From this, we do weird calculations like
"If last_writer == PUBLISHER && is_finished == true, return IS_SELLING_IN_STORES"
As you can imagine, this is ugly, and I'm refactoring it to the following:
Book
processedStatus: PUBLISHED/EDITING/PROPOSAL
workStatus: AWAITING_EDITOR_FEEDBACK/AWAITING_CHANGES/FINISHED
etc.etc.
So, I've prepared the SQL migrations that have added the necessary status columns. The way I see it, my options are to either:
1) Figure out all of the column transitions in raw SQL and run that migration on the production server like any other migration. i.e. where timestamp = blah and other timestamp = blah, set status as follows
2) Run a one-off batch job that cycles through each object and updates it.
It seems like it would be easier to do in Kotlin code via batch job vs. doing it in SQL, but what is the consensus on this? Is there a recommended to do a one-off batch job (i.e. ssh into an instance and manually run it, start it with HTTP over the API, etc.), or do I just suck it up and use SQL?
spring postgresql spring-boot kotlin database-migration
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
So, I have a specific problem and I can't find the Spring best practice for it.
I have a Spring instance in front of a Postgres DB. I have a domain entity that I'm adding some fields to (using flyway).
Essentially, imagine I have the following object:
Book
last_text_update // 2018-11-25 07:00:00
last_writer // PUBLISHER
is_finished // true
contract_closed_timestamp // NULL
From this, we do weird calculations like
"If last_writer == PUBLISHER && is_finished == true, return IS_SELLING_IN_STORES"
As you can imagine, this is ugly, and I'm refactoring it to the following:
Book
processedStatus: PUBLISHED/EDITING/PROPOSAL
workStatus: AWAITING_EDITOR_FEEDBACK/AWAITING_CHANGES/FINISHED
etc.etc.
So, I've prepared the SQL migrations that have added the necessary status columns. The way I see it, my options are to either:
1) Figure out all of the column transitions in raw SQL and run that migration on the production server like any other migration. i.e. where timestamp = blah and other timestamp = blah, set status as follows
2) Run a one-off batch job that cycles through each object and updates it.
It seems like it would be easier to do in Kotlin code via batch job vs. doing it in SQL, but what is the consensus on this? Is there a recommended to do a one-off batch job (i.e. ssh into an instance and manually run it, start it with HTTP over the API, etc.), or do I just suck it up and use SQL?
spring postgresql spring-boot kotlin database-migration
spring postgresql spring-boot kotlin database-migration
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
asked Nov 26 '18 at 5:11
JapanRobJapanRob
107210
107210
1 seems more reasonable, given 1. you are already using Flyway and 2. It seems ugly to me to pollute the entity model to allow it to do the migration work. e.g. you will need the entity model to behave given an "incomplete" persistence form, and you need to keep the obsolete fields, and etc. Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification in entity model (e.g. sending out events etc), I would rather migrate using SQL
– Adrian Shum
Nov 26 '18 at 6:04
@AdrianShum Thank you, makes perfect sense. Can you convert that to an answer so I can mark it as accepted?
– JapanRob
Nov 27 '18 at 12:37
add a comment |
1 seems more reasonable, given 1. you are already using Flyway and 2. It seems ugly to me to pollute the entity model to allow it to do the migration work. e.g. you will need the entity model to behave given an "incomplete" persistence form, and you need to keep the obsolete fields, and etc. Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification in entity model (e.g. sending out events etc), I would rather migrate using SQL
– Adrian Shum
Nov 26 '18 at 6:04
@AdrianShum Thank you, makes perfect sense. Can you convert that to an answer so I can mark it as accepted?
– JapanRob
Nov 27 '18 at 12:37
1 seems more reasonable, given 1. you are already using Flyway and 2. It seems ugly to me to pollute the entity model to allow it to do the migration work. e.g. you will need the entity model to behave given an "incomplete" persistence form, and you need to keep the obsolete fields, and etc. Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification in entity model (e.g. sending out events etc), I would rather migrate using SQL
– Adrian Shum
Nov 26 '18 at 6:04
1 seems more reasonable, given 1. you are already using Flyway and 2. It seems ugly to me to pollute the entity model to allow it to do the migration work. e.g. you will need the entity model to behave given an "incomplete" persistence form, and you need to keep the obsolete fields, and etc. Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification in entity model (e.g. sending out events etc), I would rather migrate using SQL
– Adrian Shum
Nov 26 '18 at 6:04
@AdrianShum Thank you, makes perfect sense. Can you convert that to an answer so I can mark it as accepted?
– JapanRob
Nov 27 '18 at 12:37
@AdrianShum Thank you, makes perfect sense. Can you convert that to an answer so I can mark it as accepted?
– JapanRob
Nov 27 '18 at 12:37
add a comment |
1 Answer
1
active
oldest
votes
Migration using SQL (Option 1) seems more reasonable, given that
- you are already using Flyway and
- It seems ugly to me to pollute the entity model to allow it to do the data migration work. e.g.
- you will need the entity model to behave given an "incomplete" persistence form, and
- you need to keep the obsolete fields, and etc.
Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification using entity model (e.g. sending out events etc), it seems more reasonable to do the data migration using SQL.
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53475068%2fspringboot-kotlin-running-a-data-migration-in-one-time-as-a-batch-job%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Migration using SQL (Option 1) seems more reasonable, given that
- you are already using Flyway and
- It seems ugly to me to pollute the entity model to allow it to do the data migration work. e.g.
- you will need the entity model to behave given an "incomplete" persistence form, and
- you need to keep the obsolete fields, and etc.
Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification using entity model (e.g. sending out events etc), it seems more reasonable to do the data migration using SQL.
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
add a comment |
Migration using SQL (Option 1) seems more reasonable, given that
- you are already using Flyway and
- It seems ugly to me to pollute the entity model to allow it to do the data migration work. e.g.
- you will need the entity model to behave given an "incomplete" persistence form, and
- you need to keep the obsolete fields, and etc.
Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification using entity model (e.g. sending out events etc), it seems more reasonable to do the data migration using SQL.
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
add a comment |
Migration using SQL (Option 1) seems more reasonable, given that
- you are already using Flyway and
- It seems ugly to me to pollute the entity model to allow it to do the data migration work. e.g.
- you will need the entity model to behave given an "incomplete" persistence form, and
- you need to keep the obsolete fields, and etc.
Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification using entity model (e.g. sending out events etc), it seems more reasonable to do the data migration using SQL.
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
Migration using SQL (Option 1) seems more reasonable, given that
- you are already using Flyway and
- It seems ugly to me to pollute the entity model to allow it to do the data migration work. e.g.
- you will need the entity model to behave given an "incomplete" persistence form, and
- you need to keep the obsolete fields, and etc.
Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification using entity model (e.g. sending out events etc), it seems more reasonable to do the data migration using SQL.
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
answered Nov 28 '18 at 6:23
Adrian ShumAdrian Shum
29k763107
29k763107
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53475068%2fspringboot-kotlin-running-a-data-migration-in-one-time-as-a-batch-job%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1 seems more reasonable, given 1. you are already using Flyway and 2. It seems ugly to me to pollute the entity model to allow it to do the migration work. e.g. you will need the entity model to behave given an "incomplete" persistence form, and you need to keep the obsolete fields, and etc. Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification in entity model (e.g. sending out events etc), I would rather migrate using SQL
– Adrian Shum
Nov 26 '18 at 6:04
@AdrianShum Thank you, makes perfect sense. Can you convert that to an answer so I can mark it as accepted?
– JapanRob
Nov 27 '18 at 12:37