Encoding Java object to deterministic identifier
What is the best way to encode a java object to deterministic identifier which I can decode to that object?
Basically I want to send this upstream to clients in our response so that they can report metrics on this identifier. So this identifier has to be deterministic. I've looked into JSON serialization -> GZIP compression -> base64 string. However, I'm worried that the fields in the json object will not always maintain the same order, resulting in a different set of bytes each time. Is there a way to preserve this struct order? If not is there a better way to do this?
Another related question is if there is a method which produces a string with the least number of characters since we have tight space requirements for the resulting identifier.
java json encoding base64 gzip
asked Nov 20 '18 at 22:37
nishkrish
43
add a comment |
What is the best way to encode a java object to deterministic identifier which I can decode to that object?
Basically I want to send this upstream to clients in our response so that they can report metrics on this identifier. So this identifier has to be deterministic. I've looked into JSON serialization -> GZIP compression -> base64 string. However, I'm worried that the fields in the json object will not always maintain the same order, resulting in a different set of bytes each time. Is there a way to preserve this struct order? If not is there a better way to do this?
Another related question is if there is a method which produces a string with the least number of characters since we have tight space requirements for the resulting identifier.
java json encoding base64 gzip
asked Nov 20 '18 at 22:37
nishkrish
43
Sure. Just let the database generate an ID for you, or use a UUID, and store that ID and the state of your object in the database. When you need the object back from the ID, get it from the database. See the 53402638 in the URL of this page? It's the ID of your question. The whole question isn't stored in this ID. It's just used to get the question in the database.
– JB Nizet
Nov 20 '18 at 22:41
Sorry I should have phrased that better. It's not going to be used for a database, thought that was a good example of why I need the determinism. It's for metrics reporting purposes for the BI team to use in our company. So persistence is not really optimal here on our end especially because the combination of things in this object can result in an infinitely growing table which we don't want to maintain.
– nishkrish
Nov 20 '18 at 22:54
1
How is that different from maintaining serialized full objects in logs? (other than the db being far easier to clean by throwing out all records older than some date).
– Mike 'Pomax' Kamermans
Nov 20 '18 at 23:05
You want serialization by any other name. What you do with the result afterwards: compression, encoding, etc. is irrelevant. Your key issue here is serialization and how it can match your requirements
– Perdi Estaquel
Nov 20 '18 at 23:25
add a comment |
What is the best way to encode a java object to deterministic identifier which I can decode to that object?
Basically I want to send this upstream to clients in our response so that they can report metrics on this identifier. So this identifier has to be deterministic. I've looked into JSON serialization -> GZIP compression -> base64 string. However, I'm worried that the fields in the json object will not always maintain the same order, resulting in a different set of bytes each time. Is there a way to preserve this struct order? If not is there a better way to do this?
Another related question is if there is a method which produces a string with the least number of characters since we have tight space requirements for the resulting identifier.
java json encoding base64 gzip
asked Nov 20 '18 at 22:37
nishkrish
43
What is the best way to encode a java object to deterministic identifier which I can decode to that object?
Basically I want to send this upstream to clients in our response so that they can report metrics on this identifier. So this identifier has to be deterministic. I've looked into JSON serialization -> GZIP compression -> base64 string. However, I'm worried that the fields in the json object will not always maintain the same order, resulting in a different set of bytes each time. Is there a way to preserve this struct order? If not is there a better way to do this?
Another related question is if there is a method which produces a string with the least number of characters since we have tight space requirements for the resulting identifier.
java json encoding base64 gzip
java json encoding base64 gzip
asked Nov 20 '18 at 22:37
nishkrish
43
asked Nov 20 '18 at 22:37
nishkrish
43
edited Nov 20 '18 at 22:52
asked Nov 20 '18 at 22:37
nishkrish
43
asked Nov 20 '18 at 22:37
nishkrish
43
asked Nov 20 '18 at 22:37
nishkrish
43
43
Sure. Just let the database generate an ID for you, or use a UUID, and store that ID and the state of your object in the database. When you need the object back from the ID, get it from the database. See the 53402638 in the URL of this page? It's the ID of your question. The whole question isn't stored in this ID. It's just used to get the question in the database.
– JB Nizet
Nov 20 '18 at 22:41
Sorry I should have phrased that better. It's not going to be used for a database, thought that was a good example of why I need the determinism. It's for metrics reporting purposes for the BI team to use in our company. So persistence is not really optimal here on our end especially because the combination of things in this object can result in an infinitely growing table which we don't want to maintain.
– nishkrish
Nov 20 '18 at 22:54
1
How is that different from maintaining serialized full objects in logs? (other than the db being far easier to clean by throwing out all records older than some date).
– Mike 'Pomax' Kamermans
Nov 20 '18 at 23:05
You want serialization by any other name. What you do with the result afterwards: compression, encoding, etc. is irrelevant. Your key issue here is serialization and how it can match your requirements
– Perdi Estaquel
Nov 20 '18 at 23:25
add a comment |
Sure. Just let the database generate an ID for you, or use a UUID, and store that ID and the state of your object in the database. When you need the object back from the ID, get it from the database. See the 53402638 in the URL of this page? It's the ID of your question. The whole question isn't stored in this ID. It's just used to get the question in the database.
– JB Nizet
Nov 20 '18 at 22:41
Sorry I should have phrased that better. It's not going to be used for a database, thought that was a good example of why I need the determinism. It's for metrics reporting purposes for the BI team to use in our company. So persistence is not really optimal here on our end especially because the combination of things in this object can result in an infinitely growing table which we don't want to maintain.
– nishkrish
Nov 20 '18 at 22:54
1
How is that different from maintaining serialized full objects in logs? (other than the db being far easier to clean by throwing out all records older than some date).
– Mike 'Pomax' Kamermans
Nov 20 '18 at 23:05
You want serialization by any other name. What you do with the result afterwards: compression, encoding, etc. is irrelevant. Your key issue here is serialization and how it can match your requirements
– Perdi Estaquel
Nov 20 '18 at 23:25
Sure. Just let the database generate an ID for you, or use a UUID, and store that ID and the state of your object in the database. When you need the object back from the ID, get it from the database. See the 53402638 in the URL of this page? It's the ID of your question. The whole question isn't stored in this ID. It's just used to get the question in the database.
– JB Nizet
Nov 20 '18 at 22:41
Sure. Just let the database generate an ID for you, or use a UUID, and store that ID and the state of your object in the database. When you need the object back from the ID, get it from the database. See the 53402638 in the URL of this page? It's the ID of your question. The whole question isn't stored in this ID. It's just used to get the question in the database.
– JB Nizet
Nov 20 '18 at 22:41
Sorry I should have phrased that better. It's not going to be used for a database, thought that was a good example of why I need the determinism. It's for metrics reporting purposes for the BI team to use in our company. So persistence is not really optimal here on our end especially because the combination of things in this object can result in an infinitely growing table which we don't want to maintain.
– nishkrish
Nov 20 '18 at 22:54
Sorry I should have phrased that better. It's not going to be used for a database, thought that was a good example of why I need the determinism. It's for metrics reporting purposes for the BI team to use in our company. So persistence is not really optimal here on our end especially because the combination of things in this object can result in an infinitely growing table which we don't want to maintain.
– nishkrish
Nov 20 '18 at 22:54
1
1
How is that different from maintaining serialized full objects in logs? (other than the db being far easier to clean by throwing out all records older than some date).
– Mike 'Pomax' Kamermans
Nov 20 '18 at 23:05
How is that different from maintaining serialized full objects in logs? (other than the db being far easier to clean by throwing out all records older than some date).
– Mike 'Pomax' Kamermans
Nov 20 '18 at 23:05
You want serialization by any other name. What you do with the result afterwards: compression, encoding, etc. is irrelevant. Your key issue here is serialization and how it can match your requirements
– Perdi Estaquel
Nov 20 '18 at 23:25
You want serialization by any other name. What you do with the result afterwards: compression, encoding, etc. is irrelevant. Your key issue here is serialization and how it can match your requirements
– Perdi Estaquel
Nov 20 '18 at 23:25
add a comment |
1 Answer
1
active
oldest
votes
I dont know if its a good idea to store the whole object as bytes in the database (im not sure its good practice) but if you really want to, i think the most straightforward path is to convert your object to a byte array and then convert the byte array to a base64 encoded string.
Theres already a post that deals with the issue of converting objects to byte arrays (im pretty sure the byte order will be the same every time):
Java Serializable Object to Byte Array
There is also a post that deals with the issue of converting byte arrays to base64 encoded string:
How do I convert a byte array to Base64 in Java?
Hope this helps..
Edit:
Didn't see the last part, but usually you would use hash functions like Sha256 to get a deterministic identifier of some data.
answered Nov 20 '18 at 22:55
retler
32
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53402638%2fencoding-java-object-to-deterministic-identifier%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I dont know if its a good idea to store the whole object as bytes in the database (im not sure its good practice) but if you really want to, i think the most straightforward path is to convert your object to a byte array and then convert the byte array to a base64 encoded string.
Theres already a post that deals with the issue of converting objects to byte arrays (im pretty sure the byte order will be the same every time):
Java Serializable Object to Byte Array
There is also a post that deals with the issue of converting byte arrays to base64 encoded string:
How do I convert a byte array to Base64 in Java?
Hope this helps..
Edit:
Didn't see the last part, but usually you would use hash functions like Sha256 to get a deterministic identifier of some data.
answered Nov 20 '18 at 22:55
retler
32
add a comment |
I dont know if its a good idea to store the whole object as bytes in the database (im not sure its good practice) but if you really want to, i think the most straightforward path is to convert your object to a byte array and then convert the byte array to a base64 encoded string.
Theres already a post that deals with the issue of converting objects to byte arrays (im pretty sure the byte order will be the same every time):
Java Serializable Object to Byte Array
There is also a post that deals with the issue of converting byte arrays to base64 encoded string:
How do I convert a byte array to Base64 in Java?
Hope this helps..
Edit:
Didn't see the last part, but usually you would use hash functions like Sha256 to get a deterministic identifier of some data.
answered Nov 20 '18 at 22:55
retler
32
add a comment |
I dont know if its a good idea to store the whole object as bytes in the database (im not sure its good practice) but if you really want to, i think the most straightforward path is to convert your object to a byte array and then convert the byte array to a base64 encoded string.
Theres already a post that deals with the issue of converting objects to byte arrays (im pretty sure the byte order will be the same every time):
Java Serializable Object to Byte Array
There is also a post that deals with the issue of converting byte arrays to base64 encoded string:
How do I convert a byte array to Base64 in Java?
Hope this helps..
Edit:
Didn't see the last part, but usually you would use hash functions like Sha256 to get a deterministic identifier of some data.
answered Nov 20 '18 at 22:55
retler
32
I dont know if its a good idea to store the whole object as bytes in the database (im not sure its good practice) but if you really want to, i think the most straightforward path is to convert your object to a byte array and then convert the byte array to a base64 encoded string.
Theres already a post that deals with the issue of converting objects to byte arrays (im pretty sure the byte order will be the same every time):
Java Serializable Object to Byte Array
There is also a post that deals with the issue of converting byte arrays to base64 encoded string:
How do I convert a byte array to Base64 in Java?
Hope this helps..
Edit:
Didn't see the last part, but usually you would use hash functions like Sha256 to get a deterministic identifier of some data.
answered Nov 20 '18 at 22:55
retler
32
answered Nov 20 '18 at 22:55
retler
32
answered Nov 20 '18 at 22:55
retler
32
answered Nov 20 '18 at 22:55
retler
32
32
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53402638%2fencoding-java-object-to-deterministic-identifier%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Sure. Just let the database generate an ID for you, or use a UUID, and store that ID and the state of your object in the database. When you need the object back from the ID, get it from the database. See the 53402638 in the URL of this page? It's the ID of your question. The whole question isn't stored in this ID. It's just used to get the question in the database.
– JB Nizet
Nov 20 '18 at 22:41
Sorry I should have phrased that better. It's not going to be used for a database, thought that was a good example of why I need the determinism. It's for metrics reporting purposes for the BI team to use in our company. So persistence is not really optimal here on our end especially because the combination of things in this object can result in an infinitely growing table which we don't want to maintain.
– nishkrish
Nov 20 '18 at 22:54
1
How is that different from maintaining serialized full objects in logs? (other than the db being far easier to clean by throwing out all records older than some date).
– Mike 'Pomax' Kamermans
Nov 20 '18 at 23:05
You want serialization by any other name. What you do with the result afterwards: compression, encoding, etc. is irrelevant. Your key issue here is serialization and how it can match your requirements
– Perdi Estaquel
Nov 20 '18 at 23:25