tfrecord type looks like txt or image
I'm generating tfrecords of BDD dataset from a 20000 subset. While creating tfrecords I choose 1000 images per tfrecord, and everything is fine.
However if I choose 500 images per tfrecord, some of them (very rare) have type TGA image (image/x-tga) or MATLAB script/function (text/x-matlab) while normally they should have had Program (application/octet-stream) or Binary (application/octet-stream).
Why would this happen, and does it mean the tfrecords are broken?
feature = self._get_tf_feature(
picture_id, os.path.join(full_images_path, f),
m.group(2), picture_id_annotations, new_format)
example = tf.train.Example(features=feature)
writer.write(example.SerializeToString())
python-3.x tensorflow tfrecord
edited Nov 21 '18 at 16:35
Vlad-HC
826815
asked Nov 21 '18 at 16:09
kneazlekneazle
317
add a comment |
I'm generating tfrecords of BDD dataset from a 20000 subset. While creating tfrecords I choose 1000 images per tfrecord, and everything is fine.
However if I choose 500 images per tfrecord, some of them (very rare) have type TGA image (image/x-tga) or MATLAB script/function (text/x-matlab) while normally they should have had Program (application/octet-stream) or Binary (application/octet-stream).
Why would this happen, and does it mean the tfrecords are broken?
feature = self._get_tf_feature(
picture_id, os.path.join(full_images_path, f),
m.group(2), picture_id_annotations, new_format)
example = tf.train.Example(features=feature)
writer.write(example.SerializeToString())
python-3.x tensorflow tfrecord
edited Nov 21 '18 at 16:35
Vlad-HC
826815
asked Nov 21 '18 at 16:09
kneazlekneazle
317
From the code snippet above it's not totally clear how the features are generated. But probably this problem has nothing to do with tfrecord format, because this format just stores the data. That means, that the data, which you provide while generating tfrecords are "wrong" - incorrect image type. Maybe it's worthy to check source images itself.
– Vlad-HC
Nov 21 '18 at 16:25
I was thinking the same, but I have a flag for elements per tfrecord, and I only change the number without changing the the way I get the features. Since the records are fine with 1000 samples each, I don't think the problem is caused by the features..
– kneazle
Nov 21 '18 at 16:34
add a comment |
I'm generating tfrecords of BDD dataset from a 20000 subset. While creating tfrecords I choose 1000 images per tfrecord, and everything is fine.
However if I choose 500 images per tfrecord, some of them (very rare) have type TGA image (image/x-tga) or MATLAB script/function (text/x-matlab) while normally they should have had Program (application/octet-stream) or Binary (application/octet-stream).
Why would this happen, and does it mean the tfrecords are broken?
feature = self._get_tf_feature(
picture_id, os.path.join(full_images_path, f),
m.group(2), picture_id_annotations, new_format)
example = tf.train.Example(features=feature)
writer.write(example.SerializeToString())
python-3.x tensorflow tfrecord
edited Nov 21 '18 at 16:35
Vlad-HC
826815
asked Nov 21 '18 at 16:09
kneazlekneazle
317
I'm generating tfrecords of BDD dataset from a 20000 subset. While creating tfrecords I choose 1000 images per tfrecord, and everything is fine.
However if I choose 500 images per tfrecord, some of them (very rare) have type TGA image (image/x-tga) or MATLAB script/function (text/x-matlab) while normally they should have had Program (application/octet-stream) or Binary (application/octet-stream).
Why would this happen, and does it mean the tfrecords are broken?
feature = self._get_tf_feature(
picture_id, os.path.join(full_images_path, f),
m.group(2), picture_id_annotations, new_format)
example = tf.train.Example(features=feature)
writer.write(example.SerializeToString())
python-3.x tensorflow tfrecord
python-3.x tensorflow tfrecord
edited Nov 21 '18 at 16:35
Vlad-HC
826815
asked Nov 21 '18 at 16:09
kneazlekneazle
317
edited Nov 21 '18 at 16:35
Vlad-HC
826815
asked Nov 21 '18 at 16:09
kneazlekneazle
317
edited Nov 21 '18 at 16:35
Vlad-HC
826815
edited Nov 21 '18 at 16:35
Vlad-HC
826815
edited Nov 21 '18 at 16:35
Vlad-HC
826815
826815
asked Nov 21 '18 at 16:09
kneazlekneazle
317
asked Nov 21 '18 at 16:09
kneazlekneazle
317
asked Nov 21 '18 at 16:09
kneazlekneazle
317
317
From the code snippet above it's not totally clear how the features are generated. But probably this problem has nothing to do with tfrecord format, because this format just stores the data. That means, that the data, which you provide while generating tfrecords are "wrong" - incorrect image type. Maybe it's worthy to check source images itself.
– Vlad-HC
Nov 21 '18 at 16:25
I was thinking the same, but I have a flag for elements per tfrecord, and I only change the number without changing the the way I get the features. Since the records are fine with 1000 samples each, I don't think the problem is caused by the features..
– kneazle
Nov 21 '18 at 16:34
add a comment |
From the code snippet above it's not totally clear how the features are generated. But probably this problem has nothing to do with tfrecord format, because this format just stores the data. That means, that the data, which you provide while generating tfrecords are "wrong" - incorrect image type. Maybe it's worthy to check source images itself.
– Vlad-HC
Nov 21 '18 at 16:25
I was thinking the same, but I have a flag for elements per tfrecord, and I only change the number without changing the the way I get the features. Since the records are fine with 1000 samples each, I don't think the problem is caused by the features..
– kneazle
Nov 21 '18 at 16:34
From the code snippet above it's not totally clear how the features are generated. But probably this problem has nothing to do with tfrecord format, because this format just stores the data. That means, that the data, which you provide while generating tfrecords are "wrong" - incorrect image type. Maybe it's worthy to check source images itself.
– Vlad-HC
Nov 21 '18 at 16:25
From the code snippet above it's not totally clear how the features are generated. But probably this problem has nothing to do with tfrecord format, because this format just stores the data. That means, that the data, which you provide while generating tfrecords are "wrong" - incorrect image type. Maybe it's worthy to check source images itself.
– Vlad-HC
Nov 21 '18 at 16:25
I was thinking the same, but I have a flag for elements per tfrecord, and I only change the number without changing the the way I get the features. Since the records are fine with 1000 samples each, I don't think the problem is caused by the features..
– kneazle
Nov 21 '18 at 16:34
I was thinking the same, but I have a flag for elements per tfrecord, and I only change the number without changing the the way I get the features. Since the records are fine with 1000 samples each, I don't think the problem is caused by the features..
– kneazle
Nov 21 '18 at 16:34
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53416135%2ftfrecord-type-looks-like-txt-or-image%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53416135%2ftfrecord-type-looks-like-txt-or-image%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
From the code snippet above it's not totally clear how the features are generated. But probably this problem has nothing to do with tfrecord format, because this format just stores the data. That means, that the data, which you provide while generating tfrecords are "wrong" - incorrect image type. Maybe it's worthy to check source images itself.
– Vlad-HC
Nov 21 '18 at 16:25
I was thinking the same, but I have a flag for elements per tfrecord, and I only change the number without changing the the way I get the features. Since the records are fine with 1000 samples each, I don't think the problem is caused by the features..
– kneazle
Nov 21 '18 at 16:34