Can't see updates using `less +F` to a file being written by Python
I have a Python code that does some computation and writes the data in C.csv file. The computation is huge and takes time.
So while my program is running, I want to check if data is being written or not using:
$ less +F C.csv
What I notice is that while my program is running, I do not see any output being written in C.csv but as soon as I give CTRL+C signal, all of a sudden a lot of entries appear in C.csv file.
Now, I know that disk I/O are generally buffered and probably the program will wait for the buffer to get full before it actually writes it to the file (this is my assumption). So, I googled how to check the size of Buffer which suggested me the following method:
import io
print (io.DEFAULT_BUFFER_SIZE)
This returns 8192 (bytes) in my machine. I thought writes will only happen when the data to be written does not fit into buffer, i.e., when the data size crosses 8192 bytes. But when I check the size of C.csv after CTRL+C, it shows 236540 bytes.
How did so much data fit into the buffer? Or is there something else happenning?
python python-3.x file buffer less-unix
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
add a comment |
I have a Python code that does some computation and writes the data in C.csv file. The computation is huge and takes time.
So while my program is running, I want to check if data is being written or not using:
$ less +F C.csv
What I notice is that while my program is running, I do not see any output being written in C.csv but as soon as I give CTRL+C signal, all of a sudden a lot of entries appear in C.csv file.
Now, I know that disk I/O are generally buffered and probably the program will wait for the buffer to get full before it actually writes it to the file (this is my assumption). So, I googled how to check the size of Buffer which suggested me the following method:
import io
print (io.DEFAULT_BUFFER_SIZE)
This returns 8192 (bytes) in my machine. I thought writes will only happen when the data to be written does not fit into buffer, i.e., when the data size crosses 8192 bytes. But when I check the size of C.csv after CTRL+C, it shows 236540 bytes.
How did so much data fit into the buffer? Or is there something else happenning?
python python-3.x file buffer less-unix
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
Programs generally have internal buffers before attempting a write to disk, the OS might have its own buffers before then bothering to write unless the source program force a flush operation somehow (or if it's killed as you're doing now)... If both files are only 16mb, you're better off loading them both into memory and joining them there using adict, then writing them out instead of an O(N^2) approach
– Jon Clements♦
Nov 24 '18 at 20:46
@JonClements Can you say a bit more about those internal buffers and when should I expect the data to be written on disks? What should I do if I want to monitor it on real time? Also theO(n^2)was just an example, the two files don't actually have any commonidlike column, instead I had to do some computation to figure out if they are related. I gave theO(n^2)thing to just paint a general scenario.
– Shubham
Nov 24 '18 at 20:54
Depends on your system and operating system... so nope - I can't give you more details... you can try flushing after each write to force it through, but apart from that, the OS is going to write to disk when it wants to.
– Jon Clements♦
Nov 24 '18 at 20:57
add a comment |
I have a Python code that does some computation and writes the data in C.csv file. The computation is huge and takes time.
So while my program is running, I want to check if data is being written or not using:
$ less +F C.csv
What I notice is that while my program is running, I do not see any output being written in C.csv but as soon as I give CTRL+C signal, all of a sudden a lot of entries appear in C.csv file.
Now, I know that disk I/O are generally buffered and probably the program will wait for the buffer to get full before it actually writes it to the file (this is my assumption). So, I googled how to check the size of Buffer which suggested me the following method:
import io
print (io.DEFAULT_BUFFER_SIZE)
This returns 8192 (bytes) in my machine. I thought writes will only happen when the data to be written does not fit into buffer, i.e., when the data size crosses 8192 bytes. But when I check the size of C.csv after CTRL+C, it shows 236540 bytes.
How did so much data fit into the buffer? Or is there something else happenning?
python python-3.x file buffer less-unix
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
I have a Python code that does some computation and writes the data in C.csv file. The computation is huge and takes time.
So while my program is running, I want to check if data is being written or not using:
$ less +F C.csv
What I notice is that while my program is running, I do not see any output being written in C.csv but as soon as I give CTRL+C signal, all of a sudden a lot of entries appear in C.csv file.
Now, I know that disk I/O are generally buffered and probably the program will wait for the buffer to get full before it actually writes it to the file (this is my assumption). So, I googled how to check the size of Buffer which suggested me the following method:
import io
print (io.DEFAULT_BUFFER_SIZE)
This returns 8192 (bytes) in my machine. I thought writes will only happen when the data to be written does not fit into buffer, i.e., when the data size crosses 8192 bytes. But when I check the size of C.csv after CTRL+C, it shows 236540 bytes.
How did so much data fit into the buffer? Or is there something else happenning?
python python-3.x file buffer less-unix
python python-3.x file buffer less-unix
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
edited Nov 25 '18 at 7:47
Shubham
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
asked Nov 24 '18 at 20:34
ShubhamShubham
2,01821527
2,01821527
Programs generally have internal buffers before attempting a write to disk, the OS might have its own buffers before then bothering to write unless the source program force a flush operation somehow (or if it's killed as you're doing now)... If both files are only 16mb, you're better off loading them both into memory and joining them there using adict, then writing them out instead of an O(N^2) approach
– Jon Clements♦
Nov 24 '18 at 20:46
@JonClements Can you say a bit more about those internal buffers and when should I expect the data to be written on disks? What should I do if I want to monitor it on real time? Also theO(n^2)was just an example, the two files don't actually have any commonidlike column, instead I had to do some computation to figure out if they are related. I gave theO(n^2)thing to just paint a general scenario.
– Shubham
Nov 24 '18 at 20:54
Depends on your system and operating system... so nope - I can't give you more details... you can try flushing after each write to force it through, but apart from that, the OS is going to write to disk when it wants to.
– Jon Clements♦
Nov 24 '18 at 20:57
add a comment |
Programs generally have internal buffers before attempting a write to disk, the OS might have its own buffers before then bothering to write unless the source program force a flush operation somehow (or if it's killed as you're doing now)... If both files are only 16mb, you're better off loading them both into memory and joining them there using adict, then writing them out instead of an O(N^2) approach
– Jon Clements♦
Nov 24 '18 at 20:46
@JonClements Can you say a bit more about those internal buffers and when should I expect the data to be written on disks? What should I do if I want to monitor it on real time? Also theO(n^2)was just an example, the two files don't actually have any commonidlike column, instead I had to do some computation to figure out if they are related. I gave theO(n^2)thing to just paint a general scenario.
– Shubham
Nov 24 '18 at 20:54
Depends on your system and operating system... so nope - I can't give you more details... you can try flushing after each write to force it through, but apart from that, the OS is going to write to disk when it wants to.
– Jon Clements♦
Nov 24 '18 at 20:57
Programs generally have internal buffers before attempting a write to disk, the OS might have its own buffers before then bothering to write unless the source program force a flush operation somehow (or if it's killed as you're doing now)... If both files are only 16mb, you're better off loading them both into memory and joining them there using a
dict, then writing them out instead of an O(N^2) approach– Jon Clements♦
Nov 24 '18 at 20:46
Programs generally have internal buffers before attempting a write to disk, the OS might have its own buffers before then bothering to write unless the source program force a flush operation somehow (or if it's killed as you're doing now)... If both files are only 16mb, you're better off loading them both into memory and joining them there using a
dict, then writing them out instead of an O(N^2) approach– Jon Clements♦
Nov 24 '18 at 20:46
@JonClements Can you say a bit more about those internal buffers and when should I expect the data to be written on disks? What should I do if I want to monitor it on real time? Also the
O(n^2) was just an example, the two files don't actually have any common id like column, instead I had to do some computation to figure out if they are related. I gave the O(n^2) thing to just paint a general scenario.– Shubham
Nov 24 '18 at 20:54
@JonClements Can you say a bit more about those internal buffers and when should I expect the data to be written on disks? What should I do if I want to monitor it on real time? Also the
O(n^2) was just an example, the two files don't actually have any common id like column, instead I had to do some computation to figure out if they are related. I gave the O(n^2) thing to just paint a general scenario.– Shubham
Nov 24 '18 at 20:54
Depends on your system and operating system... so nope - I can't give you more details... you can try flushing after each write to force it through, but apart from that, the OS is going to write to disk when it wants to.
– Jon Clements♦
Nov 24 '18 at 20:57
Depends on your system and operating system... so nope - I can't give you more details... you can try flushing after each write to force it through, but apart from that, the OS is going to write to disk when it wants to.
– Jon Clements♦
Nov 24 '18 at 20:57
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53462140%2fcant-see-updates-using-less-f-to-a-file-being-written-by-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53462140%2fcant-see-updates-using-less-f-to-a-file-being-written-by-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Programs generally have internal buffers before attempting a write to disk, the OS might have its own buffers before then bothering to write unless the source program force a flush operation somehow (or if it's killed as you're doing now)... If both files are only 16mb, you're better off loading them both into memory and joining them there using a
dict, then writing them out instead of an O(N^2) approach– Jon Clements♦
Nov 24 '18 at 20:46
@JonClements Can you say a bit more about those internal buffers and when should I expect the data to be written on disks? What should I do if I want to monitor it on real time? Also the
O(n^2)was just an example, the two files don't actually have any commonidlike column, instead I had to do some computation to figure out if they are related. I gave theO(n^2)thing to just paint a general scenario.– Shubham
Nov 24 '18 at 20:54
Depends on your system and operating system... so nope - I can't give you more details... you can try flushing after each write to force it through, but apart from that, the OS is going to write to disk when it wants to.
– Jon Clements♦
Nov 24 '18 at 20:57