Fastest way to delete files with 0 size












1














I am in search of the fastest possible way to delete files with 0 size using bash.



I have hundreds of thousands of files with 0 size being generated (alongside valuable output) into a single output directory from a grep command run with GNU parallel. When grep finds a match the output file contains information. When grep does not find a match the output file is empty.



parallel -j $numcores "grepfunction {} > output_{}.fastq" ::: "${input_array[@]}"


I am using the following to remove the empty files but I would like to find a faster option. In searching most of the solutions I have found use similar find based commands that work at similar speeds.



find results2/ -size 0 -delete


As suggested in the comment i have also tried the following



find ./results2 -size 0 -print0 |xargs -0 rm --


It is qualitatively slow but I will benchmark and report back.



Any suggestions are much appreciated.






linux bash delete-file







share|improve this question
























  • Maybe using find to get the files and xargs to perform the rm in parallel. Try and benchmark.
    – Poshi
    Nov 20 '18 at 20:15






  • 5




    would be overall way more efficient if you modified your parallel thing to not generate files if there is no information.
    – Mat
    Nov 20 '18 at 20:19






  • 1




    I doubt there will be anything faster than find results2/ -size 0 -delete, as that is one process generating a number of system calls. find with xargs is at least three processes (find, xargs, and one or more calls to rm), and rm is going to do the same thing find ... -delete does: make some number of system calls to delete each file individually.
    – chepner
    Nov 20 '18 at 20:24










  • I think you'll find that find ... | parallel -X rm is quicker than find ... | xargs rm ... because parallel will, respecting ARGMAX, give as many files as possible to each invocation of rm.... though @chepner's answers are normally roughly 376,892 times better than mine :-)
    – Mark Setchell
    Nov 20 '18 at 20:25








  • 1




    I've never had to use parallel, so I'm not exactly sure how it would work, but if your grep_function returns a non-zero status when not finding anything, you could do something like "grep_function {} >output_{}.fastq || rm -f output_{}.fastq" to remove the empty files within your parallel invocation.
    – Lewis M
    Nov 20 '18 at 21:21
















1














I am in search of the fastest possible way to delete files with 0 size using bash.



I have hundreds of thousands of files with 0 size being generated (alongside valuable output) into a single output directory from a grep command run with GNU parallel. When grep finds a match the output file contains information. When grep does not find a match the output file is empty.



parallel -j $numcores "grepfunction {} > output_{}.fastq" ::: "${input_array[@]}"


I am using the following to remove the empty files but I would like to find a faster option. In searching most of the solutions I have found use similar find based commands that work at similar speeds.



find results2/ -size 0 -delete


As suggested in the comment i have also tried the following



find ./results2 -size 0 -print0 |xargs -0 rm --


It is qualitatively slow but I will benchmark and report back.



Any suggestions are much appreciated.






linux bash delete-file







share|improve this question
























  • Maybe using find to get the files and xargs to perform the rm in parallel. Try and benchmark.
    – Poshi
    Nov 20 '18 at 20:15






  • 5




    would be overall way more efficient if you modified your parallel thing to not generate files if there is no information.
    – Mat
    Nov 20 '18 at 20:19






  • 1




    I doubt there will be anything faster than find results2/ -size 0 -delete, as that is one process generating a number of system calls. find with xargs is at least three processes (find, xargs, and one or more calls to rm), and rm is going to do the same thing find ... -delete does: make some number of system calls to delete each file individually.
    – chepner
    Nov 20 '18 at 20:24










  • I think you'll find that find ... | parallel -X rm is quicker than find ... | xargs rm ... because parallel will, respecting ARGMAX, give as many files as possible to each invocation of rm.... though @chepner's answers are normally roughly 376,892 times better than mine :-)
    – Mark Setchell
    Nov 20 '18 at 20:25








  • 1




    I've never had to use parallel, so I'm not exactly sure how it would work, but if your grep_function returns a non-zero status when not finding anything, you could do something like "grep_function {} >output_{}.fastq || rm -f output_{}.fastq" to remove the empty files within your parallel invocation.
    – Lewis M
    Nov 20 '18 at 21:21














1












1








1







I am in search of the fastest possible way to delete files with 0 size using bash.



I have hundreds of thousands of files with 0 size being generated (alongside valuable output) into a single output directory from a grep command run with GNU parallel. When grep finds a match the output file contains information. When grep does not find a match the output file is empty.



parallel -j $numcores "grepfunction {} > output_{}.fastq" ::: "${input_array[@]}"


I am using the following to remove the empty files but I would like to find a faster option. In searching most of the solutions I have found use similar find based commands that work at similar speeds.



find results2/ -size 0 -delete


As suggested in the comment i have also tried the following



find ./results2 -size 0 -print0 |xargs -0 rm --


It is qualitatively slow but I will benchmark and report back.



Any suggestions are much appreciated.






linux bash delete-file







share|improve this question















I am in search of the fastest possible way to delete files with 0 size using bash.



I have hundreds of thousands of files with 0 size being generated (alongside valuable output) into a single output directory from a grep command run with GNU parallel. When grep finds a match the output file contains information. When grep does not find a match the output file is empty.



parallel -j $numcores "grepfunction {} > output_{}.fastq" ::: "${input_array[@]}"


I am using the following to remove the empty files but I would like to find a faster option. In searching most of the solutions I have found use similar find based commands that work at similar speeds.



find results2/ -size 0 -delete


As suggested in the comment i have also tried the following



find ./results2 -size 0 -print0 |xargs -0 rm --


It is qualitatively slow but I will benchmark and report back.



Any suggestions are much appreciated.






linux bash delete-file




linux bash delete-file






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 20 '18 at 20:16

























asked Nov 20 '18 at 20:12









Paul

256112




256112












  • Maybe using find to get the files and xargs to perform the rm in parallel. Try and benchmark.
    – Poshi
    Nov 20 '18 at 20:15






  • 5




    would be overall way more efficient if you modified your parallel thing to not generate files if there is no information.
    – Mat
    Nov 20 '18 at 20:19






  • 1




    I doubt there will be anything faster than find results2/ -size 0 -delete, as that is one process generating a number of system calls. find with xargs is at least three processes (find, xargs, and one or more calls to rm), and rm is going to do the same thing find ... -delete does: make some number of system calls to delete each file individually.
    – chepner
    Nov 20 '18 at 20:24










  • I think you'll find that find ... | parallel -X rm is quicker than find ... | xargs rm ... because parallel will, respecting ARGMAX, give as many files as possible to each invocation of rm.... though @chepner's answers are normally roughly 376,892 times better than mine :-)
    – Mark Setchell
    Nov 20 '18 at 20:25








  • 1




    I've never had to use parallel, so I'm not exactly sure how it would work, but if your grep_function returns a non-zero status when not finding anything, you could do something like "grep_function {} >output_{}.fastq || rm -f output_{}.fastq" to remove the empty files within your parallel invocation.
    – Lewis M
    Nov 20 '18 at 21:21


















  • Maybe using find to get the files and xargs to perform the rm in parallel. Try and benchmark.
    – Poshi
    Nov 20 '18 at 20:15






  • 5




    would be overall way more efficient if you modified your parallel thing to not generate files if there is no information.
    – Mat
    Nov 20 '18 at 20:19






  • 1




    I doubt there will be anything faster than find results2/ -size 0 -delete, as that is one process generating a number of system calls. find with xargs is at least three processes (find, xargs, and one or more calls to rm), and rm is going to do the same thing find ... -delete does: make some number of system calls to delete each file individually.
    – chepner
    Nov 20 '18 at 20:24










  • I think you'll find that find ... | parallel -X rm is quicker than find ... | xargs rm ... because parallel will, respecting ARGMAX, give as many files as possible to each invocation of rm.... though @chepner's answers are normally roughly 376,892 times better than mine :-)
    – Mark Setchell
    Nov 20 '18 at 20:25








  • 1




    I've never had to use parallel, so I'm not exactly sure how it would work, but if your grep_function returns a non-zero status when not finding anything, you could do something like "grep_function {} >output_{}.fastq || rm -f output_{}.fastq" to remove the empty files within your parallel invocation.
    – Lewis M
    Nov 20 '18 at 21:21
















Maybe using find to get the files and xargs to perform the rm in parallel. Try and benchmark.
– Poshi
Nov 20 '18 at 20:15




Maybe using find to get the files and xargs to perform the rm in parallel. Try and benchmark.
– Poshi
Nov 20 '18 at 20:15




5




5




would be overall way more efficient if you modified your parallel thing to not generate files if there is no information.
– Mat
Nov 20 '18 at 20:19




would be overall way more efficient if you modified your parallel thing to not generate files if there is no information.
– Mat
Nov 20 '18 at 20:19




1




1




I doubt there will be anything faster than find results2/ -size 0 -delete, as that is one process generating a number of system calls. find with xargs is at least three processes (find, xargs, and one or more calls to rm), and rm is going to do the same thing find ... -delete does: make some number of system calls to delete each file individually.
– chepner
Nov 20 '18 at 20:24




I doubt there will be anything faster than find results2/ -size 0 -delete, as that is one process generating a number of system calls. find with xargs is at least three processes (find, xargs, and one or more calls to rm), and rm is going to do the same thing find ... -delete does: make some number of system calls to delete each file individually.
– chepner
Nov 20 '18 at 20:24












I think you'll find that find ... | parallel -X rm is quicker than find ... | xargs rm ... because parallel will, respecting ARGMAX, give as many files as possible to each invocation of rm.... though @chepner's answers are normally roughly 376,892 times better than mine :-)
– Mark Setchell
Nov 20 '18 at 20:25






I think you'll find that find ... | parallel -X rm is quicker than find ... | xargs rm ... because parallel will, respecting ARGMAX, give as many files as possible to each invocation of rm.... though @chepner's answers are normally roughly 376,892 times better than mine :-)
– Mark Setchell
Nov 20 '18 at 20:25






1




1




I've never had to use parallel, so I'm not exactly sure how it would work, but if your grep_function returns a non-zero status when not finding anything, you could do something like "grep_function {} >output_{}.fastq || rm -f output_{}.fastq" to remove the empty files within your parallel invocation.
– Lewis M
Nov 20 '18 at 21:21




I've never had to use parallel, so I'm not exactly sure how it would work, but if your grep_function returns a non-zero status when not finding anything, you could do something like "grep_function {} >output_{}.fastq || rm -f output_{}.fastq" to remove the empty files within your parallel invocation.
– Lewis M
Nov 20 '18 at 21:21












1 Answer
1






active

oldest

votes


















0














(This should be a comment, but is too hard to read as a comment).



If you are going to run the jobs again you could make it part of the generating of the file:



parallel -j $numcores "grepfunction {} > output_{}.fastq ||

  rm output_{}.fastq" ::: "${input_array[@]}"


I assume the grep function returns true if match (like grep does).






share|improve this answer





















    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400816%2ffastest-way-to-delete-files-with-0-size%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    (This should be a comment, but is too hard to read as a comment).



    If you are going to run the jobs again you could make it part of the generating of the file:



    parallel -j $numcores "grepfunction {} > output_{}.fastq ||
    
  rm output_{}.fastq" ::: "${input_array[@]}"


    I assume the grep function returns true if match (like grep does).






    share|improve this answer


























      0














      (This should be a comment, but is too hard to read as a comment).



      If you are going to run the jobs again you could make it part of the generating of the file:



      parallel -j $numcores "grepfunction {} > output_{}.fastq ||
      
  rm output_{}.fastq" ::: "${input_array[@]}"


      I assume the grep function returns true if match (like grep does).






      share|improve this answer
























        0












        0








        0






        (This should be a comment, but is too hard to read as a comment).



        If you are going to run the jobs again you could make it part of the generating of the file:



        parallel -j $numcores "grepfunction {} > output_{}.fastq ||
        
  rm output_{}.fastq" ::: "${input_array[@]}"


        I assume the grep function returns true if match (like grep does).






        share|improve this answer












        (This should be a comment, but is too hard to read as a comment).



        If you are going to run the jobs again you could make it part of the generating of the file:



        parallel -j $numcores "grepfunction {} > output_{}.fastq ||
        
  rm output_{}.fastq" ::: "${input_array[@]}"


        I assume the grep function returns true if match (like grep does).







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 21 '18 at 8:15









        Ole Tange

        19k35566




        19k35566






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400816%2ffastest-way-to-delete-files-with-0-size%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Costa Masnaga

            Image
            .mw-parser-output .avviso .mbox-text-div>div,.mw-parser-output .avviso .mbox-text-full-div>div{font-size:90%}.mw-parser-output .avviso .mbox-image div{width:52px}.mw-parser-output .avviso .mbox-text-full-div .hide-when-compact{display:block} Questa voce o sezione sull'argomento Lombardia non cita le fonti necessarie o quelle presenti sono insufficienti . Puoi migliorare questa voce aggiungendo citazioni da fonti attendibili secondo le linee guida sull'uso delle fonti. Segui i suggerimenti del progetto di riferimento. Costa Masnaga comune Localizzazione Stato   Italia Regione Lombardia Provincia Lecco Amministrazione Sindaco Sabina Panzeri (Al centro il cittadino) dall'08/06/2009 Territorio Coordinate 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Coordinate: 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Altitudine 318 m s.l.m
            Read more

            Fotorealismo

            Image
            Guerrino Boatto, Venice's St. Barnaba Church and canal Il fotorealismo è un genere di pittura basato sull'uso di una o più fotografie dalle quali l'artista prende le informazioni necessarie da utilizzare poi nel processo della creazione dell'opera, in genere ad olio o ad acrilico, al fine di avere un aspetto finale il più simile possibile alla fotografia. Indice 1 Storia 2 Lista dei fotorealisti 3 Note 4 Voci correlate 5 Altri progetti Storia | Il fotorealismo è un movimento che nasce negli Stati Uniti d'America alla fine degli anni sessanta [1] sulla scia della Pop Art [2] [3] [4] , e insieme condividono il carattere reazionario verso la travolgente ascesa dei media, che alla metà del ventesimo secolo si trasforma in un fenomeno di massa talmente imponente da minacciare il valore dell'immagine nell'arte. [5] [6] Ne consegue così che, colto il gesto d'inizio che fu della Pop Art, i fotorealisti aprono ad un
            Read more

            Create new schema in PostgreSQL using DBeaver

            Image
            0 At the beginning of my current job I had limited access to DB but recently my role is changed to superuser, admin, and owner. My colleague who was the previous DB admin is able to create a schema or change the permissions as there is a permission tab in the properties and he can change permissions by clicking easily. Would you please let me know how I can create schema and change permissions as well? Why his dbeaver looks different from me? He can easily right click on the schema folder on top of all schemas on the left side column and select create new schema while I do not have that, like the following picture I found on the internet. postgresql database-schema dbeaver share | improve this question
            Read more