Remove for loops to make it faster - numpy












-1















I have this python code using numpy:



import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])

for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]

out[-1,-1]


I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??










share|improve this question

























  • What did you try?

    – MaxPowers
    Nov 22 '18 at 8:56











  • I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible

    – mateobruno
    Nov 22 '18 at 8:59











  • Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...

    – SpghttCd
    Nov 22 '18 at 9:01


















-1















I have this python code using numpy:



import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])

for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]

out[-1,-1]


I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??










share|improve this question

























  • What did you try?

    – MaxPowers
    Nov 22 '18 at 8:56











  • I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible

    – mateobruno
    Nov 22 '18 at 8:59











  • Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...

    – SpghttCd
    Nov 22 '18 at 9:01
















-1












-1








-1


0






I have this python code using numpy:



import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])

for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]

out[-1,-1]


I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??










share|improve this question
















I have this python code using numpy:



import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])

for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]

out[-1,-1]


I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??







python performance numpy






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 22 '18 at 9:19







mateobruno

















asked Nov 22 '18 at 8:47









mateobrunomateobruno

32




32













  • What did you try?

    – MaxPowers
    Nov 22 '18 at 8:56











  • I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible

    – mateobruno
    Nov 22 '18 at 8:59











  • Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...

    – SpghttCd
    Nov 22 '18 at 9:01





















  • What did you try?

    – MaxPowers
    Nov 22 '18 at 8:56











  • I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible

    – mateobruno
    Nov 22 '18 at 8:59











  • Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...

    – SpghttCd
    Nov 22 '18 at 9:01



















What did you try?

– MaxPowers
Nov 22 '18 at 8:56





What did you try?

– MaxPowers
Nov 22 '18 at 8:56













I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible

– mateobruno
Nov 22 '18 at 8:59





I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible

– mateobruno
Nov 22 '18 at 8:59













Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...

– SpghttCd
Nov 22 '18 at 9:01







Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...

– SpghttCd
Nov 22 '18 at 9:01














2 Answers
2






active

oldest

votes


















0














This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).



This is already a huge optimization!



I don't think there is a way to deal with this without an iteration over all the coordinates.



But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.



EVEN BETTER:



You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.






share|improve this answer

































    0














    I'm not entirely sure what you're trying to accomplish.



    You're making a new out array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0]), you overwrite the element at [0, 0]).



    You then proceed to filling the rest of the out array by taking the the maximum of either the element in the row before or that of the column before.



    You're then adding the contents of the original array table at that spot to the maximum.



    Since you're looking for previous rows and columns in the out array, you'll often find yourself in a situation where you've overwritten those values before you check them.
    For instance, when you're at i=2, j=1, you've got the array



    [[ 23.  77.  89.]
    [ 40. 109. 0.]
    [ 83. *?* 0.]
    [114. 0. 0.]
    [135. 0. 0.]]


    You're at the *?* and checking for 109 and 83. And you've just inserted 109 in the previous iteration.



    Thus, you are unable to get the same result without the use of those for loops.






    share|improve this answer
























    • I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

      – mateobruno
      Nov 22 '18 at 9:28











    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426966%2fremove-for-loops-to-make-it-faster-numpy%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).



    This is already a huge optimization!



    I don't think there is a way to deal with this without an iteration over all the coordinates.



    But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.



    EVEN BETTER:



    You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.






    share|improve this answer






























      0














      This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).



      This is already a huge optimization!



      I don't think there is a way to deal with this without an iteration over all the coordinates.



      But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.



      EVEN BETTER:



      You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.






      share|improve this answer




























        0












        0








        0







        This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).



        This is already a huge optimization!



        I don't think there is a way to deal with this without an iteration over all the coordinates.



        But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.



        EVEN BETTER:



        You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.






        share|improve this answer















        This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).



        This is already a huge optimization!



        I don't think there is a way to deal with this without an iteration over all the coordinates.



        But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.



        EVEN BETTER:



        You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 22 '18 at 11:30

























        answered Nov 22 '18 at 10:43









        jlanikjlanik

        38419




        38419

























            0














            I'm not entirely sure what you're trying to accomplish.



            You're making a new out array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0]), you overwrite the element at [0, 0]).



            You then proceed to filling the rest of the out array by taking the the maximum of either the element in the row before or that of the column before.



            You're then adding the contents of the original array table at that spot to the maximum.



            Since you're looking for previous rows and columns in the out array, you'll often find yourself in a situation where you've overwritten those values before you check them.
            For instance, when you're at i=2, j=1, you've got the array



            [[ 23.  77.  89.]
            [ 40. 109. 0.]
            [ 83. *?* 0.]
            [114. 0. 0.]
            [135. 0. 0.]]


            You're at the *?* and checking for 109 and 83. And you've just inserted 109 in the previous iteration.



            Thus, you are unable to get the same result without the use of those for loops.






            share|improve this answer
























            • I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

              – mateobruno
              Nov 22 '18 at 9:28
















            0














            I'm not entirely sure what you're trying to accomplish.



            You're making a new out array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0]), you overwrite the element at [0, 0]).



            You then proceed to filling the rest of the out array by taking the the maximum of either the element in the row before or that of the column before.



            You're then adding the contents of the original array table at that spot to the maximum.



            Since you're looking for previous rows and columns in the out array, you'll often find yourself in a situation where you've overwritten those values before you check them.
            For instance, when you're at i=2, j=1, you've got the array



            [[ 23.  77.  89.]
            [ 40. 109. 0.]
            [ 83. *?* 0.]
            [114. 0. 0.]
            [135. 0. 0.]]


            You're at the *?* and checking for 109 and 83. And you've just inserted 109 in the previous iteration.



            Thus, you are unable to get the same result without the use of those for loops.






            share|improve this answer
























            • I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

              – mateobruno
              Nov 22 '18 at 9:28














            0












            0








            0







            I'm not entirely sure what you're trying to accomplish.



            You're making a new out array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0]), you overwrite the element at [0, 0]).



            You then proceed to filling the rest of the out array by taking the the maximum of either the element in the row before or that of the column before.



            You're then adding the contents of the original array table at that spot to the maximum.



            Since you're looking for previous rows and columns in the out array, you'll often find yourself in a situation where you've overwritten those values before you check them.
            For instance, when you're at i=2, j=1, you've got the array



            [[ 23.  77.  89.]
            [ 40. 109. 0.]
            [ 83. *?* 0.]
            [114. 0. 0.]
            [135. 0. 0.]]


            You're at the *?* and checking for 109 and 83. And you've just inserted 109 in the previous iteration.



            Thus, you are unable to get the same result without the use of those for loops.






            share|improve this answer













            I'm not entirely sure what you're trying to accomplish.



            You're making a new out array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0]), you overwrite the element at [0, 0]).



            You then proceed to filling the rest of the out array by taking the the maximum of either the element in the row before or that of the column before.



            You're then adding the contents of the original array table at that spot to the maximum.



            Since you're looking for previous rows and columns in the out array, you'll often find yourself in a situation where you've overwritten those values before you check them.
            For instance, when you're at i=2, j=1, you've got the array



            [[ 23.  77.  89.]
            [ 40. 109. 0.]
            [ 83. *?* 0.]
            [114. 0. 0.]
            [135. 0. 0.]]


            You're at the *?* and checking for 109 and 83. And you've just inserted 109 in the previous iteration.



            Thus, you are unable to get the same result without the use of those for loops.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Nov 22 '18 at 9:13









            Mart RMart R

            112




            112













            • I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

              – mateobruno
              Nov 22 '18 at 9:28



















            • I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

              – mateobruno
              Nov 22 '18 at 9:28

















            I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

            – mateobruno
            Nov 22 '18 at 9:28





            I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element

            – mateobruno
            Nov 22 '18 at 9:28


















            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426966%2fremove-for-loops-to-make-it-faster-numpy%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Costa Masnaga

            Fotorealismo

            Sidney Franklin