Remove for loops to make it faster - numpy
I have this python code using numpy:
import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])
for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]
out[-1,-1]
I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??
python performance numpy
add a comment |
I have this python code using numpy:
import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])
for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]
out[-1,-1]
I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??
python performance numpy
What did you try?
– MaxPowers
Nov 22 '18 at 8:56
I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible
– mateobruno
Nov 22 '18 at 8:59
Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...
– SpghttCd
Nov 22 '18 at 9:01
add a comment |
I have this python code using numpy:
import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])
for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]
out[-1,-1]
I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??
python performance numpy
I have this python code using numpy:
import numpy as np
table = np.array([[23, 54, 12],
[17, 32, 25],
[43, 19, 11],
[31, 22, 10],
[21, 19, 35]])
r, c = table.shape
out = np.zeros((r,c))
out[0, :] = np.cumsum(table[0,:])
out[:, 0] = np.cumsum(table[:,0])
for j in range(1, c):
for i in range(1, r):
out[i,j] = max( out[i-1,j], out[i,j-1] ) + table[i,j]
out[-1,-1]
I first calculate the 1st row and the 1st column of the array 'out', while the rest of the values are calculated with the equation inside the for loops. I am interested only on the last value of the table ( out[-1,-1] ) and I want to make it as fast as possible. Could I remove the two 'for' loops somehow??
python performance numpy
python performance numpy
edited Nov 22 '18 at 9:19
mateobruno
asked Nov 22 '18 at 8:47
mateobrunomateobruno
32
32
What did you try?
– MaxPowers
Nov 22 '18 at 8:56
I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible
– mateobruno
Nov 22 '18 at 8:59
Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...
– SpghttCd
Nov 22 '18 at 9:01
add a comment |
What did you try?
– MaxPowers
Nov 22 '18 at 8:56
I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible
– mateobruno
Nov 22 '18 at 8:59
Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...
– SpghttCd
Nov 22 '18 at 9:01
What did you try?
– MaxPowers
Nov 22 '18 at 8:56
What did you try?
– MaxPowers
Nov 22 '18 at 8:56
I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible
– mateobruno
Nov 22 '18 at 8:59
I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible
– mateobruno
Nov 22 '18 at 8:59
Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...
– SpghttCd
Nov 22 '18 at 9:01
Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...
– SpghttCd
Nov 22 '18 at 9:01
add a comment |
2 Answers
2
active
oldest
votes
This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).
This is already a huge optimization!
I don't think there is a way to deal with this without an iteration over all the coordinates.
But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.
EVEN BETTER:
You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.
add a comment |
I'm not entirely sure what you're trying to accomplish.
You're making a new out
array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0])
, you overwrite the element at [0, 0]
).
You then proceed to filling the rest of the out
array by taking the the maximum of either the element in the row before or that of the column before.
You're then adding the contents of the original array table
at that spot to the maximum.
Since you're looking for previous rows and columns in the out
array, you'll often find yourself in a situation where you've overwritten those values before you check them.
For instance, when you're at i=2, j=1
, you've got the array
[[ 23. 77. 89.]
[ 40. 109. 0.]
[ 83. *?* 0.]
[114. 0. 0.]
[135. 0. 0.]]
You're at the *?*
and checking for 109 and 83. And you've just inserted 109 in the previous iteration.
Thus, you are unable to get the same result without the use of those for loops.
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426966%2fremove-for-loops-to-make-it-faster-numpy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).
This is already a huge optimization!
I don't think there is a way to deal with this without an iteration over all the coordinates.
But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.
EVEN BETTER:
You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.
add a comment |
This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).
This is already a huge optimization!
I don't think there is a way to deal with this without an iteration over all the coordinates.
But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.
EVEN BETTER:
You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.
add a comment |
This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).
This is already a huge optimization!
I don't think there is a way to deal with this without an iteration over all the coordinates.
But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.
EVEN BETTER:
You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.
This is in effect finding the longest path from each coordinate to 0,0, where the cost is given by the table for each node on the path. The standard 'matrix way' to deal with such problem is Floy-Warshall algoritm, which makes n matrix multiplications, where n is the number of nodes, in your case r*c. Your algorithm is actually already optimized variant of Floyd-Warshall that uses the fact that every path that goes from (i,j) to (0,0) must pass either (i-1,j) or (i,j-1).
This is already a huge optimization!
I don't think there is a way to deal with this without an iteration over all the coordinates.
But maybe you could use some graph library to encode it as a graph problem, which would push the iteration part into the (efficient) graph library.
EVEN BETTER:
You could iterate over diagonals, using fill_diagonal for instance. This way ou would iterate only over the number of diagonals.
edited Nov 22 '18 at 11:30
answered Nov 22 '18 at 10:43
jlanikjlanik
38419
38419
add a comment |
add a comment |
I'm not entirely sure what you're trying to accomplish.
You're making a new out
array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0])
, you overwrite the element at [0, 0]
).
You then proceed to filling the rest of the out
array by taking the the maximum of either the element in the row before or that of the column before.
You're then adding the contents of the original array table
at that spot to the maximum.
Since you're looking for previous rows and columns in the out
array, you'll often find yourself in a situation where you've overwritten those values before you check them.
For instance, when you're at i=2, j=1
, you've got the array
[[ 23. 77. 89.]
[ 40. 109. 0.]
[ 83. *?* 0.]
[114. 0. 0.]
[135. 0. 0.]]
You're at the *?*
and checking for 109 and 83. And you've just inserted 109 in the previous iteration.
Thus, you are unable to get the same result without the use of those for loops.
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
add a comment |
I'm not entirely sure what you're trying to accomplish.
You're making a new out
array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0])
, you overwrite the element at [0, 0]
).
You then proceed to filling the rest of the out
array by taking the the maximum of either the element in the row before or that of the column before.
You're then adding the contents of the original array table
at that spot to the maximum.
Since you're looking for previous rows and columns in the out
array, you'll often find yourself in a situation where you've overwritten those values before you check them.
For instance, when you're at i=2, j=1
, you've got the array
[[ 23. 77. 89.]
[ 40. 109. 0.]
[ 83. *?* 0.]
[114. 0. 0.]
[135. 0. 0.]]
You're at the *?*
and checking for 109 and 83. And you've just inserted 109 in the previous iteration.
Thus, you are unable to get the same result without the use of those for loops.
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
add a comment |
I'm not entirely sure what you're trying to accomplish.
You're making a new out
array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0])
, you overwrite the element at [0, 0]
).
You then proceed to filling the rest of the out
array by taking the the maximum of either the element in the row before or that of the column before.
You're then adding the contents of the original array table
at that spot to the maximum.
Since you're looking for previous rows and columns in the out
array, you'll often find yourself in a situation where you've overwritten those values before you check them.
For instance, when you're at i=2, j=1
, you've got the array
[[ 23. 77. 89.]
[ 40. 109. 0.]
[ 83. *?* 0.]
[114. 0. 0.]
[135. 0. 0.]]
You're at the *?*
and checking for 109 and 83. And you've just inserted 109 in the previous iteration.
Thus, you are unable to get the same result without the use of those for loops.
I'm not entirely sure what you're trying to accomplish.
You're making a new out
array that copies the first(0th) row and column from the original table. (By the way, at out[:, 0] = np.cumsum(table[:,0])
, you overwrite the element at [0, 0]
).
You then proceed to filling the rest of the out
array by taking the the maximum of either the element in the row before or that of the column before.
You're then adding the contents of the original array table
at that spot to the maximum.
Since you're looking for previous rows and columns in the out
array, you'll often find yourself in a situation where you've overwritten those values before you check them.
For instance, when you're at i=2, j=1
, you've got the array
[[ 23. 77. 89.]
[ 40. 109. 0.]
[ 83. *?* 0.]
[114. 0. 0.]
[135. 0. 0.]]
You're at the *?*
and checking for 109 and 83. And you've just inserted 109 in the previous iteration.
Thus, you are unable to get the same result without the use of those for loops.
answered Nov 22 '18 at 9:13
Mart RMart R
112
112
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
add a comment |
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
I only want the last value of the table, not the entire table. I also don't change the element at [0,0], although I overwrite it, it has the same value anyway since it's cumsum of 1st element
– mateobruno
Nov 22 '18 at 9:28
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426966%2fremove-for-loops-to-make-it-faster-numpy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What did you try?
– MaxPowers
Nov 22 '18 at 8:56
I tried unsuccessfully to make it recursive but I still think it won't be fast like that. I really need to make it as fast as possible
– mateobruno
Nov 22 '18 at 8:59
Welcome to SO. Please try to think from another users perspective, who doesn't know anything about your problem, which might be for days or weeks now in your head. So, Minimal, Complete, and Verifiable example and How to Ask apply here. I'd like to state that your code doesn't even run, because you forgot the outer brackets in your table definition...
– SpghttCd
Nov 22 '18 at 9:01