Restful paginate with python
I'm getting data using Python via a rest API call as follows...
result = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc"))
There's a 100 max results built into the API so I have to paginate in python.
After making the first call, result.nextPage returns a code that I then have to pass to the next API call like...
result2 = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc&nextPage=someCode"))
And so on until I'm through all pages.
Does python have any in-built mechanism to handle pagination like this?
I could write a clumsy loop myself but am unsure about
- How to handle waiting for each page so I know I can proceed with the next call
- How to handle the 'no more pages' event without erroring
- How to combine the results of all calls into 1 variable after making all calls
Appreciate any thoughts at all on the most elegant way to go about this.
python rest pagination
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
add a comment |
I'm getting data using Python via a rest API call as follows...
result = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc"))
There's a 100 max results built into the API so I have to paginate in python.
After making the first call, result.nextPage returns a code that I then have to pass to the next API call like...
result2 = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc&nextPage=someCode"))
And so on until I'm through all pages.
Does python have any in-built mechanism to handle pagination like this?
I could write a clumsy loop myself but am unsure about
- How to handle waiting for each page so I know I can proceed with the next call
- How to handle the 'no more pages' event without erroring
- How to combine the results of all calls into 1 variable after making all calls
Appreciate any thoughts at all on the most elegant way to go about this.
python rest pagination
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
add a comment |
I'm getting data using Python via a rest API call as follows...
result = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc"))
There's a 100 max results built into the API so I have to paginate in python.
After making the first call, result.nextPage returns a code that I then have to pass to the next API call like...
result2 = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc&nextPage=someCode"))
And so on until I'm through all pages.
Does python have any in-built mechanism to handle pagination like this?
I could write a clumsy loop myself but am unsure about
- How to handle waiting for each page so I know I can proceed with the next call
- How to handle the 'no more pages' event without erroring
- How to combine the results of all calls into 1 variable after making all calls
Appreciate any thoughts at all on the most elegant way to go about this.
python rest pagination
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
I'm getting data using Python via a rest API call as follows...
result = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc"))
There's a 100 max results built into the API so I have to paginate in python.
After making the first call, result.nextPage returns a code that I then have to pass to the next API call like...
result2 = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc&nextPage=someCode"))
And so on until I'm through all pages.
Does python have any in-built mechanism to handle pagination like this?
I could write a clumsy loop myself but am unsure about
- How to handle waiting for each page so I know I can proceed with the next call
- How to handle the 'no more pages' event without erroring
- How to combine the results of all calls into 1 variable after making all calls
Appreciate any thoughts at all on the most elegant way to go about this.
python rest pagination
python rest pagination
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
asked Nov 24 '18 at 8:51
d3wannabed3wannabe
456417
456417
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Does python have any in-built mechanism to handle pagination like this?
No.
Write a while True: loop with a break statement if there are no more pages to load.
How to handle waiting for each page so I know I can proceed with the next call
urllib2.urlopen is not asynchronous. Your code will block (i.e. wait) until the request is done.
How to handle the 'no more pages' event without erroring
That depends on the API you are using. I would expect that result.nextPage is empty/not set on the last page.
How to combine the results of all calls into 1 variable after making all calls
Append them to a list as you go. Pseudo code:
url = "initial URL"
results =
while True:
current_page = request(url)
results.append(current_page)
if (another page available):
url = "new url with next page code"
else:
break
I wholeheartedly recommend that you use the requests module instead of the very bare-bones urllib2.urlopen.
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53456618%2frestful-paginate-with-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Does python have any in-built mechanism to handle pagination like this?
No.
Write a while True: loop with a break statement if there are no more pages to load.
How to handle waiting for each page so I know I can proceed with the next call
urllib2.urlopen is not asynchronous. Your code will block (i.e. wait) until the request is done.
How to handle the 'no more pages' event without erroring
That depends on the API you are using. I would expect that result.nextPage is empty/not set on the last page.
How to combine the results of all calls into 1 variable after making all calls
Append them to a list as you go. Pseudo code:
url = "initial URL"
results =
while True:
current_page = request(url)
results.append(current_page)
if (another page available):
url = "new url with next page code"
else:
break
I wholeheartedly recommend that you use the requests module instead of the very bare-bones urllib2.urlopen.
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
add a comment |
Does python have any in-built mechanism to handle pagination like this?
No.
Write a while True: loop with a break statement if there are no more pages to load.
How to handle waiting for each page so I know I can proceed with the next call
urllib2.urlopen is not asynchronous. Your code will block (i.e. wait) until the request is done.
How to handle the 'no more pages' event without erroring
That depends on the API you are using. I would expect that result.nextPage is empty/not set on the last page.
How to combine the results of all calls into 1 variable after making all calls
Append them to a list as you go. Pseudo code:
url = "initial URL"
results =
while True:
current_page = request(url)
results.append(current_page)
if (another page available):
url = "new url with next page code"
else:
break
I wholeheartedly recommend that you use the requests module instead of the very bare-bones urllib2.urlopen.
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
add a comment |
Does python have any in-built mechanism to handle pagination like this?
No.
Write a while True: loop with a break statement if there are no more pages to load.
How to handle waiting for each page so I know I can proceed with the next call
urllib2.urlopen is not asynchronous. Your code will block (i.e. wait) until the request is done.
How to handle the 'no more pages' event without erroring
That depends on the API you are using. I would expect that result.nextPage is empty/not set on the last page.
How to combine the results of all calls into 1 variable after making all calls
Append them to a list as you go. Pseudo code:
url = "initial URL"
results =
while True:
current_page = request(url)
results.append(current_page)
if (another page available):
url = "new url with next page code"
else:
break
I wholeheartedly recommend that you use the requests module instead of the very bare-bones urllib2.urlopen.
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
Does python have any in-built mechanism to handle pagination like this?
No.
Write a while True: loop with a break statement if there are no more pages to load.
How to handle waiting for each page so I know I can proceed with the next call
urllib2.urlopen is not asynchronous. Your code will block (i.e. wait) until the request is done.
How to handle the 'no more pages' event without erroring
That depends on the API you are using. I would expect that result.nextPage is empty/not set on the last page.
How to combine the results of all calls into 1 variable after making all calls
Append them to a list as you go. Pseudo code:
url = "initial URL"
results =
while True:
current_page = request(url)
results.append(current_page)
if (another page available):
url = "new url with next page code"
else:
break
I wholeheartedly recommend that you use the requests module instead of the very bare-bones urllib2.urlopen.
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
edited Nov 24 '18 at 10:15
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
answered Nov 24 '18 at 9:03
TomalakTomalak
259k51429547
259k51429547
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
add a comment |
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
thank you - that worked perfectly and I'll check out the requests module later
– d3wannabe
Nov 24 '18 at 9:48
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53456618%2frestful-paginate-with-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
