Nsryjdtyk

I'm trying to use libcurl to download the content of a webpage however the responses I get back are much smaller than expected - the documentation says that the maximum size is 16K but the responses I'm getting are much much smaller than that.

For example, requesting the en Wikipedia page "Hello" returns a size of 1239 bytes when the page is 71915 bytes in gedit. Requesting a page from the haveibeenpwned API returns a size of 554, not the actual size which is 19942 bytes.

int callback(char* resultPtr, size_t ignore, size_t resultSize) { // resultPtr points to the start of the response, ignore is always 1, resultSize varies

    printf("Response size is %i", resultSize);

    for(int i = 0; i < resultSize; i++) { printf("%c", *(resultPtr+i));} // credit @BartFriederichs

    return 0;

}



int doCurl(char* paramsPtr) {

    char url[43]; // Add space for /0

    CURL *curl = curl_easy_init(); // Initialise cURL

    strcpy(url, "https://api.pwnedpasswords.com/range/"); // Testing URL

    strcat(url, &*paramsPtr); // Always 5 characters

    curl_easy_setopt(curl, CURLOPT_URL, url); // cURL URL set to URL specified above

    curl_easy_setopt(curl, CURLOPT_BUFFERSIZE, 120000L); // Tried to set this to make the output longer

    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, callback);

    curl_easy_perform(curl);

    return 0;

}

The point of this code is to get the response from HIBP's Pwned Passwords API and compare a string given by the user to the results from the webpage. The search succeed for any strings within the first 554 bytes (first chunk of data output by cURL) but fail after that.

Check out the documentation. It says:

The callback function will be passed as much data as possible in all invokes, but you must not make any assumptions. It may be one byte, it may be thousands.

It doesn't say so explicitly, but this function might be called more than once. You can use the userdata pointer (which you haven't used) to keep track of your previous data. I advise you also take a look at the example code:

struct MemoryStruct {

  char *memory;

  size_t size;

};



static size_t

WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)

{

  size_t realsize = size * nmemb;

  struct MemoryStruct *mem = (struct MemoryStruct *)userp;



  char *ptr = realloc(mem->memory, mem->size + realsize + 1);

  if(ptr == NULL) {

    /* out of memory! */ 

    printf("not enough memory (realloc returned NULL)n");

    return 0;

  }



  mem->memory = ptr;

  memcpy(&(mem->memory[mem->size]), contents, realsize);

  mem->size += realsize;

  mem->memory[mem->size] = 0;



  return realsize;

}