Nsryjdtyk

i'm trying to read the best price from the skyscanner website using a normal get request, but i'm not getting the content that i want by using this code.

private void getRequest() throws Exception {

    StringBuilder result = new StringBuilder();

    URL url = new URL(URL);

    HttpURLConnection conn = (HttpURLConnection) url.openConnection();

    conn.addRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0");

    System.out.println(conn.getURL());

    conn.setInstanceFollowRedirects(true);

    HttpURLConnection.setFollowRedirects(true);

    conn.setRequestMethod("GET");

    BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));

    String line;

    while ((line = rd.readLine()) != null) {

        result.append(line);

    }

    System.out.println(conn.getURL());

    rd.close();

    response = result.toString();

}

The requested URL is the following:

https://www.skyscanner.com/transport/flights/fra/txl/181220/?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&rtn=0&preferdirects=false&outboundaltsenabled=false&inboundaltsenabled=false&currency=EUR&market=DE&locale=en-US

Response from the code above looks like this:

https://pastebin.com/YKh17RKE

By going to the mentioned skyscanner link in chrome i can click on inspect element and voila under

 fqs-opts-container -> <span class="fqs-price">42 €</span>

i can see the cheapest price.

How to get this information using java? What am i doing wrong here?

Thanks in advance.

Inspect shows the current HTML DOM (Document Object Model) resulting from:

• the static HTML page (see right-click + View page source) plus


• dynamic modifications by JavaScript.

If you do Inspect, tab Network and reload the page, you can see which files (and their contents) are all requested by the browser to display the page.

In this particular case, it seems that you could get the data as JSON:

In the tab Network filter for conductor/v1/fps3/search/. The query is an HTTP post request with the URL https://www.skyscanner.de/g/conductor/v1/fps3/search/?geo_schema=skyscanner&carrier_schema=skyscanner&response_include=query%3Bdeeplink%3Bsegment%3Bstats%3Bfqs%3Bpqs%3B_flights_availability. The answer is in JSON and includes a session_id which is required as part of the URL for subsequent requests for details.

Please note that even if it is technically possible to receive the data, it is in most cases forbidden to use them commercially.