Arduino ESP8266 HTTPS Request returning empty response

Question

I want to get some JSON data from a HTTPS secured Webserver using a ESP8266. For some reason I'm getting an empty response, I believe there's an error parsing the GET request. In Chrome I get the JSON response without problems.

GET Request:

httpsClient.print(String("GET ") + path + " HTTP/1.1\r\n" +
 "Host: " + host + "\r\n" + 
 "Connection: close\r\n\r\n");

Host and Path:

const char *host = "www.adler-mannheim.de";
const char *path = "/jsonapi/game/current";

UPDATE:

I tried fetching some JSON from another Server and it worked finde, so i can rule out my code. Fiddling around with the host and path I observe this:

• Hostname without "www" results in a 301

• Hostname with "https://www" results in no HTTPS connection at all

So it seems that a "www" is required although it unfortunately leads into a empty response.

The output is as follows:

headers received
reply was:
==========
==========
closing connection

In the first case, without using "www" in ahead of host there's some 301 HTML in between the two lines of "=".

Whole function:

void connect() {
 WiFiClientSecure httpsClient; //Declare object of class WiFiClient
 Serial.println(host);
 httpsClient.setInsecure();
 
 Serial.print("HTTPS Connecting");
 int r=0; //retry counter
 while((!httpsClient.connect(host, httpsPort)) && (r < 30)){
 delay(100);
 Serial.print(".");
 r++;
 }
 if(r==30) {
 Serial.println("Connection failed");
 }
 else {
 Serial.println("Connected to web");
 }
 
 String getData, Link;
 //GET Data
 Serial.print("requesting URL: ");
 Serial.println(host+Link);
 httpsClient.print(String("GET ") + path + " HTTP/1.1\r\n" +
 "Host: " + host + "\r\n" + 
 "Connection: close\r\n\r\n");
 Serial.println("request sent: ");
 Serial.println(String("GET ") + path + " HTTP/1.1\r\n" +
 "Host: " + host + "\r\n" + 
 "Connection: close\r\n\r\n");
 while (httpsClient.connected()) {
 String line = httpsClient.readStringUntil('\n');
 if (line == "\r") {
 Serial.println("headers received");
 break;
 }
 }
 Serial.println("reply was:");
 Serial.println("==========");
 String line;
 while(httpsClient.available()){ 
 line = httpsClient.readStringUntil('\n'); //Read Line by Line
 Serial.println(line); //Print response
 }
 Serial.println("==========");
 Serial.println("closing connection");
 
 delay(2000); //GET Data at every 2 seconds
}

UPDATE 2:

I tried the following code from the ESP8266HTTPClient Library examples:

void connect2() {
 WiFiClientSecure client;
 //client.setFingerprint(fingerprint);
 client.setInsecure();
 
 HTTPClient https;
 Serial.print("[HTTPS] begin...\n");
 if (https.begin(client, "https://github.com/esp8266/Arduino/issues/3417")) { // HTTPS
 Serial.print("[HTTPS] GET...\n");
 // start connection and send HTTP header
 int httpCode = https.GET();
 // httpCode will be negative on error
 if (httpCode > 0) {
 // HTTP header has been send and Server response header has been handled
 Serial.printf("[HTTPS] GET... code: %d\n", httpCode);
 // file found at server
 if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
 String payload = https.getString();
 Serial.println(payload);
 }
 } else {
 Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
 }
 https.end();
 } else {
 Serial.printf("[HTTPS] Unable to connect\n");
 }
 Serial.println("Wait 10s before next round...");
 delay(10000);
}

This one gives me a 200 but no payload. Other sites work

UPDATE 3:

I had a look on the headers.

https.headers()

This gives me 0 headers on every page, even when there's a payload.

I also had a closer look at the payload that returns from my desired api and its filled with NULL. In a weird way this is only the case with my desired page and only my desired subpages. Requesting something more at top level e.g "https://www.adler-mannheim.de/jsonapi" which is basically a SilverStripe Welcome page, I get a fully fetched beautiful HTML payload.

I got literally no Idea why. Do I have to set a specific User Agent or something so that the Server responds?

• Christopher: I assume it's an empty response since it says Reply (nothing in between), whereas from other sites theres a response in between. I'll edit the question to make it clear
• Majenko: https:// is not part of the hostname. It's the protocol indicator for a URI.
• Majenko: The 301 response will also have a "Location:" header which tells you the real URI you should be accessing. 301 is permanently moved, so it is safe for you to replace your URI in your code with the one provided in the Location header.
• Christopher: The 301 points to the exact URL which I'm trying to access (adler-mannheim.de/jsonapi/game/current). It also works fine in a Browser.
• Juraj: I guess you don't wait for the response long enough

Answer 1

Okay I have no Idea what I changed apart from deleting some imports which might have collided, but now it works. Regular call with "https://www" does the thing.

Thanks anyways

• timemage: It would be better to have a more complete answer, assuming it can be of use to someone else. Otherwise the question and answer both are probably better off deleted.
• Juraj: the version with WiFiClient or the version with HttpClient?

Answer 2

I had this same problem and I realized it was because I was using an WiFiClientSecure to try to an endpoint over HTTP instead of HTTPS. If anyone else is having this same problem, I would check that.