HTTP Problem

[theLizard]

Hello All,

Using TIdHttp and up until sometime in April this year everything working fine with the following

http->Get("http://www.pvoutput.org");

The only thing that has changed is that the url name went from an HTTP to an HTTPS which by my understanding the http://www.pvoutput.org should be automatically redirected to https://www.pvoutput.org but when attempting to connect using GET I get an 301 Moved Permanently error.

I cannot understand what is going on here, is there anything, a setting for TIdHttp component that would address this problem?

Cheers

[rlebeau]

The only thing that has changed is that the url name went from an HTTP to an HTTPS which by my understanding the http://www.pvoutput.org should be automatically redirected to https://www.pvoutput.org

Yes, and that is done using an HTTP redirect, which all web browsers handle by default.

but when attempting to connect using GET I get an 301 Moved Permanently error.

301 is one of the defined HTTP redirect response codes. An EIdHTTPProtocolException will be thrown for a 301 response if the TIdHTTP::HandleRedirects property is false (which it is by default). Make sure it is set to true instead (or handle the TIdHTTP::OnRedirect event to get the new URL, and then call TIdHTTP::Get() again with it).

Also, make sure you have an SSLIOHandler component, like TIdSSLIOHandlerSocketOpenSSL, assigned to the TIdHTTP::IOHandler property to handle the SSL/TLS aspect of the new URL when following the redirect.

And then stop using the old URL, since it is clearly no longer valid (as it has been *permanently* moved to the new URL, as opposed to other 3xx redirects which are *temporary* instead and the old URL should still be used).

[theLizard]

Also, make sure you have an SSLIOHandler assigned to handle the SSL/TLS aspect of the new URL.

Thanks Remy,

Since original post I tried HandleRedirect and assigning an SSLIOHandler which I thought would be the correct way to go about it your reply confirms this but I get an "Could not load SSL Library" error and I am not sure where to find the library on my system or what it is actually called

Cheers

[theLizard]

So, I have found libeay32 and libssl32 dll's and included these in my project but still get EidOSSLCouldNotLoadSSLLibrary error what do I need to do to get these loaded as they appear to be the libs missing?

[HsiaLin]

http://docwiki.embarcadero.com/RADStudi ... en/OpenSSL

[theLizard]

http://docwiki.embarcadero.com/RADStudi ... en/OpenSSL

Thanks for the link, I put libs in my projects folder instead, will try putting them in the right place tomorrow.

Cheers

[theLizard]

OK, the libs are loading, no longer getting "Cannot Load Library" error but getting Socket Error 10060 - Connection Timed Out

Should I be doing something with the SSLIOHandler properties or HTTP properties

Cheers

[rlebeau]

OK, the libs are loading, no longer getting "Cannot Load Library" error but getting Socket Error 10060 - Connection Timed Out

Should I be doing something with the SSLIOHandler properties or HTTP properties

Indy's defaults are usually sufficient for most cases. But not always.

When I use the latest version of Indy to connect to "http://www.pvoutput.org", using all of Indy's defaults (other than HandleRedirects, which I set to true), TIdHTTP follows the redirect to "https://www.pvoutput.org", and then the HTTPS server immediately closes the TCP connection gracefully as soon as it receives Indy's SSL/TLS handshake hello. TIdSSLIOHandlerSocketOpenSSL throws an OpenSSL exception accordingly, not a timeout error (specifically, it throws an error that says "encountered EOF that violated the protocol", which is normal for an unexpected disconnect during the SSL/TLS handshake process).

Maybe in your case, the server is not closing the connection gracefully, or your OS is not detecting it gracefully. Use a packet sniffer, like Wireshark, to verify. If the connection is not closed gracefully, that might be causing the timeout error (TIdSSLIOHandlerSocketOpenSSL does enable socket timeouts on Windows Vista+).

The same test works fine in web browsers. The difference being that TIdSSLIOHandlerSocketOpenSSL enables only TLS 1.0 by default, but modern web browsers also enable TLS 1.1 and 1.2 as well. When I enable TLS 1.1 in TIdSSLIOHandlerSocketOpenSSL, TIdHTTP is able to connect to the HTTPS URL and retrieve its HTML normally.

So clearly, the HTTPS server in question does not like Indy's use of TLS 1.0 and wants TLS 1.1+ instead.

[theLizard]

The difference being that TIdSSLIOHandlerSocketOpenSSL enables only TLS 1.0 by default, but modern web browsers also enable TLS 1.1 and 1.2 as well. When I enable TLS 1.1 in TIdSSLIOHandlerSocketOpenSSL, TIdHTTP is able to connect to the HTTPS URL and retrieve its HTML normally.

Thanks Remy,

With your input I have got my problem sorted out however, using http->Get(...) I get EIdException "Error Connecting with SSL" this in itself is not a problem because the http->Post(...) does it's job correctly and that is what I am after so all I would like to know at this point is what could be causing the "Error Connecting with SSL" and is there a specific EId Exception I could be testing for.

Cheers

[rlebeau]

however, using http->Get(...) I get EIdException "Error Connecting with SSL"

I am not able to reproduce that given the URL you provided. Are you using up-to-date versions of Indy and OpenSSL?

this in itself is not a problem because the http->Post(...) does it's job correctly and that is what I am after

Depending on the server's requirements, a GET may or may not be required before a POST, in order to receive server cookies that are required by the POST handler.

Also, how are you able to post data to the server without a successful SSL/TLS connection? Or, is the post URL using HTTP and not HTTPS?

so all I would like to know at this point is what could be causing the "Error Connecting with SSL"

I can't answer that without more details, such as a packet capture of the actual SSL/TLS handshake. There are many possibilities for why an SSL/TLS handshake would fail. Maybe there is a TLS version mismatch between your client and the server, for instance.

is there a specific EId Exception I could be testing for.

The "Error Connecting with SSL" error is thrown as an EIdOSSLConnectError exception.

[theLizard]

I am not able to reproduce that given the URL you provided. Are you using up-to-date versions of Indy and OpenSSL?

The app uses c++ 2010 so am using the indy components that came with it which I would say are not upto date.

Depending on the server's requirements, a GET may or may not be required before a POST, in order to receive server cookies that are required by the POST handler.

In my case I am using GET to establish that a connection can be made before a post is attempted, it is possible that a GET is not required but have not tested that.

Also, how are you able to post data to the server without a successful SSL/TLS connection? Or, is the post URL using HTTP and not HTTPS?

POST is using HTTP as is GET, I have not made changes to the URL code leaving the component to handle redirects. will change to https and see the result.

I can't answer that without more details, such as a packet capture of the actual SSL/TLS handshake. There are many possibilities for why an SSL/TLS handshake would fail. Maybe there is a TLS version mismatch between your client and the server, for instance.

Would love to be able to show packet capture but I cannot install wireshark (I am comfortable with it) on my dev machine it causes an hissy fit for wireshark, am in the process of migrating my dev machine to a newer updated system.

The "Error Connecting with SSL" error is thrown as an EIdOSSLConnectError exception.

Will test for this error, cheers.


Thanks again for your assistance Remy.

[rlebeau]

The app uses c++ 2010 so am using the indy components that came with it which I would say are not upto date.

Not even close. Please upgrade.

In my case I am using GET to establish that a connection can be made before a post is attempted

That is a waste of a connection, especially if HTTP keep-alives are not used. There is no difference in establishing a connection to the server for a GET vs a POST, so just do the POST unconditionally and handle any errors it may throw. HTTP is stateless, it doesn't care why a connection is being made, and it does not guarantee that the connection will remain open after the server's response is sent. Since each request will potentially require a new connection anyway, just use as few requests as needed to get the job done. Like I said, there is only really one reason to ever perform a GET before a POST, and that is if server generated cookies/token are required in the POST, such as when posting an HTML webform. Otherwise, don't do it, unless a POST-based API requires it.

POST is using HTTP as is GET, I have not made changes to the URL code leaving the component to handle redirects. will change to https and see the result.

If GET requires HTTPS, then POST most certainly will as well.

Would love to be able to show packet capture but I cannot install wireshark (I am comfortable with it) on my dev machine it causes an hissy fit for wireshark, am in the process of migrating my dev machine to a newer updated system.

Wireshark is not the only way to capture network traffic.

[theLizard]

Thanks Remy, have followed your suggestion eliminating GET all is fine now..

Cheers