AWS Signature failure - Incorrect HOST header?

[Gprsk]

Hello once more!
I've been experimenting some scenarions with AWS Authentication and I landed on a particularly painful one.

The library miscalculates the AWS Signature if, for some reason, you enable SSL and add a user+password, but AWS Authentication is disabled on AWS ElasticSearch. This causes ES to automatically reject the request and return 403 Forbidden.

It should work if I provide AWS Credentials or not - AWS is supposed to simply ignore them, as long as they are sent correctly. I could send requests without problems by using Postman in the same scenario.

I'll try to explain how to reproduce the error.

Important: ES Authentication is DISABLED on AWS - The Secret/Access Keys being used are real, enabled and valid.

My log4net configuration:

    <appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
      <Server>my-elastic-domain.com</Server>
      <Port>443</Port>
      <IndexName>index-here</IndexName>
      <IndexType>logEvent</IndexType>
      <BulkSize>100</BulkSize>
      <BulkIdleTimeout>1000</BulkIdleTimeout>
      <IndexAsync>False</IndexAsync>
      <DropEventsOverBulkLimit>False</DropEventsOverBulkLimit>
      <SerializeObjects>True</SerializeObjects>
      <ElasticSearchTimeout>10000</ElasticSearchTimeout>
      <FixedFields>Partial</FixedFields>

      <Ssl>true</Ssl>
      
      <AuthenticationMethod>
        <!--For AWS ElasticSearch service-->
        <Aws>
          <Aws4SignerSecretKey>SECRET</Aws4SignerSecretKey>
          <Aws4SignerAccessKey>ACCESS</Aws4SignerAccessKey>
          <Aws4SignerRegion>us-east-1</Aws4SignerRegion>
        </Aws>
      </AuthenticationMethod>
    </appender>

Error reported by log4net:

log4net:ERROR Got error while reading response from ElasticSearch
System.InvalidOperationException: Some error occurred while sending request to ElasticSearch.
System.Net.WebException: The SSL connection could not be established, see inner exception.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Cannot determine the frame size or a corrupted frame was received.

---

Now, experimenting with Postman, this is what I've found:

• 1 - No Auth, No SSL, Domain untouched: ES works and responds with status code 200:
  

• 2 - With Auth, No SSL, Domain untouched: ES works and responds with status code 200:
  

• 3 - With Auth, SSL, Domain untouched: Same thing as before:
  

• 4 - With Auth, SSL, Domain pointing to port 443: Immediate 403 Forbidden:
  

• 5 - With Auth, SSL, Domain pointing to port 443 and manually setting HOST header to be the same as domain: ES status code 200:
  

I think the error might be the same, however I'm not sure. If this is fixed somehow, it means a lot less downtime in my applications during the switch from non-auth to aws signature enabled.

[Gprsk]

By testing a little more, this is what i've found:

• The full error is this one. Seems not like a HOST header, but a certificate problem. Not so sure how to solve it:

log4net:ERROR Got error while reading response from ElasticSearch
System.InvalidOperationException: Some error occurred while sending request to ElasticSearch.
System.Net.WebException: The SSL connection could not be established, see inner exception.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch

If I enable AllowSelfSignedServerCert, this is the error:

log4net:ERROR Got error while reading response from ElasticSearch
System.InvalidOperationException: Some error occurred while sending request to ElasticSearch.
System.Net.WebException: The SSL connection could not be established, see inner exception.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.

Im trying to find out if this is something "solvable" or something is wrong with our certificates.

---

At the same time, could I suggest you guys to add another option to the XML Configuration?
Something along the lines of EnableInsecureSSL, so even on insecure connections (my case currently? seems like it) it would work.

This implementation can be made on file WebElasticClient - and if this option is active, just passing a return true; callback to ServicePointManager.ServerCertificateValidationCallback event should suffice.