How to fix error “[SSL: CERTIFICATE_ VERIFY_FAILED] certificate verify failed” (_ssl.c:727)
While working with one of our banking sector clients (hybrid cloud ), we encountered the error:
fatal error: SSL validation failed for https://bucket_name.s3.ap-south-1.amazonaws.com/file_name “[SSL: CERTIFICATE_ VERIFY_FAILED] certificate verify failed” (_ssl.c:727)
Scenario:
I was copying one file from s3 bucket to one of the newly launched servers x.x.x.x via AWS cli with below-mentioned command.
command: AWS_ACCESS_KEY_ID=XXXXXXX AWS_SECRET_ACCESS_KEY=XXXXXXX aws s3 cp s3://bucket_name/file_nameoutput: fatal error: SSL validation failed for https://bucket_name.s3.ap-south-1.amazonaws.com/file_name "[SSL: CERTIFICATE_ VERIFY_FAILED] certificate verify failed" (_ssl.c:727)
The above output is not giving enough information to troubleshoot this further. so I have used the OpenSSL command.
command: openssl s_client -connect bucket_name.s3.ap-south-1.amazonaws.com:443 -servername bucket_name.s3.ap-south-1.amazonaws.comOutput: Refer Fig1a and Fig1b
With the help of the OpenSSL command, I got to know that our network is blocking internet requests due to some proxy configured for all the internet requests.
But the question arises why it is giving certificate verification failed error? Well, we need to understand first how TLS/SSL communication/handshake works. While performing GET operation on s3 via s3 cp command, it is making a HTTPS request on the endpoint “ https://bucket_name.s3.ap-south-1.amazonaws.com/file_name” which is breaking at point number 3 in Fig 2 because proxy lies between client and s3 endpoint whose certificate verification is failing and also whitelisting is not done. (This type of warning also comes in a browser while accessing some sites but you click on advance > proceed further to access the site)
Note: Go through the blog, to know more about session keys and master secret.
Workaround:
So, the workaround to this is either you create a vpc-endpoint to communicate with s3 within the private network or get your endpoint whitelisted at the proxy level because all the internet requests are going via proxy in this network configured by the network proxy team.
Conclusion
So in this blog, we have seen that AWS hits a regional endpoint over SSL when you access any AWS service ( s3 in our case ) and that endpoint resolves to public IP. But any hop like a proxy ( configured for all the internet requests ) in between can restrict that request and we can bypass that proxy with VPC endpoint or whitelisting can solve this problem. Thanks for reading, I’d really appreciate your suggestions and feedback.
Originally published at http://blog.opstree.com on October 18, 2021.
