curl, or an application that uses libcurl, may have a problem with an SSL certificate that works fine when using a web browser to access the same URL. Typical error output from curl looks like this:
$ curl -v https://my-subdomain.mysecuresite.com Trying xxx.xxx.xxx.xxx:443… TCP_NODELAY set Connected to my-subdomain.mysecuresite.com (xxx.xxx.xxx.xxx) port 443 (#0) ALPN, offering h2 ALPN, offering http/1.1 successfully set certificate verify locations: CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none TLSv1.3 (OUT), TLS handshake, Client hello (1): TLSv1.3 (IN), TLS handshake, Server hello (2): TLSv1.2 (IN), TLS handshake, Certificate (11): TLSv1.2 (OUT), TLS alert, unknown CA (560): SSL certificate problem: unable to get local issuer certificate Closing connection 0 curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
Troubleshooting Strategy
- A good starting point for any SSL error on a public-facing URL is to analyze the URL at SSL Labs.
- Does the error happen for all certificates issued by a specific Certificate Authority (CA)? If so, the system running curl may need to have a root certificate for that CA added or updated in its certificate repository. That’s a relatively rare problem, but might occur if the system running curl is very old. There’s a comprehensive thread about this issue on Stack Overflow.
- If the error only happens for one specific site, it’s likely that the site is missing an intermediate certificate. The command to diagnose this issue is also found in that Stack Overflow thread:
openssl s_client -connect myhost.com:443 -servername myhost.com -showcerts
The output should show a series of certificates, starting with the site certificate, and ending with the root certificate for the Certification Authority. If this chain only shows the site certificate, that’s the problem.
Why does it work in a browser or on a Mac?
Browsers have the ability to download intermediate certificates if they’re not offered by the site. curl (at least on Linux systems) does not. curl on Linux doesn’t reference the CA certificates provided by OpenSSL. Instead, curl references its own CA vault in NSS. Interestingly, the version of curl that comes with MacOS references the CA vault provided by MacOS, so curl on a Mac is less likely to experience this error than curl on a Linux system.
Creating a Kubernetes Ingress with an SSL Certificate Chain
This issue with curl and SSL intermediate certificates will only happen more frequently as more people migrate to Kubernetes. There is a very specific procedure for creating a Kubernetes ingress with an SSL certificate chain, and it’s not well documented. Create a Kubernetes TLS secret that contains the private key and the full certificate chain. The certificate chain must have a very specific format:
-----BEGIN CERTIFICATE----- site certificate -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- intermediate certificate -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- CA root certificate -----END CERTIFICATE-----
The site certificate is the one you get from your CA; the intermediate certs and the root certificate are provided as part of a “bundle” or “chain” file that should have been provided along with the site certificate.