Certificates & the Chain of Trust

You deploy a new TLS certificate on your Kubernetes ingress controller. The cert is valid — you just bought it. You apply the Secret, the ingress reloads, and you open the site in Chrome.

"Your connection is not private — NET::ERR_CERT_AUTHORITY_INVALID."

The certificate is correct. The domain matches. It is not expired. But Chrome refuses to trust it.

The problem: you uploaded only the leaf certificate. You forgot the intermediate. Chrome cannot verify the chain of trust because a link is missing. This is the single most common TLS deployment error, and understanding certificate chains is how you fix it in minutes instead of hours.

---

Part 1: What Is Inside an X.509 Certificate

An X.509 certificate is a structured data file that binds a public key to an identity. When a server presents its certificate during the TLS handshake, the client reads these fields to decide whether to trust the connection.

Let us decode a real certificate:

# Download and decode a certificate
echo | openssl s_client -connect google.com:443 -servername google.com 2>/dev/null \
  | openssl x509 -text -noout

The output contains these critical fields:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1a:2b:3c:4d:5e:6f:...
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Google Trust Services, CN = GTS CA 1C3
        Validity
            Not Before: Mar  1 08:00:00 2026 GMT
            Not After : May 24 08:00:00 2026 GMT
        Subject: CN = *.google.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:*.google.com, DNS:google.com, DNS:*.appengine.google.com, ...
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            Authority Information Access:
                OCSP - URI:http://ocsp.pki.goog/gts1c3
                CA Issuers - URI:http://pki.goog/repo/certs/gts1c3.der
    Signature Algorithm: sha256WithRSAEncryption
         5a:3b:7c:...

The Fields That Matter

Subject — who the certificate is for. The CN (Common Name) field traditionally holds the domain name. However, modern TLS implementations use the Subject Alternative Name (SAN) extension instead. Go 1.15+ and recent browsers ignore CN entirely and require SANs.

Issuer — who signed this certificate. This is the Certificate Authority (CA) that vouches for the certificate's authenticity. If the issuer is the same as the subject, it is a self-signed certificate.

Validity — the time window when the certificate is valid. Not Before and Not After define the window. Certificates that have passed their Not After date are expired and will be rejected.

# Quick check: when does this cert expire?
echo | openssl s_client -connect devopsbeast.com:443 -servername devopsbeast.com 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=Mar  1 00:00:00 2026 GMT
# notAfter=May 30 23:59:59 2026 GMT

Subject Alternative Name (SAN) — the list of domain names this certificate is valid for. A single cert can cover multiple domains. This is how a wildcard certificate (*.example.com) and multi-domain certificates work.

# Check which domains a certificate covers
echo | openssl s_client -connect google.com:443 -servername google.com 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName
# X509v3 Subject Alternative Name:
#     DNS:*.google.com, DNS:google.com, DNS:*.youtube.com, ...

Public Key — the server's public key that clients use during the TLS handshake. This is the public half of the key pair whose private key lives on the server.

Signature — the digital signature from the issuer (CA). The CA hashed the certificate data and signed it with the CA's private key. Anyone with the CA's public key can verify this signature.

---

Part 2: The Certificate Chain

A single certificate is not enough. The client needs to verify that the certificate was legitimately issued — that the server is not just presenting a forged certificate. This verification uses a chain of trust.

The Three Levels

Root CA: Self-signed (the issuer IS the subject). Pre-installed in your operating system and browser. There are roughly 150 root CAs that major browsers trust. These root certificates are the foundation of internet trust. The root CA private key is kept in offline hardware security modules (HSMs) and is used only to sign intermediate CA certificates.

Intermediate CA: Signed by the root CA. This is the CA that actually issues certificates to websites. Intermediates exist for a critical security reason: the root CA private key stays offline and is almost never used. If an intermediate CA is compromised, only certificates from that intermediate need to be revoked — the root remains trusted.

Leaf Certificate: Your server's certificate. Signed by the intermediate CA. Contains your domain name, public key, validity dates, and the intermediate's signature.

How Verification Works

When your browser connects to a server over HTTPS:

1. The server sends its leaf certificate and intermediate certificate(s)
2. The browser reads the leaf certificate's Issuer field
3. It finds the intermediate certificate that matches that Issuer
4. It verifies the leaf certificate's signature using the intermediate's public key
5. It reads the intermediate's Issuer field
6. It finds the root certificate in its local trust store
7. It verifies the intermediate's signature using the root's public key
8. The root is self-signed and pre-trusted — chain verified

---

Part 3: The Missing Intermediate Problem

This is the most common TLS deployment error. It accounts for more "certificate authority invalid" errors than any other cause.

What Happens

1. You get a certificate from your CA (Let's Encrypt, DigiCert, etc.)
2. The CA gives you a leaf certificate and an intermediate certificate
3. You configure your server with only the leaf certificate
4. The server sends only the leaf during the TLS handshake
5. The client receives the leaf, reads the Issuer field, but does not have the intermediate
6. The client cannot verify the chain — it does not know who signed the leaf
7. Error: "unable to verify the first certificate" or "NET::ERR_CERT_AUTHORITY_INVALID"

Why It Sometimes Works Anyway

Desktop Chrome and Firefox have a feature called AIA fetching (Authority Information Access). If the intermediate is missing, they follow the AIA URL in the leaf certificate to download the intermediate. So it "works" in desktop browsers.

But these clients do NOT fetch missing intermediates:

• curl and wget
• Python requests
• Go net/http
• Node.js https
• OpenSSL s_client
• Java HttpsURLConnection
• Mobile apps
• API clients

This is why a certificate can "work in Chrome" but break everywhere else.

# Check what certificates the server actually sends
openssl s_client -showcerts -connect example.com:443 -servername example.com 2>/dev/null

# You should see TWO certificates:
# Certificate 0 (leaf): s:CN = example.com
# Certificate 1 (intermediate): s:CN = R3, O = Let's Encrypt

# If you only see Certificate 0 — the intermediate is missing

How to Fix It

The server must send the full chain: leaf certificate + intermediate certificate(s). The root is NOT included (clients already have it).

# Create a full chain file (leaf + intermediate)
cat leaf.crt intermediate.crt > fullchain.crt

# In nginx
ssl_certificate     /etc/ssl/fullchain.crt;   # leaf + intermediate
ssl_certificate_key /etc/ssl/private.key;      # private key

# In Kubernetes Ingress (the tls Secret must contain the full chain)
kubectl create secret tls my-tls \
  --cert=fullchain.crt \
  --key=private.key

# Verify the chain is correct
openssl verify -CAfile root.crt -untrusted intermediate.crt leaf.crt
# leaf.crt: OK

# Or verify a full chain file
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt fullchain.crt
# fullchain.crt: OK

---

Part 4: Certificate Types

Not all certificates are created equal. The type affects how the certificate is issued and what the CA verifies.

Domain Validation (DV)

• The CA verifies you control the domain (via DNS record, HTTP challenge, or email)
• Automated, issued in seconds
• Let's Encrypt, ZeroSSL, AWS Certificate Manager all issue DV certs
• Sufficient for 99% of use cases

Organization Validation (OV)

• The CA verifies the legal organization behind the domain
• Requires manual review, takes days
• Shows organization name in the certificate Subject field
• Used by some enterprises for compliance

Extended Validation (EV)

• The CA performs extensive verification of the organization
• Used to show a green bar with the company name in browsers
• Browsers have largely removed the visual distinction (no more green bar)
• Provides no additional cryptographic security
• Mostly obsolete for practical purposes

---

Part 5: Let's Encrypt and the ACME Protocol

Let's Encrypt fundamentally changed TLS by making certificates free and fully automated. Before Let's Encrypt (founded 2014, public 2016), certificates cost $10-300/year and required manual renewal.

How ACME Works

ACME (Automatic Certificate Management Environment) is the protocol Let's Encrypt uses:

1. Your ACME client (certbot, cert-manager, acme.sh) generates a key pair
2. It requests a certificate for your domain from Let's Encrypt
3. Let's Encrypt issues a challenge to prove you control the domain
4. Your client completes the challenge
5. Let's Encrypt issues the certificate (valid for 90 days)
6. Your client automatically renews before expiry

Challenge Types

HTTP-01: Let's Encrypt asks your server to serve a specific file at http://yourdomain.com/.well-known/acme-challenge/TOKEN. If it can fetch the file, you control the domain.

# Using certbot with HTTP challenge
certbot certonly --webroot -w /var/www/html -d devopsbeast.com

DNS-01: Let's Encrypt asks you to create a specific DNS TXT record (_acme-challenge.yourdomain.com). If the record exists, you control the domain. This is the only challenge type that supports wildcard certificates.

# Using certbot with DNS challenge
certbot certonly --dns-cloudflare -d "*.devopsbeast.com" -d devopsbeast.com

90-Day Validity

Let's Encrypt certificates expire after 90 days (compared to 1-2 years for paid certificates). This is intentional:

• Forces automation (manual renewal every 90 days is unsustainable)
• Limits damage from compromised keys (shorter window)
• Encourages modern certificate management practices

# Check cert expiry on a remote server
echo | openssl s_client -connect devopsbeast.com:443 -servername devopsbeast.com 2>/dev/null \
  | openssl x509 -noout -enddate
# notAfter=May 30 23:59:59 2026 GMT

---

Part 6: Self-Signed Certificates

A self-signed certificate is one where the Subject and Issuer are the same — the certificate signs itself. No external CA is involved.

# Generate a self-signed certificate
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes \
  -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"

Self-signed certificates provide encryption but NOT trust. Any attacker can generate a self-signed certificate for any domain. Without a CA vouching for the certificate, there is no way to distinguish the real server from an impersonator.

Use self-signed certificates ONLY for:

• Local development
• Internal testing environments
• Services where you control both client and server and can pin the certificate

---

Part 7: Wildcard Certificates and File Formats

Wildcard Certificates

A wildcard certificate (*.example.com) covers any single-level subdomain:

Domain	Covered by *.example.com?
app.example.com	Yes
api.example.com	Yes
staging.example.com	Yes
sub.app.example.com	No — wildcards do not cover nested subdomains
example.com	No — the bare domain is not covered by the wildcard

To cover both example.com and *.example.com, you need both in the SAN field:

# Let's Encrypt wildcard + bare domain
certbot certonly --dns-cloudflare \
  -d "*.example.com" \
  -d "example.com"

Certificate File Formats

Format	Extension	Encoding	Contains	Common Use
PEM	.pem, .crt, .cer	Base64 (text)	Cert and/or key	Linux, nginx, Apache
DER	.der, .cer	Binary	Single cert	Java, Windows
PKCS#12 / PFX	.pfx, .p12	Binary	Cert + key + chain	Windows, Java keystores

# Convert PEM to DER
openssl x509 -in cert.pem -outform DER -out cert.der

# Convert PEM to PKCS12 (combine cert + key)
openssl pkcs12 -export -in cert.pem -inkey key.pem -out cert.pfx

# Convert PKCS12 to PEM
openssl pkcs12 -in cert.pfx -out cert.pem -nodes

# View a DER certificate
openssl x509 -in cert.der -inform DER -text -noout

---

Key Concepts Summary

• X.509 certificates bind a public key to an identity (domain name) and are signed by a Certificate Authority
• The certificate chain has three levels: root CA (pre-trusted), intermediate CA (signs daily), leaf cert (your server)
• Chain verification walks from leaf to intermediate to root — if any link is missing or invalid, the connection is rejected
• The missing intermediate is the most common TLS error — the server must send the leaf AND intermediate certificates
• AIA fetching lets desktop browsers download missing intermediates, masking server misconfiguration — but API clients, mobile apps, and CLI tools will fail
• Let's Encrypt provides free, automated DV certificates via the ACME protocol with 90-day validity
• SAN (Subject Alternative Name) has replaced Common Name for domain validation — always include SANs in certificates
• Wildcard certificates cover single-level subdomains only — *.example.com does not cover sub.app.example.com
• PEM is Base64 text, DER is binary, PKCS#12 bundles cert + key — convert between them with OpenSSL

Common Mistakes

• Configuring the server with only the leaf certificate instead of the full chain (leaf + intermediate)
• Testing only in Chrome, which masks missing intermediates via AIA fetching
• Using Common Name (CN) without Subject Alternative Names (SAN) — Go 1.15+ and modern clients reject CN-only certs
• Forgetting that wildcard certs do not cover the bare domain or nested subdomains
• Using self-signed certificates in production instead of Let's Encrypt or an internal CA
• Not monitoring certificate expiry — Let's Encrypt certs expire every 90 days, and auto-renewal can fail silently

---

What is Next

You now understand what certificates contain, how the chain of trust works, and why the missing intermediate is the most common TLS error. But theory is not enough when production is down.

In the next lesson, we will cover the 10 certificate errors you will see in production — with the exact error message, the exact cause, the exact diagnostic command, and the exact fix for each one. Print it, bookmark it, keep it open during on-call.