mTLS (Mutual TLS): Architecture, Configuration, and Troubleshooting

Standard TLS authenticates the server to the client; mTLS authenticates both directions—critical for zero-trust, service meshes, API gateways, and device identity. This guide covers handshake mechanics, certificate requirements, reverse-proxy configuration, mesh modes, and a practical troubleshooting workflow. For pattern-level decisions and production lessons, see Mutual TLS patterns.

mTLS (Mutual TLS): Architecture, Configuration, and Troubleshooting

TL;DR: mTLS adds client authentication to TLS; you need correct SAN/EKU/KU, trust anchors on both sides, and often full chains. Configure Nginx/Envoy/HAProxy explicitly; in Kubernetes, Istio/Linkerd automate rotation—without a mesh, pair cert-manager with app or sidecar TLS.

Overview

Use this page when you are implementing or debugging mTLS at the TLS layer (proxies, handshakes, certificates). Pair it with mutual TLS patterns for when to use mTLS, service mesh certificates for mesh-wide identity, and private CA comparison for internal issuance options (Vault PKI, step-ca, EJBCA).

Problem Statement

• Two chains, double failure modes: Server and client paths both break independently—errors rarely say which side failed.
• EKU/KU mismatches: “Server” templates used for clients fail TLS Web Client Authentication checks.
• Trust store gaps: ssl_client_certificate without the issuing CA (or intermediate) → unknown CA.
• Proxies strip client certs: L7 load balancers must re-encrypt or use TCP passthrough for mTLS.
• Short-lived certs: Mesh-style rotation needs observability—see monitoring.
• Revocation: Browsers differ; internal meshes often rely on short TTL over CRL/OCSP—see revocation deep dive.

Failure scenario: New client cert verifies with openssl verify but Nginx returns 400—client sends leaf only; server cannot build path to internal intermediate trusted by ssl_client_certificate.

How mTLS works

The mTLS handshake extends the standard TLS 1.2/1.3 handshake with an additional step. In TLS 1.3, the flow is:

1. ClientHello: The client initiates the connection with supported cipher suites and key shares.

2. ServerHello + Server Certificate: The server responds with its certificate and requests a client certificate (via a CertificateRequest message).

3. Client Certificate: The client presents its certificate.

4. Client CertificateVerify: The client proves possession of the private key corresponding to its certificate by signing a hash of the handshake transcript.

5. Verification: The server verifies the client certificate against its configured trust store (the CA certificate or chain that issued the client certificate). The client verifies the server certificate against its trust store. Both sides now have cryptographically verified each other’s identity.

The result: an encrypted channel where both endpoints have authenticated identities. The server knows the client is service-a.internal because the client presented a certificate issued by a trusted CA with that identity in the Subject Alternative Name.

Certificate requirements for mTLS

Server certificate

The server certificate is a standard TLS server certificate. The Subject Alternative Name (SAN) must contain the server’s FQDN, IP address, or URI that clients will use to connect. The Extended Key Usage (EKU) must include TLS Web Server Authentication (OID 1.3.6.1.5.5.7.3.1). This is the same certificate you’d use for regular TLS.

Client certificate

The client certificate identifies the connecting client. Key requirements that are frequently misconfigured:

Subject Alternative Name: Should contain the client’s identity — a DNS name (e.g., service-a.internal), a SPIFFE ID (e.g., spiffe://cluster.local/ns/default/sa/service-a), an IP address, or an email address. The identity format depends on your authentication model.

Extended Key Usage: Must include TLS Web Client Authentication (OID 1.3.6.1.5.5.7.3.2). This is the most common misconfiguration — certificates generated for server use may not include client authentication EKU. If the server’s TLS library enforces EKU checking, the handshake fails.

Key Usage: Must include Digital Signature. The client uses the private key to sign the CertificateVerify message during the handshake.

CA certificate (trust anchor)

Both sides need a trust anchor: the server needs the CA certificate that issued client certificates, and the client needs the CA certificate that issued the server certificate. In most mTLS deployments, both certificates come from the same internal CA, so a single CA certificate serves as the trust anchor for both sides.

For hierarchical PKI: the trust store should contain the root CA or the intermediate CA that issued the peer’s certificates. If using intermediate CAs, the peer must send the full chain (leaf + intermediate) during the handshake, and the verifying side must have the root or intermediate in its trust store.

Configuration

Nginx

server {
    listen 443 ssl;
    server_name api.internal;

    # Server certificate and key
    ssl_certificate     /etc/nginx/ssl/server.pem;
    ssl_certificate_key /etc/nginx/ssl/server-key.pem;

    # Client certificate verification
    ssl_client_certificate /etc/nginx/ssl/client-ca.pem;
    ssl_verify_client on;        # Require client certificate
    # ssl_verify_client optional; # Accept but don't require
    ssl_verify_depth 2;          # Allow intermediate CAs

    # Pass client identity to backend
    proxy_set_header X-Client-DN    $ssl_client_s_dn;
    proxy_set_header X-Client-Cert  $ssl_client_escaped_cert;
    proxy_set_header X-Client-Verify $ssl_client_verify;

    location / {
        proxy_pass http://backend;
    }
}

Key variables: $ssl_client_s_dn contains the client certificate’s subject DN. $ssl_client_verify is SUCCESS if verification passed, or an error string otherwise. $ssl_client_escaped_cert contains the URL-encoded client certificate for passing to backend services that need to inspect it.

Envoy (service mesh sidecar)

clusters:
- name: backend
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain:
            filename: /certs/client.pem
          private_key:
            filename: /certs/client-key.pem
        validation_context:
          trusted_ca:
            filename: /certs/server-ca.pem

listeners:
- filter_chains:
  - transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
        require_client_certificate: true
        common_tls_context:
          tls_certificates:
          - certificate_chain:
              filename: /certs/server.pem
            private_key:
              filename: /certs/server-key.pem
          validation_context:
            trusted_ca:
              filename: /certs/client-ca.pem

HAProxy

frontend https_in
    bind *:443 ssl crt /etc/haproxy/ssl/server.pem ca-file /etc/haproxy/ssl/client-ca.pem verify required
    http-request set-header X-Client-DN %[ssl_c_s_dn]
    http-request set-header X-Client-Verify %[ssl_c_verify]
    default_backend servers

The verify required parameter enforces mTLS. Use verify optional to accept but not require client certificates.

curl (testing)

# Test mTLS connection
curl -v \
  --cert client.pem \
  --key client-key.pem \
  --cacert server-ca.pem \
  https://api.internal/health

# Debug certificate chain issues
openssl s_client \
  -connect api.internal:443 \
  -cert client.pem \
  -key client-key.pem \
  -CAfile server-ca.pem \
  -state -debug

Kubernetes service mesh mTLS

Istio

Istio automates mTLS for all service-to-service communication within the mesh. The control plane (istiod) runs a certificate authority that issues SPIFFE-based identity certificates to Envoy sidecar proxies. No application code changes are required — the sidecars handle all TLS negotiation.

PeerAuthentication resources control mTLS enforcement:

# Require mTLS for all services in the namespace
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

Modes: STRICT — all connections must use mTLS; plaintext connections are rejected. PERMISSIVE — accept both mTLS and plaintext; useful during mesh migration. DISABLE — no mTLS enforcement.

Certificate lifecycle: Istio certificates are short-lived (24 hours by default) and automatically rotated by the Envoy sidecar. This is passive revocation — certificates expire before revocation would be relevant.

Linkerd

Linkerd uses a similar model with automatically injected proxies and automatic mTLS between meshed services. Linkerd generates per-proxy identity certificates from a trust anchor and issuer certificate that you provide at installation. Certificate rotation is automatic.

Without a service mesh

For Kubernetes workloads without a service mesh, use cert-manager with a CA or ACME issuer to provision client and server certificates as Kubernetes Secrets. Mount these Secrets into Pods and configure the application or sidecar (Envoy, Nginx) to perform mTLS. This approach requires more manual configuration but avoids the operational overhead of a full service mesh.

Troubleshooting mTLS failures

mTLS failures are harder to debug than standard TLS failures because there are two certificate chains to verify. Axelspire’s diagnostic workflow:

Step 1: Identify the error

“certificate verify failed” — The server rejected the client certificate, or the client rejected the server certificate. Determine which side failed.

“no client certificate received” — The client didn’t send a certificate. The client may not have a certificate configured, the server may not have sent a CertificateRequest, or a proxy between client and server may have stripped the client certificate.

“unknown CA” — The verifying side doesn’t have the issuing CA in its trust store. The most common cause of mTLS failures.

“certificate has expired” — Self-explanatory, but common in environments with short-lived certificates and unreliable renewal.

Step 2: Verify the certificate chain

# Check the client certificate's chain
openssl verify -CAfile client-ca.pem client.pem

# Check the server certificate's chain
openssl verify -CAfile server-ca.pem server.pem

# Check Extended Key Usage on the client certificate
openssl x509 -in client.pem -noout -text | grep -A 1 "Extended Key Usage"
# Must include: TLS Web Client Authentication

# Check validity dates
openssl x509 -in client.pem -noout -dates

Step 3: Test the handshake

# Connect with full debug output
openssl s_client \
  -connect api.internal:443 \
  -cert client.pem \
  -key client-key.pem \
  -CAfile server-ca.pem \
  -state

# Look for:
# - "Acceptable client certificate CA names" — the CAs the server trusts
# - "SSL handshake has read ... and written ..." — confirms handshake completed
# - "Verify return code: 0 (ok)" — chain verification passed

Step 4: Common fixes

Missing intermediate certificate: The client sends the leaf certificate but not the intermediate. The server can’t build the chain back to the root. Fix: concatenate the leaf and intermediate certificates into a single PEM file and configure that as the client certificate.

Wrong CA in server trust store: The server’s ssl_client_certificate points to the wrong CA or an incomplete chain. Fix: ensure the CA file contains the CA that actually issued the client certificates, including intermediates if applicable.

EKU mismatch: The client certificate was generated without the TLS Web Client Authentication EKU. Fix: regenerate the certificate with the correct EKU. For OpenSSL, include extendedKeyUsage = clientAuth in the extensions configuration.

SAN mismatch (server-side): If the server validates the client certificate’s SAN against expected values (e.g., in an API gateway or service mesh), a mismatch causes rejection even if the chain verifies. Fix: ensure the client certificate’s SAN matches the identity expected by the server’s authorization policy.

For broader chain and trust issues, see chain validation errors.

mTLS architecture patterns

Internal service-to-service (zero trust)

Every service in the internal network authenticates to every other service via mTLS. No service trusts the network. All communication is encrypted and authenticated. This is the standard model for service meshes (Istio, Linkerd) and modern microservice architectures.

Certificate source: Internal CA (step-ca, Vault PKI, or the service mesh’s built-in CA). Short-lived certificates (hours to days) with automatic rotation. No CRL/OCSP needed — certificates expire before revocation would matter.

API gateway with client authentication

External clients (partners, devices, applications) present client certificates to an API gateway. The gateway verifies the client certificate and forwards the client identity to backend services. This is common in financial services (PSD2 QWAC certificates), healthcare (FHIR API authentication), and B2B API integrations.

Certificate source: A dedicated CA (potentially the organization’s enterprise CA) issues client certificates to authorized API consumers. Certificate lifetimes are typically longer (90-365 days) because external clients may not support automated renewal. Active revocation (CRL) is more important in this pattern because you need the ability to revoke a client’s access without waiting for certificate expiration.

Device identity (IoT, mobile)

Devices present certificates to authenticate to backend services. The certificate is provisioned during manufacturing (IDevID) or enrollment (LDevID) and may be stored in a hardware security element (TPM, Secure Enclave). mTLS ensures that only enrolled devices can communicate with the backend.

Certificate source: Device CA (EJBCA, step-ca Certificate Manager) with SCEP or EST enrollment. Certificate lifetimes vary — constrained devices may use long-lived certificates if they cannot reliably perform automated renewal.

Operational checklist

• Server SAN matches how clients connect; client SAN/SPIFFE matches authorization policy.
• EKU: server serverAuth, client clientAuth; KU includes Digital Signature on the client.
• Trust bundles include issuing CA (and intermediates as needed); peers send full chains.
• Renewal automated for short-lived certs (renewal automation).
• L7 proxies preserve mTLS or terminate with documented trust boundaries.
• Runbooks for rotation failures and openssl s_client steps above.

Related documentation

• Mutual TLS patterns — When to use mTLS, anti-patterns, production lessons
• Service mesh certificates — Mesh trust, rotation, cert-manager integration
• Zero-trust architecture — Context for identity-first networking
• Certificate anatomy — SAN, EKU, extensions
• Private CA comparison — AD CS, EJBCA, step-ca, Vault PKI
• HashiCorp Vault PKI — API issuance and ACME
• cert-manager for Kubernetes — ACME and secrets in clusters
• Chain validation errors — Debugging trust paths
• Certificate revocation (CRL/OCSP) — When revocation matters outside meshes