A Certification Authority can be up, healthy, answering on the network, passing every check you have pointed at it, and completely unable to issue a certificate. Not because anything on the CA is broken. Because the thing it needs is somewhere else.

This is not an exotic failure. It is the predictable consequence of a design decision that is correct and that you should not reverse. It is worth understanding before it presents itself at three in the morning.

Where the key actually lives

A CA signs certificates with a private key. On any environment built to a reasonable standard, that key does not sit on the CA. It sits in a hardware security module, an appliance whose entire purpose is that the key never leaves it.

That last part is not a figure of speech. The CA does not fetch the key and sign locally. It cannot. It packages the data to be signed, hands it to the HSM over the network, the HSM performs the signing operation internally, and the signature comes back. Every issuance, every CRL, every OCSP response signed by the CA itself, is a network round trip to a separate device.

The mental model that fits is a notary who is forbidden from keeping his own seal. The seal is in a bank vault across town and by law he may never carry it. Every document he notarizes is couriered to the vault, stamped, and couriered back. He is at his desk. He is awake. He is willing. When the road to the vault washes out, he cannot notarize anything, no matter how loudly you knock on his door.

What that means for availability

The availability of your issuance service is not the availability of your CA. It is the intersection of several things:

  • the CA host and the certificate services role
  • the HSM itself, and the state of its authenticated session with the CA client
  • every network segment, firewall rule, route, and load balancer between the two
  • the vendor client software on the CA that speaks to the appliance

Most PKI monitoring watches the first item and stops. That is why this failure mode arrives as a surprise. The service is running. The role responds. Nothing in the dashboard is red. What has failed is a dependency nobody drew on the diagram, and the symptom appears somewhere else entirely, usually as helpdesk tickets about enrolment.

In an Active Directory Certificate Services environment specifically, the CA reaches the HSM through a Key Storage Provider registered with CNG, which in turn talks to the vendor's client library, usually over PKCS #11. That is a Microsoft-specific implementation detail of a pattern that is not Microsoft-specific at all. The diagnostic that tells you the truth fastest is certutil -verifykeys, which forces the CA to exercise its signing key end to end rather than merely confirming that a certificate object exists. A CA that passes a service check and fails -verifykeys has told you exactly where the problem is.

The failure you can live with

An outage that stops issuance is unpleasant and it is mostly recoverable on its own.

Autoenrolment retries. Requests submitted through the standard paths either fail and get resubmitted or sit pending. Services with certificates that have weeks of validity left do not care that they could not have been renewed during a four hour window at two in the morning. When the path comes back, the backlog drains, usually without anyone doing anything.

If your entire exposure is deferred issuance, you have a monitoring problem and a communication problem, not an availability crisis.

The failure you cannot

Signing a certificate revocation list uses the same key, over the same path, subject to the same dependency. This is where the real risk sits, and it is the part that gets missed.

Two things make CRL publication more dangerous than issuance:

First, it is scheduled rather than demand driven. If the publication job fires during the outage window, it fails, logs an event most people are not watching, and waits for the next scheduled interval. It does not aggressively retry the way an enrolling client does. A failure that lands in the wrong window can sit unnoticed until someone runs certutil -CRL by hand.

Second, the consequences are fleet wide rather than local. A client that cannot enrol is one client inconvenienced. A CRL that has passed its NextUpdate is a validation failure for every relying party that checks revocation and is configured to fail closed. That is not a delayed certificate. That is authentication and TLS breaking across systems that were working perfectly and never asked for anything.

So the number that matters is not your CRL validity period. It is your overlap, meaning the gap between when a fresh CRL is published and when the previous one expires. That overlap is your outage budget. If you publish an issuing CA CRL every three days with a seven day validity period, you have roughly four days of cushion, and you can absorb a missed publication cycle without anyone noticing. If you publish every seven days with a seven day validity, you have no cushion at all, and a single failed job is a countdown.

Size the overlap against the worst outage you are willing to survive, then add margin, because the outage will not politely occur at the start of the window.

Delta CRLs deserve the same arithmetic and usually get less of it. And if your OCSP responder signing certificates are issued by the CA on a short validity with automatic renewal, which is a common and otherwise sensible configuration, note that their renewal path runs through the same dependency you just lost.

What to monitor instead

The lesson from any incident of this shape is not that the architecture is wrong. It is that the environment was instrumented to observe the wrong layer. Practical corrections, in the order I would implement them:

Synthetic issuance canary. On a schedule, request a certificate from a low impact template dedicated to this purpose, verify the result, and alert on failure. This is the only check that exercises the whole chain, CA to KSP to client library to network to HSM and back. Everything else is inference.

Direct dependency health. Monitor the HSM client's connectivity and session state from the CA host, not just the appliance's own status page. The appliance can be perfectly healthy while being unreachable from the one machine that needs it.

Signing latency as a leading indicator. Round trip time on signing operations degrades before it fails. A path that has quietly failed over to a longer route shows up as latency days before it shows up as an outage.

CRL freshness from the outside. Fetch the CRL from a published distribution point, on the same network path a relying party would use, and alert on time remaining to NextUpdate rather than on job success. Job success tells you the CA thinks it worked. Remaining validity tells you whether your users are about to have a bad day.

What not to do

The wrong response to this incident is to conclude that the network dependency is the problem and to move signing keys into software. It is not the problem. Keys in an HSM is the correct answer and the network round trip is the price of it. FIPS 140 validated modules at the higher assurance levels do not merely resist tampering, they actively respond to it, zeroizing key material on enclosure breach, out of range temperature or voltage, or unauthorized movement. That is the difference between a lock that shows you it was picked and a safe that destroys the contents on the way in. The notary cannot carry the seal in his pocket because the seal would not survive the trip.

If the dependency genuinely needs to shorten, the honest options are architectural, not cryptographic. Cluster or group members local to each CA site. HSMs placed in the same fault domain as the CAs they serve. Redundant paths that do not share a failure mode. Each of these carries a real cost in key custody, ceremony complexity, and operational overhead, and each is a decision to make deliberately rather than reactively at the end of a long night.

Standards footing

The principle that a private key never exists in plaintext outside a protected boundary is older than any of the products implementing it. FIPS 140-3 defines the security requirements and assurance levels for the cryptographic module itself. NIST SP 800-57 covers key management practice around it. RFC 5280 defines the CRL structure and the NextUpdate semantics that the arithmetic above depends on. PKCS #11 is the interface the CA host generally uses to reach the module.

None of those documents will tell you that a WAN event in an unrelated part of your network can present as a PKI failure. That part you have to draw on the diagram yourself.