What are Certificate Outages? How to Avoid SSL Certificate Outages with ACME?

Most people might now know about digital identity certificates, but they notice when organizations don’t handle them well.

Suppose someone tries to access your organization’s website online, but the website suddenly stops working. Firstly, they simply refresh the page and try again, but nothing happens.

It becomes frustrating, right?

This can happen when the digital identity certificate expires. These certificates verify the identity of websites and devices, ensuring a secure connection. When these certificates expire, access is denied.

This does not only happen to small organizations; organizations like Microsoft, GitHub, SpaceX, etc., have also experienced outages due to expired certificates.

Such certificate outages create server downtime, making organizations lose millions of dollars.

So, let’s begin this blog by learning about certificate outages and how organizations can avoid them with ACME.

What is a Certificate Outage?

A certificate outage or a certificate failure usually occurs when an SSL/TLS certificate becomes invalid, expired, or revoked, making the server unsuitable for creating secure connections.

During such an outage, websites and online services that rely on these certificates may face interruptions and somehow become vulnerable to cyberattacks and data breaches. This type of incident can cause a chain reaction of problems, harming user trust, reputation, and financial well-being of the harmed companies.

What Causes Outages?

Various factors cause certificate outages, including:

Expired Certificates:

One of the reasons outages happen is when website owners or system administrators don’t work to Renew SSL/TLS certificates after they expire. This leads to service or connection disruptions.

Recommended: Expired SSL Certificates Are Risky: 14.7 Million People Affected by the Mr. Cooper Data Breach

Human Error:

Another factor is human mistakes during the certificate installation, configuration, or renewal process.

Complex Ecosystem:

Moreover, the growing complexity of certificate administration in large organizations with multiple services and domains also results in misconfigurations and errors, which can cause outages.

Vendor and Third-Party Issues:

Relying on third-party services or suppliers for certificate supply and maintenance increases the risk of potential outages.

Revoked Certificates:

Some certificates are revoked due to security breaches, causing outages until new certificates are issued and implemented.

Lack of Monitoring:

Not proactively managing certificate health and expiration dates makes them invalid, leading to unplanned disruptions.

These outages can have serious consequences for SEO, trust, and regulations. SEO favors website security, but outages may affect rankings, causing users to lose trust in a website’s reliability and security.

Recommended: What is SSL Certificate Monitoring? Why Monitoring SSL Certificates Becomes Essential?

Moreover, when it comes to regulations, businesses that do not follow rules like GDPR (General Data Protection Regulation) or PCI DSS (Payment Card Industry Data Security Standard) can face legal issues and outages.

2023’s Biggest Certificate Outages

1. Microsoft’s Spotify feature

When Microsoft revamped its Clock app for Windows 11 in 2021, they added Spotify integration to play tracks suitable for intense, concentrated work. Spotify has even chosen a few productivity-focused tunes to improve the integration experience.

However, the feature was removed in February 2023 and remained unavailable for months thereafter. Users could no longer link their Spotify accounts with the Clock app. Spotify detected an issue when users complained in both the Spotify and Microsoft support forums.

This happened because the certificate expired, and the oATH header submitted to Spotify’s API was no longer valid. In other words, Microsoft was responsible for the error. This should have been a straightforward solution, but people were disappointed by how long the problem continued.

The Takeaway – This event demonstrates how a certificate outage can poison the well of a new product or integration.

In our age of digital services, it’s easy to forget that developing a new service requires some level of upkeep – certifications, in this case. When launching new products, businesses must plan ahead of time and allocate the resources required to keep features operational in the long run. After that, it’s time to go into the actual security setup.

2. Microsoft SharePoint

When a certificate outage occurred with Microsoft SharePoint, it prevented Microsoft Teams, Outlook, and other services from functioning. Even though Microsoft identified and resolved the problem in minutes, users experienced inconveniences for several hours.

Further research discovered that the sharepoint.de German TLS certificate was mistakenly added to the parent sharepoint.com domain.

The Takeaway – Though this occurrence is not catastrophic, it shows how simple it is to make a certificate error. Taking certificate handling out of human hands as much as possible decreases the possibility of human error. So, organizations striving to streamline certificate management should automate as much as feasible.

3. Cisco SD-WAN

So, when the certificates for Cisco’s Viptela and Meraki SD-WAN gear expired, it affected approximately 20,000 users and caused disruptions in the cloud, data storage, e-commerce tools, and other services.

Cisco received the certificates and hardware when it acquired Viptela in 2017. The certificates were four years into a 10-year life cycle, but Cisco failed to anticipate their expiration.

It appears that Cisco was hesitant to resolve the issue, which one Redditor dubbed “the most Cisco thing ever.”

The Takeaway – Even in 2024, it is impossible to smoothly integrate a system with an acquired system in a post-merger scenario. There are various factors, configurations, and permissions to consider, including the state of certificate lifecycles.

Finding an unaccounted-for certificate within your environment is difficult enough. Finding it in an unfamiliar, learned situation is even more challenging. This issue demonstrates the risks of M&A activity and the benefits of using an automated, centralized certificate management solution.

4. SpaceX StarLink

Not only Microsoft but when SpaceX’s Starlink satellites were down for several hours, this affected consumers throughout the world. Elon Musk, SpaceX’s CEO, took to Twitter/X to explain the problem, citing an “expired ground station certificate.”

More specifically, analysts thought the expired certificate was a TLS certificate, causing a website or web-based application to fall. Musk went on to say that the “single point vulnerability” was “inexcusable.”

The takeaway – And he was right. The average organization holds more than a quarter million credentials at any moment, and one untracked certificate expiring can bring operations to a standstill. 

On the other hand, all of Musk’s enterprises present themselves as disruptors, and Starlink expects to reach one million customers by 2022. So, in the effort to convince people to trust a new way of doing things, such interruptions can slow down the entire adoption, harm the brand precisely, and cause considerable disruptions.

5. Nvidia & GitHub repository

In January 2023, criminals got unauthorized access to several of GitHub’s source repositories and stole code signing certificates for the company’s Desktop and Atom apps. If the stolen certificates were decrypted, attackers might generate maliciously altered software versions and present them as legitimate updates from GitHub.

Fortunately, GitHub’s code signing certificates were password-protected, resulting in no damage. GitHub simply canceled the stolen certificates and published updated versions of the apps with new certificates.

Nvidia faced a similar incident in 2022 when the hacker organization Lapsus$ exposed Nvidia’s code signing certificates, allowing other hostile actors to use them to sign malware.

The Takeaway – Here, GitHub’s crypto agility proved critical in preventing such a tragedy. Cyberattacks are no longer a question of “if” but of “when.” Organizations must plan to contain threats and prevent them so that if an attack occurs, they can detect and resolve the problem without impacting users.

How To Avoid SSL Outages?

To avoid certificate outage, knowing when the certificate is expiring is essential. To know when it is about to expire, use:

• Certificate management tools for automated monitoring & alerting certificate expiration dates.
• Certificate transparency logs to record certificate issuance and expiration information.
• Manual tracking to maintain a centralized certificate inventory with a schedule to regularly review & track expiration dates manually.
• Certification authority notifications to receive notifications via email or other channels before the expiry date
• Monitoring tools to monitor certificate health and get real-time information about certificate status, including expiration dates.
• Use Certificate Management APIs to retrieve certificate details programmatically.
• Implement organizational policies requiring certificates to be renewed several days before expiration.
• Regularly check for certificate revocations, as revoked certificates are no longer valid, and immediate action should be taken to replace them.

Besides knowing the Expiry, Follow these Practices to Avoid Certificate Outages:

• Using high availability and load balancing systems to distribute traffic across servers with valid certificates, thus avoiding single points of failure.
• Creating a comprehensive incident response plan to ensure a timely response and recovery during a certificate interruption.
• Educating employees and IT personnel about the significance of certificate management, renewal procedures, and the potential risks of certificate expiration.

How Expired Certificates Can Cause Service Downtime and Financial Losses?

Expiring certificates that cause web server downtime are usually costly. According to the 11th annual hourly cost of downtime survey by Information Technology Intelligence Consulting, over 98% of organizations with over 1,000 employees say that a single hour of downtime per year costs a company over 100,000 dollars on average.

That’s around 1,667 dollars per minute of downtime for a single server, growing to 16,670 dollars when downtime affects more than 10 servers, data assets, or business applications.

Such unidentified or expired SSL certificates lead to multiple process interruptions, from a simple error message on a screen to sudden termination of service due to a protocol error.

Meanwhile, other causes of SSL certificate problems and outages include:

• It becomes a non-trusted certificate when a CA or the certificate authority does not digitally sign it, as browsers only trust certificates from a trusted organization on their list of certificates, not an untrusted site.
• An improper certificate installation on the servers hosting the site.
• The URL in question returns a name mismatch error. For example, the ‘https://www.example.com’ domain name may be included in the certificate.
  
  However, https://example.com is unique and may not be registered as part of the SSL certificate. In such cases, an SSL certificate must safeguard several subdomains as well as the original domain name.

Servers without SSL certifications are more vulnerable to hacking, as they expose visitors and customers to a higher risk of their data being stolen. So, buy SSL certificates from Certera and enable the strongest 256-bit encryption for your website.

How To Prevent Outages With Automated Certificate Lifecycle Management

Streamlined Operations and Standardisation

One of the primary issues with managing certificates manually is that there are different methods depending on the Certificate Authority or the type of certificate.

But, with ACME, standardization is at the forefront. This eliminates the potential hazards associated with different procedures during consistent domain validation, certificate issuance, and management.

Also, this consistency reduces potential outages by reducing variables from the equation.

Error Reduction using Automation

Additionally, some human errors, from typos to complex system misconfiguration, create vulnerabilities and are a major cause of disruptions. By adopting an automated approach, ACME significantly minimizes the possibility of such errors. So, with this approach, you’re safeguarding your certifications and reducing the administrative strain on your Security Team.

Automated Certificate Renewal

Expired certificates have typically been the leading source of certificate-related outages. Before ACME, certificate management was a highly laborious procedure. It necessitated expert-level record-keeping, especially in large corporations that manage several domains.

Even the most experienced Security Engineers can miss a renewal due to the administrative difficulties of doing it manually. The ACME’s automated renewal capabilities constantly monitor the certificate’s validity, assuring timely renewals and eliminating the human error component. This minimizes the possibility of outages while preserving availability and confidence.

Enhanced Monitoring and Notification

Administrators can receive rapid notifications if a certificate has a problem or is about to expire, allowing them to take corrective action before any potential disruption happens. Apart from automating certificate issues and renewal, ACME-compatible solutions frequently feature real-time monitoring and alerting capabilities.