Unfortunately zervicepoint was unavailable between 2021-05-18 11:56:54 to 2021-05-19 08:20. This was due to an expired certificate and we're sorry for any inconvenience this may have caused.
While the issue was resolved quickly, we regret that we did not catch this issue through our current monitoring.
What happend?
From our internal investigation, we've identified two issues.:
- When our certificates are pending expiration, they are automatically renewed and stored in a vault. This created a false sense of "certificate does not expire soon" as it was refreshed, though not deployed.
- Deployment of the new certificate from the vault did not occur as intended to the required endpoints.
Our plan to avoid this in the future:
- Add SSL monitoring test and alert. (Current SSL check did not monitor on all components where the certificate is distributed to, this created a false truth that certificate was not about to expire)
- Fix certificate deployment to correctly distribute certificate when it has been renewed and should be deployed.