Post Mortem of the 7th July 2023 Jenkins Infrastructure Outage
On Friday 7th of July 2023, the Jenkins infrastructure suffered a major outage from 11:05am UTC until 15:25pm UTC with complete downtime of the following public services:
accounts.jenkins.io
fallback.get.jenkins.io
get.jenkins.io
incrementals.jenkins.io
javadoc.jenkins.io
plugin-health.jenkins.io
plugin-site-issues.jenkins.io
plugins.origin.jenkins.io
plugins.jenkins.io
rating.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
weekly.ci.jenkins.io
www.origin.jenkins.io
We also had complete downtime of the following non-public services:
ldap.jenkins.io
previews of *.jenkins.io pull requests (infra.ci.jenkins.io)
In addition, there was disruption (partial or complete) of the following services:
ci.jenkins.io
infra.ci.jenkins.io
issues.jenkins.io
plugins.jenkins.io
repo.jenkins-ci.org
www.jenkins.io
The public IPs of these services changed (DNS records...
