📱

Read on Your E-Reader

Thousands of readers get articles like this delivered straight to their e-reader. Works with Kindle, Boox, and any device that syncs with Google Drive or Dropbox.

Learn More

This is a preview. The full article is published at blog.cloudflare.com.

Code Orange: Fail Small — our resilience plan following recent incidents

Code Orange: Fail Small — our resilience plan following recent incidents

By Dane KnechtThe Cloudflare Blog

On November 18, 2025 , Cloudflare’s network experienced significant failures to deliver network traffic for approximately two hours and ten minutes. Nearly three weeks later, on December 5, 2025 , our network again failed to serve traffic for 28% of applications behind our network for about 25 minutes. We published detailed post-mortem blog posts following both incidents, but we know that we have more to do to earn back your trust. Today we are sharing details about the work underway at Cloudflare to prevent outages like these from happening again. We are calling the plan “ Code Orange: Fail Small ”, which reflects our goal of making our network more resilient to errors or mistakes that could lead to a major outage. A “Code Orange” means the work on this project is prioritized above all else. For context, we declared a “Code Orange” at Cloudflare once before , following another major incident that required top priority from everyone across the company. We feel the recent events require the same focus. Code Orange is our way to enable that to happen, allowing teams to work cross-functionally as necessary to get the job done while pausing any other work. The Code Orange work is organized into three main areas: Require controlled rollouts for any configuration change that is propagated to the network, just like we do today for software binary releases. Review, improve, and test failure modes of all systems handling network traffic to ensure they exhibit well-defined behavior under all conditions, including unexpected error states. Change our internal “break glass”* procedures, and remove any circular dependencies so that we, and our customers, can act fast and access all systems without issue during an incident. These projects will deliver iterative improvements as they proceed, rather than one “big bang” change at their conclusion. Every individual update will contribute to more resiliency at Cloudflare. By the end, we expect Cloudflare’s network to be much more resilient, including for issues such as those that triggered the global incidents we experienced in the last two months. We understand that these incidents are painful for our customers and the Internet as a whole. We’re deeply embarrassed by them, which is why this work is the first priority for everyone here at Cloudflare. * Break glass procedures at Cloudflare allow certain individuals to elevate their privilege under certain circumstances to perform urgent actions to resolve high severity scenarios. In the first incident, users visiting a customer site on Cloudflare saw error pages that indicated Cloudflare could not deliver a response to their request. In the second, they saw blank pages. Both outages followed a similar pattern. In the moments leading up to each incident we instantaneously deployed a configuration change in our data centers in hundreds of cities around the world. The November change was an automatic update to our Bot Management classifier. We run various artificial intelligence models that learn from the traffic flowing through our network to build detections that identify bots. We constantly update...

Preview: ~500 words

Continue reading at Cloudflare

Read Full Article

More from The Cloudflare Blog

Subscribe to get new articles from this feed on your e-reader.

View feed

This preview is provided for discovery purposes. Read the full article at blog.cloudflare.com. LibSpace is not affiliated with Cloudflare.

Code Orange: Fail Small — our resilience plan following recent incidents | Read on Kindle | LibSpace