Rust Rewrite Enables Cloudflare to Boost CDN Performance and Enhance Security

Write for InfoQ

By adopting Rust for one of its core subsystems, Cloudflare successfully reduced response time by 10 ms and increased performance by 25%. Additionally, the company emphasized that Rust made their system more secure and reduced development time.

On the heels of its successful migration to Rust for the Pingora subsystem, Cloudflare's engineers have rewritten from scratch one of the company's oldest and most critical components, FL, the "brain of Cloudflare":

FL is the brain of Cloudflare. Once a request reaches FL, we then run the various security and performance features in our network. It applies each customer’s unique configuration and settings, from enforcing WAF rules and DDoS protection to routing traffic to the Developer Platform and R2 [Cloudflare's object store, EN].

Cloudflare's architects decided to base FL2 on Oxy, their own internal framework for building proxies, which also includes support for monitoring, soft reloads, and dynamic configuration loading and swapping.

In particular, Oxy's built-in mechanism for graceful restarts is a critical feature for a proxy, since terminating a process would otherwise break all active connections. To prevent this, when an Oxy instance needs to be terminated, it stops accepting new connections but continues serving existing ones until they end naturally.

One critical hurdle for Cloudflare's architects was how to replace a running system that had underpinned 15 years of Cloudflare products and was still evolving. To prevent their teams from having to implement each new feature twice, once for the LuaJIT-based FL and once for the new Rust-based FL2, they created a layer inside of FL that allowed new modules implemented in Rust for FL2 to run seamlessly.

Instead of maintaining a parallel implementation, teams could implement their logic in Rust, and replace their old Lua logic with that, without waiting for the full replacement of the old system.

To properly handle such a migration, Cloudflare's architects also defined a clear testing and rollout strategy. For testing, they used Flamingo, a system capable of running thousands of full end-to-end test requests concurrently against both FL1 and FL2. Each change is rolled out gradually, and at each stage, it is fully tested against increasing traffic and benchmarked to ensure that performance and resource usage remain acceptable.

Another key mechanism was FL2's ability to pass any request it could not handle to FL1. This fallback mechanism was essential for gradually increasing FL2 usage without compromising the overall stability of Cloudflare services. During the rollout, as FL2 matured, it handled a growing proportion of traffic, while the amount of traffic falling back to FL1 decreased correspondingly.

The main advantage of building FL2, say Cloudflare architects, is the performance gain it delivers. This can be attributed to two primary factors: the fact that FL2 was entirely written in a high-performance language like Rust, rather than using a combination of C, Rust, and Lua code. Additionally, FL1 had to spend quite some time converting data representation from one language to another. As a result, FL2 requires only half the CPU of FL1 and less than half the memory.

As a final note, FL2 also benefits from Rust's compile-time security guarantees, reinforced by strong linting and checking rules, as well as strict coding standards, thorough testing, and rigorous review processes. This has led to a significant reduction in crashes, with most of these being attributed to hardware failures.

Related Topics:

• Development
• Architecture & Design
• Rust
• NGINX
• Cloudflare
• Security
• Lua
• Performance

We protect your privacy.