Flaky tests led Buildkite to a Redis memory bug

A cluster of flaky tests turned out to be something much deeper: a memory corruption bug in the redis-client library.

Buildkite says the issue first surfaced when engineering manager David saw several test failures and flagged them in Slack. Within 15 minutes, the team treated the failures as serious enough to block deployments and temporarily removed the tests to unblock main. Looking at Buildkite’s Test Engine reliability score, David noticed the three worst tests had gone from roughly 100% reliable to unstable 2–4 days earlier. One recent change stood out: a Redis gem upgrade made the previous Friday afternoon.

At first, the team suspected an ordinary race condition. The failing tests lived in a feature test suite using RSpec, a Selenium headless browser, a web/API server, a database, and Redis—exactly the kind of setup where timing issues are common. They even tried inserting sleeps to check whether delayed WebSocket messages were the cause.

That theory didn’t hold. By the following Monday, more tests in the same file were failing sporadically. After building a more reliable repro—which could still take up to 30 minutes to fail—the team narrowed the problem to interactions between ActionCable and Redis. Extra logging showed that a subscribe command reached ActionCable but never arrived at Redis. Developers also reported WebSocket failures in development, suggesting the issue went beyond the test environment.

The breakthrough came after a strange allocator error appeared:

Recommended reading

Why robotics data started looking like YouTube

“malloc: Double free of object”

A few days later, another build triggered a segmentation fault and produced a core dump. That artifact existed thanks to an internal plugin another developer had built to automatically upload core dumps from failed test runs. Former Buildkite engineer Rian used it to inspect the crash and found it occurred inside hiredis, the C extension bundled into the redis-client gem.

How ASAN exposed the bug

Rian traced the crash to a call to the C function memmove. The source and destination pointers looked valid, but the size field was 0x00a0ffffffffffb6—about 45 petabytes. Combined with the odd behavior seen after the gem upgrade, that strongly suggested memory corruption.

Rian then ran the repro on a version of Ruby compiled with ASAN after seeing a conference talk earlier that year, “Finding and fixing memory safety bugs in C with ASAN.” It took a couple of hours to get running, and within 30 minutes ASAN reported a heap use-after-free.

The root cause: in some cases, hiredis’s reader thread would free the writer buffer. Hiredis uses two threads per connection, one for reading and one for writing.

Once Buildkite knew where the corruption came from, the fix was relatively straightforward. Rian disabled the C extension for Redis connections in test, development, and production and created a reproducible test case to report upstream. Buildkite says no production data was harmed.