11-byte HollowByte attack can bloat OpenSSL memory

A newly disclosed OpenSSL denial-of-service issue, dubbed HollowByte, lets unauthenticated attackers force servers to consume large amounts of memory with a malicious payload of just 11 bytes. The OpenSSL team has already fixed the bug without assigning it an identifier and has backported the patch to older branches.

According to an advisory from Okta’s Red Team, the flaw sits in how vulnerable versions handle TLS handshake messages. Each handshake starts with a 4-byte header that includes a three-byte length field describing how much data should follow. In affected releases, OpenSSL allocates memory based on that declared length before the payload arrives and before the size is validated.

“The worker thread then blocks, waiting indefinitely for data that will never arrive.”

That means an attacker can open a TLS connection, send an 11-byte input claiming a much larger body is coming, and repeat the process across many connections. The result is heavy memory allocation with very little inbound traffic.

Recommended reading

DHS missed HSIN hack twice before declaring breach

Okta said the effect is amplified by how glibc manages memory. Although OpenSSL frees buffers when connections close, glibc often keeps small and medium allocations for reuse instead of returning them to the operating system right away. By varying the claimed sizes across waves of connections, an attacker can fragment the heap and steadily increase a server’s Resident Set Size (RSS).

“By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks. The heap fragments heavily, causing the server’s Resident Set Size (RSS) to climb continuously. Even after the attacker disconnects, the server remains permanently bloated.”

In Okta’s NGINX testing, low-capacity systems were easy to exhaust, while larger servers could still lose up to 25% of memory without attack bandwidth crossing common alerting thresholds. Recovering that memory fully requires restarting the affected process.

OpenSSL is widely embedded across NGINX, Apache, Node.js, Python, Ruby, PHP, MySQL, and PostgreSQL, and ships by default on most Linux distributions for TLS and certificate handling. Fixes are available in OpenSSL 4.0.1 and backported to 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Those releases now grow the buffer only as data arrives instead of trusting the header claim.

Even though the issue was handled as a “hardening fix” rather than a formal security vulnerability, Okta urged organizations to upgrade their distribution’s OpenSSL packages immediately.

×