3 min read

OVH patched Januscape with mass reboots and a Sydney test

OVH says it fixed the critical Januscape KVM bug by backporting a Debian patch and rebooting tens of thousands of hosts after testing first in Sydney.

Image: The Register

OVH says it responded to the critical Januscape flaw in KVM by backporting a fix into its production Debian build and rebooting tens of thousands of hosts running about a million virtual machines — after first using its Sydney, Australia region as the test case.

The bug, also known as CVE-2026-53359, allowed an attacker with root access inside a guest VM to execute code as root on the host, crash that machine, or compromise other guest VMs. For cloud providers built on KVM, that is one of the worst-case failure modes: a break in isolation between tenants.

In a detailed post published Monday, OVH CISO Julien Levrard said the company ruled out several alternatives. Disabling nested virtualization with a simple config change was not practical because OVH cannot tell which customers depend on it, and the feature is also used to move VMs between physical hosts. Live patching was rejected over stability concerns, while live migration to already-patched hosts was considered too slow and could have taken months.

Recommended reading

AI’s next bottleneck may be materials, not chips

OVH instead chose a forced-reboot approach, notifying customers in advance but not asking for consent. The company said its executive committee approved that plan for three reasons:

  • the need to patch before attacks began
  • the risk of leaving the cloud exposed longer if cases were handled one by one
  • the goal of protecting the largest number of customers, even if a minority saw downtime

Levrard also said OVH deliberately kept details of the rollout quiet while systems remained exposed.

“Communicating in more detail during the execution of the mitigation plan, while the infrastructure remained unpatched, would have significantly increased the risk for our customers, potentially leading some to 'test' the publicly available exploit.”

Julien Levrard, OVH CISO

How OVH staged the reboot rollout

OVH started in Sydney, describing it as one of its smaller regions and one where the time difference let European teams work during business hours while hitting a quieter period locally. Reboots were then scheduled in waves, with a stop threshold if 15 hosts failed at once in high-density regions or five hosts in other regions.

The company said it went beyond standard anti-affinity rules. Its orchestration system built a co-location graph so that two hosts running instances from the same customer project would not be rebooted in the same window. A host had to come back online before another in the same anti-affinity class could be restarted.

The process still ran into trouble. Some VMs failed to restart after hypervisor reboots. Some saw data corruption during forced shutdowns. OpenStack APIs returned HTTP 503 errors for hours, forcing OVH to delay one wave, and at one Canadian site API traffic surged to 10 times the usual peak, overwhelming managers and support teams.

Hardware failures added to the pain. On the first night, about 20 to 30 hosts out of 6,000 did not recover automatically because of faulty memory modules, BIOS configuration issues, inactive network interfaces, and even dead CMOS batteries.

Levrard called the operation “a remarkable feat” given the scale, but said OVH will have to improve. With more kernel flaws likely, he wrote, the company expects this kind of emergency procedure may need to happen again.

Marcus Vance

Enterprise Editor

Marcus follows the money. He covers enterprise software, cloud architecture, and the tectonic shifts in Big Tech strategy. He translates dense earnings calls and complex M&A activity into actionable insights about where the industry is actually heading. If a tech giant makes a silent pivot, Marcus is usually the first to notice.

via The Register

// Keep reading