• 2 min read
Expired card took AWS-hosted client sites offline
A web firm lost access to client sites and email after an AWS payment card expired, alerts hit spam, and MFA recovery failed.
Image: The Register
A missed AWS billing alert was enough to take Christopher Bradbury’s client websites and hosted email offline last week, after an expired payment card triggered trouble on an account tied to Route 53.
Bradbury, who runs web design and development firm Digital Takumi, told The Register he discovered on Thursday, July 16 that all websites and Google Workspace email accounts connected to domains he manages were down. The domains were hosted through Route 53, and none of them were resolving.
The cause turned out to be simple at first: AWS had sent multiple warnings that the payment card on file had expired and that the account would be suspended unless action was taken. But those messages landed in a spam folder, and some were sent to an employee who had already left the business.
Recommended reading
OVH mass-rebooted hosts to patch Januscape bug
“AWS had been sending billing notifications, but unfortunately they had been filtered into a spam folder and, in some cases, were being delivered to an employee who had since left the business.”
Bradbury said he accepted responsibility, but the billing problem quickly exposed deeper issues. The account’s root login used MFA through a software authenticator stored on an old laptop that had since suffered a motherboard failure. Instead of fixing that, he had been relying on recovery codes sent by email.
That became another dead end because the registered root email address was on one of the domains hosted in the now-suspended AWS account. With DNS down, he could not receive the verification email needed to recover access.
How the account recovery failed
Bradbury then tried contacting AWS from a different email address and even opened a separate AWS account to buy business support. According to him, support staff still would not discuss the original account until he could verify ownership.
“Over the following days I spoke with several AWS teams, including Billing and Account Recovery. I was transferred between teams multiple times, but nobody was able to complete the ownership verification or restore access to the account.”
While The Register was working on the story, Bradbury said the sites were restored. He logged back in, paid the overdue invoices, updated the payment method, reset his MFA keys, and fixed the recovery setup.
His advice is blunt:
- Pay your AWS bills
- Keep the recovery email off the same domain managed by that AWS account
- Stop taking MFA shortcuts
As Bradbury put it, this was not a major infrastructure deployment, just “a single company marketing website and some domains through Route 53.” Even so, it was business-critical enough to bring everything to a halt.
Enterprise Editor
Marcus follows the money. He covers enterprise software, cloud architecture, and the tectonic shifts in Big Tech strategy. He translates dense earnings calls and complex M&A activity into actionable insights about where the industry is actually heading. If a tech giant makes a silent pivot, Marcus is usually the first to notice.
via The Register
