Cloud hosting provider Railway recently disclosed a major security breach that allowed unauthorized actors to exploit a zero-day vulnerability and rack up roughly $740,000 in fraudulent AI charges. The sophisticated intrusion began on October 20 at approximately 07:28 UTC and lasted for 26 intense hours before engineers successfully contained the threat on October 21 at 09:24 UTC. Threat actors targeted the platform's build infrastructure to steal customer AI API keys and cloud credentials.
Railway operates as a cloud hosting company designed for developers to build, deploy, and run applications without managing underlying hardware. Its services include running websites, hosting databases like PostgreSQL and MySQL, and connecting GitHub repositories for automated deployments. Many developers on the platform store API keys as environment variables to allow their applications to communicate smoothly with paid AI services. According to the company's incident report, attackers used a zero-day vulnerability in the build infrastructure to partially escape filesystem isolation.
Key takeaways
- Attackers exploited a zero-day vulnerability in Railway's build infrastructure to steal customer AI API keys.
- The unauthorized AI usage racked up roughly $740,000 in charges for coding agents and AI training pipelines.
- Railway spent 26 hours battling automated attackers before hardening infrastructure and rotating credentials.
- Up to 100,000 customer workspaces may have had their environment variables decrypted during the breach.
By bypassing filesystem isolation, the threat actors read files stored on host virtual machines and acquired credentials used by build servers to authenticate with internal systems. While initial attempts to use these credentials across various systems failed, the attackers eventually gained access to Temporal, the internal task management system. Inside Temporal, they discovered a copy of Railway's internal codebase containing a misconfigured authentication mechanism, which provided temporary cloud credentials for deeper network access.
Armed with these temporary credentials, the intruders accessed a production database containing encrypted customer environment variables and decryption information. The primary objective appeared to be harvesting API keys from businesses utilizing paid AI models. These stolen keys facilitated roughly $740,000 in unauthorized usage, supporting large-scale model training, coding agents, and AI pipelines. Analysts suggest the stolen resources likely supported model distillation, a process where a smaller AI model learns by querying a larger, more powerful model.
During the 26-hour ordeal, Railway engineers worked in shifts while facing rapid, parallel probing from automated attackers. The situation evolved into a high-speed contest of agent capability, with Railway defending against automated systems while simultaneously hardening its platform. Railway estimates that environment variables for up to 100,000 customer workspaces could have been decrypted, though this represents an upper limit rather than confirmed workspace compromises.
In response to the incident, Railway closed the exploited vulnerabilities, rebuilt the affected infrastructure, and rotated all compromised credentials. The company is actively collaborating with AI providers to secure refunds for fraudulent charges and has promised to cover any financial losses that the providers refuse to reimburse.
Comments