Railway's Breach Shouldn't Have Been Possible

Railway's November 10 incident report describes an attacker stealing customer secrets, including API keys for AI services such as OpenRouter, Anthropic, and OpenAI. We believe standard security controls should have prevented that access. This article explains where the architecture fell short and what customers should expect instead.

We consider this a serious, preventable architectural and leadership failure. A hosting provider is responsible for keeping one customer's code from reaching another customer's secrets. Its leadership must make that protection a requirement before putting customer data at risk.

The initial exploit involved a complex chain of AI-assisted probes and may have been novel. That does not excuse the missing protections that let the attacker go further. Isolating customer workloads and limiting credentials to the access they need are well-known security fundamentals. Railway should have known better.

When our team built systems at AWS, isolation through virtual machines (VMs) and tightly limited credentials were foundational requirements for running untrusted customer code. We apply those principles at Forte too. The architecture Railway describes falls short of that standard.

Being a customer does not make someone trustworthy

Renting someone a safe-deposit box doesn't mean you should leave them alone in the vault with everyone else's boxes unlocked. Paying for access to their own box gives them no right to open anyone else's.

The same principle applies to application hosting. Customers can submit code, install dependencies, run build scripts, and start processes because those are the capabilities the platform sells. An attacker can sign up and use those capabilities to attack it.

Security reviews need to start with that possibility. Assume customer code will look for credentials, try to escape its container, and probe any service it can reach. Then ask what stops it from reaching another customer's data if one of those attempts succeeds.

How a customer build led to other customers' secrets

According to Railway's attack timeline, the attacker partially escaped the filesystem restrictions around a build running in a shared VM. A stolen build credential gave them access to build histories and internal source code. The build histories exposed customer environment variables.

The internal source code helped the attacker find a separate misconfiguration in the rules for issuing cloud credentials. That route eventually led to a machine used to manage workloads, where they gained access to encrypted customer secrets and permission to decrypt them.

The diagram summarizes those two routes to customer data. It shows the reported attack sequence rather than Railway's entire infrastructure.

There were several opportunities to stop this attack. The build needed stronger isolation, and the systems it could reach needed to enforce tighter permissions.

Customer builds should never share a VM

Keeping each customer's files in a separate directory does not provide enough protection when those customers can run malicious code. Ordinary Linux containers also share the host's kernel, the core of its operating system. If an attacker gets past the restrictions around their container, other workloads on that host can be at risk.

This is why we require separate VMs for customer workloads that cannot trust each other. Each VM has its own kernel, and hardware virtualization adds another boundary an attacker would have to cross. Customers can still share physical hardware, but putting their containers inside one VM provides no VM boundary between them.

AWS makes this distinction in its container isolation guidance. Fargate tasks run in hardware-virtualized environments and MicroVMs. For containers sharing an EC2 host, AWS even explicitly warns that credentials and temporary files may be accessible to other containers on that host.

Builds need this protection just as much as running applications do. Installing a dependency or running a build script executes customer-controlled code. Treating builds as "less dangerous" leaves a gap at the very point where the platform first runs that code.

A separate VM helps contain an escape from a build container. It cannot stop an attacker from using an overly powerful credential available inside that VM, which is why permissions need their own review.

A stolen build credential should not open the rest of the platform

Least privilege has a straightforward meaning: give a job only the access it needs to do its work. For example, a build that uploads a container image needs permission to upload that image. It should not be able to read another customer's build history or the platform's internal source code.

Each service that accepts a credential must check what it allows. Recognizing a valid credential is only the first step. The service must also check whether that credential permits this action on this customer's resource. A request coming from inside the platform's network does not make those checks optional.

A security review should examine each credential that customer code could reach. If it were stolen, what could an attacker read or change? Could that access help them obtain more powerful credentials? The review needs to follow those connections and check where access would actually be denied.

Credentials that expire quickly still need limited permissions. Expiration reduces the time an attacker has to use a stolen credential; it does not limit what they can do during that time. The rules for issuing credentials must also prevent an attacker from obtaining fresh ones. AWS's IAM guidance recommends both temporary credentials and least-privilege permissions.

In our view, allowing a build credential to expose unrelated customers' data is inexcusable. A platform running untrusted code has to plan for credential theft and limit the damage it can cause.

Encryption cannot help if the attacker can also decrypt the data

Encrypting stored secrets is necessary, but the permissions around decryption matter just as much. If an attacker can read the encrypted data and ask the encryption service to decrypt it, they can still get the original secrets. They do not have to break the encryption.

The platform needs to control who can read the stored data and who can decrypt it. Those checks must also limit which customers' secrets each identity can access. If taking over one machine gives an attacker both capabilities across unrelated customers, encryption alone will not contain the breach. AWS's KMS access-control documentation explains how permissions govern access to encryption keys.

Next week's infrastructure cannot protect today's customers

Railway says it had planned to begin rolling out stronger build infrastructure during the week of October 26. Its incident report now describes dedicated, short-lived build VMs and more restricted credentials for the registry that stores container images.

Those changes address real problems. But the planned rollout is itself alarming: it suggests Railway had already identified a need to replace this design while customers were still relying on it to protect their secrets. Having a fix on the calendar is no excuse for the architecture that exposed them.

If a platform cannot safely isolate a workload, it needs to restrict how that workload runs until it can. Leaving customers exposed while basic protections wait for a rollout is an unacceptable leadership decision.

Critical vulnerabilities and architectural deficiencies ought to be foundational, launch-blocking requirements, not backlog items to be fixed later.

These protections should have been in place before the breach

Railway deserves credit for publishing a detailed report, refunding customers for AI token spend incurred due to leaked credentials, and migrating to VM level isolation. That transparency made this analysis possible. However, it does not excuse the architectural failures the report describes.

The standard customers should expect is straightforward: their workloads and secrets should be secure and isolated from other (potentially malicious) customers.

That standard applies to Forte too. Our security controls and responsibilities are a starting point for reviewing how we run customer code and protect customer data. We hold ourselves to the same standard we are asking Railway to meet, and we expect customers to hold us accountable for that.