There is a question boards have learned to ask about cloud vendors: is our data segregated from other customers'? The answer, for a decade, has been architecture diagrams showing virtual networks, identity policies, and container boundaries. Executives nodded. The diagrams were true, as far as they went.
They do not go far enough for AI infrastructure, and almost no one outside a narrow circle of ML platform engineers is saying so publicly.
The Assumption Everyone Is Making
Every major AI inference platform — the API called for completions, the "private" model endpoint a vendor sold, the GPU cluster an internal team just stood up — rests on one assumption: that a container boundary, a scheduling namespace, and an access policy are sufficient to keep one customer's workload separated from another's on the same physical GPU.
They are not, and this is not a hypothetical.
The GPU driver — the software layer that actually talks to the chip — does not run inside any customer's container. It runs in the host operating system's kernel, at the highest privilege level on the machine, shared identically by every tenant scheduled onto that node. When a workload calls into the GPU, that call passes through the same kernel code, in the same memory space, as every other customer's workload sitting on the same physical server. There is no wall there. There is a driver, written by one vendor, patched on that vendor's schedule, and trusted implicitly by every cloud platform built on top of it.
Two Ways This Fails — Neither Requires an Attacker
The driver itself gets exploited. The GPU driver is a closed-source binary several million lines long, and it has shipped multiple kernel-level vulnerabilities in recent years — memory-safety bugs that let an ordinary process running inside a customer container escalate to full control of the underlying physical server. Once that happens, every other tenant's data on that machine — model weights, customer prompts, session data — becomes readable. This is not a vendor being careless. It is the mathematics of a shared, closed-source, kernel-privileged codebase serving customers who were never meant to trust each other.
Nobody has to attack anything at all. This is the quieter problem, and the more common one. When a session ends and the GPU releases its memory back to the pool, that memory is not necessarily wiped before the next customer's workload is handed the same physical chips. Zeroing gigabytes of GPU memory on every handoff is expensive, so many platforms skip it as a performance decision made years before "AI infrastructure" was a board-level phrase. The next tenant scheduled onto those same chips can, without writing a single line of exploit code, read what is still sitting there: fragments of proprietary model weights, remnants of the previous customer's prompts. It is not theft in the legal sense of the word. It is leftovers.
A third mechanism sits underneath both. Many providers run a shared GPU scheduling service to squeeze more utilization out of expensive chips, letting several customers' workloads run on the same physical GPU simultaneously. That service is explicitly documented by the underlying hardware vendor as intended for trusted, cooperating jobs — not adversarial multi-tenancy. In that configuration, a crash triggered by one tenant's ordinary, non-malicious workload can take down every other tenant sharing that chip at that moment. No breach. No warning. A business-critical AI system goes dark because an unrelated customer's job hit a routine error three racks over.
Why This Hasn't Reached the Board Yet
This isn't classified. It is documented and discussed openly in ML infrastructure and security-research circles, and it is almost entirely absent from board decks. Three reasons explain the gap:
- It sits below the line procurement teams evaluate. Vendor security questionnaires ask about encryption, access controls, and compliance certifications. They do not ask what happens at the kernel layer when a workload shares silicon with a stranger's.
- It doesn't fit the breach narrative. There is no negligence story here — no unpatched firewall, no leaked storage bucket. It is a structural property of how the industry currently builds shared GPU infrastructure, which makes it harder to litigate and easier to ignore.
- The fix exists, and it costs money. GPU manufacturers now offer hardware-partitioned instances that enforce isolation in silicon rather than software — but partitioning a physical chip into fewer, smaller pieces means fewer customers per GPU, hitting directly at the economics that make shared inference platforms cheap. Adoption is uneven for exactly that reason.
What This Means for Your Organization
If AI workloads — internal infrastructure or a vendor's — run on shared GPU capacity without hardware-level partitioning, three things are true right now, independent of whether anyone has attacked anything:
- Model weights and customer data are exposed to any co-tenant who acquires kernel-level access to the shared server, through a vulnerability class with a recurring, multi-year track record.
- Fragments of an organization's data may already be recoverable by other tenants sharing the same physical GPUs — with no exploit required, simply by allocating memory that was recently released.
- An AI system's availability depends on the code quality of every other tenant sharing its GPU scheduling pool, not solely on the organization's own engineering discipline.
The Questions to Ask This Quarter
Before the next vendor renewal or infrastructure review, put these directly to the team or vendor responsible for AI infrastructure:
- Do our AI workloads run on hardware-partitioned GPU instances, or are we sharing GPUs with other tenants at the software layer alone?
- Is a shared GPU scheduling service enabled in our environment to improve utilization — and do we understand that a co-tenant's crash can take down our inference availability as a result?
- What is our GPU driver patch cadence, and is it tracked separately from the operating system patch cycle? Most organizations do not track it at all.
- Is GPU memory zeroed between different tenants' sessions, or only between our own?
None of these questions have satisfying answers industry-wide yet. That is precisely why they belong on this quarter's risk register rather than next year's.