Trusted Execution Environments for AI Inference Pipelines

Trusted Execution Environments do not just scramble data as it moves between a user and a server. They build a hardware-enforced boundary around the entire inference process, so that model weights, prompts, and generated outputs stay unreadable to the host operating system, the hypervisor, and the cloud provider running the physical machine. This piece maps that boundary stage by stage, from the moment a request leaves a user's device to the moment a verifiable result comes back, and shows what the current hardware landscape makes possible for enterprises that need an AI vendor to have no access to their internal prompts and data.

What TEEs do inside an AI inference pipeline

A Trusted Execution Environment is a region of a processor, and increasingly a GPU or NPU, that isolates code and data from everything else running on that machine, including privileged software like the operating system kernel. That isolation matters for AI inference because a typical cloud deployment hands enormous trust to the infrastructure provider by default. A TEE changes it by carving out a protected memory region that the host cannot inspect, so the actual computation, tokenizing a prompt, running a forward pass, generating tokens, happens somewhere the operator of the machine cannot see.

Remote attestation is what separates a TEE from an encryption scheme applied at the edges of a system. Every section that follows traces a different stage of the inference pipeline, the request, the forward pass, the output, the attestation receipt, the agentic chain, and the performance trade-offs that come with covering all of it. The phrase "end-to-end" is doing real work here, because a TEE that protects only one stage of that chain leaves the others exposed, and the rest of this piece is about what covering the whole chain actually requires.

How the hardware landscape currently supports confidential inference

Confidential inference today runs on a combination of CPU-side and GPU-side TEEs, and neither alone covers a modern inference pipeline. A pipeline with only one or the other has a gap: a CPU-protected request that is decrypted outside the GPU's protected memory before inference, or a GPU-protected forward pass fed by a prompt that sat in plaintext on an unprotected CPU host, both leave an opening.

On the CPU side, Intel TDX, Trusted Domain Extensions, protects entire virtual machines rather than the small, tightly bounded enclaves that Intel's earlier SGX technology used. That distinction matters for AI workloads specifically, because SGX's constrained memory model was built for small, security-critical code paths, not for the memory footprint of a large language model's runtime. ARM TrustZone takes a different approach, partitioning a processor into a secure world and a non-secure world, which is particularly relevant for edge inference scenarios where a device needs some guarantee of isolation without the resources of a datacenter-class chip.

On the GPU and accelerator side, the pattern established by NVIDIA's confidential computing line has started to generalize. Ascend-CC demonstrates the same confidential compute pattern implemented on Huawei's Ascend NPU architecture, which signals that hardware-enforced isolation for AI inference is becoming a feature of accelerator design generally, not a property unique to one vendor's silicon. Edge-native AI computing adds its own constraints on top of this, since an edge device balancing TEE security properties against strict power and latency budgets does not have the same headroom a datacenter GPU does, so the security guarantees and the performance trade-offs look different depending on where the inference runs.

For enterprises that do not want to build attestation infrastructure themselves, three managed cloud paths currently stand out as the dominant options: Azure Confidential VMs with H100 GPUs, Google Cloud Confidential VMs, and AWS Nitro Enclaves. Ask specifically whether the GPU, not just the CPU, sits inside the attested boundary when comparing enterprise AI platforms on this criterion, because that is where model weights and the forward pass actually live.

Protecting the request: what happens before the model sees a token

The first point at which a user's data is exposed is also the simplest to overlook: the moment a prompt leaves the user's device. Standard TLS handles part of this problem by encrypting data in transit, but TLS terminates at the server boundary. Once a connection is decrypted at that boundary, the plaintext prompt becomes visible to the host operating system and to any co-tenant process sharing that machine, unless a TEE takes custody of the data immediately upon arrival. This is the exact point where "encrypted in transit" and "protected end-to-end" stop meaning the same thing. A TEE-rooted secure channel is built so that decryption happens only inside the enclave, never in host memory, which keeps the infrastructure provider from ever seeing the plaintext prompt even for the brief window after a TLS connection ends. That principle, that a user's data should remain the user's own throughout the entire request lifecycle instead of becoming momentarily visible to whoever operates the server, is the architectural stance behind any AI system built to be privacy-preserving from the ground up.

Tokenization and pre-processing, splitting a prompt into tokens, applying a system prompt, assembling the context window, need the same protection, because these steps produce structured representations of a user's input that are arguably more useful to an attacker than the raw text. Research on guardrail verification treats system prompts themselves as sensitive intellectual property, since developers build agent behavior around carefully engineered system prompts that they cannot be expected to publish or expose. Proving that a guardrail executed correctly therefore requires a mechanism that does not force the developer to reveal the underlying agent configuration. That need for a proof mechanism that does not leak the thing it is proving something about becomes central again later in this piece, when attestation comes up as the tool for proving a pipeline ran correctly without exposing what ran.

Protecting model weights during the forward pass

The forward pass, the actual computation that turns tokens into a generated response, is where a second category of sensitive material enters the picture: the model's weights. Weights loaded into unprotected GPU memory can be read by any process with host-level access. The same machine that exposes a user's prompt can just as easily expose a company's proprietary model. That makes the forward pass a stage where user data and vendor intellectual property are at risk simultaneously, so a TEE covering this stage has to protect both at once.

In a TEE-backed inference setup, weights are stored in encrypted form and decrypted only within the protected memory region the TEE controls. Ascend-CC shows what enforcing that boundary looks like on non-NVIDIA hardware: the NPU removes DMA mappings from the host's address space before decryption begins, so weight decryption and tensor operations happen only after the host has been architecturally cut off from reaching that memory. That sequencing, unmapping before decrypting, is a genuine design trade-off. Choosing to unmap first costs a small amount of setup overhead on every inference call in exchange for closing that window.

A second, less obvious target lives alongside the weights: the KV cache, the key-value store that attention mechanisms use to avoid recomputing earlier parts of a conversation on every new turn. The KV cache accumulates across a multi-turn conversation and effectively encodes everything that happened in every prior turn. The throughline across this section is that weight protection and user data protection are not two separate problems that happen to share hardware. They are coupled: a TEE that decrypts weights in protected memory but lets the KV cache sit exposed has only solved half of what a single conversation exposes, and the performance cost of closing that gap fully is part of what the later section on overhead has to account for.

Protecting outputs and the attestation receipt that proves the pipeline ran correctly

Output generation closes the loop on the inference request, and it carries its own version of the exposure problem that opened this walk-through. Generated tokens have to be encrypted inside the TEE before they ever leave protected memory. A system that lets tokens exit the enclave in plaintext and only encrypts them afterward, at the network layer, has created a gap during which the host operating system can read the output, even if every earlier stage was handled correctly. Closing that gap is what keeps the whole chain honest: a pipeline is only as protected as its least-covered stage, and an unencrypted handoff at the exit is as serious a gap as an unencrypted handoff at the entrance.

Proving that this entire chain actually happened as described is a separate problem from making it happen, and it is where remote attestation does its most distinctive work. Phala Network's production implementation shows what this looks like as a concrete, checkable artifact: every inference response carries an x-receipt-id header, a receipt identifier for a gateway-signed record covering that response and chaining back to the TDX hardware root, along with x-aci-identity and x-aci-keyset-digest headers that bind the response to the attested gateway workload and its keyset, with the compose-hash verified on-chain. A caller can take those headers and independently verify the attestation without taking the infrastructure provider's word for any part of it.

Attestation extends naturally to proving more than just "a model ran." Attestation closes that opening by making enforcement something a user can check.

Where the root of trust for attestation actually sits is itself a design choice: Phala's DstackApp.sol contract anchors attestation on-chain, moving the root of trust from a hardware vendor's certificate authority to smart contract logic that governs enclave policy. That is a genuine alternative to vendor-rooted attestation, not a strictly superior replacement for it: anchoring trust on-chain trades reliance on a chip vendor's certificate authority for reliance on the smart contract's own correctness and governance, a trade-off this piece returns to in the discussion of side channels, where the limits of any attestation model come into sharper focus. Taken together, this is the mechanism behind AI assistants built so a service operator cannot access user prompts by design: a user is not asked to trust that a provider respects their privacy, they are given a signed, checkable record that the specific model and code they expected actually ran inside the attested boundary.

How agentic pipelines multiply the exposure surface TEEs must cover

Everything described so far assumes a single request and a single model call. An AI agent that calls external tools, retrieves documents from a knowledge base, and chains several model calls together turns that single hop into a sequence of hops, and every hop is a place where the trust boundary can break if attestation is not maintained continuously from one stage to the next. A system that attests the first model call in a chain but hands off to an un-attested tool-calling step in the middle has not actually protected the pipeline, it has protected one stage of it and left the rest to assumption.

That multiplication of stages also multiplies what an attacker can learn without ever breaking encryption directly. The deeper point this section sets up is that agentic pipelines do not just add more stages to secure, they change what "secure" has to mean at each stage, because access patterns and retrieval behavior become information-bearing in a way a single, stateless inference call is not. That is why privacy by construction has to be a system-design requirement built in from the first architectural decision, not a control bolted onto a finished pipeline.

The performance cost of confidential inference

None of the protection described above is free, and the honest version of this argument has to say so. The overhead confidential inference introduces is measurable and highly dependent on configuration, to the point that two accurate benchmark results can point in opposite directions depending on what was actually tested.

Early benchmarks run on a single GPU, on large models, at low concurrency, found overhead well under ten percent. The same study found that under closed-loop concurrency testing, throughput gaps held in the 11.5 to 20.2% range, and the larger model reached its throughput ceiling at lower concurrency under confidential mode than it did without it, while Mistral-7B showed no such earlier ceiling.

Multi-GPU deployment adds a structural constraint on top of these numbers. Confidential H100 instances do not support GPUDirect RDMA, and in single-GPU passthrough mode, all CPU-to-GPU data moves through encrypted bounce buffers that cap the available bandwidth between the two. Overhead has to be measured per model, per concurrency level, and per hardware configuration, not extrapolated from whichever benchmark looks most favorable. A team sizing capacity on the assumption that confidential inference costs under ten percent, because that is the number from an early single-GPU test, is very likely planning around the wrong configuration for its own workload.

The side-channel problem that TEE confidentiality does not automatically solve

Even a fully attested, fully encrypted TEE pipeline does not close every avenue an attacker might use, because side-channel attacks target the boundary from outside. None of those signals requires breaking the cryptography a TEE relies on, which makes this class of attack a genuine limitation that remains open.

The access-pattern leakage described in the agentic pipelines section is one instance of this broader category applied specifically to retrieval. Side-channel resistance is a separate, ongoing research problem layered on top of that guarantee, and any enterprise evaluating a zero-trust AI architecture should ask a vendor specifically how it approaches side-channel risk, rather than assuming that attestation and encryption alone have closed every gap. Confidant AI, a private AI assistant built so the service operator itself cannot reach user prompts, is one example of a system built on this architectural premise from the ground up.