What the LiteLLM Incident Revealed About AI Pipeline Risk
April 9, 2026
The Perfect Storm for Edge AI – And the Question Nobody Is Asking
May 11, 2026

Why Standard Security Fails at the Critical Moment: Encrypting AI “In Use”

By Luigi Caramico, Founder and CTO, DataKrypto

 

Why Standard Security Fails at the Critical Moment: Encrypting AI “In Use” highlights risks of unprotected AI processing and the need for in-use encryption.

Traditional encryption methods were designed years before today’s AI and high-speed, data-driven computation became central to business operations. While encryption like AES and TLS excel at protecting data at rest and in motion, these methods were not built to safeguard data in use, especially data actively processed by AI systems.

In order to achieve truly secure and compliant AI– also known as Confidential AI – organizations must have protection extending beyond the traditional “before and after.” They need technology  that keeps data secure even while it’s being computed on. Without it, the potential for catastrophic data leakage is enormous.

The Memory Exposure Problem

In a typical AI workload, data remains in memory at all times. For performance reasons, this data is not encrypted, leaving it vulnerable. While the data is stored in the Graphics Processing Unit’s (GPU’s) memory — basically the entire time the AI system is running—it stays in plaintext, creating an opportunity for attackers to steal AI models, prompts, related documents, as well as any AI responses.

This memory exposure problem runs much deeper than most executives realize. During AI processing, plaintext data and model weights are loaded into the system’s Random Access Memory (RAM) or Video Random Access Memory (VRAM) on a GPU for computation.

Most AI frameworks keep this data residing in memory long after inference completes in order to maximize performance and efficiency. This shortcut comes with a heavy security cost, leaving areas of critical exposure in two ways:

    • Lurking Data: Prompts, embeddings, and intermediate activations can linger in GPU VRAM for hours or even days. An attacker who gains control can dump memory (copy the contents of volatile memory to persistent storage) and recover sensitive data long after the transaction is complete. Memory scraping doesn’t have to be real-time; historical secrets are still there for the taking.

    • Interconnect Backdoors: In multi-GPU and distributed inference setups, most AI frameworks still do not encrypt data as it moves between GPUs on the same server or across nodes. High-speed interconnects like NVLink and PCIe transfer raw, unencrypted tensors between devices for maximum throughput.

Traditional encryption only protects data before and after computation. During inference, when the data is actually being used, it remains dangerously exposed.  In most cloud or multi-tenant environments, system administrators, cloud operators, and third-party service engineers have access to bus traffic or driver-level hooks. A single compromised account at that level could capture model weights or input data in cleartext as it flows through the cluster.

Cryptographic Guarantees for AI: The New Standard

To close this critical vulnerability, protection must extend into the runtime environment so sensitive data and models never appear in plaintext at any stage. This requirement is driving the development and adoption of advanced cryptographic frameworks, widely referred to as Confidential AI.

Confidential AI leverages technologies like Fully Homomorphic Encryption (FHE) and Trusted Execution Environments (TEEs) to achieve this “in-use” security:

    • FHE for Computation on Encrypted Data: FHE is a groundbreaking technique that enables computation on encrypted data. The data never needs to be decrypted into plaintext, even when running through the GPU. This means that even if the entire machine is compromised, the attacker only sees encrypted noise.

    • TEEs for Key and Code Isolation: TEEs (hardware-enforced secure enclaves) are used to isolate the core encryption keys and model execution logic from the underlying operating system and hypervisor. This prevents cloud operators or malicious system administrators from accessing the environment’s most critical secrets.

By implementing these technologies, organizations achieve true encryption of AI in use:

    • Model and Inputs Remain Encrypted: All proprietary model weights, user inputs, and intermediate calculations remain in ciphertext while residing in memory.

    • Key Isolation: Encryption keys are never exposed in the general operating system space.

    • End-to-End Protection: In multi-GPU setups, data moving between processors remains encrypted, eliminating the interconnect as a viable attack surface.

Confidential AI For Business Compliance 

The demand for “in-use” security is a strategic necessity for compliance and preserving enterprise value. Consider the following:

    • Compliance: Regulations such as GDPR, HIPAA, and CCPA require organizations to protect data throughout its lifecycle, including processing. Without cryptographic guarantees during inference, compliance is fundamentally at risk.

    • Intellectual Property (IP) Protection: For businesses whose primary value lies in their proprietary models (or the data used to fine-tune them), in-use encryption prevents model inversion and theft by insiders or malicious partners in multi-tenant cloud environments.

    • Market Confidence: Early adopters of robust Confidential AI frameworks gain a competitive edge, demonstrating to customers and partners that they are willing to apply the highest cryptographic standards to sensitive data, wherever it resides.

“In use” encryption is the only way to deploy AI in the cloud, across borders – confidently, and in multi-tenant settings. This approach gives you peace of mind that – your secrets stay secret, always. It is not an optional extra; it is the logical next step in AI security.

 

Read original article.

✕