Demand for private AI is rising as companies weigh frontier model performance against the risk of exposing proprietary data, according to a long-form report by IOSG. Its bottom line is direct: in the domain where a company’s alpha lives, expert-tuned open models can already beat frontier systems on both accuracy and cost, while the infrastructure needed to build and run them in private settings is starting to arrive.
Why private AI is moving up the agenda
IOSG points to a July 1 CNBC interview with Palantir CEO Alex Karp, who said enterprises are paying a token premium to frontier labs while watching their own intellectual property flow toward model providers. In his telling, the transfer happens at the architecture layer because every request sent to a closed model reaches the provider’s servers in plain text.
Days before the interview, Palantir had announced a partnership with NVIDIA to run the open Nemotron model in customer-controlled environments, alongside a nine-point AI sovereignty statement. After the CNBC appearance, PLTR rose 8%.
The report argues that the old cloud trust model is a poor fit for AI. Enterprise software vendors used to see only slices of customer data: Salesforce handled sales channels, Workday saw HR data, Jira covered development cycles, and AWS provided storage and compute. AI workflows now push users to upload broad, cross-functional context all at once in exchange for productivity gains, giving upstream service providers access to much more of a company’s operating knowledge.
That shift has not slowed the largest players. IOSG says Anthropic reached $47 billion in annualized revenue in May, up sharply from $9 billion at the end of 2025. OpenAI, it says, passed 900 million weekly active users in February. Both companies raised fresh funding this spring, with valuations nearing $1 trillion and expectations of eventual public offerings at even higher levels.
Enterprise restrictions came early
Some companies moved quickly once generative AI entered the workplace. IOSG says major Wall Street banks had already restricted ChatGPT by May 2023, less than three months after launch. That same month, Samsung banned generative AI across its network after engineers leaked chip source code into ChatGPT.
OpenAI responded in August 2023 with ChatGPT Enterprise, offering commitments not to train on business data and a zero-data-retention, or ZDR, option. IOSG says ZDR later became a standard requirement in enterprise procurement.
Even so, the contract covers only sanctioned accounts. Citing IBM, the report says shadow AI, employees feeding company data into unsanctioned tools through personal accounts, was involved in one-fifth of data leak incidents by 2025. Heavy shadow AI use added an average of $670,000 to breach costs. In a 2025 survey by security training company Anagram, 40% of employees said they would violate AI-use policies to finish tasks faster.
For consumers, the privacy question sharpened when courts stepped in
IOSG says privacy concerns remained abstract for many users until legal discovery made the stakes more visible. In May 2025, a court order forced OpenAI to retain even deleted consumer chat logs. In November, a judge ordered 20 million of those chats to be produced to lawyers for The New York Times as discovery material.
Criminal cases followed. The report says ChatGPT records entered evidence in the Palisades wildfire arson case, and an affidavit in a Florida double homicide cited a suspect’s prompts about disposing of bodies. Sam Altman also acknowledged in a July 2025 interview that ChatGPT conversations are not protected by legal privilege and that OpenAI could be required to turn them over in litigation.
IOSG cites a Kolmogorov Law survey from October 2025 of 1,000 U.S. AI users. Half did not know AI chats could be subpoenaed, and two-thirds said such conversations should receive protections comparable to speaking with a lawyer or doctor.
How privacy is implemented today
The report divides the market into protocol-level privacy, structure-level privacy, and local inference. The basic question is always the same: where does plain text appear when a prompt leaves a device, moves across the network, reaches a model, and returns as an answer?
Protocol-level privacy relies on promises
At this layer, someone other than the user can still read the prompt. Contract-based zero retention is the standard enterprise version. The provider knows who you are, processes the prompt, and promises not to keep it. Enforcement depends on contracts and reputation.
Anonymous proxies remove identity but not content. The downstream provider still sees plain text and applies its own retention rules. IOSG says Duck.ai negotiates deletion agreements with model providers, while Venice asks users to assume providers may retain everything. In both cases, users cannot verify the promise on their own.
TLS encrypts the pipe, not the endpoint. Oblivious HTTP, or OHTTP, can separate identity from content by splitting what the relay knows and what the destination knows. IOSG notes that OHTTP became an IETF standard in January 2024 and is already used in production traffic through relays rented from Cloudflare and Fastly.
Still, the report says this is roughly the privacy ceiling for access to closed models. The reason is economic. Training a flagship model now costs on the order of billions of dollars, and the near-trillion-dollar valuations of frontier labs rest on exclusive control over weights.
Meta’s LLaMA rollout is used as a case study. IOSG says the weights released to researchers in February 2023 leaked to 4chan in less than a week. A week later, llama.cpp had the smallest 7B model running locally on a MacBook. Three days after that, Stanford fine-tuned the same model into Alpaca for under $600. In July 2023, Meta formally released Llama 2 under a commercial license with a 700 million monthly active user exclusion. Once the weights escaped, the premium escaped too.
Frontier labs could, in theory, offer remote attestation for inference, but IOSG says attestation proves only which code read the prompt, not what happened to the data afterward. To verify that no retention occurred, one would need to audit the serving code and match it to the hardware-reported hash. That would also expose batching, caching, and other operational methods that support margins. Apple and Meta can afford to attest service stacks behind products such as iPhone features and WhatsApp because their economics do not depend on selling model access by token.
Structure-level privacy uses hardware or cryptography
Trusted execution environments, or TEEs, move inference into a hardware enclave. The chip signs an attestation showing which model and which code are running. But the prompt is protected only at the destination. If the request passes through a proxy, there may still be a point in the path where plain text is visible.
End-to-end encryption removes readable intermediaries. The user device encrypts the prompt with the enclave’s key, and every hop in between carries ciphertext that only the enclave can open. Yet trust shifts to the client. The code that encrypts the prompt and verifies attestation could also undermine the guarantee, so verifiable E2EE needs both an attested enclave and open, reproducible client code.
FHE and MPC take a different route by trying to remove the trusted party entirely. A server computes on encrypted data it can never decrypt, or multiple parties hold secret shares of the prompt. The tradeoff is speed. IOSG says encrypted inference still costs 10,000x to 100,000x more than plain-text inference, and small models can take seconds to minutes per token instead of milliseconds. Chips tailored for encrypted computation may narrow the gap, but the first prototype was only demoed in early 2026 and commercial versions remain years away.
Local inference removes the path altogether because the model runs on user-owned hardware. That avoids relays, servers, providers, and the need for verification. The downside is hardware cost and model capability. IOSG says gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index while taking up 65 GB, more than the VRAM of two flagship gaming GPUs combined. A full-precision GLM-5.2, it says, needs an eight-GPU datacenter node, with GPUs alone costing more than $300,000.
The cost of private inference is falling
Even with those constraints, IOSG says the penalty for enclave inference is shrinking fast. In single-GPU benchmarks from Phala, H100 throughput loss in enclave mode averaged under 7% and approached zero on larger models because the bottleneck was moving data into the chip, not computing inside the enclave.
On multi-GPU inference, NVIDIA’s Blackwell architecture supports direct encryption of chip-to-chip traffic. Older H100 systems need to route through the CPU host at one-seventh the bandwidth to reach the same effect. NVIDIA’s own Blackwell benchmarks, as cited by IOSG, showed less than 8% throughput loss for a 397B model in enclave mode.
Pricing is moving too. Azure currently rents a confidential H100 SKU at $8.90 per hour, versus $6.98 without enclave mode, a 27% premium. Phala, which specializes in enclave infrastructure, starts confidential H100 rentals at $3.80 per hour, below Lambda’s standard SXM card range of $3.99 to $4.29.
For managed APIs, IOSG says NEAR AI charges $0.15 per million input tokens and $0.55 per million output tokens for an attested gpt-oss-120b endpoint, roughly in line with plain-text routes at Amazon Bedrock, Together, and Groq. It says NEAR AI matches Fireworks pricing on GLM 5.2 and undercuts it on the larger Kimi K2.6 by 15% on input and 4% on output.
An open-model case where accuracy and cost both improved
A key example in the report comes from Bridgewater’s AIA Labs and Thinking Machines, which published a case study on June 30. Their team fine-tuned Qwen3-235B on Tinker, Thinking Machines’ hosted post-training API. It first used purchased annotations, then sent disagreement samples to investment staff for relabeling.
The training used reinforcement learning with GRPO and three changes: round-robin batching, CISPO loss, and on-policy distillation. The tasks came from day-to-day work done by investment professionals: deciding whether a news item matters to a C-suite investor, whether a central bank document implies a future direction for rates, and where template-style language begins in a document or email.
On an independent test set, frontier models averaged about 50% with simple prompts. With expert prompts, they reached 78.2%, still below the 80% threshold set by the investment team. The fine-tuned Qwen scored 84.7%. By the study’s framing, that meant 29.8% fewer mistakes than the best frontier result and 13.8x lower inference cost.
The linked study appears here: https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
IOSG stresses that the process was not private by default. Bridgewater’s expert annotations still passed through Tinker, a third-party service. The training also ran on rented compute the fund did not control. For buyers who want the recipe without those trust assumptions, choices remain limited.
Private post-training is beginning to appear
That is where attested post-training enters the picture. In March, Workshop Labs and Tinfoil released Silo, a post-training stack that runs inside Tinfoil enclaves on a single eight-GPU node, with keys controlled only by the customer.
IOSG says the cited overhead was 11 extra minutes on a two-hour training run. By freezing base weights and training only small adapters, the stack can fit a trillion-parameter model, Kimi K2 Thinking. The challenge is that reinforcement learning moves data back and forth across components, and data movement is exactly where enclaves still carry cost.
Less than a month after Silo launched, Workshop Labs was acquired by Thinking Machines. IOSG reads that as a sign that many of the parts needed to run the next Bridgewater-style reinforcement learning loop inside an enclave are now under one roof.
The biggest privacy gap sits outside inference
For IOSG, the hard problem is not only the prompt-to-model path. Every external tool call made by an agent opens a route that inference privacy cannot close. The rise of harness engineering, with models wired into tools, memory systems, and data sources, makes this much worse.
A calendar server must read a schedule. A database server must read a query. Even a fully local agent still has to send search terms in plain text if it wants information outside its training data, because a search engine cannot answer a request it cannot read.
Most current answers still rely on protocol-level trust. Companies such as Runlayer and MintMCP run central gateways that scrub personally identifiable information, decide which servers can receive requests, and log destinations and content for audit. Even with SOC 2 audits, the tool servers still need to read the query to respond, and retention depends on each provider’s own terms. The risk multiplies with every tool added to the harness.
Structure-level approaches can improve the middle layer. IOSG points to Phala hosting MCP servers inside TEEs across wallets, code execution, and data sources, allowing users to verify privacy claims via attestation instead of taking the operator’s word for it. But the endpoint problem remains: the final service provider still sees the query in plain text. The enclave protects the messenger, not the destination.
Only a few destinations have learned how to answer without reading, and those are limited to structured queries. Apple’s private information retrieval lets iPhones check incoming numbers against spam databases without exposing the number. Microsoft uses the same idea for passwords in Edge. MongoDB’s Queryable Encryption allows clients to encrypt fields before they leave the device while still supporting equality and range queries on the server.
Open-ended search is much further behind. Brave promises zero retention on its own 40 billion-page index, not Google’s, but that still sits at the promise layer. Exa built a neural index that embeds user queries semantically for ranking, yet the embedding step still starts from plain text on Exa’s servers. MIT’s 2023 Tiptoe paper achieved ranking over 360 million webpages without exposing the query, but server costs were high and ranking quality lagged standard search. Apple’s 2024 Wally paper reduced communication cost by up to 31x by hiding real queries among decoys, but only becomes economical at millions of concurrent queries, a scale no private search system currently reaches.
Private AI is real, but still tiny next to the mainstream market
IOSG offers a few demand signals. Venice AI has passed 3.5 million registered users and 1.3 trillion monthly tokens, then raised a Series A at a $1 billion valuation. Proton, a direct rival, says its Lumo chat product passed 10 million users within a year of launch.
On the infrastructure side, Phala is already processing 2 billion to 3 billion tokens per day on OpenRouter alone, according to the report. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil enclaves, giving users verifiable privacy beyond the proxy layer. IOSG adds that self-hosting is likely the largest private inference channel of all, because models running on local hardware leave no provider-side usage trace.
Still, privacy AI remains small compared with the broader market. IOSG says Google processed 3.2 quadrillion tokens across its products in May. On that basis, Venice’s monthly throughput equals roughly 18 minutes of Google’s token volume.
Google launched Private AI Compute in November 2025, placing some Gemini-powered features into sealed TPU enclaves separated from Google itself and independently audited by NCC Group. But PAC only covers a narrow set of Pixel features such as personalized recommendations and audio summaries, not the Gemini app used by hundreds of millions of people.
What IOSG thinks comes next
The report says private AI is now a real and affordable option, though not a complete one. Consumers can use open-model private chat on Lumo and Venice under no-log promises at no cost, while subscriptions priced at $18 to $20 on Venice or Tinfoil place similar chats inside enclaves, not much different from a ChatGPT subscription.
For enterprise workflows, IOSG says attested endpoints can already be cheaper than plain-text routes. It cites NEAR’s E2EE API as an example where encrypted context can reach enclaves and support memory, file uploads, and custom instructions.
On post-training, NVIDIA’s upcoming Vera Rubin NVL72 is expected to extend confidential computing from Blackwell’s eight-GPU nodes to 72-GPU racks, making frontier-scale reinforcement learning loops more workable without exposing intellectual property.
Yet the most defensible value, in IOSG’s view, lies outside the layers where pricing is already compressing. Enclave operators control a switch on standard chips, not necessarily a moat. Gateway businesses at the protocol layer are competing in territory that looks a lot like traditional middleware. The harder, more defensible frontier is the part the market still has not solved: training loops sealed inside enclaves, tool calls closed end to end, and search indexes that can answer without seeing the terms.
IOSG ends with a practical split. For execution-heavy, agent-heavy workloads, choose trust for now, because each tool call often sends plain text to destinations an enclave cannot hide and because frontier models still justify their premium in those loops. For high-level judgment that distinguishes one company from another, strategy, planning, and decisions shaped by years of experience, choose verification. That is where the alpha sits, and where expert-tuned open models are already starting to win.