Private AI is becoming a real deployment option rather than a theoretical preference, according to a new IOSG report. Its main claim is blunt: in the areas where a company’s alpha actually lives, expert-tuned open models can already beat frontier closed models on both accuracy and cost, and the infrastructure needed to build and run those systems in privacy-preserving environments is now coming together piece by piece.
Why private AI has become a board-level issue
IOSG points to a July 1 CNBC interview with Palantir CEO Alex Karp, who said companies are paying a token premium to frontier labs while letting their own intellectual property flow to model providers. Karp described that as a transfer of alpha happening at the architecture layer: every request sent to a closed model arrives at the provider’s servers in plaintext.
Days before the interview, Palantir announced a partnership with NVIDIA to run the open Nemotron model inside customer-controlled environments and published a nine-point AI sovereignty declaration. After the CNBC appearance, PLTR rose 8%.
The report contrasts today’s AI stack with the trust model enterprises accepted during the SaaS era. Salesforce saw sales channels, Workday saw HR data, Jira saw development cycles, and AWS provided storage and compute. AI workflows ask for much more. They often require users to upload broad sets of documents and structured context across departments, giving upstream providers a far more complete view of internal operations.
That shift is happening even as privacy and IP criticism has done little to slow the top labs. IOSG says Anthropic’s annualized revenue reached $47 billion in May, up from $9 billion at the end of 2025. OpenAI passed 900 million weekly active users in February. Both companies raised fresh funding this spring, with valuations nearing $1 trillion and expectations of even higher IPO valuations.
Large organizations moved early, but contracts only go so far
The report notes that major Wall Street banks had already restricted ChatGPT by May 2023, less than three months after its launch. Samsung banned generative AI across the company in May 2023 after engineers leaked chip source code into ChatGPT. OpenAI responded in August that year with ChatGPT Enterprise, promising not to train on business data and offering zero-data-retention, or ZDR, arrangements that later became a standard procurement request.
Still, contracts only cover approved corporate accounts. IOSG cites IBM data showing that by 2025, shadow AI, defined as employees feeding company data into unapproved AI tools through personal accounts, was involved in one-fifth of data leakage incidents. Heavy shadow AI use added an average of $670,000 to breach costs. In a 2025 survey from security training firm Anagram, 40% of employees said they would violate AI-use policies to finish work faster.
Consumer privacy concerns changed once courts got involved
For ordinary users, IOSG says the debate over private AI remained abstract until legal process started touching chat records directly. In May 2025, a court order forced OpenAI to retain deleted consumer chat logs. In November, a judge ordered 20 million of those chats to be produced to lawyers for The New York Times in discovery.
Criminal cases have also started to pull chat history into evidence. The report says ChatGPT records were entered in the Palisades fire arson case, and an affidavit in a Florida double-homicide case cited a suspect’s questions about how to dispose of bodies. Sam Altman said in a July 2025 interview that ChatGPT conversations are not protected by legal privilege and that OpenAI “may be required to produce” user chats in litigation.
IOSG cites a Kolmogorov Law survey from October 2025 covering 1,000 U.S. AI users. Half did not know those conversations could be subpoenaed, while two-thirds said the chats should receive protections comparable to those covering lawyers or doctors.
What private AI looks like today
The report divides the market into protocol-level privacy and structure-level privacy. Both address the same path: a prompt leaves a device, crosses the network, reaches the machine running the model, and returns a response. The difference is where plaintext exists on that path, who can read it, and whether privacy claims can be verified.
Protocol-level privacy
At this layer, someone other than the user can still read the plaintext prompt, and what happens next depends on policy and promises.
- Contractual zero retention is the enterprise version. The provider knows who you are, processes the prompt, and promises not to keep it.
- Anonymous relays remove identity but do not encrypt message content. The downstream provider still sees plaintext. IOSG points to Duck.ai, which negotiates deletion arrangements with model vendors, and Venice, which tells users to assume providers may retain everything. Neither path is verifiable.
- TLS encrypts the pipe, not the endpoint. The receiver can still read the full request. Relays often use Oblivious HTTP, RFC 9458, to split knowledge of sender identity from access to content. IOSG says OHTTP became an IETF standard in January 2024 and is already used in production traffic through relays rented from Cloudflare and Fastly.
In IOSG’s view, this is the ceiling for privacy when using closed models. The economics are simple. Training a flagship model now costs on the order of billions of dollars, and valuations approaching $1 trillion depend on exclusive control over model weights. As long as capability gaps support premium pricing, labs have every reason to protect those weights aggressively.
The report uses Meta as the key example. LLaMA launched in February 2023 with access limited to researchers. Less than a week later, the weights leaked to 4chan. A week after that, llama.cpp made the smallest 7B model usable on a MacBook, and three days later Stanford used the same model to fine-tune Alpaca for under $600. In July 2023, Meta officially released Llama 2 under a commercial license with a 700 million monthly active user exclusion. Once the weights escaped, IOSG argues, so did the premium.
Frontier labs could theoretically offer remote attestation for closed-model inference, but attestation can only show which code read the prompt. It cannot prove what that code did with the data. To verify retention practices, a provider would need to expose serving code and tie it to the hardware-reported hash. IOSG argues that would also reveal the batching and caching techniques that support margins across model generations.
Structure-level privacy
These approaches replace trust with hardware, cryptographic, or physical guarantees, though the report says they currently work mainly with open models.
Trusted execution environments, or TEEs, move inference into hardware enclaves. The chip signs an attestation stating which model and which code actually ran. Plaintext is still exposed at the endpoint, and if traffic passes through an agent or relay, that path may still contain a readable hop unless other measures are added.
End-to-end encryption closes off readable intermediaries. A user device encrypts the prompt to the enclave’s key, and every hop in between carries sealed ciphertext. But the trust burden shifts to the client. The code that encrypts prompts and verifies attestation can also undermine those guarantees, so verifiable E2EE requires not just a provable enclave but also open, reproducible client code. The tradeoff is engineering complexity. Features that rely on reading plaintext need to be rebuilt around client keys or inside the enclave.
Fully homomorphic encryption and MPC variants try to remove the trusted party entirely. Servers compute on ciphertext they can never unlock, or split prompts into secret shares across multiple parties. The bottleneck is speed. IOSG says encrypted inference costs run 10,000x to 100,000x above plaintext, and even small models can take seconds to minutes per token instead of milliseconds.
Local inference removes the path altogether. The model runs on the user’s own hardware, with no relay, no server, no service provider, and no attestation requirement. The downside is obvious: cost and capability. The report says gpt-oss-120b scores roughly half of GLM-5.2 on the Artificial Analysis index while taking up 65 GB, more than the combined VRAM of two flagship gaming GPUs. Full-precision GLM-5.2 requires an eight-GPU datacenter node, with GPUs alone costing more than $300,000.
Performance and pricing are moving in the right direction
Beyond those structural limits, IOSG says enclave inference is getting cheaper to run. In single-GPU setups, Phala benchmarked an average throughput loss of less than 7% on H100s in enclave mode, with the penalty nearing zero on larger models because the main cost comes from moving data into the chip rather than computing inside it.
For multi-GPU setups, NVIDIA’s Blackwell generation supports direct encryption of chip-to-chip traffic. Older H100 systems can reach a similar result only by routing through the CPU host at one-seventh the bandwidth. NVIDIA’s own Blackwell benchmark showed less than 8% throughput loss for a 397B model in enclave mode. IOSG’s conclusion is that privacy inference overhead is no longer the deciding constraint it once was.
Pricing is also falling. Every H100 shipped after 2023 includes enclave mode, so the added cost comes from throughput loss rather than extra silicon. On Azure, a confidential H100 SKU rents for $8.90 per hour versus $6.98 without enclave mode, a 27% premium. On providers built around enclave infrastructure, the pricing looks different. IOSG says confidential H100 instances on Phala start at $3.80 per hour, below Lambda’s normal SXM range of $3.99 to $4.29.
On hosted APIs, NEAR AI prices attested endpoints for gpt-oss-120b at $0.15 per million input tokens and $0.55 per million output tokens, which IOSG says is in line with plaintext offerings from Amazon Bedrock, Together, and Groq. For multi-chip models, the report says NEAR AI matches Fireworks on GLM 5.2 and undercuts it on the larger Kimi K2.6 by 15% on input and 4% on output.
How open models can win where alpha matters
IOSG does not deny that frontier closed models still lead top open models in general capability. What it argues is narrower and more practical: on high-value, domain-specific tasks, targeted expert tuning can overturn that ranking.
The report highlights a June 30 case from Bridgewater’s AIA Labs and Thinking Machines. Their result: an expert-labeled open model outperformed frontier models on both accuracy and inference cost.
The team fine-tuned Qwen3-235B on Tinker, Thinking Machines’ hosted fine-tuning API. It bought initial labels from vendors, trained on that set, then sent disagreement samples to the firm’s own investment professionals for relabeling. The training used GRPO reinforcement learning with three modifications: round-robin batching, CISPO loss, and on-policy distillation.
The tasks came straight from daily investor workflows: whether a news story matters to a C-suite investment professional, whether a central bank document signals the direction of future rate changes, and where templated language begins in a document or email. Scores came from an independent test set. Frontier models averaged roughly 50% under simple prompts and reached only 78.2% with expert prompts, below the investors’ 80% threshold. The fine-tuned Qwen hit 84.7%. By the paper’s own framing, that meant 29.8% fewer errors than the best frontier setup and 13.8x lower inference cost.
https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
Even so, IOSG stresses that this does not make the training process private. Bridgewater’s expert labels are private data, but they passed through Tinker, a third-party service operating at roughly the same trust layer as a ZDR agreement. The compute was also rented. The buyer did not control the machines running the training job.
Attested post-training stacks are only starting to appear. In March, Workshop Labs and Tinfoil launched Silo, a post-training system running inside a Tinfoil enclave on a single eight-GPU node, with keys controlled only by the customer. IOSG says the enclave cost worked out to 11 extra minutes on a two-hour training run. By freezing base weights and training only small adapters, the stack could fit a one-trillion-parameter model, Kimi K2 Thinking. The hard part is that reinforcement learning moves data repeatedly across components, and data movement is exactly where enclave overhead shows up.
Less than a month after Silo launched, Workshop Labs was acquired by Thinking Machines. IOSG notes that the pieces needed to run the next Bridgewater-style RL loop inside an enclave now sit under one roof.
Agent tool calls remain the weak point
The report says every private inference system runs into the same external limit: tool use. However secure the path from prompt to model may be, an agent’s call to a calendar, database, search engine, or data service opens a route that the inference layer cannot seal.
That problem has grown with the recent push toward harness engineering. Each tool, memory store, and connected data source reads its own slice of the workflow in plaintext. Calendar servers see schedules. Database servers see queries. A fully local agent still has to send a search query in plaintext if it wants information outside its training data.
Mainstream fixes still live at the protocol layer. IOSG names Runlayer and MintMCP as examples of companies using centralized gateways to manage all tool traffic, redact PII before requests go out, decide which servers can receive traffic, and log destinations and content for audit purposes. But the tool server still has to read the plaintext query to answer it, and whether it keeps a copy depends on that server’s own retention terms. The trust burden multiplies across every tool in the harness.
Structure-level approaches strike at the middle layer first. Phala, for example, hosts MCP servers directly inside TEEs, covering wallets, code execution, and data sources. Users can verify privacy claims through attestation rather than just trusting the operator. But IOSG is clear about the limit: the TEE protects the messenger, not the final destination. The downstream service provider still receives the query in plaintext.
Encrypted search is still not commercially ready
Only a small number of destinations can answer without reading plaintext at all, and those are mostly structured-query systems. Apple’s private information retrieval lets iPhones compare incoming numbers against spam databases without exposing the number. Microsoft uses a similar design for passwords in Edge. MongoDB’s Queryable Encryption lets clients encrypt fields before they leave the device, while the server performs equality and range matches on ciphertext.
Open-ended search is not there yet. Brave promises zero data retention over its own 40 billion-page index, not Google’s, but that remains a protocol-level guarantee. Exa built a neural index that embeds user keywords semantically and ranks results by semantic match, yet the embedding step is still computed from plaintext on Exa’s servers.
Academic work has pushed the boundary without making it practical. MIT’s 2023 Tiptoe paper ranked results over 360 million webpages without revealing the query, but each search consumed large amounts of server compute and ranking quality lagged unencrypted search. Apple’s 2024 Wally paper hid a real query among decoys and reduced communication cost by as much as 31x, but the math only becomes cheap at millions of concurrent queries, a scale that IOSG says no private search system currently has.
Demand is growing, but private AI is still small next to the mainstream
IOSG argues that private AI already has paying users and working infrastructure. Venice AI recently passed 3.5 million registered users and 1.3 trillion monthly tokens of throughput, then closed a Series A equity round at a $1 billion valuation. Proton, described as a direct competitor, said its Lumo chat product passed 10 million users within a year of launch.
On the infrastructure side, Phala now processes 2 billion to 3 billion tokens a day on OpenRouter, according to the report. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil enclaves, offering verifiable privacy beyond simple proxying. IOSG also says self-hosting is likely the largest channel for private inference, since models running on a user’s own hardware leave no service-side usage trace.
Still, the gap to mainstream AI volumes is huge. Google processed 3,200 trillion tokens across its product line in May. On that basis, IOSG calculates that Venice’s entire monthly throughput equals about 18 minutes of Google traffic. In November last year, Google launched Private AI Compute, or PAC, which runs some Gemini-powered features inside isolated TPU enclaves audited independently by NCC Group. But PAC only covers a limited set of Pixel features such as personalized recommendations and recording summaries, not the Gemini app used by hundreds of millions of people.
What remains unsolved
IOSG closes by saying today’s hosted private AI offerings are still incomplete. Users who want the strongest privacy through E2EE have to wait for new features to be rebuilt in places where the provider cannot read them. Private harnesses still depend on protocol-level trust in the service layer. Affordable post-training often still requires trusting third-party suppliers to get the best tuning outcome. Self-hosting removes the service provider entirely, but the cost of running the strongest open models locally can exceed what most users can bear.
That said, the report argues that private AI is already usable and affordable. For consumers, Lumo and Venice offer private chat over open models under no-log promises at no cost, while $18 to $20 subscriptions from Venice or Tinfoil place similar chat inside enclaves for about the same price as a ChatGPT subscription. For enterprise workflows, attested endpoints can already be cheaper than plaintext routes. IOSG says NEAR’s E2EE API can bring encrypted context into enclaves today, with memory, file uploads, and custom instructions already working on top of E2EE.
On the training side, NVIDIA’s upcoming Vera Rubin NVL72 is expected to extend confidential computing from Blackwell’s eight-GPU nodes to 72-GPU racks, which IOSG says should make frontier-style RL loops more viable without exposing IP.
IOSG’s bottom line
The report says the defensible value is not in simply renting enclaves or operating protocol gateways. The harder, more durable opportunities sit in the unsolved half of the stack: training loops sealed inside enclaves, end-to-end tool calls, and searchable indexes that do not expose the query terms they answer.
Its final recommendation is split by use case. For execution-heavy, agent-heavy work, trust is still the practical choice because every tool call hands plaintext to a destination an enclave cannot hide from, and frontier closed models can justify their price there. For strategic work, planning, and judgments distilled from years of domain expertise, IOSG says verification matters more. That is where a company’s real alpha lives, and where tuning open models inside controlled environments starts to make economic and technical sense.
The report ends where it began: in the domains that define a company’s edge, expert-tuned open models have already beaten frontier models on both accuracy and cost, and the infrastructure needed to build them privately is arriving node by node.