Private AI is becoming a practical option rather than a theoretical ideal, according to a long report from IOSG. Its core claim is blunt: in domains where a company’s alpha lives, expert-tuned open models can now beat frontier systems on both accuracy and cost, while the infrastructure needed to build and run those models in private is arriving piece by piece.
Written by Jeff at IOSG, the report frames the issue around a simple question: when a prompt leaves a device, who can read it, who keeps it, and what can actually be verified instead of merely promised?
Why enterprises are rethinking closed-model AI
The report opens with comments from Palantir CEO Alex Karp, who said in a July 1 CNBC interview that enterprises are paying a token premium to frontier labs while watching their own intellectual property flow outward. Karp described that as a transfer of alpha taking place at the architectural level, because every request sent to a closed model reaches the provider’s servers in plaintext.
Just days before the interview aired, Palantir had announced a partnership with NVIDIA to run the open Nemotron model inside customer-controlled environments, alongside a nine-point declaration on AI sovereignty. After the CNBC appearance, PLTR rose 8%.
IOSG contrasts today’s AI workflows with the cloud software era. For years, companies adopted SaaS products by relying on protocol-level trust, and the model mostly worked because each vendor saw only a slice of enterprise data. Salesforce saw sales funnels, Workday saw HR records, Jira saw engineering progress, AWS supplied storage and compute. AI systems now ask for much more. They pull in broad company context, often across departments, in order to raise productivity, which gives upstream providers access to far richer internal material.
The report says Anthropic’s annualized revenue reached $47 billion in May 2026, up from $9 billion at the end of 2025, while OpenAI passed 900 million weekly active users in February 2026. Both companies raised fresh funding in the spring and were described as approaching $1 trillion valuations, with expectations of even higher IPO valuations. IOSG’s point is not that privacy concerns are absent. It is that those concerns have not slowed adoption.
From enterprise restrictions to court orders
Some institutions moved early. IOSG notes that major Wall Street banks had already restricted ChatGPT by February 2023, less than three months after its release. In May 2023, Samsung banned generative AI across the company after engineers pasted semiconductor source code into ChatGPT.
OpenAI responded in August 2023 with ChatGPT Enterprise, promising not to train on business data and offering zero-data-retention, or ZDR, terms. Those provisions later became standard items in enterprise procurement.
Still, contracts only cover approved company accounts. Citing IBM, the report says shadow AI, meaning employees feeding company data into unapproved tools through personal accounts, was involved in one-fifth of data leak incidents by 2025. Heavy shadow AI usage added an average of $670,000 to the cost of a breach. In a 2025 survey by security training company Anagram, 40% of employees said they would violate AI-use rules if it helped them finish work faster.
For consumers, the issue sharpened once legal discovery entered the picture. A court order in May 2025 forced OpenAI to retain even deleted consumer chats. In November that year, a judge ordered 20 million of those records to be turned over to lawyers for The New York Times as part of discovery. IOSG also points to criminal cases: ChatGPT records were entered into evidence in the Palisades wildfire arson case, and an affidavit in a Florida double-homicide case cited a suspect’s questions about how to dispose of bodies.
Sam Altman said in a July 2025 interview that ChatGPT conversations are not protected by legal privilege and that OpenAI may be required to hand them over in litigation. A survey by Kolmogorov Law in October 2025 found that 50% of 1,000 U.S. AI users did not know these chats could be subpoenaed, while two-thirds said they should receive protections comparable to conversations with lawyers or doctors.
How private AI is being built today
IOSG breaks the current market into two broad buckets: protocol-level privacy and structure-level privacy. The first still leaves plaintext visible to someone other than the user and relies on promises after that point. The second tries to replace promises with proofs based on hardware or cryptography.
Protocol-level privacy: contracts, proxies and transport encryption
Zero-retention contracts are the standard enterprise answer. The provider knows the customer, processes the prompt, and promises not to keep the data. Anonymous proxies remove identity but do not encrypt what is being said, so downstream providers still handle plaintext under their own policies.
IOSG cites Duck.ai, which negotiates deletion arrangements with model providers, and Venice, which asks users to assume providers may retain everything. In both cases, the user cannot independently verify what happened.
At the network layer, TLS encrypts the pipe, not the recipient. Oblivious HTTP, standardized as RFC 9458 in January 2024, splits identity knowledge from content knowledge. The report compares it to handing a note to a friend: the intermediary knows who passed it along but cannot read it, while the recipient can read it but does not know who wrote it. IOSG says many companies already send production traffic through OHTTP relays rented from Cloudflare and Fastly.
That, in the report’s view, is the privacy ceiling for closed-source models. Frontier labs are built around exclusive control of weights and serving systems, and their valuations depend on keeping that edge.
Meta is used as the example. LLaMA launched in February 2023 for researchers, but its weights leaked to 4chan in less than a week. Another week later, llama.cpp made the smallest 7B version runnable on a MacBook. Three days after that, Stanford researchers used the same model to produce Alpaca for under $600 in fine-tuning costs. Meta then released Llama 2 in July 2023 under a commercial license with a 700 million monthly active user exclusion clause. Once the weights were out, IOSG argues, the premium went with them.
The report adds that remote attestation for closed models has a limit. It can show which code read the prompt, but not what that code did with the data. To verify no retention, outsiders would need to audit the serving code and match it to the hardware-reported hash. Labs have little incentive to reveal those systems because the serving stack contains the batching and caching tricks that support margins across model generations.
Structure-level privacy: TEE, E2EE, FHE and local inference
Trusted execution environments place inference inside a hardware enclave, a sealed region that even the machine operator cannot freely inspect. The chip signs an attestation stating what model and code are running.
But a TEE only seals the destination. If a request passes through a proxy, that proxy may still see plaintext. End-to-end encryption tries to remove that readable middle point by encrypting the prompt with the enclave’s key on the user device, so every network hop carries only ciphertext.
That shifts trust toward the client. The software responsible for encrypting prompts and verifying attestation must also be open and reproducible, or it can quietly break the guarantee. E2EE therefore brings more engineering burden than a plain TEE setup, especially for features that normally depend on reading plaintext outside the enclave.
Fully homomorphic encryption, along with MPC variants, goes further by trying to remove the trusted party entirely. Servers compute directly on encrypted data or across secret shares. The cost is speed. IOSG says encrypted inference runs at roughly 10,000 to 100,000 times the cost of plaintext inference, with token latencies on small models ranging from seconds to minutes instead of milliseconds. Dedicated chips may help, but the first prototype only completed a demo in early 2026 and commercial versions are still years away.
Local inference avoids the problem by removing the path. No relay, no server, no provider. But the trade-off is clear: hardware cost and model capability. The report says gpt-oss-120b scores at about half of GLM-5.2 on the Artificial Analysis index while weighing 65 GB, larger than the combined VRAM of two flagship gaming GPUs. Full-precision GLM-5.2, it says, requires an eight-GPU datacenter node, with GPU cost alone above $300,000.
Private inference is getting cheaper
IOSG argues that TEE-based inference no longer carries the kind of performance penalty many assume. Phala’s single-GPU benchmarks show less than 7% throughput loss on H100 in enclave mode on average, approaching zero on larger models because the main cost comes from moving data into the chip rather than computing inside it.
For multi-GPU inference, NVIDIA’s Blackwell generation already supports direct encrypted traffic between chips. Older H100 systems can only do that by routing through the CPU host at one-seventh of the bandwidth. NVIDIA’s own benchmark on Blackwell, according to the report, showed less than 8% throughput loss for a 397B model in enclave mode.
Pricing is also changing. Azure’s confidential H100 SKU rents for $8.90 per hour versus $6.98 without enclave mode, a 27% premium. On Phala, by contrast, confidential H100 starts at $3.80 per hour, below Lambda’s plain SXM range of $3.99 to $4.29.
On hosted APIs, IOSG says NEAR AI prices attested access to gpt-oss-120b at $0.15 per million input tokens and $0.55 per million output tokens, matching plaintext offerings from Amazon Bedrock, Together and Groq. On GLM 5.2, NEAR AI matches Fireworks exactly, while on the larger Kimi K2.6 it is 15% cheaper on input and 4% cheaper on output.
The report leaves open whether some providers are sacrificing margins to gain share. Still, it says the broader direction is obvious: privacy is getting cheaper for both operators and users.
A Bridgewater case for open models in finance
The report’s central example comes from Bridgewater’s AIA Labs and Thinking Machines. In research released on June 30, the teams showed that an open model fine-tuned with expert labels beat frontier models on both accuracy and cost in financial tasks.
The work used Thinking Machines’ hosted fine-tuning API, Tinker, to tune Qwen3-235B. The team first bought annotations from vendors, then routed disputed samples to the firm’s investment professionals for relabeling. Training used GRPO reinforcement learning with three modifications: round-robin batching, CISPO loss and on-policy distillation.
The tasks came directly from investment workflows: whether a news item mattered to C-suite-level investment professionals, whether a central bank document implied a future direction for rates, and where template language began in a document or email.
On an independent test set, frontier models scored about 50% on average with simple prompts and only 78.2% with expert prompts, still below the investors’ 80% threshold. The fine-tuned Qwen model reached 84.7%. Using the paper’s own framing, that meant 29.8% fewer errors than the best frontier setup and inference costs that were 13.8 times lower.
The original research is available here: https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
IOSG is careful not to overstate the result. The model may have been open, but the training path was not private. Bridgewater’s expert labels were proprietary data, yet they passed through Tinker, which remains a third-party service in the same trust class as ZDR contracts. The compute was also rented. Buyers who want this recipe without relying on those trust assumptions still face limited choices: use bare GPU clusters that cloud operators can inspect, or buy the hardware and absorb a large capital bill.
A more verifiable route is just beginning to appear. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running inside a Tinfoil enclave on a single eight-GPU node, with keys controlled only by the customer. IOSG says the enclave overhead added 11 minutes to a two-hour training run. By freezing base weights and training only small adapters, the stack can fit a trillion-parameter model, Kimi K2 Thinking. The hard part is reinforcement learning’s data movement across components, which is exactly where enclave costs show up.
Less than a month after Silo launched, Workshop Labs was acquired by Thinking Machines. IOSG notes that the pieces needed to run the next Bridgewater-style reinforcement learning loop inside an enclave now sit under one company.
The hardest privacy problem may be in the tool layer
Even if prompt-to-model inference is sealed, agent systems still make external tool calls. Each call opens a route that inference-layer protections do not cover. IOSG says the current harness engineering wave multiplies this issue, because every tool, memory store and data source surrounding the model becomes another plaintext destination.
Calendar servers see schedules. Databases see queries. Search engines must read the terms they are asked to search. If the server cannot see the query, it cannot answer.
The mainstream answer is still protocol-based. Companies such as Runlayer and MintMCP use central gateways to control tool traffic, strip personal data before requests leave, decide which servers can receive requests, and log destinations and contents for forensic purposes. Even with SOC 2 audits, the tool servers still need plaintext to respond, and retention depends on their own policies. The gateway itself also becomes another reader that has to be trusted.
Structure-level approaches cover the middle layer instead. Phala, for example, hosts MCP servers directly inside TEEs across wallets, code execution and data sources, so users can verify privacy claims through attestation rather than trust. But the final destination remains a problem. A TEE can protect the messenger, not the endpoint that ultimately needs the query in plain form.
Only a few destinations have learned to answer without seeing the raw input, and mostly for structured queries. Apple uses private information retrieval on iPhone so incoming phone numbers can be checked against spam databases without exposing the number. Microsoft applies a similar method to passwords in Edge. MongoDB’s Queryable Encryption lets clients encrypt fields before sending them, while servers still handle equality and range lookups on ciphertext.
Open-ended search has not crossed that line. Brave promises zero data retention on its own 40 billion-page index, but that remains a trust-based system. Exa built a neural index that ranks by semantic embeddings, yet those embeddings are still computed from plaintext on Exa’s servers. MIT’s 2023 Tiptoe paper showed private ranking across 360 million web pages without exposing the query, but each search consumed heavy server resources and ranking quality trailed unencrypted search. Apple’s 2024 Wally paper lowered communication cost by mixing real queries with decoys, by as much as 31 times, though the math only becomes cheap at millions of concurrent queries, a scale no private search system currently has.
IOSG’s takeaway is simple: encrypted search is possible in principle, but still not commercially viable on performance and price.
Demand exists, but the market is still small
On the demand side, the report points to several datapoints. Venice AI recently passed 3.5 million registered users and 1.3 trillion monthly tokens of throughput, then completed a Series A equity round at a $1 billion valuation. Proton, which IOSG calls a direct competitor, reached 10 million users for its Lumo chat product within a year of launch.
At the infrastructure level, Phala now processes 2 billion to 3 billion tokens per day on OpenRouter. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil’s enclave to give users verifiable privacy beyond a proxy arrangement.
Even so, private AI remains tiny relative to the broader market. IOSG says Google processed 3,200 trillion tokens across its products in May 2026. On that basis, Venice’s monthly throughput amounts to roughly 18 minutes of Google’s scale.
Google did introduce Private AI Compute, or PAC, in November 2025, placing some Gemini-powered features inside TPU enclaves isolated from Google itself and independently audited by NCC Group. But PAC only covers a narrow set of Pixel features, such as personalized recommendations and call recording summaries. It does not extend to the Gemini app used by hundreds of millions. IOSG argues that this is easier for Google because those features monetize through devices and advertising, not through token sales.
What IOSG thinks matters next
The report ends with a narrower investment argument. Today’s hosted setups are still incomplete. Users who want the strongest privacy through E2EE must wait for key product features to be rebuilt in places where the provider cannot read plaintext. Private harness layers still rely on trust at the service edge. Affordable post-training still often requires trusting third parties if the goal is the best possible tuning result. Self-hosting cuts providers out of the loop, but running the strongest open models locally can cost more than most users or firms will accept.
Even with those gaps, IOSG says private AI is already real and affordable. On the consumer side, Lumo and Venice offer private chat on open models under no-log promises at no cost, while Venice or Tinfoil subscriptions at $18 to $20 per month place similar chats inside enclaves for about the same price as a ChatGPT subscription. For enterprise workflows, attested endpoints can already be cheaper than plaintext alternatives. The report adds that NEAR’s E2EE API can carry encrypted context into enclaves today, supporting memory, file uploads and custom instructions.
On the training side, NVIDIA’s upcoming Vera Rubin NVL72 will expand confidential computing from Blackwell’s eight-GPU node to a 72-GPU rack, which IOSG says should make frontier-style reinforcement learning loops more feasible without exposing proprietary data.
Still, IOSG argues that the real value will not accrue to operators simply renting enclave capacity. The stronger position lies in what remains unsolved: training loops that stay sealed inside enclaves, tool calls that remain private end to end, and search indexes that can answer without seeing the terms. Whoever solves one of those gaps first will be selling something less vulnerable to commoditization than access to a hardware switch.
The report’s final split is straightforward. For execution-heavy, agent-heavy workflows, trust still wins for now because every tool call hands plaintext to endpoints that enclaves cannot hide, and frontier models still justify their price in those loops. For strategic reasoning, planning and domain judgment, the recommendation shifts toward verification: keep the boundary under company control and use proprietary insights to fine-tune open models. In the domains where a company’s alpha truly lives, IOSG argues, expert-tuned open models are already beating frontier systems on both cost and accuracy, and the private infrastructure around them is catching up.