Private AI is no longer just a policy preference. In a long report, IOSG said the market is reaching a split point: frontier closed models still offer the strongest general performance, but open models are getting cheaper, more specialized, and in some cases more effective where proprietary judgment matters most.
The piece, written by Jeff @IOSG, argues that in the domain where a company’s alpha lives, expert-tuned open models have already beaten frontier systems on both accuracy and cost. The remaining question is whether the infrastructure needed to build and run those models in private settings can catch up fast enough.
Why enterprises are rethinking AI privacy
IOSG opens with enterprise concerns. On July 1, Palantir CEO Alex Karp said in a CNBC interview that companies are paying a token premium to frontier labs while watching their own intellectual property flow to model providers. He described that as a transfer of alpha at the architecture layer, because every request sent to a closed model reaches the provider’s servers in plaintext.
Days before the interview, Palantir had announced a partnership with NVIDIA to run the open Nemotron model in customer-controlled environments, alongside a nine-point AI sovereignty statement. After the CNBC appearance, PLTR rose 8%.
The report contrasts this with the cloud software model of the last two decades. Traditional SaaS vendors usually saw only slices of a customer’s data: Salesforce handled sales channels, Workday handled HR, Jira handled development workflows, and AWS provided storage and compute. AI workflows, by contrast, often push companies to upload far more context at once, including structured information that links multiple departments.
That changes the risk profile. According to IOSG, upstream providers are no longer just storing data. They may also be in a position to use that data to build new features.
The leading labs have kept growing despite those concerns. IOSG said Anthropic’s annualized revenue reached $47 billion in May, up sharply from $9 billion at the end of 2025. OpenAI, it said, passed 900 million weekly active users in February. Both companies raised new funding this spring at valuations approaching $1 trillion and are expected to target even higher valuations in future IPOs.
From enterprise restrictions to court-ordered data retention
Some institutions moved early. IOSG said major Wall Street banks had already restricted ChatGPT use by May 2023, less than three months after its release. Samsung imposed a company-wide ban on generative AI the same month after engineers leaked chip source code into ChatGPT.
OpenAI responded in August 2023 with ChatGPT Enterprise, promising not to train on business data and offering zero-data-retention, or ZDR, agreements. IOSG said ZDR later became a standard enterprise procurement requirement.
But contracts cover company accounts, not everything employees do. Citing IBM, IOSG said that by 2025, shadow AI use had been involved in one-fifth of data leak incidents, and heavy shadow AI use added an average of $670,000 to breach costs. A 2025 survey by security training firm Anagram found that 40% of employees said they would violate AI use policies if it helped them finish work faster.
For consumers, the privacy issue became more concrete once courts got involved. IOSG said a May 2025 court order forced OpenAI to retain even deleted consumer chats. In November, a judge ordered 20 million of those chats to be handed over to lawyers for The New York Times as part of discovery.
The report also points to criminal cases. It says ChatGPT records were entered as evidence in the Palisades wildfire arson case, and an affidavit in a Florida double-homicide case cited questions from a suspect about how to dispose of bodies. In a July 2025 interview, Sam Altman said ChatGPT conversations are not covered by legal privilege and that OpenAI “may be required to produce” user chats in litigation.
IOSG cited a Kolmogorov Law survey from October 2025 of 1,000 U.S. AI users. Half did not know those conversations could be subpoenaed, while two-thirds believed they should receive protection comparable to conversations with a lawyer or doctor.
How private AI works today
The report frames private AI as a path problem. A prompt leaves a device, crosses a network, lands on the machine running the model, and comes back as a response. What changes across privacy approaches is where plaintext exists, who can read it, and whether privacy can be verified.
Protocol-layer privacy still relies on promises
Contract-based zero retention is the standard enterprise path. The provider knows who the user is, processes the plaintext prompt, and promises not to retain it. The guarantee depends on contract terms and reputation.
Anonymous proxy models remove identity but not content. IOSG said products such as Duck.ai negotiate deletion terms with model providers, while Venice tells users to assume providers may retain everything. Either way, users cannot verify what happens downstream.
Transport between machines usually runs over TLS. That protects the pipe, not the recipient. Relays often use Oblivious HTTP, or OHTTP, to separate who sent the message from what the message says. IOSG notes that OHTTP became an IETF standard in January 2024, and many companies already run production traffic through relays leased from Cloudflare and Fastly.
In IOSG’s view, that is close to the privacy ceiling for access to closed models. The reason is economic. Frontier training runs now cost on the order of billions of dollars, while near-trillion-dollar valuations are built on exclusive control over model weights.
The report uses Meta as an example. LLaMA launched in February 2023 with access limited to researchers, but the weights leaked to 4chan in less than a week. Another week later, llama.cpp had the smallest 7B model answering locally on a MacBook. Three days after that, Stanford fine-tuned Alpaca on the same model for under $600. By July 2023, Meta had formally released Llama 2 under a commercial license with a 700 million monthly active user exclusion. IOSG’s point is simple: once weights escape, the premium built on exclusivity escapes too.
Theoretically, frontier labs could add remote attestation for closed-model inference. But IOSG says attestation can only show which code read a prompt, not what that code did with it. Verifying that a provider did not retain data would require auditing the serving code itself and tying it to the hardware-reported hash. Labs are unlikely to expose that code because it contains the batching and caching techniques that support margins and carry over into future model generations.
That is why, according to IOSG, verifiable privacy is far more realistic on top of open models than proprietary ones.
Structural privacy replaces trust with hardware or cryptography
Trusted execution environments, or TEEs, move inference into a hardware enclave. The chip signs an attestation saying which model and which code were run. The problem is that the prompt is only sealed at the endpoint. If traffic passes through an intermediary, someone along the route may still see plaintext.
End-to-end encryption, or E2EE, removes readable relays. A user device encrypts the prompt to the enclave’s key, and every hop in between sees only sealed data. But IOSG says verifiable E2EE needs more than a proven enclave. It also needs open, reproducible client code, because the client that encrypts the prompt and checks attestation could also break the guarantee.
That makes E2EE harder to ship. Once a proxy becomes a blind courier, features that depend on reading plaintext have to be rebuilt around client keys or reconstructed inside the enclave.
Fully homomorphic encryption, or FHE, and MPC variants push trust reduction further. Servers compute on ciphertext they cannot open, while MPC splits prompts into secret shares across multiple parties. The tradeoff is speed. IOSG says encrypted inference costs are 10,000x to 100,000x those of plaintext inference. Even small models can take seconds to minutes per token, where unencrypted systems respond in milliseconds. Custom chips for encrypted computation may narrow the gap, but the first prototype only demoed in early 2026 and commercial versions are still years away.
Local inference removes the path entirely. The model runs on the user’s own hardware, with no relay, no server, and no provider. That is the cleanest privacy model. It is also expensive and constrained by model size. IOSG said gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index while taking up 65GB, more than the combined VRAM of two flagship gaming GPUs. Full-precision GLM-5.2, it added, only runs on an eight-GPU datacenter node, with GPU costs alone above $300,000.
Performance and pricing are moving in the right direction
Even with those limits, IOSG says the cost of enclave-based inference is falling. On single-GPU inference, Phala’s benchmarks show average H100 throughput loss below 7% in enclave mode. On larger models, the loss is near zero because the main cost comes from moving data into the chip, not computing inside it.
For multi-GPU inference, NVIDIA’s Blackwell generation supports direct encrypted traffic between chips. Older H100 systems can only achieve the same effect by routing through the CPU host at one-seventh the bandwidth. NVIDIA’s own Blackwell benchmarks, as cited by IOSG, show throughput loss below 8% for a 397B model in enclave mode.
The report says privacy inference overhead is no longer the deciding bottleneck.
It also gives price comparisons. H100s shipped after 2023 already include enclave mode, so the extra cost is mostly throughput drag rather than additional hardware. On Azure, confidential H100 instances cost $8.90 per hour, versus $6.98 without enclave mode, a 27% premium. On Phala, an H100 in confidential mode starts at $3.80 per hour, below Lambda’s plain SXM range of $3.99 to $4.29.
Hosted APIs are also getting competitive. IOSG said NEAR AI offers attested gpt-oss-120b endpoints at $0.15 per million input tokens and $0.55 per million output tokens, roughly in line with plaintext routes on Amazon Bedrock, Together, and Groq. For larger parallelized models, it said NEAR AI matches Fireworks on GLM 5.2 pricing and is 15% cheaper on input and 4% cheaper on output for Kimi K2.6.
New privacy providers may be sacrificing margins to win share, the report says, but the structural direction is still clear: privacy costs are coming down for both operators and users.
Where open models are already winning
IOSG says frontier labs still lead the best open models by roughly four months in general capability. But it highlights one case where that gap did not matter. On June 30, Bridgewater’s AIA Labs and Thinking Machines presented results showing that an expert-labeled open model beat frontier systems on both accuracy and cost in financial tasks.
The team fine-tuned Qwen3-235B on Tinker, Thinking Machines’ hosted fine-tuning API. They first bought annotations from vendors, then routed disagreement cases to the firm’s investment staff for relabeling. Training used GRPO reinforcement learning with three modifications: round-robin batching, CISPO loss, and on-policy distillation.
The tasks came directly from investor workflows: whether a news item mattered to a C-suite-level investment professional, whether a central bank document signaled the direction of future rate moves, and where template language began in a document or email.
On an independent test set, frontier models scored about 50% with simple prompts and reached 78.2% with expert prompts, still below the 80% threshold set by the investors. The fine-tuned Qwen model scored 84.7%. By the paper’s framing, that meant 29.8% fewer errors than the best frontier model and 13.8x lower inference cost.
IOSG links to the original write-up here: https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
Still, the report says that training process was not private. Bridgewater’s expert labels moved through Tinker, a third-party service, and the compute was rented on machines the buyer did not control. Anyone who wants the same recipe without those trust assumptions still has limited options. Renting bare GPU clusters leaves training visible to cloud operators. Buying a cluster solves custody at a far higher cost.
Attested post-training has only just started to appear. IOSG points to Silo, released in March by Workshop Labs and Tinfoil, as a post-training stack running inside a Tinfoil enclave on a single eight-GPU node with customer-controlled keys. The article says the enclave overhead was 11 extra minutes on a two-hour training run. By freezing base weights and training only a small adapter, Silo could fit a trillion-parameter model, Kimi K2 Thinking. The hard part is RL itself, since it moves data back and forth between components, which is exactly where enclave overhead shows up.
Less than a month after Silo launched, Workshop Labs was acquired by Thinking Machines. IOSG says the parts needed to run the next Bridgewater-style RL loop inside an enclave are now sitting under the same roof.
Agent tools and search remain unresolved
One issue sits outside all private inference approaches. They may protect the path between prompt and model, but every external tool call opened by an agent creates a new path that the inference layer does not control.
Harness engineering makes that larger. Every attached tool, memory store, and data source becomes another destination that can read some plaintext slice of the workflow. Calendar servers see schedules. Database servers see queries. Even a fully local agent must send plaintext search terms to a search engine if it wants facts beyond its training set.
Mainstream mitigations still sit at the protocol layer. IOSG says companies such as Runlayer and MintMCP route tool traffic through a central gateway that redacts personal information before requests leave, decides which servers may receive traffic, blocks unreviewed endpoints, and logs destinations and contents for forensics. Even with SOC 2 audits, tool servers still need plaintext to answer. Their own retention policies still matter, and the risk multiplies across every tool in the harness.
Structural solutions reach only part of the way. IOSG points to Phala’s TEE-hosted MCP servers, which cover wallets, code execution, and data sources and can expose an attestation instead of asking users to trust an operator. But the report is explicit: TEE can seal the messenger, not the destination. The final service provider still has to read the query in plaintext.
Only a small set of destinations can answer without reading the query, and mostly for structured lookups. Apple’s private information retrieval lets iPhones compare caller numbers against spam databases without exposing the number. Microsoft uses a similar design for passwords in Edge. MongoDB’s Queryable Encryption lets clients encrypt fields before they leave the device while still allowing equality and range queries on ciphertext.
Open-ended search is much harder. Brave promises zero data retention on its own 40 billion-page index, not Google’s, but that is still a protocol-layer promise. Exa built a neural index that embeds user keywords into semantic space and ranks results semantically, yet the embedding step still starts from plaintext on Exa’s servers. MIT’s 2023 Tiptoe paper ranked results across 360 million webpages without exposing the query, but each search consumed heavy server resources and produced lower-quality rankings than plaintext search. Apple’s 2024 Wally paper hid real queries among decoys and cut communication cost by as much as 31x, but the math only gets cheap at millions of concurrent queries, a scale no private search system currently has.
Encrypted search is possible, IOSG concludes. It just is not commercially practical yet on performance or price.
Demand is rising, but the market is still small
On the demand side, IOSG says Venice AI recently passed 3.5 million registered users and 1.3 trillion monthly tokens, then closed a new Series A equity round at a $1 billion valuation. Proton, its direct competitor, passed 10 million users within a year of launching Lumo.
Infrastructure is growing too. Phala now handles 2 billion to 3 billion tokens a day on OpenRouter, according to the report. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil enclaves to give users verifiable privacy beyond proxying. IOSG adds that self-hosting is not counted in those figures and may well be the biggest channel for private inference, since models running on local hardware leave no service-side usage trace.
Even so, private AI remains tiny relative to the broader market. IOSG says Google processed 3,200 trillion tokens in May. On that basis, Venice’s monthly throughput is roughly equal to 18 minutes of Google’s volume.
Google launched Private AI Compute, or PAC, last November, putting some Gemini-powered features into sealed TPU enclaves isolated from Google itself and audited by NCC Group. But PAC covers only limited Pixel features such as personalized recommendations and recording summaries, not the Gemini app used by a far larger audience. IOSG says Google can afford to expose that design to auditors because those features monetize through devices and ads, not token sales.
What IOSG thinks matters next
The report ends by saying today’s hosted options are still imperfect. Users who want the strongest privacy through E2EE must wait for features to be rebuilt in places the provider cannot read. Private harness systems still depend on protocol-layer trust at the service boundary. Reasonably priced post-training still often requires trusting third-party vendors to get the best fine-tuning results. Self-hosting cuts providers out entirely, but running the strongest open models locally can cost more than most people can justify.
Even with those gaps, IOSG says private AI is now a real and affordable option. On the consumer side, Lumo and Venice offer free private chat with open models under no-log promises, while Venice or Tinfoil subscriptions priced at $18 to $20 put similar chats inside enclaves for about the cost of a ChatGPT subscription. On enterprise workflows, attested endpoints can already be cheaper than plaintext ones. IOSG also says endpoints such as NEAR’s E2EE API can bring encrypted context into enclaves today, with memory, file uploads, and custom instructions already working over E2EE.
On the training side, NVIDIA’s upcoming Vera Rubin NVL72 will expand confidential computing from Blackwell’s eight-GPU node to a 72-GPU rack, making frontier RL loops more feasible without exposing IP.
Still, IOSG argues the real value does not sit in renting enclaves alone. The defensible ground is in what remains unsolved: training loops sealed inside enclaves, end-to-end private tool execution, and search indexes that can answer without seeing the terms. Whoever delivers one of those first will be selling something that is harder to commoditize than a switch on a standard chip.
Its bottom line is not that trust disappears. For execution-heavy, agent-heavy workflows, trust may still be the practical choice because every tool call hands plaintext to a destination that enclaves cannot fully shield, and frontier models still justify their price in that loop. But for strategy, planning, and expert judgment that distinguish one firm from another, IOSG says verification matters more. In that domain, the report argues, companies can use proprietary insight to fine-tune open models inside environments they control.