Private AI is no longer a niche technical idea in this IOSG report. The real split, it argues, is between trusting a model provider with proprietary context and running systems whose privacy claims can be checked.
Why private AI is back in focus
Jeff of IOSG opens with a July 1 CNBC interview featuring Palantir CEO Alex Karp. Karp said companies are paying a token premium to frontier labs while their own IP flows back to model vendors. He described that as a transfer of alpha, built into the architecture itself, because every request sent to a closed model reaches the provider’s servers in plaintext.
Days before the interview, Palantir announced a partnership with NVIDIA to run the open Nemotron model in customer-controlled environments and paired that launch with a nine-point AI sovereignty statement. After the CNBC appearance, PLTR rose 8%.
The report contrasts this with the old cloud software model. Over the past two decades, enterprises accepted protocol-level trust because each SaaS vendor usually saw only a slice of company data. Salesforce handled sales, Workday handled HR, Jira saw development workflows, and AWS supplied storage and compute. AI workflows ask for much more. They often require broad uploads of internal data plus structured context spanning multiple departments.
IOSG says Anthropic’s annualized revenue reached $47 billion in May, up from $9 billion at the end of 2025. OpenAI, it says, crossed 900 million weekly active users in February. Both companies raised fresh capital this spring, with valuations nearing $1 trillion and IPO expectations above that level. Privacy and IP complaints have not slowed their momentum.
Large organizations moved early
Some companies did not wait. The report says major Wall Street banks had already restricted ChatGPT use by May 2023, less than three months after launch. That same month, Samsung banned generative AI across its network after engineers exposed chip source code through ChatGPT.
OpenAI responded in August 2023 with ChatGPT Enterprise, promising not to train on business data and offering zero-data-retention, or ZDR, terms. IOSG says ZDR later became a standard requirement in enterprise procurement.
That still leaves shadow AI. IBM found that by 2025, shadow AI, defined as employees feeding company data into unapproved AI tools through personal accounts, was involved in one-fifth of data leakage incidents. Heavy shadow AI use added an average of $670,000 to breach costs. In a 2025 survey from security training company Anagram, 40% of employees said they would break AI usage policies to finish work faster.
Consumer privacy reached the courts
The enterprise buyer can at least pay for better terms. Ordinary users often cannot. IOSG points to a May 2025 court order that forced OpenAI to retain even deleted consumer chats. In November that year, a judge ordered 20 million of those chats turned over to lawyers for The New York Times in discovery.
The report also cites criminal cases. ChatGPT records entered evidence in the Palisades fire arson case, and an affidavit in a Florida double-homicide case referenced prompts about how to dispose of bodies. Sam Altman said in a July 2025 interview that ChatGPT conversations do not carry legal privilege and OpenAI could be required to hand them over in litigation.
That issue is not limited to criminals. IOSG cites a Kolmogorov Law survey of 1,000 U.S. AI users from October 2025 showing that 50% did not know chats could be subpoenaed, while two-thirds believed those conversations should receive protections similar to consultations with lawyers or doctors.
How privacy is being implemented now
The report frames privacy AI around a single path: a prompt leaves the device, crosses a network, lands on the machine running the model, then returns as a response. What changes from one design to another is where plaintext exists, who can read it, and what evidence exists to verify privacy.
Protocol-level privacy
At this layer, someone other than the user can still read the prompt in plaintext. Contractual zero retention is the enterprise version: the provider knows who you are, processes the request, and promises not to keep it. Anonymous proxies hide identity, but not the content itself. IOSG says products such as Duck.ai negotiate deletion terms with model vendors, while Venice asks users to assume providers may retain everything. In either case, the claim is not verifiable.
TLS secures the transport pipe, not the recipient. Oblivious HTTP, or OHTTP, splits visibility across different parties, like passing a note through an intermediary. The relay knows who sent it but cannot read the contents; the receiver can read the contents but does not know who wrote it. OHTTP became an IETF standard in January 2024, and IOSG says many companies now run production traffic through relays rented from Cloudflare and Fastly.
For closed-source frontier models, the report treats that as the ceiling. A flagship training run now costs on the order of billions of dollars, and much of the valuation premium comes from exclusive control over the weights.
Meta already lived through the downside. LLaMA launched in February 2023 with access limited to researchers. Less than a week later, the weights leaked to 4chan as a torrent. A week after that, llama.cpp made the 7B model runnable on a MacBook. Three days later, Stanford fine-tuned the same model into Alpaca for under $600. Meta later released Llama 2 in July 2023 under a commercial license with a 700 million monthly active user exclusion.
IOSG notes that frontier labs could, in theory, provide attestation for closed-model inference, but attestation can only prove which code read the prompt. It cannot prove what that code did afterward. To verify retention claims, outside parties would need the serving code and would need to match it to the hash reported by the hardware. Labs have little reason to reveal that, because those serving tricks, including batching and caching, support margins across model generations.
Structural privacy: TEE, E2EE, FHE and local inference
Structural approaches replace contractual trust with hardware or cryptographic guarantees, though each comes with tradeoffs and today largely applies to open models.
Trusted execution environments, or TEEs, place inference inside a hardware enclave that even the machine operator cannot directly inspect. The chip signs an attestation identifying the model and code being run. But plaintext can still appear before the prompt reaches that endpoint if a proxy or relay sits in the path.
End-to-end encryption closes off readable intermediaries. The user device encrypts the prompt to an enclave key, so every hop carries sealed content that only the enclave can open. IOSG says verifiable E2EE needs both a proven enclave and open, reproducible client code, because the client software that performs encryption and checks attestation could also undermine the guarantee.
Fully homomorphic encryption, along with MPC variants, tries to remove the trusted party entirely. The server computes on ciphertext it can never unlock, or the prompt is split across multiple parties. The cost remains severe. The report says encrypted inference can run at 10,000x to 100,000x the cost of plaintext, with each token on small models taking seconds to minutes instead of milliseconds. A dedicated chip for encrypted computation is expected to show a prototype in early 2026, with commercial deployment still years away.
Local inference removes the path outright. The model runs on the user’s own hardware, with no relay, no server and no provider in the loop. The tradeoff is cost and capability. IOSG says gpt-oss-120b scores roughly half of GLM-5.2 on the Artificial Analysis index while occupying 65 GB, more than the VRAM of two flagship gaming GPUs combined. Full-precision GLM-5.2 requires an eight-GPU datacenter node, with GPU cost alone above $300,000.
Confidential inference is getting cheaper
Beyond those structural limits, the cost of running inference inside enclaves is falling. Phala’s single-GPU benchmarks show average throughput loss of under 7% on H100 in enclave mode, and close to zero on large models where the main cost comes from moving data into the chip rather than computing inside it.
On multi-GPU workloads, NVIDIA’s newer Blackwell generation supports direct encryption for chip-to-chip traffic. Older H100 systems can only approximate that by routing through the CPU host at one-seventh the bandwidth. NVIDIA’s own Blackwell benchmark, according to IOSG, showed throughput loss below 8% for a 397B model in enclave mode.
The pricing picture is shifting too. Azure lists confidential H100 instances at $8.90 per hour versus $6.98 without enclave mode, a 27% premium. Phala, which focuses on enclave infrastructure, rents confidential H100 starting at $3.80 per hour, below the $3.99 to $4.29 range IOSG cites for standard Lambda SXM cards.
For hosted APIs, NEAR AI offers an attested endpoint for gpt-oss-120b at $0.15 per million input tokens and $0.55 per million output tokens, roughly in line with plaintext routes on Amazon Bedrock, Together and Groq. On GLM 5.2, IOSG says NEAR AI matches Fireworks pricing exactly. On the larger Kimi K2.6, input is 15% cheaper and output is 4% cheaper.
An open-model case from finance
The frontier gap is still visible, but IOSG highlights a June 30 example from Bridgewater’s AIA Labs and Thinking Machines. In that study, an expert-tuned open model beat frontier systems on both accuracy and cost.
The team fine-tuned Qwen3-235B on Tinker, Thinking Machines’ hosted fine-tuning API. They first purchased labeled data from vendors, then sent disagreement cases to Bridgewater investment staff for relabeling. Training used GRPO reinforcement learning plus three modifications: round-robin batching, CISPO loss and on-policy distillation.
The tasks came from real investment workflows: whether a news article matters to C-suite level investment professionals, whether a central bank document implies a future rate direction, and where template language begins in a document or email. Performance was measured on an independent test set.
Frontier models averaged about 50% on simple prompts and reached only 78.2% with expert prompting, still below the 80% threshold set by the investment team. The fine-tuned Qwen model reached 84.7%. Using the study’s own framing, that meant 29.8% fewer mistakes than the best frontier setup, at 13.8x lower inference cost.
https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
Even so, the training process itself was not private. Bridgewater’s expert labels passed through Tinker, which left the setup in a trust model similar to ZDR contracts. Compute was rented, not controlled. Buyers who want the recipe without those assumptions still face limited options: rent bare GPU clusters and expose training to the cloud operator, or buy the hardware and accept a much higher bill.
Attested post-training is just starting to appear. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running inside a Tinfoil enclave on a single eight-GPU node, with keys held only by the customer. IOSG says the enclave overhead added 11 minutes to a two-hour training job. By freezing base weights and training small adapters, the system could fit a trillion-parameter model, identified in the article as Kimi K2 Thinking. The hard part is that reinforcement learning moves data back and forth among components, and data movement is exactly where enclave costs show up.
Less than a month after Silo launched, Workshop Labs was acquired by Thinking Machines. IOSG’s point is simple: more of the stack needed to run the next Bridgewater-style RL loop inside enclaves now sits under one roof.
The agent problem sits outside inference
The report says every private inference design runs into the same wall once an agent starts calling tools. Each external tool opens a path the inference layer cannot seal. Calendar servers read schedules. Databases read queries. Search engines need plaintext queries to return useful results.
Mainstream solutions still default to protocol controls. Companies such as Runlayer and MintMCP use a central gateway to mask personally identifiable information before requests leave, decide which servers can receive traffic, and log destinations and content for forensics. Even with SOC 2 audits, the tool server must still read plaintext to answer, and retention depends on that server’s own policy. The trust burden multiplies across every tool in the harness.
Structural solutions can protect the middle layer. Phala, for example, hosts MCP servers inside TEEs across wallets, code execution and data sources, allowing users to verify privacy claims through attestation rather than trust the operator alone. But the report stresses the same limitation: the final service provider still receives the query in plaintext. The enclave hides the courier, not the destination.
A few structured query systems can answer without reading the query directly. IOSG points to Apple’s private information retrieval for spam-call matching on iPhone, Microsoft’s use of similar techniques for passwords in Edge, and MongoDB’s Queryable Encryption for encrypted equality and range matching.
Open-ended search is not there yet. Brave promises zero retention on its own 40 billion-page index, but that remains a protocol promise. Exa built a neural index for semantic ranking, yet the embedding step still starts from plaintext on Exa’s servers. MIT’s 2023 Tiptoe paper ranked results across 360 million webpages without exposing the query, but each search consumed substantial server compute and quality still trailed unencrypted search. Apple’s 2024 Wally paper reduced communication costs by up to 31x by hiding the real query among decoys, though IOSG says the economics only improve at millions of concurrent queries, a scale no private search system currently has.
Demand is rising, but it remains small versus mainstream AI
IOSG says Venice AI recently passed 3.5 million registered users and 1.3 trillion monthly tokens, then raised a Series A equity round at a $1 billion valuation. Proton, described as its direct competitor, saw its Lumo chat product reach 10 million users within a year of launch.
On infrastructure, Phala is already processing 2 billion to 3 billion tokens a day on OpenRouter alone. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil enclaves, giving users verifiable privacy beyond a proxy model. Self-hosting is not included in these figures and may be the largest channel of all, since models run on the user’s own hardware without leaving usage traces.
Yet private AI is still tiny compared with the broader market. Google processed 3.2 quadrillion tokens across its products in May. On that basis, Venice’s monthly throughput equals about 18 minutes of Google volume.
Google introduced Private AI Compute, or PAC, in November last year, placing some Gemini-powered functions inside sealed TPU enclaves isolated from Google itself, with design reviewed independently by NCC Group. But IOSG says PAC covers only a limited set of Pixel functions such as personalized recommendations and recording summaries, not the Gemini app used by hundreds of millions of people.
What exists now, and what still does not
The report’s closing view is measured. Hosted privacy systems are imperfect. E2EE users who want the strongest guarantees still have to wait while features are rebuilt in places providers cannot read. Private harness layers still depend on protocol controls. Cost-effective post-training still requires trust in outside vendors if the goal is top-end tuning results. Self-hosting removes providers from the loop, but running the strongest open models locally can be extremely expensive.
Still, IOSG argues private AI has become a real and affordable option. On the consumer side, Lumo and Venice already offer no-log private chat with open models at no cost. Venice and Tinfoil subscriptions priced at $18 to $20 put those chats inside enclaves, roughly around the price of a ChatGPT subscription. For enterprise workflows, attested endpoints can already be cheaper than plaintext routes. IOSG says endpoints such as NEAR’s E2EE API can carry encrypted context into enclaves today, supporting memory, file uploads and custom instructions.
Training infrastructure is moving as well. NVIDIA’s upcoming Vera Rubin NVL72 will extend confidential computing from Blackwell’s eight-GPU nodes to 72-GPU racks, making frontier RL loops more practical without exposing IP.
The report does not argue that every workload should move at once. For highly execution-heavy, tool-rich agentic tasks, it leans toward trust, because each tool call already reveals plaintext to destinations no enclave can protect. For strategic thinking, planning and judgments distilled from years of professional experience, it leans toward verification: fine-tuning open models with proprietary insight inside boundaries the company controls.
That is the report’s central claim. In the domains where a company’s alpha lives, expert-tuned open models have already shown they can beat frontier systems on both accuracy and cost, and the infrastructure needed to build them privately is arriving one node at a time.