Parallel, an AI search company, used real cases from its own search business to test eight decision models, including Jev, Perplexity and OpenAI Decisions. The benchmark covered four tasks: deciding whether webpage changes matter, filtering low-quality content, choosing a search index and ranking search results.
Using the original prompts, Jev ranked first on average across the four tasks. Perplexity came in second, while OpenAI ranked fifth.
Webpage-change detection was the hardest task
According to the test, the most difficult task was deciding whether a webpage change was important enough to reprocess. A job posting that only rearranges two requirements may not need an update. A change in leaderboard rankings, by contrast, may justify a fresh crawl.
The test used a combined metric that weighs both false positives and false negatives, with higher scores indicating better performance. In the initial results, Perplexity scored 93.3, Jev scored 87.9 and OpenAI scored 79.0.
Claude-assisted prompt tuning lifted several models
Parallel then used Claude to improve prompts for the models. In the webpage-change task, the team first prepared 800 manually labeled examples, asked Claude to inspect incorrect model judgments and added rules for cases where the models were prone to errors. The process ran for five rounds, and only the better-performing version from each round was kept.
On an independent test set, OpenAI improved from 79.0 to 88.8 in webpage-change detection. Rune 26B rose from 74.1 to 83.9, and Nimble 9B increased from 65.1 to 75.3. All three models gained about 10 points.
When averaging three tasks together — webpage-change detection, low-quality content filtering and query routing — Perplexity improved from 97.3 to 97.6, Jev from 95.5 to 96.5 and OpenAI from 88.7 to 95.1.
Parallel said all eight models posted higher average scores after prompt optimization, but the gains were limited for the models that were already ahead. OpenAI, however, narrowed the gap significantly and moved into third place after tuning.
Ranking task was measured separately
The search-ranking task was calculated on its own. After prompt optimization, Jev reached a 71.45% overlap rate between its top-ranked search results and the human-labeled results. Perplexity posted 70.68%.
Parallel also said a small ranking model it fine-tuned in-house reached 75.21%, about 3.8 percentage points higher than Jev. Its inference cost per token was also only about one-quarter of Jev’s.
The trade-off, according to Parallel, is that this approach requires collecting labeled data, training the model and deploying it independently. For large volumes of repeated fixed tasks, a specially trained small model may still be more cost-effective.