Keenable AI open-sources NEEDLE, a live search benchmark that rebuilds queries every hour

Keenable AI has released NEEDLE, an open-source benchmark for real-time search systems. Instead of relying on a fixed evaluation dataset, NEEDLE rebuilds its query set every hour using public sources including RSS feeds and Google Trends. The stated aim is to reduce distortions that can appear when models read answers directly from static benchmarks or rely on parameter memory. The benchmark covers several query categories: news, finance, academic, legal, and rare-entity lookups. It evaluates 15 search APIs under a unified protocol and adds a "ceiling" metric designed to capture the best retrievable result available across the field. NEEDLE is available as an open-source evaluation tool that can be installed and run through a Python CLI. Judging requires an OpenRouter key, and the tool supports use on laptops or in CI environments. Results cited in the report show finance queries are close to being solved. Exa scored 0.910, followed by Keenable at 0.872, Perplexity at 0.871, and Google at 0.847. The reported "ceiling" score was 0.965. The item was cited by MarkTechPost.