Redis has introduced LangCache, a managed semantic caching service now in public preview on Redis Cloud. Instead of checking whether prompts are identical at the text level, the service uses semantic matching to identify similar requests. If a match is found, the cached response is returned and the application can skip a fresh large language model call. Redis said this can reduce API costs by as much as 90% and improve response speed by up to 15x on cache hits.
LangCache works in a two-call loop. Before sending a prompt to an LLM, the application first sends it to LangCache for vector similarity search. If a semantically similar cached entry exists, LangCache returns that response directly. If not, the application proceeds with the LLM call, then stores the prompt and response pair for future reuse. The service includes built-in embedding model generation and also supports user-supplied models.
Redis said LangCache is available through a REST API and comes with Python and JavaScript SDKs. Users can configure similarity thresholds, TTL, and eviction policies, while cache hit rates and cost savings can be tracked in the Redis Cloud console. Customer Mangoes.ai reported a 70% cache hit rate in its patient care application.
Redis has launched LangCache, a managed semantic caching service that is now in public preview on Redis Cloud.
The service identifies similar prompts through semantic matching rather than text matching. When a request hits the cache, an application can skip a large language model call, cutting API spending and returning results faster. Redis said LangCache can reduce API costs by up to 90%, while response speed on cache hits can improve by as much as 15x.
How LangCache works
LangCache runs on a two-call loop. Before an application sends a prompt to an LLM, it first sends that prompt to LangCache for vector similarity search.
- If LangCache finds a semantically similar cached entry, it returns the cached response directly and no LLM call is made.
- If there is no match, the application proceeds with the normal LLM request, then stores the prompt and response in the cache for later use.
The service includes built-in embedding model generation and also supports bring-your-own models.
Access and configuration
LangCache is available through a REST API and also offers Python and JavaScript SDKs. Cache behavior can be configured through settings such as similarity thresholds, TTL, and eviction policies.
Users can monitor cache hit rates and cost savings in the Redis Cloud console.
Customer example
According to the announcement cited by MarkTechPost, customer Mangoes.ai reported that its patient care application has reached a 70% cache hit rate.
This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan. Disclaimer:
The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.
Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.
