Shypd.aiThe largest open index of AI toolsLLMLingua is an AI Frameworks & Infra tool that compresses prompts and KV-caches for Large Language Models. It achieves up to 20x compression with minimal performance loss, speeding up inference and enhancing key information perception.
Also listed in
About
LLMLingua is a powerful tool designed to optimize the performance of Large Language Models (LLMs) by significantly compressing prompts and KV-caches. This innovative approach, detailed in research papers presented at EMNLP'23 and ACL'24, allows for up to 20x compression while maintaining minimal performance degradation. The tool helps overcome common LLM challenges such as token limits, high API costs, and the 'lost in the middle' issue for long contexts. LLMLingua offers various versions, including LongLLMLingua for enhanced long-context processing and LLMLingua-2 for faster, task-agnostic compression. It also features SecurityLingua, a safety guardrail model for detecting and mitigating jailbreak attacks through security-aware prompt compression. The tool is integrated into popular frameworks like LangChain and LlamaIndex, making it accessible for developers working on LLM-based applications.
Capabilities
Pricing & Plans
Open Source
Free
