Shypd.aiThe largest open index of AI toolsSqueezeLLM is an AI Frameworks & Infra tool that provides dense-and-sparse quantization for efficient LLM serving. It enables deployment of large language models with reduced memory footprint and improved accuracy.
Trending
About
SqueezeLLM is a post-training quantization framework designed to optimize the serving of large language models (LLMs) through a novel Dense-and-Sparse Quantization method. This approach addresses the significant memory requirements of LLMs by splitting weight matrices into a dense component, which can be heavily quantized without performance loss, and a sparse component that preserves sensitive outlier parts. This allows for serving larger models with a smaller memory footprint, maintaining the same latency, and achieving higher accuracy and quality compared to baseline models. For instance, SqueezeLLM's variant of Vicuna models can operate within 6 GB of memory, surpassing FP16 baseline models in MMLU accuracy despite the latter requiring twice the memory. The framework supports various LLMs including LLaMA, LLaMA-2, Mistral, Vicuna, XGen, and OPT, with options for 3-bit and 4-bit quantization and different sparsity levels.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
Also listed in
