Llm-Compressor

Visit Tool

llm-compressor is an open-source library that optimizes Large Language Models for deployment with vLLM. It offers a comprehensive set of quantization algorithms for weight-only and activation quantization, seamlessly integrating with Hugging Face models.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is llm-compressor?

llm-compressor is a powerful, open-source library designed to optimize Large Language Models (LLMs) for efficient deployment, particularly with vLLM. It provides a comprehensive suite of quantization algorithms, supporting both weight-only and activation quantization, including advanced schemes like FP8, MXFP8, and NVFP4. The library ensures seamless integration with Hugging Face models and repositories, utilizing a safetensors-based file format compatible with vLLM. Key features include support for large models via accelerate, distributed GPTQ for faster calibration, and updated offloading capabilities for very large models that exceed CPU memory. It also introduces a model-free PTQ pathway for quantization without Hugging Face model definitions, making it versatile for various optimization needs.

Best used for

Ideal for developers and data scientists who need to optimize Large Language Models for efficient deployment, reduce model size, and accelerate inference. Especially valuable for those working with vLLM and Hugging Face models, requiring advanced quantization techniques like FP8 or NVFP4.

Common actions

optimize LLMs

compress models

quantize models

deploy LLMs

automated workflowworkflowsdeepfakelow-code/no-codecollaboration"AI Agents"face swappinggithub copilotopen-source

Capabilities

Key features

Quantization algorithms
Hugging Face model integration
safetensors file format
Large model support
Distributed GPTQ
Model-free PTQ pathway
KV Cache quantization

Target Audience

developerdata scientist

Integrations

hugging-facevllm

Pricing & Plans

Open Source

Free

FAQs

What types of quantization algorithms does llm-compressor support?

llm-compressor supports a comprehensive set of quantization algorithms including weight-only and activation quantization. This includes W8A8 (int8 and fp8), MXFP8, W4A16, W8A16, MXFP8A16, and NVFP4. It also supports algorithms like GPTQ, AWQ, SmoothQuant, and AutoRound.

Can llm-compressor be used with very large models that don't fit in CPU memory?

Yes, llm-compressor supports offloading functionality to handle very large models. This allows users to load and compress models that would normally exceed CPU memory, facilitating optimization for even the largest LLMs. This functionality is now handled through model loading utilities in compressed-tensors.

How does llm-compressor integrate with vLLM?

llm-compressor is designed for optimized deployment with vLLM. The checkpoints created by llm-compressor, using its safetensors-based file format, can be directly loaded and run within the vLLM inference engine, ensuring a seamless workflow for deploying compressed models.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce