Shypd.aiThe largest open index of AI toolsLMDeploy is an open-source toolkit for compressing, deploying, and serving Large Language Models (LLMs). It offers efficient inference, effective quantization, and effortless distribution server capabilities for LLMs and VLMs.
Trending
About
LMDeploy is a comprehensive open-source toolkit developed by the MMRazor and MMDeploy teams, designed for compressing, deploying, and serving Large Language Models (LLMs) and Vision Language Models (VLMs). It boasts core features such as efficient inference, achieving up to 1.8x higher request throughput than vLLM through innovations like persistent batching, blocked KV cache, and high-performance CUDA kernels. The toolkit also provides effective quantization, supporting weight-only and k/v quantization, with 4-bit inference performance 2.4x higher than FP16. Furthermore, LMDeploy offers an effortless distribution server, enabling easy deployment of multi-model services across multiple machines and cards. It supports a wide range of LLMs and VLMs, including popular models like Llama, InternLM, Qwen, and DeepSeek, and features two inference engines: TurboMind for ultimate performance optimization and PyTorch for developer accessibility and rapid experimentation.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
