Shypd.aiThe largest open index of AI toolsPunica is an open-source coding and development tool that serves multiple LoRA finetuned LLMs as one. It enables efficient execution of numerous LoRA models with minimal overhead, achieving up to 12x throughput compared to state-of-the-art systems.
Trending
About
Punica is an open-source project designed to efficiently serve multiple LoRA (Low-Rank Adaptation) finetuned Large Language Models (LLMs) simultaneously. While pretrained LLMs can be massive, LoRA finetuned models add only about 1% storage and memory overhead. Punica leverages a custom CUDA kernel, called Segmented Gather Matrix-Vector multiplication (SGMV), to efficiently compute the LoRA addon, preserving the strong batching effect of the pretrained model. This approach allows Punica to achieve significantly higher throughput, up to 12 times, compared to other systems like HuggingFace Transformers, DeepSpeed, FasterTransformer, and vLLM, across various LoRA model popularity settings. It can be installed from binary packages or built from source, supporting different CUDA, Python, and PyTorch versions.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
