Shypd.aiThe largest open index of AI toolsPaddler is an open-source LLM/VLM load balancer and serving platform that allows self-hosting of large language models at scale. It provides a simple deployment alternative to other LLM serving solutions, running on both CPU and GPU.
Trending
About
Paddler is an open-source load balancer and serving platform designed for self-hosting Large Language Models (LLMs) and Vision Language Models (VLMs) at scale. It offers a streamlined alternative to existing solutions like llm-d or Docker Model Runner, focusing on fewer moving parts and simpler deployments built around the ggml ecosystem. Key features include a built-in llama.cpp engine for inference, LLM-specific load balancing, dynamic model swapping, and request buffering for scaling from zero hosts. Paddler also provides a web admin panel for management, monitoring, and testing, along with observability metrics. It runs efficiently on both CPU and GPU, catering to product teams needing LLM inference, DevOps/LLMOps teams deploying models at scale, and organizations with high compliance and privacy requirements.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
Also listed in
