Shypd.aiThe largest open index of AI toolsMooncake is an open-source DevOps & Infrastructure tool that serves as a KVCache-centric disaggregated platform for large language models. It optimizes LLM serving performance and efficiency, particularly for long-context scenarios.
Trending
About
Mooncake is an open-source serving platform designed for large language models (LLMs), notably powering Kimi by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates prefill and decoding clusters, leveraging underutilized CPU, DRAM, and SSD resources to implement a disaggregated KVCache pool. The core component is the Transfer Engine (TE), which provides a unified interface for batched data transfer across various storage devices and network links, supporting multiple protocols like TCP, RDMA, and NVMe over Fabric. Mooncake also includes P2P Store for sharing temporary objects and Mooncake Store for distributed pooled KVCache, enhancing resource utilization and system performance. It integrates with leading LLM inference systems like vLLM and SGLang for prefill-decode disaggregation and hierarchical KV caching, significantly improving inference efficiency for large-scale distributed tasks.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
