Shimmy

Visit Tool

shimmy is an AI Frameworks & Infra tool that provides a Python-free Rust inference server. It is OpenAI-API compatible, supports GGUF and SafeTensors, and offers hot model swapping and auto-discovery in a single binary.

Claim this tool

1View

At a glance

Pricing

Free · Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is shimmy?

shimmy is a Python-free Rust inference server designed for local AI model deployment, offering full OpenAI API compatibility. It supports GGUF and SafeTensors models, enabling users to point existing AI tools to shimmy for local, private, and free inference. Key features include hot model swapping, automatic model discovery from Hugging Face cache, Ollama, and local directories, and zero configuration. It also boasts advanced Mixture of Experts (MOE) support for running large models (70B+) on consumer hardware through intelligent CPU/GPU hybrid processing. shimmy is distributed as a single binary, including all GPU backends for various platforms, simplifying installation and ensuring automatic GPU detection.

Best used for

Ideal for developers who need to deploy and run large language models locally, integrate with existing OpenAI-compatible applications, and optimize inference performance on consumer hardware. Especially valuable for ensuring privacy, reducing costs, and achieving sub-second response times without external API dependencies.

Common actions

deploy LLM locally

run AI inference

integrate OpenAI API

manage AI models

optimize GPU usage

face swapping"AI Agents"github copilotdeepfakeautomated workflowopen-sourcelow-code/no-codeworkflowscollaboration

Capabilities

Key features

Python-free Rust server
OpenAI API compatible
GGUF + SafeTensors support
Hot model swap
Auto-discovery models
Single binary distribution
MOE CPU/GPU offloading

Target Audience

developer

Integrations

Not yet documented

Pricing & Plans

Free · Open Source

Free

FAQs

What kind of models does shimmy support?

shimmy supports GGUF and SafeTensors models, providing 100% OpenAI-compatible endpoints. It can automatically discover models from Hugging Face cache, Ollama, and local directories, making it versatile for various local inference needs.

Does shimmy require Python or other dependencies?

No, shimmy is a Python-free Rust inference server. It is distributed as a single binary that includes all necessary GPU backends, eliminating the need for complex installations or external dependencies like Python.

How does shimmy handle GPU acceleration and large models?

shimmy includes automatic GPU detection for CUDA, Vulkan, OpenCL, and MLX. It also features advanced Mixture of Experts (MOE) support, allowing it to run 70B+ models on consumer hardware by intelligently offloading layers between CPU and GPU for optimal performance and memory efficiency.

Trending

Subcategories trending in AI Agents & Automation

Chatbots & Conversational AI General-Purpose Agents Workflow Agents Personal Assistants RAG & Document AI Voice Agents

Trending

Also listed in

This tool also appears in

Coding & Development › Backend & APIs Coding & Development › Code Assistants Coding & Development › Open Source & Models

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce