Ctransformers

Visit Tool

ctransformers provides Python bindings for Transformer models implemented in C/C++ using the GGML library, enabling efficient local execution of large language models. It supports integration with Hugging Face Transformers and LangChain, offering GPU acceleration for improved performance.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is ctransformers?

ctransformers is a Python library designed to provide efficient bindings for Transformer models, leveraging C/C++ implementations and the GGML library. This allows for optimized local execution of various large language models, including GPT-2, LLaMA, Falcon, and more. The library offers a unified interface for model loading and generation, supporting both local files and Hugging Face Hub repositories. It integrates seamlessly with popular AI frameworks like Hugging Face Transformers and LangChain, making it versatile for different development workflows. ctransformers also provides GPU acceleration for compatible models, with support for CUDA, Metal, and ROCm, and includes experimental GPTQ quantization support for LLaMA models, enhancing performance and reducing memory footprint.

Best used for

Ideal for developers and data scientists who need to efficiently run large language models locally, integrate them into Python applications, and leverage GPU acceleration for faster inference. Especially valuable for those working with Hugging Face Transformers or LangChain and seeking optimized performance for various model types.

Common actions

run transformer models

integrate LLMs

accelerate AI inference

develop AI applications

collaborationopen-sourceworkflows"AI Agents"low-code/no-codegithub copilotautomated workflowface swappingdeepfake

Capabilities

Key features

Python bindings
GGML library integration
Hugging Face Hub support
GPU acceleration
LangChain integration
GPTQ support

Target Audience

developerdata scientist

Integrations

hugging-facelangchain

Pricing & Plans

Open Source

Free

FAQs

What Transformer models are supported by ctransformers?

ctransformers supports a variety of Transformer models including GPT-2, GPT-J, GPT-NeoX, StableLM, Falcon, LLaMA, LLaMA 2, MPT, StarCoder, StarChat, Dolly V2, and Replit. It provides a unified interface for these models, allowing for flexible integration into projects.

How can I enable GPU acceleration with ctransformers?

GPU acceleration can be enabled by installing specific CUDA, ROCm, or Metal libraries. For CUDA, use `pip install ctransformers[cuda]`. For ROCm, set `CT_HIPBLAS=1` before installing. For Metal, set `CT_METAL=1` before installing. You can then specify `gpu_layers` when loading a model.

Does ctransformers support quantized models like GPTQ?

Yes, ctransformers offers experimental support for GPTQ quantized models, specifically for LLaMA models using ExLlama. To use this feature, you need to install additional dependencies via `pip install ctransformers[gptq]` and then load the GPTQ model.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Also listed in

This tool also appears in

AI Agents & Automation › AI Frameworks & Infra Research & Education › Scientific Computing

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce