Delayed-Streams-Modeling

Visit Tool

delayed-streams-modeling provides Kyutai's Speech-To-Text and Text-To-Speech models based on the Delayed Streams Modeling framework. It offers real-time streaming inference and word-level timestamps for voice AI systems.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is delayed-streams-modeling?

delayed-streams-modeling offers Kyutai's advanced Speech-To-Text (STT) and Text-To-Speech (TTS) models, built upon the innovative Delayed Streams Modeling framework. These models are optimized for real-time usage, supporting streaming inference and efficient batch processing, making them ideal for interactive applications. Key features include word-level timestamps and a semantic Voice Activity Detection (VAD) component in the 1B STT model, useful for building responsive voice agents. The repository provides flexible implementations in PyTorch for research, Rust for production-grade servers with websocket access, and MLX for on-device inference on Apple silicon, catering to diverse development and deployment needs.

Best used for

Ideal for developers and researchers who need to implement real-time speech-to-text and text-to-speech functionalities, build interactive voice agents, and deploy models on various platforms including Apple silicon. Especially valuable for creating responsive and efficient voice AI systems.

Common actions

transcribe speech

generate speech

build voice agents

process audio streams

github copilotface swappingworkflows"AI Agents"deepfakecollaborationautomated workflowopen-sourcelow-code/no-code

Capabilities

Key features

Speech-To-Text models
Text-To-Speech models
Streaming inference
Word-level timestamps
Semantic VAD
PyTorch implementation
Rust server

Target Audience

developersmachine learning engineersai researchers

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What is Delayed Streams Modeling?

Delayed Streams Modeling is a technique for solving streaming X-to-Y tasks (with X, Y in {speech, text}) that formalizes the approach used in previous models like Moshi and Hibiki. It enables efficient, real-time processing for speech and text applications.

What are the key advantages of Kyutai STT models?

Kyutai STT models offer streaming inference for real-time transcription, easy batching for maximum efficiency (e.g., 400 streams on an H100), and return word-level timestamps. The 1B model also includes a semantic VAD for detecting when a user is speaking.

Which implementation should I use for Kyutai STT/TTS?

For research and experimentation, use the PyTorch implementation. For production environments requiring robust, scalable servers, the Rust implementation is recommended. For on-device inference on Apple silicon (iPhone/Mac), choose the MLX implementation.

Are there different models available for Speech-To-Text?

Yes, Kyutai provides two STT models: 'kyutai/stt-1b-en_fr' (English and French, ~1B parameters, 0.5s delay, semantic VAD) and 'kyutai/stt-2.6b-en' (English-only, ~2.6B parameters, 2.5s delay).

What licenses apply to this project?

The Python parts and web client code are provided under the MIT license, while the Rust backend is under the Apache license. The weights for the speech-to-text models are released under the CC-BY 4.0 license.

Trending

Subcategories trending in Content & Design

Image Generation AI Writing Assistants Video Generation Photo Editing Graphic Design Video Editing

Trending

Also listed in

This tool also appears in

AI Agents & Automation › Voice Agents Customer Support & CX › Voice & Call Center

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce