Shypd.aiThe largest open index of AI toolsAudioCLIP extends the CLIP model to handle audio in addition to text and images, enabling bimodal and unimodal classification and querying. It achieves state-of-the-art results in environmental sound classification.
Trending
Also listed in
About
AudioCLIP is an advanced AI model that expands the capabilities of the Contrastive Language-Image Pre-training (CLIP) framework to include audio processing. This innovative extension allows for joint representation learning across image, text, and audio modalities, facilitating tasks such as bimodal and unimodal classification and querying. Built upon prior research in robust time-frequency transformation of audio and environmental sound classification, AudioCLIP integrates the ESResNeXt audio-model with the CLIP framework using the AudioSet dataset. This combination enables the model to generalize to unseen datasets in a zero-shot inference fashion, achieving new state-of-the-art results in Environmental Sound Classification (ESC) tasks on datasets like UrbanSound8K and ESC-50.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
