ER-NeRF

Visit Tool

ER-NeRF is a video generation tool that synthesizes high-fidelity talking portraits. It leverages efficient region-aware neural radiance fields for realistic video output.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is ER-NeRF?

ER-NeRF is an open-source project providing Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis, as presented at ICCV 2023. This tool is designed for computer vision and graphics research, enabling users to generate realistic talking portraits from input videos and audio. It includes functionalities for processing custom training videos, extracting facial features like AU45 for eye blinking, and pre-processing audio using DeepSpeech, Wav2Vec, or HuBERT models. The repository offers detailed instructions for installation, data preparation, training, and testing, supporting both head-only and head-plus-torso synthesis. It also allows for inference with target audio, making it a comprehensive solution for advanced talking portrait generation.

Best used for

Ideal for researchers and developers who need to synthesize high-fidelity talking portraits, process custom training videos, and extract detailed facial and audio features. Especially valuable for academic projects and advanced applications in computer vision and graphics.

Common actions

synthesize talking portraits

generate realistic videos

process facial features

extract audio features

conduct computer vision research

open-sourceautomated workflowcollaborationworkflowsdeepfakelow-code/no-codegithub copilot"AI Agents"face swapping

Capabilities

Key features

High-fidelity talking portrait synthesis
Region-aware neural radiance fields
Custom video processing
AU45 extraction for blinking
DeepSpeech/Wav2Vec/HuBERT audio processing
Head and torso synthesis
Inference with target audio

Target Audience

professor

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What are the system requirements for running ER-NeRF?

ER-NeRF has been tested on Ubuntu 18.04 with Pytorch 1.12 and CUDA 11.3. It requires specific Python packages and dependencies, including PyTorch3D and TensorFlow-GPU, which are detailed in the installation guide on the GitHub repository.

Can ER-NeRF process custom videos for talking portrait synthesis?

Yes, ER-NeRF supports custom training videos. Users need to place their 25FPS video (512x512 resolution, 1-5 min duration) in the specified data directory and run a processing script. This prepares the video for subsequent training and synthesis.

What audio feature models does ER-NeRF support for synthesis?

ER-NeRF supports multiple audio feature models for evaluation and synthesis, including DeepSpeech, Wav2Vec (like RAD-NeRF), and HuBERT (borrowed from GeneFace). Users can specify their preferred model during training and testing for optimal results.

Trending

Subcategories trending in Content & Design

Image Generation AI Writing Assistants Audio & Music Photo Editing Graphic Design Video Editing

Trending

Also listed in

This tool also appears in

Research & Education › Academic Research Coding & Development › Open Source & Models

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce