Spaces:
Paused
Paused
| license: apache-2.0 | |
| title: audio2kineticvid | |
| sdk: gradio | |
| emoji: 🚀 | |
| colorFrom: red | |
| colorTo: yellow | |
| # Audio2KineticVid | |
| Audio2KineticVid is a comprehensive tool that converts an audio track (e.g., a song) into a dynamic music video with AI-generated scenes and synchronized kinetic typography (animated subtitles). Everything runs locally using open-source models – no external APIs or paid services required. | |
| ## ✨ Features | |
| - **🎤 Whisper Transcription:** Choose from multiple Whisper models (tiny to large) for audio transcription with word-level timestamps. | |
| - **🧠 Adaptive Lyric Segmentation:** Splits lyrics into segments at natural pause points to align scene changes with the song. | |
| - **🎨 Customizable Scene Generation:** Use various LLM models to generate scene descriptions for each lyric segment, with customizable system prompts and word limits. | |
| - **🤖 Multiple AI Models:** Select from a variety of text-to-image models (SDXL, SD 1.5, etc.) and video generation models. | |
| - **🎬 Style Consistency Options:** Choose between independent scene generation or img2img-based style consistency for a more cohesive visual experience. | |
| - **🔍 Preview & Inspection:** Preview scenes before full generation and inspect all generated images in a gallery view. | |
| - **🔄 Seamless Transitions:** Configurable crossfade transitions between scene clips. | |
| - **🎪 Kinetic Subtitles:** PyCaps renders styled animated subtitles that appear in sync with the original audio. | |
| - **🔒 Fully Local & Open-Source:** All models are open-license and run on local GPU. | |
| ## 💻 System Requirements | |
| ### Hardware Requirements | |
| - **GPU**: NVIDIA GPU with 8GB+ VRAM (recommended: RTX 3080/4070 or better) | |
| - **RAM**: 16GB+ system RAM | |
| - **Storage**: SSD recommended for faster model loading and video processing | |
| - **CPU**: Modern multi-core processor | |
| ### Software Requirements | |
| - **Operating System**: Linux, Windows, or macOS | |
| - **Python**: 3.8 or higher | |
| - **CUDA**: NVIDIA CUDA toolkit (for GPU acceleration) | |
| - **FFmpeg**: For audio/video processing | |
| ## 🚀 Quick Start (Gradio Web UI) | |
| ### 1. Install Dependencies | |
| Ensure you have a suitable GPU (NVIDIA T4/A10 or better) with CUDA installed. Then install the required Python packages: | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ### 2. Launch the Web Interface | |
| ```bash | |
| python app.py | |
| ``` | |
| This will start a Gradio web interface accessible at `http://localhost:7860`. | |
| ### 3. Using the Interface | |
| 1. **Upload Audio**: Choose an audio file (MP3, WAV, M4A, etc.) | |
| 2. **Select Quality Preset**: Choose from Fast, Balanced, or High Quality | |
| 3. **Configure Models**: Optionally adjust AI models in the "AI Models" tab | |
| 4. **Customize Style**: Modify scene prompts and visual style in other tabs | |
| 5. **Preview**: Click "Preview First Scene" to test settings quickly | |
| 6. **Generate**: Click "Generate Complete Music Video" to create the full video | |
| ## 📝 Usage Tips | |
| ### Audio Selection | |
| - **Format**: MP3, WAV, M4A, FLAC, OGG supported | |
| - **Quality**: Clear vocals work best for transcription | |
| - **Length**: 30 seconds to 3 minutes recommended for testing | |
| - **Content**: Songs with distinct lyrics produce better results | |
| ### Performance Optimization | |
| - **Fast Generation**: Use 512x288 resolution with "tiny" Whisper model | |
| - **Best Quality**: Use 1280x720 with "large" Whisper model (requires more VRAM) | |
| - **Memory Issues**: Lower resolution, use smaller models, or reduce max segments | |
| ### Style Customization | |
| - **Visual Style Keywords**: Add style terms like "cinematic, vibrant, neon" to influence all scenes | |
| - **Prompt Template**: Customize how the AI interprets lyrics into visual scenes | |
| - **Consistency Mode**: Use "Consistent (Img2Img)" for coherent visual style across scenes | |
| ## 🛠️ Advanced Usage | |
| ### Command Line Interface | |
| For batch processing or automation, you can use the smoke test script: | |
| ```bash | |
| bash scripts/smoke_test.sh your_audio.mp3 | |
| ``` | |
| ### Custom Templates | |
| Create custom subtitle styles by adding new templates in the `templates/` directory: | |
| 1. Create a new folder: `templates/your_style/` | |
| 2. Add `pycaps.template.json` with animation definitions | |
| 3. Add `styles.css` with visual styling | |
| 4. The template will appear in the interface dropdown | |
| ### Model Configuration | |
| Supported models are defined in the utility modules: | |
| - **Whisper**: `utils/transcribe.py` - Add new Whisper model names | |
| - **LLM**: `utils/prompt_gen.py` - Add new language models | |
| - **Image**: `utils/video_gen.py` - Add new Stable Diffusion variants | |
| - **Video**: `utils/video_gen.py` - Add new video diffusion models | |
| ## 🧪 Testing | |
| Run the basic functionality test: | |
| ```bash | |
| python test_basic.py | |
| ``` | |
| For a complete end-to-end test with a sample audio file: | |
| ```bash | |
| python test.py | |
| ``` | |
| ## 📁 Project Structure | |
| ``` | |
| Audio2KineticVid/ | |
| ├── app.py # Main Gradio web interface | |
| ├── requirements.txt # Python dependencies | |
| ├── utils/ # Core processing modules | |
| │ ├── transcribe.py # Whisper audio transcription | |
| │ ├── segment.py # Intelligent lyric segmentation | |
| │ ├── prompt_gen.py # LLM scene description generation | |
| │ ├── video_gen.py # Image and video generation | |
| │ └── glue.py # Video stitching and subtitle overlay | |
| ├── templates/ # Subtitle animation templates | |
| │ ├── minimalist/ # Clean, simple subtitle style | |
| │ └── dynamic/ # Dynamic animations | |
| ├── scripts/ # Utility scripts | |
| │ └── smoke_test.sh # End-to-end testing script | |
| └── test_basic.py # Component testing | |
| ``` | |
| ## 🎬 Output | |
| The application generates: | |
| - **Final Video**: MP4 file with synchronized audio, visuals, and animated subtitles | |
| - **Scene Images**: Individual AI-generated images for each lyric segment | |
| - **Scene Descriptions**: Text prompts used for image generation | |
| - **Segmentation Data**: Analyzed lyric segments with timing information | |
| ## 🔧 Troubleshooting | |
| ### Common Issues | |
| **GPU Memory Errors** | |
| - Reduce video resolution (use 512x288 instead of 1280x720) | |
| - Use smaller models (tiny/base Whisper, SD 1.5 instead of SDXL) | |
| - Close other GPU-intensive applications | |
| **Audio Processing Fails** | |
| - Ensure FFmpeg is installed and accessible | |
| - Try converting audio to WAV format first | |
| - Check that audio file is not corrupted | |
| **Model Loading Issues** | |
| - Check internet connection (models download on first use) | |
| - Verify sufficient disk space for model files | |
| - Clear HuggingFace cache if models are corrupted | |
| **Slow Generation** | |
| - Use "Fast" quality preset for testing | |
| - Reduce crossfade duration to 0 for hard cuts | |
| - Use dynamic FPS instead of fixed high FPS | |
| ### Performance Monitoring | |
| Monitor system resources during generation: | |
| - **GPU Usage**: Should be near 100% during image/video generation | |
| - **RAM Usage**: Peak during model loading and video processing | |
| - **Disk I/O**: High during model downloads and video encoding | |
| ## 🤝 Contributing | |
| Contributions are welcome! Areas for improvement: | |
| - Additional subtitle animation templates | |
| - Support for more AI models | |
| - Performance optimizations | |
| - Additional audio/video formats | |
| - Batch processing capabilities | |
| ## 📄 License | |
| This project uses open-source models and libraries. Please check individual model licenses for usage rights. | |
| ## 🙏 Acknowledgments | |
| - **OpenAI Whisper** for speech recognition | |
| - **Stability AI** for Stable Diffusion models | |
| - **Hugging Face** for model hosting and transformers | |
| - **PyCaps** for kinetic subtitle rendering | |
| - **Gradio** for the web interface |