Trending Model:#1Inklingthinkingmachines⬇13kTrending Model:#2Ternary-Bonsai-27B-ggufprism-ml⬇339kTrending Model:#3Bonsai-27B-ggufprism-ml⬇1263kTrending Model:#4Unlimited-OCRbaidu⬇2123kTrending Model:#5GLM-5.2zai-org⬇532kTrending Model:#6Qwythos-9B-Claude-Mythos-5-1M-GGUFempero-ai⬇2117kTrending Model:#7krea2-identity-editconradlocke⬇0kTrending Model:#8Qwen3.6-35B-A3B-Uncensored-HauhauCS-AggressiveHauhauCS⬇2007kTrending Model:#9Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUFDavidAU⬇17kTrending Model:#10OvisOCR2ATH-MaaS⬇15kTrending Model:#1Inklingthinkingmachines⬇13kTrending Model:#2Ternary-Bonsai-27B-ggufprism-ml⬇339kTrending Model:#3Bonsai-27B-ggufprism-ml⬇1263kTrending Model:#4Unlimited-OCRbaidu⬇2123kTrending Model:#5GLM-5.2zai-org⬇532kTrending Model:#6Qwythos-9B-Claude-Mythos-5-1M-GGUFempero-ai⬇2117kTrending Model:#7krea2-identity-editconradlocke⬇0kTrending Model:#8Qwen3.6-35B-A3B-Uncensored-HauhauCS-AggressiveHauhauCS⬇2007kTrending Model:#9Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUFDavidAU⬇17kTrending Model:#10OvisOCR2ATH-MaaS⬇15k

Xiaomi Research Orchestrates ControlFoley For Video Soundtracks

By vramkickedin | | 2 min read
A combined waveform is uniquely constructed from dozens of tiny interconnected rectangular film frames and small floating translucent text bubbles.

ControlFoley transforms video clips into synchronized soundtracks by combining visual scenes, written descriptions, and existing audio samples into a single generation system. This new framework produces matching sound effects and background elements while following specific creative directions from multiple inputs simultaneously.

Xiaomi Research developed the project to solve a common issue where generated audio clashes with on-screen action or ignores written instructions. Independent creators and production teams can now run reliable audio synthesis directly on their own machines without relying on external cloud services.

Model Size: varied & VRAM GPU: requirements vary

Multimodal audio generation

  • Generates synchronized audio using video footage combined with text prompts or reference sound clips.
  • Resolves conflicting instructions by prioritizing written commands while keeping sound aligned with visual timing.
  • Separates acoustic style from pacing to match the exact tone of uploaded reference tracks.
  • Includes a built-in test suite to measure how well the system handles mismatched video and text pairs.

Independent filmmakers can draft complete soundscapes without leaving their editing environment. Archival specialists also test audio directions while keeping raw footage completely offline.

Development approach and performance data

The developers addressed a frequent limitation in existing synthesis tools by training the system with random input removal and a unified alignment process. This method teaches the software to maintain quality even when some guidance is missing.

"Unlike existing methods that rely on a single modality or struggle under conflicting inputs, ControlFoley is designed to handle complex multimodal interactions and maintain strong controllability even when modalities are inconsistent,"

stated the research team in their project overview. Testing across several standard benchmarks showed a twenty-seven percent improvement in overall audio quality compared to previous open options. You can explore the full ControlFoley code repository on GitHub, review the detailed technical paper, or download the pre-trained weights directly from Hugging Face.