HarmonyDreamer: Rectifying and Harmonizing

Dynamic Objects with Backgrounds for 4D Video

Generation via Inversion Scores

Anonymous Author(s)
Affiliation
Address
email

Abstract

Recent advancements in score distillation have markedly enhanced the generation of both static and dynamic 3D tasks by effectively harnessing the power of 2D pretrained diffusion models for artistic applications. However, traditional distillation methods primarily concentrate on object-centric representations without adequately addressing the integration of objects with their backgrounds, which includes matching rendering styles, shadows, and motion. This paper presents a new framework that incorporates a background-enhanced distillation score along with an iterative editing-reconstruction process tailored for 4D video generation from basic dynamic 4D Gaussian splatting. Our experiments demonstrate that our method not only excels in producing high-fidelity 4D videos that harmoniously blend objects with varied backgrounds, but also facilitates the creation of multiview videos. Additionally, our approach outperforms existing baselines in both spatiotemporal and semantic consistency, demonstrating its effectives in 4D video generation.

Method overview

Semantic Harmonization Examples

Prompt "running lion"

Reference

Harmonized View 1

Harmonized View 2

Harmonized View 3

Prompt "flying cat like a bee"

Reference

Harmonized View 1

Harmonized View 2

Harmonized View 3

Prompt "robot on the moon"

Reference

Harmonized View 1

Harmonized View 2

Harmonized View 3

BG1+rough GS

Harmonized style 1

BG2+rough GS

Harmonized style 2

Reference

Harmonize wt water

Harmonize wt sand

Harmonize wt fish

Reference

Turn and raise head

Lower head

Stare

4D harmonization result

Prompt "robot on the moon"

Input

Phase 1

Phase 2

Prompt "robot"

Input

Phase 1

Phase 2

Prompt "Running cat"

Input

Phase 1

Phase 2

Prompt "Running lion"

Input

Phase 1

Phase 2

Generated with these models: SVD/Stability-AI SD2.1/Stability-AI lllyasviel/Lvmin Zhang

BibTeX

 @misc{**************,

              title={***************************************************************************},

              author={***************************************************************************},

              booktitle={************************************************},

              year={2024},

              archivePrefix={***},

              primaryClass={*.*}

            }