Probability Flow ODE

For every forward SDE $\mathrm{d}x=f\mathrm{d}t+g\mathrm{d}W$ there is a deterministic ODE with the same marginals $p_t$:

$$\frac{\mathrm{d}x}{\mathrm{d}t}=f(x,t)-\frac{1}{2}g(t{)}^2{\nabla }_x\log p_t(x).$$

For the Variance-Preserving SDE:

$$\frac{\mathrm{d}x}{\mathrm{d}t}=-\frac{1}{2}\beta (t)(x+{\nabla }_x\log p_t(x))=-\frac{1}{2}\beta (t)(x-\frac{{\epsilon }_{\theta }(x,t)}{\sqrt{1-\bar{\alpha }(t)}}).$$

Derivation sketch. Differentiating $x(t)=\sqrt{\bar{\alpha }}{x}_0+\sqrt{1-\bar{\alpha }}\epsilon$ with $\epsilon$ fixed, and using $\dot{\bar{\alpha }}=-\beta \bar{\alpha }$, gives $\dot{x}=-\frac{1}{2}\beta x+\frac{\beta }{2\sqrt{1-\bar{\alpha }}}\epsilon$. Replacing $\epsilon$ by $\mathbb{E}[\epsilon \mid x_t]=-\sqrt{1-\bar{\alpha }}\nabla \log p_t$ yields the ODE. Both the Fokker–Planck equation of the SDE and the continuity equation of the ODE are solved by the same $p_t$.

Properties.

• Deterministic: a bijection between noise and data. It enables exact likelihoods (as a continuous normalising flow; see Neural ODEs), latent interpolation, and fast high-order ODE solvers (DDIM is a discretisation of it).
• Same marginals, different paths: the SDE and the ODE agree at each time but not as path measures. A diffusion is not determined by its marginals.

References

1. Y. Song et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021. arXiv:2011.13456
2. J. Song, C. Meng, S. Ermon (2021). Denoising Diffusion Implicit Models. ICLR 2021. arXiv:2010.02502