Tweedie's Formula

Tweedie’s formula. If $y=\mu +s\eta$ with $\eta \sim \mathcal{N}(0,I)$ and $y$ has marginal density $p$, then the posterior mean of the clean signal is

$$\mathbb{E}[\mu \mid y]=y+s^2{\nabla }_y\log p(y).$$

The MMSE denoiser only needs the score of the noisy marginal. The formula goes back to Robbins (1956), who credited Tweedie; Efron (2011) gives a modern account.

For diffusion models. With $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}\epsilon$, apply the formula to $\mu =\sqrt{{\bar{\alpha }}_t}{x}_0$ and $s^2=1-{\bar{\alpha }}_t$:

$${\hat{x}}_0(x_t):=\mathbb{E}[x_0\mid x_t]=\frac{1}{\sqrt{{\bar{\alpha }}_t}}(x_t+(1-{\bar{\alpha }}_t){\nabla }_{x_t}\log p_t(x_t)).$$

With the learned score $s_{\theta }=-{\epsilon }_{\theta }/\sqrt{1-{\bar{\alpha }}_t}$:

$${\hat{x}}_0(x_t)\approx \frac{1}{\sqrt{{\bar{\alpha }}_t}}(x_t-\sqrt{1-{\bar{\alpha }}_t}{\epsilon }_{\theta }(x_t,t)).$$

This is the forward formula solved for $x_0$ with the predicted noise.

Role in inverse problems. A data-fidelity term $\Vert y-\mathcal{A}(x)\Vert$ only makes sense for clean images, but during sampling only noisy $x_t$ is available. Tweedie gives a clean estimate in closed form at every step, so the likelihood can be evaluated at ${\hat{x}}_0(x_t)$:

• as a gradient with respect to $x_t$, backpropagating through ${\hat{x}}_0$ (Diffusion Posterior Sampling);
• as the starting point of a data-consistency prox (DiffPIR).

${\hat{x}}_0$ is a posterior mean. It is blurry at high noise levels, when many clean images are compatible with $x_t$, and sharp at low noise levels.

References

1. B. Efron (2011). Tweedie’s Formula and Selection Bias. J. Amer. Statist. Assoc. 106(496), 1602–1614. doi:10.1198/jasa.2011.tm11181
2. H. Robbins (1956). An Empirical Bayes Approach to Statistics. Proc. Third Berkeley Symp. Math. Stat. Prob. 1, 157–163. projecteuclid.org
3. H. Chung, J. Kim, M. T. McCann, M. L. Klasky, J. C. Ye (2023). Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR 2023. arXiv:2209.14687