Expectation Particle Belief Propagation

The paper proposes an update scheme
for the proposal distribution in the message computation of particle
belief
propagation. The
proposal distribution is in the exponential family and is iteratively
updated in an expectation propgation (EP) framework. The computation at
each iteration is quadratic in the number of particles.

Particle Belief Propagation

Let $\psi_{u}$ and $\psi_{uv}$ be node and edge potentials,
respectively. Let $\Gamma_{u}$ be the set of neighbouring variables
connected to node $u$. The loopy belief propagation (LBP) message
updates are written as

where the superscript $\cdot^{t}$ denotes iteration number, $B_{u}$
denotes the belief at node $u$, and $m_{uv}$ is the message from node
$u$ to node $v$. Particle belief propgation (PBP) uses importance
sampling to compute the messages $m_{uv}$. Given a proposal distribution
$q_{u}$ on node $u$, and a set of $N$ particles
${x_{u}^{(i)}}{i=1}^{N}\sim q{u}(x_{u})$, PBP messages
$\hat{m}_{uv}^{\mathrm{PBP}}$ are written as:

where the importance weight
$w_{uv}^{(i)}:=\frac{1}{N}\frac{\psi_{u}(x_{u}^{(i)})}{q_{u}(x_{u}^{(i)})}\prod_{w\in\Gamma_{u}\backslash v}m_{wu}^{t-1}(x_{u}^{(i)})$.
The choice of $q_{u}$ determines the approximation quality.

Expectation Particle Belief Propagation

Let $\hat{m}_{uv}(x_{v})$ be the particle approximation of the exact
message $m_{uv}$. The particle approximation to the belief at node $v$
is

where $\delta_{x}$ is a delta measure at $x$. It can be seen that the
best proposal $q_{u}$ is the belief itself, which is unknown. An idea to
approximately achieve that is to use a tractable exponential family
distribution for $q_{u}$ instead:
where $\eta_{u}$ and $\eta_{wu}$ are exponential family approximations
of $\psi_{u}$ and $\hat{m}_{wu}$ respectively. The exponential family
distribution for $q_{u}$ is computed with EP. Specifically, each
$\eta_{wu}$ is iteratively updated with

where the cavity distribution
. It is known that the
solution $\eta_{wu}$ is the one such that the moments (expected
sufficient statistics) of the tilted distribution
match that of
$\eta q_{u}^{\backslash w}$. The computation of moments can be performed
crudely as $q_{u}$ is only used as a proposal distribution.

It is unclear if this scheme will work in the case that $q_{u}$ is high
dimensional.