How to Implement Expectation Propagation for BNNs

Introduction

Expectation Propagation (EP) approximates posterior distributions in Bayesian Neural Networks by iteratively refining factor approximations, delivering fast uncertainty estimates without costly Monte Carlo sampling.

Key Takeaways

• EP replaces the true posterior with a tractable Gaussian factorized approximation.
• Updates rely on local moment matching for each factor.
• Training converges in a few passes, scaling to large networks.
• Provides calibrated predictive variance, essential for risk‑sensitive applications.

What is Expectation Propagation for BNNs?

EP is a message‑passing framework that decomposes the likelihood and prior into independent factors and updates each factor’s sufficient statistics against the current global approximation. In BNNs, this yields a Gaussian mixture posterior over weights, enabling closed‑form predictions with uncertainty. Wikipedia: Expectation Propagation describes the general algorithm.

Why Expectation Propagation Matters

Bayesian Neural Networks need reliable posterior estimates to quantify model confidence. Traditional Markov Chain Monte Carlo (MCMC) is accurate but slow; Variational Inference (VI) trades speed for expressiveness. EP balances speed and fidelity, making uncertainty‑aware deep learning feasible for production systems. arXiv: EP for BNNs demonstrates competitive results on benchmark tasks.

How Expectation Propagation Works

EP alternates three operations for each factor f_i(θ):

1. Remove: Subtract the current approximate factor q_i(θ) from the global approximation q(θ).
2. Project: Compute the cavity distribution q_-i(θ) = q(θ) / q_i(θ).
3. Match: Update q_i</