Classical estimation treats \(\theta\) as a fixed unknown. Bayesian estimation draws \(\theta\) from a prior, updates to a posterior, then picks one number from that posterior. Square cost gives the mean. Hit-or-miss gives the mode. They are different functions.
You never see \(\theta\). You see data \(\mathbf{x}\) that depend on \(\theta\). Classically the dependence is a PDF parametrized by a constant, written \(p(\mathbf{x};\theta)\) with a semicolon. Bayesian estimation puts a prior \(p(\theta)\) on the parameter, writes the same formula as a likelihood \(p(\mathbf{x}\mid\theta)\) with a bar, and updates by Bayes:
$$ p(\theta\mid\mathbf{x})=\frac{p(\mathbf{x}\mid\theta)\,p(\theta)}{p(\mathbf{x})}=\frac{p(\mathbf{x}\mid\theta)\,p(\theta)}{\int p(\mathbf{x}\mid\theta)\,p(\theta)\,d\theta}. $$The denominator does not depend on \(\theta\). For finding an estimator you can drop it and work with \(p(\mathbf{x}\mid\theta)p(\theta)\), or with the log \(\ln p(\mathbf{x}\mid\theta)+\ln p(\theta)\).
The number you return is an estimate of this realization of \(\theta\), after seeing \(\mathbf{x}\). Performance is a cost \(C(e)\) of the error \(e=\hat\theta-\theta\), averaged under the joint \(p(\theta,\mathbf{x})\): the Bayes risk \(\mathbb{E}[C(e)]\). Because \(p(\mathbf{x})\ge 0\), it is enough to minimize the posterior risk \(\int C(\hat\theta-\theta)\,p(\theta\mid\mathbf{x})\,d\theta\) for each \(\mathbf{x}\).
| Cost | \(C(e)\) | Estimator |
|---|---|---|
| Square (MMSE) | \(\|e\|_2^2\) | mean of \(p(\theta\mid\mathbf{x})\) |
| Absolute (MAE) | \(\|e\|_1\) | median of \(p(\theta\mid\mathbf{x})\) |
| Hit-or-miss | 0 if \(\|e\|_\infty<\varepsilon\), else 1 | mode (MAP), as \(\varepsilon\to 0\) |
Differentiate the posterior square risk in the free variable \(\hat\theta\):
$$ \frac{\partial}{\partial\hat\theta}\int(\hat\theta-\theta)^2\,p(\theta\mid\mathbf{x})\,d\theta =2\int(\hat\theta-\theta)\,p(\theta\mid\mathbf{x})\,d\theta=0. $$The integral of the posterior is 1, so
$$ \hat\theta_{\mathrm{MMSE}}=\int\theta\,p(\theta\mid\mathbf{x})\,d\theta=\mathbb{E}[\theta\mid\mathbf{x}]. $$If \(\theta\) is discrete the integral becomes a sum: \(\mathbb{E}[\theta\mid\mathbf{x}]=\sum_i \theta_i\,\Pr(\theta=\theta_i\mid\mathbf{x})\). That is the \(X=\pm 1\) problem below.
A uniform / hit-or-miss cost is 1 minus the posterior mass in an \(\varepsilon\)-window around \(\hat\theta\). For small \(\varepsilon\), put the window on the peak:
$$ \hat\theta_{\mathrm{MAP}}=\arg\max_\theta p(\theta\mid\mathbf{x})=\arg\max_\theta p(\mathbf{x}\mid\theta)\,p(\theta). $$If the log is differentiable and the max is interior to the support of the prior, a necessary condition is
$$ \frac{\partial}{\partial\theta}\big(\ln p(\mathbf{x}\mid\theta)+\ln p(\theta)\big)=0. $$If the unconstrained critical point falls outside the support, the maximum is on the boundary. A compact uniform prior is the exam case of that.
MAE: the derivative of the absolute cost is \(\mathrm{sign}(\hat\theta-\theta)\). Splitting the integral, it vanishes iff \(\hat\theta\) is a median. You almost never need MAE on this exam.
MMSE is not “the maximum of the posterior”. That is MAP. Resit 2022 2d: the official correct statement is “MMSE is the mean”. “MMSE and MAP always differ” is also false: a symmetric unimodal posterior (Gaussian) has mean = median = mode.
Skewed posterior: three different numbers. MAP sits at the peak, MAE at the median, MMSE is pulled toward the tail.
DC in WGN: \(x[n]=\theta+w[n]\), \(w[n]\sim\mathcal{N}(0,\sigma^2)\) iid, prior \(\theta\sim\mathcal{N}(\mu_\theta,\sigma_\theta^2)\). The likelihood is
$$ p(\mathbf{x}\mid\theta)\propto\exp\Big(-\frac{1}{2\sigma^2}\sum_{n=0}^{N-1}(x[n]-\theta)^2\Big). $$Split \(x[n]-\theta=(x[n]-\bar x)+(\bar x-\theta)\). Cross terms vanish, leftover \(N(\bar x-\theta)^2\). The posterior kernel is therefore
$$ \exp\Big(-\frac{N}{2\sigma^2}(\theta-\bar x)^2-\frac{1}{2\sigma_\theta^2}(\theta-\mu_\theta)^2\Big). $$Expand the quadratics. The coefficient of \(\theta^2\) is the posterior precision; the coefficient of \(\theta\) locates the mean:
$$ \frac{1}{\sigma_{\theta\mid\mathbf{x}}^2}=\frac{N}{\sigma^2}+\frac{1}{\sigma_\theta^2},\qquad \mu_{\theta\mid\mathbf{x}}=\sigma_{\theta\mid\mathbf{x}}^2\Big(\frac{N\bar x}{\sigma^2}+\frac{\mu_\theta}{\sigma_\theta^2}\Big). $$The posterior is Gaussian, so MMSE = MAP = MAE \(=\mu_{\theta\mid\mathbf{x}}\). Precisions add. The posterior mean is a precision-weighted average of the sample mean and the prior mean.
Same calculation with a cosine regressor (SLT4 2024–25 Ex. 4.3). Prior \(A\sim\mathcal{N}(0,\sigma_A^2)\), WGN variance \(\sigma^2\):
$$ \hat A_{\mathrm{MMSE}} =\frac{\sum_n\cos(2\pi f_0 n)\,x[n]/\sigma^2}{\sum_n\cos^2(2\pi f_0 n)/\sigma^2+1/\sigma_A^2}. $$Large \(\sigma_A^2\): \(\to\) the LS / MLE amplitude. Small \(\sigma_A^2\): shrinks to 0. (The SLT display of the posterior exponential has a sign OCR error; keep the mean formula.)
Exam Nov 2022 2f–g. Prior \(p(\theta)=a e^{-a\theta}\) for \(\theta\ge 0\). One observation \(p(x\mid\theta)=\theta e^{-\theta x}\) for \(x\ge 0\). The exam gives the posterior
$$ p(\theta\mid x)=(x+a)^2\,\theta\,e^{-(x+a)\theta},\qquad\theta\ge 0. $$This is Gamma / Erlang, shape 2, rate \(b=x+a\). Check the constant: \(\int_0^\infty\theta e^{-b\theta}\,d\theta=1/b^2\), so \(b^2\theta e^{-b\theta}\) is a PDF.
MMSE is the Gamma mean, shape/rate:
$$ \hat\theta_{\mathrm{MMSE}}=\mathbb{E}[\theta\mid x]=b^2\int_0^\infty\theta^2 e^{-b\theta}\,d\theta=b^2\cdot\frac{2}{b^3}=\frac{2}{b}=\frac{2}{x+a}. $$(Substitute \(u=b\theta\) if you do not want to quote the Gamma moment. \(\int_0^\infty u^2 e^{-u}\,du=2\).)
MAP: \(\ln p(\theta\mid x)=\mathrm{const}+\ln\theta-b\theta\), derivative \(1/\theta-b=0\), so
$$ \hat\theta_{\mathrm{MAP}}=\frac{1}{x+a}. $$Mode of Gamma(2, \(b\)) is \(1/b\). The posterior is skewed: MMSE \(\neq\) MAP.
Exam 21/01/2019 Q2. Likelihood \(p(x\mid\lambda)=\lambda e^{-\lambda x}\) (\(x\ge 0\)). Prior \(p(\lambda)=\tfrac12\) on \([0,2]\), 0 else. Part (a) of that paper is an \(N\)-sample MLE; the Bayesian parts in the published key are written as functions of a single \(x\). Treat them as \(N=1\) unless the question keeps the sum.
The prior is flat on \((0,2)\), so maximizing the posterior on that interval is maximizing the likelihood on that interval. For one sample, \(\ln p(x\mid\lambda)=\ln\lambda-\lambda x\), derivative \(1/\lambda-x=0\), unconstrained MLE \(\hat\lambda_{\mathrm{ML}}=1/x\). Second derivative \(-1/\lambda^2<0\): a maximum.
The function \(\ln\lambda-\lambda x\) is increasing for \(\lambda<1/x\). If \(1/x>2\), i.e. \(x<1/2\), it is still increasing on the whole of \((0,2]\), so the constrained maximum is the right endpoint.
$$ \hat\lambda_{\mathrm{MAP}}=\begin{cases} 2 & x\lt 1/2,\\[4pt] 1/x & x\ge 1/2. \end{cases} $$For \(N\) iid samples the unconstrained MLE is \(N/\sum_i x_i\). Project that onto \([0,2]\):
$$ \hat\lambda_{\mathrm{MAP}}=\min\Big(2,\;\max\Big(0,\;\frac{N}{\sum_i x_i}\Big)\Big). $$The 21/01/2019 key writes only \(\hat\lambda_{\mathrm{MAP}}=N/\sum_i x_i\). That is the MLE, and it is MAP only when it already lies in \([0,2]\). A compact uniform prior puts MAP on the boundary whenever the unconstrained MLE sits outside the support. Write the projection.
The prior constant \(1/2\) cancels in the posterior mean. For a single \(x\),
$$ \hat\lambda_{\mathrm{MMSE}}=\mathbb{E}[\lambda\mid x] =\frac{\int_0^2\lambda\cdot\lambda e^{-\lambda x}\,d\lambda}{\int_0^2\lambda e^{-\lambda x}\,d\lambda} =\frac{\int_0^2\lambda^2 e^{-x\lambda}\,d\lambda}{\int_0^2\lambda e^{-x\lambda}\,d\lambda}. $$Antiderivatives, integrate by parts (or differentiate the Gamma integral in the rate):
$$ \int\lambda e^{-x\lambda}\,d\lambda=-e^{-x\lambda}\Big(\frac{\lambda}{x}+\frac{1}{x^2}\Big),\qquad \int\lambda^2 e^{-x\lambda}\,d\lambda=-e^{-x\lambda}\Big(\frac{\lambda^2}{x}+\frac{2\lambda}{x^2}+\frac{2}{x^3}\Big). $$Denominator, bounds \(0\) to \(2\):
$$ Z=\Big[-e^{-x\lambda}\Big(\frac{\lambda}{x}+\frac{1}{x^2}\Big)\Big]_0^2 =\frac{1}{x^2}-e^{-2x}\frac{2x+1}{x^2} =\frac{1-(2x+1)e^{-2x}}{x^2}. $$Numerator:
$$ \int_0^2\lambda^2 e^{-x\lambda}\,d\lambda =\frac{2}{x^3}-e^{-2x}\frac{4x^2+4x+2}{x^3} =\frac{2-(4x^2+4x+2)e^{-2x}}{x^3}. $$Ratio, cancel one \(x\):
$$ \hat\lambda_{\mathrm{MMSE}} =\frac{2}{x}\cdot\frac{1-(2x^2+2x+1)e^{-2x}}{1-(2x+1)e^{-2x}}. $$The published 21/01/2019 last line is \(\frac{2}{x}\Big(1-\frac{x^2 e^{-2x}}{1-e^{-2x}(2x+1)}\Big)\), which equals \(\frac{2}{x}\frac{1-(x+1)^2 e^{-2x}}{1-(2x+1)e^{-2x}}\). That replaces \(2x^2+2x+1\) by \(x^2+2x+1\). The algebra is wrong. Use the formula above. Intermediate fragments in that key that mix \(p(x)\) are unreliable too.
SLT4 2024–25 Ex. 4.2. The posterior is already given:
$$ p(\theta\mid x)=\begin{cases}e^{-(\theta-x)} & \theta\ge x,\\ 0 & \theta\lt x.\end{cases} $$MMSE, substitute \(u=\theta-x\):
$$ \mathbb{E}[\theta\mid x]=\int_x^\infty\theta\,e^{-(\theta-x)}\,d\theta=\int_0^\infty(u+x)e^{-u}\,du=1+x. $$Official: \(\hat\theta_{\mathrm{MMSE}}=1+x\). The density is strictly decreasing on \([x,\infty)\), so the mode is the left endpoint: \(\hat\theta_{\mathrm{MAP}}=x\). Clean example that MMSE \(\neq\) MAP.
SLT4 2025–26 Ex. 4.2. \(x[n]=A+w[n]\), WGN variance \(\sigma^2\),
$$ p(A)=\lambda e^{-\lambda A}\ (A>0),\qquad 0\ (A<0). $$For \(A>0\),
$$ \ln\big(p(\mathbf{x}\mid A)p(A)\big)=-\frac{1}{2\sigma^2}\sum_{n=0}^{N-1}(x[n]-A)^2-\lambda A+\mathrm{const}. $$ $$ \frac{\partial}{\partial A}=\frac{1}{\sigma^2}\sum_{n=0}^{N-1}(x[n]-A)-\lambda=\frac{N}{\sigma^2}(\bar x-A)-\lambda\stackrel{!}{=}0, $$ $$ A=\bar x-\frac{\sigma^2\lambda}{N}. $$Prior support \(A\ge 0\): if that critical point is negative, the maximum is at 0.
$$ \hat A_{\mathrm{MAP}}=\max\Big(0,\;\bar x-\frac{\sigma^2\lambda}{N}\Big). $$Compared with the MLE \(\bar x\), the exponential prior shrinks toward 0 by \(\sigma^2\lambda/N\). Large \(N\): MAP \(\approx\) MLE. Small \(N\): trust the prior, and the hard wall \(A\ge 0\).
The cousin on 04/11/2019 Q3c is Rayleigh scale with the same exponential prior. The stationarity condition is the cubic \(\lambda\theta^3+2N\theta^2-\sum x^2[n]=0\). No tidy closed form; official: “\(\hat\theta_{\mathrm{MAP}}\) depends on \(\lambda\)”. Also project onto \(\theta>0\).
SLT4 2025–26 Ex. 4.3 and resit 24/01/2022 2f–h. Same official key.
Model: \(Y=X+N\), \(X\in\{+1,-1\}\) equally likely, \(N\sim\mathcal{N}(0,\sigma^2)\) independent of \(X\). So \(Y\mid X=x\sim\mathcal{N}(x,\sigma^2)\).
\(X\) is discrete, so the posterior mean is a sum:
$$ \hat X_{\mathrm{MMSE}}=\mathbb{E}[X\mid Y] =(+1)\Pr[X=1\mid Y]+(-1)\Pr[X=-1\mid Y] =\Pr[X=1\mid Y]-\Pr[X=-1\mid Y]. $$ $$ \hat X_{\mathrm{MMSE}} =\frac{e^{-(y-1)^2/(2\sigma^2)}-e^{-(y+1)^2/(2\sigma^2)}}{e^{-(y-1)^2/(2\sigma^2)}+e^{-(y+1)^2/(2\sigma^2)}}. $$Divide numerator and denominator by \(e^{-(y+1)^2/(2\sigma^2)}\):
$$ =\frac{e^{-[(y-1)^2-(y+1)^2]/(2\sigma^2)}-1}{e^{-[(y-1)^2-(y+1)^2]/(2\sigma^2)}+1}. $$Expand the squares: \((y-1)^2-(y+1)^2=(y^2-2y+1)-(y^2+2y+1)=-4y\). The exponent is
$$ -\frac{-4y}{2\sigma^2}=\frac{2y}{\sigma^2}, $$ so $$ \hat X_{\mathrm{MMSE}}=\frac{e^{2y/\sigma^2}-1}{e^{2y/\sigma^2}+1}. $$The identity printed on the exam is \((e^{2u}-1)/(e^{2u}+1)=\tanh u\) with \(u=y/\sigma^2\):
$$ \hat X_{\mathrm{MMSE}}=\tanh\Big(\frac{y}{\sigma^2}\Big). $$Official (16). Soft estimate in \((-1,1)\), not \(\pm 1\). Large \(|y|/\sigma^2\): saturates at \(\pm 1\).
(The other division, by \(e^{-(y-1)^2/(2\sigma^2)}\), produces \((1-e^{-2y/\sigma^2})/(1+e^{-2y/\sigma^2})\), which is the same \(\tanh\).)
Decide \(+1\) iff this ratio \(>1\) iff \(e^{2y/\sigma^2}>1\) iff \(y>0\).
$$ \hat X_{\mathrm{MAP}}=\begin{cases}+1 & y\gt 0,\\ -1 & y\lt 0,\end{cases} \qquad y_{\mathrm{th}}=0. $$Equal prior, symmetric Gaussians: MAP \(=\mathrm{sign}(y)\). The tie at 0 has probability zero.
The printed resit 2.h writes \(e^{-2y/\sigma^2}\) in one line and then \(2y/\sigma^2\gtrless 0\). Two sign slips that cancel. SLT4 2025–26 has \(e^{2y/\sigma^2}\). Final threshold \(y=0\) is official and correct.
\(\tanh(y/\sigma^2)\) and \(\mathrm{sign}(y)\) have the same sign and different magnitude. MAP is a hard decision. MMSE is a soft estimate (useful as an equalizer output).
Close the algebra above. Write yours. Then open the spoiler.
Resit 2022 2d style. Which statements are true?
Only (b). (a) is MAP. (c) fails for a Gaussian posterior, where mean = mode. Official: MMSE is the mean.
Nov 2022 2f–g. Posterior given: \(p(\theta\mid x)=(x+a)^2\theta e^{-(x+a)\theta}\) for \(\theta\ge 0\). Find \(\hat\theta_{\mathrm{MMSE}}\) and \(\hat\theta_{\mathrm{MAP}}\).
Shape 2, rate \(b=x+a\). Mean \(2/b\), mode \(1/b\):
$$ \hat\theta_{\mathrm{MMSE}}=\frac{2}{x+a},\qquad\hat\theta_{\mathrm{MAP}}=\frac{1}{x+a}. $$They differ (skew). No official numeric key for 2022; this is the Gamma calculation.
Likelihood \(\lambda e^{-\lambda x}\), prior uniform on \([0,2]\).
(a) Unconstrained MLE \(1/x\), then project:
$$ \hat\lambda_{\mathrm{MAP}}=\begin{cases}2 & x\lt 1/2,\\ 1/x & x\ge 1/2.\end{cases} $$For \(N\) samples, project \(N/\sum x_i\) onto \([0,2]\). The key’s \(\hat\lambda_{\mathrm{MAP}}=N/\sum x_i\) drops that projection.
(b) \(\mathbb{E}[\lambda\mid x]=\int_0^2\lambda^2 e^{-x\lambda}\,d\lambda\big/\int_0^2\lambda e^{-x\lambda}\,d\lambda\), which evaluates to
$$ \frac{2}{x}\cdot\frac{1-(2x^2+2x+1)e^{-2x}}{1-(2x+1)e^{-2x}}. $$(c) The factor \(2x^2+2x+1\), not \((x+1)^2=x^2+2x+1\).
\(Y=X+N\), \(X=\pm 1\) equally likely, \(N\sim\mathcal{N}(0,\sigma^2)\) independent of \(X\). Derive \(\hat X_{\mathrm{MMSE}}\) and the MAP threshold. The identity \((e^{2u}-1)/(e^{2u}+1)=\tanh u\) is given.
Posterior mean:
$$ \mathbb{E}[X\mid Y]=\frac{e^{-(y-1)^2/(2\sigma^2)}-e^{-(y+1)^2/(2\sigma^2)}}{e^{-(y-1)^2/(2\sigma^2)}+e^{-(y+1)^2/(2\sigma^2)}}. $$\((y-1)^2-(y+1)^2=-4y\), so the ratio of exponentials is \(e^{2y/\sigma^2}\), and
$$ \hat X_{\mathrm{MMSE}}=\frac{e^{2y/\sigma^2}-1}{e^{2y/\sigma^2}+1}=\tanh\Big(\frac{y}{\sigma^2}\Big). $$MAP: the same ratio of posteriors is \(e^{2y/\sigma^2}>1\) iff \(y>0\). Threshold \(y_{\mathrm{th}}=0\), \(\hat X_{\mathrm{MAP}}=\mathrm{sign}(y)\). Official for both the resit and SLT4 2025–26.
5CTA0 Q1 2026–27 · from the course reader, lectures and official keys.