Pick the \(\theta\) that makes this dataset most plausible.

You never see \(\theta\). You see noisy data that depend on \(\theta\). An estimator is a recipe \(g(\mathbf{x})\). The estimate \(\hat{\theta}=g(\mathbf{x})\) is a random variable, because \(\mathbf{x}\) is. Maximum likelihood is the recipe that maximises the likelihood of the data you actually saw.

• Three words, bias, MSEyardstick
• What a likelihood isfrom zero
• Exam derivationswrite these
• Invariance and the boundarytraps

Parameter, estimator, estimate

An experiment is a probabilistic mapping from the parameter to the observation: the PDF \(p(\mathbf{x};\theta)\). An estimator is a deterministic mapping the other way: \(g:\mathbf{x}\mapsto\hat{\theta}\). The recipe exists before you see the data. Talking about “the bias of \(\theta\)” is nonsense; bias is a property of \(g\).

Object	Symbol	Meaning
Parameter	\(\theta\)	Classical: a fixed unknown. Bayesian: random. Not a function of the data.
Data	\(\mathbf{x}=(x[0],\ldots,x[N-1])^\top\)	Random. What you see.
Estimator	\(g(\mathbf{x})\)	The recipe. Sample mean, MLE, LS, \ldots
Estimate	\(\hat{\theta}=g(\mathbf{x})\)	What the recipe returns on this dataset.
Classical PDF	\(p(\mathbf{x};\theta)\)	Parametrized by a deterministic \(\theta\). Semicolon.
Bayesian likelihood	\(p(\mathbf{x}\mid\theta)\)	Same formula, \(\theta\) random. Bar. Next chapter after CRLB.
Log-likelihood	\(\ln p(\mathbf{x};\theta)\)	What you actually differentiate.
WGN	\(w[n]\sim\mathcal{N}(0,\sigma^2)\) iid	White Gaussian noise.

Slides sometimes write \(p(x\mid\theta)\) for classical MLE. On the exam, semicolon = classical (no prior), bar = Bayesian (prior exists). Mixing them is a free point lost. The reader misspells “Carmér-Rao”. It is Cramér–Rao.

Running example: DC in WGN

Reader Example 5.1 / VL4. Unknown DC voltage \(A\), \(N\) noisy samples:

$$x_n=A+w_n,\qquad w_n\sim\mathcal{N}(0,\sigma^2)\ \text{iid},\quad n=0,\ldots,N-1.$$ $$p(\mathbf{x};A)=\frac{1}{(2\pi\sigma^2)^{N/2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{n=0}^{N-1}(x_n-A)^2\right].$$

Three recipes from the reader:

$$\hat{A}=\frac{1}{N}\sum_{n=0}^{N-1}x_n,\qquad \tilde{A}=x_0,\qquad \bar{A}=\frac{1}{N-2}\left(2x_0+\sum_{n=1}^{N-2}x_n-2x_{N-1}\right).$$

Unbiasedness

$$\mathbb{E}[\hat{\theta}]=\theta\qquad\text{for all }\theta$$

in the classical parameter range. “On average you hit the true value.” It does not say a single realization is close. All three DC recipes are unbiased:

$$\mathbb{E}[\hat{A}]=\frac{1}{N}\sum_{n=0}^{N-1}\mathbb{E}[A+w_n]=A,$$ $$\mathbb{E}[\tilde{A}]=\mathbb{E}[x_0]=A,$$ $$\mathbb{E}[\bar{A}]=\frac{1}{N-2}\big(2A+(N-2)A-2A\big)=A.$$

Unbiasedness does not pick a winner.

Variance

$$\mathrm{Var}[\hat{\theta}]=\mathbb{E}\big[(\hat{\theta}-\mathbb{E}[\hat{\theta}])^2\big]=\mathbb{E}[\hat{\theta}^2]-\big(\mathbb{E}[\hat{\theta}]\big)^2.$$

For iid \(x_n\) with \(\mathbb{E}[x_n]=A\), \(\mathrm{Var}[x_n]=\sigma^2\), so \(\mathbb{E}[x_n^2]=\sigma^2+A^2\), and independence gives \(\mathbb{E}[x_n x_m]=A^2\) for \(n\neq m\):

$$ \begin{aligned} \mathrm{Var}[\hat{A}] &=\mathbb{E}\left[\left(\frac{1}{N}\sum_n x_n\right)^2\right]-A^2\\ &=\frac{1}{N^2}\left(\sum_n\mathbb{E}[x_n^2]+\sum_{n\neq i}\mathbb{E}[x_n]\mathbb{E}[x_i]\right)-A^2\\ &=\frac{1}{N^2}\big(N(\sigma^2+A^2)+N(N-1)A^2\big)-A^2 =\frac{\sigma^2}{N}. \end{aligned} $$

Reader: \(\mathrm{Var}[\tilde{A}]=\sigma^2\), \(\mathrm{Var}[\bar{A}]=\dfrac{(N+6)}{(N-2)^2}\sigma^2\). The sample mean wins by averaging noise. Do not confuse \(\bar{A}\) with the SLT estimator below: the reader puts a minus on the last sample and divides by \(N-2\).

MSE = variance + bias squared

The natural figure of merit is mean squared error \(\mathrm{MSE}(\hat{\theta})=\mathbb{E}\big[(\hat{\theta}-\theta)^2\big]\). Write \(b(\theta)=\mathbb{E}[\hat{\theta}]-\theta\) and split the error:

$$\hat{\theta}-\theta=\big(\hat{\theta}-\mathbb{E}[\hat{\theta}]\big)+b(\theta).$$ $$ \begin{aligned} \mathrm{MSE} &=\mathbb{E}\big[(\hat{\theta}-\mathbb{E}[\hat{\theta}]+b(\theta))^2\big]\\ &=\mathbb{E}\big[(\hat{\theta}-\mathbb{E}[\hat{\theta}])^2\big] +2b(\theta)\mathbb{E}[\hat{\theta}-\mathbb{E}[\hat{\theta}]] +b(\theta)^2\\ &=\mathrm{Var}[\hat{\theta}]+b(\theta)^2, \end{aligned} $$

because the cross term is the expectation of a zero-mean quantity.

Minimising MSE over all estimators is not a well-posed classical problem: the minimiser is allowed to depend on the unknown \(\theta\), so you get unrealizable estimators. Restrict to unbiased estimators (\(b=0\)), then MSE = variance, and you look for the minimum-variance unbiased estimator (MVUE): unbiased, and \(\mathrm{Var}[\hat{\theta}]\le\mathrm{Var}[\tilde{\theta}]\) for every other unbiased \(\tilde{\theta}\), for all \(\theta\). An MVUE need not exist. When it does, it is unique.

A slightly biased estimator can beat an unbiased one in MSE (bias–variance). The course still uses unbiasedness + CRLB as the classical yardstick. CRLB is the next chapter.

What a likelihood is

Fix the data you actually observed. The function of \(\theta\)

$$\mathcal{L}(\theta)=p(\mathbf{x};\theta)\qquad(\mathbf{x}\text{ frozen})$$

is the likelihood. It is not a PDF in \(\theta\): \(\int\mathcal{L}(\theta)\,d\theta\) need not be 1. It answers: “if the true parameter were this \(\theta\), how plausible would this dataset be?”

IID samples: independence turns the joint into a product, the log into a sum.

$$p(\mathbf{x};\theta)=\prod_{n=0}^{N-1}p(x[n];\theta),\qquad \ln p(\mathbf{x};\theta)=\sum_{n=0}^{N-1}\ln p(x[n];\theta).$$

The log is strictly increasing, so the maximiser is the same:

$$\hat{\theta}_{\mathrm{ML}}=\arg\max_{\theta}p(\mathbf{x};\theta)=\arg\max_{\theta}\ln p(\mathbf{x};\theta).$$

If \(\ln p\) is differentiable and the max is interior, a necessary condition is the likelihood equation

$$\left.\frac{\partial}{\partial\theta}\ln p(\mathbf{x};\theta)\right|_{\theta=\hat{\theta}_{\mathrm{ML}}}=0.$$

Necessary, not sufficient: check it is a max (second derivative \(<0\), or inspect the function). If the max is on the boundary of the parameter set, the derivative need not vanish — look at the function (uniform example below).

Exam recipe, every time:

1. Write \(p(\mathbf{x};\theta)\). IID \(\Rightarrow\) product.
2. Take \(\ln\). Drop constants that do not depend on \(\theta\).
3. Differentiate. Set the score to 0. Solve for \(\theta\).
4. Second derivative \(<0\), or argue the function. If support depends on \(\theta\), skip the derivative and look at the function.

If an efficient estimator exists, it is the MLE. The efficiency identity (next chapter) is \(\partial\ln p/\partial\theta=\mathcal{I}(\theta)(g(\mathbf{x})-\theta)\). At the MLE the left side is 0 and \(\mathcal{I}(\theta)>0\), hence \(g(\mathbf{x})=\hat{\theta}_{\mathrm{ML}}\).

Asymptotics

Under the CRLB regularity conditions, as \(N\to\infty\):

• Consistent: \(\Pr\big(|\hat{\theta}_{\mathrm{ML}}-\theta|>\varepsilon\big)\to 0\).
• Asymptotically unbiased: \(\mathbb{E}[\hat{\theta}_{\mathrm{ML}}]\to\theta\).
• Asymptotically efficient: \(\mathrm{Var}[\hat{\theta}_{\mathrm{ML}}]\to\mathcal{I}(\theta)^{-1}\).
• Asymptotically Gaussian: \(\hat{\theta}_{\mathrm{ML}}\sim\mathcal{N}\big(\theta,\mathcal{I}(\theta)^{-1}\big)\). Vector: \(\hat{\boldsymbol{\theta}}_{\mathrm{ML}}\sim\mathcal{N}(\boldsymbol{\theta},\mathbf{I}(\boldsymbol{\theta})^{-1})\).

Consistency is stronger than asymptotic unbiasedness: the random variable itself concentrates, not just its mean.

VL slides write Chebyshev as \(\Pr[|\hat{\theta}-\theta|>\varepsilon]\le\mathbb{E}[(\hat{\theta}-\theta)^2]/\varepsilon\). Chebyshev is \(\mathrm{MSE}/\varepsilon^2\). The square is missing; the conclusion (MSE \(\to 0\) \(\Rightarrow\) consistency) still holds.

Finite-sample MLE can be biased (Gaussian variance \(1/N\) instead of \(1/(N-1)\); uniform max; Rayleigh scale; exponential rate). You still use it because of the asymptotics, and because if an efficient estimator exists it is the MLE.

Derivations the exam actually uses

Exponential rate 21/01/2019 · Q2a

IID \(p(x;\lambda)=\lambda e^{-\lambda x}\) for \(x\ge 0\), \(\lambda>0\) (reader App. B.1). Joint:

$$p(\mathbf{x};\lambda)=\lambda^N\exp\left(-\lambda\sum_{i=0}^{N-1}x_i\right),$$ $$\ln p=N\ln\lambda-\lambda\sum_i x_i,$$ $$\frac{\partial}{\partial\lambda}\ln p=\frac{N}{\lambda}-\sum_i x_i \stackrel{!}{=}0 \qquad\Rightarrow\qquad \hat{\lambda}_{\mathrm{ML}}=\frac{N}{\sum_{i=0}^{N-1}x_i}=\frac{1}{\bar{x}}.$$

Second derivative \(-N/\lambda^2<0\): a maximum. Official key: \(\hat{\lambda}_{\mathrm{ML}}=N/\sum_i x_i\). Mean of one sample is \(1/\lambda\), so the MLE is “one over the sample mean”.

For \(N>1\), \(\mathbb{E}[\hat{\lambda}_{\mathrm{ML}}]=N\lambda/(N-1)\neq\lambda\): biased, asymptotically unbiased. The score \(N/\lambda-\sum x_i\) cannot be written as \(\mathcal{I}(\lambda)(g(\mathbf{x})-\lambda)\) with \(g\) independent of \(\lambda\). No finite-sample efficient estimator. CRLB \(=\lambda^2/N\) (next chapter). The MVUE \((N-1)/\sum x_i\) has variance \(\lambda^2/(N-2)>\lambda^2/N\).

Rayleigh-type 04/11/2019 · Ex. 3

IID, \(x\ge 0\),

$$p(x;\theta)=\frac{x}{\theta^2}\exp\left[-\frac{x^2}{2\theta^2}\right].$$

This is Rayleigh with scale \(\theta\); \(\mathbb{E}[x^2]=2\theta^2\). One sample:

$$\ln p(x;\theta)=\ln x-2\ln\theta-\frac{x^2}{2\theta^2}.$$

\(N\) iid:

$$\ln p(\mathbf{x};\theta)=\sum_n\ln x[n]-2N\ln\theta-\frac{1}{2\theta^2}\sum_{n=0}^{N-1}x^2[n],$$ $$\frac{\partial}{\partial\theta}\ln p=-\frac{2N}{\theta}+\frac{1}{\theta^3}\sum_n x^2[n] \stackrel{!}{=}0.$$

Multiply by \(\theta^3\) (\(\theta>0\)):

$$-2N\theta^2+\sum_n x^2[n]=0 \qquad\Rightarrow\qquad \hat{\theta}_{\mathrm{ML}}=\sqrt{\frac{1}{2N}\sum_{n=0}^{N-1}x^2[n]}.$$

Official (9)/(3): same formula. Keep the positive square root (\(\theta>0\)).

Practice 28/10/2021 2b is the cousin \(p=\frac{x}{\theta}\exp(-x^2/(2\theta))\), i.e. \(\theta\) is the scale-squared: MLE \(\hat{\theta}=\frac{1}{2N}\sum x^2[n]\). Do not mix the two PDFs.

Same exam, part b, asked the CRLB. Second derivative of the log-likelihood:

$$\frac{\partial^2}{\partial\theta^2}\ln p=\frac{2N}{\theta^2}-\frac{3}{\theta^4}\sum_n x^2[n].$$ $$\mathcal{I}(\theta)=-\mathbb{E}\left[\frac{\partial^2}{\partial\theta^2}\ln p\right] =-\frac{2N}{\theta^2}+\frac{3}{\theta^4}\cdot N\cdot\mathbb{E}[x^2].$$

\(\mathbb{E}[x^2]=2\theta^2\) (integrate, or: \(x/\theta\) is standard Rayleigh, \(\mathbb{E}[(x/\theta)^2]=2\)). So

$$\mathcal{I}(\theta)=-\frac{2N}{\theta^2}+\frac{3}{\theta^4}\cdot 2N\theta^2 =\frac{-2N+6N}{\theta^2}=\frac{4N}{\theta^2}, \qquad \mathrm{Var}[\hat{\theta}]\ge\frac{\theta^2}{4N}.$$

Official key writes \(\mathrm{VAR}(\hat{\theta})\ge\hat{\theta}^2/(4N)\). Wrong object: CRLB is a function of the true \(\theta\), not of the estimate. Report \(\theta^2/(4N)\). The score is not linear in \((\hat{\theta}_{\mathrm{ML}}-\theta)\), so this MLE is not finite-sample efficient.

Same exam, part c, MAP with exponential prior \(p(\theta)=\lambda e^{-\lambda\theta}\), \(\theta>0\). Maximize \(p(\mathbf{x}\mid\theta)p(\theta)\):

$$\ln\big(p(\mathbf{x}\mid\theta)p(\theta)\big) =\mathrm{const}-2N\ln\theta-\frac{1}{2\theta^2}\sum x^2[n]-\lambda\theta,$$ $$\frac{\partial}{\partial\theta}=-\frac{2N}{\theta}+\frac{1}{\theta^3}\sum x^2[n]-\lambda\stackrel{!}{=}0 \qquad\Rightarrow\qquad \lambda\theta^3+2N\theta^2-\sum x^2[n]=0.$$

Cubic, no tidy closed form. Official: “\(\hat{\theta}_{\mathrm{MAP}}\) depends on the value of \(\lambda\)”. Also project onto \(\theta>0\).

DC level in WGN — the workhorse

Reader Example 7.2 / VL6. \(x_n=A+w_n\), \(w_n\sim\mathcal{N}(0,\sigma^2)\) iid, \(\sigma^2\) known.

$$\ln p(\mathbf{x};A)=-\frac{N}{2}\ln(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_n(x_n-A)^2,$$ $$\frac{\partial}{\partial A}\ln p=\frac{1}{\sigma^2}\sum_n(x_n-A)=\frac{N}{\sigma^2}\left(\frac{1}{N}\sum_n x_n-A\right)\stackrel{!}{=}0,$$ $$\hat{A}_{\mathrm{ML}}=\frac{1}{N}\sum_{n=0}^{N-1}x_n=\bar{x}.$$

This is the efficiency identity with \(\mathcal{I}(A)=N/\sigma^2\) and \(g=\bar{x}\). Sample mean is MLE, efficient, and MVUE. \(\mathrm{Var}=\sigma^2/N=\mathrm{CRLB}\).

Both mean and variance unknown (reader Example 7.4). Score in \(A\) still gives \(\hat{A}=\bar{x}\). Score in \(\sigma^2\):

$$\frac{\partial}{\partial\sigma^2}\ln p=-\frac{N}{2\sigma^2}+\frac{1}{2\sigma^4}\sum_n(x_n-A)^2=0 \Rightarrow \hat{\sigma}^2_{\mathrm{ML}}=\frac{1}{N}\sum_n(x_n-\bar{x})^2.$$

Biased: \(\mathbb{E}[\hat{\sigma}^2]=\frac{N-1}{N}\sigma^2\). Unbiased cousin uses \(N-1\). Asymptotically \(\hat{\sigma}^2\sim\mathcal{N}(\sigma^2,2\sigma^4/N)\), matching the Fisher-matrix bound \(\mathrm{Var}[\sigma^2]\ge 2\sigma^4/N\).

DC with variance equal to the mean (reader Example 7.3): \(x_n=A+w_n\), \(w_n\sim\mathcal{N}(0,A)\), \(A>0\). No efficient estimator. Likelihood equation reduces to the quadratic \(A^2+A-\frac{1}{N}\sum x_n^2=0\), positive root

$$\hat{A}_{\mathrm{ML}}=-\frac12+\sqrt{\frac{1}{N}\sum_n x_n^2+\frac14}.$$

Jensen \(\Rightarrow\) biased; LLN \(\Rightarrow\) consistent.

Angle from two Gaussians Nov 2022 · 2d–2e

No official numeric key in the old-exam answers file. Derive from the paper.

$$\mathbf{x}=A\begin{bmatrix}\cos\theta\\\sin\theta\end{bmatrix}+\mathbf{w}, \qquad p(\mathbf{x};\theta)=\frac{1}{2\pi\sigma^2}\exp\left(-\frac{1}{2\sigma^2}\Big[(x_1-A\cos\theta)^2+(x_2-A\sin\theta)^2\Big]\right).$$

\(A>0\) and \(\sigma^2\) known. Expand the quadratic:

$$(x_1-A\cos\theta)^2+(x_2-A\sin\theta)^2 =x_1^2+x_2^2+A^2-2A(x_1\cos\theta+x_2\sin\theta).$$

Maximizing the likelihood \(\Leftrightarrow\) maximizing \(x_1\cos\theta+x_2\sin\theta\) (the only \(\theta\)-dependent term). Write

$$x_1\cos\theta+x_2\sin\theta=R\cos(\theta-\phi),\qquad R=\sqrt{x_1^2+x_2^2},\quad\phi=\mathrm{atan2}(x_2,x_1).$$

Maximum at \(\theta=\phi\):

$$\hat{\theta}_{\mathrm{ML}}=\mathrm{atan2}(x_2,x_1).$$

\(\arctan(x_2/x_1)\) loses the quadrant. Use two-argument atan2. Setting \(\partial/\partial\theta=0\) gives \(\tan\theta=x_2/x_1\), same ambiguity.

Fisher (2e). Write \(Q\) for the quadratic. Then

$$\frac{\partial Q}{\partial\theta}=2(x_1-A\cos\theta)(A\sin\theta)+2(x_2-A\sin\theta)(-A\cos\theta)=2A(x_1\sin\theta-x_2\cos\theta),$$ $$\frac{\partial\ln p}{\partial\theta}=-\frac{A}{\sigma^2}(x_1\sin\theta-x_2\cos\theta),$$ $$\frac{\partial^2\ln p}{\partial\theta^2}=-\frac{A}{\sigma^2}(x_1\cos\theta+x_2\sin\theta).$$

Hint on the exam: \(\mathbb{E}[x_1]=A\cos\theta\), \(\mathbb{E}[x_2]=A\sin\theta\). So

$$\mathcal{I}(\theta)=-\mathbb{E}\left[\frac{\partial^2\ln p}{\partial\theta^2}\right] =\frac{A}{\sigma^2}\big(A\cos^2\theta+A\sin^2\theta\big)=\frac{A^2}{\sigma^2}.$$

CRLB \(=\sigma^2/A^2\). Large amplitude or small noise: easier angle. One snapshot, so no factor \(N\).

Two groups sharing a mean SLT3 2025–26 · 3.2

\(X_i\stackrel{\mathrm{iid}}{\sim}\mathcal{N}(\mu,\sigma^2)\), \(Y_i\stackrel{\mathrm{iid}}{\sim}\mathcal{N}(\mu,\lambda\sigma^2)\), \(i=1,\ldots,N\). Shared mean \(\mu\) is known. Unknown: \(\lambda>0\), and in (b) also \(\sigma^2\).

(a) \(\sigma^2\) known. Independence of all \(2N\) samples:

$$ \begin{aligned} p(\mathbf{x},\mathbf{y};\lambda) &=\frac{1}{(2\pi\sigma^2)^{N/2}}\exp\left(-\frac{1}{2\sigma^2}\sum_i(x_i-\mu)^2\right)\\ &\quad\cdot\frac{1}{(2\pi\lambda\sigma^2)^{N/2}}\exp\left(-\frac{1}{2\lambda\sigma^2}\sum_i(y_i-\mu)^2\right). \end{aligned} $$

Drop terms without \(\lambda\):

$$\mathcal{L}=-\frac{N}{2}\ln\lambda-\frac{1}{2\lambda\sigma^2}\sum_i(y_i-\mu)^2+\mathrm{const},$$ $$\frac{\partial\mathcal{L}}{\partial\lambda}=-\frac{N}{2\lambda}+\frac{1}{2\lambda^2\sigma^2}\sum_i(y_i-\mu)^2\stackrel{!}{=}0,$$ $$\hat{\lambda}_{\mathrm{ML}}=\frac{1}{N\sigma^2}\sum_{i=1}^N(y_i-\mu)^2.$$

Official (4). The \(X\)-sample does not enter: given \(\sigma^2\), it carries no information about \(\lambda\).

(b) \(\sigma^2\) and \(\lambda\) unknown. Same likelihood, now \(p(\mathbf{x},\mathbf{y};\lambda,\sigma^2)\). The \(\lambda\)-equation is still (a). Differentiate in \(\sigma^2\):

$$\frac{\partial\mathcal{L}}{\partial\sigma^2} =-\frac{N}{2\sigma^2}+\frac{1}{2\sigma^4}\sum_i(x_i-\mu)^2 -\frac{N}{2\sigma^2}+\frac{1}{2\lambda\sigma^4}\sum_i(y_i-\mu)^2.$$

Set to 0 and substitute \(\lambda=\frac{1}{N\sigma^2}\sum(y_i-\mu)^2\). The last term becomes \(N/(2\sigma^2)\), so

$$-\frac{N}{\sigma^2}+\frac{1}{2\sigma^4}\sum_i(x_i-\mu)^2+\frac{N}{2\sigma^2}=0 \qquad\Rightarrow\qquad -\frac{N}{2\sigma^2}+\frac{1}{2\sigma^4}\sum_i(x_i-\mu)^2=0,$$ $$\hat{\sigma}^2_{\mathrm{ML}}=\frac{1}{N}\sum_{i=1}^N(x_i-\mu)^2, \qquad \hat{\lambda}_{\mathrm{ML}}=\frac{\sum_i(y_i-\mu)^2}{\sum_i(x_i-\mu)^2}.$$

Official (12)–(13). Intuition: \(X\) estimates \(\sigma^2\); \(Y\) estimates \(\lambda\sigma^2\); the ratio estimates \(\lambda\).

Invariance, and MLE on a boundary

Invariance

Reader 7.4 / VL7. If \(\alpha=h(\theta)\) and \(h\) is one-to-one,

$$\hat{\alpha}_{\mathrm{ML}}=h(\hat{\theta}_{\mathrm{ML}}).$$

You maximize in the convenient parametrization, then transform. VL7 also states the property for \(h\) not necessarily one-to-one, via the induced likelihood \(p_T(x;\alpha)=\max_{\{\theta:h(\theta)=\alpha\}}p(x;\theta)\). Exam MCQ (resit 2022 2b, official c): if \(\alpha=\psi(\theta)\) one-to-one then \(\hat{\theta}_{\mathrm{ML}}=\psi^{-1}(\hat{\alpha}_{\mathrm{ML}})\).

Worked (VL / EstimationTheory slides). DC in WGN, \(\hat{A}_{\mathrm{ML}}=\bar{x}\). For \(\alpha=e^{A}\), invariance gives \(\hat{\alpha}_{\mathrm{ML}}=e^{\bar{x}}\). Do not re-derive a likelihood in \(\alpha\) unless asked.

Efficiency is not invariant unless \(h\) is affine. Counter-example: sample mean \(\bar{x}\) is efficient for \(A\), but \(\bar{x}^2\) is biased for \(A^2\):

$$\mathbb{E}[\bar{x}^2]=\big(\mathbb{E}[\bar{x}]\big)^2+\mathrm{Var}[\bar{x}]=A^2+\frac{\sigma^2}{N}\neq A^2.$$

CRLB of the transform is still valid as a bound; the transformed estimator need not achieve it.

Uniform: MLE on a boundary

Reader Example 7.1 / VL6. \(x_i\stackrel{\mathrm{iid}}{\sim}\mathrm{Unif}[0,\theta]\), \(\theta>0\):

$$p(\mathbf{x};\theta)=\begin{cases}\theta^{-N}&0\lt\max_i x_i\lt\theta,\\0&\text{else.}\end{cases}$$

Likelihood is 0 if \(\theta<\max x_i\), and \(\theta^{-N}\) (strictly decreasing) if \(\theta\ge\max x_i\). So

$$\hat{\theta}_{\mathrm{ML}}=\max_i x_i.$$

No derivative: support depends on \(\theta\). Same setup breaks the CRLB (next chapter).

If the max sits on the boundary of the parameter set, the likelihood equation need not hold. Look at the function. Uniform, \(x\in[0,\theta]\), \(x\ge\theta\): MLE is usually \(\max\) or \(\min\).

5CTA0 Q1 2026–27 · from the course reader, lectures and official keys.