Fisher information is the expected curvature of the log-likelihood. The Cramér–Rao lower bound says that every unbiased estimator has variance at least \(1/\mathcal{I}(\theta)\), if the support of the PDF does not depend on \(\theta\). An unbiased estimator that attains the bound is efficient, hence the MVUE.
Unbiased estimators cannot be arbitrarily sharp. The data PDF, as a function of \(\theta\), has a certain curvature. That curvature is Fisher information. More information \(\Rightarrow\) lower bound on variance.
Two equivalent definitions (reader (6.2)–(6.3)):
$$\mathcal{I}(\theta) =\mathbb{E}\left[\left(\frac{\partial\ln p(\mathbf{x};\theta)}{\partial\theta}\right)^2\right] =-\mathbb{E}\left[\frac{\partial^2\ln p(\mathbf{x};\theta)}{\partial\theta^2}\right].$$The second is almost always easier: differentiate twice, take expectation, flip sign. \(\mathcal{I}(\theta)\ge 0\). It is not strictly negative (2022 exam 2a trap). The expected second derivative of the log-likelihood is negative; Fisher is the positive number \(-\mathbb{E}[\partial^2\ln p/\partial\theta^2]\).
CRLB. For every unbiased estimator \(g(\mathbf{x})\), if regularity holds,
$$\mathrm{Var}[g(\mathbf{x})]\ge\frac{1}{\mathcal{I}(\theta)}.$$An unbiased estimator that attains the bound is efficient. Efficient \(\Rightarrow\) MVUE. MVUE need not be efficient (bound not attained).
“Variance of any unbiased estimator is lower bounded by the Fisher information, \(\mathrm{Var}\ge\mathcal{I}(\theta)\)” is false: it is \(1/\mathcal{I}\). For \(N\) iid observations \(\mathcal{I}_N(\theta)=N i(\theta)\), so the bound shrinks as \(1/N\).
You must interchange derivative and integral:
$$\frac{d}{d\theta}\int p(\mathbf{x};\theta)\,d\mathbf{x} =\int\frac{\partial}{\partial\theta}p(\mathbf{x};\theta)\,d\mathbf{x}.$$This holds when the support of \(p(\mathbf{x};\theta)\) does not depend on \(\theta\), the parameter space is an open interval, and the derivative exists and is finite (reader 6.2). If the support does depend on \(\theta\), Leibniz’s rule adds boundary terms:
$$\frac{d}{d\theta}\int_{a(\theta)}^{b(\theta)}f(x,\theta)\,dx =f(b(\theta),\theta)b'(\theta)-f(a(\theta),\theta)a'(\theta)+\int_{a(\theta)}^{b(\theta)}\partial_\theta f\,dx.$$Slides (EstimationTheory part 1) do the same with \(\mathrm{Unif}(1,\theta)\), \(p=1/\theta\) on \([1,\theta]\): Leibniz leaves the extra \(g(\theta)/\theta\), so derivative and integral do not commute.
If you see \(x\in[0,\theta]\) or \(x\ge\theta\): stop. No CRLB. MLE is usually \(\max\) or \(\min\). Do not grind Fisher.
One observation with
$$p(x;\theta)=\frac{2}{\theta}x,\qquad 0\le x\le\theta$$(and 0 else). Can you derive a CRLB for unbiased estimators of \(\theta\)? Why, or why not?
Support is \([0,\theta]\), which depends on \(\theta\). Regularity fails: you cannot pass \(d/d\theta\) under the integral. Official key: “The CRLB cannot be derived because the domain of integration depends on the parameter \(\theta\). Thus, the regularity condition is violated.” Gold-standard trap. Same story as \(\mathrm{Unif}[0,\theta]\) in the previous chapter.
Reader 6.2, every step. Unbiased: \(\int(g(\mathbf{x})-\theta)p(\mathbf{x};\theta)\,d\mathbf{x}=0\). Differentiate in \(\theta\), use regularity, and \(\int p=1\):
$$\int(g-\theta)\frac{\partial p}{\partial\theta}\,d\mathbf{x}-\int p\,d\mathbf{x}=0 \qquad\Rightarrow\qquad \int(g-\theta)\frac{\partial p}{\partial\theta}\,d\mathbf{x}=1.$$Identity \(\partial p/\partial\theta=p\cdot\partial\ln p/\partial\theta\):
$$\int(g-\theta)\,p\,\frac{\partial\ln p}{\partial\theta}\,d\mathbf{x}=1.$$Rewrite as an inner product of \((g-\theta)\sqrt{p}\) and \((\partial\ln p/\partial\theta)\sqrt{p}\). Cauchy–Schwarz:
$$1\le\int(g-\theta)^2 p\,d\mathbf{x}\cdot\int\left(\frac{\partial\ln p}{\partial\theta}\right)^2 p\,d\mathbf{x} =\mathrm{Var}[g]\cdot\mathcal{I}(\theta).$$Hence \(\mathrm{Var}[g]\ge 1/\mathcal{I}(\theta)\).
Equality in Cauchy–Schwarz iff the two functions are proportional, i.e.
$$\frac{\partial\ln p}{\partial\theta}=a(\theta)\big(g(\mathbf{x})-\theta\big)$$with \(a\) independent of \(\mathbf{x}\). Differentiate again, take \(\mathbb{E}\), use unbiasedness: \(a(\theta)=\mathcal{I}(\theta)\). That is the efficiency condition below.
Why the two Fisher formulae match. \(\int p=1\Rightarrow\int\partial p/\partial\theta=0\Rightarrow\mathbb{E}[\partial\ln p/\partial\theta]=0\). Differentiate again:
$$\mathbb{E}\left[\left(\frac{\partial\ln p}{\partial\theta}\right)^2\right]+\mathbb{E}\left[\frac{\partial^2\ln p}{\partial\theta^2}\right]=0.$$CRLB shrinks as \(1/N\). Official resit 2022 2a: “for IID observations the CRLB decreases linearly with the number of observations” is the correct MCQ (the bound is \(1/(N i)\), so yes, linear in \(1/N\)).
Reader 6.5. \(x_n=s[n;\theta]+w_n\), \(w_n\) WGN variance \(\sigma^2\). Then
$$\mathrm{Var}[g]\ge\frac{\sigma^2}{\sum_{n=0}^{N-1}\big(\partial s[n;\theta]/\partial\theta\big)^2}.$$Sketch: \(\ln p=-\frac{1}{2\sigma^2}\sum(x_n-s_n)^2+\mathrm{const}\). Differentiate twice. The expectation of each \((x_n-s_n)\) is 0, leftover \(\frac{1}{\sigma^2}\sum(\partial s/\partial\theta)^2\). DC: \(s[n;A]=A\), \(\partial s/\partial A=1\), CRLB \(=\sigma^2/N\).
\(x[n]=A r^n+w[n]\), \(r>0\) known, WGN variance \(\sigma^2\). Use the WGN shortcut with \(s[n]=A r^n\), \(\partial s/\partial A=r^n\):
$$\mathcal{I}(A)=\frac{1}{\sigma^2}\sum_{n=0}^{N-1}r^{2n}.$$Geometric sum: if \(r\neq 1\), \(\sum_{n=0}^{N-1}(r^2)^n=(r^{2N}-1)/(r^2-1)\). Official:
$$\mathrm{Var}(\hat{A})\ge\sigma^2\frac{r^2-1}{r^{2N}-1}.$$For \(0<r<1\) both numerator and denominator are negative; equivalent positive form \(\sigma^2(1-r^2)/(1-r^{2N})\). For \(r=1\) the sum is \(N\), CRLB \(=\sigma^2/N\).
An efficient estimator exists. Score:
$$\frac{\partial\ln p}{\partial A}=\frac{1}{\sigma^2}\sum_n(x[n]-A r^n)r^n =\frac{\sum_n r^{2n}}{\sigma^2}\left(\frac{\sum_n x[n] r^n}{\sum_n r^{2n}}-A\right).$$So \(g(\mathbf{x})=\hat{A}=\dfrac{\sum_n x[n] r^n}{\sum_n r^{2n}}\) is efficient, variance \(=\) CRLB. Setting the score to 0 shows it is also the MLE (SLT3 2024–25 Ex. 3.1d).
\(N\to\infty\) (official 2025 3.1c):
$$\lim_{N\to\infty}\mathrm{Var}(\hat{A})=\begin{cases} 0 & r\ge 1,\\ \sigma^2(1-r^2) & 0\lt r\lt 1. \end{cases}$$If \(|r|<1\) the signal dies; extra samples are almost pure noise and the variance floors. If \(r\ge 1\) energy grows (or stays, \(r=1\)) and variance \(\to 0\).
IID, \(x\ge 0\),
$$p(x;\theta)=\frac{x}{\theta^2}\exp\left[-\frac{x^2}{2\theta^2}\right],\qquad \mathbb{E}[x^2]=2\theta^2.$$\(N\) samples. Compute \(\mathcal{I}(\theta)\) as \(-\mathbb{E}[\partial^2\ln p/\partial\theta^2]\) and write the CRLB. (MLE was \(\hat{\theta}_{\mathrm{ML}}=\sqrt{\frac{1}{2N}\sum x^2[n]}\).)
\(\ln p(\mathbf{x};\theta)=\sum_n\ln x[n]-2N\ln\theta-\frac{1}{2\theta^2}\sum_n x^2[n]\). First derivative \(-2N/\theta+(1/\theta^3)\sum x^2[n]\). Second:
$$\frac{\partial^2}{\partial\theta^2}\ln p=\frac{2N}{\theta^2}-\frac{3}{\theta^4}\sum_n x^2[n].$$ $$\mathcal{I}(\theta)=-\frac{2N}{\theta^2}+\frac{3}{\theta^4}\cdot N\cdot 2\theta^2=\frac{4N}{\theta^2},\qquad\mathrm{Var}[\hat{\theta}]\ge\frac{\theta^2}{4N}.$$Official key writes \(\mathrm{VAR}(\hat{\theta})\ge\hat{\theta}^2/(4N)\). Wrong object: the bound depends on the true parameter, not on the estimate. Report \(\theta^2/(4N)\). The score is not linear in \((\hat{\theta}_{\mathrm{ML}}-\theta)\), so the MLE is not finite-sample efficient.
\(x[n]=A r^n+w[n]\), \(r>0\) known, WGN variance \(\sigma^2\), \(n=0,\ldots,N-1\). Write the CRLB for unbiased estimators of \(A\). Exhibit an efficient estimator. What is \(\lim_{N\to\infty}\mathrm{Var}(\hat{A})\)?
WGN shortcut: \(\mathcal{I}(A)=\sigma^{-2}\sum_{n=0}^{N-1} r^{2n}\). Geometric sum, \(r\neq 1\): official \(\mathrm{Var}(\hat{A})\ge\sigma^2(r^2-1)/(r^{2N}-1)\). Equivalent positive form for \(0<r<1\): \(\sigma^2(1-r^2)/(1-r^{2N})\). For \(r=1\): \(\sigma^2/N\).
Score factors as \(\mathcal{I}(A)\big(\hat{A}-A\big)\) with \(\hat{A}=\sum_n x[n] r^n\big/\sum_n r^{2n}\). Efficient, hence MLE, variance = CRLB.
$$\lim_{N\to\infty}\mathrm{Var}(\hat{A})=\begin{cases}0& r\ge 1,\\\sigma^2(1-r^2)& 0\lt r\lt 1.\end{cases}$$If the geometric dies, extra samples do not help past a floor. Official 2025 3.1c.
\(\alpha=h(\theta)\) one-to-one, unbiased for \(\alpha\). Same Cauchy–Schwarz with an extra \(h'(\theta)\):
$$\mathrm{Var}[\hat{\alpha}]\ge\frac{\big(h'(\theta)\big)^2}{\mathcal{I}(\theta)}.$$Vector: \(\boldsymbol{\alpha}=h(\boldsymbol{\theta})\), Jacobian \(J=\partial h/\partial\boldsymbol{\theta}\) (\(r\times p\)),
$$\mathbf{C}_{\hat{\boldsymbol{\alpha}}}\ge J\,\mathbf{I}(\boldsymbol{\theta})^{-1} J^\top.$$Affine \(h(\theta)=a\theta+b\): efficiency is preserved. Nonlinear: generally not (the \(\bar{x}^2\) example in the previous chapter: \(\mathbb{E}[\bar{x}^2]=A^2+\sigma^2/N\)).
Reader 6.7. \(\mathbf{C}_{\hat{\boldsymbol{\theta}}}\ge\mathbf{I}(\boldsymbol{\theta})^{-1}\) in the PSD sense: \(\mathbf{a}^\top(\mathbf{C}-\mathbf{I}^{-1})\mathbf{a}\ge 0\) for all \(\mathbf{a}\neq 0\). In particular the diagonals \(\mathrm{Var}[\hat{\theta}_i]\ge[\mathbf{I}^{-1}]_{ii}\).
$$[\mathbf{I}]_{ij} =\mathbb{E}\left[\frac{\partial\ln p}{\partial\theta_i}\frac{\partial\ln p}{\partial\theta_j}\right] =-\mathbb{E}\left[\frac{\partial^2\ln p}{\partial\theta_i\partial\theta_j}\right].$$DC and variance together (reader Example 6.2). Off-diagonals 0, so
$$\mathbf{I}(A,\sigma^2)=\begin{bmatrix}N/\sigma^2&0\\0&N/(2\sigma^4)\end{bmatrix}, \qquad \mathbf{I}^{-1}=\begin{bmatrix}\sigma^2/N&0\\0&2\sigma^4/N\end{bmatrix}.$$Unknown extra parameter \(\sigma^2\) does not worsen the bound on \(A\) here (orthogonal scores). That is special, not general.
Reader (6.20) / VL5. An unbiased estimator \(g(\mathbf{x})\) is efficient iff
$$\frac{\partial\ln p(\mathbf{x};\theta)}{\partial\theta}=\mathcal{I}(\theta)\big(g(\mathbf{x})-\theta\big).$$Recipe on an exam:
Vector: \(\nabla_{\boldsymbol{\theta}}\ln p=\mathbf{I}(\boldsymbol{\theta})(g(\mathbf{x})-\boldsymbol{\theta})\).
Finding the MVUE in general is hard. Two routes this course actually uses:
There is no MVUE-finding machine for arbitrary PDFs.
Worked check, DC in WGN (previous chapter): the score was already \(\frac{N}{\sigma^2}(\bar{x}-A)\). Prefactor \(\mathcal{I}(A)=N/\sigma^2\), \(g=\bar{x}\). Efficient, MVUE, variance \(\sigma^2/N\).
Worked check, exponential rate: score \(N/\lambda-\sum_i x_i\). You cannot factor this as \(\mathcal{I}(\lambda)(g-\lambda)\) with \(g\) independent of \(\lambda\). No finite-sample efficient estimator. Second derivative \(-N/\lambda^2\), so \(\mathcal{I}(\lambda)=N/\lambda^2\) and CRLB \(=\lambda^2/N\). The MVUE \((N-1)/\sum x_i\) has variance \(\lambda^2/(N-2)>\lambda^2/N\).
IID exponential \(p(x;\lambda)=\lambda e^{-\lambda x}\), \(N\) samples. Write the score. Can you factor it as \(\mathcal{I}(\lambda)(g(\mathbf{x})-\lambda)\) with \(g\) free of \(\lambda\)? Compute \(\mathcal{I}(\lambda)\) from the second derivative and quote the CRLB.
\(\ln p=N\ln\lambda-\lambda\sum_i x_i\). Score \(N/\lambda-S\) with \(S=\sum_i x_i\). Second derivative \(-N/\lambda^2\), so \(\mathcal{I}(\lambda)=N/\lambda^2\) and \(\mathrm{Var}\ge\lambda^2/N\). If the score factored as \(\mathcal{I}(\lambda)(g-\lambda)\) with \(g\) free of \(\lambda\), then
$$g=\lambda+\frac{N/\lambda-S}{N/\lambda^2}=2\lambda-\frac{\lambda^2}{N}S,$$which still depends on \(\lambda\). No finite-sample efficient estimator. Contrast DC in WGN, where the score is already \(\frac{N}{\sigma^2}(\bar{x}-A)\).
MLE does not always achieve the CRLB — only if the efficiency identity holds, or asymptotically. Fisher is \(\ge 0\), not “strictly negative”. Attaining the CRLB means efficient, not “variance lower than CRLB” (that is impossible under regularity).
5CTA0 Q1 2026–27 · from the course reader, lectures and official keys.