Pick a signal model and minimise residual energy. No PDF is required. Linear in \(\boldsymbol{\theta}\): the normal equation. Known noise covariance: weight by \(\mathbf{C}^{-1}\) and you have the BLUE. Gaussian on top of that: BLUE = MLE = MVUE.
A linear signal model is linear in \(\boldsymbol{\theta}\), not necessarily in time:
$$ \mathbf{s}=\mathbf{H}\boldsymbol{\theta},\qquad\mathbf{x}=\mathbf{H}\boldsymbol{\theta}+\mathbf{w}. $$\(\mathbf{H}\) is \(N\times K\), known, full column rank. Polynomial trend: \(s[n]=\theta_0+\theta_1 n+\cdots+\theta_{K-1}n^{K-1}\), \(\mathbf{H}\) Vandermonde (\(0^0:=1\)). Two known-rate exponentials, sinusoid of known frequency: same idea, one column per unknown amplitude.
If additionally \(\mathbf{w}\sim\mathcal{N}(\mathbf{0},\mathbf{C})\),
$$ \ln p=-\frac12\ln|2\pi\mathbf{C}|-\frac12(\mathbf{x}-\mathbf{H}\boldsymbol{\theta})^\top\mathbf{C}^{-1}(\mathbf{x}-\mathbf{H}\boldsymbol{\theta}). $$The gradient of the quadratic is the score. Completing it (or matrix calculus) gives the efficiency identity:
$$ \frac{\partial\ln p}{\partial\boldsymbol{\theta}} =\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{x}-\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H}\,\boldsymbol{\theta} =\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H}\Big((\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{x}-\boldsymbol{\theta}\Big). $$Fisher \(\mathbf{I}(\boldsymbol{\theta})=\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H}\), and the estimator in the parentheses is efficient, hence MLE, hence MVUE:
$$ \hat{\boldsymbol{\theta}}=(\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{x},\qquad \mathrm{Cov}(\hat{\boldsymbol{\theta}})=(\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H})^{-1}. $$WGN special case \(\mathbf{C}=\sigma^2\mathbf{I}\):
$$ \hat{\boldsymbol{\theta}}=(\mathbf{H}^\top\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{x},\qquad \mathbf{I}(\boldsymbol{\theta})=\frac{\mathbf{H}^\top\mathbf{H}}{\sigma^2}. $$Same formula as linear LS. Under WGN, MLE = LS = BLUE = MVUE.
Known Fourier frequencies, multiples of \(1/N\), WGN: columns of \(\mathbf{H}\) are orthogonal, \(\mathbf{H}^\top\mathbf{H}=(N/2)\mathbf{I}\), and the amplitudes decouple into DFT-style sums
$$ \hat a_p=\frac{2}{N}\sum_{n=0}^{N-1}x[n]\cos\Big(2\pi\frac{k_p}{N}n\Big),\qquad \hat b_p=\frac{2}{N}\sum_{n=0}^{N-1}x[n]\sin\Big(2\pi\frac{k_p}{N}n\Big). $$Amplitude-and-phase form is nonlinear in the phase. Linearize with \(\cos(\alpha-\varphi)=\cos\alpha\cos\varphi+\sin\alpha\sin\varphi\), estimate \(a_p,b_p\), then \(c_p=\sqrt{a_p^2+b_p^2}\), \(\varphi_p=\mathrm{atan2}(b_p,a_p)\) (invariance).
SLT3 2024–25 Ex. 3.3, two exponentials \(r_0=-1\), \(r_1=1\), \(N\) even:
$$ \mathbf{H}=\begin{bmatrix}1&1\\-1&1\\1&1\\\vdots&\vdots\\-1&1\end{bmatrix},\quad \mathbf{H}^\top\mathbf{H}=N\mathbf{I},\quad \hat A_0=\frac{1}{N}\sum_{n=0}^{N-1}(-1)^n x[n],\quad \hat A_1=\frac{1}{N}\sum_{n=0}^{N-1}x[n]. $$LS does not use a PDF. You pick \(s[n](\theta)\) and minimise the residual energy
$$ J(\theta)=\sum_{n=0}^{N-1}\big(x[n]-s[n](\theta)\big)^2=\|\mathbf{x}-\mathbf{s}(\theta)\|^2. $$No claim of optimality, no variance formula, unless you add assumptions later (WGN \(\Rightarrow\) LS coincides with MLE; known \(\mathbf{C}\) \(\Rightarrow\) WLS / BLUE).
Exam says “no noise statistics” / “no PDF”: LS. Official 20/01/2020 Q4a: “Least squares, since it requires no function for the noise.” If the covariance is known, do not stay with ordinary LS — that is BLUE, next section.
\(\mathbf{s}=\mathbf{H}\boldsymbol{\theta}\), \(N>K\), \(\mathbf{H}\) full rank.
$$ \begin{aligned} J(\boldsymbol{\theta}) &=\|\mathbf{x}-\mathbf{H}\boldsymbol{\theta}\|^2 =(\mathbf{x}-\mathbf{H}\boldsymbol{\theta})^\top(\mathbf{x}-\mathbf{H}\boldsymbol{\theta})\\ &=\mathbf{x}^\top\mathbf{x}-2\mathbf{x}^\top\mathbf{H}\boldsymbol{\theta}+\boldsymbol{\theta}^\top\mathbf{H}^\top\mathbf{H}\boldsymbol{\theta}. \end{aligned} $$\(\mathbf{H}^\top\mathbf{H}\) is symmetric. Differentiate: \(\partial(\boldsymbol{\theta}^\top\mathbf{A}\boldsymbol{\theta})/\partial\boldsymbol{\theta}=2\mathbf{A}\boldsymbol{\theta}\), \(\partial(\mathbf{b}^\top\boldsymbol{\theta})/\partial\boldsymbol{\theta}=\mathbf{b}\).
$$ \frac{\partial J}{\partial\boldsymbol{\theta}}=-2\mathbf{H}^\top\mathbf{x}+2\mathbf{H}^\top\mathbf{H}\boldsymbol{\theta}\stackrel{!}{=}0 \qquad\Rightarrow\qquad \mathbf{H}^\top\mathbf{H}\,\hat{\boldsymbol{\theta}}=\mathbf{H}^\top\mathbf{x}. $$That is the normal equation. Invert:
$$ \hat{\boldsymbol{\theta}}_{\mathrm{LS}}=(\mathbf{H}^\top\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{x}. $$Scalar through the origin (\(\mathbf{H}=\mathbf{h}\) a vector — electric car; sinusoid amplitude below):
$$ \hat\theta_{\mathrm{LS}}=\frac{\mathbf{h}^\top\mathbf{x}}{\mathbf{h}^\top\mathbf{h}}=\frac{\sum_n h[n]\,x[n]}{\sum_n h[n]^2}. $$Columns of \(\mathbf{H}\) span a \(K\)-dimensional subspace of \(\mathbb{R}^N\). The candidate signals \(\mathbf{s}=\mathbf{H}\boldsymbol{\theta}\) live in that subspace. \(J=\|\mathbf{e}\|^2\) with \(\mathbf{e}=\mathbf{x}-\mathbf{s}\). The shortest error is orthogonal to the subspace (orthogonality principle):
$$ \mathbf{H}^\top(\mathbf{x}-\mathbf{H}\hat{\boldsymbol{\theta}})=\mathbf{0}, $$which is again the normal equation. The fitted signal is the orthogonal projection of the data onto \(\mathrm{col}(\mathbf{H})\):
$$ \mathbf{P}=\mathbf{H}(\mathbf{H}^\top\mathbf{H})^{-1}\mathbf{H}^\top,\qquad \hat{\mathbf{s}}=\mathbf{P}\mathbf{x},\qquad \mathbf{e}=(\mathbf{I}-\mathbf{P})\mathbf{x}\perp\mathrm{col}(\mathbf{H}). $$\(\mathbf{P}\) is symmetric and idempotent: \(\mathbf{P}^\top=\mathbf{P}\), \(\mathbf{P}^2=\mathbf{P}\).
Resit 2022 2c: “the solution vector \(\hat{\boldsymbol{\theta}}\) is orthogonal to the span of the columns of \(\mathbf{H}\)” is false. \(\hat{\mathbf{s}}=\mathbf{H}\hat{\boldsymbol{\theta}}\) lies in the span; the residual is orthogonal to the span. Official correct MCQ that year was (a) (nonlinear LS is sometimes linearizable). Linearization-by-transform was “no longer part of the content” in that grading; Gauss–Newton still is.
Replace the Euclidean inner product by a positive-definite weight \(\mathbf{W}\):
$$ J=(\mathbf{x}-\mathbf{H}\boldsymbol{\theta})^\top\mathbf{W}(\mathbf{x}-\mathbf{H}\boldsymbol{\theta}), \qquad \hat{\boldsymbol{\theta}}_{\mathrm{WLS}}=(\mathbf{H}^\top\mathbf{W}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{W}\mathbf{x}. $$If \(\mathbb{E}[\mathbf{x}]=\mathbf{H}\boldsymbol{\theta}\) (linear mean), WLS is unbiased for any PD \(\mathbf{W}\):
$$ \mathbb{E}[\hat{\boldsymbol{\theta}}_{\mathrm{WLS}}] =(\mathbf{H}^\top\mathbf{W}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{W}\,\mathbf{H}\boldsymbol{\theta} =\boldsymbol{\theta}. $$Let \(\mathbf{C}_x=\mathrm{Cov}(\mathbf{x})\). Then \(\hat{\boldsymbol{\theta}}=\mathbf{A}\mathbf{x}\) with \(\mathbf{A}=(\mathbf{H}^\top\mathbf{W}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{W}\), so
$$ \mathbf{C}_{\hat{\boldsymbol{\theta}}} =\mathbf{A}\,\mathbf{C}_x\,\mathbf{A}^\top =(\mathbf{H}^\top\mathbf{W}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{W}\,\mathbf{C}_x\,\mathbf{W}\mathbf{H}\,(\mathbf{H}^\top\mathbf{W}\mathbf{H})^{-1}. $$Choose \(\mathbf{W}=\mathbf{C}_x^{-1}\). The sandwich collapses:
$$ \mathbf{H}^\top\mathbf{W}\,\mathbf{C}_x\,\mathbf{W}\mathbf{H} =\mathbf{H}^\top\mathbf{C}_x^{-1}\mathbf{C}_x\mathbf{C}_x^{-1}\mathbf{H} =\mathbf{H}^\top\mathbf{C}_x^{-1}\mathbf{H}, $$ $$ \mathbf{C}_{\hat{\boldsymbol{\theta}}}=(\mathbf{H}^\top\mathbf{C}_x^{-1}\mathbf{H})^{-1}. $$Gauss–Markov: this is the smallest covariance among all linear unbiased estimators. Write any other linear unbiased estimator as \(\mathbf{A}'=\mathbf{A}+\mathbf{B}\) with \(\mathbf{B}\mathbf{H}=\mathbf{0}\) (unbiasedness). The extra term \(\mathbf{B}\mathbf{C}_x\mathbf{B}^\top\) is PSD, so you cannot beat \(\mathbf{A}\) inside the linear class. That estimator is the BLUE:
$$ \hat{\boldsymbol{\theta}}_{\mathrm{BLUE}}=(\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{C}^{-1}\mathbf{x}. $$Same formula as the Gaussian linear-model MLE. BLUE only needs mean and covariance, not Gaussianity, and only claims optimality inside the linear class. If the noise is Gaussian, BLUE = MLE = MVUE.
If \(\mathbf{C}=\sigma^2\mathbf{I}\), BLUE = ordinary LS.
If \(\mathbf{C}\) is known, ordinary LS ignores information you have. Use \(\mathbf{W}=\mathbf{C}^{-1}\). Ordinary LS is BLUE only when the noise is white (or already whitened).
\(y[n]=\alpha\sin(\omega n)+w[n]\), \(\omega\) known. Scalar LS with \(h[n]=\sin(\omega n)\):
$$ J(\alpha)=\sum_i\big(y[n_i]-\alpha\sin(\omega n_i)\big)^2, $$ $$ \frac{\partial J}{\partial\alpha}=-2\sum_i\big(y[n_i]-\alpha\sin(\omega n_i)\big)\sin(\omega n_i)\stackrel{!}{=}0, $$ $$ \alpha_{\mathrm{LS}}=\frac{\sum_i\sin(\omega n_i)\,y[n_i]}{\sum_i\sin^2(\omega n_i)}. $$Data: \(\omega=2\), \(y[1..8]=2.31,-2.01,-1.33,3.23,-1.28,-1.66,3.28,-0.88\). Official: \(\alpha_{\mathrm{LS}}=2.99\). Direct evaluation \(\sum\sin(2n)\,y[n]=12.2785\), \(\sum\sin^2(2n)=4.1045\), ratio \(2.991\).
Collect \(A\): the first sum contributes \(M A-\sum_{n=0}^{M-1}x[n]\) after dividing by 2 and flipping sign; the second contributes \((N-M)A+\sum_{n=M}^{N-1}x[n]\). Together
$$ N A-\sum_{n=0}^{M-1}x[n]+\sum_{n=M}^{N-1}x[n]=0, \qquad \hat A=\frac{1}{N}\sum_{n=0}^{M-1}x[n]-\frac{1}{N}\sum_{n=M}^{N-1}x[n]. $$If \(x[n]=s[n]+\)WGN, \(\mathbb{E}[\hat A]=A\), \(\mathrm{Var}[\hat A]=\sigma^2/N\), so \(\hat A\sim\mathcal{N}(A,\sigma^2/N)\). Official (4). Linear model with \(\mathbf{H}=[+1,\ldots,+1,-1,\ldots,-1]^\top\), so under WGN this LSE is also the MLE / MVUE.
\(x[n]=A\cos(2\pi f_1 n)+w[n]\), \(\mathbf{C}=\sigma^2\mathbf{I}\). Then BLUE = LS, \(\mathbf{H}=[1,\cos(2\pi f_1),\ldots,\cos(2\pi f_1(N-1))]^\top\):
$$ \hat A_{\mathrm{BLUE}}=\frac{\sum_{n=0}^{N-1}\cos(2\pi f_1 n)\,x[n]}{\sum_{n=0}^{N-1}\cos^2(2\pi f_1 n)}, \qquad \mathrm{Var}[\hat A]=\frac{\sigma^2}{\sum_n\cos^2(2\pi f_1 n)}. $$\(\mathbb{E}[\hat A\mid s_1]=A\), \(\mathbb{E}[\hat A\mid s_0]=0\), same variance under both. Threshold in the middle: \(\gamma=A/2\). Minimize variance \(\Leftrightarrow\) maximize \(\sum\cos^2\): official best frequency in \([0,1/2)\) is \(f_1=0\), \(\mathrm{Var}\to\sigma^2/N\) (a DC; max signal energy). A communications engineer may still not want \(f_1=0\); the estimation question is only about variance.
If \(s(\boldsymbol{\theta})\) is nonlinear, \(J\) need not have a closed-form min. Two course devices.
A. Transform / separate. If \(\boldsymbol{\alpha}=g(\boldsymbol{\theta})\) makes \(\mathbf{s}=\mathbf{H}\boldsymbol{\alpha}\), do linear LS in \(\boldsymbol{\alpha}\), invert \(g\) (phase via \(a,b\) above).
B. Iterate. Gauss–Newton linearizes the signal about the current guess \(\boldsymbol{\theta}_k\):
$$ \mathbf{s}(\boldsymbol{\theta})\approx\mathbf{s}(\boldsymbol{\theta}_k)+\mathbf{H}_k(\boldsymbol{\theta}-\boldsymbol{\theta}_k),\qquad [\mathbf{H}_k]_{n,i}=\frac{\partial s[n;\boldsymbol{\theta}]}{\partial\theta_i}\Big|_{\boldsymbol{\theta}_k}. $$Then one linear LS step for the increment:
$$ \boldsymbol{\theta}_{k+1}=\boldsymbol{\theta}_k+(\mathbf{H}_k^\top\mathbf{H}_k)^{-1}\mathbf{H}_k^\top\big(\mathbf{x}-\mathbf{s}(\boldsymbol{\theta}_k)\big). $$Newton–Raphson iterates on \(J\) itself:
$$ \boldsymbol{\theta}_{k+1}=\boldsymbol{\theta}_k-(\nabla^2 J)^{-1}\nabla J. $$The Hessian of \(J\) includes second derivatives of \(s\). Gauss–Newton approximates that Hessian by \(2\mathbf{H}^\top\mathbf{H}\) (the factor 2 cancels with \(\nabla J\)) and drops residual \(\times\) second-derivative terms.
Resit 2022 2c: “only Gauss–Newton, not Newton–Raphson” is false. Both are legal. Gauss–Newton uses the Jacobian of \(s\). Newton uses \(\nabla^2 J\). Do not swap the two formulae.
\(s[n]=A e^{\phi n}\), \(\phi<0\). (a) LS, no noise PDF. (b) \(\phi\) known: linear in \(A\),
$$ \hat A_{\mathrm{LS}}=\frac{\sum_{n=0}^{N} x[n]e^{\phi n}}{\sum_{n=0}^{N}e^{2\phi n}}. $$(The key writes the sums \(n=0..N\).) (c) Both unknown: \(J(\phi,A)=\sum_n(x[n]-A e^{\phi n})^2\),
$$ \nabla J=\begin{bmatrix} \sum_n 2A n e^{\phi n}(A e^{\phi n}-x[n])\\ \sum_n 2 e^{\phi n}(A e^{\phi n}-x[n]) \end{bmatrix}. $$(d) The question asked Gauss–Newton. The official key wrote Newton–Raphson on \(J\):
$$ \begin{bmatrix}\phi_{k+1}\\ A_{k+1}\end{bmatrix} =\begin{bmatrix}\phi_k\\ A_k\end{bmatrix} -(\nabla^2 J)^{-1}\nabla J. $$Teach both. Gauss–Newton Jacobian, row \(n\): \(\big[A_k n e^{\phi_k n},\; e^{\phi_k n}\big]\), then the increment formula with \(\mathbf{H}_k\). If the grader follows the 2020 key, the Hessian form is what they printed.
Learning goal: choose the method from the signal model and from what is known (noise stats? prior?).
| Method | Prior | Claim | |
|---|---|---|---|
| MLE | yes | no | asy. efficient; efficient if the identity holds. \(\arg\max p(x;\theta)\) |
| LS | no | no | none in general. \(\arg\min\|x-s\|^2\); linear: \((H^\top H)^{-1}H^\top x\) |
| BLUE | mean+cov | no | best linear unbiased. \((H^\top C^{-1}H)^{-1}H^\top C^{-1}x\) |
| MMSE | yes, as \(p(x\mid\theta)\) | yes | min posterior 2nd moment. \(\mathbb{E}[\theta\mid x]\) |
| MAP | yes | yes | min hit-or-miss cost. \(\arg\max p(\theta\mid x)\) |
Coincidences worth the A4:
Exam 20/01/2020 Q4a, official: unknown noise law, exponential signal \(\Rightarrow\) LS.
MCQ traps that sit next to this table: CRLB is \(1/\mathcal{I}\), not \(\mathcal{I}\); Fisher is \(\ge 0\), not “strictly negative”; MLE does not always achieve the CRLB; MLE can be found whenever the PDF is known, not only for linear models; Bayesian estimators do use the prior; uniform cost \(\to\) MAP, not MMSE; residual \(\perp\mathrm{col}(H)\), not \(\hat\theta\) itself.
20/01/2020 Q4a. \(s[n]=A e^{\phi n}\), \(\phi<0\), iid samples, noise PDF not given. Which estimator, and why? If \(\phi\) is known, write \(\hat A\).
Least squares: it requires no function for the noise. Official Q4a. With \(\phi\) known the model is linear in \(A\), \(h[n]=e^{\phi n}\),
$$ \hat A_{\mathrm{LS}}=\frac{\sum_n x[n]e^{\phi n}}{\sum_n e^{2\phi n}}. $$Key writes the sums \(n=0..N\).
SLT4 2025–26 Ex. 4.1. \(s[n]=A\) on \(0\le n\le M-1\) and \(s[n]=-A\) on \(M\le n\le N-1\). Derive \(\hat A_{\mathrm{LS}}\). If the noise is WGN, give mean and variance.
Differentiate \(J(A)=\sum_{n=0}^{M-1}(x[n]-A)^2+\sum_{n=M}^{N-1}(x[n]+A)^2\):
$$ \hat A=\frac{1}{N}\sum_{n=0}^{M-1}x[n]-\frac{1}{N}\sum_{n=M}^{N-1}x[n]. $$Under WGN, unbiased, \(\mathrm{Var}=\sigma^2/N\), so \(\hat A\sim\mathcal{N}(A,\sigma^2/N)\). Official (4). Also MLE / MVUE (linear model, white Gaussian).
SLT3 2025–26 Ex. 3.3. \(x[n]=A\cos(2\pi f_1 n)+w[n]\), \(\mathbf{C}=\sigma^2\mathbf{I}\). Write \(\hat A_{\mathrm{BLUE}}\) and its variance. Which \(f_1\in[0,1/2)\) minimises that variance?
Minimise variance \(\Leftrightarrow\) maximise \(\sum\cos^2\). Official: \(f_1=0\), variance \(\sigma^2/N\).
\(s[n]=A e^{\phi n}\), both unknown. Write (i) the Gauss–Newton step, including the Jacobian row, and (ii) the Newton–Raphson step on \(J\). Which one did the 20/01/2020 key print?
(i) Jacobian row \(n\) at the current guess: \(\big[A_k n e^{\phi_k n},\; e^{\phi_k n}\big]\). Then
$$ \boldsymbol{\theta}_{k+1}=\boldsymbol{\theta}_k+(\mathbf{H}_k^\top\mathbf{H}_k)^{-1}\mathbf{H}_k^\top\big(\mathbf{x}-\mathbf{s}(\boldsymbol{\theta}_k)\big). $$(ii) Newton on \(J\):
$$ \begin{bmatrix}\phi_{k+1}\\ A_{k+1}\end{bmatrix} =\begin{bmatrix}\phi_k\\ A_k\end{bmatrix}-(\nabla^2 J)^{-1}\nabla J, $$with \(\nabla J\) as in Q4c. The 2020 key printed (ii) even though the question said Gauss–Newton. Both are legal (resit 2022 2c).
5CTA0 Q1 2026–27 · from the course reader, lectures and official keys.