Question

Consider a location model $$ X_{i}=\theta+e_{i}, \quad i=1, \ldots, n $$ where \(e_{1}, e_{2}, \ldots, e_{n}\) are iid with pdf \(f(z)\). There is a nice geometric interpretation for estimating \(\theta\). Let \(\mathbf{X}=\left(X_{1}, \ldots, X_{n}\right)^{\prime}\) and \(\mathbf{e}=\left(e_{1}, \ldots, e_{n}\right)^{\prime}\) be the vectors of observations and random error, respectively, and let \(\boldsymbol{\mu}=\theta \mathbf{1}\), where \(\mathbf{1}\) is a vector with all components equal to 1 . Let \(V\) be the subspace of vectors of the form \(\mu\); i.e., \(V=\\{\mathbf{v}: \mathbf{v}=a \mathbf{1}\), for some \(a \in R\\}\). Then in vector notation we can write the model as $$ \mathbf{X}=\boldsymbol{\mu}+\mathbf{e}, \quad \boldsymbol{\mu} \in V $$ Then we can summarize the model by saying, "Except for the random error vector e, X would reside in \(V\)." Hence, it makes sense intuitively to estimate \(\boldsymbol{\mu}\) by a vector in \(V\) that is "closest" to \(\mathbf{X}\). That is, given a norm \(\|\cdot\|\) in \(R^{n}\), choose $$ \widehat{\boldsymbol{\mu}}=\operatorname{Argmin}\|\mathbf{X}-\mathbf{v}\|, \quad \mathbf{v} \in V $$ (a) If the error pdf is the Laplace, \((2.2 .4)\), show that the minimization in \((6.3 .27)\) is equivalent to maximizing the likelihood when the norm is the \(l_{1}\) norm given by $$ \|\mathbf{v}\|_{1}=\sum_{i=1}^{n}\left|v_{i}\right| $$ (b) If the error pdf is the \(N(0,1)\), show that the minimization in \((6.3 .27)\) is equivalent to maximizing the likelihood when the norm is given by the square of the \(l_{2}\) norm $$ \|\mathbf{v}\|_{2}^{2}=\sum_{i=1}^{n} v_{i}^{2} $$

Short answer

For the Laplace error distribution, the location model minimization problem under the L1 norm is equivalent to the likelihood maximization problem, while for N(0,1) error, the equivalent occurs under the square of the L2 norm.

Step-by-step solution

Problem Analysis and Set Up

We start by noting down the given location model \(X_{i} = \theta + e_{i}\), where \(e_{1}, e_{2}, ..., e_{n}\) are iid with pdf \(f(z)\). The goal is to find \(\widehat{\mu}\) such that it minimizes the distance \(||X - v||\), where \(v\) belongs to the subspace \(V\) and \(X\) is the vector of observations. Now, for Laplace and Normal distributions, we will set up their likelihood functions.

Maximization for Laplace Error

For the Laplace distribution, the pdf is given by \(f(z)= 1/2 * e^{-|z|}\). Using this pdf, we write down the likelihood for n observations \(L= (1/2)^n * e^{-\sum |x_{i} - a|}\). On taking natural logarithm, we get log likelihood as \(-n log(2) - \sum |x_{i} - a|\). As maximizing the likelihood is equivalent to minimizing the negative of log-likelihood, we see that |\(x_{i} - a|_1\) is minimized, which is consistent with L1-norm minimization.

Maximization for Normal Error

For \(N(0,1)\), the pdf is \(f(z) = e^{-z^2/2} / sqrt(2*\pi)\), giving likelihood \(L = (1/sqrt(2*\pi))^n * e^{- \sum (x_{i} - a)^2 / 2}\). Log likelihood becomes \(-n/2 * log(2\pi) - \sum (x_{i}-a)^2 / 2\). We see that in order to maximize this function, we need to minimize \(||(x_{i}-a)||_2^2\), which is equivalent to minimizing the square of L2-norm.

Key concepts

Location Model

In mathematical statistics, the location model represents a situation where observations differ from some central value, known as the location parameter, by some random error. Imagine you have a target and each shot lands at a slight random distance from the bullseye, which in this context is the location parameter.

The simplest form of a location model can be expressed as \( X_{i} = \theta + e_{i} \), where \( X_{i} \) represents the observed data, \( \theta \) is the location parameter we're trying to estimate, and \( e_{i} \) represents the random error added to each observation. These errors are considered independent and identically distributed (iid), meaning they follow the same probability distribution and are statistically independent from each other. A powerful aspect of the location model is that it can provide insights into the structure of a dataset by isolating the 'signal' (\(\theta\)) from the 'noise' (\(e_{i}\)).

When considering a dataset in vector form, the goal is often to estimate the vector \( \boldsymbol{\mu} \) in a simpler subspace \( V \). Here, we aim to find a value that best represents the central tendency of our data given the random variations. This means we are looking for a point in our subspace \( V \) that is closest to our observational vector \(\mathbf{X}\), hence the connection to 'norms' which measure distances in vector spaces.

Likelihood Maximization

The principle of likelihood maximization is central to statistical inference. To estimate parameters of a statistical model, we use observed data to find the parameter values that make the observed data most probable - this process is what we call maximizing the likelihood.

Consider a scenario where we have a certain model, and we've observed some data points. How do we know which parameter value is the best? We turn to likelihood maximization. For each candidate parameter value, we calculate the likelihood; that is, the probability of observing our data given that parameter value according to our model. The 'best' estimate of the parameter is then the value that gives us the highest likelihood.

This is analogous to finding the right setting on a finely tuned radio: twist the dial until the signal comes through loud and clear with the least amount of static. In the context of the location model, the process is mathematically similar to minimizing a certain norm of the residuals between the assumed model and the observed data. For different error distributions, this maximization takes different forms, aligning with different norms.

L1 and L2 Norms

In the realm of mathematics, L1 and L2 norms are methods of quantifying the 'size' or 'length' of vectors in some space, which, in statistics, corresponds to the distance between two points - typically between our data and our model predictions.

Imagine you are plotting a course through a city. The L1 norm, also known as the Manhattan norm or taxicab norm, is like navigating by driving strictly along the grid of streets and avenues - you only move vertically or horizontally. Mathematically, for the vector \( \mathbf{v} = (v_{1}, ..., v_{n}) \) the L1 norm is defined as \[ ||\mathbf{v}||_{1} = \sum_{i=1}^{n}|v_{i}| \], summing the absolute values of the components.

Conversely, the L2 norm is akin to taking a direct path, as the crow flies, to your destination. The L2 norm is also known as the Euclidean norm, and for the same vector \( \mathbf{v} \) it is \[ ||\mathbf{v}||_{2}^{2} = \sum_{i=1}^{n} v_{i}^{2} \], which is essentially the square of the direct distance from the origin to the point in space represented by \( \mathbf{v} \). In statistics, these norms help in estimating parameters by finding the 'shortest' distance in their respective manners, which aligns with the intuition behind least squares and absolute deviation minimization.