Question

Consider a linear regression model (8.1) in which the errors \(\varepsilon_{j}\) are independently distributed with Laplace density $$ f(u ; \sigma)=\left(2^{3 / 2} \sigma\right)^{-1} \exp \left\\{-\left|u /\left(2^{1 / 2} \sigma\right)\right|\right\\}, \quad-\infty0. $$ Verify that this density has variance \(\sigma^{2} .\) Show that the maximum likelihood estimate of \(\beta\) is obtained by minimizing the \(L^{1}\) norm \(\sum\left|y_{j}-x_{j}^{\mathrm{T}} \beta\right|\) of \(y-X \beta\). Show that if in fact the \(\varepsilon_{j} \stackrel{\text { iid }}{\sim} N\left(0, \sigma^{2}\right)\), the asymptotic relative efficiency of the estimators relative to least squares estimators is \(2 / \pi\).

Short answer

The variance is \( \sigma^2 \), MLE for \( \beta \) minimizes \( L^1 \) norm, and relative efficiency is \( 2/\pi \).

Step-by-step solution

Verify Variance of Laplace Distribution

To find the variance of the given Laplace distribution, recall that the variance of a Laplace distribution with scale parameter \( b \) is \( 2b^2 \). Here, \( b = \sigma / \sqrt{2} \), so the variance becomes \( 2(\sigma/\sqrt{2})^2 = \sigma^2 \).

Set Up Maximum Likelihood

Given the density function, the likelihood for multiple independent errors is the product of their densities. For \( n \) observations, this is:\[ L(\beta, \sigma) = \prod_{j=1}^{n} (2^{3/2} \sigma)^{-1} \exp\left(-\left|\varepsilon_j / (2^{1/2} \sigma)\right| \right) \]where \( \varepsilon_j = y_j - x_j^T \beta \).

Derive Log-Likelihood Function

Take the natural logarithm of the likelihood function to derive the log-likelihood:\[ \log L = \sum_{j=1}^{n} \left[ -\log(2^{3/2} \sigma) - \frac{|\varepsilon_j|}{2^{1/2} \sigma} \right] \]

Find Maximum Likelihood Estimator for \( \beta \)

The term involving \( \beta \) in the log-likelihood is:\[ -\sum \frac{|\varepsilon_j|}{2^{1/2} \sigma} \]which is minimized when the \( L^1 \) norm \( \sum | y_j - x_j^T \beta | \) is minimized. Hence, minimizing the \( L^1 \) norm gives the maximum likelihood estimator for \( \beta \).

Asymptotic Relative Efficiency for Normally Distributed Errors

If errors are normally distributed \( \varepsilon_j \sim N(0, \sigma^2) \), compare the variances of least squares estimator and the \( L^1 \) estimator asymptotically. Least squares estimator has variance \( \sigma^2/n \) and the \( L^1 \) estimator's variance is \( \pi \sigma^2 / 2n \). Therefore, the asymptotic relative efficiency is \( 2/\pi \).

Key concepts

Laplace Distribution

The Laplace distribution is a continuous probability distribution that is often used in various statistical models, especially when dealing with data that demonstrates sharper peaks and heavier tails compared to the normal distribution. It is particularly useful in robust regression methods where errors might not follow a normal pattern.

The given probability density function (PDF) for a Laplace distribution is centered around its mean. For a scale parameter denoted as \( b \), the PDF is expressed as: \[f(u ; \, \sigma) = \frac{1}{\sqrt{2}b} \exp \left(-\frac{|u|}{b}\right), \] where, in this exercise, \( b = \sigma / \sqrt{2} \).

The variance for this distribution is calculated as \( 2b^2 \). Substituting the relationship between \( b \) and \( \sigma \), we verify the variance: \[ 2 \left(\frac{\sigma}{\sqrt{2}}\right)^2 = \sigma^2. \] This characteristic makes the Laplace distribution suitable for modeling symmetric data with potentially large deviations.

Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) is a method of estimating the parameters of a statistical model. For the exercise, MLE is used to estimate the coefficients \( \beta \) in a linear regression model, where the errors follow a Laplace distribution. This technique finds the parameter values that make the observed data most probable.

To set up the MLE for this linear model, we start by defining the likelihood function, which is the product of the probability densities for all observed data points, given the parameters we want to estimate. Here, the likelihood \( L(\beta, \sigma) \) is: \[L(\beta, \sigma) = \prod_{j=1}^{n} (2^{3/2} \sigma)^{-1} \exp\left(-\left|\varepsilon_j / (2^{1/2} \sigma)\right| \right) \]where \( \varepsilon_j = y_j - x_j^T \beta \).

After transforming this to the log-likelihood function, the term involving \( \beta \) becomes crucial: \[-\sum \frac{|\varepsilon_j|}{2^{1/2} \sigma}. \] Minimizing this term corresponds to minimizing the \( L^1 \) norm \( \sum | y_j - x_j^T \beta | \), leading us to the maximum likelihood estimator for \( \beta \).

Asymptotic Efficiency

Asymptotic efficiency refers to the performance of an estimator when the sample size approaches infinity. It is a crucial concept in statistics for comparing different estimation methods. If two estimators are asymptotically efficient, they will perform similarly well in large samples.

In the context of this exercise, we compare the asymptotic behavior of estimators derived from the Laplace setting with those from a normal error model. Specifically, the \( L^1 \) estimator's variance in an asymptotic sense is calculated as \( \pi \sigma^2 / 2n \) for normally distributed errors \( \varepsilon_j \sim N(0, \sigma^2) \).

In comparison, the well-known least squares estimator's variance is \( \sigma^2/n \). Therefore, the ratio of these variances, known as the asymptotic relative efficiency (ARE), is given by: \[\frac{\sigma^2/n}{\pi \sigma^2 / 2n} = \frac{2}{\pi}. \] This indicates that the \( L^1 \) norm-based estimator retains a lower efficiency compared to the least squares method, but it gains robustness, offering advantages in the presence of outliers.

Least Squares Estimator

The least squares estimator is a standard method in linear regression which aims to minimize the sum of the squared differences between the observed data and the predicted values: \[\sum (y_j - x_j^T \beta)^2. \] This method is often used when the error terms are assumed to follow a normal distribution.

Due to its reliance on squared differences, the least squares estimator is sensitive to outliers or non-normal error distributions as it can be heavily influenced by large deviations in certain data points.

Despite this, the least squares estimator remains popular due to its simplicity and the ease with which it can be implemented. Moreover, under normal distribution assumptions, it results in estimators for \( \beta \) that are unbiased, efficient, and consistent, meaning they converge to the true parameter values as the sample size increases. It also serves as a benchmark for evaluating other estimators, such as the \( L^1 \) norm-based estimator, particularly in terms of asymptotic properties like efficiency.