Question

The Laplace or double exponential distribution has density $$ f(y ; \mu, \sigma)=\frac{1}{2 \sigma} \exp (-|y-\mu| / \sigma), \quad-\infty0 $$ Sketch the log likelihood for a typical sample, and explain why the maximum likelihood estimate is only unique when the sample size is odd. Derive the score statistic and observed information. Is maximum likelihood estimation regular for this distribution?

Short answer

MLE is unique for odd samples; derivatives show non-regularity due to non-differentiability at median values.

Step-by-step solution

Understanding the Log Likelihood Function

Given the density function of the Laplace distribution, the log likelihood for a sample \( y_1, y_2, \... , y_n \) is derived by taking the natural logarithm of the product of individual densities:\[ \log L(\mu, \sigma) = \sum_{i=1}^{n} \log \left( \frac{1}{2\sigma} \exp\left(-\frac{|y_i - \mu|}{\sigma}\right) \right). \]This simplifies to:\[ \log L(\mu, \sigma) = -n \log(2\sigma) - \frac{1}{\sigma} \sum_{i=1}^{n} |y_i - \mu|. \]

Sketching the Log Likelihood

The log likelihood function \( \log L(\mu, \sigma) \) depends on the absolute differences \( |y_i - \mu| \). Visually, it resembles a peak at the median of the sample minus a scaling factor for \( \sigma \). For an even sample size, multiple medians make the peak not sharp, leading to non-uniqueness.

Explaining Uniqueness of MLE

The maximum likelihood estimate (MLE) is the value of \( \mu \) that minimizes \( \sum_{i=1}^{n} |y_i - \mu| \), which is typically attained at the sample median. When the sample size is odd, the median is unique, whereas for even sample sizes, any value between the two central data points is a median, leading to multiple possible ML estimates.

Deriving the Score Statistics

The score function for \( \mu \) is:\[ \frac{\partial}{\partial \mu} \log L = \frac{1}{\sigma} \sum_{i=1}^{n} \text{sign}(y_i - \mu). \]For \( \sigma \):\[ \frac{\partial}{\partial \sigma} \log L = -\frac{n}{\sigma} + \frac{1}{\sigma^2} \sum_{i=1}^{n} |y_i - \mu|. \]

Calculating the Observed Information

The observed information, the second derivative of the log likelihood, for \( \mu \) is zero everywhere except at data points, indicating jumps. For \( \sigma \):\[ \frac{\partial^2}{\partial \sigma^2} \log L = \frac{2n}{\sigma^2} - \frac{2}{\sigma^3} \sum_{i=1}^{n} |y_i - \mu|. \]

Regularity of MLE

MLE is regular if the Fisher information matrix is non-singular everywhere. In this case, due to the dependency of uniqueness on sample size and non-differentiability at certain points, MLE is not considered regular for this distribution.

Key concepts

Laplace Distribution

The Laplace distribution, also known as the double exponential distribution, is a common probability distribution. It is defined by its probability density function:

• \[ f(y; \mu, \sigma) = \frac{1}{2\sigma} \exp\left(-\frac{|y-\mu|}{\sigma}\right) \]

In this function:

• \(y\) is the random variable, which can take any value from \(-\infty\) to \(\infty\).
• \(\mu\) is the location parameter, also ranging from \(-\infty\) to \(\infty\). It identifies the peak of the distribution, much like the mean in the normal distribution.
• \(\sigma\) is the scale parameter, determining the spread of the distribution. It must be a positive number.

The Laplace distribution is named for its characteristic shape, which is peaked at its mean (or median) \(\mu\), and falls off more sharply than a Gaussian distribution. It is widely used in statistics for modeling data that tend to have fatter tails or more extreme values than a normal distribution.

The log likelihood function for a sample from the Laplace distribution plays a crucial role in estimating parameters such as \(\mu\) and \(\sigma\). Understanding this function is key to applying Maximum Likelihood Estimation (MLE) to this distribution.

Score Statistic

The score statistic is an important concept in maximum likelihood estimation. It helps us determine the sensitivity of the likelihood function to changes in the parameter estimates.For the Laplace distribution, we derive the score function by differentiating the log likelihood function with respect to the parameters \(\mu\) and \(\sigma\). Here's how this works:

• The score function for \(\mu\) is given by:\[\frac{\partial}{\partial \mu} \log L = \frac{1}{\sigma} \sum_{i=1}^{n} \text{sign}(y_i - \mu),\]where \(\text{sign}(y_i - \mu)\) represents the sign function, returning -1, 0, or 1 based on whether the argument is negative, zero, or positive, respectively.
• The score function for \(\sigma\) is given by:\[\frac{\partial}{\partial \sigma} \log L = -\frac{n}{\sigma} + \frac{1}{\sigma^2} \sum_{i=1}^{n} |y_i - \mu|.\]

These derivatives indicate how much the likelihood function changes as \(\mu\) or \(\sigma\) changes. The score statistic helps us find the parameter values that maximize the likelihood, as these occur when the scores are zero.

Understanding and deriving the score statistic is crucial because it not only aids in parameter estimation but also provides insights into the behavior of the likelihood function around the estimated values.

Log Likelihood Function

The log likelihood function is an essential tool in statistical estimation, especially in maximum likelihood estimation. For the Laplace distribution, the log likelihood function helps us estimate the parameters \(\mu\) and \(\sigma\). Here's how it works:Starting from the density function of the Laplace distribution, the log likelihood for a sample from this distribution is obtained by taking the natural logarithm of the product of individual probabilities:

• \[\log L(\mu, \sigma) = \sum_{i=1}^{n} \log \left( \frac{1}{2\sigma} \exp\left(-\frac{|y_i - \mu|}{\sigma}\right) \right) = -n \log(2\sigma) - \frac{1}{\sigma} \sum_{i=1}^{n} |y_i - \mu|.\]

This expression involves two main components:

• The term \- \(n \log(2\sigma)\) represents the constant parts of the likelihood that are not directly influenced by changes in the sample.
• The term \- \(\frac{1}{\sigma} \sum_{i=1}^{n} |y_i - \mu|\) influences the shape of the log likelihood and depends heavily on the difference \(|y_i - \mu|\), which are the absolute deviations of the data from the parameter \(\mu\).

Understanding this log likelihood function is crucial because it helps us find the MLE for \(\mu\), which is typically the sample median. The uniqueness of the MLE for even sample sizes is an interesting property, as it can lead to multiple estimates being equally optimal.

Observed Information

Observed information is a concept that measures the curvature of the log likelihood function at the MLE. It helps us understand the precision of our estimates.For the Laplace distribution, the observed information is related to the second derivative of the log likelihood function. The calculation for the observed information involves differentiating the score statistic:

• For \(\mu\), the second derivative and therefore the observed information, remains zero except at the data points, where jumps occur. This reflects the non-smoothness of the log likelihood function with respect to \(\mu\).
• For \(\sigma\), the observable information is:\[\frac{\partial^2}{\partial \sigma^2} \log L = \frac{2n}{\sigma^2} - \frac{2}{\sigma^3} \sum_{i=1}^{n} |y_i - \mu|.\]

This information is helpful as it allows us to assess how well the parameters are estimated. The function provides insights into whether small differences in parameter estimates greatly change the likelihood.The irregularity noted for the MLE of the Laplace distribution is linked with this characteristic. The non-singularity of the Fisher information matrix is crucial for regular estimates, which, in the case of the Laplace distribution, is not always achievable due to the non-smooth nature of the log likelihood function.