Question

Let \(X_{1}, X_{2}, \ldots, X_{n}\) be a random sample from a \(N\left(\theta, \sigma^{2}\right)\) distribution, where \(\sigma^{2}\) is fixed but \(-\infty<\theta<\infty\) (a) Show that the mle of \(\theta\) is \(\bar{X}\). (b) If \(\theta\) is restricted by \(0 \leq \theta<\infty\), show that the mle of \(\theta\) is \(\widehat{\theta}=\max \\{0, \bar{X}\\}\).

Short answer

For unrestricted \(\theta\), the MLE is \(\bar{X}\). When \(\theta\) is restricted to non-negative values, the MLE is \(\max\{0, \bar{X}\}\).

Step-by-step solution

Write the Likelihood Function

The likelihood function for a set of observations \(X_{1}, X_{2}, \ldots, X_{n}\) from a normal distribution \(N(\theta, \sigma^2)\) is given by: \[ L(\theta) = \prod _{i=1} ^n \frac{1}{\sqrt{2\pi}\sigma} e^{-(X_{i}-\theta)^2 / (2\sigma^2)} \]

Logarithm of the Likelihood Function

Taking the natural logarithm of the likelihood function simplifies the task of finding the maximum of the function. The log-likelihood function is given by: \[ l(\theta) = \ln(L(\theta)) = -\frac{n}{2} \ln(2\pi\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^{n} (X_i - \theta)^2 \]

Derive the MLE for Unrestricted θ

Take the derivative of \(l(\theta)\) with respect to \(\theta\) and set equal to zero, to locate the maximum of the function. Solve for θ: \[ \frac{d}{d\theta} l(\theta) = \frac{1}{\sigma^2} \sum_{i=1}^{n} (X_i - \theta) = 0 \] Solving for \(\theta\) gives the MLE for \(\theta\) as \(\bar{X}\), the sample mean.

Derive the MLE for Restricted θ

When \(\theta\) is restricted to non-negative values, the MLE is found by maximizing the likelihood over the interval [0, ∞). Therefore, the MLE will be the maximum of 0 and \(\bar{X}\) - i.e., if \(\bar{X} < 0\), then the MLE of \(\theta\) will be 0, otherwise it is \(\bar{X}\). Therefore, \(\widehat{\theta} = \max\{0, \bar{X} \}\).

Key concepts

Likelihood Function

The likelihood function is a fundamental concept in statistical inference, especially in the context of maximum likelihood estimation (MLE). It represents the probability of observing the sample data given specific values of the parameters of the model. In a more technical sense, for a set of independent and identically distributed observations, the likelihood function is the product of the probability density functions (PDFs) for each observed value.

When dealing with continuous distributions such as the normal distribution, the likelihood function for observing a particular sample \(X_1, X_2, ..., X_n\) given a parameter \(\theta\) is expressed as \(L(\theta) = \prod_{i=1}^{n} f(X_i|\theta)\), where \(f(x|\theta)\) is the PDF of the distribution. To compute the MLE, one needs to find the parameter value \(\theta\) that maximizes this likelihood function. The act of maximizing the likelihood function indirectly maximizes the chances of observing our sample, which makes the estimated parameter more plausible.

Normal Distribution

The normal distribution, also known as the Gaussian distribution, is a continuous probability distribution that is symmetric about the mean, showing that data near the mean are more frequent in occurrence than data far from the mean. It is characterized by its mean \(\mu\) and variance \(\sigma^2\), and its PDF is defined as \(f(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-(x-\mu)^2 / (2\sigma^2)}\).

The normal distribution is widely used in statistics due to the Central Limit Theorem, which states that the sum of a large number of independent and identically distributed random variables will be approximately normally distributed, regardless of the original distribution of the variables. This property is particularly useful when dealing with sample means and inference about population parameters.

Sample Mean

The sample mean, denoted as \(\bar{X}\), is the arithmetic average of a set of sample values. It is computed by summing up all the observed values in a sample and then dividing by the number of observations \(n\): \(\bar{X} = \frac{1}{n} \sum_{i=1}^{n} X_i\).

The sample mean is of paramount importance when it comes to estimating parameters of a population, especially the population mean. In the context of MLE, when the distribution under consideration is the normal distribution, the sample mean becomes the MLE of the population mean \(\mu\) under certain conditions, particularly when the variance \(\sigma^2\) is known and the mean \(\mu\) is the parameter to be estimated. The sample mean serves as a central concept in both descriptive and inferential statistics due to its property of being the best unbiased estimate of the population mean when the samples are randomly drawn.

Log-Likelihood Function

The log-likelihood function is simply the natural logarithm transformation of the likelihood function and is widely used for ease of computation and better numerical stability. For the normal distribution, the logarithm transformation turns the product of exponentials into a sum, which makes differentiation with respect to the parameter \(\theta\) simpler and more manageable.

The general form of the log-likelihood function for a sample from a normal distribution is given by \(l(\theta) = \ln(L(\theta))\), which, after applying the logarithm, becomes a sum of the logged PDFs. When deriving the MLE, we take the derivative of this function with respect to \(\theta\) and set it equal to zero to find the maximum. This process is known as maximization of the log-likelihood, and it often yields the same estimates as the original likelihood function, with the added benefits of simpler algebraic manipulation and a reduced risk of computational underflow or overflow.