Question

One model for outliers in a normal sample is the mixture $$ f(y ; \mu, \pi)=(1-\pi) \phi(y-\mu)+\pi g(y-\mu), \quad 0 \leq \pi \leq 1, \infty<\mu<\infty $$ where \(g(z)\) has heavier tails than the standard normal density \(\phi(z)\); take \(g(z)=\frac{1}{2} e^{-|z|}\) for example. Typically \(\pi\) will be small or zero. Show that when \(\pi=0\) the likelihood derivative for \(\pi\) has zero mean but infinite variance, and discuss the implications for the likelihood ratio statistic comparing normal and mixture models.

Short answer

When \(\pi=0\), the derivative's mean is zero, but its variance is infinite, complicating the use of likelihood ratio tests.

Step-by-step solution

Understand the model and likelihood

The given model is a mixture model where the main component is a normal distribution represented by \(\phi(y - \mu)\), and the outlier component is \(g(y-\mu)\), which in this case has heavier tails than the normal distribution, specified as \(g(z)=\frac{1}{2} e^{-|z|}\). The mixing parameter \(\pi\) determines the weight of the outlier component in the overall distribution, where \(0 \leq \pi \leq 1\). The model simplifies to a normal distribution when \(\pi=0\).

Derive the likelihood function

The likelihood function of a sample \(y_1, y_2, \ldots, y_n\) under this mixture model is \( L(\mu, \pi) = \prod_{i=1}^{n} \left( (1-\pi) \phi(y_i - \mu) + \pi g(y_i - \mu) \right) \). When \(\pi=0\), this reduces to the product of normal densities: \( L(\mu, \pi=0) = \prod_{i=1}^{n} \phi(y_i - \mu) \).

Compute the derivative of the likelihood with respect to \(\pi\)

The derivative of the likelihood function with respect to \(\pi\), evaluated at \(\pi=0\), is given by: \[ \left. \frac{\partial L(\mu, \pi)}{\partial \pi} \right\vert_{\pi=0} = \sum_{i=1}^{n} \left( g(y_i-\mu) - \phi(y_i-\mu) \right) \]. This derivative shows how the likelihood changes with small changes in \(\pi\) at \(\pi=0\).

Check the mean of the derivative

The expected value of the derivative in Step 3, when \(\pi=0\), is \[ E\left( \sum_{i=1}^{n} \left(g(y_i-\mu) - \phi(y_i-\mu)\right) \right) = 0 \] over the normal distribution \(\phi(y_i - \mu)\), because \(g(y - \mu)\) and \(\phi(y - \mu)\) are balanced by their construction, maintaining the property of zero mean. This follows from the properties of the normal distribution when \(\pi=0\).

Examine variance of the derivative

Calculate the variance of \(\sum_{i=1}^{n} \left( g(y_i-\mu) - \phi(y_i-\mu) \right) \). Because \(g(z)\) has heavier tails than the normal distribution \(\phi(z)\), \(Var(g(y_i - \mu) - \phi(y_i - \mu)) = \infty\). This property arises from the infinite expectation of the absolute deviation of heavy-tailed distributions.

Discuss implications for the likelihood ratio statistic

Because variance is infinite when \(\pi=0\), the likelihood ratio statistic, which is based on comparing log-likelihoods in different models, might not converge according to typical assumptions. This affects hypothesis testing using the likelihood ratio, making it unreliable when differentiating between normal and mixture models due to instability from outliers.

Key concepts

Outliers in Statistics

In statistics, outliers are data points that differ significantly from other observations in the dataset. They can represent variability in the measurement or may indicate experimental errors. In mixture models designed to handle outliers, such as the one presented in the exercise, the data is thought to arise from a combination of a standard model and an additional model that accounts for these unusual observations.

Outliers can heavily influence the results of statistical analyses, leading to biased estimates and non-representative conclusions. Recognizing this, statisticians often employ mixture models to accommodate data distributions that deviate from the norm, allowing for a more flexible model that considers these outliers appropriately.

Likelihood Function

The likelihood function is vital in statistical inference, as it connects parameters of statistical models to observed data. In the context of mixture models, it often includes components for both the main distribution and the outlier distribution. For the exercise at hand, the mixture model has its likelihood function set as a product of contributions from a normal distribution and a heavy-tailed distribution for outliers.

The likelihood function helps us understand how probable our data is, given different values of the mixture parameter \( \pi \). Adjusting this parameter changes the weight between standard data and outliers, providing insight into how much data variance is possibly due to these rare or extreme points. This flexibility is critical when deciding whether or not to incorporate an outlier adjustment in the model.

Heavy-Tailed Distributions

Heavy-tailed distributions, such as the one introduced for handling outliers in the original exercise, differ from normal distributions because they have 'tails' that do not decline as quickly. This means that they assign greater probability to extreme outcomes. For instance, the chosen outlier component \( g(z) = \frac{1}{2} e^{-|z|} \) has heavier tails compared to a normal distribution.

Inferences drawn from heavy-tailed distributions are particularly important when dealing with anomalous data as they can represent extreme variability or measurement error. Such distributions can thus better accommodate outliers, as their properties allow for infinite variance, capturing the presence of extreme values within the data without severely impacting the model fit for the main distribution component.

Likelihood Ratio Test

The likelihood ratio test (LRT) is a statistical test employed to compare the goodness-of-fit between two competing models. It is particularly useful in the mixture model context to decide whether a simple model (e.g., normal) or a more complex model (e.g., mixture) better fits the data.

In our exercise, the LRT examines the log-likelihoods of both models, shedding light on whether including an outlier component provides a significant improvement. Due to the infinite variance in the derivative of the likelihood function when \( \pi = 0 \), typical assumptions of the LRT may fail, causing test results to be unstable or unreliable. This highlights the importance of evaluating the LRT cautiously when outliers are present, as incorrect assumptions can skew hypothesis testing outcomes.