Question

Let \(X_{1}, X_{2}, \ldots, X_{n}\) be a random sample from a distribution with pmf \(p(x ; \theta)=\theta^{x}(1-\theta), x=0,1,2, \ldots\), zero elsewhere, where \(0 \leq \theta \leq 1\). (a) Find the mle, \(\hat{\theta}\), of \(\theta\). (b) Show that \(\sum_{1}^{n} X_{i}\) is a complete sufficient statistic for \(\theta\). (c) Determine the MVUE of \(\theta\).

Short answer

The MLE, \(\hat{\theta}\), can be found by taking the log of the pmf, differentiating it with respect to \(\theta\), and equating it to zero. A statistic is complete and sufficient if it satisfies the conditions of the Factorization Theorem, which states that the joint pmf can be expressed as \(g(T(x), \theta)h(x)\), where \(g\) does not contain \(x\) except through \(T\) and \(h\) does not contain \(\theta\). To find the MVUE, an unbiased estimator of the parameter is determined. If this estimator achieves equality in the Cramér-Rao lower bound, which is the minimum possible variance of any unbiased estimator, then it is the MVUE.

Step-by-step solution

Finding the Maximum Likelihood Estimate (MLE)

To find the MLE, take the log of the probability mass function (pmf) first and differentiate it with respect to \(\theta\). Equate it to zero and solve for \(\theta\). This should give you the MLE \(\hat{\theta}\). This is done because the log function is a monotone transformation, and it simplifies the derivative.

Show that \(\sum_{1}^{n} X_{i}\) is a complete sufficient statistic

To show that a statistic is a sufficient statistic, we use the Factorization Theorem. According to the theorem: A statistic \(T(X)\) is a sufficient statistic for \(\theta\) if and only if the joint pmf/pfd of \(X\) can be expressed as \(g(T(x), \theta)h(x)\), where \(g\) does not contain \(x\) except through \(T\) and \(h\) does not contain \(\theta\). If we can write the joint pmf in this form, then the statistic is sufficient.

Determining the Minimum Variance Unbiased Estimator (MVUE)

To find the MVUE, we first find an unbiased estimator of the parameter. Once we find this, we determine whether it's the MVUE. Anyone estimator is MVUE if it achieves equality in the Cramér-Rao lower bound (CRLB). CRLB is the minimum possible variance of any unbiased estimator. If our unbiased estimator has the same variance as the CRLB, then it is the MVUE.

Key concepts

Probability Mass Function

A probability mass function (pmf) is a function that gives the probability that a discrete random variable is exactly equal to some value. In the exercise, the pmf is given by \(p(x ; \theta)=\theta^{x}(1-\theta)\), which describes the probability of observing \(x\) in a single trial of a distribution parameterized by \(\theta\). For the given pmf, \(x\) represents the number of successes, and \(\theta\) the probability of success. It's crucial to note that pmfs must satisfy two conditions: first, the sum of all probabilities must equal one, and second, every probability must be non-negative.

Understanding the pmf can provide insight into the random variable's behavior, which is essential for solving advanced statistical problems such as finding the maximum likelihood estimate (MLE) of \(\theta\).

Complete Sufficient Statistic

A statistic is deemed sufficient for a parameter if it captures all the information needed to estimate that parameter. In other words, once you have the sufficient statistic, you don't need any more data from your sample to estimate your parameter effectively. A statistic is 'complete' if it does not allow for any unbiased estimators of zero, except the zero function itself. This means that there are no other statistics that can provide additional information about the parameter.

In the exercise, students are asked to show that \(\sum_{1}^{n} X_{i}\) is a complete sufficient statistic for \(\theta\). This means that this summation captures all the information about \(\theta\) contained in the data, and there are no other functions of the observed data that provide any additional information on \(\theta\) that isn't already obtained from the sum. The implication is profound because it simplifies the process of estimation significantly.

Minimum Variance Unbiased Estimator (MVUE)

The minimum variance unbiased estimator (MVUE) is the best possible estimator for a parameter because it has the smallest variance among all unbiased estimators. Being 'unbiased' means that, on average, the estimator hits the true parameter value; it doesn't systematically overestimate or underestimate it. The 'minimum variance' aspect is critical because it ensures the estimates are as close as possible to the true parameter value across different samples.

In the context of the exercise, the goal is to determine the MVUE for \(\theta\). This involves not only finding an unbiased estimator but also one that has the smallest variance possible. Once an estimator meets these criteria, it can be considered the most reliable and efficient estimator given the data.

Cramér-Rao Lower Bound (CRLB)

The Cramér-Rao Lower Bound (CRLB) is a fundamental result in estimation theory that quantifies the lowest variance that an unbiased estimator can possibly have. The CRLB hinges on the concept of Fisher information, which measures the amount of information that an observable random variable carries about an unknown parameter. The greater the Fisher information, the lower the bound on the variance of unbiased estimators – implying more precise estimates are possible.

In practice, if an unbiased estimator reaches the CRLB, it is the MVUE as no other unbiased estimator can have a lower variance. When tackling statistical problems, the Cramér-Rao Bound is instrumental in judging the quality of an estimator. By comparing the variance of a given unbiased estimator to the CRLB, one can determine if the estimator is the best possible (i.e., MVUE) or if a search for a better estimator is warranted.

Factorization Theorem

The Factorization Theorem provides a practical criterion for identifying sufficient statistics. According to this theorem, a statistic \(T(X)\) is sufficient for a parameter \(\theta\) if the joint probability mass function (pmf) or probability density function (pdf) of the sample can be expressed as a product of two functions: \(g(T(x), \theta)\) and \(h(x)\), where \(g\) only depends on the data through the statistic \(T\) and \(\theta\), and \(h\) does not depend on \(\theta\) at all.

In the provided exercise, applying the Factorization Theorem shows that the sum of the sample \(\sum_{1}^{n} X_{i}\) is indeed a sufficient statistic for \(\theta\). This is a key step in simplifying the estimation process. Understanding how to apply the Factorization Theorem is crucial for students to solve similar problems and understand the underlying principles of statistical estimation.