Question

An experiment consists of a large, but unknown, number \(n(\gg 1)\) of trials in each of which the probability of success \(p\) is the same, but also unkown. In the \(i\) th trial, \(i=1,2, \ldots, N\), the total number of successes is \(x_{i}(\gg 1)\). Determine the log-likelihood function. Using Stirling's approximation to \(\ln (n-x)\), show that $$ \frac{d \ln (n-x)}{d n} \approx \frac{1}{2(n-x)}+\ln (n-x) $$ and hence evaluate \(\partial\left({ }^{n} C_{x}\right) / \partial n\) By finding the (coupled) equations determining the ML estimators \(\hat{p}\) and \(\hat{n}\), show that, to order \(N^{-1}\), they must satisfy the simultaneous 'arithmetic' and 'geometric' mean constraints $$ \hat{n} \hat{p}=\frac{1}{N} \sum_{i=1}^{N} x_{i} \quad \text { and } \quad(1-\hat{p})^{N}=\prod_{i=1}^{N}\left(1-\frac{x_{i}}{\hat{n}}\right). $$

Short answer

The ML estimators \(\hat{n}\) and \(\hat{p}\) satisfy \(\hat{n} \hat{p} = \frac{1}{N} \sum_{i=1}^N x_i\) and \((1-\hat{p})^N = \prod_{i=1}^{N}\left(1-\frac{x_i}{\hat{n}}\right)\).

Step-by-step solution

Log-likelihood function

The probability of obtaining exactly \(x_i\) successes in any given trial follows the binomial distribution given by \({n \brace x} p^{x} (1-p)^{n-x}\). The full log-likelihood function is therefore the sum over all \(N\) trials: \[ \ln L(n, p) = \sum_{i=1}^{N} \left[ x_i \ln(p) + (n-x_i) \ln(1-p) + \ln \left(\frac{n!}{x_i!(n-x_i!)}\right) \right] \].

Applying Stirling's approximation

Stirling's approximation states that \(\ln(k!) \approx k\ln(k) - k\). This gives \[ \ln \left(\frac{n!}{x_i!(n-x)!}\right) \approx n \ln(n) - n - x_i \ln(x_i) + x_i - (n-x_i)\ln(n-x_i) + n - x_i \].

Simplifying log-likelihood function

Simplifying using Stirling’s approximation, we get \[ \ln L(n, p) = \sum_{i=1}^{N} \left[ x_i \ln(p) + (n-x_i) \ln(1-p) + n \ln(n) - x_i \ln(x_i) - (n-x_i) \ln(n-x_i) \right]. \]

Derivative of log-likelihood

First, find the derivative of the log-likelihood with respect to n. Applying Stirling's approximation, we approximate \(\frac{d \ln(n-x)}{d n} \approx \frac{1}{2(n-x)} + \ln(n-x)\). Partial derivative is then: \[ \frac{\partial}{\partial n} \ln L(n, p) \approx \sum_{i=1}^{N} \left[ -\ln(1-p) - x_i \left( \frac{1}{2(n-x_i)} + \ln(n-x_i)\right) \right]. \]

Evaluate \(\partial\left(\binom{n}{x}\right) / \partial n\)

We have the term \(\binom{n}{x}\) involved in log-likelihood. The derivative with respect to \(n\) can now be evaluated considering the approximation: \[ \frac{\partial}{\partial n}\binom{n}{x} \approx \sum_{i=1}^{N} \left[ \binom{x_i}{n} \left( \frac{1}{2(n-x_i)} + \ln(n-x_i)\right) \right] \]

Setup coupled equations

To find ML estimators \(\hat{p}\) and \(\hat{n}\), solve the equations: \[ \hat{n} \hat{p}=\frac{1}{N} \sum_{i=1}^{N} x_{i} \text{ and } (1-\hat{p})^{N} = \prod_{i=1}^{N}\left(1-\frac{x_{i}}{\hat{n}}\right). \]

Validate equations to order \(N^{-1}\)

Lastly, ensure the equations meet the simultaneous 'arithmetic' (\(\hat{n} \hat{p}\)) and 'geometric' ((1-\hat{p})^{N}) mean constraints to order \(N^{-1}\). Hence, verify: \[ \hat{n} \hat{p}=\frac{1}{N} \sum_{i=1}^{N} x_{i} \text{ and } (1-\hat{p})^{N} = \prod_{i=1}^{N}\left(1-\frac{x_{i}}{\hat{n}}\right).\]

Key concepts

Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE) is a method used for estimating the parameters of a statistical model. In the context of the binomial distribution, we are trying to find the values of the parameters that make the observed data most probable. Here, the parameters are the number of trials \(n\) and the probability of success \(p\).
To perform MLE, we start by defining a likelihood function, which expresses the probability of observing the given set of data as a function of the parameters to be estimated. We then find the parameter values that maximize this likelihood function. This is often done by taking the natural logarithm of the likelihood function (resulting in the log-likelihood function) for easier differentiation. We solve the partial derivatives of the log-likelihood function with respect to each parameter and set them to zero. This gives us the estimators for the parameters.

The log-likelihood function for the binomial distribution is particularly useful because its additive properties make it easier to differentiate.

Stirling's Approximation

Stirling's Approximation is a mathematical formula used to approximate the factorial of a large number. It is especially handy in probability and statistics, particularly when dealing with the log-likelihood function of the binomial distribution. According to Stirling's Approximation, \(\ln(k!) \approx k\ln(k) - k\).
This simplifies complex factorial expressions that appear in the binomial coefficients. For example, the term \(\frac{n!}{x!(n-x)!}\) in the binomial distribution's probability mass function can be cumbersome for large values of \(n\) and \(x\). By applying Stirling's Approximation, this term becomes manageable:
\[ \ln(\frac{n!}{x!(n-x)!}) \approx n \ln(n) - n - x \ln(x) + x - (n-x) \ln(n-x) + (n-x). \]
This approximation eases the differentiation process in the next step of finding the MLE.

Binomial Distribution

The binomial distribution is a discrete probability distribution that models the number of successes in a fixed number of independent and identically distributed Bernoulli trials. Each trial results in a success with probability \(p\) and a failure with probability \(1-p\). The binomial distribution is denoted by \(B(n, p)\), where \(n\) is the number of trials, and \(p\) is the probability of success.
The probability mass function for the binomial distribution is:
\[ P(X = x) = {n \choose x} p^x (1-p)^{n-x}, \]
where \({n \choose x}\) is the binomial coefficient, representing the number of ways to choose \(x\) successes out of \(n\) trials.
In MLE, our goal is to find the values of \(n\) and \(p\) that make our observed data most probable, by maximizing the likelihood of observing a specific sequence of successes and failures.

Log-likelihood Function

The log-likelihood function is the natural logarithm of the likelihood function. It is used in maximum likelihood estimation for easier differentiation. When dealing with the binomial distribution, the log-likelihood function is:
\[ \ln L(n, p) = \sum_{i=1}^{N} \left[ x_i \ln(p) + (n - x_i) \ln(1-p) + \ln \left(\frac{n!}{x_i!(n-x_i)!}\right) \right]. \]
Applying Stirling's Approximation, the log-likelihood function simplifies to:
\[ \ln L(n, p) = \sum_{i=1}^{N} \left[ x_i \ln(p) + (n-x_i) \ln(1-p) + n\ln(n) - x_i\ln(x_i) - (n-x_i)\ln(n-x_i) \right]. \]
The log-likelihood function helps us find the MLE by simplifying the process of taking derivatives and solving for the parameters.

Partial Derivatives

Partial derivatives are used in MLE to find the points where the log-likelihood function reaches its maximum, which correspond to the estimates of the parameters. For our problem, we compute the partial derivatives of the log-likelihood function \(\ln L(n, p)\) with respect to \(n\) and \(p\):
- The partial derivative with respect to \(p\) is:
\[ \frac{\partial \ln L(n, p)}{\partial p} = \sum_{i=1}^{N} \left( \frac{x_i}{p} - \frac{n - x_i}{1 - p} \right). \]
- The partial derivative with respect to \(n\) involves approximations and can be expressed as:
\[ \frac{\partial \ln L(n, p)}{\partial n} \approx \sum_{i=1}^{N} \left[ -\ln(1-p) - x_i \left( \frac{1}{2(n - x_i)} + \ln(n - x_i) \right) \right]. \]
We set these partial derivatives to zero and solve the resulting system of equations to find the MLE for \(n\) and \(p\).