Question

Consider the Bayes model \(X_{i} \mid \theta, i=1,2, \ldots n \sim\) iid with distribution \(b(1, \theta), 0<\theta<1\) (a) Obtain Jeffrys prior for this model. (b) Assume squared error loss and obtain the Bayes estimate of \(\theta\).

Short answer

For the Jeffreys prior, we obtained a Beta distribution with parameters 0.5 and 0.5, and for squared error loss, the Bayes estimator of \(\theta\) is \(\frac{\Sigma{x_{i}}+0.5}{n+1}\).

Step-by-step solution

Find the likelihood function and its derivatives

The likelihood function for a Bernoulli distribution is \(\prod_{i=1}^{n} \theta^{x_{i}}(1-\theta)^{1-x_{i}}\). The first derivative of the log-likelihood is \(\frac{\partial \log L(\theta)}{\partial \theta} = \frac{\Sigma{x_{i}}}{\theta} - \frac{n - \Sigma{x_{i}}}{1 - \theta}\) and the second derivative is \(\frac{\partial^{2} \log L(\theta)}{\partial \theta^{2}} = -\frac{\Sigma{x_{i}}}{\theta^{2}} - \frac{n - \Sigma{x_{i}}}{(1 - \theta)^{2}}\).

Compute Jeffreys Prior

The Jeffreys prior is proportional to the square root of the Fisher information, which in this case is \(-E[\frac{\partial^{2}}{\partial \theta^{2}}logL(\theta)]\). Using the derivative calculated in the first step, the Fisher information simplifies to \(\frac{n}{\theta(1-\theta)}\). The Jeffreys prior then is \(\sqrt{E[\frac{n}{\theta(1-\theta)}]} = \frac{\sqrt{n}}{\theta(1-\theta)}\) (unnormalized). This is a Beta distribution function where both the shape parameters are 0.5.

Calculate Bayes Estimator with squared error loss

We know that for squared error loss and a Beta prior, the Bayes estimator of \(\theta\) is given by \(\frac{\Sigma{x_{i}}+\alpha}{n+\alpha+\beta}\), where \(\alpha\) and \(\beta\) are shape parameters from the prior Beta distribution. Plugging in \(\alpha=\beta=0.5\), we get \(\frac{\Sigma{x_{i}}+0.5}{n+1}\).

Key concepts

Jeffreys Prior

The Jeffreys prior is a non-informative prior used in Bayesian statistics that has a unique role in ensuring invariant parameter estimation under reparameterization, meaning it is not affected by transformations of the parameters. This prior is based on the concept of Fisher information, which we'll discuss later.

When constructing a Jeffreys prior, we start by considering the Fisher information, which itself is rooted in the likelihood function of the data given the parameter. For a given model, the Jeffreys prior is proportional to the square root of the Fisher information. Thus, for the Bernoulli distribution in our exercise, with a likelihood function for a sample of size n, the Jeffreys prior is given by \( \sqrt{E\left[\frac{n}{\theta(1-\theta)}\right]} \), which is unnormalized. Significantly, in this particular problem, it takes the form of a Beta distribution with both shape parameters equal to 0.5, often referred to as a \'reference prior\'.

This choice of prior reflects a state of ignorance about the parameter and assigns equal weight to all possible values, which helps in scenarios where we do not want any subjective prior information to influence our inferences.

Squared Error Loss

In Bayesian statistics, the concept of \'squared error loss\' represents the cost associated with an incorrect estimate of a parameter. It quantifies the accuracy of an estimate by squaring the difference between the estimated value and the actual value of the parameter.

The formula for squared error loss is given by \( (\hat{\theta} - \theta)^2 \), where \( \hat{\theta} \) is the estimated value and \( \theta \) is the true parameter value. The squared nature of this loss function means that larger errors are penalized more severely than smaller ones, emphasizing the preference for more accurate estimations. In the context of our problem, when the squared error loss is applied, the Bayes estimate for \( \theta \) is the value that minimizes the expected squared error.

Fisher Information

Fisher information is a key concept in both frequentist and Bayesian statistics, providing a measure of the amount of information that an observable random variable carries about an unknown parameter upon which the probability of the random variable depends.

In mathematical terms, the Fisher information is the variance of the score, or the expected value of the squared gradient of the log-likelihood with respect to the parameter. It can be expressed as \( -E\left[\frac{\partial^2}{\partial \theta^2} \log L(\theta)\right] \), where \( L(\theta) \) is the likelihood function of the data. A key property of Fisher information is that, under certain regularity conditions, the inverse of the Fisher information matrix provides a lower bound on the variance of any unbiased estimator (known as the Cramér-Rao bound). The higher the Fisher information, the smaller the lower bound on the variance, indicating more precise estimation.

Bernoulli Distribution

The Bernoulli distribution is one of the simplest discrete probability distributions, governing experiments that result in a binary outcome — typically success (1) or failure (0). Mathematically, a random variable \( X \) follows a Bernoulli distribution with parameter \( \theta \) if it takes value 1 with probability \( \theta \) and value 0 with probability \( 1 - \theta \).

The likelihood function of a set of independent and identically distributed (iid) Bernoulli trials is the product of individual probabilities, yielding \( \prod_{i=1}^{n} \theta^{x_i}(1-\theta)^{1-x_i} \), where \( x_i \) are the observed values. This distribution forms the foundation of the model in our exercise and plays a central role in the calculation of both the Fisher information and the Bayes estimator.