Question

Two independent samples \(Y_{1}, \ldots, Y_{n} \stackrel{\text { iid }}{\sim} N\left(\mu, \sigma^{2}\right)\) and \(X_{1}, \ldots, X_{m} \stackrel{\text { iid }}{\sim} N\left(\mu, c \sigma^{2}\right)\) are available, where \(c>0\) is known. Find posterior densities for \(\mu\) and \(\sigma\) based on prior \(\pi(\mu, \sigma) \propto 1 / \sigma\).

Short answer

The posterior for \( \mu \) is normal, and for \( \sigma^2 \), it is inverse gamma.

Step-by-step solution

Define the Likelihoods

The likelihood for the first sample \( Y_1, \ldots, Y_n \) is a normal distribution \( N(\mu, \sigma^2) \) for each \( Y_i \). Therefore, the joint likelihood for all \( Y_i \) is: \[ L_Y(\mu, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{(Y_i - \mu)^2}{2\sigma^2}\right) \].

Compute the Likelihood for Second Sample

Similarly, for the second independent sample \( X_1, \ldots, X_m \) with \( N(\mu, c\sigma^2) \), the joint likelihood is: \[ L_X(\mu, \sigma^2) = \prod_{j=1}^{m} \frac{1}{\sqrt{2\pi c \sigma^2}} \exp\left(-\frac{(X_j - \mu)^2}{2c\sigma^2}\right) \].

Combine Likelihood Functions

The combined likelihood for both samples is the product of the two individual likelihoods: \[ L(\mu, \sigma^2) = L_Y(\mu, \sigma^2) \cdot L_X(\mu, \sigma^2) \]. This simplifies to a function of \( \mu \) and \( \sigma^2 \) involving sums of squared deviations around \( \mu \) weighed according to their variances.

Apply the Prior

Given the prior \( \pi(\mu, \sigma) \propto \frac{1}{\sigma} \), the posterior density up to proportionality is: \[ \pi(\mu, \sigma | Y, X) \propto \frac{1}{\sigma} L(\mu, \sigma^2) \].

Determine Posterior for \(\mu\)

Integrate out \( \sigma \) from the posterior density to determine the marginal posterior for \( \mu \). This involves calculus and simplifying the expression, typically resulting in a normal distribution for \( \mu \).

Determine Posterior for \(\sigma\)

To find the marginal posterior for \( \sigma \), integrate out \( \mu \) from the joint posterior. This will typically result in an inverse gamma distribution for \( \sigma^2 \) due to the conjugate nature of the priors.

Key concepts

Posterior Distribution

In Bayesian inference, the posterior distribution is the probability distribution that represents what we know about an unknown parameter after considering the evidence or data at hand. This is the core of Bayesian analysis, as it allows us to update our beliefs about the parameters based on the observed data.

Think of the posterior distribution as combining prior beliefs with new evidence. Specifically, the posterior is formed by multiplying the prior distribution and the likelihood function. In mathematical terms, the posterior distribution for parameters like \(\mu\) and \(\sigma\) can be expressed as:

• \( \pi(\mu, \sigma | Y, X) \propto \text{Prior}(\mu, \sigma) \times \text{Likelihood}(Y, X | \mu, \sigma) \)

Once we have the posterior distribution, we can make more informed decisions and predictions about \(\mu\) and \(\sigma\). Often, this will result in a refined belief, typically expressed with familiar distributions, as shown in Steps 5 and 6 of the solution, where \(\mu\) follows a normal distribution and \(\sigma^2\) is characterized by an inverse gamma distribution.

Likelihood Function

The likelihood function is a fundamental component in statistical modeling and forms the backbone of Bayesian inference. It describes how probable the observed data is, given specific parameter values.

In our exercise, the likelihood function for two independent samples is derived based on the normal distribution assumptions. For each sample group, the likelihood is:

• For sample \(Y_1, \ldots, Y_n\): \[ L_Y(\mu, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{(Y_i - \mu)^2}{2\sigma^2}\right) \]
• For sample \(X_1, \ldots, X_m\): \[ L_X(\mu, \sigma^2) = \prod_{j=1}^{m} \frac{1}{\sqrt{2\pi c \sigma^2}} \exp\left(-\frac{(X_j - \mu)^2}{2c\sigma^2}\right) \]

The combined likelihood \(L(\mu, \sigma^2)\) is found by multiplying these functions. This accounts for the probability of observing the data across both samples, which is crucial for forming the posterior distribution.

Prior Distribution

Prior distribution represents the initial beliefs about the parameters before any data is observed. In Bayesian inference, it's essential because it influences the posterior distribution.

For this exercise, the given prior is:

• \( \pi(\mu, \sigma) \propto \frac{1}{\sigma} \)

This is a non-informative prior, meaning it doesn't favor any particular values of \(\mu\) or \(\sigma\), except to enforce that \(\sigma\) must not be zero. Non-informative priors like this are useful when we don't have strong prior knowledge about the parameter values and want the data to have a more significant influence on the posterior distribution.

By using this prior in conjunction with the likelihood function, we derive the posterior distribution, which enables us to update our beliefs about \(\mu\) and \(\sigma\) after observing the data.