Question

Find the optimal estimating function based on dependent data \(Y_{1}, \ldots, Y_{n}\) with \(g_{j}(Y ; \theta)=\) \(Y_{j}-\theta Y_{j-1}\) and \(\operatorname{var}\left\\{g_{j}(Y ; \theta) \mid Y_{1}, \ldots, Y_{j-1}\right\\}=\sigma^{2} .\) Derive also the estimator \(\tilde{\theta}\). Find the maximum likelihood estimator of \(\theta\) when the conditional density of \(Y_{j}\) given the past is \(N\left(\theta y_{j-1}, \sigma^{2}\right) .\) Discuss.

Short answer

The optimal estimator \( \tilde{\theta} \) is \( \frac{\sum_{j=2}^{n} Y_{j}Y_{j-1}}{\sum_{j=2}^{n} Y_{j-1}^2} \), maximizing the likelihood under normality.

Step-by-step solution

Understanding the Estimating Function

The function provided is a measure of the difference between the observed data point, \( Y_j \), and its prediction based on the previous data point, \( \theta Y_{j-1} \). Our task is to estimate \( \theta \) such that this difference is minimized or makes the data most probable.

Expression of the Conditional Variance

The conditional variance \( \operatorname{var}\{g_j(Y ; \theta) | Y_1, \ldots, Y_{j-1}\} \) is given as \( \sigma^2 \). This implies that the difference \( g_j(Y ; \theta) = Y_j - \theta Y_{j-1} \) is homoscedastic, or constant variance, which will be useful for simplification during estimation.

Form of the Maximum Likelihood Function

The density function of \( Y_j \) given the past data \( Y_{j-1} \) is distributed as \( N(\theta y_{j-1}, \sigma^2) \). Thus, the likelihood function for all data is the product of these densities:\[ L(\theta; Y) = \prod_{j=2}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left(-\frac{(Y_j - \theta Y_{j-1})^2}{2\sigma^2}\right) \]

Maximizing the Log-Likelihood

To find the estimator \( \tilde{\theta} \), we take the logarithm of the likelihood function to simplify:\[ \log L(\theta; Y) = -\frac{n-1}{2} \log(2\pi \sigma^2) - \frac{1}{2\sigma^2} \sum_{j=2}^{n} (Y_j - \theta Y_{j-1})^2 \]

Deriving the Optimal \( \theta \)

Taking the derivative of the log-likelihood with respect to \( \theta \) and setting it to zero gives:\[ \frac{\partial}{\partial \theta} \log L(\theta; Y) = \frac{1}{\sigma^2} \sum_{j=2}^{n} Y_{j-1} (Y_j - \theta Y_{j-1}) = 0 \]Solving for \( \theta \), we find:\[ \tilde{\theta} = \frac{\sum_{j=2}^{n} Y_{j}Y_{j-1}}{\sum_{j=2}^{n} Y_{j-1}^2} \]

Discussion on the Estimator

The estimator \( \tilde{\theta} \) is the sample approximation of the slope of regression of \( Y_j \) on \( Y_{j-1} \). It leverages past data to best fit the expected relation assumed in the model. This makes it a reasonable estimate under normality of residuals and fixed variance assumptions.

Key concepts

Estimating Functions

Estimating functions are crucial in statistical modeling when dealing with dependent data. They differ from typical objective functions because they focus on finding an estimator that aligns closely with the data pattern. In the outlined exercise, the estimating function is expressed as \( g_j(Y ; \theta) = Y_j - \theta Y_{j-1} \). This function measures how well our model's prediction aligns with the observed data.
To optimize our estimation of \(\theta\), we aim to minimize the discrepancy expressed by \( g_j(Y ; \theta) \). This is conceptually similar to matrix transformations in linear models, which measure differences or errors. By choosing a \(\theta\) that minimizes these differences, we improve the fit and accuracy of our predictions. Thus, the estimating function becomes a guide, showing where our model aligns well and where it needs adjustments. It allows us to incorporate historical data to refine our parameter estimation continually.

Conditional Density

When dealing with time series or sequential data, understanding the conditional density is fundamental. The concept involves considering the distribution of an observation given its past data points. In mathematical terms, for any given time point \( Y_j \), the exercise specifies \( Y_j \mid Y_{j-1} \sim N(\theta Y_{j-1}, \sigma^2) \).
This indicates that \( Y_j \) follows a normal distribution where the mean is based on the past observation \( Y_{j-1} \) scaled by \( \theta \). The variance remains constant at \( \sigma^2 \). This characteristic distribution pattern means that the data points have a predictable spread and concentration due to their normality assumption. Hence, when estimating\( \theta \), this predictability is a tool for improving precision. Understanding and utilizing conditional densities ensures that we model relationships between data points accurately, leveraging dependencies effectively.

Log-Likelihood

Log-likelihood functions simplify the process of parameter estimation in complex data models by transforming the likelihood into a summative form. Given the likelihood function in the exercise: \[ L(\theta; Y) = \prod_{j=2}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left(-\frac{(Y_j - \theta Y_{j-1})^2}{2\sigma^2}\right) \]The log-likelihood simplifies this product into a manageable sum: \[ \log L(\theta; Y) = -\frac{n-1}{2} \log(2\pi \sigma^2) - \frac{1}{2\sigma^2} \sum_{j=2}^{n} (Y_j - \theta Y_{j-1})^2 \]The advantage is distinct: addition is mathematically more tractable than multiplication. By utilizing the log-transformed version, it becomes easier to differentiate and optimize. For our case, finding \( \theta \) is just about dealing with derivatives of sums, making procedures efficient and results more accessible to interpret. The log-likelihood turns the noisy complexity of data into a map to find the parameters that make our model plausible.

Homoscedasticity

Homoscedasticity refers to the assumption of constant variance across different data points. This concept is crucial for regression models, and it is applied in our exercise through \(\operatorname{var}\{g_j(Y ; \theta) | Y_1, \ldots, Y_{j-1}\} = \sigma^2\).
This assumption suggests that no matter the time point or observation considered, the variability of \(g_j(Y; \theta)\) stays the same. Such uniform property simplifies the process for estimating and interpreting results, as it negates potential data distortions related to unequal spread of residuals across the data set. By assuming homoscedasticity, our model primarily focuses on capturing patterns and relationships rather than being misled by variable variance levels. Nonetheless, it’s important to verify this assumption in real data scenarios, as ignoring heteroscedasticity (variable variance) can lead to inefficient estimates. Additionally, fitting models with homoscedastic variance provides a solid baseline for comparing more complex models later with tools such as diagnostic tests.