Question

Consider the approximation to the first derivative $$ f^{\prime}(x) \approx \frac{f(x+h)-f(x)}{h} $$ The truncation (or discretization) error for this formula is \(\mathcal{O}(h)\). Suppose that the absolute error in evaluating the function \(f\) is bounded by \(\varepsilon\) and let us ignore the errors generated in basic arithmetic operations. (a) Show that the total computational error (truncation and rounding combined) is bounded by $$ \frac{M h}{2}+\frac{2 \varepsilon}{h}, $$ where \(M\) is a bound on \(\left|f^{\prime \prime}(x)\right|\). (b) What is the value of \(h\) for which the above bound is minimized? (c) The rounding unit we employ is approximately equal to \(10^{-16}\). Use this to explain the behavior of the graph in Example \(1.3\). Make sure to explain the shape of the graph as well as the value where the apparent minimum is attained. (d) It is not difficult to show, using Taylor expansions, that \(f^{\prime}(x)\) can be approximated more accurately (in terms of truncation error) by $$ f^{\prime}(x) \approx \frac{f(x+h)-f(x-h)}{2 h} $$ For this approximation, the truncation error is \(\mathcal{O}\left(h^{2}\right)\). Generate a graph similar to Figure \(1.3\) (please generate only the solid line) for the same function and the same value of \(x\), namely, for \(\sin (1.2)\), and compare the two graphs. Explain the meaning of your results.

Short answer

In conclusion, by analyzing the total computational error caused by truncation and rounding errors, we can find the optimal value of \(h\) that minimizes this error. The behavior of the graph in Example 1.3 illustrates the trade-off between rounding and truncation errors as \(h\) varies. When comparing the original formula to the improved formula, we find that the new formula provides a more accurate approximation of the derivative due to its lower truncation error term. Understanding the sources of error in these approximation methods allows us to make better choices when implementing them.

Step-by-step solution

Consider truncation error

To find the truncation error, consider the Taylor expansion of \(f(x + h)\): $$ f(x + h) = f(x) + hf'(x) + \frac{h^2}{2}f''(x) + \mathcal{O}(h^3) $$ Rearrange this expression for \(f'(x)\) as follows: $$ f'(x) = \frac{f(x + h) - f(x)}{h} - \frac{h}{2}f''(x) + \mathcal{O}(h^2) $$ Comparing this to the approximation formula, we find that the truncation error is the additional terms not accounted for in the approximation: $$ TE = \left|\frac{h}{2}f''(x) + \mathcal{O}(h^2)\right| \leq \frac{Mh}{2} $$ where \(M\) is a bound on \(\left|f^{\prime \prime}(x)\right|\).

Consider rounding error

In evaluating the function \(f(x)\), the absolute error is bounded by \(\varepsilon\). Therefore, we can approximate the rounding error: $$ RE \approx \left|\frac{f(x + h) - f(x)}{h}\right| \leq \frac{2 \varepsilon}{h} $$

Combine errors and show bound

Now, to find the total computational error, combine the truncation error and the rounding error: $$ TCE = TE + RE \leq \frac{Mh}{2} + \frac{2 \varepsilon}{h} $$ (b) What is the value of \(h\) for which the above bound is minimized?

Find minimal h

To find the optimal value of \(h\), take the derivative of the total computational error bound with respect to \(h\) and set it equal to 0: $$ \frac{d}{dh}\left(\frac{Mh}{2} + \frac{2 \varepsilon}{h}\right) = M - \frac{4 \varepsilon}{h^2} = 0 $$ Solve for \(h\): $$ h = \sqrt{\frac{4 \varepsilon}{M}} $$ (c) The rounding unit we employ is approximately equal to \(10^{-16}\). Use this to explain the behavior of the graph in Example \(1.3\). Make sure to explain the shape of the graph as well as the value where the apparent minimum is attained.

Interpret behavior and minimum

The total computational error bound increases as the value of \(h\) approaches zero since the division by \(h\) results in a larger rounding error. As \(h\) increases, the truncation error becomes more significant, causing the bound to also increase. The shape of the graph reflects the balance between these two errors. The minimum value where the apparent minimum is attained is given by the previously found optimal value of \(h\): $$ h_{min} = \sqrt{\frac{4 \varepsilon}{M}} = \sqrt{\frac{4 \times 10^{-16}}{M}} $$ Since we don't know the exact value of \(M\), we cannot find the exact value of \(h_{min}\), but we can infer that it varies depending on the bound on the second derivative. (d) Generate a graph similar to Figure \(1.3\) (please generate only the solid line) for the same function and the same value of \(x\), namely, for \(\sin (1.2)\), and compare the two graphs. Explain the meaning of your results.

Generate the graph

To generate the graph, compute the approximation for \(f'(x)\) using both the original formula and the new formula for various values of \(h\). Plot the results and compare the graphs. The graph can be easily plotted using any graphing software or programming language (e.g. Python, MATLAB).

Compare and interpret graphs

The graph for the new formula will have a smaller overall error, as the truncation error is of order \(\mathcal{O}(h^{2})\) compared to \(\mathcal{O}(h)\) for the original formula. This means that the new formula will have a more accurate approximation for the same value of \(h\), and the graph will be closer to the true derivative.

Key concepts

Truncation Error

In numerical differentiation, the concept of truncation error is important to understand when approximating derivatives using finite differences. Truncation error occurs when we cut off or "truncate" part of a mathematical process, typically due to using an approximation instead of an exact formula.

The approximation formula given in the exercise, \[ f'(x) \approx \frac{f(x+h)-f(x)}{h} \]introduces a truncation error because it does not account for all terms in the Taylor expansion of \(f(x+h)\).

• The Taylor expansion of \(f(x+h)\) is given by:

\[f(x + h) = f(x) + h f'(x) + \frac{h^2}{2} f''(x) + \mathcal{O}(h^3)\]

• When rearranged for \(f'(x)\) to form our derivative approximation, the resulting formula omits higher-order terms like \(\frac{h^2}{2}f''(x) + \mathcal{O}(h^3)\), which results in a truncation error.

This error becomes significant as \(h\) is increased. It is often expressed in terms of \(M\), a bound on \(|f''(x)|\), leading to a truncation error expressed as:

• \(TE = \left|\frac{h}{2}f''(x) + \mathcal{O}(h^2)\right| \leq \frac{M h}{2}\)

Understanding truncation error helps us decide the appropriate size for \(h\) while interpreting the accuracy of derivative approximations.

Rounding Error

Rounding error is another form of error that arises in numerical differentiation. Unlike truncation errors that depend on mathematical approximations, rounding errors are due to the limitations of representing numbers precisely in digital computers.

• When functions like \(f(x+h)\) and \(f(x)\) are evaluated on a computer, the actual values are approximated to the limits of the computer's precision, such as float or double data types.
• These approximations can amplify the overall error in numerical computations, especially when small differences like \(f(x+h) - f(x)\) are divided by \(h\).

Given that the error in function evaluation is bounded by \(\varepsilon\), the rounding error can be approximated by:

• \(RE \approx \left| \frac{f(x+h) - f(x)}{h} \right| \leq \frac{2 \varepsilon}{h}\)

This formulation indicates that rounding errors increase when \(h\) becomes very small, because the division by a small \(h\) magnifies the impact of imprecise function evaluations. In practical terms, choosing an optimal \(h\) involves balancing both truncation and rounding errors to minimize the total error.

Taylor Expansion

The Taylor expansion is a key mathematical tool that helps us understand and compute derivatives more accurately in numerical contexts. It allows us to express functions as an infinite sum of terms, each involving derivatives of the function evaluated at a single point.

The first-order Taylor expansion for the function \(f(x+h)\) is written as:\[f(x + h) = f(x) + h f'(x) + \frac{h^2}{2} f''(x) + \mathcal{O}(h^3)\]

• This serves as a foundation for deriving reliable numerical approximation methods.
• By utilizing these additional terms, numerical methods like forward and central difference formulas can be developed.

For a more accurate derivative approximation involving reduced truncation error, consider the central difference formula:\[f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}\]

• This utilizes both \(f(x+h)\) and \(f(x-h)\), effectively canceling out higher-order error terms due to symmetry.
• The truncation error is reduced from \(\mathcal{O}(h)\) in the forward difference formula to \(\mathcal{O}(h^2)\).

Taylor expansion thus not only provides insight into the behavior of functions but also aids in designing better numerical techniques with improved accuracy, making it an essential concept in numerical analysis.