Question

Software defects in NASA spacecraft instrument code. Portions of computer software code that may contain undetected defects are called blind spots. The issue of blind spots in software code evaluation was addressed at the 8th IEEE International Symposium on High AssuranceSoftware Engineering (March 2004). The researchers developed guidelines for assessing methods of predicting software defects using data on 498 modules of software code written in “C” language for a NASA spacecraft instrument. One simple prediction algorithm is to count the lines of code in the module; any module with more than 50 lines of code is predicted to have a defect. The accompanying file contains the predicted and actual defect status of all 498 modules. A standard approach to evaluating a software defect prediction algorithm is to form a two-way summary table similar to the one shown here. In the table, a, b, c, and d represent the number of modules in each cell. Software engineers use these table entries to compute several probability measures, called accuracy, detection rate, false alarm rate, and precision.

a. Accuracy is defined as the probability that the prediction algorithm is correct. Write a formula for accuracyas a function of the table values a, b, c, and d.

b. The detection rate is defined as the probability that the algorithm predicts a defect, given that the module actually is a defect. Write a formula for detection rate as a function of the table values a, b, c, and d.

c. The false alarm rate is the probability that the algorithm predicts a defect, given that the module has no defect. Write a formula for false alarm rates as a function of the table values a, b, c, and d.

d. Precision is defined as the probability that the module has a defect, given that the algorithm predicts a defect. Write a formula for precision as a function of the table values a, b, c, and d.

e. Access the accompanying file and compute the accuracy, detection rate, false alarm rate, and precision values. Interpret the results.

Short answer

1. The probability is$P=\frac{a+d}{a+b+c+d}$
2. The probability is$P=\frac{d}{b+d}$
3. The probability is$P=\frac{c}{a+c}$
4. The probability is$P=\frac{d}{c+d}$
5. The values of accuracy, detection rate, false alarm rate, and precision are$84.34\%,40.82\%, 10.91\%\text{and}28.99\%$

Step-by-step solution

Important formula

The formula for probability is$\mathbf{P}\mathbf{=}\frac{\mathbf{favourableoutcomes}}{\mathbf{totaloutcomes}}$

(a) The probability that the prediction algorithm is correct

Here the outcomes are a, b, c, d., and thefavorable outcome are a and d.

So, the probability is $P=\frac{a+d}{a+b+c+d}$.

(b) Write a formula for detection rate as a function of the table values a, b, c, and d

Here the outcomes are b, d. and the favorable outcome is d.

Thus, the probability is $P=\frac{d}{b+d}$.

(c) Write a formula for false alarm rate as a function of the table values a, b, c, and d

Here the outcomes are a, c., and the favorable outcome is c.

Hence, the probability is$P=\frac{c}{a+c}$ .

(d) Write a formula for precision as a function of the table values a, b, c, and d

Here the outcomes are c, d, and the favorable outcome is d.

Accordingly, the probability is$P=\frac{d}{c+d}$ .

(e) Access the accompanying file and compute the accuracy, detection rate, false alarm rate, and precision values. Interpret the results

From the algorithms, the values are:

The value for accuracy:

$\begin{array}{rcl}P& =& \frac{a+d}{a+b+c+d}\\ & =& \frac{400+20}{400+29+49+20}\\ & =& \frac{420}{498}\\ & =& 84.34\%\\ & & \end{array}$

The value for detection rate:

$\begin{array}{rcl}P& =& \frac{d}{b+d}\\ & =& \frac{20}{29+20}\\ & =& 40.82\%\end{array}$

The value for false alarm rate:

$\begin{array}{rcl}P& =& \frac{c}{a+c}\\ & =& \frac{49}{400+49}\\ & =& 10.91\%\\ & & \end{array}$

The value for precision:

$\begin{array}{rcl}P& =& \frac{c}{a+c}\\ & =& \frac{49}{400+49}\\ & =& 10.91\%\\ & & \end{array}$

Therefore, the values of accuracy, detection rate, false alarm rate, and precision are $84.34\%,40.82\%, 10.91\%\text{and}28.99\%$.