Vulnerability Engineering
In this article we are going to use some metrics from Software Engineering and apply them to the Vulnerability Research World. We are going to define a new term which will allow us get a probabilty showing how likely is an application to have a vulnerability during its lifetime and also will give an idea of the Software Reliability.
Let's start defining two well-known metrics MTTR and MTBF but within our context:
- MTTR (Mean Time To Repear): Specify the average time a company takes to fix a vulnerability.
- MTBF (Mean Time Between Failures): We are going to use this metric as the average time between to reported vulnerabilities.
Software Reliability can be calculated by the next formula:
to give prospective to the article we are named this value VFP (Vulnerability Free Probability) which shows how likely is NOT an application to have a bug during its lifetime.
The complementary probablity VEP (Vulnerabilty Exposed Probablity). Application's probability of having a vulnerability during its lifetime.
Now it is time to apply these metrics and get some real measurements. Thanks to Hispasec and its paper about Vendor's vulnerabilities this task is going to be easier for us, since they have already compiled all the bugs information.
We are going to apply this metrics to 5 applications Microsoft Explorer, SUN Java JRE, HP NodeManager, Apple Quicktime and Adobe Reader.
First we calculate the MTTR and MTBF for the above applications, as you can see in the spreadsheet.
The following figure shows the VFP or Software Reliability for the above applications.
As can be seen in the figure the reability of these products is far from good, in fact they are worst than I expected. Even a vulnerability free probability of 40% is too large for me.
Only Adobe Reader is above the 50% and QuickTime is only 1% of its lifetime free of vulnerabilities!
Apple's QuickTime users are exposed to at least one vulnerability the 99% of the time they use it.
These metrics should be taken more seriosly when choosing certain software for politicians and people which high risk of being attacked.
Part II:
Besides these metrics, we can retrieve some patterns from the data we first had. The next step is to see which is the trend in the response-time when fixing vulnerabilities.
Let's calculate the MTTR for each bug reported so we can plot a scattered bunch of points. Afterwards, we try to linearly fit this points so we can see the trend.
As can be seen in the figures shown above, Microsoft's Explorer and Sun's Java JRE have reduced their time to fix vulnerabilties since the first report date we have.
On the other side, Quicktime seems to keep in its average time. While Adobe Reader looks like its response time is getting worst, the linear approximation is strongly biased by the first measurement, so if we remove this outlier the linear fit is similar to QuickTime's
