More on SDL & Counting Vulnerabilities

This will be my last post on this topic, or at least I hope this is my last post.  But, Dave Litchfield clarified his comments and so did Ryan Naraine (via my Facebook wall) so I thought I would add some of mine.  I mean, I could be in worse company than Litchfield and Naraine right?  😉
 
Ryan said the following:

 “the sdl works (no argument there at all!). my contention is that counting vulns is the worst metric you can use to prove that. in fact, because of silent patching, rolling up fixes into service packs, etc., the stats are misleading. it’s security-by-pr.”

 I agree with this comment from Ryan.  In fact I would be a hypocrite if I didn’t.  That said, I think we are all talking about two very different things — Vulnerabilities found during the SDL process and vulnerabilities that are silently fixed by a vendor.

As I said in my last post, SDL found vulnerabilities should not be counted nor do they really matter because they do not affect the end user in the least.  So read that again before you comment or flame me.  VULNERABILITIES FOUND IN THE SDL PROCESS.  These are issues that have no need to be public because they are not found in public code.

But, as Ryan said — issues found in public code that are fixed silently are a real issue.  While I have picked on Microsoft specifically for this practice the sad reality (that I quickly learned after publicly picking on MS) is that pretty much all vendors do this.

So I ask you, what is the real risk?  Vulnerabilities in public software that we all know about or vulnerabilities in that same code that we have no clue exists and are patched without us being notified….

Personally, I would rather know about all the issues including those found internally by vendors in code they have released to the public.  The fact that ALL VENDORS feel that they do not need to disclose this is a real problem and sadly I don’t think this will change until one big vendor steps up and takes the lead for the rest to follow.

So while you can never get a proper count of vulnerabilities because of vendors practice of silently fixing issues I would still argue that there have been less vulnerabilities in newly released software and (perhaps I am dreaming) I would like to think that the SDL is at least mostly responsible for this.