Builders and Breakers and Security Evangelists

Security Buddha posted a while back about the lack of "builders" in the security industry. Are there too many breakers in security - people obsessed with finding the latest vulnerabilities and finding exploits? Does it make sense to build software by having a bunch of people build something as fast as possible and then have a bunch of other people try to break it? Or are breakers unnecessary if there are enough builders doing their job properly? 

I got to thinking about this after reading Software Security Top 10 Surprises, a great look at software security in 9 very large software shops. The survey showed that most of these companies have a Software Security Group composed of about 1% of the total development staff (which averaged a whopping 7795 in the companies surveys).  I had the opportunity to talk to David Mortman (Echelon One) and a few others about this data.  One percent seems surprisingly low, but for me the biggest surprise was that these SSGs act mainly as facilitators and evangelists within their companies.  They usually don't perform code reviews or penetration tests.

The emphasis on evangelizing is interesting. Some CISOs act as security evangelists for the organization as a whole. But for CISOs evangelism is only effective after deeply rooted business practices have been re-engineered to reduce the risk of data breach. An effective CISO must drill deep into the organization's DNA to understand how and where data flows from point to point. Only after detailed policies and procedures have taken root can the focus shift from analysis and enforcement to evangelism. A security evangelist in the software world will also only be effective if the underlying development lifecycle enables security.

But even with a security friendly SDLC, can secure products be built without substantial audit and review?  Is empowering and motivating developers enough? In the product world, we are seeing the decline of standalone security products in favor of built-in security functionality.  Will we see the same thing in software development? Will every developer become a mini-security expert, encouraged and guided by security evangelists and management types?

My gut feeling is that outside of highly regulated and very large organizations, the tension between getting things built quickly and building them securely will be too great for this approach to work. Some form of detailed review outside the development team is the only way to ensure security for anything but the lowest security applications. Developers aren't going to become security experts and vice-versa anytime soon. Especially in small companies of 10,20, or 100 developers, there is no room for evangelists. Its all hands on deck all the time. If a security code reviewer or pen tester is needed then bring 'em on.  But there's no room for someone to stand there and just tell people to do a pen test.

Over the years the gap between the builders and breakers has grown bigger.  Its no secret that security is not really an accepted discipline within the development world. There are very few security talks at the leading software conferences. At a recent visit to Barnes and Noble, I flipped through the index pages of a bunch of O'Reilly titles. Security was hardly mentioned in these books, if at all. There is a security shelf, and then row after row of software titles. The two don't mix, and the Java security book was on the security shelf and not on the software shelf next to the Java programming title.  Search the words "secure" or "security" in any of the leading development blogs and you will get almost no hits. 

It is in any case very encouraging that metrics are finally starting to emerge in this area. The OWASP Security Spending Benchmarks project will help us as an industry quantify the cost of security in building software. My guess is that there is a significant difference between companies with thousands of developers and those with a few dozen.