Random musings on software quality
Posted by aogTuesday, 30 March 2004 at 21:50 TrackBack Ping URL

This post at Winds of Change about electronic voting crystallized some thoughts I’ve had about ISO-9000 Certification. I have deep concerns about electronic voting, because, as far as I can tell, the proponents and manufacturers are basically clueless about security. In my real job I work with financial companies on network security. Those guys are hard core and will spend what it takes to be secure, so I’ve seen what real computer security looks like1 and electronic voting today doesn’t. Some of the commentors called for “ISO-9000” certification, as if that would ameloriate the problem. However, all an IS0-9000 certification means is

  • The company has official procedures
  • These procedures are (mostly) followed in actual practice

That’s it. If your procedure is “the programmer writes code and when it compiles, we ship it” then there’s no reason you can’t get ISO-9000 certification. Because of this, ISO-9000 is frequently the target of much mockery, but I’ve come to realize over the last couple of years that it’s actually useful. It’s in fact very similar to financial audits. If your company is successfully financially audited, this says nothing about the financial health of your company. It simply means that your books are in order. In exactly the same way, ISO-9000 certification means that the books of procedures are in order and reasonably approximate what goes on in the company. Whether these procedures are good or bad is a different issue. It’s much easier to judge whether procedures are being followed (which is somewhat objective) vs. judging the procedures themselves. But for a certified company, you can read those procedures and have some confidence that they’re not complete fiction, which makes the procedures themselves a good basis for forming your opinion of the company.

One problem is that the ISO community really hypes this reality. The core message is something I competely agree with - you can’t improve what you’re doing if you don’t know what it is you do. There’s no hope of improving internal processes if you don’t know what actually goes on. If actually following the official procedures grinds the business to a halt, that tends to indicate that the procedures are flawed and should be fixed rather than simply being ignored. Where the ISO boosters go wrong, in my opinion, is viewing getting knowledge and control of your procedures as sufficient. It’s not stated outright but it’s certainly the impression I get from reading the materials. However, process is just a tool. You master it in order to do something else. If you don’t know what that is, then control of process won’t be much help.

1 Such as, there is no single person in the entire company who can change the network fabric. It always takes two (sometimes more). Generally the two people have to have completely different reporting chains as well. I’ve seen no evidence that individuals can’t change the code running in the voting machines, which is simply not secure. No auditable logfiles of voting activity? If we told the financial guys that our system didn’t have that feature they’d have just one word for us - “next!”.