Certifications are Evil.....By John McCash

Note: The following does not represent the opinion of Mark McKinnon. He merely had the good grace to allow me a forum in which to post it after it was respectfully declined (for obvious reasons) by the SANS Institute's Forensic Blog. I wrote it chiefly because I hadn't seen anything recently, or as I recall, ever, that so much as acknowledged any downside to certification. I respect the pro-certification viewpoint, but I do disagree with it. And so, without further ado...

Certifications are Evil

by John McCash

Folks, this is an opinion piece, and it's going to be a controversial one. Some of you started composing a scathing rebuttal to it as soon as you read the title. Normally I restrict myself to what I hope are useful technical tidbits, but like most of you out there, I'm a forensic practitioner, and I have little patience for time sinks which provide no benefit (no I'm not including the training in that category, save your flames for the end). I've always begrudged the time commitment (over and above what's required to actually take the training and learn the included material) required to attain certifications, despite which I'm in possession of five, soon to be six, not counting my master's degree, so I like to think I speak from some degree of experience.

I do understand the arguments used by the proponents of certification. In essence, they allow people who have no understanding of a technical discipline to discriminate between other people who do and don't have that understanding. At least that's what they're supposed to do. Let me list two of the most egregious counterexamples that I have found in my own personal experience (with no disrespect intended to either Microsoft or the International Information Systems Security Certification Consortium). I have met, in my career, an extraordinarily large number of clueless CISSPs and MCSEs. These are people who were apparently able to pass the test, but who were unable to, respectively, secure or administer their way out of wet paper bags. To state it in more general/inflammatory terms, one problem with certifications is the number of idiots who are in possession of them. On the flip side of this, I personally oversaw the hiring of a system administrator back in 1996 who had nothing but a High School Diploma and a clue. I still work with him on occasion, and his hiring was one of the smartest decisions I ever made.

One logical response to this issue is simply to make certifications more difficult to get, but there we run into a second fundamental problem. When a certification raises its difficulty in order to exclude a certain percentage of unqualified people, they also exclude a certain percentage of qualified people. As the difficulty raises more and more, the incremental number of unqualified people being excluded gets smaller, and the incremental number of qualified people being excluded becomes larger. The amount of work required in order to to pass increases substantially as well. Qualified people get excluded for several reasons. For one, the more difficult a certification, the more training is typically required before attempting the exam. One forensic certification I heard about last week, the one which finally prompted me to write this posting, requires six months of training and six exams. That's a tremendous amount of time committed to obtaining a fancy certificate and some alphabet soup to put on your resume. Don't get me wrong, I'm not saying that training is useless. But what do you do if you're already in possession of 75% of the knowledge this training is intended to pass on? It's in the financial interest of the certification providers to make it more difficult to pass the certification if you haven't attended their custom-designed training program. Review guides may be available, but typically cover more material than the certification vendor's training, without the subtle emphasis often provided by that training. The practical upshot of this is that an individual who may know 75% of the material on the exam off the top of his head, substantially better than a graduate of the certification course will be (probably) after six months or so, may still have to complete a long and expensive training course just to get to a point where he can reliably pass the certification exam. For many of us, it's simply not worth it. We resign ourselves to being filtered out because we don't have the requisite alphabet soup, even though we're otherwise qualified.

You'd think that at some point, an exam would filter out all the idiots, but that's much harder than you'd think. That's why IQ tests have fallen out of vogue, and why an actual interview is still the best way to select a new employee. This brings me to the third reason certifications, or more specifically certification exams are bad. Many standardized tests consist of simple regurgitation of facts. They don't require that the subject really be able to think, just memorize. Personally, I believe that any idiot can pass such a test if they put sufficient time into preparation. It's possible to design questions to test problem solving ability, but it's difficult. One tactic that's often resorted to, and this is a personal hot button of mine, is to provide the subject limited information, allow him to assume the rest, and make him pick the 'most reasonable' or 'best' solution from the list. The problem with this occurs when the test subject is smarter or knows more than the individual who designed the question. I personally have run into this several times on various certification exams (I got a couple of the questions changed), and I find it intensely frustrating.

Finally, certifications are bad because they provide lazy people with a tool that can be easily misused. Rather than read 100 resumes to determine the 15 most qualified for a particular position (which he may lack the expertise to do anyway), an HR person can simply filter out all those lacking a specific certification. If this still results in a number of resumes that is too large, he can filter on another certification. This sort of data reduction can easily remove more qualified people than unqualified. In my opinion, it's better to pass all 100 resumes down to the hiring manager.

Certifications are bad for hiring managers, because they reduce their pool of qualified candidates, and they're bad for the candidates, because they enable those candidates' resumes to be filtered out before the manager sees them. In the end, they provide the most benefit to the vendors who provide them and their associated training, and to HR organizations, who are able to get by with fewer and less expert people.

Once a certification is accepted as required in a certain area, this fact can be used by people who lack training in that area to obtain it. The downside of this is that people who are already qualified sometimes must forgo more advanced training to take training just to get the certification. I'm not suggesting they don't learn anything in this training, but typically it will be much less than they could have learned had they been able to attend training of their choice.

So, you might ask, what's the alternative? Isn't there some other low-overhead way to reliably tell if a candidate knows anything about a given specialty without actually reading his resume or interviewing him? Well, I have a suggestion. Maybe somebody out there can make it work. It's based on word of mouth, and the PGP web of trust. Basically, there are a number of people who's word I trust if they say somebody has a clue. If everybody had one or more PGP keys with a comment that said "I am an expert in X", then people could sign that key, and the subject could publish the result. If Rob Lee, Ed Skoudis, & Josh Wright all say I'm an Uber Geek (and I'd like to think they might), I tend to think most people would buy into it. Maybe we could call this the web of cluefulness.

As always, please feel free to leave commentary if you liked this article or want to call me on the carpet for some inaccuracy.

Let the flames commence!

John