Category Archives: Certification

Post navigation

It’s been about a month since the CCIE Continuing Education program was announced ahead of Cisco Live. There was a fair amount of discussion about it both on this blog as well as other places, like Jeff Fry’s post. Overall, the response has been positive. However, there are a few questions and ideas about the program that are simply not true. And no, this is not The Death Of The CCIE Program (just Google it). So, let’s take a look at this edition of Mythbusters for the CCIE CE program.

Myth #1: The CE Program Is Just A Way For Cisco To Sell More Training

This was a good one. The list of CE classes that was release at the beginning of the program included Cisco Live classes as well as Cisco Authorized training classes. Those were the only thing on the list as of right now. When some people saw the list, they jumped to the conclusion that the reason why the CE program exists is because Cisco wants to push their training courses. Let’s look at that.

Let’s say you want to start a global program that requires people to keep track of their training credits to turn them in for some kind of reward, whether it be money or credit for something else. Do you:

Open the program for submissions of any kind and then hire a team to sort through them all to verify that they are legitmate

Use a small list of verified submissions that can be audited at any time internally and are known to be of good quality based on existing metrics

I can only imagine that you would pick #2 every time. Remember that the CCIE CE program is barely a month old. It was announced so people could start taking advantage of it at Cisco Live. The list of classes included on the list was small on purpose. They were Cisco affiliated classes on purpose. The CCIE team can audit these classes easily with internal metrics. They can drop in on them and ensure the content is high quality and appropriate for learners. They can revoke classes deemed too easy or add advanced classes at any time.

The list of training classes looks the way it does because Cisco thinks that these are classes that CCIEs would learn from. They weren’t picked at random to get class sizes higher or to make more profit for Cisco. These classes are something that people would benefit from. And if you’re going to be taking the class anyway or are looking to take a class on a subject, wouldn’t you rather take one that you can get extra credit for?

Myth #2: The CCIE CE Program Was Designed to Sell More Cisco Live Conference Passes

Another chuckle-worthy conclusion about the CCIE CE program. People assumed that because Cisco Live courses were included in the acceptable courses for CE credits, Cisco must obviously be trying to push people to register for more Cisco Live courses, right?

It is true that the CCIE CE program was announced right before Cisco Live 2017. I personally think that was so the CCIEs attending the conference could get credit toward any classes they had booked already. Yes, the courses count. And yes, the longer 4-hour and 8-hour Techtorial classes count for more credits than the 1-hour sessions. But, there is a limit to how many classes count for credit at Cisco Live in total. And there is a cap of 70 credits per cycle on Cisco Live credits in total.

Even if Cisco wanted to use the CCIE CE program to push Cisco Live attendance, this isn’t the best way to do it. The Cisco Live option was to reward those that went anyway for things like advanced training classes and the CCIE NetVet lunch with the CEO. If Cisco wanted to make the CCIE dependent on Cisco Live, they could easily go back to the model of a specific conference just for CCIE recert as they did in the past. They could also just require a specific number of 3000-level classes be taken to recertify, again as in the past, instead of awarding points for other things like Techtorials. Thanks to Terry Slattery for helping me out with these last two points.

Additionally, tying CCIE CE credits to Cisco Live is a horrible way to push conference attendance. Most of the “cool stuff” happening at Cisco Live right now is happening in the DevNet Zone. Many people that I talked to ahead of the conference this year are strongly considering getting Explorer or Social passes next year and spending the whole time in the DevNet Zone instead of the conference proper. If Cisco wanted to push Cisco Live conference pass purchases, they would lock the DevNet Zone behind a more expensive pass.

Myth #3: There Are No Third Party CCIE CE Credits Because Cisco Hates Competition

This myth is currently a half truth. Yes, there are no third party CCIE CE options as of July 2017. Let’s go back to myth #1 and take a look at things. Why would Cisco open the program to the whole world and deal with all the hassle of auditing every potential source of CE credits just after launching the program? Sure, there are a lot of great providers out there. But, for every Narbik bootcamp there’s a bunch of shady stuff going on that isn’t on the up-and-up. But investigating the difference requires time and manpower, which aren’t easy to come by.

Ask yourself a simple question: Do you think Cisco will never have third party options? I can almost guarantee you the answer is no. Based on conversations I had with CCIE program people at Cisco Live this year, I would speculate that the CCIE CE program will expand in the future to encompass more training options, including third parties. I would bet the first inclusions will be certified trainers offering official courses. The next step will be auditing of classes for inclusion, like bootcamps and other semi-official classes. Expansion will be slow, but the classes that make the grade will help enhance the program.

What won’t be included? Youtube videos. Training webinars that are just cleverly disguised promotional pitches. Anything that is given without any way to track down the author and verify their knowledge level. And, as much as it pains me, I can almost guarantee that blog posts won’t count either. Cisco wants to be able to verify that you learned something and that you put in the effort. The only way to do that is through class attendance auditing and verification. Not through Youtube views or blog post counters.

Tom’s Take

For a program that’s less than a month old, there were a lot of people rushing to pass judgement on the hard work put into it. To pronounce the death of something that has endured for more than 20 years is a bit presumptuous. Is the current version of the CCIE CE program perfect? Nope. However, it’s better than the lack of a CE program we had three months ago. It’s also a work-in-progress that will only get better over time. It’s a program that Cisco is going to put significant investment into across the entire certification portfolio.

Rather than tearing down the hard work of so many people for the sake of ego stroking, let’s look at what was delivered and help the CCIE program managers build a bigger, better offering that helps us all in the long run. Cisco wants their CCIEs to succeed and go far in the networking world. And that’s no myth.

Like this:

Studying for a big exam takes time and effort. I spent the better part of 3 years trying to get my CCIE with constant study and many, many attempts. And I was lucky that the CCIE Routing and Switching exam is offered 5 days a week across multiple sites in the world. But what happens when the rug gets pulled out from under your feet?

Not Appearing In This Testing Center

The Cisco Certified Design Expert (CCDE) is a very difficult exam. It takes all of the technical knowledge of the CCIE and bends it in a new direction. There are fun new twists like requirements determination, staged word problems, and whole new ways to make a practical design exam. Russ White made a monster of a thing all those years ago and the team that continues to build on the exam has set a pretty high bar for quality. So high, in fact, that gaining the coveted CCDE number with its unique styling is a huge deal for the majority of people I know that have it, even those with multiple CCIEs.

The CCDE is also only offered 3-4 times a year. The testing centers are specialized Pearson centers that can offer enhanced security. You don’t have to fly to RTP or San Jose for your exam, but you can’t exactly take it at the local community college either. It’s the kind of thing that you work toward and get ready for, not the kind of spur-of-the-moment sales test that you have to take today to certify on a new product.

So, what happens if the test gets canceled? Because that’s exactly what happened two weeks ago. All those signed up to take the May 17, 2017 edition of the exam were giving one week’s notice that the exam had been canceled. Confusion reigned everywhere. Why had Cisco done this? What was going to happen? Would it be rescheduled or would these candidates be forced to sit the August exam dates instead? What was going on? This was followed by lots of rumors and suggestions of impropriety.

According to people that should know, the February 2017 exam had a “statistically signification pass rate increase”. For those that don’t know, Cisco tracks the passing exam scores of all their tests. They have people trained in the art of building tests and setting a specific passing rate. And they keep an eye on it. When too many people start passing the exam, it’s a sign that the difficulty needs to be increased. When too few people are able to hit the mark, a decision needs to be made about reducing difficulty or examining why the pass rate is so low. Those are the kinds of decisions that are made every day when the data comes back about tests.

For larger exams like the CCDE and CCIE that have fewer test taking opportunities, the passing rate is a huge deal. If 50% of the people that take the CCIE pass, it doesn’t say very much for the “expert” status of the exam. Likewise, if only 0.5% of test takers pass the CCDE on a given attempt, it’s an indicator that the test is too hard or overly complex and less about skill demonstration and more about being “really, really tough”. The passing rate is a big deal to Cisco.

According to those reports, there was a huge spike in the passing rate for the February CCDE. Big enough to make Cisco shut down the May attempts to find out why. If you have a few percentage points variance in passing or failing rates now and then, it’s not a huge deal. But if you have a huge increase in the number of certified individuals for one of the most challenging exams ever created, you need to find out why. That’s why Cisco went into damage control mode for May and maybe even for August.

The Jenga Problem

So, if your still with me at this point, you’ve probably figured out that there’s a good possibility that someone got their hands on a copy of the CCDE exam. That’s why Cisco had to stop the most recent date in order to ensure that whatever is out in the wild isn’t going to cause issues with the exam passing rates going forward. It makes sense from a test giver perspective to plug the leak and ensure the integrity of the exam.

Yes, it does suck for the people that are taking the exam. That’s a lot of time and effort wasted. One of the things I kept hearing from people was, “Why doesn’t Cisco just change the test?” Sadly, changes to the CCDE are almost impossible at this point in the game.

Even written exams with hundreds of test bank questions are difficult to edit and refresh. I know because I’ve been involved the process for many of them in the past. Getting new things added and old things removed takes time, effort, and cooperation among many people on a team. Imagine the nightmare of trying to get that done for a test where every question is hand-crafted and consists of multiple parts. Where something that is involved in Question 2 of a scenario has an impact on Question 8.

Remember, exams like the CCIE and CCDE are infamous for word choices mattering a lot in the decision making process of the candidate. Changing any of the words is a huge deal. Now, think about having to change a question or two. Or a scenario or two out of four. Or a whole exam revision. Now, do it all in two weeks. You can see what the CCDE team was faced with and why they had to make the decision they did. Paying customers are angry, but the possibility that the integrity of a complicated exam is compromised is worth more to Cisco in the long run.

Tom’s Take

What’s going to happen? It’s difficult to say. Cisco has to find out what happened and stop it immediately. Then, they have to assess what can be done to salvage the exam as it exists today. Scenarios must be created to replace known bad versions. Evaluation of those new scenarios could take a while. And in the interim, Cisco can’t just shut down the testing and certification process. It would be the end of the CCDE and cause a huge backlash in the elite community that exists to spread the good word of Cisco throughout networking circles. Cisco isn’t acting from a position of confusion, but instead from a position of caution. The wrong step at the wrong time could be disastrous.

Share this:

Like this:

In CCIE news this week, Cisco has raised the price of their exams across the board. The CCNA has moved up to $325, and the CCIE Written moves from $400 to $450. It goes without saying that there is quite a bit of outcry in the community. Why is the price of the CCIE Written exam surging so high?

No Such Thing As A Free Test

The most obvious answer is that the amount of work going in to development of the exam has increased. The number of people working behind the scenes to create a better exam has caused the amount of outlay to go up, hence the need to recover those costs. This is the simplest explanation of all the cost increases.

As Cisco pours more and more technology into the tests, the amount of hands and fingers touching them has gone down. At the same time, the quality of the eyeballs that do look at the exam has gone up. It’s a lot like going to a specialist doctor. The quality of the care you receive for your condition is high, but the costs associated with that doctor are higher than a regular general practice doctor. Cisco’s headcount is now focused on keeping exam quality high. That kind of expertise is always more expensive per capita, even if the number of those people is fewer.

The odd thing here is that even if the costs of the people doing the work are going up, the amount that the test is increasing doesn’t seem to correlate. It’s been less than two years since the formal introduction of the current version of the CCIE written exam at the then-unheard of price point of $400. We’re two and a half years removed from the CCIE 4.0 Written exam and it’s lofty $350 price point. Has the technology changed so much in less than three years?

The Great Barrier Test

Going back to the introduction of the 5.0 version of the CCIE Written, there was also a retake policy change introduced. Cisco wanted to create a “backoff timer” to reduce the amount of times that a person could take the exam before needing to wait. The change still allowed you to take the second attempt after 30 days, but then the third attempt must wait an additional 90 days after that. So, instead of being able to get three exam attempts in 60 days, those same three attempts would have taken 120 days.

This change was rolled back about six months ago due to outcry from the community. CCIEs trying to recertify were stymied by the exam and forced to wait longer and longer to pass it, with their certification hanging in the balance. With the increased timeouts and limit of four retakes per year, some long time CCIEs were in danger of exhausting their attempts and watching their certification slide away without any recourse to fix it.

Now, the increased price behind the CCIE Written could indeed be attributed to the increased overhead. But it could also be an attempt to keep people from rushing in to take the test every 30 days. Making a policy change to keep people out the exam is one way to do it. But making the exam financially painful to continually fail is another. If you’re willing to drop $1350 in three months to try and pass then you either have money to burn or you’re desperate to pass.

In addition, a higher exam fee would cause test takers to be absolutely certain of their knowledge level before attempting the exam. Creating an initial barrier to entry that will make people think twice before scheduling an exam on a whim does create a situation where the first-time pass rate will improve significantly. This will also help drive funding to certification materials and classes, as candidates will want to know that they will pass before stepping into a certification exam center.

Tom’s Take

I’d really like to think that Cisco is just trying to cover their overhead with the recent price increases. Everything goes up in price. Some things go up faster than others. But the conspiracy theorist in me wonders if Cisco isn’t trying to use the increased price of the exam to help raise the pass rates and discourage folks from rushing the test repeatedly to see the exam question pool. $450 is a tough pill to swallow even if you pass. I think we’re going to see a lot more people taking advantage of the free Cisco Live exam as well as the half price cert exams there. And I sincerely hope the rumored options for recertification take flight soon. Because I don’t know how ready I am to go all out to study when there’s that much money on the line.

Like this:

I stirred up quite the hornet’s nest last week, didn’t I? I posted about how I thought the CCIE Routing and Switching Written Exam needed to be fixed. I got 75 favorites on Twitter and 40 retweets of my post, not to mention the countless people that shared it on a variety of forums and other sites. Since I was at Cisco Live, I had a lot of people coming up to me saying that they agreed with my views. I also had quite a few people that weren’t thrilled with my perspective. Thankfully, I had the chance to sit down with Yusuf Bhaiji, head of the CCIE program, and chat about things. I wanted to share some thoughts here.

Clarity Of Purpose

One of the biggest complaints that I’ve heard is that I was being “malicious” in my post with regards to the CCIE. I was also told that it was a case of “sour grapes” and even that the exam was as hard as it was on purpose because the CCIE is supposed to be hard. Mostly, I felt upset that people were under the impression that my post was designed to destroy, harm, or otherwise defame the CCIE in the eyes of the community. Let me state for the record what my position is:

I still believe the CCIE is the premier certification in networking. I’m happy to be a CCIE and love the program.

Why did I write the post? Not because I couldn’t pass the written. Not because I wanted people to tell me that I was wrong and being mean to them. I wrote the post because I saw a problem and wanted to address it. I felt that the comments being made by so many people that had recently taken the test needed to be collected and discussed. Sure, making light of these kinds of issues in a public forum won’t make people happy. But, as I said to the CCIE team, would you rather know about it or let it fester quietly?

Yusuf assured me that the CCIE program holds itself to the highest standards. All questions are evaluated by three subject matter experts (SMEs) for relevance and correctness before being included in the exam. If those three experts don’t sign off, the question doesn’t go in. There are also quite a few metrics built into the testing software that give the CCIE team feedback on questions and answer choices. Those programs can index all manner of statistics to figure out if questions are creating problems for candidates. Any given test can produce pages worth of valuable information for the people creating the test and trying to keep it relevant.

Another point that was brought up was the comment section on the exam. If you have any problem with a question, you need to fill out the comment form. Yes, I know that taking time out of the test to provide feedback can cause issues. It also interrupts your flow of answering questions. But if you even think for an instant that the question is unfair or misleading or incorrect, you have to leave a detailed comment to make sure the question is flagged properly for review. Which of the following comments means more to you?

Trivia question

or

This question tests on an obscure command and isn’t valid for a CCIE-level test.

I can promise I know which one is going to be evaluated more closely. And yes, every comment that has purpose is reviewed. The exam creators can print off every comment ever left on a question. The more detailed the comment, the more likely to trigger a review. So please make sure to leave a comment if you think there is a problem with the question.

Clarity Of Vision

Some of the conversations that I had during Cisco Live revolved around the relevance of the questions on the test to a CCIE candidate. Most of the people that I talked to were CCIEs already and using the test for recertification. A few came to me to talk about the relevance of the test questions to candidates that are qualifying for the lab.

While I’m not able to discuss any of the specific plans for the future of the program, I will say that there are ideas in place that could make this distinction matter less. Yusuf told me that the team will be releasing more details as soon as they are confirmed.

The most important point is that the issues that I have with the CCIE Written exam are fixable. I also believe that criticism without a suggestion solution is little more than whining. So I decided to put my money where my mouth is with regard to the CCIE written exam.

I volunteered to fix it.

I stepped up and offered my time as an SME to review the questions on the written exam for relevance, correctness, and grammar. That’s not a light undertaking. There are a ton of questions in the pool that need to be examined. So for every person that agreed with my post or told me that they thought the exam needed to be fixed, I’m putting you all on the spot as well.

It’s time for us as a community of CCIEs to do our part for the exam. Yusuf told me the easiest way to take part in the program is to visit the following URL:

Sign up for the SME program. Tell them that you want to help fix the CCIE. Maybe you only have to look at 5-10 questions. If the hundred or so people that agreed with me volunteered today, the entire test question pool could be analyzed in a matter of weeks. We could do our part to ensure that people taking the exam have the best possible test in front of them.

But I also challenge you to do more. Don’t just correct grammar or tell them they spelled “electricity” wrong in the question. Challenge them. Ask yourself if this is a question a CCIE candidate should know the answer to. There’s a chance that you could make a difference there. But you can’t do that unless you step up the plate.

Tom’s Take

I had at least ten people tell me that they would do whatever it took to fix the CCIE test last week after I talked to the CCIE cert team. They were excited and hopeful that the issues they saw with the test could be sorted out. I’ll admit that I stepped out on a pretty big limb here by doing this in public as opposed to over email or through official channels. And I do admit that I didn’t clarify my intent to build the program up as opposed to casting the whole exam team and process in a bad light.

Mea culpa.

But, my motivation succeed in getting people to talk about the CCIE written. There are many of you that are ready to do your part to help. Please, go sign up at the link above to join the SME program. Maybe you’ll never look at a single question, Maybe you’ll look at fifty. The point is that you step up and tell Cisco that you’re willing. If even fifteen people come forward and agree to help then that message will sound loud and clear that each and every one of us is proud of being a CCIE and want the program to continue long past the time when we’re retired and telling our grandchildren about the good old days of hard but fair tests.

If you have any questions about participating in the program or you want to reach out to me with your thoughts, don’t hesitate to contact me. Let’s put the power of community behind this!

I’m having a great time at Cisco Live this year talking to networking professionals about the state of things. Most are optimistic about where their jobs are going to fit in with networking and software and the new way of doing things. But there is an undercurrent of dissatisfaction with one of the most fundamental pieces of network training in the world. The discontent is palpable. From what I’ve heard around Las Vegas this week, it’s time to fix the CCIE Written Exam.

Whadda Ya Know?!?

The CCIE written is the bellwether of network training. It’s a chance for network engineers that use Cisco gear to prove they have what it takes to complete a difficult regimen of training to connect networks of impressive size. It’s also a rite of passage to show others that you know how to study, prep, and complete a difficult practical examination without losing your cool. But all that hard work starts with a written test.

The CCIE written has always been a tough test. It’s the only barrier to entry to the CCIE lab. Because the CCIE has never had prerequisites and likely never will due to long standing tradition, the only thing standing in the way of you ability to sit the grueling lab test is a 100 question multiple choice exam that gauges your ability to understand networking at a deep technical level.

But within the last year or so, the latest version of the CCIE written exam has begun to get very bad reviews from all takers of the test. There are quite a few people that have talked about how bad the test is for candidates. Unlike a lot of “sour grapes” cases of people railing against a test they failed, the feedback for the CCIE written is entirely different. It tends to fall into a couple of categories:

The Test Is Poorly Written

The most resounding critique of the exam is that it is a poorly constructed and executed test. The question quality is subpar. There are spelling mistakes throughout and test questions that have poor answer selections. Having spent a large amount of time helping construct the CCNA exam years ago, I can tell you that you will spend the bulk of your time creating wrong answers as distractors to the right ones. Guidelines say that a candidate should have no better than a 25% chance to guess the correct answer from all the choices. If you’ve ever taken a math test that has four multiple choice answers with three being correct for various mistakes in working the problem, you know just how insidious proper distractors can be (and math teachers too).

The CCIE written is riddled with bad distractors according to reports. It also has questions that don’t have a true proper answer or a set of answers that are all technically correct with no way to select them all. That frustrates test takers and makes it very difficult to study for the exam. The editing and test mechanics errors must be rectified quickly in order to restore confidence to the people taking the test.

The Test Doesn’t Cover The Material

Once people stop telling me how bad the test is constructed, they start telling me that the questions are bad on a conceptual level as well. No NDAs are violated during these discussions to protect everyone involved, but the general opinion is that the test has skewed in the wrong direction. Cisco seems to be creating a test that focuses more on the Cisco and less on the Internetworking part of the CCIE.

The test has never been confused for being a vendor-neutral exam. Any look at the blueprint will tell you that there a plenty of proprietary protocols and implementation methods there. But the older versions of the exam did do a good job of teaching you how to build a network that could behave itself with other non-Cisco sections. Redistributing EIGRP and OSPF is a prime example. But the focus of the new exam seems to be skewed toward very specific Cisco proprietary protocols and the minutia around how they operate. I’ve always thought that knowing the hello and dead timers of OSPF NBMA areas is a huge time sink and really only justified for test takers, but I also see why knowing that would be important in multi-vendor operations. But knowing the same thing for an EIGRP DMVPN seems a bit pointless.

The other problem is that, by the admission of most test takers, the current CCIE Written Exam study guide doesn’t cover the areas of the blueprint that are potentially on the test. I feel very sorry for my friend Narbik Kocharians here. He worked very hard to create a study guide that would help test takers pass the exam with the knowledge necessary to do well on the lab. And having a test over a completely different area than his guide makes him look bad in the eyes of testers without good cause. It’s like a college class when the professor tells you to study the book but gives you a test over his or her lectures. It’s not fair because you studied what you were told and failed because they tested something else.

CCIEs Feel There Are Better Recert Options

This is the most damaging problem in my mind. About half the test takers for the CCIE written are candidates looking to qualify for the lab. That requires them to take the written exam for their specific track. But the other half of the test takers are CCIEs that have passed the lab and are looking to recertify. For these professionals, any CCIE written exam is valid for recertification.

Many CCIE candidates look to broaden their horizons by moving to different track to keep their CCIE current while they study for service provider, data center, or even collaboration as a topic area of study. For them, the CCIE is a stepping stone to keep the learning process going. But many CCIEs I’ve spoken to in the past few months are starting to take other exams not because they want to learn new things, but because the CCIE Routing and Switch written exam is such a terrible test.

Quite a few CCIEs are using the CCDE written to recertify. They feel it is a better overall test even though it doesn’t test the material to the level that the CCIE R&S written exam does. They would even be willing to take the chance of getting a question on an area of technology that they know nothing about to avoid having to deal with poor questions in their areas of study. Still more CCIEs are choosing to become Emeritus and “retire” so as to avoid the pain of the written exam. While this has implications for partner status and a host of other challenges for practicing engineers, you have to wonder how bad things must be to make retirement of your CCIE number look like a better option.

Tom’s Take

I took the CCIE R&S written last year at Cisco Live. I was so disgusted with the exam that I immediately switched to the CCDE written and recertified my number while simultaneously vowing never to take the R&S written again. From what I’ve heard this year, the test quality is still slipping with no relief in sight. It’s a sad state of affairs when you realize that the flagship test for Cisco engineers is so horribly broken that those same engineers believe it can’t be fixed. They feel that all the comments and feedback in the world are ignored and their expertise in taking exams is pushed aside for higher cut scores and a more exclusive number of candidates. The dark side of it all is the hope that there isn’t an agenda to push official training materials or other kinds of shortcuts that would help candidates while charging them more and/or locking out third party training providers that work hard to help people study for the lab.

Cisco needs to fix this problem now. They need to listen to feedback and get their written problems under control. If they don’t, they may soon find the only people taking the R&S written test are the same kinds of dumpers and cheaters they think they are trying to keep out with a poorly constructed test.

Like this:

Congratulations to Ryan Booth (@That1Guy_15) on becoming CCIE #50117. It’s a huge accomplishment for him and the networking community. Ryan has put in a lot of study time so this is just the payoff for hard work and a job well done. Ryan has done something many dream of and few can achieve. But where is the CCIE program today? And where will it be in the future?

Who Wants To Be A CCIE?

A lot of virtual ink has been committed to opinions in the past couple of years about how the CCIE is become increasingly irrelevant in a world of software defined DevOps focused non-traditional networking teams. It has been said that the CCIE doesn’t teach modern networking concepts like programming or building networks in a world with no CLI access. While this is all true, I don’t think it diminishes the value of getting a CCIE.

The CCIE has never been about building a modern network. It has never been focused on creating anything other than a medium-sized enterprise network in the case of the routing and switching exam. It is not a test of best practices or of greenfield deployment scenarios. Instead, it has been a test of interoperability with an exisiting architecture. It tests the ability of the candidate to add devices and protocols to a stable existing network.

Other flavors of the CCIE test over different protocols or technologies, but the idea is still the same. The only one that even comes close to requiring programming is the CCIE Collaboration, which tests over the ability to customize Cisco Contact Center scripts. Otherwise, each test focuses on technology implementation and not architecture or operation.

Current logic dictates that people don’t want to take the CCIE because it doesn’t teach programming or API interaction. Yet candidates are showing up in droves. It’s almost as if the networks we have today are going to need to be maintained and built out over the coming years. These are the kinds of tasks that are well suited to a support-focused certification like the CCIE. The ideal CCIE candidate isn’t using Vagrant and Chef in a lab somewhere. They’re muddling through OSPF to RIP distribution somewhere in the dark corners of a network that got welded on after an acquisition.

Is Everyone A CCIE?

One thing I have noticed about the CCIE is the fact the number climb seems to have leveled off. It’s not the rapid explosion of certifications that it has been in the past, nor is it the eventual cliff of increased difficulty. Things seem to be marching more toward steady growth. I don’t know how much of that can be attributed to factors like the Cisco official CCIE training program or the upgrade to version 5 almost two years ago.

Lots of CCIEs doesn’t necessarily mean that the test has lost meaning. Microsoft had several thousand MCSEs by the time the certification became a punchline to countless call center jokes. Novell had a virtual army of Certified NetWare Engineers (CNEs) before software changes locked many of them into CNE 5 or CNE 6. Having a lot of certified individuals doesn’t devalue the certificaiton. It’s what people do with it that creates the reputation. Ask and Novell Certififed Directory Engineer (CDE) about the reputation garnered by a test and they can give you a lesson in hard exams that breed bright engineers.

Does that mean that we should brace ourselves for even more CCIEs in the future? It likely won’t be as bad as has been imagined. The written exam for version 5 has pointed out to me that Cisco is going to start closing ranks around technologies in the near future. The written exam serves as a testing ground for potential new topics on the exam. MPLS was a written topic long before it became a potential lab exam topic. The current written exam is full of technologies that make me think Cisco is starting to put more emphasis on the Cisco and less on the Internetworking in CCIE.

Cisco wants to have a legion of certified individuals that think about Cisco technology benefits. That’s why we’re starting to see a shift toward things like DMVPN and GETVPN in testing. In place of industry standard protocols, we get the Cisco improved versions. This locks candidates into the Cisco method of thinking and ensures that their go-to solutions will include some form of proprietary technology.

If this shift in thinking is really the start of the new way of certification testing, I worry for the future of the CCIE. Not because there are 50,000 CCIEs, but because the new inductees into the CCIE group will be focused on creating islands of Cisco in the sea of interoperable data center networks. That’s good for Cisco’s bottom line, but bad for the reputation of the CCIE. Could you imagine what would happen if a CCIE walked in and told you they couldn’t fix your MPLS VPN configuration issues because “I only know how to work on DMVPN”?

Tom’s Take

Every time someone I know passes the CCIE it makes me happy that they’ve completed a rigorous exam testing process. It tells me this person knows how to follow the lab instructions to create an interoperable enterprise network based on constraints. It also tells me that this person knows how to study material and doesn’t give up. Those are the kinds of people I would want in my networking group.

CCIEs are the perfect people to learn more modern network techniques like programmability and SDN. Not because they learned how to do it on their test. But because they are the kinds of people that learn well and will apply everything they have to picking up a new concept. But it needs to be pointed out here that Cisco must foster that kind of interoperable learning experience with CCIEs. Focusing too heavily on proprietary solutions to help create an army of unknowing Cisco SEs in the field will only serve to hurt Cisco in the future when that group of certified individuals must learn to work in the world of networking post-SD.

Like this:

If you haven’t had the chance to read Jeff Fry’s treatise on why the CCIE written should be dropped, do it now. He raises some very valid points about relevancy and continuing education and how the written exam is approaching irrelvancy as a prerequisite for lab candidates. I’d like to approach another aspect of this whole puzzle, namely the growing need to get that extra edge to pass the cut score.

Cuts Like A Knife

Every standardized IT test has a cut score, or the minimum necessary score required to pass. There is a surprising amount of work that goes into calculating a cut score for a standardized test. Too low and you end up with unqualified candidates being certified. Too high and you have a certification level that no one can attain.

The average cut score for a given exam level tends to rise as time goes on. This has a lot to do with the increasing depth of potential candidates as well as the growing average of scores from those candidates. Raising the score with each revision of the test guarantees you have the best possible group representing that certification. It’s like having your entire group be members of the honor roll.

A high cut score ensures that unqualified personnel are washed out of the program quickly. If you take a test with a cut score of 800 and you score a 500, you quickly know that you need to study quite a bit more before you’re ready to attempt the exam again. You might even say to yourself that you don’t know the material in any kind of depth to continue your quest for certification.

What happens if you’re just below the cut score? If you miss the mark by one question or less? How much more studying can you do? What else do you need to know? Sure, you can look at the exam and realize that there are many, many more questions you can answer correctly to hit the right score. But what if the passing score is absurdly high?

Horseshoes and Hand Grenades

I believe the largest consumer of purloined test questions is not the candidate that is completely clueless about a subject. Instead, the largest market of these types of services is the professional that has either missed the mark by a small margin or is afraid they will not pass even after hours of exhaustive study.

Rising cut scores lead to panic during exams. Why was a 790 good enough last time but now I need an 850 to pass? It’s easy to start worrying that your final score may fall in between that gray area that will leave lacking on the latest attempt. What happens if you miss the mark with all of the knowlege that you have obtained?

Those are the kinds of outcomes that drive people to invest in “test aids”. The lure is very compelling. Given the choice between failing an exam that costs $400 or spending a quarter of that to have a peek at what might be on the test, what is stopping the average test taker besides morality? What if your job depended on passing that exam? Now that multi-hundred dollar exam becomes a multi-thousand dollar decision.

Now we’re not talking about a particular candidate’s desire to fleece a potential employer or customer about knowledge. We’re talking about good old fashioned fear. Fear of failure. Fear of embarassement. Fear of losing your livelyhood because of a test. And that fear is what drives people to break the rules to ensure success.

Cut Us Some Slack

The solution to this issue is complicated. How can you ensure the integrity of a testing program? Worse yet, how can you stem the rising tide of improper behavior when it comes to testing?

The first thing is to realize what drives this behavior. Should a test like the CCIE written have higher and higher cut scores to eliminate illicit behavior? Is that really the issue here? Or is it more about the rising cut score itself causing a feedback loop that drives the behavior?

Companies need to take a hard look at their testing programs to understand what is going on with candidates. Are people missing the mark widely? Or are they coming very close without passing? Are the passing scores in the 99th percentile? Or barely above the mark? Adjustments in the cut score should happen both up and down.

It’s easy to look at testing groups and say, “If you just stuided a bit harder, you’d hit this impossibly high mark.” It’s also very easy to look at scores and say, “We see that many of you are missing the mark by less than ten points. Let’s lower that and see how things go from here.”

Certification programs are very worried about diluting the pool of certified candidates. But is having more members of the group with scores within a question or two of passing preferable to having a group with absurdly high passing scores thanks to outside help?

Tom’s Take

I’ve taken exams with a 100% cut score. It’s infuriating to think that even a single wrong answer could cost you an entire exam. It’s even worse if you are financing the cost of your exam on your own. Fear of missing that mark can drive people to do lots of crazy things.

I’m not going to say that companies like Cisco need to lower the cut scores of exams to unrealistically low levels. That would cheapen the certifications that people have already earned. What needs to happen is that Cisco and other certification bodies need to learn what they are trying to accomplish with their programs and adjust all the parameters of the tests to accomplish those goals.

Perhaps raising the cut scores to more than 900 points isn’t the answer. Maybe instead more complex questions or more hands-on simulations are required to better test the knowledge of the candidates. These are better solutions that take time and research. They aren’t the false panacea of raising the passing score. The rising tide can’t be fixed by making the buoys float just a little higher.