Brian Dennis and I attended Cisco Live! – Networkers this week, and both enjoyed the privilege of sitting down to talk privately with Yusuf Bhaiji (Program Manager over the entire CCIE program) and Ben Ng (Program Manager over the CCIE Voice track) for roughly 45 minutes. It was quite an enjoyable and spirited talk, and I believe it benefited both sides – our side to gain a better understanding of why some of the choices have been made, and theirs possibly to see things a bit more ‘through the eyes of the typical hard-studying student’. I would like to take a moment to jot down some of the highlights from our conversation, and then unpack them in a bit more detail, so that you may benefit from the open conversation.

Highlights

I’ll jot down some very simple, high-level topics that were discussed during our conversation, and then unpack them in more detail in the following section.

Upcoming changes to every CCIE Lab Exam

Protecting the integrity of the CCIE certification

Robust, matured results-based grading engine

Heuristic logic embedded into task wording

Accuracy and detail of lab score reports

Cisco’s CCIE Lab Delivery System and virtualization for mobile labs

No re-reads

CCIE Voice

Next blueprint version expectation

Topics for current and next blueprint versions

CCIE Data Center

CCIE Storage grows up

Reason behind Cisco.com CCIE Statistics web page being removed

Let’s Unpack This a Bit More

Firstly, if I could sum up our entire conversation into one, clear theme, it would be “Protecting the Integrity of the CCIE Program”. Indeed, both Yusuf and Ben clearly stated that central theme was the primary guiding principle behind every meeting they have regarding any aspect of the program, every planning session for the future of the program, and every decision made to add, remove and/or change anything in any of the tracks.

The main focus is to ensure that candidates who are attempting the lab exams are what Cisco is certifying them as: Experts. To ensure that no one is able to memorize any aspect of the exam, and to make sure that those of us that are teaching these potential candidates about the technologies involved, are in fact teaching our students everything they need to know about a given group of technologies necessary to become a true expert – something INE has always worked incredibly hard to do. This focus (and the changes it will affect) implicitly deals with the problems that have arisen in the past with a small minority of people who dishonestly try just to memorize parts of the exam, as well as companies who blatantly cater to that small minority by attempting to gain access to a given exam and publish entire exams, or even parts of them.

Now, this has always been a driving focus for the program, however they realized that while some of their efforts in the past to this end have been successful, that many also may not have succeeded in the manor in which they had intended for them to. Case in point: Core Knowledge/Open-Ended Questions (and their complete removal from all tracks). Another attempt at this from long ago -and still in place but soon to possibly change- was the ‘detailed score report’ (or shall I say ‘lack’ of detail in the score report). More on that last bit in a moment.

Upcoming Changes to Every Lab

So, onto the specifics of how they plan to accomplish this. With no definites given as to exactly how or when the implementation of any of these initiatives would be implemented, some things were quite frank and very clearly stated. They talked about how their lab grading engine has been in use for some time now, and has reached quite a mature level with it’s results-based scoring (both terms of it’s accuracy as well as it’s modularity). Now would be remiss if I didn’t take a quick aside to say when I mention that it has reached a high level of accuracy, some might immediately jump to the conclusion that it wasn’t accurate before, and that might have been the reason for their failed attempt. Just to quell those fears, a (human) proctor has always, and will continue to look over and verify a candidate’s lab and scripted grading results before issuing the final grade. Now with the fact that the engine grades based on results-based testing (and not on any specific commands input into the configuration), as well as the key point that it is extremely modular (due to it’s structured XML nature), any given task can be tested in the same way, however worded in 10-15 different ways on different iterations of any given lab exam. This leads naturally to the ability for them to have upwards of 25 or 50 labs in rotation at any given time for any given CCIE track and blueprint, which makes it nearly imposible for anyone to actually memorize any/all of the questions or versions of lab exams. Thus making those that pass, truly validated as experts. They spoke of a sort-of heuristic logic woven into all of the task wording to accomplish this. Also talked a lot about was how troubleshooting was one of the best things that they added (added ‘back’ I should say, since it used to be there in the 2-day exam). So look for that to not only continue, but permeate its way through more exams in more ways. No mention was specifically made as to whether any other track outside of R/S would go to the same sort of segregated TS section that it utilizes – so it may happen, it may not. Just have to wait and see.

They have been developing towards this goal for some time now, they know exactly how they will carry it out (however to disseminate that specific knowledge would be completely counter-intuitive to the very idea of integrity protection – so anyone telling you they ‘know’ how it’s being done clearly doesn’t know what they are talking about). They mentioned that it will be occurring soon, though when exactly won’t ever be known or disclosed, and it won’t be necessary to ‘announce’ it per-se, since it won’t require the upgrade/change of any hardware/software or technology-specific topics.

All of what we’ve talked about up till this point, this obviously doesn’t negatively impact any of our readers here, as you all are true learners, true knowledge-seekers – those who wish to not only know, but to truly understand (to paraphrase Einstein, if I may). However, it may have some positive implications that you have not yet considered. Aside from the obvious fact that it preserves the integrity of the prestigious certification that you have (and continue to) worked so diligently towards achieving, another possible benefit from this is that since the CCIE program has/will-have this new ability to have so many exams in rotation (and thus exponentially-if-not-entirely reducing the possibility of memorization of tasks), they can relax a bit on their very ambiguous failed score reports. For those of you who are not yet aware, while some candidates pass on their first attempt, most do not. In fact the average (very loose average) is roughly 2-3 attempts at any given CCIE lab before a ‘Pass’ is awarded, and you only receive a broken down score report if you fail – if you pass you simply receive a ‘PASS’ (and frankly, you don’t care that you didn’t receive a report). Now they didn’t state for certain that this would in fact occur (more detailed reports), but they did indicate that it was being discussed internally as a strong possibility – since the grading engine obviously reports back in extreme detail – although he (Yusuf) said it would never include specifically which tasks you got right vs wrong. However, he did also mention that the purpose of the CCIE Program is not to train candidates or provide feedback – it is to test them. It is the responsibility of us, the CCIE training providers, to not only teach, but pre-test and provide accurate feedback to students before they make the actual journey to the Lab and truly become candidates for admission into the prestigious certification that is the CCIE. That by the way, is something that INE is committed to, and indeed already provides more than adequate resources for CCIE R/S, and will soon be adding for the CCIE Voice as well (more on that later).

Another one of the initiatives of the program is to make it much more widely accesible to everyone, everywhere. This was the impetus behind the original idea of having the lab able to be taken from any Prometric/Vue testing center. But Vue wasn’t really ready to handle the stringent requirements, and thus the attempt didn’t ultimately succeed. Then came the CCIE Mobile Testing Labs. Those worked. Well. Really well in fact. And now the push is to make every single track able to be tested in that same mobile fashion. However, there are a few challenges to overcome first, before that can become a reality. Take CCIE Voice for instance – in order for the Voice lab to become truly mobile, everything has to be ported to Cisco’s CCIE Lab Delivery System (where everything: the tasks, the desktop for CLI/GUI, etc. are all completely virtualized). This means the phones themselves as well. Softphones won’t do. They don’t behave at all like hardware phones. So they are working on creating a completely virtualized hardware-like phone, that behaves exactly like a hardware phone (but doesn’t remotely control one). When they get that completed, then you will begin to see the Voice lab become available in the mobile testing centers (along with every other lab once their similar challenges are met).

One last thing I asked Yusuf about regarding specifically the CCIE Voice lab exam, was why they hadn’t yet allowed for re-reads (the ability that, if you fail an exam but think you should have passed, you can pay $300 USD to have your lab re-graded manually). He mentioned that since the overall percentage since they began offering that option was so extremely low of those who request a re-read actually results in a grade being overturned (we’re talking like 1 out of every 5,000), not to mention that that number has dropped even more with results-based scripted grading, that the focus was not to add more of that unnecessary burden on the already taxed proctors, but to reduce it. So while he didn’t provide any guidance on when (if ever) they will completely eliminate that practice from the other tracks (R/S, Sec, SP), he did say that the focus is on removing that option from those tracks vs. adding that option to other tracks (Voice, WiFi, Storage).

CCIE Voice

There was a very interesting breakout session this year entitled “CCIE Voice: Cryptography in Cisco Unified Communications”, which inevitably led to the question by participants: “If this class is prefaced by the title ‘CCIE Voice’, does this mean that Cryptography/Security is going to begin being tested in the lab exam?”. The answer to this when asked by the attendee and then later in a bit more detail by me was answered with the basic answer of (and I’ll paraphrase): “This has been tested in the written exam for some time now, however there is absolutely nothing stopping us from testing this in the lab today with version 7 of the various UC platform servers, and obviously no problem testing it moving forward to UC version 8 (or 9, etc), as even more security has been added to the new version of UC platforms”.

Of course, in the main CCIE Voice 8-hour Technical Seminar, the question was asked by some participants (as well as by me in more detail later in private) when we might anticipate the Voice lab being updated to UC platform version 8.x (or beyond, if FCS’d for better than 6 months by time of announcement), to which Ben gave no real guidance publicly, yet in private alluded to (reverting back to our previous mention in this blog post) the desire to virtualize the hardware phones and deliver the exam with the new virtual Lab Delivery System, so that they could support the mobile labs.

Now of course 8.x has been in production for well over a year now in many networks around the globe, but 9.x doesn’t FCS (First Customer Ship) until April ’12, meaning that they could possibly update the lab to UC platform version 8 anytime (with a standard 6-month pre-announcement of course), but to update to version 9 would mean that they couldn’t even announce a new lab based on that version until Oct ’12, with it going live around Apr ’13 at the earliest. I have no idea at all which they will end up doing, and from talking with Ben, he made it seem like they hadn’t reached an internal decision yet either. In fact it probably will largely depend on how quickly they get that virtualized hardware-like phone put into production in all reality.

Either way, it doesn’t really matter. INE has you covered. The major new things in CUCM ver 8.x that can be tested are the following:

Call Control Discovery over the new Services Advertisement Framework (CCD / SAF) with RSVP SIP Preconditions

They can’t really test the Intercompany Media Engine (IME) – it’s just not technically possible. EMCC is easy, and within the day of them announcing this being tested I’ll have a minimum 2-hour video ready. I already have over 4 hours recorded on every aspect of CCD over SAF and another half hour of the RSVP with SIP Preconditions in our new CCNP Voice product (which should be posted to our CDN in about a week). SIP Normalization with Lua — eewww — let’s hope they don’t test you on it – but either way, we’ll have a video for you the week they announce it.

There are way too many small new features in UC 9.x to list here, — I’ll do a post covering those in the not-too-distant future, but needless to say, it won’t be hard to add content to what we have to cover it. The best news is that everything you are studying today, is 100% completely relevant to any new version of the lab that could be announced, whenever they decide to do so — so don’t loose an moment of sleep over a possible upgrade — you’ll be 98% ready anyhow.

As for the possible testing of Cryptography/Security, I will be adding labs to our Volume II Workbook here in the upcoming month, with at least one of which will specifically address this topic. Our racks will allow you to test those features related to security to coincide with the release of that lab.

CCIE Data Center

Now, onto Data Center. Data Center was easily, hands-down, the single largest topic discussed at this year’s Cisco Live! event. Not to mention that VMware timed the announcement of version 5 of their suite of virtualization products (including vSphere, vCenter Site Recovery Manager, vSphere Storage Appliance, and vCenter Heartbeat Center) to be exactly 1 hour prior to (and thus ending with the beginning of) John Chamber’s Keynote address at Cisco Live!.

But here’s how everything relates to the CCIE program. It was very clear that the CCIE Storage is going to become the CCIE Data Center. In fact, aside from that being very clearly stated, the breakout this year was entitled: “Cisco Data Center/Storage Certification”. It was stated that the following would be what comprised the new CCIE DC.

We will consider adding DC solutions and technologies that can be deployed on the following Cisco Products:

MDS SAN Switches

Nexus 7000, 5000 and 2000

Cisco Unified Computing Systems (UCS)

Application Control Engine (ACE)

Global Site Selector (GSS) in case of DR scenario

Wide Area Application Services (WAAS)

Data Center Management for both LAN & SAN

Virtualization with Nexus 1000v

Of all those things that can be tested, the most obvious ones that would almost have to be included are the Nexus line of DC switches and the UCS blade servers, complete with Fabric Interconnects and Virtual Interface Cards. So those are the things that INE will begin immediately to record video-based lessons on, adding them to our All Access Pass. Guidance wasn’t provided on exactly when this track might go into live testing, though sources tell us that we may be less than 12 months away from it going live. Watch this blog for announcements soon on when you might expect the first of those Nexus and UCS training videos. Storage wasn’t big. Data Center is already huge. The CCIE Data Center is going to be as well.

UPDATE

I mentioned above two things that I forgot to include, so I will add them in here.

First off I mentioned that we will be adding graded mock labs to the Voice track. Please email me directly if you would be interested in participating in a graded mock lab. It would involve using a dedicated rack for 8.5 hours (8 hours for config and .5 for lunch, just like the real lab) with basic minimal access to the proctor (myself) for basic question clarification, but no real assistance (again, just like the real lab).

Second thing was the reasoning behind the removal of the CCIE stats page from cisco.com. This might sound like a strange reason – I thought so, but after listening to Yusuf talk a bit more about it, it did make good sense in the end. The reason was completely centered around the fact that when one updates his/her cisco.com testing profile with the proper home mailing address, and most importantly home country, and then takes and passes any CCIE exam, his/her CCIE number is forever associated with that country. The problem was, people didn’t always stay in that country. They sometimes moved, as is a reasonable assumption. However, the CCIE stats page wasn’t designed as a synchronous page that would do a real-time DB lookup each time it was loaded, and so it would always report X number of CCIEs in X country. Yusuf used himself as an example. He’s from Pakistan, and so his CCIE was basically ‘registered’ there (at least so far as that stats page went). Problem is, he moved to Australia for a number of years, but according to that stats page, his CCIE was still in Pakistan. Then he moved to Dubai in the UAE. CCIE? Still in Pakistan. So why should any of that matter? Well, to you and me – it might not. However, when a Cisco Partner in Pakistan (just continuing our example of Yusuf) is told by their Channels team: “You must have 4 CCIEs to become a Gold Partner” (or CCDE’s, they count now too for that metric), and maybe the parter reports back: “But there aren’t enough CCIEs in Pakistan to accomplish that!”, the Cisco Channels team would just pull up that stats page, point to it and show the Partner and say: “Yes there are, see here?” (Again, Pakistan is just an example country, I have no idea how many CCIE/CCDEs there are there, so please don’t think I’m being partial for/against that or any other country in any way – nothing is implied). It actually became a very, very big issue with Channels and Partner certifications. And Cisco is a large organization. If any of you have worked for one, you know that a team like the CCIE program has nothing to do with web page programming – that’s a completely different part of the business. And to get something changed there requires a requisition to be submitted, go through various levels of approval, and finally implementation. And believe it or not, it took about 4 months for that process to occur, and by the time the implementation of it was being carried out (the removal of that page off of cisco.com), it truly just happened to coincide with a period just following a CCIE R/S change that was resulting in less people passing the lab at that specific point in time. Yusuf stated that it couldn’t have been worse timing, however it truly was purely coincidental. He also mentioned that – yes, for a period of time after any type of a change to any lab track, there is always a fall-off in the number of CCIEs awarded for that track, but then it always picks back up. This is perfectly natural for any type of change for a number of reasons. 1) People stop booking a given lab en-masse right after a new version is announced – their basically afraid of what they don’t know (aren’t we all to some degree?). 2) If you sit a brand new lab version, and have no idea what to expect, you might be thrown for a bit of a loop, and therefore loose a bit of time you would expect to be productive during that lab attempt. By the way, this doesn’t have to be. Take a bootcamp course from veterans of the lab such as Brian & Brian (and possibly counting myself as a veteran at this point, I guess I’m getting up there in years , my first lab attempt was in ’02, so I’m going on 10 years next year …. wow — although Brian Dennis has his 15 year anniversary coming up in just a few months!), and anyway, you’ll be prepared no matter what they throw at you, and there will be no need for you to be counted in with the stats of people that ‘don’t know what to expect after a new version change’. Anyway, he finally mentioned that while Cisco doesn’t (and won’t) publish the exact statistics of CCIE Pass/Fail, if you look at the overall average number of passing scores over the life of any blueprint version, those numbers have always been, and will continue to keep trending upward.

———————-

Well, that’s about all I can think of at the moment. I just finished a long flight from Las Vegas to Minneapolis sitting next to Louie Anderson, and to be honest, I’m not sure how I got any writing done. That guy’s funny. I need to see him next time I hit the strip.

About Mark Snow, CCIE #14073:

Mark Snow has been actively working with data and traditional telephony as a Network Consulting Engineer since 1995, and has been working with Cisco Call Manager and voice-over technology since 1998. Mark has been actively teaching and developing content for the CCIE Voice track since 2005, and the Security track since 2007.
Mark's story with both data and voice technology started out quite young, as he began learning around the age of five from his father who was a patented inventor and a research scientist at AT&T Bell Laboratories. Mark started out on Unix System V and basic analog telephony, and went on from there to large data networking projects with technologies such as Banyan Vines, IPX and of course IP, and large phone systems such as Nortel 61c, Tadiran Coral, Avaya Definity and of course Cisco Unified Communications Manager in both enterprise and 911 PSAP environments across the US and internationally.
Mark is also an accomplished pilot and punched his ticket in 2001. When Mark isn't learning, labing, consulting or teaching, he can be found either piloting or possibly jumping out of a perfectly good airplane, hanging off a rock somewhere or else skiing out west. He also might just be enjoying a quiet day at the beach with his wife and two wonderful young kids, Ryleigh and Judah.

Hello Mark, this CCIE DC stuff is really exciting. All of my work for the past 12 months has been on deploying 7018s,5000,2000s and FEXs. What I am really hoping is for INE to do some training on the storage side of things as almost every tech meeting turns to a storage (FC or IP) discussion – it is such an integral part of the DC.

What about “Reason behind Cisco.com CCIE Statistics web page being removed”, you seem to have left it out.

I’m all for the protection of the Integrity of the CCIE Program with regards to the lab, and I welcome the upcoming changes (especially the removal of OEQs replaced by troubleshooting), but what about the written exams which are a joke in this regard, their integrity being 200% compromised?

All of the written exams across all tracks (CCIE and lower levels as well) are being very recently updated with real-time heuristic detection mechanisms looking for people to answer with the exact same patterns as the various cheat-type exams have (meaning, they have certain answers, that are not all correct answers, and if you match a specific pattern, then further examination is done on your test answers and results). So be assured that they are being addressed with vigor. But it is also the reason why there is a difficult practical (lab) exam for the expert level certification.

wow, i’m going to purchase ine’s ccna voice package. I have a voice package that seems to be ok, but i’m wanting to know more technical info about voice, and i know ine doesn’t hold back when it comes to technical. i’m on the r&s track right now, but i’m gonna take a few months off to study voice.

[...] There was some good stuff about the directions of the CCIE labs and even voice in particular, but I won’t speak to that when Mark Snow from http://www.INE.com has provided a very good breakdown of that very subject found here. [...]

Great blog and very informative, thanks! Regarding the graded mock labs, I would be definitely interested to participate and looking forward to it. Also is there any plan for creating solution guides for CCIE Voice DD Volume 1 labs?

Sounds great, I’ll unicast you about getting one setup. Solutions for the Volume 1 labs have been finished for quite some time now. Have you purchased this product and just aren’t seeing them in your members account? Assuming that’s the case, I’ll talk to some internal folks and see what the deal might be. Look for an email from me over the weekend.

We are considering it. Don’t know for sure at this time.
Might want to email cs (at) ine (dot) com for more info. Or keep watching this blog, as here is where we would most certainly announce support for that or any other new product/service.

Interresting news.
The origin country issue could be solved by allowing the CCIE to correct his country in Cisco’s database: Login, change country … done. As a protection against “fake CCIE hire from companies” they could limit this feature to once per year.

I believe the biggest issue for Cisco is dumps. After the last major change in R&S lab no one could pass the lab. Perhaps one per month (R&S) Then new dumps became available, since then the curve raised dramatically. I believe that 40 people passes each day now. (I’ve heard).

But the medicine Cisco applies is perfect, a lot of labs. Impossible to memorize.

CCIE’s can change their country, that wasn’t the problem. The problem was that Cisco originally had the web team build this very simple page to keep track of the overall number of IEs, which track they had received their IE in, and which country the IE was registered to in their profile when they first obtained their IE. The problem was, the web page logic never goes back and checks the updated DB to see if the IE has moved countries. Now, while you and I know this to be a fairly trivial task to program into the web page logic, and the CCIE program people do as well, they don’t have any access to the web page development team directly, nor the simple .js driven page source code. And getting that change approved – well it just isn’t a priority at all to them or the web team. So the reasoning was a very true one, no matter what anyone chooses to speculate about it. But I do understand your thinking, and it was mine as well until I spoke at length with Yusuf about it.

And I agree, their new medicine is by far the best step in the right direction I think they could have made.

The upcoming INE’s Voice Mock Lab is a great way for any candidate to check their level of uderstanding. Count me in as soon as it is available. I have already mailed you about this new feature.
Regards
Hristos

Thanks for your answer.
Funny how large organizations can make simple stuff complicated. I used to work for a large company, and it’s incredible how complicated work-flow gets ((c;
Let’s hope the medicine works, I recently meet a guy who just had failed the lab with 5 points. He asked me what a dot1q trunk was ((c;

It won’t really become a different track, it will be the same track – renamed. Much how the original “CCIE Communications and Services” became the “CCIE Service Provider”. Those who originally had earned the cert of CCIE Communications and Services simply became known as CCIE Service Provider if they chose to refer to themselves that way. At least that is the current plan, but it certainly could change. I just meant for lunch two days ago with one of the people on the Cisco team for determining the requirements for it, and they still haven’t even determined what hardware will for certain be on it yet or not.

Will certainly keep everyone posted as I find out more information on the topic!

[...] interesting technology coming in the future for the CCIE lab. Mark Snow hinted at some of it in his Cisco Live 2011 recap post. The plan going forward is to port all the lab workbooks to the CCIE Lab Delivery System that the [...]

i spent 3 years of my life, to find a way to be eligible to take my exam. it is unbelievable, but it is true. if you live in Iran, you don’t have any chance to be considered. so the mobile labs, facilitates the test process to everybody, every where, ecept, if you are live in the middle of nowhere, like Iran. anyway, i would like to thank u, the INE team for your great effort and support. even i don’t have access to your great materials, (just because, nobody sells the stuffs to Iranians at all and even we cannot use any credit card to order your materials), but I’m always dreaming a day, that everybody, every human, has the same rights in our small world. I’m also always dreaming to be there, and participate in your GREAT bootcamps too. God bless u.

[...] Mobile CCIE Voice Labs 0 Comments Posted by Mark Snow, CCIE #14073 in CCIE Voice TweetAs we first reported to you back in July, Cisco was planning on expanding their offering of Mobile CCIE Labs to the Voice track as well. [...]

Leave a Reply

Currently you have JavaScript disabled. In order to post comments, please make sure JavaScript and Cookies are enabled, and reload the page.Click here for instructions on how to enable JavaScript in your browser.