The World of JNCIE Exam Development - An Inside Look
Jul 5, 2013
Something I have always struggled with in the networking world is the meaning of the word ‘smart’. What defines ‘smart’ in our industry – is it vast knowledge gained through experience, proven knowledge through certification, some mix of the two, or something else entirely? However it is measured, you can be sure that ‘smart’ is a relative thing; it seems like any time you start getting that ‘smart’ feeling is about the same time someone or something comes along to put you back in your place.
Such was the case at a recent JNCIE exam development workshop. For all of our exam workshops we strive to bring together a strong and diverse group of subject matter experts – SMEs – to provide broad expertise in the given topic area; for lab exam workshops, the stakes are even higher to get that mix of folks just right. So it was quite something to look around the workshop table that week and take note of the talent level in the room… there were folks with significant TAC experience, others with significant service provider design experience, one took time off his consulting gig with a major SP in Europe to fly in, while two others were the creators of the original JNCIE-ER Enterprise lab exam. All told, the folks in the room represented over 20 JNCIE credentials, including four 3xJNCIEs. As a Program Manager for Juniper’s exam workshops, it’s a terrific feeling to be able to bring together a strong group of SMEs and feel confident that the outcome will be an equally strong exam. But as a member of the networking world, it’s a very good reminder of just how much ‘smart’ is a relative thing. Being a JNCIE and yet the least-smart person in the room is definitely a lesson in humility.
The process of creating a lab exam is a remarkable thing. It starts with a form of job task analysis, which essentially asks the question, “What tasks do engineers in this job role need to be able to perform?” Watching this process, it immediately becomes clear why getting that right mix of folks in the room is so important, as they literally help define the scope of the exam. With the tasks collected and categorized into sections, a blueprint was created – a roadmap from which to build the entire exam. Then it was on to designing a network topology, and this is where fun started for these guys. The group was split into teams and tasked with designing a topology to serve as the basis for the entire exam. Whiteboard it and then pitch it to the group – best design wins. Nearly as fun as the brainstorming for them was the fun for me in seeing what they came up with. And in the end, all agreed that a single team’s design wouldn’t cut it – there were just too many good ideas that would be left behind. So we pulled the best from each design to create a single topology that met all the needs of the exam, while also providing a good challenge for exam candidates.
With a blueprint and topology in place, it was time to get down to writing the tasks for the exam. This is the part that represents the actual ‘exam’ as test-takers know it – the actual step-by-step instructions on what to build, what to ensure is working, what you may do, what you may not do … all the elements that make up the full day that is a JNCIE lab exam. It was also the toughest part of the week for the SMEs. Creating a strong exam task in isolation is a reasonable challenge; creating a strong exam task that takes into account the overall exam, whether the task will break some other task, and whether that is a good or bad thing for the exam, is an entirely different endeavor. Nevertheless, the SMEs’ experience shone through and they worked through the process well, coming up with some really terrific scenarios and challenges.
With the tasks written, it was on to tech-review to ensure all tasks were in line for correctness, relevance, and appropriateness of difficulty. As a function of running through all the tasks, we also got our first look at how the exam was coming together. Similar to watching a house being built from pre-constructed components, it was very neat to watch the exam start to take shape as each layer was added, and to get a sense of how the exam would feel to candidates, as a single entity. We were fortunate that the exam came together very well. A tweak here and there and we were done – voila, a JNCIE exam! Well, not quite.
There is still a long road ahead. One of our Tech Leads (a 3xJNCIE) here on the JNCP Cert team has the job of taking the raw elements from the workshop and shaping them into finished elements. From incorporating the exam tasks into a single cohesive package, to physically building the topology in our lab environment, to creating grading guides and scripts, there is much work still to be done. But work is progressing well, and when it’s all ready to go we will have an updated exam we can all be proud of.
For my part, I’m into studies towards a second JNCIE. Anyone who has ventured down this path knows very well how much lab exam prep involves discovering at least as much about what you don’t know as what you do. Between that and being in rooms full of top-level SMEs, one thing I do know is this: whatever ‘smart’ is, it’s a ways off yet.