VDX Troubleshooting Course

The material available also feels very short, same as the beta material available for the CEF300 , like only the parts of the slides that were updated for the BCEFP 2015 beta were included.
When a slide says “(cont.)” but there was no previous slides on this topic, that’s a hint :)
Take the (currently free) course on Brocade’s SABA – it’s under Education on my.brocade.com. It has way more slides and info.

The NOS 4.1.1 Admin Guide

I’ve been reading the pages on paper (together with a highlighter :) that I printed with the help of my script below and there is lots of goodness in there.
For sure some topics are brought up without any preamble so for these I just make a note in the paper that I need to check out this other thing later.
Especially the Fibre Channel things take up quite a lot of pages. I thought in these devices FC would not be with so much focus but it seems like they do re-use a lot of the things in FC that works.

Notes and acronyms (page in NOS Admin Guide):

DCB – lossless. Able to allocate bandwidth on links.

TRILL – transparent interconnections of lots of links.

RBridge – Routing Bridge. Lowest WWN or priority.

Looks like on p54 only the text about Logical Chassis cluster config is applicable.

Trunking between VDX8770 and B8000 are not supported (B8000 is some early version of FCoE from Brocade, not visible on Brocade’s page where they list their switches)

lacp system-priority 25000 # For deciding which system is in charge of resolving LAG conflicts. (p437)

nas server-ip IP/PREFIX # Set IPs for AutoQoS for NAS (p506)

address-family ipv4 unicast # Used to enter IPv4 config in a VRF (p609)

debug lacp pdu # turn on debug (p714)

terminal monitor # view debug messages in terminal

Printing the NOS Admin Guide relevant pages:

Because the slides for the BCEFP course were insufficient I would get a lot of the basic information about the NOS from the NOS Admin Guide.
In the materials provided the NOS Admin Guide was separated into two documents. The guide is of course available in one pdf. Go to the web version and click on the pdf icon.
This makes printing based on the numbers provided easier. However the NOS Admin Guide for v4.1.1 referenced was one version below the one on the html version.

Now the numbers referenced are the numbers in the document, not the one told by the pdf viewer. So actually page 11 is page 13. Page 135 is 137. 311 is 313. 425 is 427. 517 is 519. 661 is 663. 714 is 716.
I checked a few to make sure there were no major increase due to version difference or elsewhere. One could with a bit of scripting increase each number with two like:

In a previous post I listed a some of the sources Brocade listed that one should use when studying for the BCEFP exam. Here I’m going through a those I found some comments on what what they are and what I think of them.

Beta Course Material

The first of the beta material available is something called “Brocade Ethernet Fabric Administration“. This is a few pdfs/slides with notes on them. Introduction of various features and components. Not much detail in the first 10 modules and basically all the modules are awfully short, some are one slide even. Hopefully this is just because it’s a beta. Progressively they become more detailed, which is good to not overwhelm the reader I guess. Checking out the data sheet for the CEF 300 course should give you some idea what you should learn after going through the materials. There are free materials available for the Ethernet Fabric Specialist Accreditation– it’s even on the tube. The youtube video is quite long but it’s an introduction to the thought behind the Ethernet Fabrics. It’s a bit outdated already I hope as they the talk talks about immaturity a lot, less than a year old. The presenter – Chip Copper – also mentions a Fabric Essentials 201 that should be out “later on down the line” – which is not out yet. Boo Urns!

Questions I got while reading material:

What is a hard-drop option in an extended ACL?

What does “override the control packet trap entries” mean? Brocade communities to the rescue. Is for normal transit traffic and traffic to the CPU == the management interface?

After ~49 posts on this blog on the topic Brocade the first larger block is finally complete: the Brocade Certified Professional Data Center Track (BCPDC)!

What’s that? So Brocade has several (4) tracks which consist of certifications/accrediations, some are shared between the tracks and some are only in one track.
Currently, after completing 3 out of 4 you gets the title Brocade Distinguished Architect! Woop!

It took me ~3.5 years (counting since first blog post about BCFA (certified fabric administrator)) to complete all the prerequisites for BCPDC, but naturally I didn’t do it as fast as I could. I was patient and many of the certificates I got by being signing up for Brocade’s beta tests of their certs.

Not that many certificates left to take actually before I can complete another track.
Most of the remaining ones are labeled accreditations, which are unproctored tests one does at home.

For Brocade Certified Professional Converged Networking (BCPCN) I have 3 accrediations left (Fabric Specialist, FCoE Specialist and Ethernet Fabric Support Specialist) and 1 certification: Ethernet Fabric Professional 2013. The certification I have signed up for the free one I mentioned in an earlier blog post.

Either register on Brocade’s site and get the download that way. Or get it via HP’s public page – for example here. Click on Download.
Because I’m lazy I’m installing it in a Windows 7 x64 VM 2 cores and 4GB RAM is much faster than 2GB. For just installing it you’ll need 3-4GB disk space.
Find install.exe within na1214_hp_windows.zip

The default user/password is: Administrator/password
The user/password you set during installation is for the database.

Spoke a bit with some people in the Brocade stand at HP Discover in Barcelona. The open source / core http://www.vyatta.org/download will be kept, but could not get any commitment to what will happen to it or if they will update it. The 5400 (VSE, 6.6) and 5600 (VR, 7.x) are however available for a free 60-day trial. With the 5600 having a new architecture and is interfacing more closely with the hardware (using Intel’s DPDK to for example dedicate processes to cores) that improves performance “quite a bit”.

// Update, I since found out about VyOS: http://vyos.net/wiki/Main_Page which is a community fork of the Vyatta Core OS

For training these I set up networks. Many.
Drawing the networks first in LibreOffice Draw and then setting them up with virtual machine templates and LAN segments.

The exam I took in October and because it was a beta exam the results aren’t out until December :)

The BCvRP has the below objectives (included for free are some of my comments on each topic).
None of this should be taken as a replacement for taking the actual course and actually doing these things on a vrouter.
And honestly, the various concepts and technologies described in the objectives below can become very complex. So before taking this course/exam you at a minimum want to know the basics of BGP and setting up an OSPF network should be a breeze.

OSPF Multi-Area Concepts

Describe OSPF routing concepts

Stub area – replace external routes with a default route

NSSA – not so stubby – can have a local external route inside a stub area

no-summary : exclude inter-area routes

LSA – link state advertisements

1 All OSPFs: Lists subnets/links directly connected, does not cross area boundaries

2 from DR: Lists routers connected to a network, does not cross

3 from ABR: Lists networks from outside the local area

4 from ASBR: Summary, lists location of ASBR

5 from ASBR: AS external, list networks outside OSPF AS. 7 for NSSA.

Summarization: Good to have continuous addresses in an area, easier to summarize.

exact match must be in the router’s table: create a static route to blackhole on the router

iBGP = same AS on the BGP peer (the neighbor)

iBGP – a full mesh is necessary. iBGP does not forward routes learned from other iBGP peers.
One can use “next-hop-self” so that iBGP router’s change the next-hop address to a network whenever it propagates the route.
update-source – this needs to be the same as the router-id.

bgp does not reset advertised routes after an administrator’s changes.
Changes to eBGP does not come into affect until you run the reset:
‘reset ip bgp external out‘. The BGP table can be large – gigabytes.
Use the word soft to only request updates and not reset the peer connection.

reset ip bgp external [ipv4 address]

Tuning attributes and priority

Local preference – only included within an AS. Default is 100. Higher is better.

AS Path – always forwarded – shorter is better

Origin – lowest

Multi-exit discriminator # modified by an ISP to indicate preference

eBGP preferred over iBGP

Lowest Peer ID

Community # group of prefixes with a common property. Can be used in filters.

Prepending: insert your AS number in the AS in the beginning of the AS path.
Communities are created with: set policy community list

BGP troubleshooting

An active peer – not good. Trying to actively set up a session.

iBGP design

Does not have to be physically connected (as in BGP).

Connectivity over BGP

Peer to loopback address

Full mesh is required

Doesn’t scale. You can use a Route reflector (“concentrator”) and have other iBGP routers as clients.

route reflectors must be meshed

You can also create multiple private AS within your AS. Reduces members in the mesh. Called a confederation.

Public AS number is only visible in the config

The Private numbers are visible in the show ip bgp commands.

Create a peer group, set BGP settings on the peer group. Then assign peers to the group.

This post will be continuously updated with my short notes under each concept.
It’s not meant to be a replacement of the official training materials.
I’m just starting out playing with the vRouter Core / open source version and installing it in a VM and set up some networks and firewalls is probably one of the best way to learn this.
Learn by doing!

The Brocade Certified vRouter Engineer 2013 exam has these objectives:

Brocade Vyatta vRouter System Operations

Describe show command system usage

show – in operational mode shows status of components

show – in configurational mode shows the configurations

run show – in configurational mode shows status of components

Identify key CLI operations

set/delete

copy (configs)

renew (new dhcp IP)

install (to disk)

Describe the commit and save processes

Ethernet Concepts

Identify Ethernet operations

Identify VLAN operations and settings

set interface ethernet eth0 vif <vlanid> # this creates eth0.<vlanid> a subinterface. This looks like a normal ethernet interface.

set interface pseudo-ethernet # these can be used if you want to set the MAC-address. Some features are not allowed for these peth devices though (VLAN, bonding).

Identify bonded interface operations

Two NICs on the same network

set interface bonding (IP address, mode)

set interface ethernet (bond-group)

Demonstrate knowledge of configuration and operation using show commands

TCP/IP

Demonstrate knowledge of the relationship between Layer 2, IP and TCP/IP

I tried installing Vyatta vRouter 6.6 amd64 Live ISO to disk first in a Virtualbox VDI file and then uploading said file to openstack. This works, but:

Ethernet interfaces might get renamed but a startup, log in and save, poweroff and another boot should get the first interface back to eth0.

In the openstack available to me I could set up my own networking topology like this:

Create one network (VLAN) and define several subnets inside (these are still kind of firewalled based on IP and MACs).

Then create machines and add the network.

Power off and start the machines again (or the links stay DOWN).

VMs should see an individual eth interface per subnet.
The machines still get an IP assigned to each interface/subnet even if DHCP is disabled. If DHCP is disabled you still have to statically assign only this assigned address on the interface.
The interfaces are in order: the IP listed at the top is the IP you need to put on the first interface (eth0).

Because a lot of the things you can do with a router involves creating networks and assigning IP addresses, which openstack would block for security reasons – it was much easier to do all of these in VMWare Workstation:

DHCP/DNS

Install a Vyatta VM – bridged and a private network (without a DHCP).

Install another OS in a VM – this will be a client – only on the private network.

OSPF!

Adding host V4 that is in Network B and Network C.
Basically Vyatta, V2 and V4 are routers.
V1 and V3 do not run OSPF, they have their default gateway to one of their local routers.
So V3 has 10.2.2.20 and V1 has 10.1.1.1.

Public (192.168.1.0/24, bridged):

Vyatta – eth0: 192.168.0.23

Network A (10.1.1.0/24, vmnet2):

Vyatta – eth1: 10.1.1.1

V1 – eth0: 10.1.1.10

V2 – eth1: 10.1.1.20

Network B: (10.2.2.0/24, vmnet3)

V2 – eth2: 10.2.2.20

V3 – eth0: 10.2.2.30

V4 – eth0: 10.2.2.40

Network C: (10.3.3.0/24, vmnet4)

V4 – eth1: 10.3.3.40

Remove all static routes we did previously on Vyatta and V[1-2,4]:

delete protocols static route
commit
save
show proto

Set up OSPF – define the networks on each router that that router share with another router:

ALL: set loopback interface IP to something unique and with a /32
ALL: set protocols ospf redistribute connected
V4: set protocols ospf area 0 10.2.2.0/24
V2: set protocols ospf area 0 10.2.2.0/24
V2: set protocols ospf area 0 10.1.1.0/24
Vyatta: set protocols ospf area 0 10.1.1.0/24
V3: set system gateway 10.2.2.20
V1: set system gateway 10.1.1.1

Brocade has a beta exam up for BCVRE – Certified vRouter Engineer – which is on the Vyatta software from the company with the same name that Brocade bought last year.

There is the free open source core. Download from here: http://vyatta.org/downloads (no you don’t have to register). The evaluation/subscriber version has the API and web gui available, I’ll probably check those out closer to the exam date.

I grabbed VC6.6 – Virtualization ISO. Use it in a VM and assign 5GB disk (install only requires 1G, or you could just run it on the iso, but then it doesn’t keep state between reboots) and 1GB RAM. Two NICs: One NAT and one private. But to get more acquainted with it you’ll likely have to do a bit more configuration on the hypervisor side. Such as turn off dhcpd in your virtual networks.

To install it to disk: hit “install system” at the CLI after it’s booted.

The server is basically Debian with a more recent kernel (6.6 has 3.3) and a shell to make it more switch-like. It actually uses the bash completion to make it look like this. Check out /etc/bash_completion.d/vyatta-*

To remove a setting use “delete” (comparable to no in other CLIs). There is a web interface, but this is only for subscribers. Core version allows SNMP though if you want to use that :)

But no SDN stuff (separate data and the control plane). It looks like it’s not possible to modify the flow table of a switch via Vyatta. This looks like a software router/VPN/firewall with some extras added to it.

Using the “configDefault –all” does not clear everything, for example it doesn’t clear: system name, zoning, etc.

Setting the switch to AG mode (Access Gateway) – will clear more things as it basically dumbs down the switch, it does not remove the licenses, IP and password.

ag --modeenable
ag --modedisable

The ‘ag –modedisable’ (puts switch back in normal switch mode) command sets the default zones access to No Access, so if you want to merge this switch into a fabric you’ll most likely need to change that and disable/enable the E_Ports.

In early April of 2013 Brocade had a great offer – ask for it and you’ll get a voucher to an exam – for free!

I took them up on their offer and scored a voucher for the BCNE – Brocade Certified Network Engineer.

After that I noticed that Brocade also has a limited offer for BCNE http://www.brocade.com/education/CNE_250.page , you can take them up on it if you already have a CCNA. By doing that you also get a free voucher to the BCNE exam..

I chose to try it without the recommended course. A bit risky but a long time ago I took the CCNA and passed. For me this exam was probably more about remembering and looking at improvements to all the things in CCNA back in 2005. This post is about my study technique or perhaps more of a record of how I did things. To find places for improvement.

Brocade IP Primer – this is a great refresher on most Ethernet things if you’ve been out of touch.

Go through the manuals – but read the material in the newer released manuals.

IP Quick Reference – CLI Quick and quite comprehensive overview not only of commands but also of technologies.

http://community.brocade.com/docs/DOC-2613 has the list of pages and manuals and guides, but to get the newest documents you have to look elsewhere.
One place to get them is on each Product’s page on brocade.com, at the bottom there is a place to get some manuals.

First thing I did before diving into the materials was to take the BCNE Knowledge Assessment test. Get some sort of idea of what kind of topic the exam is about.

Then I read the nutshell guide and marked the things I needed to learn more about (basically all). Last time I took an exam with Brocade I only read the nutshell in the beginning of my study time, this time I’m re-reading it every now and then to see if I catch something that is not clear and I want to focus extra on. I’m also keeping a focus on the objectives of the exam. Reading the objectives and trying to answer them with as much detail as I can.-The objectives are general so there’s quite a lot of room for freedom there. As a bonus, if you can’t describe something in the objectives well, you just found something you do not know well enough.

After going through the nutshell guide and checking up on a few acronyms and technologies I hadn’t heard about I read through the IP Primer and did the same things there: Mark the things that I thought would be of interest and what I would need to dig deeper into.

Then went through the NetIron and FastIron configuration guides. Not only did I have a peak at all the pages that were listed as relevant, but also read chapters that was not listed. Either because I found them interesting or perhaps because the subject in those chapters are touched upon in Nutshell. To me that just means the more you know about the subject the better.

Rehash objectives/previous notes and dig deeper. Perhaps first time you read it you glanced over some part. By digging deeper I mean finding the chapters in all the manuals that touch on this subject and reading them, making more notes. Could also be surfing the Internets or Wikipedia for basic overview of how a technology operates. Eventually all of this crystallizes into a view that describes things in your own words.

To me there are parts of IT exams that you just can’t know even if you’ve been working with it for a long time. For example license options or feature differences between all the products. To learn things like these (also other types of questions I thought would come on the exam) I made flashcards in a spreadsheet and printed it on normal A4 so that the question is on one side and the answer is on the back. This was no easy feat.

After going through all these documents you should be able to figure out yourself which areas are being focused on – which you should be making sure that you know.

Thursday today, the day after the Day. It was a real long day, and to my surprise it said ‘press’ on my pass – so I had to try to ask some questions :)

Some things picked up:

* New VDX 8770 product released– a modular Ethernet switch. Room for 384 10GbE ports. 100GbE ready and also ready for SDN protocols like VXLAN (vmware) and NVGRE (windows 2012). The VDX 8770 chassi is called “Mercury” internally in Brocade. I found it very similar to the DCX chassis except that the supervisor modules are half-height.

* Today Brocade opened up registrations for the BCEFP certification – Brocade Certified Ethernet Fabric Professional (which include the VDX8770), It looks advanced and you probably want to take the previous exam – BCEFE – before.

* SDN – storage-defined network was the main focus of the day. Fibre Channel was barely mentioned at all.Ken Cheng‘s (one of the VPs of Brocade) definition of SDN:

“A set of technologies which are focused on achieving three objectives: network virtualization (vxlan), programmatic control (openflow) and cloud orchestration (openstack).”

It was quite obvious that Brocade’s VCS is the technique/medium which they intend to enable these new technologies. SDN is still quite immature (even though internet2 are already using it in their production network) – so be prepared to wait if you want ready solutions.

* VCS seems quite similar to QLogic’s/Juniper’s QFabric. They had a hands-on lab where we could connect four smaller vdx switches and a vdx8770 (4-slot version). The switches had only had a unique ID set on them and their were end-devices (web-servers, web cams and a tablet) on different IP subnets on each switch. All I needed to do to connect switches (and devices) was to connect two switches via a fibre pair. Quite easy. Almost too easy to be true. This is something I really enjoy that’s part of Fibre Channel. The technology has quite a few features, self-forming trunks being one of them (with frames being striped over all members of a trunk). It also gets rid of spanning tree (so no more unused links).

* Quite soon we should see Brocade’s OEMs release embedded VDX switches for their blade chassis. No news yet about which but lately IBM have been quick to release new Brocade products. As a side note: Brocade from start only sold their gear through OEMs, this is no longer always the case and they are trying to communicate more directly with customers.

* Cost per bit was really important to push down for internet exchanges.

* It’s a lot easier to write a blog post on my wordpress blog via Chrome (on android) than via the native browser. Using my asus transformer tf101 as a note taking device for the day worked out great. Success!

This accrediation has the most questions of all the current ones, but it has the same amount of time allotted (one hour) so this exam will have a lot less time available for each question.

The curriculum for this accreditation are also free, they are called Introduction to HBA and Introduction to CNA. There’s also some docs about the 1860 Fibre Adapter. They can be found on Brocade’s Saba/training website under my.brocade.com.

// Update 20140422: This accrediation has been replaced with something else. See the current list here: http://www.brocade.com/education/certification-accreditation/index.page?

The only part I was initially not entirely sure about is the “Given a scenario, describe when portlog dumps are required”. The objectives indicate that a Fibre Channel theory knowledge is necessary, so the FC-101 course seems like a very good idea to study. I doubt many people remember specific FC mechanisms/theory if they don’t work with these occasionally. Like the well-known addresses – who remembers the address of the name-server or controller? =)

My general tip for the BADCS: Learn the material of the FC-101 course. Really. Learn. it.

You may be tricked into thinking that Brocade’s accrediations are easy because you can do them from home.

Say what??

These are all names for the basic idea / functionality but as there’s no standard the vendors have made up their own names for it.

A switch in Access Gateway (AG) mode does not consume Domain IDs, you can do port mapping, needs NPIV on the port in the switch that it connects to. AG requires a switch / fabric to connect to as it doesn’t run the normal fibre channel services.

It is very useful in case you are going to mix vendors in your fabric. Meaning you can populate the core with Brocade switches and then connect other vendors’ switches in the above modes to the Brocade switches.

On some QLogic switches you can also set a port into TR-mode, see this post on HP’s EBC forum about how to do it. It is not exactly the same as AG or NPV, because you still need to do zoning on the QLogic switch.

There is also the IPM by Qlogic for IBM – it looks like a module that you cannot switch between ‘fabric’ and ‘IPM’ mode. Which is what you can do on a Cisco or on a Brocade switch.

HP just published an advisory describing how to tune some parameters for Emulex, Qlogic and Brocade Fibre Channel HBAs: c02518189. It sounds like these are new, but these changes have been around for at least 6 months in all three vendors’ HBAs.

Emulex

“Emulex driver version 2.42.002 or later, along with OneCommand Manager version 5.1.53.2 or later,”