This is the White Rhino Security blog, an IT technical blog about configs and topics related to the Network and Security Engineer working with Cisco, Brocade, Check Point, and Palo Alto and Sonicwall. I hope this blog serves you well. -- May The Lord bless you and keep you. May He shine His face upon you, and bring you peace.

Saturday, January 31, 2015

Friday, January 30, 2015

Thank you to all the Telcos out there that make me keep you honest. One of my customers told me that they came into work this morning to find that they could not make voice calls and could not receive calls. So I get onsite and I can see that the PRI module has an alarm on it. I pull out my trusty T1 loopback and I get green (meaning it loops up) on the csu/dsu module. Green is good. So I go back to the AT&T smartjack where I find no power on the HTU-R that is inside the case. Ok, that is a problem for sure.
Well, the responsible carrier says that they can not get to smartjack. I tell them they are correct, because it doesnt have power on it. They say they will call AT&T. AT&T tells them everything looks good. Good? No power is good? I tell the responsible carrier that its not good. We still cant call out and make calls. Still no power on the smartjack. How can things be good? The responsible carrier says Im right, that they cant reach the smartjack. I tell them that is because there is NO POWER ON THE SMARTJACK! Responsible carrier says they will have to contact AT&T again.
You kind of get the idea on how this is going. Anyway, eventually, they fix it.

Thursday, January 29, 2015

Why are Voice carriers always able to get by telling you whatever they want, without ever having to prove anything to you??? Doesn't anyone think that is just not right? Well, its ok. I can prove things for them. So I have a DID number that I cant call into. I mean, the customer internal phone doesn't ring anymore when you call its external DID number. A little frustrating for the customer for sure. We looked into the Cisco CUCM and the translation pattern looks ok. Lets debug on the voice gateway, to make sure the call is getting that far. I mean, it has to come into the gateway to get to the customer CUCM, right?
So I get on the Cisco router (the voice gateway), and I do a "debug isdn q931". That turns on debugging for incoming/outgoing calls on the PRI. Lets make a test call into one we know works. Ive changed the numbers for customer privacy.

Wednesday, January 28, 2015

Ok, have you ever had to deal with another technical 'professional' that drove you to this point below? This is me, below, after dealing with someone who didn't understand that in order for packets to get to your internal web server (from the Internet), that they have to hit the firewall first (the one I configured). And if the packets are not hitting the firewall, they aren't going to get to the web server!!! So, logically, Miss Techie Support, I don't need your tech guy to come onsite to troubleshoot! I need you to send your data to the right public address!
Packet captures are the network guy's best friend. Keep your friends close, and your packet captures closer. On the Cisco ASA, see this link for how to do a packet capture in CLI.

Tuesday, January 27, 2015

I really hate the Check Point doesn't document this well, at least as far as I have seen. Check Point says that when you go from one major version to another (like R76 to R77), they say that you have to use the version of migrate export tools of the version you are going to. That is mostly correct. I have seen this also be the case when going from R77.10 to R77.20, which is not a major version change. In fact, I got myself in trouble one night when I relied on the documentation, only to be told by Check Point TAC that I had to use the R77.20 upgrade export tools instead. This meant, for me, re-installing R77.10 and re-importing again just to get back to my starting point. Then go through the process. So keep that in mind. But, how, exactly, do you get the right version?
There are two ways to do this. First, you can download the migration tools from the Check Point site, based on the version you are going to. If you are going to R77.20, then you have to download those tools (and since their website change recently, its anything but fast on the front end). I downloaded "Check_Point_migration_tools_R77.20_B020.Gaia_SecurePlatform_Linux.gz" for my upgrade export. You have to get this file on the management station, gtar it, and then run THIS "migrate export" from the upgrade_tools folder (in /var partition) instead of the one of the current version install. This is how you get the right version that you will need when going to do an upgrade. Its probably good just to do this anytime you do an upgrade.
The second way, which I have not done, is to download the upgrade_tools folder from another management station to the box you are trying to get an export from. This seems like too much trouble to me, but do what you like.
Hope this is helpful, as I have not really seen this documented in Check Point.

Monday, January 26, 2015

I have the privilege of having Justin Jocewicz as a guest poster on the Network Fun!!! blog today. He brings some very nice experience to us all today, and I want to thank him for posting. Very nice job Justin! ~~Shane Killen

Cisco ASA Firewall
Cluster Member Replacement

So one of your firewalls in your highly available cluster
died. It happens. It’s not your fault. But, you have to put humpty dumpty back
together again. Do it the wrong way, and
you can erase your configuration and bring the cluster down!

Sunday, January 25, 2015

Have you ever taken the time to read Jude? In the Bible, Jude is toward the end. I find it interesting to me, that Jude was going to write about salvation. Instead, he wrote a warning to us. But, what was he going to say about salvation? I just think that would have been an interesting topic from Jude. What would you say about salvation to others? Or, a better question, what DO you say?
Its one chapter. Its a good read.

Friday, January 23, 2015

What I meant to say was, what comes first in a router when a packet hits, PBR or the routing table? So think of it this way: If you want to manipulate traffic, like setting the next hop, can the routing table come first? The answer is no, since it will take PBR to make the change BEFORE decision is made to look at the routing table. Just FYI. That will be the case no matter what vendor equipment you are configuring. Check Point, Cisco, Brocade, Palo Alto etc. Its all the same on this one.

Thursday, January 22, 2015

There have been plenty of times, as a consultant, where I go into a customer to help with an environment, only to think to myself that some sales guy really oversold to the customer. It really aggravates me to some extent. Its just not necessary. Yes, have a five year growth plan. Yes, expect some unexpected growth. But dont sell the top of the line equipment when in reality, the customer only needs something middle of the road. Switching gear is where I see this the most.
Properly size the gear with the current network needs, plus the five year growth plan, plus some extra for unexpected growth. Not the twenty year growth plan. Be honest and fair to your customers.

Tuesday, January 20, 2015

I have a customer that I had to go onsite and prove that the firewall was not blocking SIP traffic. The phone guy there was saying that the firewall was blocking the SIP traffic going out to the carrier. Now, I know it wasn't the firewall that was the problem. After all, I configured it. But, as you know, the network guy has to prove everything. So, I went onsite and did a packet capture. The internal IP address of 10.11.1.250 is actually hitting the inside interface of the ASA. Here is how I know:

Now, I want to see the traffic being sent out the ASA on the outside interface. I mean, I need to know that the traffic is actually getting THROUGH the firewall, right? Looks like the NAT'ed IP address of 40.40.40.250 is making it out to the destination of IP 50.50.50.38:5060. Lets see below:

Monday, January 19, 2015

I think this is important for the network guy to really get good at. Its a solid, proven technique. I recommend this to anyone troubleshooting intermittent ISP issues. I realize that I just wrote about this Friday, but this is an important topic. Its just too common.
Use two different methods of ICMP for testing. As I mentioned Friday, I use ping plotter pro and a DOS prompt for ICMP. I NEVER rely on just one method when troubleshooting issues like this. You will have some false positives, but you will learn which ones are false and which ones are not over time. If they both fail (ping plotter pro and ping through DOS prompt), then you got what you need. Capture several time outs to prove it THEM, not you.
Here is another example I just came across. Customer complained about dropped calls between sites, across the VPN.

Here is a different customer that I did some troubleshooting for. Same thing, different ISP. Again, a solid, proven technique.

Sunday, January 18, 2015

There is an interesting book called The Story. It's selections of the Bible in chronological order, without the Bible chapter/verse numbers. I find that I can't stop reading it. It's like a novel, or at least written like one. I suggest it to you if you don't like the chapter and verse numbers in the Word. This isn't a substitute for reading the Bible, but a companion. I highly encourage you to take a look at it.

Saturday, January 17, 2015

I have added a "About Me - Career Story" page to this blog. If you are interested in reading it, click on the link across the top or Click Here. I hope that this is an encouragement to young IT professionals.

Friday, January 16, 2015

How do you do it? You know something isnt right, but you can still get on the Internet. Everything works ok, but sometimes it gets slow. Sometimes you get a "page not found". But if you refresh, it seems to work. What is going on???
Well, here is a pretty reliable way to find unreliable ISP connectivity. Get yourself a program that will do a graph of ICMP. I use ping plotter pro. I also use DOS with it. You will see what I mean below. Start a ping to a remote site IP address like google or norton or somewhere (not the URL). Also, get your graphing program (ping plotter pro in my case) and ping the same IP within that program also. Then, watch them both closely.

Now keep in mind this, one sporatic "time out" is really not enough for reliably saying that there is a problem.

Now in the below, the speed test starts dropping out when a consecutive "time out" occurs.

Anyway, if you suspect problems with your Internet connectivity, try this above. Its pretty reliable in finding these sorts of issues. I have even found VPN drops this same way. That turned out to be dirty fiber, which I found just like this above.

Thursday, January 15, 2015

I have been getting prepared to do an upgrade of a Check Point management station to R77.20. One of the things I am getting ready to do is to move the log files over to an external hard drive, so that we can import them back in and no logs are lost. Ill do the migrate import in after I do a fresh install of R77.20 on the new management station. Then Ill import in the logs so that no logs are lost.
One problem I have run into is when I connect the 2T byte drive in the USB connection, I can mount the drive just fine, and read it contents. However, I can not write to it. On read access is available. This drive is formatted via NTFS.
After a while of trying to figure this out, it turns out that I have to have some other install package (Fuse, I think) to make it work. Well, I dont want to do that on my Check Point management station, or any enforcement module for that matter. So, I formatted my 2T drive to be exFAT instead. This seems to have worked for me. I can now write to this drive.
Here are some things that I ran into that you might recognize if you are running into this issue:
One of the error messages I got when trying to write to the external drive:cp: cannot create regular file `/usb-storage/2014-10-01_235900.logptr': Read-only file system

3. Let the stack build. Verify with "show stack" command.
Then do the following commands:config thitless-failover enablestack mac XXXX.XXXX.XXXX <--- (mac of the primary switch when you do a show stack)exitwr mem

Tuesday, January 13, 2015

Sometimes you just need to see more that what SmartTracker will give you. Well, I do. To effectively troubleshoot, you need to know where things are failing in the firewall, IF its failing there. I've written about packet captures on the Check Point before, but if you can do this in CLI in any firewall, its preferable. In this case, I needed to troubleshoot if a packet was actually making it through or not. Sure, I could see this in Tracker, but I wanted to see the whole picture. See the packets from 192.168.50.10 destined to 172.16.2.59? Also, in Check Point, you can tell what stage you are in by the lettering (in orange). Refer to this post for more info on that.

Monday, January 12, 2015

I have, on occasion, had to unconfigure a Brocade ICX out of a stack before. Sometimes, you just have to re-purpose a switch. Here is the command that you will want to run, on the ICX, when you have disconnected the stacking cables off the back (of an ICX 6610)(front of an ICX 6450).
6610#stack unconfigure clean
This command is not available on standalone or Active Controller
6610#stack unconfigure me
This unit will recover pre-stacking startup config. Are you sure? (enter 'y' or 'n'): y
remove stack bootup flash
6610#

Friday, January 9, 2015

There are times when you just need to monitor multiple ports at the same time. One example is when you have and active/active firewall setup, and you need to monitor the traffic going to both internal interfaces. Can you do this on the Brocade ICX switches? Yes, you can. Here is a piece of the config as an example:
...mirror-port ethernet 1/1/37
!
interface ethernet 1/1/46mon ethe 1/1/37 both
!
interface ethernet 1/1/47mon ethe 1/1/37 both

Thursday, January 8, 2015

I'm an IT consultant, and I do go around from customer to customer on a daily basis. Sometimes, I see multiple customers in one day. I guess you can say that in the IT services business, you kind of get around.
One of the things I think is interesting is the titles that people get. Anything from Network Analyst to VP of IT. But have you ever wondered how a lot of times, IT titles don't necessarily match the skillset of the guy holding that title? I think this is just misleading.
For example, I met a guy once in the education sector that had a title of Network Analyst. But if the truth was told, he couldn't tell you what a switch really did. He could tell you it connects the network together, and I guess that is true. But he couldn't tell you what a CAM table is. He couldn't tell you what ARP is. And I'm sure he couldn't configure a port on the switch to be an access port. Would you call that guy a Network Analyst? I wouldn't. I would really consider this guy a PC tech, based on what I know.
Now I'm not knocking the guy. He has experience that is valuable to the company, but not network experience. His title really should have been something like IT technician I, PC technician, or something like that. I mean, you set the guy up for failure one day when he applies for a real network position at another company when he really doesn't have any real world experience in network related things.
I also know people who are very talented in specific areas, like security. But they fall under the network group and therefore, have a network title. How is that fair or accurate? I mean, if you have a guy whose focus is security in the company, you would expect he might have a title like Security Engineer, Security Analyst, etc. This could affect his money making, meaning he should be making more.
On the flip side, I actually worked for a guy who was a VP of IT. His background was a cable puller. I think he might have touched a few servers at one time, but he was absolutely worthless as a "VP of IT". He made terrible decisions that affected the way IT worked in that area. He would get the opinions of the real technical people, only to ignore them and make a decision on his own or based on pressure from upper management, who also had no clue. In fact, not one person had any respect for this guy. I certainly did not. (I still laugh at this one.)
I suppose you can find this all over the place. I see it all the time. I even know people who have an Architect title, but rarely do they do any architecture of a network.
I think companies need to really consider the titles of the employees they have. Its not like HR has any real clue as to what a real title should be for someone with particular skillset. Its really the IT manager who should give this consideration. I mean, what does HR know about different areas of IT?
Its just interesting how you can go into companies and find the high powered titles, when in reality, their skillset doesn't match. It sometimes leave you, after a first impression of a technical guy, with questions like "Shouldn't you know this?" Anyway, I just find this interesting. Maybe all fields are like this, but I know for sure IT is like this.

Wednesday, January 7, 2015

Im pretty sure I have posted about this before, but one thing I had to do recently was to check some fiber. Being a network guy and not a cable guy, I really dont have any good electronic tools to check fiber runs. But, what I do have is a cell phone with a camera on it. I personally never look into a GBIC or a fiber where a laser could be coming out. I know there are arguments that say its ok, but I just like to NOT do it. My cell phone is much more forgiving than my eyes. So I use my cell phone to check for the laser. You can see below that I do have a laser coming out of the fiber run in this case (circled). Anyway, glad that was not the problem in this case.

Tuesday, January 6, 2015

If you have access into the Check Point, and for whatever reason, you need to figure out who you are logged in as (which has happened to me once before), then you can always run the following command in Expert mode:

Monday, January 5, 2015

Sometimes, what looks like one problem, can actually be another. I got a call from a school system saying that their network was down for one particular school. No Internet. No server access. Nothing.
So when I got there, things seemed to be ok. They could get to the Internet, servers, etc. Everything appeared to be ok. Then it went down again. This problem was very inconsistent. You couldnt even ping anything. But then, after some time, it would be fine.
Well, when I was there, I really couldn't find anything wrong. Except once, when the problem actually happened. It appeared to me, at that moment, like the layer 3 core was acting up. When the problem happened, I couldnt even ping the next hop MPLS router (on the same subnet). This problem showed some really odd symptoms. So, I broke the switch stack (Brocade ICX 6610s) and left the last switch in place to run as the core by itself. The problem seemed to go away. Until the next day, when the problem happened again.
I showed up and guess what. The problem was gone again. So I sit down and just start looking at configs, spanning tree, interface statistics, cpu utilization, etc. What in the world is going on here. Then it happened right in front of me. CPU shot up to 33% (from 1%). Internet was down and network was hosed again. So I ran a packet capture. Nothing on the first 4 VLANs that I looked at. Then I found it. The 5th VLAN I looked at was flooded with broadcasts.
6610#sho cpu-utilization
32 percent busy, from 9 sec ago
1 sec avg: 32 percent busy
5 sec avg: 32 percent busy
60 sec avg: 32 percent busy
300 sec avg: 32 percent busy

So, I disconnected the fiber to the closet that VLAN went to, and CPU dropped back down to 1% on the core. Ok, leave that disconnected, and go to that closet.
I got to that closet and plugged in my packet analysis tool (seen above). Again, flooded when I hit the right VLAN, but no other VLAN. So, what do I do? Its a stack of 2 ICX6450s. I start unplugging one port at a time until I find the flooding settles down. It just so happens the second port I unplugged calmed the network down right away. It was a PC that was connected at the other end. It appears the NIC was flaking out. Intermittent broadcast storms. Not a good situation, but with the right tools, I found the problem as it was happening. Packet captures are your friend!

Sunday, January 4, 2015

I read an article about reading each book of the Bible twenty times in a row. That's right. x20! I'm going to start with a small book first, like Colossians. Read it twenty times in a row, before I read any other book of the Bible. Let's see what results that will produce.

Saturday, January 3, 2015

Friday, January 2, 2015

Just a little job thought for us all. I came across this pro bono thing where I really wanted to help this guy out. He had a worthy cause and it really bothered me that I really just didnt have anything to offer this guy to help out his cause. I feel like I have a pretty decent IT skillset, none of which he needed.
I did, however, know a guy who could help him out. TRUTH: Its ok to not be able to help someone with your skillset. Sometimes in life, it is enough just to know someone else who can help.