Blog Actions

Looks like some time on August 6, web.de changed something in the STARTTLS configuration of their mail system (possibly enabling it for the first time, as I don't see TLS info in headers of older mails), which was leading to SSL errors on my system, and mails were not being delivered:

It was a 5000 series card, one of the first affordable "real" IDE RAID controllers (unlike the other cheap ones that did most of the heavy lifting in the driver software). I last used it in the single PCI slot of an Intel Atom board, with two 1TB SATA drives connected via IDE-SATA bridge boards - a setup that worked, but with several drawbacks (like no S.M.A.R.T support, which was added only in later 3ware products):

I've currently replaced it with a pure software solution - using the mirroring feature of Linux's new btrfs filesystem on two SATA disks connected directly to the Atom board. Remains to be seen if that works as reliable as the old setup.

With Windows Server 2008, Microsoft has introduced a Read-Only Domain Controller (RODC) role that adds an additional layer of security to protect an Active Directory infrastructure from member systems that are located in an not completely trusted environment (say, a DMZ or perimeter network). Configured correctly, using an RODC can prevent unauthorized changes to the ADS from a compromised member.

Let's assume we have done all that, and end up with an RODC living happily in his small firewalled DMZ, waiting for other AD systems to use his services. We move a server from the main network to the same DMZ, assuming everything will just work. And then it doesn't:

Active Directory has the nifty concept of AD sites, which allows clients and member servers to find appropriate Domain Controllers that are assigned to a specific location, using DNS. As detailed in the Planning and Deployment Guide, an RODC will be placed in it's own site, with appropriate AD site links to allow access to specific writable DCs (and the appropriate firewall rules to permit just that communication).
A member server moved to the RODC DMZ will not be able to contact any of the Domain Controllers it knows of from it's old site though, which causes the usual site selection mechanism to fail. With no knowledge of the site membership, our server is unable to find an assigned RODC, and all AD services will fail.

So we need a way to bootstrap site selection without AD, and lo - Microsoft has provided a way to do just that: There's a Registry key to tell an AD member which site it is in. Unfortunately the information is somewhat hidden in a subsection of the Deploying RODCs in the Perimeter Network document...

With that information, a DMZ system can create the appropriate DNS requests to find the DCs for it's site, and most things will be ok.
One of the things that needs additional work is DNS registration, though: As an Read-Only DC doesn't have the permission to update DNS entries through Active Directory, systems in an RODC site will need to send DNS updates to a writable Domain Controller. Microsoft has documented that part in DNS updates for clients that are located in an RODC site.

1. Intro

Moving the external interface of a Juniper SA 4500 SSL-VPN appliance to a different IP subnet is not an entirely illogical procedure, but due to the many interdependencies between configuration elements, the exact workflow is somewhat non-obvious... It's also not mentioned in the official system documentation. In this case, I was working on an SA 4500 active/passive - cluster running R7.1 sofware.

The main problem is that all IP addresses on the external interface have to be from the same subnet - and it's not possible to define additional VLANs with other address ranges on the external side. It's also not possible to move any part of the external interface configuration to a new subnet as long as there's still configuration from the old address range somewhere on the system.

2. Useful preparation steps

Sort of a migration plan: A mapping from old to new external IP addresses on the system, and preparation for the required DNS changes. You'll need Netmask and Default Gateway of your external port, external IP addresses from all cluster members, the external cluster VIP, and any addresses used on external virtual ports.

As long as you are only using SSL transport (and not IPSEC), having a NAT gateway in front of the SA is also a great help - just do destination NAT from old to new addresses, and DNS changes can be postponed to any time later on without further interruption of service.

Configuration -> Certificates -> Device Certificates has a list of SSL certs for all hostnames on the appliance. Each Certificate is bound to one or more Internal or External Virtual Ports. Remove all Selected Virtual Ports from any Certificate that's bound to an External Virtual Port.

Now that all External Virtual Ports are gone, it's possible to disable the External Interface entirely (in fact, it can be disabled any time, but then the Virtual Ports are locked and can't be deleted): Go to Network -> External Port -> Settings, make shure to select Settings For: Entire Cluster and change Use Port? to Disabled. When navigating away from the page, note that the UI likes to jump to the configuration of the active cluster member, and Settings For: Entire Cluster has to be reselected from the Network Settings menu.

On the disabled port, all IP configuration can be removed - just delete Netmask and Default Gateway for the Cluster.

That's it for the SSL-VPN configuration - now you just need to make shure your destination NAT is in place (or that all DNS entries for addresses on your gateway have been changed to the new addresses).

Jloader will in fact not be installed after this error. JunOS has to be upgraded first (10.4R8.5 in my case). It will still boot with the old boot loader (just dual-root support won't work).
After a reboot, the jloader package has to be downloaded and installed a second time with the usual request system software add .... With JunOS 10.4 as a base, the "upgrade address wrong" message will disappear. Reboot again, then remove the jloader installer with request system software delete jloader-ex-2200 to get rid of the "At least one package installed on this device has limited support" - warning. Reboot a third time.

This needs quite some time on EX2200 (CPU on these boxes is really, really slow) - maybe 45 minutes all in all - but works without further problems in the end.

It's implicitly in the apt-get manpage, but I didn't see it until now - downgrading Ubuntu/Debian packages (for example to remove packages from an Ubuntu-PPA or a manually installed .deb with a newer version): Just add /distribution-name to the package when reinstalling.

So I have this rather old Celeron-300 laptop that I install a new version of Ubuntu on from time to time to see if it still works.

It's had the problem that DPI gets set to a wrong value (20) and consequently I get a really tiny font for quite some time, and after the old way of forcing --dpi on the X comand line doesn't seem to work anymore, it was finally time to find out why.

Turns out the newly created /etc/X11/xorg.conf contains garbage in the form of a wrong DisplaySize and PanelSize:

Turns out netdot does a lot of useful things via SNMP::Info(cache), from pulling inventory data to reading out VLAN, IP, ARP and forwarding tables, and then puts all that in a neat web interface for further classification. It's possible to document just about everything around network systems in netdot, down to the cabling setup.
On the logical side, netdot can be used for IP address and VLAN management, and to create configurations files for RANCID, ISC bind and dhcpd, and Nagios.

All in all it seems like a pretty nifty tool, and I've only scratched the surface of it's functionality until now - just having the MAC database (and by that the information where each end system is attached) makes setting up netdot worthwhile.

2. basic setup

My installation is based on netdot 0.9.10 on a Debian Squeeze LAMP system.
Setup is quite straightforward, just follwing the available documentation: Dowload, unpack, run make installdeps-apt-get to install all missing packages, run make installdeps to fetch anything that might still be missing from CPAN.
Then copy etc/Default.conf to etc/Site.conf and adapt all settings for your installation, especially those for the database connection. A make installdb will take care of setting up the DB. (I'm not very fond of the way that's done, as the routine needs database admin credentials and fails if you create a database in advance with a user that just has admin rights on that.)
After that, make install copies everything to /usr/local/netdot. I've used checkinstall to create a basic debian package:

As a last step, netdot needs to be added to the apache configuration, by copying a template: cp /usr/local/netdot/etc/netdot_apache2_local.conf /etc/apache2/conf.d/

3. installation woes

At this point, everything works in principle, including logging in to the web interface. Unfortunately, that's also where the problems start...

3.1. Can't add any system to the database

updatedevice.html produced the following error:
Device::_get_snmp_session: Cannot connect to <Device>. Tried communities: public

Looking at network traffic with tcpdump, no snmp queries go out at all... After tinkering with SNMP::Info internals for two hours, found the unexpected solution in the netdot bugtracker, bug #1622(cache):
Two debian packages are missing in the installdeps phase: apt-get install snmp-mibs-downloader smistrip

3.2. Alcatel MIBs missing

netdot contains a collection of MIB files that's probably derived from the netdisco-mibs package. When trying to add Alcatel OmniSwitch systems on our network, SNMP::Info barfs - it has an AlcatelLucent.pm to handle those systems, but that doesn't work without the MIBs. In the end I found a relatively complete set of OmniSwitch MIBs in the Observium SVN(cache) (because, you know, those files are so secret that you can't download them from the Alcatel-Lucent web site - I'll be so happy when we're rid of the last of these things).
To make things somewhat more fun, Alcatel provides a customized AaIETF_HUBMIB_POWER_ETHERNET_DRAFT.mib that uses the same module name as the official POWER-ETHERNET-MIB. SNMP::Info::Layer3::AlcatelLucent documentation advises to change it's module name to ALU-POWER-ETHERNET-MIB by editing the file. Which is fine, except that the corresponding error also comes up when there's another problem (like a dependant mib still missing).

3.3. Juniper EX non-obviously not supported

Adding our Juniper EX 4200 and EX 4500 switches to the database seemed to work fine at first, until it turned out that just the data for the system that was added last is in the database.
I did not have a look at netdot's data model, but obviously the problem was caused by IP addresses used on internal interfaces that are identical on all EX systems (128.0.0.x). Fortunately, Site.conf provides a configuration directive to weed out these pseudo interfaces:

Next I noticed that the VLAN information gathered from Juniper switches was broken: Instead of the VLAN ID, the database just contained sequence numbers.
Obviously EX switches use a different method to provide VLAN information via SNMP than Juniper routers. Someone has already provided an JuniperEX.pm extension to SNMP::Info, but it's not integrated into the distribution yet.
I ended up reinstalling a current version of SNMP::Info directly from CPAN, and then just copy the JuniperEX.pm file into /usr/local/share/perl/5.10.1/SNMP/Info/Layer3/ (path on my system) and patch SNMP/Info.pm:

4. unsolved problem

After all that, my netdot installation happily crunches data from about 90 switches in our central location, and I'll most probably add our external sites in the coming weeks.
One last problem remains - SNMP::Info doesn't read inventory information from Juniper EX switches, so I'll have a look at exending the existing JuniperEX.pm. More news when that actually works.

At last, a thanks to the developers - got a detailed description on where to look when I posted my various problems to the netdot-users mailing list.

Feels somewhat different from last time over (25C3, has it really been three years?):

Less crowded. Maybe just because it's day one, but most things run smoothly (exception: trying to get a r0ket, I think 50% of the attendands were in that queue). No probelm to find place with a free power socket.

Less laptops in the lectures. The desks in Saal1 have been removed, and I have the impression that the same happened to lots of wall sockets. Ok, people switched to tablets and smartphones, but even that seems at a reduced level.

I'm not going to write summaries of speeches, but I noticed that the topic of (OSI) layer boundary violations came up twice - once in the session on PHY attacks on wireless devices, and when Dan Kaminsky mentioned something in the line of "applications built on top of TCP assume reliability" (and that assumption can be broken). It seems there's some surprises still waiting in that direction.

Cory Doctorow was good for a host of quotes, but I was hit by his description of networked activism, and why it seems so disparate without being desolate, which went something like this: Earlier on, the main work was actually organizing the activist movement, and so people agreed on common goals first and then went on organizing around that ("2% figuring out what to do, 98% stuffing envelopes"). Nowadays, organization comes mostly for free, and it's easy to form up around a topic, start discussing things, and maybe move off in different directions without losing touch. Which is exactly what confuses people about the pirate party in germany, for example (or Anonymous, or Occupy). Now that maybe oversimplified, but it rings true in principle.

I've now seen several installations of an Checkpoint R70 management GUI where the details pane in SmartView Monitor doesn't work because of Javascript errors like "'statSection' is undefined" or similar. The URL usually is "file:///C:/Program%20Files/CheckPoint/SmartConsole/R70.40/PROGRAM/data/htdocs/overview.html?", as in the following screenshot (german version):

I've found several references to this problem on the Checkpoint support forums and elsewhere, but none with a conclusive solution.

Obviously, this error occurs because SmartView Monitor can't find several files in PROGRAM/data/htdocs.

I have a somewhat convoluted workaround for this:

There are two .tar files in PROGRAM/data which contain the required files, namely SmartViewMonitor.tar and SmartViewTracker.tar.

For some reason it's impossible to untar those two files directly in the data folder, even as Administrator - I have the impression that a htdocs directory already exists, but is invisible or has very strange access restrictions (have found no way to make it visible).

So copy the two files to somewhere else, and untar them using either an archiver like 7-Zip, or with the gtar.exe that's included in PROGRAM/util with any SmartDashboard installation.

Move the resulting htdocs folder back to PROGRAM/data within the Checkpoint GUI directory. You'll need administrative privileges, and force overwriting any existing files.