I strong recommend reading, or at least skimming:

I have known this attack is possible for some time—and have even performed this attack in consentual environments without an intermediate certificate.

These types of attacks are simple to implement in principle, but difficult to execute because root certificates are well controlled by root certificate authorities. This is changing, however, since well-known certificate authorities will provide signed intermediate certificates for a price—or—perhaps a writ from a judge. I will compile CAs providing this service on this blog entry as I find them. For now, this is a start:

The FriendlyArm mini2440 and Nokia N900

Both the FriendlyArm 2440 and Nokia N900 run armel-compatible CPU’s, but neither are very powerful in of themselves. Today I wrote a tutorial on setting up an armel distcc+ccache compile farm under the Qemu VM environment. Give it a swing if you need more compute power than your cute-lil’-device can provide.

Getting started

I have had quite the challenge recently using Digium’s multi-port T1 cards for link bonding. The plan is to have multiple links from multiple cards go to multiple locations and provide aggregate bandwidth and fault tolerence. This means I wish to unplug any wire from any bonded link and—except for reduced bandwidth from the missing link—the network should continue to operate as if nothing happened: telephone conversations must continue and open TCP sockets must stay open.

Prior Work

I found this guide in my search for multi-link PPP, however, it was written for the Zaptel device driver, which was renamed to DAHDI due trademark issues back in 2008. This means the Zaptel documentation is at least 2 years old and DAHDI has come a long way since then.

Still, this guide is nearly sufficient to provide redundant links. The challenge one might experience in using this Zaptel guide on modern DAHDI drivers is the lack of detailed documentation. Thus, the motivation for this article.

Sangoma, a competing T1/E1/J1 card manufacturer released a modified version of pppd to manage multi-link PPP under Linux in a more reliable way. From reading the code, it appears that Sangoma modified pppd such that it will exit if it looses its multilink bundle—and uses a wrapper script, pppmon, to restart the daemon upon failure.

This is not ideal, since we would like the pppd daemon to keep the pppX interface up even if the multilink bundle drops in order to keep routes in place, as the Linux kernel will drop all routes through an interface when that interface is removed. As it turns out, this is a much more difficult challenge than it sounds.

The Case of the Missing Route

There are two complete-failure scenarios (that is, multilink-bundle-failure complete) that we would like to seamlessly recover from:

One pppd daemon drops (perhaps the endpoint reboots), and the other side stays on.

The first does does not need LCP negotiation, and existing PPP state can continue upoon recovery. Simply using the pppd persist argument is sufficient here after removing Sangoma’s “exit-on-error” logic, however, this presents a challenge for the latter scenario:

If the side that remains available (ie, turned on) keeps its state, then the remote side will attempt to LCP handshake when it reboots (or re-launches pppd). Since the existing side is assuming existing state, it does gets confused by LCP request frames coming down the PPP link, and the link becomes inoperable.

Thinking, “well, why not just re-initialize the link whenever a multilink-bundle fails” I added new_phase(PHASE_INITIALIZE) at the point that the multipoint bundle is lost; this is nearly the same as re-executing pppd, but it keeps the associated pppX interface—and its associated routes—alive and kicking. This worked well when the remote-end reboots complpetely but, then, the first failure scenario of unplug/replug does not recover: The DAHDI pppd plugin attempts to re-initialize the master channel carrying the PPP link and throws “device or resource busy” errors.

The “Solution”

It turns out the easiest “fix” for this is to write a wrapper shell script around pppd with no-persist and no-fork. You can background the scripts and manage them in a hand-crafted way or perhaps a SysV script. I added the config to the end of /etc/rc.local using the “hub server” to assign IP address, allowing all of the clients to dynamically pick up IP addresses. This means any tech can plug a unit in and it will train up with the proper addressing no matter which port the T1 lines are plugged into. I also used the watchdog daemon to reboot the PPP router if it hasn’t gotten a ping response in X seconds, X*2 seconds after having booted.

This is perhaps not ideal since it drops routes when pppX is downed, but the watchdog will reboot the system if access is really lost.

Future Work

It would be great if someone were to patch the mainline pppd to support graceful recovery from the two failure scenarios listed above—without bring the pppX interface down and losing the network routes. Email me if you’re curious and I will point you in the right direction. After a few hours poking at the pppd code, this change may not be trivial since multi-link PPP apears to be a hack into pppd at the moment.

We’ve been using Linux Global to take care of our web server and network for several years now. In short, they are great. Always quick to respond, and always fix the problem. They are super easy to deal with and to understand- and very fairly priced. With Linux Global on board, our network and server worries are way down. If something happens, they will be right on it. Very highly recommended.

Pat FranzTerraCycle, Inc.

Eric is very knowledgable in his area of expertise. He is fast with communication and very easy to work with.

Matt Prados, FounderGotcha Local

Working with Global Linux Security has been a true pleasure. When I joined MedXCom, I had a huge task in stabilizing and fixing our software while migrating it all into the cloud. Looking back over the last year and a half, I can’t see how we could have ended up with a better solution without his help.

Jason Berry, Lead DeveloperMedXCom

I know you are always on top of things, and I trust you. I know I am being taken care of and I know you always do the best you can under the different circumstances you are confronted with. You always follow through, and follow-up. You communicate well. Your prices and charges are fair. You work quickly.

Joe Crestuk, President/CEO
webSURGE digital marketing

Our experience with your service was excellent. I appreciated the quick, professional response. The Linux world can be challenging but your expertise made the project seem simple!

Mark Woodbury, IT ManagerSlidematic Precision Components

We have been completely satisfied with the level of service, expertise, and professionalism that we receive from you on an ongoing basis. I sleep better at night knowing that our servers and network are constantly updated, secured, and monitored. We are still a small company but we are growing fast and we look forward to expanding our relationship with you as we continue to grow.

Bob DuncanBlindster.com

We have always received great service from Eric and his team when we have issues to deal with.

Sande CaplinSande Caplin & Associates

I have used Eric Wheeler and his team at Global Linux Security multiple times. There is a reason for this: if you want the highest level of professional skill and a take charge attitude, Global Linux Security is the company for you. We’ve truly found that Eric and his company just get it done––once you give them a problem to tackle they solve it and they solve it quickly.

Wayne D. McFarland, Managing PartnerVexillum LLC

Eric and his team are top-notch Linux experts. We had a server compromised with malware a few years ago which Eric was able to migrate to a new server, patch and mitigate in fairly short order.

Joe Crestuk, President/CEOwebSURGE digital marketing

Eric is extremely knowledgable and efficient. His understanding of servers and security is top notch.

Eric SteenstraCommerce Strategies

You have been able to recover logs and to help us ferret out hacking issues which have been a continuing problem for us, especially since we are in the elections industry. I feel very confident in the security of our web servers as you have protected them with certificates and monitoring.

Jon Winchester
Logicworks Systems, Inc.

I requested help to configure an email server and Eric did an excellent job of setting that up for me.

Charlie KreiderAviation Weather

The experience with your company has been extraordinary; the response time has been quick and everything is resolved timely and professionally. Every solution has included an explanation which has been very helpful. Another point is on your website you make the comment “There is not a problem we cannot resolve”. Well in my case this has been accurate.