My Life as a Sys Admin

Category Archives: FreeSwitch

Ever since the entry of Docker, everyone is busy porting their applications to Docker Containers. Now with the tools like Mesos, CoreOS etc we can easily achieve scalability also. @Plivo we always dedicate ourselves to play around such new technologies. In my previous blog posts, i’ve explained how to containerize the Freeswitch, how to perform some basic load test using simple dialplans etc. My previous load tests required a bunch of basic Freeswitch servers to originate calls to flood the calls to the FreeSwitch container. So this time i’m going to use a simple method, which everyone can use even from their laptops.

Enter SIPp. SIPp is a free Open Source test tool / traffic generator for the SIP protocol. But the main issue for beginer like me is in generating a proper XML for SIPp that can match to my exact production scenarios. After googling, i came across a super simple ruby wrapper over SIPp called sippy_cup. SIPpy_cup is a simple ruby wrapper over SIPp. We just need to create a simple yaml file and sippy_cup parses this yml file and generates the XML equivalent which will be then used to generate calls. sippy_cup can also be used to generate only the XML file for SIPp.

Setting up sippy_cup is very simple. There are only two dependencies

1) ruby (2.1.2 recomended)
2) SIPp

Another important dependency is our local internet bandwidth. Flooding too many calls will definitely result in network bottlenecks, which i faced when i generated 1k calls from my laptop. Now let’s install SIPp.

Once we have installed SIPp and ruby, we can install sippy_cup via ruby gems.

gem install sippy_cup

Configuring sippy_cup

First we need to create yml file for our call flow. There is a good documentation available on the Readme on various options that can be used to create the yml to suit to our call flow. My call flow is pretty simple, i’ve a DialPlan in my Docker FS, which will play an mp3 file. So below is a simple yml config for this call flow

I tried to perform a large scale load test by making 1k calls with 250 concurrent calls. My local internet was flooding with network traffic as there was real Media packets coming from the servers, though it bottlenecked my internet, but still i was able to make 994 successfull calls. I suggest to do such heavy load test on machines wich has good network throughput. Below are the output for this test.

Docker is a very powerfull tool for managing Linux containers. In my previous blog i’ve explaind on how to setup a Docker Freeswitch. Docker is very mature now, version 1.0 has already been released. Docker is now supported by all major cloud vendors. Docker was showing promising results when i was performing my initial testing. So this time i decided to perform a heavy load test on the Freeswitch container to ensure that Docker can really enter Telephony. Like any normal sys admin, i was googling for Freeswitch load test, and most of the results were pointing to Sipp, an Open Source test tool / traffic generator for the SIP protocol. For me Sipp didnt helped me as it started throwing errors beyond 320 simultaneous calls. The UDP connections were timing out. I tried increasing the timeout, which didn’t helped much.

So next choice is to use a Freeswitch itself, to generate calls. Using the FreeSwitch’s originate command to generate simultaneous calls and hit the Docker Freeswitch container. I also decided to collect all system metrics, so that i knows how the machine behaves under various load tests conditions. For this i deciced to use CollectD and Graphite combo. Collectd 5+ has an inbuild graphite plugin which can send the collectd metrics to a graphite server.

I’ve already setup an Ubuntu-Freeswitch Docker image. First we need to pull the images from the Docker hub.

The privilieged option was enabled because, the FreeSwitch init script sets some custom ulimit values, so the container has to be given special privileges. Corresponding SIP and RTP ports are forwarded from the host to the container.

Now before starting the Freeswitch service, we can set up the CollectD agent. By default, the Ubuntu repostiry contains CollectD versio 4.10, but the Graphite plugin is available from version 5.0+ onwards. So we can use somne PPA which has the corresponding version available.

I’ve enabled a custom freeswitch plugin, which will extract the current ongoing calls count from freeswitch and sends it to the graphite server. Once the config changes are done we can restart the CollectD service. Now we can check our graphite UI to see if the default metrics like memory, load, cpu etc. are reaching the graphite server. Once CollectD-Graphite setup is ready, we can go ahead with our load test. So, once the call has reached the server, we need some Dialplan to continue the calls. So the simplest method is to create an infinite loop of playing some file, or some conference. Below are some dialplans that i’ve created in the public.xml

Now, we have the dialplans ready, next is authentication. By default there are two ways, Digest auth and IP Whitelist. Here i’m going to use IP whitelist, so we need to whitelist our IP in the acl.conf file.

<list name="domains" default="deny">
<!-- domain= is special it scans the domain from the directory to build the ACL -->
<node type="allow" domain="$${domain}"/>
<node type="allow" cidr="xxx.xxx.xxx.xxx/32"/> # IP of FS from which we are going to send the calls
<!-- use cidr= if you wish to allow ip ranges to this domains acl. -->
<!-- <node type="allow" cidr="192.168.0.0/24"/> -->
</list>

Now freeswitch is ready to accept the connection. We can start sending the calls from our Load test freeswitch. Below is the script that was used to originate the calls from the load test Freeswitch machine. This will create simultaneous calls towards the Docker FS.

While bulk calls are being made from the Load test freeswitch machines, to test the Quality in real time, it’s better to dial to the extension directly from a Sip Phone/Client and ensure that voice quality is good. Below is my Graphite dashboard for the load test.

The FS was stable till 500 simultaneous calls, after that there was a sudden drop in calls and also the voice quality started dropping and in a minute the Freeswitch crashed due to Segmentation fault. I’m going to analyze the core dump file to understand more about the crash. The other smaller drops that we see in the graph was caused by the Load test Freeswitch machine, as the load was getting high when the number of calls was increased. But 500 simultaneous calls are pretty decent and the there was no issue in voice quality till the number of calls crossed 500. Though it’s very difficult to make a final confirmation, i decided to go ahead with phase 2 load test.

In the phase 2 test, i’m planning to use multiple FS load test machines to generate large simultaneous calls + running 2 separate FS containers on the same host and split the incoming calls to both these containers. Once the phase 2 test is completed, ill share the test results in an another blog post. Docker is still under heavy development, and i’m sure Docker will be entering Telephony soon.

Docker has became one of the hottest topics in IT now a days. Docker is an open-source project that automates the deployment of applications inside software containers. Docker extends a common container format called Linux Containers (LXC), with a high-level API providing lightweight virtualization that runs processes in isolation.Docker uses LXC, cgroups, and the Linux kernel itself. Though i coudn’t make out to the DockerCon 2014 in SF, a lot new developments were announced on the DockerCon. Especially three new Opensource Projects libcontainer, libchan and libswarn. Docker is indeed creating a revolution in the container space, creating a next generation of scalable platform management. There are a lot PAAS services like Deis, resin.io, Dokku which are already using Docker in production. Another important and exciting project is CoreOS. CoreOS uses tools like SystemD, Fleet, EtcD to build a fully scalabale docker based cluster management system. I definitely need a separate blog to write about CoreOS, it’s really a super exciting project to play with.

Last week Docker Team released Version 1.0 of Docker. So i’ll be using the same in this new set up. It’s been almost 6 Month’s since i’ve been working @ Plivo as a DevOps Engineer. Telephony was really a very new platform for me. And my first companion was offcourse FreeSwitch,a scalable open source cross-platform telephony platform designed to route and interconnect popular communication protocols using audio, video, text or any other form of media. I was heavily using Vagrant for all my experiments in my mac. But after started using Docker, it really made me crazy. I’ve played for some time wiht LXC’s long back. So this was like a leap back to the container world.

There are a lot of concerns on using Virtual Machines in Telephony world. Especially for the server’s that handles the Real Time voice packets, as voice quality is pretty important in Telephony. Docker’s again more light weight isolated environment, and i decide to see how Docker can perform with such issues. If Docker handle Freeswitch smoothly, then i’m sure that we can use Docker for other telephony app’s like OpenSIPS/Kamailio etc, as they handle only sessions not the Media traffic. I know there are a lot of concerns like CPU load, Network etc, but this is like an initial move to test Docker into Telephony.

Now Docker is installed, but we need some OS images to use with docker. We can build custom images using debootstrap etc. But there are official minimal images available in Docker HUB. We can search for the repositories and can pull those images via docker binary itself.

For example to pull the entire Ubuntu images, we can just do,

$ docker pull ubuntu

But this will download all the ubuntu images available in the repo. We can also do selective download by using the tag.

$ docker pull ubuntu:14:04

Once the images are downloaded, we can use images option in docker binary to see all the downloaded images.

Here i’m not going to daemonize the container, i’ll be using the interactive option. But first, let’s start a new container.

$ docker run -t -i ubuntu:14.04 /bin/bash

This command will start a conatiner and will open up a bash session for us and we will be inside the bash session. Now to use an application we need to open up corresponding ports to outside world. We can use the “-p” option while starting a docker container to enable port forwarding. Under the hood, docker is using IPtables for the same. In the case of Freeswitch, we need to open 5060,5080 for the default Sofia profiles (Internal and External). Also we need to open the RTP ports. In this test i’ll be opening a predefined set of ports ie from “16384” to “16394”. (As my Docker host resides on Azure, creating an Endpoint for each port forward is really a pain, so i decided to open only a few). And also i’ll be opening port 22, so that we can have an ssh server inside the container.

Now we can go ahead with Freeswitch compilation. In my previous blog, i’ve mentioned how to compile and set up freeswitch. Once freeswitch is ready, we need to make a few changes. By default, Freeswitch uses STUN to route through NAT, but this doesn’t work with Docker. So we have to set the external IP manually. In the Freeswitch installed folder, edit conf/autoload_configs/switch.conf.xml. In this file we can set the External IP manually. Add the below lines to switch_conf.xml.

Also we need to modify the Default Sofia Profiles and need to set the ext-rtp-ip and ext-sip-ip to use our external IP added in the switch_conf.xml file while establishing connections. Add the below lines to the conf/sip_profiles/internal.xml and conf/sip_profiles/external.xml

Once the changes are made, we can start the FreeSwitch service. Now to make sure that the External IP is working properly, we can check the sofia profile status using fs_cli. below is a sample output of the sofia profile status.

Now freeswitch ahs started successfully. We can test some basic calls using softphones like Xlite, Telephone etc. By default, there are some default extensions and user’s available, so we can use the same for testing the calls. But i really wanted to try trunkning also and wanted to see the quality of the voice. So i created SIP trunking in Freeswitch using Plivo. And i tested a couple of calls to US and India DID’s and no issues were detected in the quality. But again i need to test the laod of the server’s when it startes handling concurrent calls and also the voice quality. But i decied to d oit as a Phase II. But as of now, Docker FreeSwitch is working perfectly like a physical machine with out issues.

So now we have a working FreeSwitch container, now here comes the main advantage of the Docker. We can create a new image with all these changes, so that nex time i dont need to work from scratch. I can use this saved image and a readymade Docker Freeswitch container can be launched in seconds. Since we are in interactive mode, we should not quit the session before it’s saved or else all the things will be lost,becoz dokcer will destroy the same. So open up a new shell on the docker host and use the commit option. But to use the commit command, we need to know the container id, so here docker ps command comes handy.

In my case “e7f3c02346d4” is the container ID. So i can use the same for commit. I won’t be commiting to the base Ubuntu image, as i can use the same for other purposes, so here i’ll commiting to a new image say “ubntu-fs-docker”

$ docker commit -m "<commit message>" e7f3c02346d4 ubntu-fs-docker

Now we can use this “ubntu-fs-docker” image to launch a ready made FreeSwitch server’s.

Docker is a very juvenile project about more than a year old. But the use cases are expanding heavily in the Modern IT world. Docker is fueling up a new generation of scalable servers. Wishing all the best for Docker and kudos to Solomon Hykes and the DotCloud team for opensourcing such a powerfull project

Kamailio aka OpenSER is one of the most powerfull and popular Open Source SIP server. It can be used as SIP Proxy/ Registrar/ LB/ Router etc. It also provides a lot of features like WebSocket support for WebRTC, ; SIMPLE instant messaging and presence with embedded XCAP server and MSRP relay,IMS extensions,ENUM and offcourse AAA (accounting, authentication and authorization) also. Kamailio is a modular system, ie, it has lot of modules which corresponds to particular functions. These modules can be easily installed and can be used easily in Kamailio. In this blog i’m going to use Kamailio as a proxy server. All the user’s are created in the Kamailio and FreeSwitch will be acting as a relay server for outbound calls. So Kamailio performs authentication and all the outbound calls wil be relayed to FreeSwitch.

Now we have the dependencies installed, we can go ahead with the build.

$ make cfg # generates config files for build system

Now open modules.lst and add the modules to be installed in ”include_modules” section. In my case i’m going to use MySQL backend so it will be ”include_modules= db_mysql” and then we can run the ”make all”. The other way is we can mention the modules directly while running the “make” rather editing the modules.lst file.

$ make include_modules="db_mysql" cfg

Now we can install,

$ make install

The above command will install Kamailio to our system. There are four main binaries for Kamailio,

Setting up Kamailio

All the Kamailio configurations are mentioned in only one single file /usr/local/etc/Kamailio/Kamailio.cfg. All the logics are defined in this file, and Kamailio blindly executes this logics and perform the actions. It’s very important that the logics defined in the config should suit to our VOIP platform requirement.

First we need to enable the modules and the necessary features, so add the below lines in the Kamailio.cfg

Now we can go ahead to the ”request_route” section which performs the routing logic. Here i’m going to add two more routing logic for the FreeSwitch relay. After the ”request_route” section, we can see the definition for each routing options. Below that we need to add our new route definitions.

By default, all the routes mentioned in the “request_route” will be executed line by line. There is a default route called ”Location”, which splits the user part from the request URI and verifies if the user exists in the location table. But when we dial an outside number/user, this location check will fail, so i’m going to add a condition which checks if the user in the request URI contains a number with a length 9-15 will be relayed to the FreeSwitch. Again this is just a simple condition, we can create a more complex condition, like check the domain part, if the domain part contains a domain which doesnot belong to our Domain list, we can either decline the request, or we can relay to FreeSwitch or we can make DNS query and we can make Kamailio to process the request to that domain’s Proxy server. Like this we can define our own conditions in the config file, and Kamailio will execute it line by line.

So now all the calls coming with numbers of length 9-15 in the Request URI will be relayed to the FreeSwitch, and FreeSwitch will process the call based on the DialPlan configured in the FreeSwitch. Since i’m going to use IP authentication, i need to whitelist the Kamailio ip in “acl_conf.xml” file in the FreeSwitch autload conf directory, so that FreeSwitch will accept the invites from Kamailio. Again i’m not defining any Voicemail options here. If we have a Voice mail server, then we can create another route option and when the caller doesn’t pick the call we can route the call to the Voice Mail server.
For example the below condition will route the failures to FreeSwitch Voice Mailbox.

Kamailio has a lot of modules which really comes in handy. For example we can use LDAP module to use LDAP as a backend. There is a PRESENCE module which helps us to setup an Instant Messaging server using Kamailio. I’ll be writing a blog on how to use Kamailio as an IM server soon. One of the main advantage of Kamailio over OpenSIP is the WebSocket support. This just a basic configuration, but we can design much complex system using Kamailio. We can even remove the default route logics, and we can use our own routing logics. Kamailio doesnot depend on default routing logics, it blindly reads the route and executes it for each incoming connections.