Jekyll2018-02-12T14:20:54+00:00http://blog.simonmetzger.de/blog.simonmetzger.denetworking, automation, virtualization and moreNetwork-Automation with Salt, NAPALM and Kubernetes2018-02-06T00:00:00+00:002018-02-06T00:00:00+00:00http://blog.simonmetzger.de/2018/02/salt-napalm-k8s-network-automation<p>Since the native integration of <a href="https://github.com/napalm-automation/napalm">NAPALM</a> into the <a href="https://saltstack.com/salt-open-source/">Salt</a> core which was officially done since Salt Carbon (2016.11) by <a href="https://mirceaulinic.net/aboutme/">Mircea Ulinic</a> it is possible to manage network-devices directly with Salt. I thought about how to scale this out and how to manage a lot of (hundreds or even thousands) network-devices with this solution. To be more clear my goal was to manage legacy devices that are not able to install software natively on themselves. In this case the salt-minion client can’t be used. The job has to be done by proxy minions instead. Per salt-managed network-device a salt-proxy process which reserves about 60 MB of memory is required. The following image shows the high-level logic of this construct.</p>
<p><img src="/images/salt-minion-proxy.png" alt="salt-minion-proxy" title="salt-minion-proxy" /></p>
<p>If we think about possible solutions for an environment running this there are different options:</p>
<ol>
<li>One very big salt-master server with a huge amount of memory/CPU and all of the salt-proxies running locally on it</li>
<li>One salt-master and a lot of additional instances (e.g. VMs) where the salt-proxies are running</li>
<li>Multiple salt-masters which could be for example in different locations so that the salt-proxies can communicate with the master next to them</li>
<li>One salt-master and one container which only runs the salt-proxy process per managed network-device</li>
<li>Same approach like the fourth but with multiple salt-masters</li>
</ol>
<p>In the following sections I describe how I tried to implement the fourth possibility.</p>
<h3 id="the-components">The components</h3>
<p>Because I wanted the solution to be very scalable and I like containers a lot for me it was a good idea to continue with a containerized solution. But managing hundreds or even thousands of containers is no easy task.</p>
<p>This was when <a href="https://kubernetes.io/">Kubernetes</a> came to my mind. Some days ago I was just reading the great book <a href="https://www.amazon.de/dp/B072TS9ZQZ/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1">The Kubernetes Book</a> which brought the key concepts of Kubernetes nearer to me. Kubernetes is great for managing a huge number of containers and provides a lot of very cool stuff to do this in a best manner. If you look at the features of it you can find super helpful things like: automatic binpacking, self-healing, horizontal-scaling, service discovery and load balancing, automated rollouts and rollbacks, secret and configuration management, storage orchestration, batch execution. Wow! - especially the self-healing and automated rollouts/rollbacks is what I looked for.</p>
<p>So I decided to spin up a test-environment with all of these components: One salt-master and a Kubernetes-Cluster running one container per salt-managed network-device. The platform I used for testing was just my local Mac OS installation and some cool software on it: <a href="https://www.vagrantup.com/">Vagrant</a> with an ubuntu/xenial64 virtual-machine acting as salt-master, <a href="https://github.com/kubernetes/minikube">minikube</a> as a local Kubernetes cluster and <a href="https://www.docker.com/docker-mac">Docker for Mac</a> with <a href="https://github.com/bitnami/minideb">minideb</a> to build a small container image which can be used by the salt-proxy containers.</p>
<p>The picture shows these pieces connected together to build the full solution. Not every single detail is contained in the picture, so please consider it as a high-level overview. In the next sections I will describe the setup of the different tools and what to take account of when configuring them for our use-case. In one of the last sections I will show some working examples.</p>
<p><img src="/images/high-level-network.png" alt="high-level-network" title="high-level-network" /></p>
<p>If you look at the routing table of the minikube-VM you will see something like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ ip route
default via 192.168.64.1 dev eth0 proto dhcp src 192.168.64.2 metric 1024
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.64.0/24 dev eth0 proto kernel scope link src 192.168.64.2
192.168.64.1 dev eth0 proto dhcp scope link src 192.168.64.2 metric 1024
</code></pre>
</div>
<p>Each Pod (which is the smallest piece in Kubernetes and contains the container) will get its own IP and has a default-route to 172.17.0.1. Communications to the outside will be source-NATTed with 192.168.64.2 (that’s the ip of the Kubernetes node running the Pods). This is done inside the minikube-VM with <a href="https://docs.docker.com/engine/userguide/networking/#links">iptables</a>. If traffic coming from the bridge100 interface, where the minikube-VM is connected, wants to communicate with outside networks it also gets source-NATTed to the dhcp-allocated IP of the Macbook. There are NAT-rules in place on Mac OS which were automatically built when installing minikube:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>nat on en0 inet from 192.168.64.0/24 to any -&gt; (en0:0)
no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24
</code></pre>
</div>
<p>For communication to the salt-master only the two required TCP-ports are forwarded from the Macbook (not limited to one IP) to the Vagrant-VM (TCP 4505/4506).</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ vagrant port
The forwarded ports for the machine are listed below. Please note that
these values may differ from values configured in the Vagrantfile if the
provider supports automatic port collision detection and resolution.
22 (guest) =&gt; 2222 (host)
4505 (guest) =&gt; 4505 (host)
4506 (guest) =&gt; 4506 (host)
</code></pre>
</div>
<p>Thats how the traffic gets to my “Macbook” and from there communications with the salt-master, device1 and vice versa are made possible.</p>
<h3 id="set-up-the-salt-master-vm-with-vagrant">Set up the salt-master VM with Vagrant</h3>
<p>We need a server where the salt-master will run. In our testing environment <strong>Vagrant</strong> will be used to setup an ubuntu/xenial server. Follow the instructions on the <a href="https://www.vagrantup.com/">Vagrant-Homepage</a> to install it on your operating system. Also install a vagrant provider, which is mostly <a href="https://www.virtualbox.org/">virtualbox</a>. I recommend you to follow the install-instructions on the links and not to install the software with package-managers of your operating system.</p>
<p>After installing Vagrant and virtualbox now create a new folder called <code class="highlighter-rouge">salt-master</code> and create the <code class="highlighter-rouge">Vagrantfile</code>. We will forward the ports 4505 and 4506 from our host-system to the salt-master VM and start the VM.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mkdir salt-master
$ cd salt-master
$ echo "VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.network "forwarded_port", guest: 4505, host: 4505
config.vm.network "forwarded_port", guest: 4506, host: 4506
end" &gt;&gt; Vagrantfile
$ vagrant up
[... snipped output ...]
$ vagrant ssh
</code></pre>
</div>
<p>Vagrant will download the needed ubuntu image and spin up the VM. As soon as it is finished you can connect to it with <code class="highlighter-rouge">vagrant ssh</code>. Now we need to set up salt-master and NAPALM. At the moment of writing the blog-post Salt 2017.7.3 and NAPALM 2.3.0 will be installed.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ wget -O bootstrap-salt.sh https://bootstrap.saltstack.com
[... snipped output ...]
$ sudo sh bootstrap-salt.sh
[... snipped output ...]
$ wget https://bootstrap.pypa.io/get-pip.py
[... snipped output ...]
$ python get-pip.py
[... snipped output ...]
$ pip install napalm
[... snipped output ...]
$ pip install -U pyOpenSSL
[... snipped output ...]
</code></pre>
</div>
<p>We will configure the <code class="highlighter-rouge">file_roots</code> and <code class="highlighter-rouge">pillar_roots</code> variables for our master and set up a pillar file for our test-device which we will name <em>device1</em> and which is reachable in the network with 192.168.0.201. We will also configure the salt-proxy to use it as an initial test directly on our master. The proxy configuration file will also be needed later when the container image is built. I will not describe details about how Salt works in this blog post. For the final testing I also included a <em>formula</em> for NTP, which could be found with install-instructions on <a href="https://github.com/saltstack-formulas/napalm-ntp-formula">GitHub</a>. The formula uses the NTP parts of the <a href="https://github.com/openconfig/public/blob/master/release/models/system/openconfig-system.yang">OpenConfig system YANG model</a> which makes it vendor agnostic and perfect for integration with NAPALM. If you are new to Salt I recommend you reading the documentation on the <a href="https://saltstack.com/salt-open-source/">Salt-Homepage</a> or on <a href="https://mirceaulinic.net/aboutme/">Mircea Ulinic’s Blog</a>.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat /etc/salt/master
file_roots:
base:
- /etc/salt/pillar
- /etc/salt/states
- /etc/salt/reactors
- /etc/salt/templates
- /etc/salt/extmods
- /etc/salt/formulas/napalm-ntp-formula
pillar_roots:
base:
- /etc/salt/pillar
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat /etc/salt/proxy
master: localhost
pki_dir: /etc/salt/pki/proxy
cachedir: /var/cache/salt/proxy
multiprocessing: false
mine_enabled: true
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat /etc/salt/pillar/top.sls
base:
'*':
- openconfig_ntp_servers
'device1':
- device1_pillar
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat device1_pillar.sls
proxy:
proxytype: napalm
driver: ios
hostname: 192.168.0.201
username: admin
password: password
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt-run pillar.show_pillar device1
[... snipped output ...]
proxy:
----------
proxytype:
napalm
driver:
ios
hostname:
192.168.0.201
username:
admin
password:
password
</code></pre>
</div>
<p>We can now do a first test: start a local salt-proxy for <em>device1</em>, accept the exchanged key and check if the device is managed by Salt. Surely you have to ensure that the user/password combination defined in the pillar-file is correct for connecting to the Cisco IOS device.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ sudo salt-proxy --proxyid device1 -l debug
[... snipped output ...]
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
device1
Rejected Keys:
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt-key -A device1
[... snipped output ...]
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt-key -L
Accepted Keys:
device1
Denied Keys:
Unaccepted Keys:
Rejected Keys:
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt-run manage.status device1
down:
up:
- device1
</code></pre>
</div>
<p>Your salt-master is now in a state which is good enough to continue with the setup of our other components.</p>
<h3 id="set-up-docker-and-build-the-salt-proxy-container-image">Set up Docker and build the salt-proxy container image</h3>
<p>For easy availability of Docker on my Macbook I decided to use <a href="https://www.docker.com/docker-mac">Docker for MAC</a> which installs a complete docker-environment with the help of a docker-helper-VM in which the docker-daemon runs and in which you can spin up and manage your docker containers. The whole thing gets managed locally with the known <code class="highlighter-rouge">docker</code> cli-commands. Thats an easy solution to get started with docker locally on your client and helps a lot when developing things with containers. To install Docker for MAC please look at the given documentation on the Docker-Homepage.</p>
<p>When thinking about our container image there are different important things to consider: the image should be small, salt-minion and salt-proxy should be installed and ready for use, the proxy configuration file has to be configured for the correct master, NAPALM has to be installed. There is also one more special thing: Because of the self-healing functions of Kubernetes it is absolutely possible that in a failure-case the Pod/Container with our running salt-proxy just gets deleted and replaced by a completely new Pod. If we let everything like it is we would run into problems with the Salt key-exchange. A salt-minion which was well known before would appear with the same name and another key after the Pod was deleted and another one re-deployed. This would lead to a non-functional master/minion relationship and the Kubernetes self-healing would badly impact our solution. Thats why in this case I decided to use the same key-pair for every Pod which gets deployed. The easiest way to accomplish this is to just include the two files into the container-image. If you want to do this in production be sure to use a private container-registry which is secured. The files to use are located on our salt-master VM: <code class="highlighter-rouge">/etc/salt/pki/proxy/minion.pub</code> and <code class="highlighter-rouge">/etc/salt/pki/proxy/minion.pem</code>.</p>
<p>To get a small, but also stable salt-proxy image I decided to use <a href="https://github.com/bitnami/minideb">minideb</a> which is a small image based on Debian designed for use in containers. The following <code class="highlighter-rouge">Dockerfile</code> contains all information needed to build our container image. Be sure to have the three files in the same directory: <code class="highlighter-rouge">proxy</code> (the proxy configuration file -&gt; take the one from our master), <code class="highlighter-rouge">minion.pub</code>, <code class="highlighter-rouge">minion.pem</code>. In the proxy configuration file we change the line <code class="highlighter-rouge">master: localhost</code> to <code class="highlighter-rouge">master: 192.168.64.1</code>. Remember: 192.168.64.1 is the IP of the bridge100-interface built by minikube on our Macbook. Because of our Vagrant configuration the two TCP ports 4505/4506 are forwarded to the salt-master VM. So the proxies are able to use the mentioned IP as salt-master IPv4-address.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat ~/code/docker/salt-minion-proxy
FROM bitnami/minideb:stretch
MAINTAINER No reply smnmtzgr@gmail.com
RUN apt-get update &amp;&amp; install_packages iputils-ping wget gnupg ca-certificates python-setuptools &amp;&amp; wget -O gpgkey https://repo.saltstack.com/apt/debian/9/amd64/latest/SALTSTACK-GPG-KEY.pub &amp;&amp; apt-key add gpgkey &amp;&amp; echo 'deb http://repo.saltstack.com/apt/debian/9/amd64/latest stretch main' &gt;&gt; /etc/apt/sources.list.d/saltstack.list &amp;&amp; apt-get update &amp;&amp; install_packages salt-minion=2017.7.3+ds-1 python-pip &amp;&amp; pip --no-cache-dir install -U pyOpenSSL
ADD ./proxy /etc/salt/proxy
ADD ./minion.pub /etc/salt/pki/proxy/minion.pub
ADD ./minion.pem /etc/salt/pki/proxy/minion.pem
</code></pre>
</div>
<p>This <code class="highlighter-rouge">Dockerfile</code> will pull the minideb base-image, install salt-minion (and proxy), install napalm and copy the needed files into the correct directory. I have built and pushed the image to <a href="https://hub.docker.com/">Docker Hub</a> with the following <code class="highlighter-rouge">docker</code> cli-commands:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ docker build -t salt-minion-proxy .
[... snipped output ...]
$ docker tag salt-minion-proxy smnmtzgr/salt-minion-proxy:0.1.14
[... snipped output ...]
$ docker push smnmtzgr/salt-minion-proxy:0.1.14
[... snipped output ...]
</code></pre>
</div>
<p>If you want to build this lab/solution for your own you could directly use my container image and skip the image build parts: https://hub.docker.com/r/smnmtzgr/salt-minion-proxy/.
The container image is now ready to be used by Kubernetes.</p>
<h3 id="set-up-of-the-kubernetes-cluster-with-minikube">Set up of the Kubernetes cluster with minikube</h3>
<p>When you install <strong>minikube</strong> it builds up a VM on your host-system and runs the Kubernetes-master/node inside of the VM. Inside the VM this is done with a <a href="https://github.com/kubernetes/minikube/tree/master/pkg/localkube">localkube</a> construct which runs a Kubernetes node and a Kubernetes master. The master includes the API server and all other components of the Kubernetes control plane. The container runtime is pre-installed and defaults to docker.</p>
<p>Kubernetes will be managed directly from your host-system via the <code class="highlighter-rouge">kubectl</code> cli-tool which is normally also used for all other kinds of Kubernetes installations. For the VM-hypervisor we use the lightweight <a href="https://github.com/mist64/xhyve">xhyve</a>. The following steps are required to install all of these components:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ brew install kubectl
[... snipped output ...]
$ brew cask install minikube
[... snipped output ...]
$ brew install docker-machine-driver-xhyve
[... snipped output ...]
$ sudo chown root:wheel $(brew --prefix) /opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ sudo chmod u+s $(brew --prefix) /opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ minikube start --vm-driver=xhyve
[... snipped output ...]
$ kubectl config current-context minikube
$ kubectl get nodes
[... snipped output ...]
</code></pre>
</div>
<h3 id="configuration-of-the-kubernetes-deployment">Configuration of the Kubernetes Deployment</h3>
<p>Now that we have a working Kubernetes-cluster installation directly available on our system we can continue and fill it with some life. Our goal is to use it for running one container per salt-managed legacy network-device. We also want to benefit of the self-healing, update- and rollback-features of Kubernetes. The thing we want to use is named a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a>. It is a construct that wraps an object which is called <a href="https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/">ReplicaSet</a>, and the ReplicaSet itself wraps the <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Pod</a> object (which is our container). The Deployment gives us update and rollback possibilities. The ReplicaSet ensures that our Pod is self-healing, scalable and has the desired state (which is READY). The following picture shows these relationships.</p>
<p><img src="/images/k8s-deployment.png" alt="k8s-deployment" title="k8s-deployment" /></p>
<p><em>source: <a href="https://leanpub.com/thekubernetesbook">the-k8s-book</a></em></p>
<p>The best-practice to do something in Kubernetes is to use <em>manifest-files</em>. You can define your <em>desired state</em> in this files in a well-structured format, which is normally <a href="http://yaml.org/">yaml</a>.
In production environments it is highly recommended to manage these manifest-files with a version-control system like git. For our use-case it is enough to just create the file locally on the client, lets say we use the folder <code class="highlighter-rouge">~/code/kubernetes/Deployments</code> and create the file <code class="highlighter-rouge">deploy_device1.yml</code> in it.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat ~/code/kubernetes/Deployments/deploy_device1.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: device1
spec:
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: device1
minReadySeconds: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: device1
spec:
containers:
- name: device1-container
image: smnmtzgr/salt-minion-proxy:0.1.14
command: ["salt-proxy", "--proxyid", "device1", "-l", "debug"]
resources:
limits:
cpu: 70m
memory: 70Mi
requests:
cpu: 60m
memory: 60Mi
</code></pre>
</div>
<p>The file contains the Deployment completely defined in yaml. The Deployment will be named <em>device1</em>, we will have one replica which means one Pod, and the Pod will contain one container named <em>device1-container</em>. Here we also reference to the container image we built before and which is now located on Docker Hub. Thats where Kubernetes will pull the image from and spin up the containers with. It is also important that we define the command which to run in the container. In our case this is <code class="highlighter-rouge">salt-proxy --proxyid device1 -l debug</code> which starts the salt-proxy, gives it the id <em>device1</em> and activates logging at debug level. If we wouldn’t give a command the Pod would just start up, don’t know what to do and shut down. This would lead to a loop: Kubernetes would destroy the Pod and deploy a new one, because the Pod-state was not READY. Thats the self-healing mechanism. We also limit the resources the container can use.</p>
<p>If you really want to use such a solution with hundreds or thousands of devices it would for example be possible to build the manifest file in a more generic way so that you don’t use the hard-coded <em>device1</em> name, but rather than that use a variable which you fill with content when using the file.</p>
<p>We also have to talk about the self-healing functionality here. If we deploy the Deployment like defined in the manifest Kubernetes would only re-deploy the Pod when the command we call will return an exit-code of 1. This is for example the case when the command just fails. But it’s also possible that it throws some ERRORS, but doesn’t exit completely. This happens for example when the IP of the network-device is simply not reachable at the moment. The proxy-process will just try the connection again and again. But for Kubernetes this means that everything is fine and it considers the state of the Pod as READY. In a production environment we should be more clear at this point and define a <em>Liveness</em> state in the Deployment manifest. Details are described in the official <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/">Kubernetes documentation</a>.</p>
<p>The last remaining step is to use the manifest-file to deploy the Deployment to the Kubernetes-cluster. If you want to do such things in Kubernetes you use normally the <code class="highlighter-rouge">kubectl</code> tool to do a HTTP-POST against the <em>Kubernetes API Server</em> which provides a RESTful API. The API Server runs as part of the Kubernetes control-plane on the Kubernetes master. After receiving the POST it inspects the content of the file and knows exactly what to do to get to the desired state defined in the file. This desired state will also be saved to the Kubernetes <em>cluster store</em> which is based on <a href="https://github.com/coreos/etcd">etcd</a> and acts as the brain of the cluster. To ensure the desired state ReplicaSets implement a background reconciliation loop that is constantly monitoring the cluster. It checks if the current state matches the desired state. If that’s not the case it wakes up the control-plane and Kubernetes fixes the situation with whatever steps are needed to get back to the desired state. The following command does the HTTP-POST:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kubectl create -f deploy_device1.yml
[... snipped output ...]
</code></pre>
</div>
<p>This is all what is needed to instruct Kubernetes to set up all of these cool objects. After successful execution you can check the state with different commands:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
device1 1 1 1 1 1d
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kubectl get rs
NAME DESIRED CURRENT READY AGE
device1-859ffc6cb7 1 1 1 1d
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kubectl get pod
NAME READY STATUS RESTARTS AGE
device1-859ffc6cb7-fbwrc 1/1 Running 0 1d
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kubectl logs &lt;pod_name&gt;
[... snipped output ...]
</code></pre>
</div>
<p>If you continue to check the logs of the Pod you should see after some time that the salt-proxy is active, connected as a minion to the master and acting as proxy for the network-device.
On the master you have to accept the key for device1 (if not already done in our first steps):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt-key -A device1
[... snipped output ...]
</code></pre>
</div>
<h3 id="manage-the-network-device-with-salt">Manage the network-device with Salt</h3>
<p>After setting up all these things we should now be able to manage device1 with Salt. On the salt-master we execute some commands to show this. Most of them should be self-explaining. If you need more examples or ideas what Salt is able to do with network devices I recommend you to read through the great book <a href="https://www.cloudflare.com/media/pdf/network-automation-at-scale.pdf">Network Automation at Scale</a>.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt device1 net.load_config text='ntp server 172.17.18.1'
device1:
----------
already_configured:
False
comment:
diff:
+!
+ntp server 172.17.18.1
+switch-01#^@
loaded_config:
result:
True
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt device1 grains.get interfaces
device1:
- Vlan1
- FastEthernet0/1
- FastEthernet0/2
- FastEthernet0/3
- FastEthernet0/4
- FastEthernet0/5
- FastEthernet0/6
- FastEthernet0/7
- FastEthernet0/8
- GigabitEthernet0/1
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt device1 grains.get model
device1:
WS-C2960-8TC-L
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt device1 grains.get model --output=json
{
"device1": "WS-C2960-8TC-L"
}
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ salt device1 state.sls ntp.netconfig
device1:
----------
ID: oc_ntp_netconfig
Function: netconfig.managed
Result: True
Comment: Configuration changed!
Started: 20:31:50.924403
Duration: 23426.954 ms
Changes:
----------
diff:
+!
-no ntp
+ntp server 172.17.19.1 prefer
+ntp server 172.17.19.2
Summary for device1
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 23.427 s
</code></pre>
</div>
<p>Note: For the last command to work you have to set up the <a href="https://github.com/saltstack-formulas/napalm-ntp-formula">napalm-ntp-formula</a>.</p>
<h3 id="running-the-solution-in-production">Running the solution in production?</h3>
<p>If you really think about running a solution like this in production there are surely things you have to go deeper and make decisions. I want to mention three sticking points.</p>
<p>The first is how to handle the key-exchange between salt-master and salt-minions (proxies). In my solution we just used the same key-pair over and over again. We even integrated it into the container image we built. This works fine but if you think about securing your application that’s not the best way to go.</p>
<p>The second thing is how you want to set up your Kubernetes cluster. That’s no easy task and has to be discussed with Kubernetes and networking experts. There are a lot of questions how to design it, where to run it and what pieces to use. For example Kubernetes has its own model how networking should work. It imposes the following fundamental requirements on any networking implementation:</p>
<ul>
<li>all containers can communicate with all other containers without NAT</li>
<li>all nodes can communicate with all containers (and vice-versa) without NAT</li>
<li>the IP that a container sees itself as is the same IP that others see it as</li>
</ul>
<p>In practice this means that you can’t just take two docker-hosts and run Kubernetes on top of it to manage it. That’s not how docker-networking is implemented. In the <a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/">Kubernetes documentation</a> you can read more about these aspects and go into detail. An interesting part is how to achieve this behaviour. Kubernetes gives some ideas and references for setting up the networking in a <em>Kubernetes-model</em> way. Those are: Cisco ACI, Cilium, Contiv, Contrail, Flannel, Google Compute Engine (GCE), Kube-router, L2 networks and linux bridging, Multus, NSX-T, Nuage Networks VCS, OpenVSwitch, OVN, Project Calico, Romana, Weave Net from Weaveworks, CNI-Genie from Huawei.</p>
<p>The third is how you design the salt-master. I’ve just used a single Vagrant-VM which was enough for testing purposes. But in production you should eventually setup a HA salt-master construct or think about running multiple salt-masters in different locations. This depends a bit on what you want to do with the solution.</p>
<h3 id="summary">Summary</h3>
<p>Seeing all these tools and components working together is very cool, makes a lot of fun and builds a robust framework for network-automation purposes. At the end I want to mention that I’m neither a Salt expert nor a Kubernetes guru. I started using these tools two weeks before writing this blog-post. I just wanted to integrate them into my overall solution. So certainly there are other and even better ways to do the stuff I’ve done here. I would be happy if you get in touch with me in such cases. You can reach me (@smnmtzgr) via Twitter or directly in the networktocode slack community (networktocode.slack.com).</p>smnmtzgrSince the native integration of NAPALM into the Salt core which was officially done since Salt Carbon (2016.11) by Mircea Ulinic it is possible to manage network-devices directly with Salt. I thought about how to scale this out and how to manage a lot of (hundreds or even thousands) network-devices with this solution. To be more clear my goal was to manage legacy devices that are not able to install software natively on themselves. In this case the salt-minion client can’t be used. The job has to be done by proxy minions instead. Per salt-managed network-device a salt-proxy process which reserves about 60 MB of memory is required. The following image shows the high-level logic of this construct.Blog hosted on heroku2018-02-05T00:00:00+00:002018-02-05T00:00:00+00:00http://blog.simonmetzger.de/2018/02/blog-hosted-on-heroku<p>My blog is now hosted on <a href="https://heroku.com">heroku</a>. I decided to switch from <a href="https://pages.github.com/">GitHub Pages</a>. Heroku has also support for git and jekyll, which is perfect for my blog. So like before I can just make my changes, watch and check them locally with <code class="highlighter-rouge">jekyll serve</code> and if everything is okay just push them via <code class="highlighter-rouge">git</code> to my heroku repository. It also gives me some additional stuff, which enable my blog to use some kind of authentication at some parts of it.</p>
<p>These are the steps required for the workflow:</p>
<div class="highlighter-rouge"><pre class="highlight"><code># install heroku client
$ brew install heroku/brew/heroku
# login
$ heroku login
# clone the repository
$ heroku git:clone -a smnmtzgr-blog
$ cd smnmtzgr-blog
# make changes and deploy them
$ git add --all
$ git commit -am "make it better"
$ git push heroku master
</code></pre>
</div>smnmtzgrMy blog is now hosted on heroku. I decided to switch from GitHub Pages. Heroku has also support for git and jekyll, which is perfect for my blog. So like before I can just make my changes, watch and check them locally with jekyll serve and if everything is okay just push them via git to my heroku repository. It also gives me some additional stuff, which enable my blog to use some kind of authentication at some parts of it.Ansible-playbook wrapped in docker container2017-10-11T00:00:00+00:002017-10-11T00:00:00+00:00http://blog.simonmetzger.de/2017/11/ansible-playbook-docker<p>Today I’ve tested some of the new Cisco ACI modules which get shipped with Ansible 2.4. So I built my local ansible project, created the inventory and the playbook. I run into an authentication issue which was based on the python version ansible was using. I corrected this with the solution mentioned in this <a href="http://blog.simonmetzger.de/2017/10/ansible-python-interpreter/">blog post</a>.</p>
<p>But I was bugged out of this issue and thought about how to prevent this in the future. After talking with a cool <a href="https://twitter.com/dirkwoellhaf">Cisco SE</a> he gave me the hint that he is using a docker container to run his ansible test-environment. Thats a very cool alternative and just a moment later I wanted to built this too.</p>
<p><strong>My steps for using this on my client Mac OS X:</strong></p>
<ul>
<li>Install <a href="https://www.docker.com/docker-mac">docker for MAC</a></li>
<li>Build a Dockerfile to create a custom image with ansible installed (I was choosing centos as my base image)</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>FROM centos:latest
RUN curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
RUN python get-pip.py
RUN pip install ansible --trusted-host=pypi.python.org
RUN mkdir -p /ansible/playbooks
WORKDIR /ansible/playbooks
ENTRYPOINT ["ansible-playbook"]
</code></pre>
</div>
<ul>
<li>Create the custom image: docker build -t simon-ansible-image .</li>
<li>Use the image as an <em>ansible-playbook</em> wrapper. So a container gets spawned, the playbook executed (interactive because of the -it parameter) and after execution the container gets deleted (–rm parameter). The -v parameter maps your local ansible-project directory - where the playbook and inventory and other ansible-related stuff is located - to a directory into the container.</li>
<li>Have fun executing your playbooks:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>docker run --rm -it -v $(pwd):/ansible/playbooks simon-ansible-image aci-tests.yml -i inventory
PLAY [try out aci modules] ***************************************************************************************************************************************************************************************************************************************************
TASK [add a new tenant] ******************************************************************************************************************************************************************************************************************************************************
ok: [apic-dev.local]
PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************
apic-dev.local : ok=1 changed=0 unreachable=0 failed=0
</code></pre>
</div>
<p>I also use an alias in my .zshrc to make the use of this a log easier: <em>alias ap24=’docker run –rm -it -v $(pwd):/ansible/playbooks smnmtzgr/ansible-playbook:2.4’</em></p>
<p>Now I can run ansible-playbooks in the container just by using this command: <em>ap24 playbook_name -i inventory_name</em></p>
<p>The docker image is available on <a href="https://store.docker.com/community/images/smnmtzgr/ansible-playbook">docker store</a>.</p>smnmtzgrToday I’ve tested some of the new Cisco ACI modules which get shipped with Ansible 2.4. So I built my local ansible project, created the inventory and the playbook. I run into an authentication issue which was based on the python version ansible was using. I corrected this with the solution mentioned in this blog post.Ansible python interpreter2017-10-10T00:00:00+00:002017-10-10T00:00:00+00:00http://blog.simonmetzger.de/2017/10/ansible-python-interpreter<p>If you run into problems with the python which ansible uses to execute playbooks you can ensure that a specific python version gets used with the <em>ansible_python_interpreter</em> value set in the inventory file.</p>
<p>It is possible to hardcode the path to a custom python executable or what I prefer is to set the interpreter to the one that’s found first in the remote (or local) machine’s PATH. <em>ansible_python_interpreter</em> can be used to address that case as well if you have the env program installed in a reliable location. Here’s how you can do that in an inventory file:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[all:vars]
ansible_python_interpreter="/usr/bin/env python"
</code></pre>
</div>smnmtzgrIf you run into problems with the python which ansible uses to execute playbooks you can ensure that a specific python version gets used with the ansible_python_interpreter value set in the inventory file.Private vs. public vs. hybrid Cloud2017-08-21T00:00:00+00:002017-08-21T00:00:00+00:00http://blog.simonmetzger.de/2017/08/private-vs-public-cloud<p>Because of this interesting article written by <a href="https://cumulusnetworks.com/learn/private-cloud-vs-public-cloud/">Cumulus Networks</a> I decided to think about some questions regarded to this topic.
Below you can read my personal thougths about the different kinds of a cloud. Some information are out of the mentioned article.</p>
<h2 id="my-definition-of-a-cloud">My definition of a Cloud</h2>
<p>A cloud is a bunch of computing systems which together build a closed system that provides computing, networking, storage, security, applications and services. People or other computer systems can make use of this things in an automated manner, use them on demand as single parts or combine them to get a fully working system.
The key point for the following differentiations is the surface where the resources are offered: in your own infrastructure/data-center (private), in the internet (public) or both (hybrid).</p>
<p><img src="/images/types-of-cloud.jpg" alt="workflow" title="types of cloud" /></p>
<p><em>source: <a href="http://www.tek-nologysolutions.co.uk/uploads/images/article-images/Solutions/Cloud/types-of-cloud.jpg">tek-nologysolutions</a></em></p>
<h3 id="the-correct-choice">The correct choice?</h3>
<p>Todays companies have to decide where to run their applications. Build a private cloud? Use a public cloud? Do both? Which is the best for my applications? Who wants to use them? What is needed? SaaS? Or only a pool of resources? All these questions require a very good planning and oversight. Each application can be different. But the important thing is: the application has to run with focus-first to the customers/users. The decision how to run applications and IT systems should be driven by the needs of customers/users. Whats the purpose of the best-running private cloud or the perfect network solution if in the end none of the end users can profit of it? If this happens something went wrong and a lot of unnecessary expense were generated.</p>
<h3 id="public-cloud">Public Cloud</h3>
<p>Public Clouds are hosted and maintained by an external provider. Some of the biggest are Microsoft Azure, Google Cloud Platform or Amazon AWS. Such big clouds are separated into a tenant per customer. So every customer gets some pieces of computer resources which are logically reserved and only available for him. How much resources you get depends on how much you are willing to pay for.</p>
<p>The big benefit of public clouds is the fact that you can fully concentrate on your application. The things in behind are maintained and run by the cloud provider. If something fails you can open tickets or the provider has some self-healing/failover mechanisms. But don`t forget that you still have to administrate the cloud and to understand how the cloud provider works. If this approach fits for your application a public cloud could be a good choice and can lead to time and money savings.</p>
<h3 id="private-cloud">Private Cloud</h3>
<p>Private Clouds run in own environments/datacenters: on-premise. They can be configured to support any needs of your applications in the best possible manner. You have full control over the cloud and can use whatever you want to build it. Often it is a high barrier of planning and beginning such a project. But if you have the people and knowledge in your company in most cases it is a very good choice to build and run a private cloud. The cloud can offer everything your people and customers need.</p>
<p>Another key point is that you know where your applications are running. You don`t have to think about a lot of compliance topics. Does it for example make a difference if my company data is stored on a system in Europe or the USA?</p>
<h3 id="hybrid-cloud">Hybrid Cloud</h3>
<p>With Hybrid Cloud solutions you can get the best out of both worlds. Use the cloud you need at the moment! You have a legacy application? Go for your private cloud. You want to use an application which perfectly runs on AWS or even better is fully integrated and available there? Choose AWS. You need a Microsoft Sharepoint test environment? Choose Azure.</p>
<p>Of course a hard point of this approach is again the compliance. Now you have to consider not only one cloud-environment, but rather two, three or more. And you have to know the guidelines and specifics of each of them. So in sum a hybrid approach usually leads to the most overhead of resources and management.</p>
<h3 id="benefits-of-moving-to-a-private-cloud">Benefits of moving to a private cloud</h3>
<p>The following points are referenced from the mentioned article of <a href="https://cumulusnetworks.com/learn/private-cloud-vs-public-cloud/">Cumulus Networks</a>.</p>
<ul>
<li>cost efficiency</li>
<li>improved security (“single tenant”)</li>
<li>web-scale IT agility and continuity</li>
<li>resource optimization (no vendor locking)</li>
<li>greater control</li>
<li>more customization</li>
<li>less compliance issues</li>
<li>unlimited scalability and adaptability</li>
</ul>
<h3 id="technical-networking-trends-that-may-help-you-move-to-a-private-cloud">Technical networking trends that may help you move to a private cloud</h3>
<p>The following points are referenced from the mentioned article of <a href="https://cumulusnetworks.com/learn/private-cloud-vs-public-cloud/">Cumulus Networks</a>.</p>
<ul>
<li>devops and netdevops</li>
<li>hyper-converged infrastructure</li>
<li>disaggregation (e.g. with Cumulus Linux)</li>
</ul>smnmtzgrBecause of this interesting article written by Cumulus Networks I decided to think about some questions regarded to this topic. Below you can read my personal thougths about the different kinds of a cloud. Some information are out of the mentioned article.ACI 3.0 released2017-08-11T00:00:00+00:002017-08-11T00:00:00+00:00http://blog.simonmetzger.de/2017/08/aci-3-0<p>I just installed ACI 3.0(1k). The improved GUI looks very cool :)</p>
<p><img src="/images/aci-3-0-gui.png" alt="aci" title="ACI 3.0 login screen" /></p>smnmtzgrI just installed ACI 3.0(1k). The improved GUI looks very cool :)Career thoughts2017-04-10T00:00:00+00:002017-04-10T00:00:00+00:00http://blog.simonmetzger.de/2017/04/career-thoughts<p>Inspired by the book <a href="https://leanpub.com/unintendedfeatures">Unintended Features</a> (written by Daniel Dib and Russ White) I decided to think about some questions regarded to my career as a network engineer. The questions are mentioned in the third chapter (Education) and are in my opinion very helpful to think about. You should answer them to get a clearer picture of yourself and your (career) goals: Do I go forward on the right way? Which next step is the best for me? Do I make the (hopefully) best decisions at the right time?</p>
<p>Below you can read my personal thougths and answers to the questions.</p>
<h3 id="where-do-i-want-to-go-in-my-life--who-do-i-want-to-be-as-a-person-overall">Where do I want to go (in my life)? / Who do I want to be as a person overall?</h3>
<p>There is the private part: I want to be a father for my family (wife and daughter). For me it is important to live in the near of my roots. So it doesn’t work for me to have a job which leads to massive traveling the whole week. Because I am a passionated christian I also want to put a lot of energy into the church where I go and their work with young people.
And there is the job-related part: I also want to become a more experienced network engineer and one of my goals is to get a CCIE Routing&amp;Switching in the next 3 years. In parallel my interest for python and network automation is there and I want to contribute to at least one open source project.</p>
<h3 id="what-am-i-not-very-good-at-today-skills-as-a-network-engineer-what-do-i-need-to-get-to-where-i-want-to-go-and-what-do-i-need-there">What am I not very good at today (skills as a network engineer)? What do I need to get to where I want to go and what do I need there?</h3>
<p>After my CCNA I stopped focusing on certifications and fully concentrated on my work as a network engineer. I tried to do a good job and to get a lot of real life work experience. Now after about 5 years of experience I think it would be good to learn some technologies deeper with the help of a expert certification. This is why I decided to begin my way of becoming a CCIE in routing&amp;switching. As first steps I want to re-certify my CCNA R&amp;S at least until September and get a CCNP R&amp;S in 2018.</p>
<h3 id="what-motivates-me-to-learn">What motivates me to learn?</h3>
<p>What motivates me the most is to see results. I love to learn things with different methods and ways: read blogs, read books, read howtos, watch videos. And after reading and thinking about one thing I try to understand it and “build” something related to the new knowledge I got. If there is a working end result which ideally could be used by others and gets documented by me this leads to a very comfortable feeling and motivation for myself to go on and learn more. So it is important to me to interrelate the theoretical and practical parts of a learning topic.</p>
<h3 id="what-is-the-best-way-to-learn-this">What is the best way to learn this?</h3>
<p>Like I mentioned there are different ways to learn something. And I think the best method to learn something is dependend on what exactly you want to achieve. If you for example want to be informed about the latest trends and news in networking, read some blogs, build a twitter network or participate in a slack team/group. If your goal is to get a certification you should mix it a bit more and think about what motivates you to learn constantly. For me this would be a combination of: reading books and blog posts + watch videos about the different topics and make structured notes.</p>
<h3 id="what-will-have-the-biggest-financial-impact">What will have the biggest financial impact?</h3>
<p>There are to parts here. One part is what will have the biggest financial impact to my salary. The other part is what are the costs for learning material to get the information and knowledge I need for successfully finishing the certifications.</p>
<h3 id="what-do-i-already-have-today">What do I already have today?</h3>
<p>I hold a degree (Bachelor of engineering) in communication technology, have 5 years experience as a network engineer and got a CCNA R&amp;S certification straight after finishing my degree.</p>smnmtzgrInspired by the book Unintended Features (written by Daniel Dib and Russ White) I decided to think about some questions regarded to my career as a network engineer. The questions are mentioned in the third chapter (Education) and are in my opinion very helpful to think about. You should answer them to get a clearer picture of yourself and your (career) goals: Do I go forward on the right way? Which next step is the best for me? Do I make the (hopefully) best decisions at the right time?Event-driven network automation using Slack, hubot, StackStorm and Ansible2016-04-01T00:00:00+00:002016-04-01T00:00:00+00:00http://blog.simonmetzger.de/2016/04/event-driven-network-automation<p>Event-driven automation is a very interesting topic. Think about executing task X when a POST-request gets fired to an application. Or about executing task Y as soon as a special mail gets obtained. I thought about bringing this topic together with network-automation and the things I have learned about Ansible and Co. I also got inspired by some great guys who are active in the <a href="https://networktocode.slack.com/">networktocode</a> slack-team. For example there is already a <a href="https://www.dravetech.com/blog/2016/03/30/chatops-demo.html">blog post</a> published by <a href="https://twitter.com/dbarrosop">@dbarrosop</a> which covers a similar topic. If you are interested in network-automation <em>networktocode</em> is the place to go. <a href="https://stackstorm.com/">StackStorm</a> is a tool for event-driven automation. The tool is very successful and has a big company behind. A few days before the start-up just got <a href="https://stackstorm.com/2016/03/29/stackstorm-joining-brocade/">bought by Brocade</a>, which let me hope about getting more networking-content within StackStorm in the future. The community-edition will still be available as open-source product at no charge.</p>
<p>I started a project in my development environment with the goal to automate Cisco IOS-devices over simple chat-messages in <a href="https://slack.com/">Slack</a>. At the moment of writing this post I had build actions and aliases in StackStorm to add or delete vlans and to ensure a base config on my network switches.
In this blog post I want to show you the possibilities and how you can get this working in your own environment. The following picture illustrates the automation workflow.</p>
<p><img src="/images/HWN8T78.png" alt="workflow" title="event-driven automation workflow" /></p>
<p>Summarized we have to install and configure the following components:</p>
<ul>
<li>Slack team</li>
<li>Slack hubot plugin</li>
<li>An Ubuntu 14.04. server with the help of vagrant</li>
<li>StackStorm</li>
<li>Custom StackStorm packs</li>
<li>Ansible</li>
</ul>
<h3 id="lets-get-started-with-the-slack-components">Lets get started with the Slack components</h3>
<p>Go to <a href="https://slack.com">slack.com</a> and create a new team. In my environment the team is named <em>itsnetwork</em>. After creating the team go to <strong>team settings-&gt;Configure Apps</strong>, search for <strong>hubot</strong> and add it to your team. Also copy the <strong>API Token</strong> to a temporary text file. <a href="https://hubot.github.com/">Hubot</a> is an open source bot, written in CoffeeScript on Node.js and a standardized way to share scripts between everyones robots. We will use it as a gateway between Slack and StackStorm.</p>
<p><img src="/images/slack-hubot.png" alt="slack-hubot" title="hubot in Slack" /></p>
<p>Now invite your hubot to one of your team-channels (replace the bot-name with your own):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/invite @simons-hubot
</code></pre>
</div>
<h3 id="set-up-ubuntu-1404-with-vagrant">Set up Ubuntu 14.04. with vagrant</h3>
<p>We need a server where StackStorm will run. In our testing environment <em>vagrant</em> will be used to setup an Ubuntu 14.04. server. Follow the instructions on the <a href="https://www.vagrantup.com/">Vagrant-Site</a> to install Vagrant on your operating system. Also install a vagrant provider, which is mostly <a href="https://www.virtualbox.org/">virtualbox</a>. I recommend you to follow the install-instructions on the links and not to install the software with package-managers of your operating system.</p>
<p>After installing Vagrant and virtualbox now create a new folder called <em>st2</em> and create the <strong>Vagrantfile</strong>. Edit the public_network ip to an ip which is reachable in your local LAN. This will be the ip-address of your server. Then build the VM.</p>
<div class="language-sh highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>mkdir st2
<span class="gp">$ </span><span class="nb">cd </span>st2
<span class="gp">$ </span><span class="nb">echo</span> <span class="s2">"VAGRANTFILE_API_VERSION = "</span>2<span class="s2">"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "</span>ubuntu/trusty64<span class="s2">"
config.vm.box_download_insecure = true
config.vm.network "</span>public_network<span class="s2">", ip: "</span>192.168.0.21<span class="s2">"
config.vm.provider "</span>virtualbox<span class="s2">" do |vb|
vb.memory = "</span>2404<span class="s2">"
end
end"</span> &gt;&gt; Vagrantfile
<span class="gp">$ </span>vagrant up
</code></pre>
</div>
<h3 id="install-stackstorm-community-edition">Install StackStorm community edition</h3>
<p>SSH to your vagrant VM with <em>vagrant ssh</em> and follow the official <a href="https://docs.stackstorm.com/install/deb.html">install instructions</a> for StackStorm on Ubuntu. If you have trouble installing or starting mistral please check your installed locales. In the <strong>Setup ChatOps</strong> section you will also install <strong>hubot</strong> and the <strong>StackStorm-hubot-plugin</strong>. These things are included in the <em>st2chatops</em> package which gets installed with apt.</p>
<p>Now also update the chatops pack files to the <a href="https://github.com/StackStorm/st2/tree/master/contrib/chatops">most recent version</a>, because the one installed via apt-get includes a bug (replaces \n with @).</p>
<p>After setting up all those things please play around with some <em>st2 commands</em> on the StackStorm-VM or look at the web-interface.</p>
<p><img src="/images/stackstorm-webui.png" alt="stackstorm-webui" title="StackStorm Webui" /></p>
<h3 id="install-the-custom-stackstorm-packs">Install the custom StackStorm packs</h3>
<p>I have built two custom StackStorm packs which we will install now. Packs could contain StackStorm actions, aliases, rules and sensors. For our purpose the ansible playbooks and roles will be included additionaly. The packs could be found on github:</p>
<ul>
<li><a href="https://github.com/smnmtzgr/st2-netops-vlan">smnmtzgr/st2-netops-vlan</a></li>
<li><a href="https://github.com/smnmtzgr/st2-netops-base_config">smnmtzgr/st2-netops-base_config</a></li>
</ul>
<p>Install them on your VM with:</p>
<div class="language-sh highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>st2 run packs.install <span class="nv">packs</span><span class="o">=</span>st2-netops-vlan <span class="nv">repo_url</span><span class="o">=</span>https://github.com/smnmtzgr/st2-netops-vlan
<span class="gp">$ </span>st2 run packs.install <span class="nv">packs</span><span class="o">=</span>st2-netops-base_config <span class="nv">repo_url</span><span class="o">=</span>https://github.com/smnmtzgr/st2-netops-base_config
</code></pre>
</div>
<p>Edit the following things:</p>
<ul>
<li>insert your hosts to the ansible-host file located at <em>/opt/stackstorm/packs/st2-netops-base_config/playbooks/hosts</em></li>
<li>insert your hosts to the ansible-host file located at <em>/opt/stackstorm/packs/st2-netops-vlan/playbooks/hosts</em></li>
<li>change user and password in the ansible group_vars file located at <em>/opt/stackstorm/packs/st2-netops-base_config/playbooks/group_vars/ios.yaml</em></li>
<li>change user and password in the ansible group_vars file located at <em>/opt/stackstorm/packs/st2-netops-vlan/playbooks/group_vars/ios.yaml</em></li>
</ul>
<p>Now we have to setup Ansible on the StackStorm VM. The devel-branch includes all needed network-modules:</p>
<div class="language-sh highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>git clone http://github.com/ansible/ansible.git
<span class="gp">$ </span><span class="nb">cd </span>ansible
<span class="gp">$ </span>git checkout devel
<span class="gp">$ </span>git submodule init
<span class="gp">$ </span>git submodule update
<span class="gp">$ </span>pip uninstall ansible
<span class="gp">$ </span>python setup.py install
<span class="gp">$ </span>ansible --version
ansible 2.0.1.0 <span class="o">(</span>stable-2.0-network 49f7e6a12c<span class="o">)</span> last updated 2016/03/01 20:57:24 <span class="o">(</span>GMT +200<span class="o">)</span>
lib/ansible/modules/core: <span class="o">(</span>detached HEAD 72540d1f0c<span class="o">)</span> last updated 2016/03/01 20:57:27 <span class="o">(</span>GMT +200<span class="o">)</span>
lib/ansible/modules/extras: <span class="o">(</span>detached HEAD 51cddf2b35<span class="o">)</span> last updated 2016/03/01 20:57:28 <span class="o">(</span>GMT +200<span class="o">)</span>
</code></pre>
</div>
<h3 id="setup-slack-integration-within-the-stackstorm-hubot-plugin">Setup Slack-integration within the StackStorm-hubot-plugin</h3>
<p>If not already done while following the StackStorm install instructions now edit the <em>/opt/stackstorm/chatops/st2chatops.env</em> file, comment out the Slack-lines and insert your hubot API Token. Also insert the password for your st2admin-user.
My file looks like this (API Token is changed):</p>
<div class="language-sh highlighter-rouge"><pre class="highlight"><code><span class="c">#####################################################################</span>
<span class="c"># Hubot settings</span>
<span class="c"># set if you don’t have a valid SSL certificate.</span>
<span class="nb">export </span><span class="nv">NODE_TLS_REJECT_UNAUTHORIZED</span><span class="o">=</span>0
<span class="c"># Hubot port - must be accessible from StackStorm</span>
<span class="nb">export </span><span class="nv">EXPRESS_PORT</span><span class="o">=</span>8081
<span class="c"># Log level</span>
<span class="nb">export </span><span class="nv">HUBOT_LOG_LEVEL</span><span class="o">=</span>debug
<span class="c"># Bot name</span>
<span class="nb">export </span><span class="nv">HUBOT_NAME</span><span class="o">=</span>stanley
<span class="nb">export </span><span class="nv">HUBOT_ALIAS</span><span class="o">=</span>!
<span class="c">######################################################################</span>
<span class="c"># StackStorm settings</span>
<span class="c"># StackStorm api endpoint. (Don’t use `localhost` as it would point to the Docker container).</span>
<span class="nb">export </span><span class="nv">ST2_API</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">ST2_HOSTNAME</span><span class="k">}</span>/api
<span class="c"># StackStorm auth endpoint. (Don’t use `localhost` as it would point to the Docker container).</span>
<span class="nb">export </span><span class="nv">ST2_AUTH_URL</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">ST2_HOSTNAME</span><span class="k">}</span>/auth
<span class="c"># ST2 credentials</span>
<span class="nb">export </span><span class="nv">ST2_AUTH_USERNAME</span><span class="o">=</span>st2admin
<span class="nb">export </span><span class="nv">ST2_AUTH_PASSWORD</span><span class="o">=</span>&lt;INSERT YOUR PASSWORD&gt;
<span class="c"># Public URL of StackStorm instance: used it to offer links to execution details in a chat.</span>
<span class="c">#export ST2_WEBUI_URL=https://${ST2_HOSTNAME}</span>
<span class="nb">export </span><span class="nv">ST2_WEBUI_URL</span><span class="o">=</span>https://192.168.0.21
<span class="c">######################################################################</span>
<span class="c"># Chat service adapter settings</span>
<span class="c"># Uncomment one of the adapter blocks below.</span>
<span class="c"># Currently supported: slack, hipchat, xmpp, yammer, irc, flowdock.</span>
<span class="c"># Slack settings (https://github.com/slackhq/hubot-slack):</span>
<span class="c">#</span>
<span class="nb">export </span><span class="nv">HUBOT_ADAPTER</span><span class="o">=</span>slack
<span class="nb">export </span><span class="nv">HUBOT_SLACK_TOKEN</span><span class="o">=</span>&lt;INSERT YOUR API TOKEN HERE&gt;
</code></pre>
</div>
<p>Now restart st2chatops with <strong>sudo service st2chatops restart</strong>.</p>
<h3 id="test-it">Test it</h3>
<p>Go back to your Slack-team and execute some commands. After playing around check on your network-devices if the base_config is ensured and if the vlans you added/deleted via chat-commands are really added/deleted to/from your devices. Here are some screenshots and outputs of my results:</p>
<p><img src="/images/hubot-help.png" alt="hubot-help" title="hubot-help" />
<img src="/images/hubot-addvlan.png" alt="hubot-addvlan" title="hubot-addvlan" /></p>
<div class="highlighter-rouge"><pre class="highlight"><code>switch-01#sh vlan brief
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5, Fa0/6, Fa0/7, Fa0/8, Gi0/1
5 test active
10 LX active
22 blogpost active
700 fw active
1002 fddi-default act/unsup
1003 token-ring-default act/unsup
1004 fddinet-default act/unsup
1005 trnet-default act/unsup
</code></pre>
</div>
<p><img src="/images/hubot-deletevlan.png" alt="hubot-deletevlan" title="hubot-deletevlan" /></p>
<div class="highlighter-rouge"><pre class="highlight"><code>switch-01#sh vlan brief
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5, Fa0/6, Fa0/7, Fa0/8, Gi0/1
5 test active
10 LX active
700 fw active
1002 fddi-default act/unsup
1003 token-ring-default act/unsup
1004 fddinet-default act/unsup
1005 trnet-default act/unsup
</code></pre>
</div>
<p><img src="/images/hubot-ensureconfig.png" alt="hubot-ensureconfig" title="hubot-ensureconfig" /></p>
<div class="highlighter-rouge"><pre class="highlight"><code>switch-01#sh run | i snmp
snmp-server community public RO
snmp-server community private RW
snmp-server community operations RW
snmp-server location data center
snmp-server contact network operations
switch-01#sh run | i ntp
ntp source Vlan1
ntp server 1.1.1.1
ntp server 2.2.2.2
ntp server 3.3.3.3
ntp server 4.4.4.4
</code></pre>
</div>
<h2 id="summary">Summary</h2>
<p>This use-case is only a very small and easy one but shows a lot of the power and possibilities you have with todays tools. StackStorm is easily said an orchestration tool which can control everything what happens on the basis of events. To clear things up: in our case we simply used some chat-commands as manual API-calls. Real events are normally not manually created by a human being, they are generated by another machine. For example in a real-world scenario it would be possible that a monitoring tool like icinga - in the case of a special event (host down, interface down…) - generates a HTTP-POST request to StackStorm which will then trigger some actions. Your fantasy should not have any constraints! :)</p>
<h2 id="miscellaneous">Miscellaneous</h2>
<p>Good sources I used to get started with those things are:</p>
<ul>
<li><a href="https://stackstorm.com/2015/06/24/ansible-chatops-get-started-%F0%9F%9A%80/">Stackstorm blog post about getting started with chatops</a></li>
<li><a href="https://stackstorm.com/2015/06/12/integrating-chatops-with-stackstorm/">StackStorm blog post about integrating chatops with stackstorm</a></li>
<li><a href="https://stackstorm-community.slack.com">The StackStorm Slack-Team</a> (you should be invited if you follow @Stack_Storm on twitter)</li>
</ul>smnmtzgrEvent-driven automation is a very interesting topic. Think about executing task X when a POST-request gets fired to an application. Or about executing task Y as soon as a special mail gets obtained. I thought about bringing this topic together with network-automation and the things I have learned about Ansible and Co. I also got inspired by some great guys who are active in the networktocode slack-team. For example there is already a blog post published by @dbarrosop which covers a similar topic. If you are interested in network-automation networktocode is the place to go. StackStorm is a tool for event-driven automation. The tool is very successful and has a big company behind. A few days before the start-up just got bought by Brocade, which let me hope about getting more networking-content within StackStorm in the future. The community-edition will still be available as open-source product at no charge.Using the official network modules with Ansible 2.0.X2016-03-13T00:00:00+00:002016-03-13T00:00:00+00:00http://blog.simonmetzger.de/2016/03/ansiblem-network-modules<p>About one month ago Ansible released official core modules to work with network equipment. With Ansible 2.1 they will be included in the stable-release. Their functionality is described in the <a href="http://docs.ansible.com/ansible/list_of_network_modules.html" target="_blank">documentation</a>. Different vendors and platforms like Cisco IOS, Cisco NX-OS, Cisco IOS-XR, Juniper JUNOS or Arista EOS are supported.</p>
<p>Typically there are 3 types of modules available per platform. The platform I use in this blog post will be Cisco IOS:</p>
<ul>
<li>command: Run arbitrary commands on IOS devices.</li>
<li>config: Manage configuration sections.</li>
<li>template: Manage device configurations.</li>
</ul>
<p>At the moment of writing this post there is a special network branch which includes the new Ansible network modules. The branch is called <strong>stable-2.0-network</strong>. In this post I want to show how a Cisco-switch can be managed with this new possibilites. My goal is to ensure the basic configuration of the switch (like hostname, logging, ntp, snmp…) with the help of configuration templates. As a templating system <a href="http://jinja.pocoo.org/" target="_blank">jinja2</a> is used. The ansible playbook ensures that the templates are used for configuring the switch, and writes the config if something changed. As an example I will create a role for common-configurations like the hostname and a role for the logging-configurations.</p>
<p>To get started you have to download and install Ansible with the branch <strong>stable-2.0-network</strong>:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone http://github.com/ansible/ansible.git
$ cd ansible
$ git checkout stable-2.0-network
$ git submodule init
$ git submodule update
$ pip uninstall ansible
$ python setup.py install
$ ansible --version
ansible 2.0.1.0 (stable-2.0-network 49f7e6a12c) last updated 2016/03/01 20:57:24 (GMT +200)
lib/ansible/modules/core: (detached HEAD 72540d1f0c) last updated 2016/03/01 20:57:27 (GMT +200)
lib/ansible/modules/extras: (detached HEAD 51cddf2b35) last updated 2016/03/01 20:57:28 (GMT +200)
</code></pre>
</div>
<p>Now we are ready to create our project structure:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mkdir ansible-blog-ios-network-modules
$ cd ansible-blog-ios-network-modules
$ mkdir group_vars
$ mkdir roles
$ cd roles
$ mkdir common
$ mkdir logging
$ mkdir writecfg
$ cd common
$ mkdir meta
$ mkdir tasks
$ mkdir templates
$ cd ../logging
$ mkdir meta
$ mkdir tasks
$ mkdir templates
$ cd ../writecfg
$ mkdir handlers
</code></pre>
</div>
<p>Next let us create the needed files.</p>
<p><img src="/images/directory-listing.png" alt="directory-listing" title="needed files in the ansible-project" /></p>
<p>This is the yaml-file we are using as initial playbook.</p>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat site.yml</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">Ensure basic configuration of switches</span>
<span class="s">connection</span><span class="pi">:</span> <span class="s">local</span>
<span class="s">hosts</span><span class="pi">:</span> <span class="s">all</span>
<span class="s">gather_facts</span><span class="pi">:</span> <span class="s">false</span>
<span class="s">roles</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">role</span><span class="pi">:</span> <span class="s">common</span>
<span class="s">tags</span><span class="pi">:</span> <span class="s">common</span>
<span class="pi">-</span> <span class="s">role</span><span class="pi">:</span> <span class="s">logging</span>
<span class="s">tags</span><span class="pi">:</span> <span class="s">logging</span>
</code></pre>
</div>
<p>In our inventory only one device is defined for testing. In a real world scenario this can also be a dynamic inventory like the one I mentioned in my last <a href="http://blog.simonmetzger.de/2016/02/apicem-ansible-dynamic-inventory/" target="_blank">blog post</a>.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat inventory
[all]
switch-01 device_ip=192.168.0.200
</code></pre>
</div>
<p>We need some variables for our configuration templates and for connecting to the device.</p>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat group_vars/all.yml</span>
<span class="nn">---</span>
<span class="s">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">username"</span>
<span class="s">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">password"</span>
<span class="s">secret</span><span class="pi">:</span> <span class="s2">"</span><span class="s">password"</span>
<span class="s">logserver</span><span class="pi">:</span> <span class="s">192.168.1.5</span>
<span class="s">mgmt_vlanid</span><span class="pi">:</span> <span class="s">1</span>
</code></pre>
</div>
<p>The <em>common/meta/main.yml</em> of our common-role has the writecfg-role as a dependency. Only if the common-role changes something on the device also the writecfg-role will be called. I use this to ensure that the configuration of the devices only gets written, if there were some changes in the running-configuration.</p>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat roles/common/meta/main.yml</span>
<span class="nn">---</span>
<span class="s">dependencies</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">role</span><span class="pi">:</span> <span class="s">writecfg</span>
</code></pre>
</div>
<p>The <em>common/tasks/main.yml</em> executes a task in which the ios_template module of Ansible is used to ensure that the configuration defined in common.j2 exists on the IOS-device.</p>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat roles/common/tasks/main.yml</span>
<span class="nn">---</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">ensure common configuration exists</span>
<span class="s">ios_template</span><span class="pi">:</span>
<span class="s">src</span><span class="pi">:</span> <span class="s">common.j2</span>
<span class="s">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">username</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">password</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_ip</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">notify</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">write config</span>
</code></pre>
</div>
<p>Thats the <em>jinja2-template</em> which contains the common-configurations we want to ensure on our devices. For testing purposes this is only the hostname at the moment.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat roles/common/templates/common.j2
hostname {{ ansible_host }}
</code></pre>
</div>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat roles/logging/meta/main.yml</span>
<span class="nn">---</span>
<span class="s">dependencies</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">role</span><span class="pi">:</span> <span class="s">writecfg</span>
</code></pre>
</div>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat roles/logging/tasks/main.yml</span>
<span class="nn">---</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">ensure logging configuration exists</span>
<span class="s">ios_template</span><span class="pi">:</span>
<span class="s">src</span><span class="pi">:</span> <span class="s">logging.j2</span>
<span class="s">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">username</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">password</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_ip</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">notify</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">write config</span>
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>$ cat roles/logging/templates/logging.j2
logging buffered 128000
no logging console
no logging monitor
logging {{ logserver }}
logging source-interface Vlan{{ mgmt_vlanid }}
</code></pre>
</div>
<p>This is the handler of the writecfg-role. It only gets called by the other roles if there were some changes triggered by the common- or logging-role.</p>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="s">$ cat roles/writecfg/handlers/main.yml</span>
<span class="nn">---</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">write config</span>
<span class="s">ios_command</span><span class="pi">:</span>
<span class="s">commands</span><span class="pi">:</span> <span class="s">write</span>
<span class="s">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">username</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">password</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_ip</span><span class="nv"> </span><span class="s">}}"</span>
</code></pre>
</div>
<p>Now lets execute this by calling the initial playbook <em>site.yml</em>.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>» ansible-playbook site.yml -i inventory
PLAY [Ensure basic configuration of switches switches] *************************
TASK [common : ensure common configuration exists] *****************************
ok: [switch-01]
TASK [logging : ensure logging configuration exists] ***************************
ok: [switch-01]
PLAY RECAP *********************************************************************
switch-01 : ok=2 changed=0 unreachable=0 failed=0
</code></pre>
</div>
<p>Ok all configurations seems to already have been on the device. Lets change the logserver-ip in <em>group-vars/all.yml</em> to 192.168.0.20 and execute it again.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>» ansible-playbook site.yml -i inventory
PLAY [Ensure basic configuration of switches switches] *************************
TASK [common : ensure common configuration exists] *****************************
ok: [switch-01]
TASK [logging : ensure logging configuration exists] ***************************
changed: [switch-01]
RUNNING HANDLER [writecfg : write config] **************************************
ok: [switch-01]
PLAY RECAP *********************************************************************
switch-01 : ok=3 changed=1 unreachable=0 failed=0
</code></pre>
</div>
<p>Notice that the status of logging is <em>changed</em> and the handler for writing the configuration gets called. Now two logging-servers are configured on your device. If you want to change the logging server you can delete it before adding a new one or execute a <em>cleanup-role</em> at the end of all roles, which deletes all configuration-snippets you dont want to use anymore.
At this point you could also add other roles like ntp, aaa, ssh… be creative. If your templates are cleverly designed it would be enough to change the variables in <em>group-vars/all.yml</em> to the new values. With this method a basic-configuration on all your devices could be ensured very easy.</p>smnmtzgrAbout one month ago Ansible released official core modules to work with network equipment. With Ansible 2.1 they will be included in the stable-release. Their functionality is described in the documentation. Different vendors and platforms like Cisco IOS, Cisco NX-OS, Cisco IOS-XR, Juniper JUNOS or Arista EOS are supported.Cisco APIC-EM as an Ansible dynamic-inventory2016-02-10T00:00:00+00:002016-02-10T00:00:00+00:00http://blog.simonmetzger.de/2016/02/apicem-ansible-dynamic-inventory<p>APIC-EM is a product of Cisco which delivers SDN to the Enterprise WAN, Campus and Access networks. It provides centralized automation of policy-based application profiles. Through programmability, automated network control helps IT rapidly respond to new business opportunities. APIC-EM can be downloaded at no additional charge using a free membership to the <a href="https://developer.cisco.com" target="_blank">Cisco DevNet community</a>. On the DevNet community there are also a lot of tutorials and sandbox learning labs. <br />
It can discover devices in your network using ip-ranges or CDP-neighbors.</p>
<p><img src="/images/apicem-discovery.jpg" alt="discovery" title="device discovery in APIC-EM" /></p>
<p>After discovering devices APIC-EM manages the inventory and triggers data collections periodically. For discovering the so called southbound-API of the APIC is used to connect via CLI to the devices and gather facts. In the future there will be additional southbound mechanisms for connecting to devices (like REST or NX-API). <br />
Every device can be tagged with <em>tags</em> or marked with a <em>location</em>. In our example we will set only the location of some devices to the value <strong>location1</strong>.</p>
<p><img src="/images/apicem-location.jpg" alt="location" title="set location in APIC-EM" /></p>
<p>If you like to manage network-devices with Ansible and your network consists primarily of Cisco equipment APIC-EM is perfect as a dynamic inventory. My python script uses code snippets of <a href="https://github.com/joelwking/ansible-apic-em" target="_blank">this project</a> but does not function as a separate Ansible module. It is a regular Ansible dynamic inventory which can be specified as an inventory when calling playbooks. A dynamic inventory is built with the help of a python script and has to return <a href="http://docs.ansible.com/ansible/developing_inventory.html" target="_blank">a specially formatted</a> JSON hash/dictionary of all the groups to be managed. So the script has to return something like this to the Ansible-playbook (This is only a static example. Normally the python script creates this structure dynamically after querying APIC-EM):</p>
<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="p">{</span>
<span class="s">"location1"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s">"hosts"</span> <span class="p">:</span> <span class="p">[</span> <span class="s">"device1"</span><span class="p">,</span> <span class="s">"device2"</span><span class="p">,</span> <span class="s">"device3"</span> <span class="p">]</span>
<span class="p">},</span>
<span class="s">"location2"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s">"hosts"</span> <span class="p">:</span> <span class="p">[</span> <span class="s">"device4"</span> <span class="p">]</span>
<span class="p">},</span>
<span class="s">"location3"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s">"hosts"</span> <span class="p">:</span> <span class="p">[</span> <span class="s">"device5"</span> <span class="p">]</span>
<span class="p">},</span>
<span class="s">"_meta"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s">"hostvars"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s">"device1"</span> <span class="p">:</span> <span class="p">{</span> <span class="s">"device_ip"</span> <span class="p">:</span> <span class="mf">192.168</span><span class="o">.</span><span class="mf">2.1</span> <span class="p">},</span>
<span class="s">"device2"</span> <span class="p">:</span> <span class="p">{</span> <span class="s">"device_ip"</span> <span class="p">:</span> <span class="mf">192.168</span><span class="o">.</span><span class="mf">2.2</span> <span class="p">},</span>
<span class="s">"device3"</span> <span class="p">:</span> <span class="p">{</span> <span class="s">"device_ip"</span> <span class="p">:</span> <span class="mf">192.168</span><span class="o">.</span><span class="mf">2.3</span> <span class="p">},</span>
<span class="s">"device4"</span> <span class="p">:</span> <span class="p">{</span> <span class="s">"device_ip"</span> <span class="p">:</span> <span class="mf">192.168</span><span class="o">.</span><span class="mf">2.4</span> <span class="p">},</span>
<span class="s">"device5"</span> <span class="p">:</span> <span class="p">{</span> <span class="s">"device_ip"</span> <span class="p">:</span> <span class="mf">192.168</span><span class="o">.</span><span class="mf">2.5</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>Our dynamic inventory script will query the APIC-EM REST-API (the northbound API) and the received results will be formatted correctly to an interpretable JSON hash/dictionary. A very cool thing about the script is the fact that it only returns devices with the <em>reachable</em> state. So you can be sure that the returned devices were reachable a few minutes ago by APIC-EM. The script can be found on my <a href="https://github.com/smnmtzgr/ansible-apicem-dynamic-inventory" target="_blank">github project ansible-apicem-dynamic-inventory</a>.
<img src="/images/ansible-apic-em.jpg" alt="dynamic-inventory" title="dynamic ansible inventory" /></p>
<p>The playbook <strong>regex.yml</strong> we are executing uses the dynamic inventory and as an example calls the module <strong>ntc_show_command</strong> (with <em>show clock</em>) which is part of the amazing project <a href="https://github.com/networktocode/ntc-ansible" target="_blank">ntc-ansible</a>. The module connects to network-devices with the help of <a href="https://github.com/ktbyers/netmiko" target="_blank">netmiko</a>, executes <em>show clock</em>, parses the output with <a href="https://code.google.com/archive/p/textfsm/" target="_blank">googles textfsm</a> and returns it to Ansible where we can now use the JSON-structured data. The used playbook <strong>regex.yml</strong> outputs the data for demonstration purposes with the debug command:</p>
<div class="language-yaml highlighter-rouge"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">GET STRUCTURED DATA BACK FROM CLI DEVICES</span>
<span class="s">hosts</span><span class="pi">:</span> <span class="s">all</span>
<span class="s">connection</span><span class="pi">:</span> <span class="s">local</span>
<span class="s">gather_facts</span><span class="pi">:</span> <span class="s">False</span>
<span class="s">tasks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">GET DATA</span>
<span class="s">ntc_show_command</span><span class="pi">:</span>
<span class="s">connection</span><span class="pi">:</span> <span class="s">ssh</span>
<span class="s">platform</span><span class="pi">:</span> <span class="s">cisco_ios</span>
<span class="s">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">show</span><span class="nv"> </span><span class="s">clock'</span>
<span class="s">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_ip</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">username</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">secret</span><span class="nv"> </span><span class="s">}}"</span>
<span class="s">register</span><span class="pi">:</span> <span class="s">mydata</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">DISPLAY mydata</span>
<span class="s">debug</span><span class="pi">:</span> <span class="s">msg="{{ mydata.response }}"</span>
</code></pre>
</div>
<p>Username and password are variables of a <a href="http://docs.ansible.com/ansible/playbooks_vault.html" target="_blank">vault-file</a> I use with my Ansible-scripts. <strong>device_ip</strong> is one of the variables received from APIC-EM. The following example shows how to call the Ansible-playbook and displays the debugging output after using the playbook with APIC-EM as dynamic inventory. I limit the hosts to our defined <strong>location1</strong> which currently contains 3 devices:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>» ansible-playbook regex.yml -i dynamic-inventory --limit "location1" --ask-vault
Vault password:
PLAY [GET STRUCTURED DATA BACK FROM CLI DEVICES] *******************************
TASK [GET DATA] ****************************************************************
ok: [device1]
ok: [device2]
ok: [device3]
TASK [DISPLAY mydata] **********************************************************
ok: [device1] =&gt; {
"msg": [
{
"day": "9",
"dayweek": "Tue",
"month": "Feb",
"time": "21:52:50.265",
"timezone": "MET",
"year": "2016"
}
]
}
ok: [device2] =&gt; {
"msg": [
{
"day": "9",
"dayweek": "Tue",
"month": "Feb",
"time": "21:52:50.332",
"timezone": "MET",
"year": "2016"
}
]
}
ok: [device3] =&gt; {
"msg": [
{
"day": "9",
"dayweek": "Tue",
"month": "Feb",
"time": "21:52:50.271",
"timezone": "MET",
"year": "2016"
}
]
}
PLAY RECAP *********************************************************************
device1 : ok=2 changed=0 unreachable=0 failed=0
device2 : ok=2 changed=0 unreachable=0 failed=0
device3 : ok=2 changed=0 unreachable=0 failed=0
</code></pre>
</div>
<p><strong>Summarized my project structure is:</strong></p>
<ul>
<li>The dynamic inventory script: <em>dynamic-inventory/dynamic.py</em></li>
<li>The vault yaml with user/password for my IOS devices: <em>group-vars/all/vault.yml</em></li>
<li>The ntc_show_command Ansible module: <em>library/ntc_show_command.py</em></li>
<li>The ntc_template for show clock: <em>ntc_templates/cisco_ios_show_clock.template</em></li>
<li>The playbook: <em>regex.yml</em></li>
</ul>smnmtzgrAPIC-EM is a product of Cisco which delivers SDN to the Enterprise WAN, Campus and Access networks. It provides centralized automation of policy-based application profiles. Through programmability, automated network control helps IT rapidly respond to new business opportunities. APIC-EM can be downloaded at no additional charge using a free membership to the Cisco DevNet community. On the DevNet community there are also a lot of tutorials and sandbox learning labs. It can discover devices in your network using ip-ranges or CDP-neighbors.