In a previous blog, I hinted at a network configuration life cycle management library called hierarchical_configuration. I’ve been meaning to write about it for a while, but we’ve been super busy at work. I also wanted to ensure that we get our latest version of the library out in the public for general consumption before I wrote about it.

As your fleet routers and switches grow, it becomes pretty natural to place these devices into a set of categories. For example, core, aggregation, and access. Each of these categories typically have a standard configuration. Hopefully each of these standard configurations exists as templates, so that new deployments can be rolled out quickly. But, what about making changes to the templates? Do you make changes to these templates, then continue to roll them out to new deployments, leaving the existing install base with an outdated configuration? Or do you return to the install base and remediate the devices with updated configurations? What if you have thousands of devices? This has been a problem that my colleagues and I have set out to solve. This is how hierarchical_configuration has evolved.

So, what is hierarchical_configuration? hierarchical_configuration is a python library that allows you to compare the running configuration and the intended configuration from a network device, then generate a set of commands that it will bring the network device into compliance with the intended configuration. hierarchical_configuration also has an extensive configuration file, so that you can define how specific commands or sections of commands get remediated.

Most utilities that performs a similar function as hierarchical_configuration, apply command remediation by negating a command, then applying the new command. For instance, if you wanted to change the interface description of an interface, most utilities will do something like:

1

2

3

interfaceEthernet0/1

no description ROUTER1

description ROUTER2

That works, but it’s wasteful on CPU cycles, which slows down the over all application run time when you are attempting to apply interface descriptions to thousands of interfaces. What if the command was something that could be impactful, if it were negated? Maybe something like changing ‘transport input ssh telnet’ to ‘transport input ssh’, under your line vty? Negating the command could potentially cause you to lose management access.

hierarchical_configuration gives you several configuration options for dealing with such scenarios. You define those as a YAML file under hier_options. Here is a sample of hier_options:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

---

hier_tags:

-lineage:

-startswith:

-ip access-list extended TEST

-no ip access-list extended TEST

add_tags:NEW_ACL

-lineage:

-startswith:interface

-startswith:ip access-group TEST

add_tags:NEW_ACL

-lineage:

-startswith:

-ip domain-name

-no ip domain-name

-ip domain-lookup

-no ip domain-lookup

-logging

-no logging

-snmp-server

-no snmp-server

-ntp server

-no ntp server

-ip tcp path-mtu-discovery

-ip access-list resequence

add_tags:safe

-lineage:

-startswith:line

-startswith:exec-timeout

add_tags:safe

-lineage:

-startswith:interface

-startswith:ip access-group

add_tags:unsafe

-lineage:

-startswith:router ospf

new_in_config:false

-startswith:

-ispf

-nsf

-log

add_tags:safe

-lineage:

-startswith:router ospf

new_in_config:false

-startswith:

-network

-area

add_tags:unsafe

hier_options:

#Indicates the style of the configuration

style:ios

#if there is a delta, overwrite these parents instead of one of their children

sectional_overwrite:

-lineage:

-startswith:ipv6 access-list

ordering:

-lineage:

-startswith:

-ip access-list

-access-list

order:300

-lineage:

-startswith:

-tacacs-server host

order:400

-lineage:

-startswith:interface

-startswith:

-ip access-group

-no ip access-group

order:400

-lineage:

-startswith:

-no ip access-list

-no access-list

-no ip prefix-list

-no tacacs-server host

order:590

-lineage:

-contains:ip spd queue min-threshold

order:601

-lineage:

-contains:ip spd queue max-threshold

order:602

sectional_overwrite_no_negate:[]

#adds +1 indent to lines following start_expression and removes the +1 indent for lines following end_expression

indent_adjust:[]

parent_allows_duplicate_child:[]

sectional_exiting:

-lineage:

-startswith:router bgp

-startswith:template peer-policy

exit_text:exit-peer-policy

-lineage:

-startswith:router bgp

-startswith:template peer-session

exit_text:exit-peer-session

-lineage:

-startswith:router bgp

-startswith:address-family

exit_text:exit-address-family

#substitions against the full multi-line config text

full_text_sub:

-search:'banner exec (\S+)\n(.*\n)+\\1\s*\n'

replace:''

-search:'banner motd (\S+)\n(.*\n)+\\1\s*\n'

replace:''

#substitions against each line of the config text

per_line_sub:

-search:^Building configuration.*

replace:''

-search:^Current configuration.*

replace:''

-search:^!Last configuration change.*

replace:''

-search:^!NVRAM config last updated.*

replace:''

-search:^ntp clock-period.*

replace:''

-search:.*message-digest-key.*

replace:''

-search:^version.*

replace:''

-search:.*password.*

replace:''

-search:^logging event link-status$

replace:''

-search:^logging event subif-link-status$

replace:''

-search:^\s*ipv6 unreachables disable$

replace:''

-search:^\s*key.*

replace:''

-search:^end$

replace:''

-search:'^\s*[#!].*'

replace:''

-search:^no ip address

replace:''

-search:^exit-peer-policy

replace:''

-search:^exit-peer-session

replace:''

-search:^exit-address-family

replace:''

-search:^crypto key generate rsa general-keys.*$

replace:''

-search:.*key-string.*

replace:''

idempotent_commands_blacklist:[]

#These commands do not require negation, they simply overwrite themselves

idempotent_commands:

-lineage:

-startswith:hostname

-lineage:

-startswith:logging source-interface

-lineage:

-startswith:interface

-startswith:description

-lineage:

-startswith:interface

-startswith:ip address

-lineage:

-startswith:line vty

-startswith:

-transport input

-access-class

-ipv6 access-class

-lineage:

-startswith:interface

-re_search:standby\d+(priority|authentication md5)

-lineage:

-startswith:router bgp

-startswith:bgp router-id

-lineage:

-startswith:router ospf

-startswith:router-id

-lineage:

-startswith:ipv6 router ospf

-startswith:router-id

-lineage:

-startswith:router ospf

-startswith:log-adjacency-changes

-lineage:

-startswith:ipv6 router ospf

-startswith:log-adjacency-changes

-lineage:

-startswith:router bgp

-re_search:neighbor\S+description

-lineage:

-startswith:snmp-server community

-lineage:

-startswith:snmp-server location

-lineage:

-equals:line con0

-startswith:exec-timeout

-lineage:

-startswith:interface

-startswith:ip ospf message-digest-key

-lineage:

-startswith:logging buffered

-lineage:

-startswith:tacacs-server key

-lineage:

-startswith:logging facility

-lineage:

-startswith:vlan internal allocation policy

#Default when expression: list of expressions

negation_default_when:[]

#- lineage:

# - startswith: interface

#Negate substitutions: expression -> negate with

negation_negate_with:[]

#- lineage:

# - startswith: interface

# use: command

Lets break down the individual sections of hier_options. The first section is ‘sectional_overwrite’. sectional_overwrite does exactly like it sounds. It over-writes an entire section of configuration if there is a change. In the example, it tags ipv6 access-lists as a section of code that should use sectional_overwrite. If any changes are made to the intended configuration for ipv6 access-list, then hierarchical_configuration over writes the entire section of configuration, rather than targeting individual lines of children configuration in the section.

The next section is ‘ordering’. Ordering is a very handy configuration option. It allows you to weight the order in which commands are presented in hierarchical_configuration. The default weight is 500. The smaller the number, the higher up in the configuration the commands are presented. While the commands tagged with larger numbers are presented lower in the configuration.

For instance, assume that you have an access-list called TEST, which is applied to Ethernet0/1:

1

2

3

4

ip access-list TEST

permit ip any host1.1.1.1

interfaceEthernet0/1

ip access-group TEST in

Let’s say that you want to create a new access-list called TESTING and apply it to Ethernet0/1, rendering the access-list TEST as un-needed. When you go to apply the configuration, you don’t want to remove the access-list TEST before you’ve created access-list TESTING and applied it to interface Ethernet0/1. Doing so may impact traffic that is flowing across the interface. The preferable order of operation is:

Create the new access-list

Apply the new access-list to the interface

Remove the old access-list

To do so, you will want to the command ‘no ip access-list’ closer to the bottom of the list of commands. You would do this, by setting the order of the negation of ip access-list higher than 500.

1

2

3

4

5

ordering:

-lineage:

-startswith:

-no ip access-list

order:525

In this example, any command generated that starts with ‘no ip access-list’ gets tagged with an order of 525, which moves that section of configuration lower into the generated config. The generated configuration would look like:

1

2

3

4

5

ip access-list TESTING

permit ip any host1.1.1.1

interfaceEthernet0/1

ip access-group TESTING in

no ip access-list TEST

The next two sections are ‘full_text_sub’ and ‘per_line_sub’. When you pull a running config from a device, it will typically contain some fluff, such as:

1

2

3

4

5

6

7

#sh run

Building configuration...

Current configuration:9574bytes

!

!Last configuration change at19:51:43CST Mon Apr252016by jtdub

version15.1

That kind of text is just back ground noise, when we are attempting to determine the difference between the running config and the intended config. So, per_line_sub attempts to resolve that by ignoring it when comparing the configurations.

1

2

3

per_line_sub:

-search:^Building configuration.*

replace:''

As you can see, it will find any line that contains ‘Building configuration” and replace it with no data, effectively deleting the line. full_text_sub performs a similar task, but for entire sections of code. In our example, we ignore the banners on the device, as those can have information that is unique to the device.

1

2

3

4

5

full_text_sub:

-search:'banner exec (\S+)\n(.*\n)+\\1\s*\n'

replace:''

-search:'banner motd (\S+)\n(.*\n)+\\1\s*\n'

replace:''

Hierarchical_configuration understands sections of config. It does this by assuming that a line of configuration that doesn’t have any indentation to be a parent and any lines of configuration under the parent that have indentation are children of the parent. When the config reaches another line without indentation, the section of configuration ends. An example would be an interface configuration.

With ‘sectional_exiting’, you can define sections of configuration that have sub-children of children, as explained above.

1

2

3

4

5

6

7

8

9

10

11

12

13

sectional_exiting:

-lineage:

-startswith:router bgp

-startswith:template peer-policy

exit_text:exit-peer-policy

-lineage:

-startswith:router bgp

-startswith:template peer-session

exit_text:exit-peer-session

-lineage:

-startswith:router bgp

-startswith:address-family

exit_text:exit-address-family

‘idempotent_commands’ is the section where you define what should be over-written, rather than negated, then re-applied with new configuration. Commands such as hostname, description, ip address, etc should all be over-written, rather than negated.

1

2

3

4

5

6

7

8

9

10

11

idempotent_commands:

-lineage:

-startswith:hostname

-lineage:

-startswith:logging source-interface

-lineage:

-startswith:interface

-startswith:description

-lineage:

-startswith:interface

-startswith:ip address

Another very handy set of options is command tagging, which resides in a different config section from hier_options, called hier_tags. Being able to tag commands allows you to generate remediation commands which target very specific commands, such as creating a new access-list, applying it, then removing the old access-list. We’ll continue to use the examples that I’ve have above with replacing the access-list TEST with TESTING. The first thing we need to do is set up our tagging, which will look like:

1

2

3

4

5

6

7

8

9

10

hier_tags:

-lineage:

-startswith:

-ip access-list extended TEST

-no ip access-list extended TEST

add_tags:NEW_ACL

-lineage:

-startswith:interface

-startswith:ip access-group TEST

add_tags:NEW_ACL

Assume that your running config is:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

hostname router

!

interfaceEthernet0/1

ip address10.0.0.0/31

ip access-group TEST in

!

interfaceEthernet0/2

ip address10.0.0.2/31

!

ip access-list extended TEST

permit ip any host1.1.1.1

permit ip any host4.4.4.4

permit ip any host5.5.5.5

permit ip any host6.6.6.6

!

router ospf1

network10.0.0.00.0.255.255area0

!

snmp-server community private

!

ntp server11.22.33.44

and your intended config is:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

hostname router

!

interfaceEthernet0/1

ip address10.0.0.0/31

ip access-group TESTING in

!

interfaceEthernet0/2

ip address10.0.0.2/31

ip access-group SOMEACL in

ipv6 enable

ipv6 filter TEST out

!

ip access-list extended TESTING

permit ip any host1.1.1.1

permit ip any host4.4.4.4

permit ip any host5.5.5.5

permit ip any host6.6.6.6

!

ip access-list extended SOMEACL

permit ip any host7.7.7.7

!

ipv6 access-list TEST

permit ipv6 any2001::1/128

!

router ospf1

network10.0.0.00.0.255.255area0

!

snmp-server community private

!

ntp server11.22.33.44

We can use hierarchical_configuration compare the two configurations, make the appropriate configuration tag (NEW_ACL), then spit out a configuration plan based on the NEW_ACL tag.

Let’s break down the script. The very first thing we do is import yaml and hierarchical_configuration:

1

2

3

import yaml

from hierarchical_configuration import HierarchicalConfiguration

Next, we read the hierarchical_configuration config file and define the options and tags variables:

1

2

3

config_options=yaml.load(open('main.yml','r'))

hier_tags=config_options['hier_tags']

hier_options=config_options['hier_options']

Now, we define an instance of hierarchical_configuration for the running config and load the running config from a file:

1

2

3

running_config_hier=HierarchicalConfiguration(

options=hier_options)

running_config_hier.from_file('running.config')

Then, we do the same for the intended config:

1

2

3

compiled_config_hier=HierarchicalConfiguration(

options=hier_options)

compiled_config_hier.from_file('compiled.config')

Once that is done, we can perform the comparison:

1

2

remediation_config_hier=compiled_config_hier.deep_diff_tree_with(

running_config_hier)

Now, we load the ordering, sectional exiting, and tags options:

1

2

3

remediation_config_hier.set_order_weight()

remediation_config_hier.add_sectional_exiting()

remediation_config_hier.add_tags(hier_tags)

At this point, everything is loaded into memory. The next two portions of code are simply to have a visual of what is happening. The first portion is a for-loop, which displays the raw list of dictionaries:

1

2

3

print('\nPython dictionary of the comparison results\n')

forcommand inremediation_config_hier.to_detailed_ouput():

print(command)

Finally, the finished product. Generating a config plan, based on the NEW_ACL tag.

1

2

3

4

print('\nConfig plan based on NEW_ACL tag\n')

forcommand inremediation_config_hier.to_detailed_ouput():

if'NEW_ACL'incommand['tags']:

print(command['text'])

As you can see, hierarchical configuration is a very powerful life-cycle management tool for network gear. We’ve been using it successfully on IOS, IOS-XR, IOS-XE, NX-OS, and EOS devices. It has made our work less risky – from an outage perspective – more consistent, and allows us to automate and move faster than we have in previous years.

I’ve implemented some new changes to pyMultiChange and netlib. The biggest change affects both netlib and pyMultiChange. In netlib, I ripped out both the ‘simple_creds’ and ‘simple_yaml’ methods, as both stored user credentials in plain text on the computer that you used them on.

Instead, utilizing the ‘keyring’ python module, I created a new class called ‘KeyRing’. Using ‘KeyRing’ is simple. It implements three methods. These methods are ‘get_creds’, ‘set_creds’, and ‘del_creds’.

‘get_creds’ will attempt to retrieve the credentials for a username, if the credentials exist. If they do not exist, ‘get_creds’ will automatically call the ‘set_creds’ method.

1

2

3

4

5

6

7

8

9

10

11

&gt;&gt;&gt;from netlib.user_keyring import KeyRing

&gt;&gt;&gt;creds=KeyRing(username='testuser')

&gt;&gt;&gt;creds.get_creds()

No credentials keyring exist.Creating newcredentials.

Enter your user password:

Confirm your user password:

Enter your enable password:

Confirm your enable password:

&gt;&gt;&gt;print creds.get_creds()

{'username':'testuser','enable':u'test','password':u'testpass'}

&gt;&gt;&gt;

‘set_creds’ does exactly what it sounds like. It allows you to set your credentials. If no user credentials exist, it creates a new keyring. However, if user credentials do exist, it over-writes the credentials.

1

2

3

4

5

6

7

&gt;&gt;&gt;creds.set_creds()

Enter your user password:

Confirm your user password:

Enter your enable password:

Confirm your enable password:

&gt;&gt;&gt;print creds.get_creds()

{'username':'testuser','enable':u'enable123','password':u'test123'}

Finally, ‘del_creds’ deletes a user’s credentials from an existing keyring.

1

2

3

&gt;&gt;&gt;creds.del_creds()

Enter your user password:

Deleting keyring credentials fortestuser

The ‘keyring‘ python library utilizes your operating systems native methods for storing passwords. For Example, in Mac OS X, it will utilize the KeyChain functionality, In Linux, it will use dbus, and in Microsoft Windows, it utilizes the Credential Vault. There are also methods available for you to create your own backend. Much more secure than the ~/.tacacslogin or ~/.tacacs.yml files that used to be created from the old methods.

As expected, this change was implemented into pyMultiChange. Doing so, required that new command line arguments needed to be implemented. There are actually several new command line arguments, since the last time that I wrote about pyMultiChange. I’ll go over them here.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

usage:multi_change.py[-h]-uUSERNAME[--delete-creds[DELETE_CREDS]]

[--set-creds[SET_CREDS]][-dDEVICES][-cCOMMANDS]

[-s[SSH]][-t[TELNET]][-o[OUTPUT]][-v[VERBOSE]]

[--delay DELAY][--buffer BUFFER]

[--threaded[THREADED]][-mMAXTHREADS]

Managing network devices with python

optional arguments:

-h,--help show thishelp message andexit

-uUSERNAME,--username USERNAME

Specify your username.

--delete-creds[DELETE_CREDS]

Delete credentials from keyring.

--set-creds[SET_CREDS]

set keyring credentials.

-dDEVICES,--devices DEVICES

Specifiesahost file

-cCOMMANDS,--commands COMMANDS

Specifiesacommands file

-s[SSH],--ssh[SSH]

Default:Usethe SSH protocol

-t[TELNET],--telnet[TELNET]

Usethe Telnet protocol

-o[OUTPUT],--output[OUTPUT]

Verbose command output

-v[VERBOSE],--verbose[VERBOSE]

Debug script output

--delay DELAY Change the defaultdelay exec between commands

--buffer BUFFER Change the defaultSSH output buffer

--threaded[THREADED]

Enable process threading

-mMAXTHREADS,--maxthreads MAXTHREADS

Define the maximum number of threads

The first, is that multi_change.py now requires that you specify a username for all actions. This allows multi_change.py to interact with the keyring to set, extract, and delete credentials.

1

2

3

4

5

6

usage:multi_change.py[-h]-uUSERNAME[--delete-creds[DELETE_CREDS]]

[--set-creds[SET_CREDS]][-dDEVICES][-cCOMMANDS]

[-s[SSH]][-t[TELNET]][-o[OUTPUT]][-v[VERBOSE]]

[--delay DELAY][--buffer BUFFER]

[--threaded[THREADED]][-mMAXTHREADS]

multi_change.py:error:argument-u/--username isrequired

When you specify a username, multi_change.py immediately attempts to extract the credentials for the user. If they don’t exist, multi_change.py will prompt you to set the credentials.

1

2

3

4

5

6

$multi_change.py-utestuser

No credentials keyring exist.Creating newcredentials.

Enter your user password:

Confirm your user password:

Enter your enable password:

Confirm your enable password:

You can also utilize the ‘–set-creds’ command line argument to either set credentials for a new user or over-write the credentials for an existing user.

1

2

3

4

5

$multi_change.py-utestuser--set-creds

Enter your user password:

Confirm your user password:

Enter your enable password:

Confirm your enable password:

Like wise, you can use the ‘–delete-creds’ to delete existing creds.

1

2

3

$multi_change.py-utestuser--delete-creds

Enter your user password:

Deleting keyring credentials fortestuser

Beyond that, the option to utilize threading was created. With the threading ability, you also get the ability to specify the number of threads and the delay factor between command execution.

For example, the below series of command line arguments enable threading, utilizing 50 threads, create a delay factor of 5 seconds, and will display the command output.

This is very handy for running a common command set across a large number of devices very quickly.

Beyond that, there are a few under the hood enhancements.

Protocol failover will no longer happen. Meaning that if a device fails to login via SSH, it will no longer failover to attempt to login via telnet and vise versa.

Login failures are logged in a file called ‘failure.log’. This file is created in the local folder that you’re running ‘multi_change.py’ in.

multi_change.py will now only read the commands file once, rather than reading it for every device that it attempts to make changes on.

pyMultiChange and netlib are now python installable packages. Meaning that you can run their ‘setup.py’ files and they will be installed as native python packages, allowing them to be called from anywhere on the OS.