How to manage access in a web-scale data center

One of the consistent questions that arises during the web-scale transition is the impact of managed access to networking infrastructure. How do we take traditional management techniques and adapt them to the new operational paradigm of web-scale networking, where automation drives the majority of changes and the infrastructure is treated as a holistic entity rather than node-by-node?

Local privileges

In the most basic way, we can migrate existing workflows to the new paradigm. Though inefficient, the old way of doing things still works with the new web-scale paradigm. The easiest way to do this is to restrict access to your switches using local privileges. In Linux, users are controlled using the adduser command, and the permissions for that user are controlled using the chmod commands.

After a user is created, the new user is only part of their own group.

Privilege level

This brings us to the first major paradigm shift. As the industry moves away from proprietary operating systems and closed CLIs, the networking managed access model needs to shift and adapt more closely to the server management model. Instead of treating managed access as a series of commands that are or are not allowed to execute, the restriction in privilege migrates to the files and programs themselves that are being executed.

Linux permissions aren’t the same as traditional networking permissions. We can create the same paradigm, but each file has its own read, write and execute functionality. It isn’t about command authorization, but rather about file authorization.

For example, the two most commonly edited files on Cumulus Linux are /etc/network/interfaces and /etc/frr/frr.conf. The former is owned by root and the later is owned by frr:

These files can only be edited by those respective users and groups. In order to edit these files, I have to use elevated privileges using the sudo command:

test@leaf02:/etc/network$ sudo nano interfaces
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for test:
Sorry, user test is not allowed to execute '/usr/bin/nano interfaces' as root on leaf02.
test@leaf02:/etc/network$
test@leaf02:/etc/network$ groups
test

Since my test user is only part of the test group, I have no privileged access to edit these files.

This circles back to the concept of privilege levels. In traditional networking, user access is always controlled via two methods:

Privilege level allowing read/write vs. read-only access

Role based access control restricting which commands can be executed

In open networking, this concept is expanded and allows for more flexibility by migrating it to user/group privileges per file.

Centralized user management

Moving along, the next level up is integrating the user access with using a centralized user authority like TACACS or RADIUS. Centralized user management can come in three different flavors:

User authentication

User authorization

Per command authorization

Applying these to the open networking paradigm, all files have local user and group permissions. So the user authentication and authorization will dictate what groups the user has associated. Generally speaking, most centralized management systems will control which users have sudo privilege, as that is the differentiation between read-only and read-write users.

In conjunction with general open networking access privileges, the privileges can be applied to restrict a user’s capability in NCLU. NCLU has its own user privilege control file:

In the above example, the test user is not part of users_with_edit privilege, and as a result this user cannot make changes:

test@leaf02:/etc/network$ net add interface swp1
ERROR: User test does not have permission to make networking changes.

This control can be included as a field stored in the centralized server. In this example, the ops user only has show privileges (read-only) via NCLU:

cumulus@oob-mgmt-server:~$ ssh ops@leaf01
ops@leaf01's password:
Welcome to Cumulus VX (TM)
Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:
http://community.cumulusnetworks.com
The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.
ops@leaf01:~$ groups
radius_users netshow

The admin user has edit and show privileges (read-write) via NCLU:

cumulus@oob-mgmt-server:~$ ssh admin@leaf01
admin@leaf01's password:
Welcome to Cumulus VX (TM)
Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:
http://community.cumulusnetworks.com
The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.
admin@leaf01:~$ groups
radius_users netedit

Access control with automation

When using automation, the configurations are normally stored on a git server. So at the highest level, the permissions on the repo server itself can restrict how the configuration is edited and deployed. At the next level, users can have isolated access to the management server and the policy can allow users to execute only necessary automation playbooks. The playbooks can be written with a service account, and that service account can have restricted access on each of the network devices. Take a look at this workflow to understand it better:

This moves all the access control responsibility to the management server, which can be tied into the existing access control infrastructure applied to all other servers in the environment.

Share this blog post!

Rama Darbha is a Senior Consulting Engineer at Cumulus Networks, helping customers and partners optimize their open networking strategy. Rama has an active CCIE #22804 and a Masters in Engineering and Management from Duke University.