Viewing Keystone CADF notifications with Ceilometer and RabbitMQ

This post is long overdue, but better late than never! This post will show how a user can view Keystone CADF notifications in both Ceilometer and RabbitMQ. To start, let’s go over some basic concepts first.

Definitions:

Events: An event is triggered when a user accesses certain APIs (i.e.: an administrator creates a new user)Auditing: We want to be able to figure out who did what, from where, and to what resource.Ceilometer: OpenStack’s Telemetry service – provides support for metering, alarms, notifications, and other services.Keystone: OpenStack’s Identity service – provides authentication, authorization and identity managementCADF: The Cloud Auditing Data Federation (CADF) standard defines a full event model anyone can use to fill in the essential data needed to certify, self-manage and self-audit application security in cloud environments.pyCADF: A python implementation of CADF events, maintained by OpenStack.

Background, or Why is Keystone so special?

Why not just use pyCADF’s audit middleware?
Most OpenStack services (nova, glance, cinder, etc…) use keystonemiddleware to establish communication with Keystone. This is performed by creating a service user and saving credentials in the [keystone_authtoken] section and including keystonemiddleware in the service’s pipeline. Services can also include pyCADF‘s middleware, which piggybacks off the content it sees in keystonemiddleware to easily include auditing support for every API request that hits the service.

We can’t take this approach in Keystone. keystonemiddleware gets its information from a Keystone token (using the service user). If we try to include pyCADF‘s middleware into Keystone’s pipeline, then we’ll run into a circular reference. What the Keystone team has decided to do is to instead emit notifications for specific events.

I thought Keystone already emits notifications, why do I need CADF?
Keystone did already emit notifications, but these were lacking in content and made auditing impractical. Take for instance this example of a basic notification.

We can tell a user was created, but who triggered the event? From which host machine? What project did they use as their scope? Using CADF notifications will tell us this information (and more), and additionally it’ll standardize the way events are seen in OpenStack. On to the fun stuff!

Update Keystone’s configuration file
Make the following changes to keystone.conf. Use cadf as the notification_format instead of basic. Also, set notification_driver twice, once to messaging and again to log. This will make the notifications be emitted to the rabbitmq service, and be written to Keystone’s log.

You’ll see here that we have access to the necessary information we need for auditing. The initiator block includes: the user who performed the action, the project they used to authenticate with, and the host machine they used. For a full list of actions that will emit notifications, refer to Keystone’s notification documentation.

Viewing CADF notifications in RabbitMQ

Alternatively instead of using Ceilometer, you can use RabbitMQ’s management plugin to view the same notifications. Note, you should stop all Ceilometer processes to ensure that it won’t read the messages on the notification queue.

Setup RabbitMQ management plugin
When installed, the RabbitMQ management plugin will automatically add a web interface, users will be able to easily read notifications.

You’ll need to trigger another event, so create a user, project or role using OpenStackClient. When that is done, navigate to: http://10.0.2.15:15672 and log in with your RabbitMQ credentials. The default username and password are guest/guest. Note, in the screenshots shown below RabbitMQ is shown running on port 8081, this because I had port forwarding enabled in my VM.

1. Check the Queues tab at the top

2. Look for unread messages in the notifications.info queue.

3. Get the messages, there will be authentication messages and a identity.user.created message

Steve Martinelli is a Development Manager focused on delivering Cognitive Journeys that empower developers worldwide. Previously he served as the Project Team Lead on OpenStackâ€™s Identity, Authentication and Authorization service, code named Keystone. He continues to serve on OpenStackâ€™s Technical Committee. Steve has spoken at various OpenStack Summits, Centre for Advanced Students Conference (CASCON), and at the Cloud Identity Summit. Steve is a co-author of Identity, Authentication & Access Management in OpenStack, a book published by O'Reilly Media in 2015. Steve received his B.ASc. in Computer Engineering from York University in Toronto, Ontario, where he continues to reside.

[…] But if you had an async way to get events from Keystone, you could solve this yourself. That was my idea with my Keystone CADF Event Logger tool. Before we dive into the tool, some quick background on CADF events. You can read the DMTF mumbo-jumbo at the link in the previous sentence, but just know, Keystone CADF events log anything interesting that happens in Keystone. They also tell you who did it, from where they did it, and when they did it. All important things for auditing. (This article from Steve Martinelli has some more great background) […]