OpenShift

“Red Hat® OpenShift is a container application platform that brings Docker and Kubernetes to the enterprise. Regardless of your applications architecture, OpenShift lets you easily and quickly build, develop, and deploy in nearly any infrastructure, public or private. ” [1]

I’ve helped a lot of customers find their way to OpenShift. I’ve helped them develop and refine use case as well as figure out how it fits into their environment.

Cockroach DB

Other than having an unfortunate name I really like Cockroach Labs CockroachDB from the little I’ve used it. I am by far not an expert on CockroachDB. I first learned about it at OpenStack Summit 17 in Boston. Kudo to the guys for putting this presentation together and presenting it on the big stage.

What is CockroachDB?

“CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.” [2]

There is a lot more to be said about the statement above that I am not going to cover here. To summarize, you can partition and replicate your database while still making sure queries are only getting the latest data, called “Strong Consistancy”.

In Closing

On a personal level I like the notion of OpenShift and CockroachDB together for a couple reasons. Chiefly, they are both Open Source. You can experiment locally with no up front cost on a developer macine and eventually roll it into production. When you are ready for production both projects have Enterprise support offerings. It’s not critical you invest time, money and resources into figuring out if it’s going to work for you just to find out 6-9 months down the line it’s not what you hoped it would be. Experiment now! My next post will give you the framework to start….

2018 is upon us and I’ve renewed my interest in posting to this blog. I’ve been doing a ton of learning in 2017 and I think 2018 is going to be the year I start spreading that knowledge.

I’m always pushing myself to look for new ground or enhancing new concepts that we are familiar with already. Look for more of this in coming weeks. I think I have things to say about data storage (Databases) in the kubernetes space.

During a meeting it was brought up to me that the OpenShift/Kubernetes logging strategy isn’t very concise. Though looking into this I wanted to put some context around the technology. “How does OpenShift capture logs?” “What is captured and logged?” “What is my recommendations for using the logging system?”

EFK Stack

EFK stands for Elastic Search (E), Fluentd (F), and Kibana (K). This is a modification on the traditional ELK stack that has become popular in recent years for log aggregation, collection and sorting. Kibana acts as the user interface for the collected logs. Elastic Search is the search and analytics engine. Fluentd is a unified logging system with hundreds (500+ as of the time if this writing [1]) of plugins.

What is captured

Looking thought the Kubernetes documentation, it’s made a bit more clear what is captured where and how applications logs are managed from the container level. In the section titled ‘Logging at the node level’ [2] it is explained that “Everything a containerized application writes to stdout and stderr is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to a logging driver, which is configured in Kubernetes to write to a file in json format.” It is said in OpenShift documentation “Fluentd reads from /var/log/messages and /var/log/containers/.log for system logs and container logs, respectively. You can instead use the systemd journal as the log source. There are three deployer configuration parameters available in the deployer ConfigMap.” [3]. For additional information and resources on Fluentd, I strongly recommend watching the ‘OpenShift Commons’ videos from May 17, 2017 [4].

Cluster wide -vs- Project logging

This is not a simple question to answer. I’m drawing upon my experience working with other clustered technologies and customers that have implemented OpenShift. My recommendation is to do what’s right for your environment. I know… not very useful. Hopefully my heuristics will lead you to your answer.

Business Reasons

More often than not your company, organization, group and team have their structure. I’ve worked with companies and agencies that have had every sort of organically grown business structure. Some extremely independent, some centralized, and some ignorant to the structure entirely. We have to consider how you do business today and what will actually work and how we can fit into that system. What are the security requirements? What are the data retention rates? What is the disaster recovery strategy?

Technical Reasons

If I were to propose that every project have their own EFK stack to manage only their logs. A customer running 100+ project will have a LOT of redundancy and the overhead for a security team to manage and track logs could be prohibitory expensive/complicated. How does a security team monitor the creation of new projects, validated their access and ultimate ensure the security and compliance of the systems?

If I proposed one giant company-wide EFK stack it would lighten the burden for some but could cause data management and growth complications. Our security team is happy, because they have one log on to one server to see all the system and application logs being generated by the containers and applications. Let me assume for a minute a non-common use case for OpenShift, batch processing. I want to use this platform to ETL function on a file I have stored out in S3. That project or job that live and die on my whim, might introduce long stale data into my logging system and tool chain. The point of a job is to run and be gone, so I might not care about the details.

While working as a US Army consultant in 2011, we were implementing Splunk. Working though the data ingest rates and figuring out what was good and stale data was complicated and we had fairly static workloads. Working though all the requirements will likely guide you the right direction. I suggest pruning what is important and measuring them often, high signal to noise ratio. This typically means smaller units or project based logging. It becomes quite daunting to measure every job, application and container in your environment on an ongoing basis. Off load that responsibility to the application and project owners.

Since I mentioned Splunk, I thought it is important to include the following section as well. ‘Configuring Fluentd to Send Logs to an External Log Aggregator’. You can configure Fluentd to send a copy of its logs to an external log aggregator, and not the default Elasticsearch, using the secure-forward plug-in. From there, you can further process log records after the locally hosted Fluentd has processed them [3].

Tags

I think tags are one of the most important features of Red Hat CloudForms. CloudForms ability to tag resources for later use in reporting, chargeback/showback and automation is critical for getting more in-depth knowledge and generating laser focused reports that provide value.

In this article I am going to touch on general guidelines I use when building a tag schema. I believe there are two rules when talking about CloudForms tags. It’s better too over tag your resources than under tag. If you can measure it; you can manage and monitor it. Just like any data structure, a well thought out schema will save you a lot of work.

Tag Schema Recommendations

Business Tags:

The most important thing about your business tag schema is they make sense to you and your companies. The examples I list below are a very rough estimation of what your business will look like and how it will operate. Think about logically grouped business resources and come up with a tag for them.

Business Unit

– Sales (North America)

– Engineering

– QA

– Marketing
– IT Development
– IT Operations

Business Project

– oVirt

– OpenStack

– Project Phoenix

Business Owner

– VP – Linux Torvalds

– Project Owner – Richard Stallman

– Manager of Marketing (East) – Doris Hopper

IT Operations Tags

If you’re reading this blog post, these tags probably matter most to you. Remember, measure what matters most to you. Help the business understand your value and what you do. I know it, you hopefully know it, let them know it too.

Imagine if IT and the business came to an agreement on service windows based on what worked for each business unit. This can happen. Maybe you want to have test deployments on production resources. Tagging change windows into your resources will help with reporting and also automation.

A service level agreement (SLA) is a contract between a service provider (either internal or external) and the end user that defines the level of service expected from the service provider.

– Diamond

– Gold

– Silver

– Bronze

Security Tags

Security tagging is something I’m still working through. I know there is value in creating a Security Role, Group and Users for Dashboard and Reporting. I’m looking at linking this into Policies and exporting log events to a SIM (Security Information Management).