#critical thinking #science #puppet #devops #linux #vmware #cloudops

DevOps in the Enterprise – Oracle Financials “Flow”

Is your eBusiness suite environment full of multiple development instances all running their own mini projects of sorts with unique efforts and no alignment? Do you catch yourself thinking that you need more environments because you have more projects that need to go on? Are you adding more projects to the mix because you’re stuffing your projects with fill time and dependencies and seeing that as idle time to do more?

Lets talk about FLOW.

In my experience, the typical Oracle Financials flow is chaos, managed chaos. Its heavily based upon very inflexible and highly managed plans and projects. These plans and projects often have layers of people management, process management, requirements, expectations, deadlines, goals and ambitions that are based upon assumptions and waste – processes that add no value to our end goal of delivering or supporting the financials applications stack. Come to think of it, I’ve been in this industry for 10+ years and its always been a struggle against unrecognized waste! We build our procedures on waste without really knowing it.. almost habitually!

15 Years of eBusiness suite deployments and the bulk of my work has been waste.

Let that settle in. (I feel anxious just thinking about it!)

I’m not here to point fingers, lay blame or say it was all wrong, I just need this to sink in before you can move to understanding flow, understanding your own waste and understand what it means to operate with “DevOps” value or “lean” operations. We can’t possibly work towards DevOps patterns in Enterprise systems until you realize what your own “Flow” of work is and what waste really is.

So about this waste, lets get into more details so you understand why its called “waste”.

Waste

Warning: This is mostly thoughts flowing out of my head.. maybe not the most cohesive.. most grammatically or syntactically correct, but just thoughts & experiences & ideas. I hope these get you thinking about your “waste” more than they are meant to document all waste.

Learned helplessness – Procedures are in place to trigger punishment, there is no reward for stopping production to do things right, there is no value consensus in the project or collaboration & reflection on getting things done. Someone else has planned things out according to someone’s idea of a deliverable that someone else forecast and the core value sold to the business it that the project will be done by a specified time and budget. Where is the value in this?

Overproduction – The tendency of projects to keep people busy by losing focus on what the customer has requirements for. In general over production is producing goods for which there are no orders, but in an Enterprise IT shop, this could translate to having too many environments operating that doesn’t fit your main flow essentially overproducing in order to keep the appearance of utilization. This also contains Excess Inventory, adding new expensive instances that you then have to have capital appropriations for because the storage, CPU and memory is expensive. You start passing those costs on to your customers or absorbing them as cost of doing the project / business. You start building Unnecessary movement with too many flows of work because now you have to maintain, patch and clone these environments on disparate schedules thus, creating more slippage in projects, more waste time in cycles. All of which adds up to Big Buffers,Defects, and Unused Employee Creativity. I’ll let those speak for themselves 🙂

With poor flow you build insurmountable technical debt; we start choosing the “least” concerning option, the “least” risky option, the “safest” route – and generally, not even based upon a consensus but someone actually signing off on the risk as a personal responsibility or PUSHING responsibility up/down the “People who are paid enough” chain. I quote all these “safe” and “least” routes because they’re often done on assumptions and learned behaviors more than they are chosen on facts and the discussions thereof.

How then do you then start to approach your eBusiness suite flow?

The conceptual practices that people generally speak of in DevOps circles stem from the works of Kevin Behr, Gene Kim & George Spafford (and many.. many others!) through the metaphor of the Three ways they hint of in the Phoenix Project. I say metaphors because conceptualizing these is fairly simple, but applying them is a different story and they don’t really “Speak” to the typical “Enterprise” shop. (if you ask me..) – BTW, let me be clear, this is mostly paraphrasing the Three ways!

The First Way – Create a fast flow of work that moves from development into IT.

The Second Way – Amplify feedback loops, so we can fix quality at the source and avoid rework.

The Third Way – Create a culture that simultaneously fosters experimentation, learning from failures, and understanding that repetition and practice are the prerequisites to mastery.

I believe the approach to thoughtfully applying and understanding these three ways successfully is to learn from Operations Management, Lean processes and to borrow heavily from lean JIT manufacturing and the Toyota Production System (TPS).

How does Lean workflow improve our flow?

Here are seven traits of lean operations, made possible by well studied and observed lean practitioners..

Builds in Quality

Flexibility

Higher Productivity

Frees up floor space

Improves Safety

Improves Morale

Reduces cost of inventory

Did you read those and ponder on them a bit? Did you think about them in your own environment? I wanted to get them out there to get you thinking about them so the next part makes more sense.. It can be confusing to talk principles, culture and concepts when you don’t feel any bearing to them yourself.

Developing lean / DevOps values & principles.

Principle 1. Long term philosophy even at the expense of short term goals. If your goal is to generate value to your customers then you can’t achieve that by taking short cuts to appease near term goals. I had a big paragraph describing concepts here, but deleted it because this will make more sense as you get through the rest of the principles. Simply put, your financial application investment is a HUGE long term investment, don’t put it at risk for near term gain! Keep that in mind as you build your values & culture!

Principle 2. Create continuous process flow to bring problems to the surface. Those project plans, are they full of hand-offs, waits & idle time? (Waste?) Make flow evident in your organization so the project itself is improved iteratively to optimize its own value. In a continuous flow, you don’t promote your problems to the next “Cell” for them to work on, you surface them to fix them!

Principle 3. Use Pull systems to avoid overproduction Provide your customers with what they want, when they want it and in the amount they want. Minimize your WIP (Work in Progress/Process) so you can be responsive to day-by-day shifts in demand rather than relying on schedules. Too much WIP and you start losing your flow and spiral out of control.

Principle 4. level out the workload Eliminate waste, eliminate overburden. Remove the painful “start & stop” of implementations – strive to work through to completion smaller, more iterative processes. The more you start and stop work because you have so many inter-dependencies, the more you’re breaking up the workload, the more you’re adding waste & re-work. Projects & WIP generally get out of hand if you don’t value the leveling out of workloads!

Principle 5. Build a culture of stopping to fix problems – get quality right the first time. Don’t build up technical debt! The quality of service we provide is our value proposition to our customers and the business. Use quality assurance methods available to help get quality right. This is one of the areas where I’ll mention tools as you can build a process around tools to remove complexity, remove dependencies (decouple!), remove guess work and improve the flow. Continuous integration environments, Unit testing, Desired State Configs. These are all concepts that embrace the culture of getting quality right the first time and embracing a culture of stopping to fix problems. “Prod” is no longer just the production instance of your financials application, it is the entire flow!

Principle 6. Standardize tasks Build foundations for continuous improvement and employee empowerment. Use stable, repeatable methods everywhere to maintain the predictability, regular timing and output of your processes. Those “controls” you have in place that you thought did this, actually get in the way. Rethink those! The tasks are more important than the controls because the tasks empower people to improve them, collaborate on them, test them and analyze them. Obviously I’m not saying to throw out controls, companies have them in place for Audit, legislative, legal and financial accountability reasons, but use them wisely. Look at your values and see how your “controls” are part of the value chain!

Principle 7. Use visual controls so no problems are hidden. Many people recommend Kanban style “pull” visuals so you can visually represent the flow, but you can expand these into providing visual data across your flows. We often call them “KPI’s”, we call them “Dashboards”, We call them “Portals”, They’re representations of flow! Don’t hide problems by keeping people unaware from them, shoving them off to other groups or pushing the blame around, keep them front and center so you know what you have to apply your efforts on!

Principle 8. Use reliable and thoroughly tested technology that serve your people and processes. Isn’t this the expectation we have in buying and paying for Oracle eBusiness Suite? We often imply reliability and thoroughness by forcing long lead times, forcing compliance, forcing validation, forcing checkpoints and buffering time in our projects. We’re doing it wrong! Build up reliable OS platforms, so you can do OS patching without your Business Analyst having to approve something they have no bearing on. (You know, those change requests that get promoted to 5 different people!). Build up reliable infrastructure so you can have alerting, collect metrics, analyze data and visualize your flow. You want to have the trust to do things right as well as the trust to be able to fix things that break. If you’re not innovating, its time to review your process!

Principle 9 – Grow leaders who thoroughly understand the work, live the philosophy and teach it to others. I personally believe that growing an agile / lean / DevOps culture is much more powerful than buying one. The best way to grow people is to empower them and the best way to empower them is to motivate them by making them part of the process, showing them how they contribute to the process and reflecting their value.

Principle 10. Develop exceptional people and teams who follow your company’s philosophy What is your goal? Are you working with these values? Are you looking to “iterate to innovate”? Do you have a policy of “If you’re not innovating, its time to review your process”? Are you understanding these values? Are you developing them in your teams & company culture?

Principle 11. Respect your extended network of partners and suppliers by challenging them and helping them improve Work with your partners, help them understand your requirements and your flow. Make sure Oracle knows your patterns, make sure your OS vendor knows your strategy, make sure you’re choosing people who support YOUR business. Don’t let “best practices” or other buzz phrases get in the way. Your best practices are the practices you foster, you measure, you analyze, you validate, you test and you improve. Validate what your partners are doing through the same process you validate your own organization!

Principle 12. Go and see for your self to thoroughly understand the situation! Don’t leave people hanging! Simple as that. If there is a problem, go and discuss it. If something breaks, you should go and stop the flow, see the problem for yourself, get the right people involved to fix it and take ownership.

Principle 13. Make decisions slowly by consensuses, thoroughly considering all options; implement decisions rapidly. If you ask me, this is the best principle ever because its a complete reversal of experience I’ve had where there is very little discussion on doing this well. Reverse your patterns by striving for excellence in design & implementation, increase your flow so the implementation of decisions is rapid! Often times people assume a project manager has done all of this in advance and the entire project is implementation, that seems entirely backwards and counter productive to me but a trap I see all to often! Make sure stakeholders are holding their stake! 😀

Principal 14. Become a learning organization through relentless reflection and continuous improvement This is it folks. The culture you develop should be based on relentless reflection and continuous improvement. This means experimentation, A/B Analysis, trial and error. This means blameless postmortems. This means collaboration. This means cross domain participation. This is “DevOps” folks!

What do these principals do for your flow?

These principals help you develop new values and new learned behaviors. They really have clarity when you tie them back to those seven traits I bullet pointed earlier.

Flexibility – Reduced lead times, faster deployments, faster fixes. You’re now empowering people to be creative, flexible because the process is creative.

Higher Productivity – Single flow means that the work is getting done and done right. You have less over production, less environment/instances, less coordination, less process management, more focus on agility and very little non value add work.

Frees up floor space – While we don’t typically have “floor space” in the manufacturing sense, we do have “floor” space in the technology. Single flow means less VM sprawl, server sprawl, less capital tied up in idle hardware. We’re no local focusing on local optimums of optimizing infrastructure use and VM use by using MORE of it to make it appear utilized.

Improves Safety – People usually don’t get hurt physically in eBusiness suite deployments and upgrades but safety can be distilled in many ways. Safety can be seen as stability, Safety can be seen as respect, partnerships. Safety can be the protocol by which you solve the insatiable appetite of managers to strive for consistency.

Improves Morale – People are focused on single flows, people feel empowered to make a difference, people are challenged to do better, people feel the reward of a job well done. People appreciate adding value and creating value!

Reduces cost of inventory – Just like freeing up floor space, you can reduce sprawl, hardware, processes and controls, reducing the cost of inventory, reducing the cost of WIP and reducing the cost of your flow. You don’t need to have environments on standby, don’t need to have excess capacity just in case. You can use these excess capacities for other purposes that add value! Use them to experiment, test, verify. Use them for proof of concepts. The idle time you measure should be idle time you allocate to improvement. If there is one thing you want to stock up on, its an excess of pride, excess of trust, excess of ownership. The ROI on that is through the roof!

The Three ways

Once you start understanding the metaphors, values and principals of lean operations you can start building your own metaphor for the “Three ways”. Understand your flow, make feedback part of your process and wrap it all up with culture!

My Story

If after reading my opening statements you feel I’m being snarky in a way, it’s probably because I am. Not only have I ran into exceedingly costly and ever failing efforts and projects but as a person in this failure loop, its hurt me personally. Its built up bad behaviors, bad experiences, bad attitudes and so much more! Poor flow projects turn collaboration inside out, people don’t want to talk, they don’t want to work together, blame takes over, people fight for the wrong types of responsibility and the demands on your technical staff to pull of thankless miracle after miracle become detrimental to not only their attitudes, but their behaviors and even their health. We start striving for the wrong goals, we start striving for deadlines and dates, we hide risk behind the successes of deadlines, we build technical debt not creating a process that allows us to improve our deficiencies and do things “The right way” and when we say the “right way” – that doesn’t mean “My way as I see fit” but the “Collaborative way with continuous improvement in mind so that we know we’re all working towards creating and adding value”.

The end result of determining your flow isn’t just a beautiful “Self-healing”, “self-maturing”,”reflective” project and infrastructure – it’s a cultural shift for your organization that is downright empowering!

I don’t have all the answers, but what I wanted to do was offer an honest critique of where we are failing and ways that we can fix it, building upon the successful values of others. Isn’t that really where our flow stems from? I didn’t invent this, don’t claim to be a master of it, and i’m always having to practice it and think about it. I believe in sharing it and writing about it since that is part of my philosophy and how I learn as well!

As always, comments/feedback is HIGHLY appreciated! I know this was a long one so thanks for staying with it!