Featured in AI, ML & Data Engineering

In this article, author shows how to use big data query and processing language U-SQL on Azure Data Lake Analytics platform. U-SQL combines the concepts and constructs both of SQL and C#. It combines the simplicity and declarative nature of SQL with the programmatic power of C# including rich types and expressions.

Featured in Culture & Methods

The book Agile Leadership in Practice - Applying Management 3.0 by Dominik Maximini is an experience report of the agile transformation journey of NovaTec. Maximini shares his experiences from applying principles and practices from Management 3.0, success stories, failure stories, and learnings from experiments.

Featured in DevOps

Yuri Shkuro presents a methodology that uses data mining to learn the typical behavior of the system from massive amounts of distributed traces, compares it with pathological behavior during outages, and uses complexity reduction and intuitive visualizations to guide the user towards actionable insights about the root cause of the outages.

SpringOne 2017 Key Announcements

The Second Annual SpringOne 2017 Platform Conference has kicked off in full regalia! The Spring team has evolved its signature framework to where it is as important to the Java platform as Java itself, and the keynote did not disappoint. Mentioning their responsibility to help enterprises standardize on Pivotal Cloud Foundry, Pivotal Sr. VP Head of R&D for Cloud, Onsi Fakhouri, declared their mission to transform enterprises.

12 minutes were scarcely sufficient for the dozens of important announcements that were made in the Spring space as well as for Pivotal in general, including what they labeled “the most important product announcement in Pivotal’s history” (more on this later).

I will abstract from Fakhouri’s announcements and motivations below:

“What do you do in a world that is complicated and filled with obstacles?

“The best is to start by exploring, making a choice, then moving and doing it again. That core feedback loop, that way of handling reality is called learning. We want to empower our customers to become learning organizations, and we believe the best way to do that is to help our customers achieve velocity. Velocity enables learning; the ability to move fast drives efficient learning, and the ability to change direction drives specific learning.

“We have tools and frameworks that allow you to write clean consistent code, that you can change rapidly. And we have a platform that lets you push to production really quickly, and then push again and again as you learn. With all of these things combined, you can go fast forever. You can have velocity.

“Our customers have had a lot of success doing this. This conference is a diverse group of people, all here to learn from us and from each other. I want to share some of the things we learned in the last year.

Spring programming model has been a synchronous blocking programming model, that works well for many problems. But then there are cases where you have a lot of blocking IO and this model can be very inefficient, increasing memory usage, increasing latency. Now Asynchronous non blocking models are typically more efficient. But it is a totally different technology model. Picking between these models can be an expensive shift. Well the Spring team has been working on Reactor, and we are bringing async non blocking io paradigm to Spring, and it is going to be a familiar consistent programming stack, with powerful reactive patterns, that gives you a choice. And it is all packaged up in SpringBoot 2.0 and Spring 5.

We also learned that there are a lot of choices when it comes to IDEs so we are pleased to announce SpringTools 4, that is built for Boot, is IDE agnostic, and super-fast.

We’ve also learned that you need choice so we are partnering with IBM to bring Open Liberty as a commercially supported option for SpringBoot. And to bring SpringCloud stream connectivity into the IBM ecosystem.

This is just the tip of the iceberg.

But Spring needs a home, and we believe the best home is CloudFoundry. The heart of CloudFoundry is the Elastic Runtime. This is the platform that runs all of your applications, and this year we are rebranding Elastic Runtime, calling it the Pivotal Application Service (PAS), to emphasize that it knows how to run your applications really well. It is the driving force behind our customers’ success; it enables developer productivity through the cf push command, a simple elegant command that says “here is my source code, run it on the cloud for me. I do not care how".

“We’ve learned a lot about PAS in the last year, and we have invested a lot in security, observability, performance, and stability, in the local development environment, in extensibility. But at the heart of the platform is the ability to run polyglot languages. So again we are working with IBM to bring commercial IBM Liberty support to PAS, which will round out the collection of tools and languages you can use on the platform.

“There is the new Pivotal Function Service (PFS), which provides the ability to say “here is some code, deploy it, I don’t care where or how!”

“We’ve also been investing in running Windows/.Net workloads and we are going to be increasing that investment and so I am pleased to announce that we are bringing support for Windows Server 2016 along with native Windows Server containers, and it is coming in our next release, and it is going to bring feature parity to .Net workloads because we want to become your premier flagship platform to run your .Net workloads.

“We’ve seen our customers run PAS at ridiculous scales, folks running many thousands of applications on hundreds of VM’s, and we have learned that PAS runs really well at scale, in fact it was built to run at scale. And because of that it’s default footprint is a fat 20 VM’s, which is pretty heavy. But again, we need choice. So we are pleased to announce “Small Footprint” PAS, a full featured HA horizontally scalable PAS in just six VM’s. It’s available now on all our supported clouds, and you can try it with one click on the Azure Marketplace.

“Those are some of the PAS announcements, but PAS itself is just a part of the broader Pivotal CloudFoundry. PAS is powered by Bosh lifecycle management platform, which makes the impossible possible. With Bosh a small team of operators can manage PAS running thousands of applications. They can run regular platform upgrades and apply security patches at the click of a button.

“But we also learned that PAS can be difficult to understand what it is doing. And so we’ve invested in “Health Watch”, a beautiful dashboard that lets you understand exactly what your PAS is doing. The Pivotal operators that are running Pivotal tracker on top of PAS love it. And so we are excited to bring it to market as a GA product in the next release.

“We’ve also learned that although Bosh makes the impossible possible, upgrades can take long, and so we’ve been investing in upgrade speed and stability, and the next release will shave hours off the upgrade time. We are also bringing Concourse’s CI/CD automation to the platform, so you will be able to bring installation and upgrade pipelines that automatically download and apply upgrades to the platform, making it super easy to continue to maintain and upgrade your CloudFoundry installation.

“Bosh is at the heart of our multi-cloud story; it is what enables CloudFoundry to run across all of these different clouds. And here again we have learned that you need choice. And so we have been working with Microsoft to bring beta support for Azure Stack hybrid cloud in our next release.

“We have invested in these based on feedback from our customers, and we are always open to learning from you, so if there is a target you want us to hit, we welcome your feedback.

“So that’s PAS and Bosh, the heart of CloudFoundry. But there is more to CloudFoundry. We also have a Data Services ecosystem. Pivotal has a number of data services in that marketplace. We have our Redis offering, MySQL, RabbitMQ and our CloudCache offering powered by GemFire. These are fully managed dedicated services, and they’re easy to spin up with cf create-service. We’ve learned that delivering a fully managed service on premise is incredibly complex, and we’ve been on this learning journey gradually evolving the feature set for these products. We’ve started by providing customizable on‑demand services, so with Redis we know that you need choice, and you can pick the Redis of your size to run your application; no tickets to file at all, just cf create-service. We’ve also been allowing increased complexity from our product, so we have been going from single node services to multi node services, so for MySQL our next release will support leader-follower, and for Rabbit, our current release supports Rabbit clusters.

“And finally, we have been layering on richer feature sets, so with CloudCache we are going to allow you to deploy custom code to your GemFire cluster. And we are going to be supporting LAN replication to enable true multi-site application development.

“Now what you can see is that PCF is really an umbrella term that covers a broad set of workloads. But we’ve been learning that it doesn’t cover all workflows. PAS is optimized for developer productivity. It is opinionated, to help developers move faster, and developers love this. But because it’s opinionated, it can be challenging to run certain classes of workloads. For example, commercial off the shelf software, or software that has complex networking, persistence and lifecycle requirements- these are typically some legacy workloads. So what do we do? One option would be to add a lot of complexity to PAS to enable these workloads. But that would make the platform less opinionated, and we would run the risk of lowering developer productivity, which is not something we are going to do; we are your trusted partner. We want to enable our customers to maintain developer productivity. But we also want to be the trusted partner for all of your workflows. So what do we do?

“Well, it turns out that Kubernetes is a great solution for running some of these workloads. It is a great low-level container orchestrator that can handle these sorts of workflows, and we have observed that the industry seems to be having an OR conversation, where you have to choose between PCF and its developer productivity, and Kubernetes. We think that this is wrong. There should be an opportunity to turn an OR conversation into an AND conversation, and to bring some of our learnings from running PCF at scale behind the firewall, take those lessons, and apply them to Kubernetes. So we have been partnering with Google and VM Ware to bring Kubernetes to the PCF family, and we call this Pivotal Container Service or PKS. PKS is Kubernetes powered by Bosh. It allows a small team of operators to deploy fleets of on demand vanilla Kubernetes clusters. You can manage seamless upgrades and security patches; you can stay up to date with open source Kubernetes and something that we call constant compatibility. And we have been working again with IBM to bring commercially supported IBM Docker images to PKS.

“PKS rounds out this family, and what we want to deliver is a unified platform experience, with shared logging and metrics, shared networking, shared security, that is seamless and just works. But here again we have learned that networking can be quite challenging, especially in this new world of container networking. So we have been partnering with VM Ware to bring NSX-T to the platform. So you’ll be able to use NSX to manage your container networking across our entire stream of products on PCS. This is what is going to allow us to be the trusted partner for all of our customer workloads. It is our biggest release ever and that is why we are calling it PCF 2.0. It brings all of these pieces together, and sets us up for the future, a future where we continue to learn from our customers.

“For example, we have learned that you want to explore Functions as a Service. Well, we want to explore FaaS service too, so we are happy to introduce an open source FaaS platform, Riff, sponsored by Pivotal. Our plans in 2018 are to bring a commercial offering Pivotal Function Service powered by Riff to the PCF."

He concluded with the new Pivotal mantra - "Enjoy your time here as we all learn how to go fast forever."