Blogs

How to navigate—and conquer—the world of Hadoop and Spark

Interview with Julie Elliot and Mike Ashwell

While working with organizations at various stages of Apache Hadoop and Apache Spark adoption, IBM has identified a number of common problems that many businesses find themselves facing. Those problems led to IBM developing Lifecycle Support Services for IBM Open Platform (IOP) to help organizations accelerate time to value outcomes and reduce risk when building Hadoop and Spark applications. This offering includes a set of three specific services:

Install and Planning: Deploy, configure and manage a scalable Hadoop cluster to move quickly into production

Developer Assist: Explore and develop specific use cases

Designated Support Engineer: Provide ongoing advice and support to help maintain and expand the use of these technologies

We spoke recently with Mike Ashwell and Julie Elliot about how these services can be combined to help organizations. Ashwell is on the offering management team at IBM Lab Services and specializes in advanced analytics. Elliot is a solution architect at IBM, who works on developing services offerings for the IBM Analytics brand. Using Lifecycle Support Services for IOP, not only can customers take first steps with Hadoop and Spark, but they can also build a strong, in-house capability for delivering long-term big data analytics projects rapidly and at minimal risk.

Can you tell us a bit about the thinking behind Lifecycle Support Services for IOP?

Julie Elliot: In the area we work in, customers are often looking for a consistent relationship with an engineer who will get to know their application, get to know their people and understand what they’re doing and why they’re doing it. For those reasons, IBM came up with an offering that can give organizations the continuity of relationship with a consultant who can provide on-demand assistance whenever they need it.

Mike Ashwell: We’ve designed these three support offerings to target different elements of the lifecycle with IOP. This design gives customers a palette of services they can choose from to support what they’re trying to do with the software. And depending on what they’re doing, the services help them design and develop use cases, assist them in actually installing a cluste, or support them more generally across the whole lifecycle of their project.

“We’re helping customers build their own intellectual capital within their organizations, to help them expand their use of IOP, adopt the technology more quickly and make good decisions along the way. We’re not just providing a resource that will do the job for them; we’re transferring expertise to help them operate and develop the environment for themselves in the long term.” —Julie Elliot

Hadoop and Spark expertise is hard to find and expensive to hire, as many companies know. Does the new Lifecycle Support Services for IOP effectively provide a substitute for having a full-time resource in house, or does it offer something more than that?

Elliot: I think it’s more than that. What we’re doing is helping customers build their own intellectual capital within their organizations, to help them expand their use of IOP, adopt the technology more quickly and make good decisions along the way. We’re not just providing a resource that will do the job for them; we’re transferring expertise to help them operate and develop the environment for themselves in the long term.

Ashwell: Of course, we also offer turnkey services for organizations that want someone to come in and lead an entire implementation. But Lifecycle Support Services for IOP focuses on helping the customer grow its skill set. First, it helps customers expand skills by getting started and getting up and running quickly. Second, it gives them guidance about best practices and how to keep focused on achieving their goals for their application.

How did you go about designing the services to make sure they would really meet customers’ needs?

Ashwell: Well, one thing to note is that the Developer Assist and Designated Support Engineer services are based on similar offerings that we’ve developed for other products in the past that have been very successful. And the initial Install and Planning service was designed around an obvious need in the market. When customers start thinking about how to put their Hadoop strategy into practice, one of the first and most challenging issues they run into is how to build their first cluster and get up and running. So we’ve seen a lot of interest in this offering already.

One of the nice things about what we’re doing with these three offerings is bringing our IOP services together in a unified view, so our customers and our sales teams can see that we’re offering an end-to-end approach with well-defined pieces. In the past, we have delivered each of these types of services separately for IOP customers, but we think bringing them together into a coherent set will really prove to be popular.

“Expert services to guide your planning and help you leverage well-tested practices can set the right direction early and really mitigate risk. That guidance means you don’t go down the wrong path, and you don’t create problems that you have to correct down the road at much greater cost.” —Mike Ashwell

We’ve spoken about how these services are designed to accelerate the development of analytics applications with IOP, but another key objective for customers is to mitigate risk in these kinds of projects. How does Lifecycle Support Services for IOP help to achieve that?

Ashwell: When customers are working with technologies such as Hadoop, Spark and IBM BigIntegrate, even if they have some in-house experience already, the frontier is still relatively new. Inevitably, gaps in their knowledge of the field in general will exist, and they may not fully understand how IBM’s specific offerings help to fill those gaps.

Of course, that situation opens them up to risk. If they make bad decisions early in their adoption of these technologies, and don’t have the expertise to recognize those bad decisions until later, they can end up with problems festering beneath the surface that become really challenging to resolve.

A simple example that is unfortunately common with Hadoop is that you might have some problems in your original installation in which you improperly set up some of the configuration. If you don’t catch the problem until later when you’re trying to leverage a particular component, going back to correct those early errors can create a huge amount of extra work. Expert services to guide your planning and help you leverage well-tested practices can set the right direction early and really mitigate risk. That guidance means you don’t go down the wrong path, and you don’t create problems that you have to correct down the road at much greater cost.

Elliot: And that kind of guidance isn’t only important at the beginning of a project; it’s important all the way through its lifecycle. Once customers have their platform in production and they’re achieving success with it, they might want to add new capabilities and tools, or expand to different populations of users. As a result, they’ll have an ongoing need to consider capacity management, performance and operational stability.

The Designated Support Engineer offering helps give organizations the ongoing advice they need to maintain and enhance their applications and ensure they can take advantage of the best new features that the ever-evolving Hadoop ecosystem has to offer. And they can do so without the risk of running into a dead end because of incompatibilities among components or other architectural issues.

“Having a partner who can really help guide you through these complexities and smooth out some of the problems that early adopters often face is critical. You can start getting value from these technologies rapidly, and that outcome is absolutely essential in the software world today.” —Mike Ashwell

Some recent Gartner research notes that although digital transformation is vital for business growth, working out what to do and how to do it remains challenging for IT and business leaders. Can these new support offerings help with that challenge?

Ashwell: Absolutely. Although this bundle of services is primarily focused on what I would call the how-to-do-it angle, it still contains a strong element of what to do. So we’re aiming to fire on both cylinders. The Install and Planning service is very much on the practical, how-to side; and then at the other extreme, the Designated Support Engineer service is about helping customers understand what they should be doing and how to plan successfully.

Why do you think companies are finding this kind of digital transformation so challenging to navigate and build expertise in? Is the technology more complex, is the landscape changing more quickly or is some other industry trend the cause?

Ashwell: Well, being in the IT business for a number of years, I’m seeing a significant explosion of technologies over the last few years, especially due to the leading role open source communities are playing in the software industry’s evolution. Many more moving parts are evident because it’s not just a handful of large vendors who are building enterprise products; practically any small open source community could potentially develop a technology that becomes the next big thing. Just look at Hadoop. So many projects within the Hadoop ecosystem are in progress, and new ones are being added all the time. Keeping up with them takes considerable effort.

In addition, we’re simultaneously seeing the industry re-orientate itself around cloud-delivery models, which is another gigantic change for customers to grasp. The rise of cloud computing is partly a reflection of vendors trying to help customers handle the increasing complexity of the IT world, and trying to make consuming these services easier for them. But at the same time, moving to the cloud can create its own challenges, and customers are challenged to devote resources to all these different areas at once.

Therefore, having a partner who can really help guide you through these complexities and smooth out some of the problems that early adopters often face is critical. You are still building the core skills for yourself, but with a partner you can do it in a more accelerated way and without needing to focus on understanding all the intricacies before you even get started. So you can start getting value from these technologies rapidly, and that outcome is absolutely essential in the software world today.

So where can I learn more about Lifecycle Support Services for IOP?

Elliot: The IBM Analytics Hadoop webpage offers a good overview of IOP and Hadoop and Spark support services. It also acts as a hub where you can find lots of other resources.