Industry

Employees

Location

Use Case

Gain transparency across entire AWS cloud technology stack and third-party services to rapidly deliver seamless, world-class fan experiences for Major League Baseball and other high-profile clients.

Why New Relic

Access to powerful out-of-the-box visualizations, helpful alerts, great ease of use, advanced drilldown functions, and a single source of truth across multiple environments.

Highlights

Visibility into software performance, customer experience, and business metrics

Empowers developers with a single source of truth

Improves the accuracy of capacity planning ahead of high-traffic events

MLBAM–Partnering with New Relic to Ensure Great Fan Experiences (Technical Deep-Dive)

It’s the bottom of the ninth. Two on, two out, the home team is down by a single run. Millions of people are watching, in the park, on television, and now, because of Major League Baseball Advanced Media’s (MLBAM) world-class streaming technology operations, on a wide variety of computers and mobile and connected devices. Each fan is glued to the action as the star closer goes into the stretch and fires the ball toward…

What if the action froze or buffered right there for the people around the world watching the live streaming video? What if they couldn’t see whether the pitch turned into a game-ending out or a walk-off win? There’d be an awful lot of very unhappy baseball fans.

Those are the stakes for MLBAM, the 16-year-old technology company of Major League Baseball dedicated to delivering world-class digital experiences and distributing content through all forms of interactive media. While it’s perhaps best known for developing the platform that lets people watch virtually any Major League Baseball game from virtually anywhere in the world on virtually any device, it does much more, for Major League Baseball and a variety of other high-profile media partners—all managed with help from New Relic across its entire technology stack.

Born in 2000, and now New York’s largest born-and-bred technology company, MLBAM began with a prescient mission to build and run technology services for Major League Baseball (MLB) and its 30 teams. MLBAM provides all technical capabilities, including running the websites (MLB.com and team sites), ticket purchasing, game and video distribution, and statistics collection and distribution. MLBAM also provides over-the-top streaming infrastructure, and in some instances app development, for major clients in sports, news and entertainment.

To make sure it delivers the best possible viewing experiences for all users and minimizes glitches, freezes, and other problems, MLBAM (insiders commonly use the nickname “BAM”) partnered with New Relic for end-to-end visibility into how its applications are performing and the quality of the experience users are getting. It’s difficult to be perfect at MLBAM’s scale and velocity, but experts credit the company with building a “gold standard” digital streaming video infrastructure—with New Relic baked into its full technology stack, including mobile, browser, and core cloud infrastructure.

Today, MLBAM is an online giant, powering some 10 million live streams daily. MLB.com estimates its owned and operated properties attracted roughly 25 million daily visitors and its wildly popular and successful MLB.com At Bat app logged more than 6.9 billion total minutes used by fans, making it the top mobile sports app in 2015. MLBAM also tracks state-of-the-art game and player analytics via Statcast powered by Amazon Web Services —made available to teams and fans through digital platforms and TV broadcasts. When the TV play-by-play announcer says a ball thrown from the shortstop to the first baseman traveled at 76 mph and hit the first baseman’s glove 37.4 milliseconds before the base runner got there, that’s data MLBAM helped measure, collect, and manage in real time.

Leveraging its market-leading expertise in digital media distribution, MLBAM has become “one of the world’s top tech players,” according to CBS News.

“[New Relic] is more than a developer tool, at this point it’s an operational tool as well,”

“With an environment like Amazon Web Services, you can be very elastic with it. That elasticity is only really helpful if you truly understand what's going on inside your applications.” — Joe Inzerillo, Executive Vice President and Chief Technology Officer, MLBAM

Like all highly successful endeavors, success means scaling, and MLBAM is no different. As MLBAM continues to grow its digital fan base, and its roster of big league clients, the company relies on Amazon Web Services for scale and agility, and New Relic for smooth integration and visibility.

Whether it’s for MLB fans, or MLBAM’s other customers like HBO, WWE, NHL, or PGA, “The cloud lets us maximize the potential of our product and engineering teams by letting us focus on building the most compelling digital products and experiences,” Curtis explains. “We leverage AWS and its solutions so we don't have to reinvent the wheel.” As Curtis notes, “our teams can focus on measurement, collection, distribution, and prioritization, and leverage AWS for things like storage, compute, redundancy, and distribution.”

MLBAM teams naturally turned to New Relic APM and New Relic Browser to help ensure success of their cloud strategy. “New Relic plays an important role for us in being the partner that helps us refine our ability to measure the effectiveness of our applications,” says Curtis.

“Partnering with Amazon from an infrastructure standpoint and New Relic from the instrumentation standpoint, we’re not diverting resources from our primary mission which is to build something,’ says Joe Inzerillo, executive vice president and chief technology officer at MLBAM. “Instead, we’re adding value to the ecosystem.”

With New Relic and AWS, scalability is a whole new ball game.

New Relic supports MLBAM in the cloud

Clearly, scaling is critical for MLBAM, which has led the company to embrace DevOps, the cloud, Amazon Web Services—and New Relic.

In 2014 an influx of large scale live events that needed to be executed in a short window, even sometimes simultaneously, led MLBAM to engage with AWS to handle “overflow” scaling needs, and based on that experience subsequently settled on AWS as its primary scaling strategy. However, MLBAM’s cloud vision was broader than simply accessing on-demand compute, storage, and networking resources.

“The cloud allowed us to not only scale our applications, but to scale our organization as well,” says Christian Villoslada, vice president of software engineering at MLBAM. To speed application development, the company began taking advantage of new engineering and operations tools and approaches, including DevOps and agile development. New Relic dashboards play a critical role here.

"All of a sudden it was just like the stadium lights went on and we could see the whole field. We could see every single play that was going on. It was just an awesome amount of visibility into how our stacks were performing. That’s the biggest gain that we had. I would not want that to go away."

Paolo Vaca, ‎Director
Web Software Engineering, MLBAM

How New Relic adds visibility

In this new model, MLBAM saw a critical need for shared, end-to-end visibility into how its apps are performing and the customer experience it was delivering. “Everything is sending data to New Relic,” says Paolo Vaca, ‎Director of Web Software Engineering at MLBAM. “As developers work on new features, they can go to the dashboard and see if their features are having any performance impacts or if there are errors that they just can’t catch sifting through the AWS logs.”

“There were two major holes for my team,” notes Mike Lenner, vice president of software engineering at MLBAM. “One was visualization. There wasn't a way to make performance data clear, crisp, and commonly available across teams. The second issue was troubleshooting complex applications with APIs to other systems. For instance, we’d see an increase in latency, but couldn’t easily pinpoint why.”The key is to identify and fix potential problems before users complained about them. “The earlier we can find out there’s an issue, the better our ability to preserve the customer experience,” explains Sean Curtis, senior vice president of engineering at MLBAM. “We're competing against the biggest and best platforms in the world, so you have to provide the best customer experience.”

In the beginning, that wasn’t always happening. “Information about performance issues often came from users,” recalls Shawn Will Smith, associate director at MLBAM. “Then the engineers would start trying to figure out what was going wrong.”

But with New Relic, “all of a sudden it was just like the stadium lights went on and we could see the whole field. We could see every single play that was going on,” says Vaca. “It was just an awesome amount of visibility into how our stacks were performing. That’s the biggest gain that we had. I would not want that to go away.”

New Relic helps empower MLBAM’s development teams

To better manage the cloud transition, MLBAM transferred internal ownership for all of its services away from its central operations team to the individual development teams that build each service. These largely autonomous development teams now are responsible for everything about their services: code, test, deployment, and production support.New Relic helps the teams work together and take ownership of their services, giving them a place to compare and drill into metrics like error reporting and page load times. “What I really like,” adds Lenner, “is that it has been a cultural change too. We've definitely been more transparent amongst teams about what's going on, especially if a team that's dependent on another team's services. But also wanting to know prior to end-user customers alerting us to their issues, is first and foremost.”

As Curtis puts it: “Each team has to understand that they have to drive transparency around how well they're doing, how well they are able to measure their effectiveness within that area.”

MLBAM’s automated scoreboard

Unlike the latest fashion in ballpark design, there’s nothing manual about the performance scoreboards MLBAM built with New Relic. The company relies on a suite of New Relic products for access to an automated, full-team view of how its applications are performing.

The rollout started with using New Relic APM™ to optimize the server-side stack by gathering “out of the box” metrics, delivering transparency into performance, and generating alerts as needed. It expanded over time to include New Relic Browser™—at first to confirm if the server side was functioning correctly but eventually for its ability break out factors like page-load and library-load times across all different browsers, to instrument usage more granularly, and to more easily track down and “squash” JavaScript errors.

New Relic Insights™, meanwhile, helps MLBAM to more easily surface data from other New Relic products. And because mobile app performance is so often affected by factors beyond MLBAM’s own backend, the company is now getting started with New Relic Mobile™ to track performance of its many mobile apps, says MLBAM’s Mike Leonard. New Relic Servers™, and New Relic Synthetics are also playing increasing roles, including for transactional synthetics monitoring, and MLBAM is currently considering generating its own custom metrics. Put it all together, and MLBAM now has access to automated cross-team views and metrics of how its applications are performing.

Because each production team owns its products and services, each is free to choose the tools that work best for them. Tellingly, however, more than 80% of MLBAM applications running in the cloud are now monitored by New Relic.

“New Relic empowers our developers to experiment and work fast without compromising on the quality of the MLB fan experience."

Sean Curtis
Senior Vice President of Engineering, MLBAM

MLBAM uses New Relic from development through production

“We use New Relic throughout the entire development workflow,” in both pre-deployment and production servers, says Vaca. “As developers work on certain features, they can go to the New Relic dashboard and see how performance is affected,” both in testing and in the real world. New Relic is also integrated into MLBAM’s continuous delivery process, where it increases confidence in deployments, and consequently improves the frequency of software releases and updates. “New Relic empowers our developers to experiment and work fast without compromising on the quality of the MLB fan experience,” says Curtis.

Once applications are in production, New Relic helps MLBAM keep a watchful eye on performance, with New Relic Alerts™ notifying the appropriate technical teams any time a critical threshold is exceeded. “New Relic is normally thefirst thing that gives us that heads up,” Curtis says. “We can immediately address whatever is going on at that point without necessarily having users call in and complain,” he adds.

New Relic also helps MLBAM pinpoint issues regarding interactions with third-party services and APIs—and like many modern operations, MLBAM works with many, many outside services. “The Service Maps feature in New Relic becomes hugely important to us when there’s something critical happening,” explains Curtis. “To have a service map provide a visual of red, yellow, or green on status, including third-party services, is great because it really increases our ability to respond quickly and consistently deliver a good experience."

If things do head south, MLBAM puts customers ahead of technology. “We expect our downstream dependencies to fail.” Krone says. “We build our systems so that if we encounter a failure trying to determine if you have access, we just give you access. Customer experience first. It's so much easier to explain after the fact, ‘We had a failure, and we let some odd number of people watch when they shouldn't have," versus, ‘We blocked everyone out.’”

“There’s always a state of change, so measurement becomes even more important, and harder to do well, in an environment that's entirely dynamic. New Relic collects data across the entire ecosystem that allows us to understand how we're doing and how we can make it better.”

New Relic helps MLBAM come through in the clutch

With New Relic helping MLBAM identify the sources of performance issues, the company continues to reduce its mean time to resolution (MTTR) and improve the customer experience. “It raises the bar in many ways, including how our teams collaborate and the ability to resolve incidents in real time,” says Curtis. “We’re often able to realize these types of issues before they’ve had any adverse effect on our customers. We can route [users] around a problem until the root cause has been addressed.”

“The a-ha moment was really when a developer can see what's going on, diagnose what the actual issue is, and be able to remediate it and restore it, Villoslada says. Just as important, “It just makes for a better overall product.”

Critically, “New Relic makes it easier for us to accurately plan and prepare for big events,” says Curtis. “Using load testing, we can see and understand the influx of traffic and the effect of users coming through the system.”

“The earlier we can find out there’s an issue, the better our ability is to preserve the customer experience. In our business, we're competing against the biggest and best platforms in the world, so you have to provide the best customer experience.”

Sean Curtis
Senior Vice President of Engineering, MLBAM

MLBAM: fans using New Relic to better serve fans

Joe Inzerillo, executive vice president and chief technology officer at MLBAM, explains that “New Relic helps the organization collect data across the entire ecosystem that allows us to understand how we're doing and how we can make it better. You can’t make something super high quality until you can measure it, and if you can’t measure it, how do you know it’s super high quality?”

“We’re very fortunate to work with passionate folks who are true baseball fans,” concludes Krone. “They eat, live, and breathe our products. New Relic gives them the visibility they need to support their passion and build something fans will truly enjoy.” And to help make sure customers won’t be left in the dark during the most important moments of the big game.

Major League Baseball trademarks and copyrights are used with permission of the applicable MLB entities. Visit MLB.com