When an organization gets to the point where they realize they need an Application Performance Management (APM) solution, things are usually on fire and getting more unmanageable by the day. For any organization, the purchase of an APM toolset is a large budget item and the pressure is getting these new tools into production and delivering value today, not tomorrow. In these circumstances, it is often difficult to make the time to really think through the process and do the implementation correctly. I know all too well how this feels, because I have been in this position with APM solutions on several occasions. I have deployed almost every major APM solution in this exact situation – executives breathing down my neck to both fix problems and make sure that the money spent delivers value quickly. An alternate title for this post may be “Don’t Make the Same Mistakes I Did” or even “My Pain, Your Gain”.

Tactical Deployment vs. Strategic Enablement

When it comes to enabling agent-based APM solutions in your environment, it’s typically a relatively easy task. Put agent binaries on the machines you want to monitor, modify your application startup parameters to use the agent binaries, restart the application, and presto, you have an application monitored by APM – or so you can tell your bosses. However, the reality is, if you don’t think through how the agents are organized within the solution, and you don’t plan for how you will automate the deployment and enablement of those agents, you will get nowhere near the value you could get from the money spent on your solution. It is very hard to put the brakes on and wait to do things right, but even with the smallest level of effort ahead of time, you can reap 10 times that effort in lost productivity and greater visibility down the line. This small amount of effort is what enables your APM solution rather than merely deploying a product.

A Tale of Two Deployments

I currently spend my time in the field helping AppDynamics enterprise customers get the most value from their APM investment. However, these stories come from my past experience deploying APM solutions at large Fortune 500 companies, pre-AppDynamics.

At one of these companies, we had just made a significant investment in an APM solution, and the pressure was on to deploy the solution overnight. I thought it was really as simple as I described earlier, so I went about making plans to get this deployment done. The mindset was purely tactical (get agents out, turn agents on, PROFIT). When it came to questions like how the application should organized within the solution (what is a tier, what is a node, how should tiers be grouped together) it seemed as simple as just grouping tiers owned by one group together into a single bucket, one “Application” in AppDynamics terms. So this is what I did. In a matter of a few days, after some herculean efforts to get change requests in, planning for application restarts, getting change management exceptions, etc, all of the agents were up and reporting in to the APM solution. However, when I started looking at the visibility I gained from this solution I was perplexed because calls that should be flowing from one tier to another tier simply weren’t showing up as I expected. As it turns out, data that flows from one tier to another tier that was part of another “Application” were showing as a call to an external system rather than a call flowing through the other tier as I expected. Consequently, a significant reconfiguration effort had to made in order to get this data flow corrected. Even a little time spent initially understanding how to design the hierarchy in the APM tool would have avoided this significant headache, nevermind the credibility lost by deploying something that didn’t deliver the value promised.

At another company, I took the time to make a bit of a sketch of how I thought the data should flow from tier to tier. One meeting with the application team who owned the monitored application was sufficient to whiteboard this high level picture, and from here determine what the proper agent configuration should be. With this knowledge, I then developed scripts that did the work of pushing the agent binaries out to the servers and configuring the relevant properties to enable them. In a few days longer than was required to simply push the agents manually and ask questions later, I had the agents deployed, and a fully enabled APM solution.

Lessons Learned and Applying Lessons to AppDynamics Deployments

The most important thing to consider when deploying any APM solution is how the tool enables you to understand true application performance. One key element of this is the application hierarchy inside of your APM tool. A good way to think of this in AppDynamics terms is as follows:

Application – This is the top level of the hierarchical configuration. All services that talk to each other should be in the same AppDynamics Application. For most customers, from very small to the biggest customers using AppDynamics, the starting point should be a single Application. Only when it is absolutely known that a monitored process communicates with no other monitored processes should they be broken out into a separate application.

Tier – A tier should combine all runtime elements that implement similar services. If there are several runtimes that are load balanced, this is a good indication that they belong in the same tier. If two runtimes implement the same services, they should be grouped together.

Node – This is the lowest level element in the AppDynamics monitoring hierarchy and the easiest to understand. This is a single monitored run time, whether it be a JVM, a .NET CLR, or a PHP server.

Take the time to briefly sketch out a high-level flow map, and this will get you thinking in the right way about how to lay these things out. If you need assistance, reach out to AppDynamics field enablement teams to help you understand how this should be laid out. Additional information can be found in the AppDynamics Center of Excellence Best Practices Guides.

Lastly, take the time to automate your agent rollout and enablement. Building an APM solution should be an iterative process, where you can easily deploy and remove agents, and reconfigure them as needed. Doing this small amount of pre work will save you much of the pain I experienced in my earliest APM deployments.

Take five minutes to get complete visibility into the performance of your production applications with AppDynamics Pro today.

The Old Enterprise Sales Routine

When I was an IT Architect at a financial services company I got to interact with a lot of software vendors. Sometimes I had a great experience, sometimes it was terrible, most of the time I didn’t want to be bothered by the vendor. I really just wanted to test out their software to see if it met my needs.

In order to get my hands on their software I had to go through an initial sales call, scoping call, and convince the vendor that I was not just kicking the tires (even if that is exactly what I wanted to do). There were times when I was interested in certain technologies and wanted to understand how they worked so that I would be knowledgable and prepared when right time presented itself to implement the given technology. Regardless of my reasoning, I wanted to get my hands on certain software to try it out in my environment without all of the extra BS that the vendor was going to put me through.

When I finally did get the software into my lab environment I could usually see why the vendor was so guarded about letting people try to use it on their own. It simply wasn’t user friendly and it usually took a very deep understanding of the underpinnings to make it work (customization)in most environments. This is a major problem for most traditional enterprise software and it has been a significant inhibitor to the adoption of self service trials for many years.

There’s a Better Way

Thankfully there is a new breed of enterprise software emerging that is both easy to use and loaded with powerful features and functionality. This new paradigm in enterprise software has allowed vendors to implement self service strategies at varying levels. Most vendors are offering watered down, simplistic versions of their software to satisfy a check box on a requirements list while other vendors market their SaaS offering as something that it really isn’t.

Today AppDynamics has released a full featured self-service trial of it’s enterprise Application Performance Monitoring solution. There is no marketing fluff, no watered down functionality, just the full AppDynamics Pro software that our paying customers use every day. You sign up, you download, you implement, you use it, you determine if it’s right for your use cases. Want some help along the way? Fine, let us know and we’ll be there for you. Want us to stay away while you test it out? That’s fine too, just let us know if you have any questions.

Introducing AppDynamics Pro Self Service

Let’s walk through the process so you can see how easy it is…

Step 1: Head over to www.appdynamics.com, click the link to the the free trial, and fill in all of the relevant details.

Step 2: Verify your email address, click the continue button, and choose whether you will use a SaaS or on-premise installation (I’m going to use SaaS).

Step 4: Watch as your application appears on the management server screen and follow the helpful instructions that pop-up to guide you along.

AppDynamics Pro is a fully functional self service SaaS platform that monitors your distributed applications. It’s not watered down; it’s not a marketing gimmick; it’s the full product that you can use to solve your application problems today. Click here to start monitoring your application in just minutes.

Unicorns, those magical mythical creatures that many have searched for but never actually found. One of our customers recommended AppDynamics to their associates and compared us to “Unicorns … only real.” This analogy is really great since Enterprises have been searching for “software that just works” but up until recently haven’t been able to find it. So now that we’ve found them, lets talk about 2 awesome Unicorns, AppDynamics and PagerDuty.

Recently we released a couple of blogs about the AppDynamics and PagerDuty integration. If you haven’t had the chance yet you can check them out here and here. I had some time to sit back and really think about what these two companies and our integration mean to the IT world and I want to share those thoughts with you.

I’m a person that has worked in many sizes of company from really small startups (less than 20 employees) to really large enterprises (more than 250,000 employees) and a few in between. IT support levels vary greatly within these different size organizations. In particular, the ability to detect problems and notify the right people quickly is an issue in the SMB world (at the companies I worked for anyway).

One of the reasons for this problem lies in the costs associated with traditional monitoring and alerting systems. Beyond the up front purchase price there is typically the ongoing configuration and maintenance costs which can drive TCO excruciatingly high in no time. When thinking about SMB, taking into account the high purchase price, high setup cost, and high maintenance costs it’s no wonder very few companies invest in the software they need to monitor and manage their environment properly.

Taking it a step further, it’s a shame that large enterprises have to pay these exorbitant costs and suffer through “Enterprise Class Software” that takes an army of highly paid consultants and/or employees to setup and maintain.

This is why AppDynamics and PagerDuty is a big deal to me. Enterprise quality software that is as easy to use, configure and maintain as consumer software while not sacrificing functionality. This was unheard of 5 years ago. Thankfully, things are changing rapidly for the better. AppDynamics and PageryDuty allow any company to quickly deploy, configure, manage, identify, isolate, alert, troubleshoot, automate, repair, etc… All of this done better than the Enterprise Class products of 5 years ago and at fraction of the TCO.

Specifically, here are a few of the things that are way better when you use AppDynamics and PagerDuty:

Monitoring

90% less configuration and management work with better results.

Isolation of problems down to the node, page, transaction, or line of code level.

Automatic remediation of known problems.

Reduced dependency on “The Expert” who actually knows how to set up and use the monitoring tool.

Alerting

Ability to interface with modern devices (like sending push notifications to iOS and Android)

Easy to use graphical interface for configuration of advance rules.

On call scheduling so you don’t have to “pass the pager”. Yep, there are still pagers out there.

Automated escalation of alerts that have not been responded to yet.

When it comes right down to it we are in a time where software is being re-invented and every company from the biggest to the smallest need to re-evaluate their strategy and take advantage of the amazing tools at their disposal. Here’s your chance to catch a Unicorn, don’t miss out by looking the other direction.

Click here to start your free trial of AppDynamics and catch a Unicorn for yourself.

It’s been about 12 years since I last scripted in PHP. I pretty much paid my way through college building PHP websites for small companies that wanted a web presence. Back then PHP was the perfect choice, because nearly all the internet service providers had PHP support for free if you registered domain names with them. Java and .NET wasn’t an option for a poor smelly student like me, so I just wrote standard HTML with embedded scriplets of PHP code and bingo–I had dynamic web pages.

Today, 244 million websites run on PHP which is almost 75% of the web. That’s a pretty scary statistic. If only I’d kept coding PHP back when I was 21, I’d be a billionaire by now! PHP is a pretty good example of how open-source technology can go viral and infect millions of developers and organizations world-wide.

Everyone and their mother is talking about big data these days – how to manage it, how to analyze it, how to gain insight from it – but very few organizations actually have big data that they have to worry about managing or analyzing. That’s not the case for FamilySearch, the world’s largest genealogy organization. FamilySearch has 10 petabytes of census records, photographs, immigration records, etc. in its database, and its data grows every day as volunteers upload more documents. Ironically, this organization that’s tasked with cataloging our past is now on the forefront of the big data trend, as they’re being forced to find new and innovative ways to manage and scale this data.

From 2011 to 2012, FamilySearch scaled almost every aspect of their application, from data to throughput to user concurrency. According to Bob Hartley, Principal Engineer and Development Manager at FamilySearch, AppDynamics was instrumental in this project. Hartley estimates that FamilySearch saved $4.8 million over two years by using AppDynamics to optimize the application instead of scaling infrastructure. That’s a pretty big number, so we broke it down for you in this infographic:

From 12 application releases per year to 20 application releases per year

From 10 PB of data to approaching 20 PB of data

No additional infrastructure

Response time reduced from minutes to seconds

Before AppDynamics

227 Severity-1 incidents/year took 33 hours each to troubleshoot

300 pre-production defects per year took 49 hours each to troubleshoot

This amounts to a total of 36,891 man-hours spent on troubleshooting every year

After AppDynamics

FamilySearch estimates that they saved $4.8 million with AppDynamics in two years. That’s a huge number, so let’s break it down:

Infrastructure Savings:

FamilySearch would have had to purchase 1,200 servers at approx. $1,000 each, amounting to $1,200,215 in savings

Those 1,200 servers would cost $2,064,370 in power and air conditioning

Those 1,200 servers would cost $200,000 in administrative costs over two years

Productivity Savings:

FamilySearch estimates that they’ve reduced troubleshooting time for both pre-production defects and production incidents by 45%, amounting to $885,170 in savings for pre-production and $460,836 in savings for production incidents (based on average salaries for those positions).

To learn more about what FamilySearch accomplished and how they use AppDynamics, check out their case study and Bob Hartley’s video interview on the FamilySearch ROI page.

This week we finally arrive at the end of my series. If you’ve implemented the suggestions in my other posts you’re probably an up and coming rockstar in your organization. Now that you’re on the right track you need to avoid becoming a one hit wonder. This post is dedicated to all the one hit wonders out there, and will act as your guide on how to avoid this scenario. So in honor of the one hit wonders of past years and decades I’ll be scattering some song names and artists throughout this post as a reminder of their fleeting fame and glory.

Build and Lead a Center of Excellence (CoE)

The Reason – Hoobastank

“And the reason is you…” – When you’re at the top of your game it can be difficult to figure out what you want to do next. In most organizations you wont maintain your rockstar status for long without diversifying your skill set and securing your next big win. The “What have you done for me lately?” mentality is alive and well in the corporate world. Give the masses what they want (your next hit) by starting a CoE, inviting performance geeks from all across your organization, and exchanging information. You’ll probably hear about all kinds of performance nightmares and have the opportunity to help solve them with your tools and expertise. You will also probably get exposed to cool projects that other performance geeks are working on and get a chance to improve your skills. The CoE is a powerful tool for any organization.

Architect the Future

Epic – Faith No More

“You want it all but you can’t have it…” unless you are the Architect deciding the direction of tooling for your organization. Larger organizations will have architecture review boards or something similar that make decisions on the technology direction for the company. Attend these meetings and find out what it takes to become a board member.

“If you don’t have a seat at the table you’re probably on the menu.” – I have no idea who to attribute this quote to but it’s right on the money. Go get your seat at the table!!!

The Trusted Advisor

You’re a Friend of Mine – Clarence Clemons

“Oh you can depend on me, over and over…” – Being a trusted adviser means that you have built a strong relationship with someone and they value your viewpoint. If you are responsible for solving a problem for the business make sure you take time to nurture that relationship through conversation, email, chat, etc… Be sure to check in occasionally and ask is there is anything that you can help resolve. The more problems you help out with, the stronger your trusted advisor status becomes, the brighter your rockstar shines. It’s a cycle you want to be caught up in. This is a role that adds tremendous value to your organization and to your personal career so make sure you give this the attention it deserves.

Customer/Vendor Relationship

I Can Help – Billy Swan

“When I go to sleep at night, you’re always a part of my dream” – I’ve had my work invade my dreams too many times. It’s definitely a sign that you are immersed in what you are doing and is scary and pretty cool at the same time.

It’s really important that you have a running dialogue with your software vendor after you make a purchase. You need to make sure your vendor keeps you updated on new product features, best practices, and their product roadmap. It’s not all up to the vendor though, you need to be engaged at multiple levels described below…

Enhancement Requests

You will find that there are features and functionality that could really help in certain circumstances but that are missing from the product you purchased. This happens with every product if you use it enough and you need to keep a list of these enhancements along with use cases to justify and clarify each item. It also helps the vendor tremendously if you can keep the list prioritized and assign a high/medium/low importance to each line item. This benefits both you and the vendor, a true win-win.

Support Calls

Another certainty when using a software product long enough is that you will need some support when the product doesn’t work as expected. Be sure to stay involved with support cases so that you know how responsive the vendor is as well as what problems have already been encountered so you will recognize them if they crop up again in the future. This level of engagement can also help you avoid issues because you already know the scenario that causes a certain problem.

It’s also vitally important to make sure your vendor is responding in a timely manner. Problems that drag on for too long with too little communication create a bad overall impression and can impact the business. This is bad for you, your business, and the vendor. If you notice a pattern of problems that take too long to resolve or just too many issues be sure to address it with your vendor as soon as possible.

User Conferences

Most vendors will have at least annual users conferences. These conferences are a great time to network with other users and learn how they are solving problems, what kind of issues they experience, and understand their best practices for deploying and using the product. It’s also a time where you can learn about upcoming product features and even get to meet the folks who are directly responsible for your tool. There is a wealth of very specific knowledge at users conferences for you to take advantage of so make sure to get approval and reserve your spot early.

Tracking Your Success

The Future’s So Bright I Gotta Wear Shades – Timbuk 3

(No quote needed!) – I’ve said it before and I’ll say it again. Document your successes, ALL of them, and translate each success into business value. By doing this you get the following benefits:

Documentation of your value to the business for raises, bonuses, reviews, etc…

Track record of past success used in business justification for new products.

Business justification when it comes time to renew your existing product licenses.

Credibility in presentations when you are meeting with new organizations or convincing application owners to implement your solutions.

Becoming a rockstar in your organization is a great accomplishment that can be easily tarnished by becoming a one hit wonder. Don’t let it happen to you. Become and expert and constantly look for opportunities to share and improve upon your knowledge and skills.

If you’ve read all of the posts in this series you probably deserve some sort of prize (as if your growing rockstar status isn’t enough). Feel free to tweet to me (@HirschOnAPM) to let me know of this gargantuan accomplishment and I’ll see if I can hook you up with some AppDynamics swag as a prize. Now go forth and be a rockstar!

This week we focus on Dashboards and Reports. I’m a huge believer in unlocking the information and intelligence contained within your monitoring tools, and turning that data into actionable information. Over time your tooling ecosystem will collect a wealth of data that can be used for capacity planning, business intelligence, development planning, and many other business and IT activities that require information about usage patterns and operational statistics. By unlocking this value, and surfacing it to the business and IT organizations in a meaningful way, you take another step up the maturity ladder and solidify your rockstar status within your organization. A huge benefit of providing useful information to the business is that they will fight for your tools and projects if they are ever in question!

Dashboards

Dashboards should be used to provide real-time insight into critical business and IT indicators. A failed server doesn’t always mean that there is impact to the business.

Several of our customers are using AppDynamics dashboards to better understand business activity. For example, we have multiple e-commerce customers that are tracking revenue and order volumes in real time.

Let’s take a look at some different perspectives within the organization and explore what type of dashboard each role requires.

Management

Most managers don’t need to know the sordid details about the infrastructure that supports the business applications. For each application within the managers realm of responsibility they need to understand key business indicators and have an overall indication of their application performance and health. Take a look at the dashboard below…

This is the type of dashboard that managers want to keep an eye on throughout the workday. Any impact to the business will be easily identified on this dashboard so that the manager can make sure the impact is being addressed. There should be alerts associated with these metrics so that the operations center is notified of business impact but it’s not always an IT problem when business metrics deviate from normal behavior. It’s possible that sales volume is impacted by a competitor offering lower prices on the same product. This will show up as business impact and there is nothing that the IT staff can do to fix it. This is a business problem that needs to be addressed by the appropriate department.

Of course any IT infrastructure problems that impact the business will also show up on the managers dashboard but only by way of business metrics that have deviated from normal behavior. This fosters communication between IT and the business which should lead to improved cooperation and coordination over time.

Application Support

Application support rides the fence between the business and IT. If something goes wrong with their application, the business makes sure that app support hears about it and gets the problem resolved in a timely manner. Application support teams need a view of key business indicators (similar to what the managers are looking at) as well as key IT indicators so they know when there are problems with the application or the infrastructure they depend upon.

Operations

The operations teams need to know when infrastructure components are nearing capacity, throwing errors, or failing completely. It is their responsibility to ensure the proper functionality of the infrastructure and as such need an infrastructure centric view of the components they are responsible for. This is a very classic enterprise monitoring view that has been long established and still has its place in the modern monitoring world. With that said, its beneficial to show the operations teams a few key business indicators so they know how urgent it is to replace that failed piece of hardware.

Reports

Given that dashboards are best utilized to provide real-time status information, reports are the go-to solution for information that drives action in the future. Reports can be about the application, infrastructure, business, or anything else that makes sense given the data you ave to work with. What I want to focus on in this blog are the insights I have picked up over the years when developing and utilizing reports at my past companies.

Reports should contain information that is actionable. There is little value in receiving a report that you cannot use to decide if action is required.

Reports need to be concise. Very few people are interested in reading a 50 page report. Try to keep them to a few pages or at least summarized in the first couple of pages with supporting details in the rest of the report.

Don’t send an avalanche of reports. People usually don’t have the time or desire to read multiple reports per day. Ideally you should send reports only when there is something in the report that needs attention.

If you can, include the source of the report and a description of what they report is used for. I’ve received reports in the past where I didn’t know what system sent them and why I needed to see them.

Know your audience. Make sure the business gets business related information and IT gets technology related information.

Don’t blast reports to email groups (usually). Most reports need to be seen by only a few people in your organization. Email distribution groups tend to contain way more people than are truly interested in your report. No need to clog up the email system with a 50 page PDF sent to 2000 addresses.

Reports are important and can help your organization avoid issues down the road but you need to implement them carefully or they become just another piece of internal spam. The best report that I can ever receive is the one that only shows up when there is a problem brewing within the next few weeks, clearly identifies the problem, and I am the right person to make sure it gets addressed. That would be reporting done right!

Dashboards and reports need to be implemented properly to get the most out of your monitoring investments. You can amplify the value of all of your monitoring investments by combining, analyzing, and displaying the data contained within each disparate repository. The most mature organizations will build out their monitoring ecosystem and then invest in analytics to derive business and technology value and to create the best dashboards and reports possible.

Thanks for taking the time to read this series. The final post will be next week and will focus on how to keep up the momentum and stay relevant when it comes to monitoring within your organization.

This week we’ll take a look at one of my favorite aspects of deploying APM properly … showing off (just a little bit) and helping others find their inner rockstar!

How do I help others find their inner rockstar and why would I want to do that?

When it comes right down to it you need to show results. You most likely played a big part in convincing someone to spend money on APM software, or had a big enough problem with an application that someone decided to help you out by purchasing APM software, or you just really want to show your company how good you are at your job. Whatever your motivation, a major key to success lies with getting solid results across your organization. If you are the only person using a tool successfully the odds are stacked against you when it comes time to renew your subscription or maintenance for that tool.

Welcome to Part 4 of my series Deploying APM in the Enterprise. In the last installmentwe covered how you find, test, and justify purchasing an APM solution. This blog will focus on what to do after you’ve made a purchase and started down the path of deploying your coveted APM tool (ahem, ahem, AppDynamics, ahem). Just clearing my throat, let’s jump right in…

It’s time for a celebration, time to break out the champagne, time to spike the football and do your end zone dance (easy there Michael Jackson, don’t hurt yourself). All of the hours you spent turning data into meaningful information, dealing with software vendors, writing requirements, testing solutions, documenting your findings, writing business justifications, and generally bending over backwards to ensure that no objection would stand in your way has culminated in management approving your purchase of APM software. Now the real work begins…