Thoughts on Java, OSGi and DevOPS

I’ve been using the RTCamp Ubuntu package for Nginx because it had ngx_cache_purge and ngx_pagespeed modules builtin. The problem with is that it’s still stuck on Nginx 1.8 version which doesn’t support HTTP/2 so I had to figure out how to do my own build based on the latest Nginx mainline version. Continue reading →

I recently had to move a project from one git repository to another existing repository under different source tree and I wanted to retain the history of each file. This is rather easy once you know how to do it but you can easily jack up things if you don’t know. So I wanted to write clear instructions for the next time I have to do this and hopefully it also helps someone else. Continue reading →

Share this:

Let’s Encrypt is an awesome free, automated and open way of protecting your site with https. As you may have noticed this site is using Let’s Encrypt certificate and I’ve started rolling it out to all my other sites too. With free https certificate there’s really no excuse not to use https only. In fact if you want to take advantage of HTTP/2 you’ll need https since no one currently supports it unencrypted even though the spec doesn’t mandate it. Continue reading →

Share this:

When figuring out what’s wrong with a sites performance it’s important to get facts about every aspect and component involved with that site. Apache is quite often used in front of java applications and it’s the app server for php applications. Bad apache configuration can make a site seem sluggish even when there are plenty of other resources available so it’s important to see what’s going on here. From the first screen of Apache HTTPd plugin you’ll get a nice overview of all your monitored apaches.

Drilling down to a single server overview shows request velocity, cpu load, busy/idle workers and even bytes sent over time.

Going to throughtput shows throughtput details over time.

Workers section shows you what is happening with the workers. If you have a lot of busy workers you can see in what state their are and that might provide some insight to what is going on.

Installing Apache HTTPd agent for New Relic

1) I’m using the MeetMe New Relic Agent to monitor Apache HTTPd. It’s written in python and we’ll need to install pip. The following is using the Ubuntu python-pip package. You can find alternative install methods from pip docs.

apt-get install python-pip libyaml-dev python-dev

2) Next use pip to install newrelic-plugin-agent. When I ran it I got some errors but it still worked.

pip install newrelic-plugin-agent

3) Next we’ll need to create the configuration file for the agent. You can start by copying /opt/newrelic-plugin-agent/newrelic-plugin-agent.cfg or just use what I have posted below. The first thing you need is to set license_key. You can find your license key from your account settings page on rpm.newrelic.com. The second one is to add apache_httpd configuration. You can add multiple httpds to monitor.

Share this:

Apache HTTPd has always been my goto httpd, reverse proxy and load balancer but lately I’ve grown more interested in Nginx. It’s very high performance and lightweight not to mention easy to configure. Of course with my currently a single Nginx I wanted to see how could I hook it up my monitoring. Turns our there’s a New Relic agent directly from Nginx.

From the overview you can see the number of active and idle connections as well as the request rate.

From connections you’ll even more connection details. With very little connections and requests my graphs are currently slightly boring. In addition to connection details you can find more details about requests, upstreams, servers and cache.

Installing New Relic Monitoring Agent for Nginx

1) First you need to add the Ubuntu package repository for Nginx. If you’ve done this already when you installed Nginx you can skip to next step. If you are not using Ubuntu 14.04 like I am you can find the other Linux packages from Nginx website.

3) Next you’ll need to edit the agent configuration file in /etc/nginx-nr-agent/nginx-nr-agent.ini. You need to add your license key which you can find from your account settings page on rpm.newrelic.com.

newrelic_license_key=YOUR_LICENSE_KEY

Additionally you need to add a new source which points to your Nginx status url.

[source1]
name=localhost
url=http://localhost/nginx_stub_status

4) You’ll need to add a server block to Nginx for the status. Since I had very simple configuration in my Nginx I just added the following to /etc/nginx/sites-enabled/default

Share this:

In order to size your system correctly you need metrics and you need those also from your database. Not only is it important to know what’s happening in your database for capacity planning it’s also invaluable information when something goes wrong. If you have a database related performance problem in your code seeing how the fix effects the database can help you see if you actually solved the problem. Also looking at database statistics you might be able to spot a issue before it becomes a serious problem.

New Relic has a wonderful plugin framework and there’s a ton of ready made plugins and also SDK and API for things it doesn’t already support. MySQL plugin is one of those ready made plugins and it provides all the key information you’ll need. The MySQL plugin page quickly shows what’s going on all monitored databases.

When you drill down to a individual database server the overview shows the SQL volume and how it’s split between reads and writes. More key metrics are displayed under Key Utilizations. You’ll also find database connections and network traffic on this page.

Going further down to Query analysis you’ll see in more details about the queries.

If you are using InnoDB there’s a separate page to show key metrics from InnoDB.

Installing MySQL / MariaDB Monitoring

1) MySQL plugin can easily be installed with New Relic platform installer. So the first thing you need to do is install the platform installer. You’ll need your New Relic license key which you can find from account settings on rpm.newrelic.com. Once you have that you can install it with following one liner which is for 64bit Debian and Ubuntu.

2) Next go to the newly created newrelic-npi directory and run install. You’ll want to answer yes to all the questions and when prompted to configure the plugin grab the configuration from the next step.

./npi install nrmysql

3) If you skipped configuration you can configure the plugin afterward too. You can find the configuration file under newrelic-npi from plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/config/plugin.json. Below is a sample configuration for MariaDB (works for MySQL) running on localhost and we’ll be creating a separate user newrelic with password somepassword which the plugin will use to gather data. You can connect to multiple databases with the same agent. I usually install this agent on the same server my nagios is running on.

5) Last thing is to start the service but before we do make sure you have Java installed as this agent is written in Java. If you don’t have Java installed check my unattended Java install script. Otherwise you can proceed to start the service that should have been created during npi install if you answered all the questions correctly.

service newrelic_plugin_com.newrelic.plugins.mysql.instance start

Now it may take few minutes before you see your server under Plugins MySQL in rpm.newrelic.com. If it doesn’t check the log under plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/logs/ for hints and make sure the agent actually started.

Share this:

With New Relic Server Monitoring you’ll see all the important information about your system with just one glance. This is a essential tool for troubleshooting performance issues and also seeing that your system is properly sized. Sometimes poor application performance has nothing to do with the application but rather the system it’s running. If the system is not correctly sized you might be running out of memory, cpu or the bottle neck could be disk io. Without proper monitoring it is very hard to pinpoint the cause.

The servers listing gives a nice overview of all servers and you an easily see if there’s any issues.

New Relic Servers

When looking at a specific server you’ll see a history of it’s CPU and memory usage as well as load average and network I/O. If you have any APM enabled applications installed you’ll see a overview of their response times, throughput and error rate. You’ll also see some of the top processes running on the server.

New Relic Server Overview

When you drill down to processes listing you’ll quickly see the top memory and cpu consumers. You can also look at the history of individual processes.

3) Next you’ll need to tell it your license key so that it reports the data to your account. You can find your license key from your account settings page on rpm.newrelic.com. You can either edit the configuration file or you can set the license like shown below:

/usr/sbin/nrsysmond-config --set license_key=YOUR_LICENSE_KEY

4) Finally once everything is configured you can start the system monitor daemon.

service newrelic-sysmond start

Now in few minutes you should start seeing your server listed under Servers on rpm.newrelic.com

Share this:

New Relic is a wonderful software analytics suite that is 100% SaaS. I love it because it’s so easy to setup compared to Nagios, MRTG and other on premise software. Also their Lite edition is free with 24 hour data retention and for 30 days you’ll get to see the power of the Pro version. I still use nagios for my main monitoring and create some key graphs with MRTG but the data junkie in me loves all the data New Relic gathers and shows in nice graphs.

New Relic has seven parts or products as they call them. Those are APM, Insights, Mobile, Browser, Synthetics, Servers and Plugins. I have myself used only APM, Browser, Server and Plugins which are included in the free Lite edition.

APM

APM is the application monitoring part. It focuses on providing information about the application itself. The Lite edition shows you response times, throughput, web transaction information. It’s basically a low impact profiler. With the Pro subscription you get much deeper analysis of time spent on executing SQL, JVM statistics etc.

Browser

The browser provides insights on client side performance. Even though your application might respond quickly the users perceived performance could be poor because of network performance or even how the page is rendering on the browser.

Servers

Servers as the name suggests provides performance information about the actual server your applications are running on.

Plugins

There’s a ton of plugins to provide monitoring capabilities to systems not otherwise supported by New Relic and with it’s SDKs and API you can build your own plugins. Some of the plugins I have used are for MySQL/MariaDB, Nginx and Apache.

Mobile

Mobile is APM for mobile applications.

Synthetics

Synthetics allows you to test your application from around the world. It can check business critical user flows and interaction to make sure your site available and functioning from around the world.

Insights

Insights is a paid feature that combines business metrics with performance data. It can combine data from APM, Browser, Mobile and Synthetics for deeper analysis and segmentation and filtering.

Since I recently installed bunch of new servers and I had to refresh my memory on how I installed and configured each of the agents so I decided to write a series of articles on each of them. Here’s a list of topics I’m going to publish and as I publish them I’ll link the topic to the article. These topics will cover APM with Java and PHP, Servers and Plugins for MySQL, Nginx and Apache

Share this:

I like to to automate all the tasks I do often and of the things many of my virtual servers need is Java JDK. Unfortunately the Oracle JDK is not available as debian package but there’s a way to make it. This is where WebUpd8 Team PPA comes in as they provide installer for java6, java7 and java8.

Below is the script I use to install it unattended. You can download it also from github gist. If you want Java6 then just use oracle-java6-installer and for Java 8 oracle-java8-installer. This also works other ubuntu versions just substitute trusty with the code of your ubuntu release like precise for Ubuntu 12.04. Hope you find this useful.

Share this:

It’s been a while since I last posted anything and the site has been quite stale and out dated. So I thought I’d update it to a more modern responsive site that looks good on both desktop and mobile. As I wanted to focus more on actual writing and less on building a site I moved back to WordPress which is still the best blogging platform. My only concern is now security as php apps are much more prone to being hacked than Java ones.

With this new focus I’m also going to write about much more broad topics than just Liferay. As Liferay is moving it’s core towards OSGi I’ve studied it a lot and grown to love it although it doesn’t come without it’s own challenges especially for some moving to it from Java EE. As some of you may know I’ve been running and administering my own Linux servers for more than 15 years so some of the new topics will be about virtualization, containers, monitoring etc. I still read a lot of books so when I read one that deserves a mention I’ll write a book review. I hope you enjoy what’s to come.

As I built this new site I only migrated posts that had been read in the past year so if you run into something that’s no longer available do email me I still have them saved. Also all the old urls should be automatically redirected to the new ones.