For real world applications, we need a logger that gets out of the way. Logging is not the true value of our application. Rather, we need logger that is fast, to free up the maximum computing cycles for our business logic. And we need to collect the output of that logger (and others).

Caveat Emptor: Pino && Hapi && TypeScript

Pino also provides server.logger().info(‘some info’) as a form of logging with Hapi. However, if you have a TypeScript project then you are restricted to the type definitions supplied by @types/hapi, and they do not allow this form of logging yet.

Log aggregation with Event Hubs

Now we have a logger working, but a feed on a local machine is not that useful. We need to be able to get this information into a dashboard.

Here we need to take a step back and consider our architecture. We are deploying an app in the cloud, i.e., on a VM or container. Presumably, there are multiple instances of your app, and their other instances of, e.g., databases. There will also be other services if you are doing a micro-services deploy. You might also be collecting client-side or IoT data. Connecting all of these to a dashboard in parallel might overwhelm it. It makes more sense to connect to a log aggregation service. Event Hubs is the obvious candidate since this is part of our series on Azure.

Microsoft describes Event Hubs as:

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters. With the ability to provide publish-subscribe capabilities with low latency and at massive scale, Event Hubs serves as the “on ramp” for Big Data.

To collect our pino logs, we will provide a Web App and an Event Hub. Then pipe the logs from the Web App to the Event Hub:

Event Hubs can be used even if part of your architecture is on another cloud or own infrastructure. We will make use of this during development below to test the connection from our local machine to an Event Hub.

That said, it is still necessary for us to connect a dashboard by subscribing it to our Event Hub. We will do so in the next article.

Note that when using Event Hub there is a delay of ±10 minutes when viewing data on its dashboard. Keep this in mind as you build your own application.

Provision an Event Hub

In our previous article we demonstrated how to provision a Web App. Assuming you have an Azure and a deployment user, then this simple bash script will get you started:

Piping stdout to an Event Hub

We can now successfully post to an Event Hub, now all we need is to send all the Pino logs to such a posting mechanism. Since Pino logs to stdout, all we need is something like node yourapp.js | eventhub.

What we have is a stream of output from pino. First, we can break those into discrete events using split2. Then, POST these lines in batches (for better performance) to an Event Hub with a writable stream in object mode using https.

The alternative: AMQP

Alternatively, AMQP could be used to send the logs to an Event Hub. We stuck to https for simplicity.

Deploying to Web App

We have our basic Web App setup done already. However, to keep in line with Twelve-Factor App development, we want to simplify our npm run start command to node build/index.js | pino-eventhub. We have setup our npm module to take environment variables, so this can be done using the Azure CLI:

Again, you can go to your Event Hubs dashboard on the Azure Portal to see your events arrive.

Alternative: Application Insights

Instead of using Pino with Event Hubs, we could’ve opted for a solution like Application Insights (AI).

According to Microsoft:

Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms. Use it to monitor your live Web App. It will automatically detect performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app. It’s designed to help you continuously improve performance and usability.

All the functionality AI ties into looks great. AI, however, tightly couples to your application through monkey-patching. At the moment AI is instrumented for Bunyan, Console, MongoDB, MongoDB-Core, Mysql and Redis. However, for this application, we are looking at the lightest/fastest possible logging solution. Hence, Event Hubs made more sense for us.

Conclusion

Building on our previous article, we wanted to be able to collect the log files from our application. We showed that it is easy to provision an Event Hub on the Azure portal, using Pino as our logger. After creating credentials (using the tool we provided), your Web App can stream to an Event Hub from a single-line command.

In the next article in the series, we will select from the logs collected in an Event Hub, and represent them visually.