OMS Log Analytics: Collect, Visualize and Analyze Log File Data

This blog post uses a simple test scenario to show how we can collect, visualize and analyze data collected from a log file, with OMS Log Analytics. This test scenario can be used in a POC (Proof-Of-Concept) to demonstrate how the following features in OMS Log Analytics – Custom Logs, Custom Fields, and Direct Agent, can be configured to work in coherence to provide a custom monitoring solution.

Here is a high level view of what needs to be configured:

Preparing the Log File:

For our test scenario, lets say we have a customer facing application that logs information about each critical transaction at a frequent basis into a text file in the following format:

This format satisfies the criteria for the text file to become a Custom Log data source in Log Analytics. For more information on Custom Logs and the data format criteria, refer to Brian Wren’s guide on Custom Logs in Log Analytics.

The text file can be stored in a folder of a Windows or Linux computer, eg. C:\TempPath

Install OMS Direct Agent:

The Microsoft Monitoring Agent for OMS Log Analytics can be installed on the Windows computer where the text file is stored, and configured to report and send data directly to a specific OMS Workspace.

Configuring the Custom Logs Feature:

On the Overview > Settings Dashboard page, select the Data tab, then select the Custom Logs option and click the Add+ button to open the Custom Log configuration page. Upload a sample version of the text file, select a record delimiter (New line or Timestamp), enter the path of the text file, eg. C:\TempPath\SampleApplication5.log, and provide a name as the “Type” to categorize the data collected from the text file. eg SampleApplication5_CL. Additional information can be added into the Description field. Refer to Custom Logs in Log Analytics for more information on how to configure the Custom Logs feature.

Configuring the Custom Fields Feature:

First, validate that the data in the text file is being collected successfully and searchable in OMS Log Analytics by running a search query of Type=<FileDataTypeName>_CL, eg. Type=SampleApplication5_CL

Then, create searchable fields by highlighting and extracting the words or strings of interest in the raw data field of an existing record by using the Custom Fields feature. For searchable fields with numerical values, use the Numerical Field Type while for fields with strings or names, use the Text Field Type instead. For more information on how to configure Custom Fields, visit Brian’s guide.

Important Note:Custom fields will only show up on data that comes into OMS after you configure them to be extracted. It does not retroactively extract custom fields, it only performs extractions during ingestion time.

Hence only the data collected after the custom fields have been configured for the specific type will have their new searchable fields populated with the required values from the rawdata field.

With the Custom Logs and Custom Fields features configured, and data being collected successfully from the text file every time new data is written to it, the records can be visualized and analyzed using search queries in the OMS Log Analytics workspace portal.

Here are some examples:

a. A query to display the maximum response time recorded (in ms) for each transaction over time on a performance view on an hourly interval:

To simulate a scenario where the text file is updated with application transaction information on a frequent basis, here is a PowerShell script that generates random numerical values within a given minimum and maximum range for each transaction’s response time. Every time the script is run, it writes the information into the SampleApplication5.log file located in the server installed with the Microsoft Monitoring Agent for OMS. The PowerShell script can then be triggered on an interval of 10 – 15 minutes with the task scheduler to update the contents of the text file and allow new data to be collected into OMS Log Analytics via the Direct Agent.

Disclaimer: All information on this blog is provided on an as-is basis with no warranties and for informational purposes only. Use at your own risk. The opinions and views expressed in this blog are those of the author and do not necessarily state or reflect those of my employer.