Splunk’ing Netsuite is the act of aggregating logs from Netsuite into Splunk

– Roi

So what is this blog post about? Allow me to introduce the problem – we have Netsuite. We use it extensively in REA. We also have Splunk which is used, even more extensively, I reckon, throughout the company. We would like to have our Netsuite execution logs in Splunk. Sounds simple? well, as it turns out – that’s not that simple.

Introduction

Over the years we have created a lot of software at REA ranging from internal tools used by our customer experience team, to mobile apps used by millions of users each month, to data processing engines that crunch hundreds of gigabytes of data. While each piece of software has different functional and non-functional requirements we still have consistent expectations of that software as well as alignment on architectural principles. By taking this general consensus floating in the ether and capturing it using succinct language within high-level categories we have formed a foundation for discussion to ensure we can keep producing good software regardless of team, technology, and timeframe. Additionally, we routinely use this framework to assess all new and existing software to ensure our software continues to meet expectations and, where concerns are detected, to drive change.

The categories – overview

When considering the qualities of good software we use the following lenses:

There’s a pattern that I keep recommending to teams over and over again, because it helps separate concerns around I/O; sending and receiving things over a network, interacting with AWS, saving and loading things from data stores. It’s an old idea; you’ll find it filed under “Decorator” in your Gang of Four book.

For our purposes here, this is a compositional pattern that allows us to stack onion-rings of concern around an I/O boundary; strings on the wire, transfer protocols (eg HTTP) , formats (eg JSON), business concepts. There is no code in existence that should be concerned about more than one of these at once; they can be structured as stackable layers, and other modules can pick which level of detail they wish to talk to.

In REA, we have a list of highly sought after internal courses available on various topics including AWS, Docker and Elasticsearch. On 7th and 8th May 2018, we added Scala to the list, with over 20 people attending!

How did this all start?

Scala and functional programming have always had some mystique surrounding it. There is this idea that it is too hard or not pragmatic enough. Existing courses are very technical and many people including myself have struggled to complete them.

“Do we (still) need QAs?”

… and flavours thereof, is a question I have been hearing for years now.

The same has been asked of Tech Leads, Operations Engineers, “Front End” devs, “Back End” devs, Security, Iteration Managers/Scrum Masters, Business Analysts, etc. Anything that is not a full-stack dev. The #NoOps conversation is interesting research material.

It seems to me that this question stems from a misinterpretation of agile and lean startup materials.

Idealistic hopes of cutting costs, removing waste (and blindly classifying some roles as such), delivering faster, etc., have caused some to think that role specialisations can all be generalised into being “Engineers” and that we can all self-organise and just do it all.