Category: Big Data

Point to point integration is hard to maintain when consumers increase and have slightly different demands. And how can we keep a fast delivery of our own service without the synchronized upgrades of multiple services at the same time to solve point to point integration dependency. One way is Continue Reading

Avro is a very good record oriented compact format and is easy to work with, this processor is a version of the xml2csv processor that I published a few weeks ago, but is improved and is now generating avro files instead of csv files. All code used in this Continue Reading

Upgrade all instances of your NiFi process groups with automation script NiFi has a nice registry function to manage versioning of process groups it is called nifi-registry https://nifi.apache.org/registry.html In this article I will show how you can maintain versions with nifi-registry and how you can upgrade or downgrade all Continue Reading

In GDPR Article 32 and Article 4 anonymization & pseudonymisation is mentioned as methods of securing personal information. http://www.privacy-regulation.eu/en/article-32-security-of-processing-GDPR.htm http://www.privacy-regulation.eu/en/article-4-definitions-GDPR.htm Anonymization of the data secures that the data cannot be used to identify an individual by masking or encrypting the data in a way that it cannot be reversed Continue Reading

Avro is a very commonly used binary row oriented file format, it has a very small footprint compared to text formats like CSV. Many processors like ExecuteSql with reads data from a database are returning the result in avro format. In this article I share a small groovy template Continue Reading

All code used in this article can be downloaded from https://github.com/maxbback/nifi-xml/ Problem with XML and design for converting XML to CSV XML is a common format used by many applications, so it will be one of the formats you need to support in your Big Data platform. XML is Continue Reading

XMl is a common format used by many applications, so it will be one of the formats you need to support in your Big Data platform. I have divided up this article in two part Part 1 Preparation Part 2 Coding and a complete working setup in NiFi Continue Reading

If you want to use a lookup table in NiFi to mask or complement the data in a feed you can build a simple processor with Groovy. The groovy code can also be found here https://github.com/maxbback/nifi_lookuptable In this processor I use a DB pool service for looking up addresses to Continue Reading

When you want your users to bring their own data, you soon realize that they will bring any kind of data and you need to figure out what they want to load. As an analyst you like to work with structured data if you can and many times you Continue Reading

I am found of quick solutions and groovy is a convenient way of developing small scripts for extending NiFi when you need it. NiFi has good processors for extracting data from a database, but sometimes you need your own, which I will show in another post where I will Continue Reading