Introduction

Spotless is a web service to filter incoming feeds according to business rules as they come into your data platforms. It's easy to incorporate into your applications to ensure that you always have clean data to work from. There are different types of rules to validate whether data is correct and some various options for how invalid data is handled:

Number validation checks the formatting of the number and the range of the number

String validation checks the length of the string and can also verify it using a regular expression

Date validation checks the data format and can also check the range of dates

Lookup validation checks that the value in the field is one of a specific number of values in a separate file. This could be a fixed list like city names or link to an external file for database reference validation

Uniqueness rules check that one or more columns are unique and can remove duplicates

When you execute a filter, Spotless creates a job which then provides a report on the cleanliness of the file and actions that have been taken.

Getting started

We have several demonstration videos, including the one below that shows how you can use Spotless. In essence, you should start by uploading a CSV file here, customise the rules to suit your needs, and then hit “run.” There is an example file you can download here for your testing.