Merchant Product Datafeeds/Data Feeds for Affiliates - 101

Article by Carsten Cumbrowski, January 2006, August 26, 2006, September 9 and November 15 2006.
This is an updated and greatly extended version of my original Scrap. Carsten, 11/15/2006
You can also download this article in PDF format .

Note: Datafeeds or Data Feeds are the same thing. Both spellings are being used in the industry (including the the grammatically wrong version). If you search for additional resources to the topic at forums or search engines, make sure that you use the singular and plural version (feed/feeds) in addition to the two different spellings. Search for "Datafeeds", "Datafeed", "Data Feeds" and "Data Feed" plus your additional keywords to get the best results.

Introduction

I have dealt with feeds from hundreds of merchants, including every U.S. affiliate network: Commission Junction, Linkshare, BeFree, Performics, ShareASale, Kolimbo, Dark Blue, you name it! The feeds come in all kinds of formats and via all types of delivery methods.

I have also, as an affiliate manager, implemented datafeeds for merchants. With this in mind, you can believe me when I say that I know what I am talking about. If you are an affiliate manager and want to provide a datafeed to your affiliates, you might want to print this article and show it to your IT department.

You can also contact me directly to ask any questions to affiliate datafeeds or give feedback on this document.

1. Datafeed Format

Delimited Text File. Please note: this is not CSV, which stands for comma separated values, already a very specific delimited text file format which I do not recommend for automated affiliate product datafeeds due to possible issues it can cause.

Using the tab character (chr(9)) as column delimiter makes a lot of sense for affiliate datafeeds, since it can't be used on the web and should be replaced in the content as multiple spaces anyway. Tilde ( ~ ) and pipe ( | ) will do fine as well. I would avoid using commas or semicolons.

I will provide a clarification in a moment.

Row delimiter should be either
(A) a standard windows line-break: carriage return/cr (chr(13)) + linefeed/lf (chr(10)) OR
(B) a standard Unix linebreak: linefeed/lf (chr(10)) only without a carriage return.

If a quotation mark (") is not specified as a text qualifier, don't enclose strings with it. Simply avoid using a text qualifier such as a quotation mark or double quotes because people often misuse them and there are no real benefits for using them in affiliate datafeeds.

Quotes and double quotes are often part of the feed content itself (product names like 19" monitor or Joe 'Jay' Doe Doll). If you use a text qualifier and the text qualifier itself is part of the content, the character in the content has to be replaced by 2x the character - ' becomes '' (2 single quotes) and " becomes "" to prevent the characters from being mistaken as the beginning or end of a column's content. The only case in which the text delimiter occurs doubly is when the column has no value. There is no fixed rule; if text qualifiers can be used for empty text values, but it is a special case that must be taken care of manually to prevent problems.

Column and row delimiters must be removed from the feed content itself or be replaced. They have a meaning for the feed structure, and will be mistaken as errors in the feed structure if they appear in the content.

You should provide column headers in the first row. Do NOT use SPECIAL CHARACTERS. You should only use 0-9, a-z, A-Z and, if you have to, the characters "-" or "_" to separate individual words. Do NOT use: !,?,&,%,#,',",(,),\,/ etc. Those are usually reserved characters in applications or programming languages. Also, avoid blank spaces.

Delimited files MUST have the same number of columns for every single record!

If information is not available for a product, send the initial value of the column instead (0 for numbers, 0.00 or 0 for currency, nothing for text and character values and 1/1/1900 for dates).

Yes, this comment is directed straight at Linkshare. To begin with, Linkshare uses no column headers. Further, using a first row with four columns ("HDR," MID, merchant name, date last updated) and last row with two columns ("TRL," number of products) is against the definition of "delimited."

(By the way, thanks for letting me know at the end how many products I just read in sequential order. My computer cannot count.)

XML format is fine, too, but only if you provide the data via a web services (SOAP/WSDL, REST etc.) or if you can generate correct XML (most people can't). Do not use only XML if you provide the feed as a file to download.

Use GZip to compress the datafeed file if necessary (for large feeds), don't use Winzip (not all affiliates are using Microsoft Windows).

2. Delivery Method

Offer FTP download, make the datafeed available via HTTP on the site (if the feed is < 5 MB) or set up a web service (SOAP/XML). If you can, do all three.

Don't provide FTP upload as the only delivery option (it's very kind if a merchant offers that, but most of the time it's impractical for affiliates). It's fine if it is an option, but it should not be the only option.

If you make it available as a file, don't make the name dynamic (make the creation date part of the file name for example). You might do that if the file is provided via FTP (NOT HTTP) for which affiliates can write a script that checks what file(s) are available.

The file is scheduled to be generated automatically on A RELIABLE SCHEDULE: daily, weekly, monthly, whatever. For files generated daily, make sure to consistently generate the file during the same time window - which should be as small as possible and during the night. Do it as early as possible so that affiliates can also pick it up (and process it) during the night.

When I say "night," I do not necessarily mean "at night where you live," but "at night" for your targeted customers, also known as "low business hours". Remember to consider that your customers may be in different time zones.

If you can, provide affiliates the information about a change of an HTTP datafeed that affiliates can check first if they need to update their content, before they waste time and resources downloading and processing the whole and unchanged datafeed for no reason.

Have a second file with a fixed filename available via HTTP that contains information like: most recent date/time the feed was updated, filename of current feed, size, number of records, etc.

Affiliates have different levels of sophistication and available tools. Some build links to selected products manually, some use simple scripts or software, and others have a high-end enterprise server at their disposal. Some run their stuff under windows, others use Linux.

The more formats and delivery methods you provide, the more affiliates can and will use your datafeed.

Documentation is always king. A description of the datafeed structure and format takes the guesswork out of working with feeds.