In the previous Yahoo Pipes Tutorial, I briefly described what every module does. In this article, I demonstrate how to do many useful things with Yahoo Pipes.

Remember, the ultimate goal is to create great custom feeds that will automatically get posted on our websites. I showed you how to do this in a previous article How to Make Money Blogging. With those techniques, I was able to make an extra $11 bucks, after all fees, for each site I made. Each site took less than 20 minutes to put together and then required no maintanence. With Yahoo Pipes pulling even better articles, I’m pulling in $26 on average.

Pull Entire Article Using Yahoo Pipes

Normally, when you pull information from an RSS feed, you only get an abbreviated version of the article. Yahoo Pipes allow you to pull the entire article using the following steps. This is what the Yahoo Pipe page looks like after everything is set up. I’ll then describe the steps.

[adsense]

This Yahoo Pipe example, pulls every article from the current Huffington Post RSS feed, along with the whole article, if the title contains the word Hollywood. Here is how it’s built.

The Fetch Site Feed Module

I drug this module from the left sidebar and onto the working area

I plugged in the address, that I wanted it to pull feeds from

It pulled all of the information from the main feed

If I would have wanted all of the feed’s, I would have instead used the Feed Auto Discovery Module

Fetch the Whole Article for Each Feed Item

To grab the whole article, rather than just the abbreviated description in the feed, you need the Loop and Fetch Page Modules.

Drag the Loop module, from the left side bar, under Operators

Drag the Fetch Page Module, into the Loop Module

Under URL in Fetch Page, select item.link

item.link refers to the link for the entire article

To get the whole article, I need the tags that surround the article

Finding the Tags that Surround an Article

Go to the Huffington Post and click on any article

Copy the first few words of the article

View the articles page source

Search for the article words you copied, until you find the beginning of the article

Copy the tag that proceeds the article

Paste it in the Cut Content From section in the Fetch Page Module

You can find the other tag, by looking at which tag follows the ending for an article. To finish off the Fetch Page Module:

Select the radio button next to assign

Select the all option

Select item.description as the target

You could stop there, but I’ll explain how the Yahoo Pipes Filter Module works.

Yahoo Pipes Filter Module

With the Filter Module, you can either Block or Allow certain articles based of whether a certain word lies within some part of a feed item. Here I choose to only include articles that contained “Hollywood”, in the title. Here is how I did that.

Pull the Filter Module out on the stage

Choose the Permit option

Choose the All option

I then told the filter to only select items with Hollywood in the title

item.title, refers to the Title of every article in the feed

I then connected all of my modules, as you see above. Then I clicked save and done.

Manipulating CSV Data with Yahoo Pipes

CSV stands for Comma Separated Values. Here I’ll show you how to pull data from a CSV, and manipulate it however you see fit.

Here I created an RSS feed from images that I pulled from the Flickr website.

All I did was drag the Flickr Module on to the stage and asked it to return 50 photos of Pittsburgh, taken in Pittsburgh. It returned a map detailing where the photos were taken. It also returned a list of the photos and stuck them in a new RSS feed.

You could also use the Flickr module to populate your articles with interesting photos.

That’s All Folks

There are 3 interesting ways to use Yahoo Pipes. In the next article, I create a massive Yahoo Pipe that will use most of the modules. I’ll then describe how those modules work and you’ll be a Yahoo Pipe Master. Leave Questions Below

Well the feed is up and running. I checked into a few things and found that you get this error because of DNS issues. This can happen for numerous reasons and it is normally a temporary problem that will go away on its own.

I do know personally that Yahoo gets really irritated if you retrieve data from the feed to often (Every 5 Minutes). I’d only retrieve from it twice per hour at most.

Excellent post! Grabbing the full article with Yahoo Pipes instead of just the summary is great for autoblogs. But use this method at your own risk because there are some obvious legal issues by scraping the content. You might tick off a few people! Great technique none the less though.

Thanks for pointing that out. I only grab the first 800 characters from sites. My goal being to provide readers with just enough information to find out if they want to continue reading. I also always place a link to the original article and leave a reference to the original poster in the title.

I only provide original content on newthinktank, but make good money off of my autoblogs. Thanks for stopping by 🙂

I’ve tried to accomplish the same results as it has come out for you using Y-pipes grabbing from “huffingtonpost.com”.

So when I made this firs pipe by drag and drop in the “fetch feed” and type in the site url and then drop in the loop en put the “fetch page” inside it and filled in the div information, it showed me an error as the result…

This was the error message:
===========================
Error fetching http://www.huffingtonpost.com: Invalid XML document: org.xml.sax.SAXParseException: The entity name must immediately follow the ‘&’ in the entity reference.

B.t.w. I have tried some other sites to grab from them also, but the error is showing all the time as the result.

I know that I filled in all the fields correctly, so I’m confused here because of it.

Please Derek, can you tell me how to get around this so i can get the full article anyway?