Introduction

This article describes WebResourceProvider, a simple yet powerful framework for retrieving useful information from publicly available web services. I use the term "web service" in a generic, non-Microsoft sense, to mean information providers such as:

The demo application included with this article shows how you can easily create objects to get:

stock quotes

the weather for a US zip code

the list of locations served by a US zip code

the translation of a piece of text

the list of broken links on an HTML page

the list of top posters at CodeProject

A Word of Caution

Before you use WebResourceProvider to write the next killer app, be aware that there are legal and ethical issues regarding the use of information obtained from other sources. In particular, the terms of service (TOS) of content providers such as Yahoo, CNN, etc. clearly state what you can and cannot do with information retrieved from their sites. Even if you write a web resource provider for personal use only, you should take into consideration any undue stress that your object may put on a web server. The CodeProjectTopPosters example in the demo won't let you get at more than the top 40 CodeProject posters. Further, it pauses between multiple accesses to the CodeProject server in order to not overload it.

How it Works

WebResourceProvider works by initializing itself, constructing a URL to be retrieved, downloading the resource, and extracting useful information from the downloaded content. The process is repeated until no more data needs to be downloaded.

You use WebResourceProvider by deriving your own resource provider class from it, and overriding any of these virtual methods (shown in red in the diagram on the right):

init

constructUrl

isPost()

getPostData()

parseContent()

moreAvailable()

WebResourceProvider provides an assortment of methods to help parse downloaded content. They are:

Acknowledgement

A Call for Interesting WebResourceProviders!

This is an invitation to the CP community to come up with interesting and useful web resource providers. Let your imagination (and coding prowess) flow! Please post your cool WebResourceProvider derived classes at CodeProject.

Share

About the Author

Ravi Bhavnani is an ardent fan of Microsoft technologies who loves building Windows apps, especially PIMs, system utilities, and things that go bump on the Internet. During his career, Ravi has developed expert systems, desktop imaging apps, marketing automation software, EDA tools, a platform to help people find, analyze and understand information, trading software for institutional investors and advanced data visualization solutions. He currently works for a company that provides enterprise workforce management solutions to large clients.

His interests include the .NET framework, reasoning systems, financial analysis and algorithmic trading, NLP, HCI and UI design. Ravi holds a BS in Physics and Math and an MS in Computer Science and was a Microsoft MVP (C++ and C# in 2006 and 2007). He is also the co-inventor of 2 patents on software security and generating data visualization dashboards. His claim to fame is that he crafted CodeProject's "joke" forum post icon.

Ravi's biggest fear is that one day he might actually get a life, although the chances of that happening seem extremely remote.