Dealing With the RSS

Bandwidth Problem">
SourceForge.net and its news cousin Slashdot.org are on the lookout for overly zealous newsreaders as a way to combat bandwidth hogging by RSS. Slashdot.org has an aggressive policy where it will ban newsreaders that check for its feeds more than once every half hour.
SourceForge.net, on the other hand, watches RSS traffic as part of its overall check for abuse. If any IP address is checking too many pages in too short of a time, it could be banned. McGovern declined to offer details on the threshold for banning but said the problem is not RSS itself.

What is needed is better caching of news feeds so that they are not constantly being pulled from the single server of the publisher, he said.

"We like RSS feeds because theyre a great way to get our content out to more sites and get more traffic to our sites," McGovern said.
Winer also suggested that one component of the RSS 2.0 specification already allows better cooperation between publishers and newsreaders. Called "time to live," or ttl, the element lets a content publisher tell a newsreader how often it should check for fresh content, Winer explained.
However, the ttl element is based on the honor system, Winer said. For it to be successful, a newsreader must abide by the time-interval set in the ttl tag. Winer said one way to enforce it could be for a publisher to give newsreaders a period of time, say 60 days, in which to support the interval or face being banned.
While a useful tool, ttl has other limitations, Reinacker said. It only covers one XML syndication specification, RSS 2.0, and not the other RSS varieties or the Atom format.
Another way feed publishers can limit too much polling is the use of HTTP caching headers, where sites can notify readers whether any new content has been published since the last check, Reinacker said. NewsGator, and most major newsreaders, support HTTP caching headers, he said.
NewsGator also sets updating by default at every hour but lets users change the setting. Reinacker said that newsreaders have to be careful about limiting users ability to poll more frequently for updates, or they could undermine the value of RSS. Some enterprises, for example, are using RSS for real-time notifications.
"Publishers are paying a bandwidth cost right now, but in doing so theyre delivering content in a way that hasnt been done before," Reinacker said. "They are able to notify users of changes to their site as opposed to waiting for them to come to their site."
Winer agreed that publishers have to expect to pay some cost for the extra bandwidth of RSS traffic. While he doesnt think that most blogs and publishers are facing an undue bandwidth burden, he said that the full industry does need to discuss a more collaborative solution.
"Its basically when this industry is ready to face these issues that its going to happen," Winer said.
Read more here about some enterprises are eyeing RSS for content distribution.
Until then, perhaps those most effected by RSS bandwidth hogging wont be large Web sites such as MSDN but smaller bloggers who develop a decent following, such as Gary Lawrence Murphy.
Murphy, who publishes the blog Teledyn, last November began noticing traffic spikes from his RSS feed. It reached a point where he was exceeding his ISPs daily bandwidth limit under his hosting plan.
He also noticed that despite using HTTP caching, many of the implementations of newsreaders didnt follow the specification correctly, leading him to both "break" the way his feed delivered its time stamp and to pare it down by removing HTML and limiting posts to 180 characters.
It stemmed the bandwidth tide, but Murphy said it should serve as a warning to other smaller blogs like his.
"Any site that becomes popular is going to be killed by their RSS," Murphy said.
Check out eWEEK.coms Messaging & Collaboration Center at http://messaging.eweek.com for more on IM and other collaboration technologies.

As an online reporter for eWEEK.com, Matt Hicks covers the fast-changing developments in Internet technologies. His coverage includes the growing field of Web conferencing software and services. With eight years as a business and technology journalist, Matt has gained insight into the market strategies of IT vendors as well as the needs of enterprise IT managers. He joined Ziff Davis in 1999 as a staff writer for the former Strategies section of eWEEK, where he wrote in-depth features about corporate strategies for e-business and enterprise software. In 2002, he moved to the News department at the magazine as a senior writer specializing in coverage of database software and enterprise networking. Later that year Matt started a yearlong fellowship in Washington, DC, after being awarded an American Political Science Association Congressional Fellowship for Journalist. As a fellow, he spent nine months working on policy issues, including technology policy, in for a Member of the U.S. House of Representatives. He rejoined Ziff Davis in August 2003 as a reporter dedicated to online coverage for eWEEK.com. Along with Web conferencing, he follows search engines, Web browsers, speech technology and the Internet domain-naming system.